Multimodal Embodied Plan Prediction Augmented with Synthetic Embodied Dialogue
Authors: Aishwarya Padmakumar
Mert Inan
Spandana Gella
Patrick L. Lange
Dilek Hakkani-Tur
Venue: EMNLP 2023 Main (2023)
Keywords: Embodied AI
Embodied Task Completion
Language and Robotics
Plan Prediction
Dialog Simulation
TL;DR: We propose a method to simulate embodied dialogs including natural language utterances and actions in a simulated environment and demonstrate its value in improving plan prediction

Abstract: Embodied task completion is a challenge where an agent in a simulated environment must predict environment actions to complete tasks
based on natural language instructions and ego-centric visual observations. We propose a variant of this problem where the agent predicts
actions at a higher level of abstraction called a plan, which helps make agent actions more interpretable and can be obtained from the appropriate prompting of large language models. We show that multimodal transformer models can outperform language-only models for this problem but fall significantly short of oracle plans. Since collecting human-human dialogues for embodied environments is expensive and time-consuming, we propose a method to synthetically generate such dialogues, which we then use as training data for plan prediction. We demonstrate that multimodal transformer models can attain strong zero-shot performance from our synthetic data, outperforming language-only models trained on human-human data.

Multimodal Embodied Plan Prediction Augmented with Synthetic Embodied Dialogue
Aishwarya Padmakumar
Mert Inan"
Spandana Gella
Amazon padmakua@amazon. com
Northeastern University
Amazon sge sgella@amazon.com com
inan www.mootttttteeennnnnneeeeeeettt
Patrick L. Lange
Dilek Hakkani-Tur
Amazon
University of Illinois TrrbanaChhappaiee
patlange@amazon.ccom
dilekeillinois edu
Abstract
dedicated component for task planning compos- ing sequence of fine-grained motor skills into a more complex task (Chen et al., 2010; Lemaignan et al., 2017; Jiang et al., 2019). In such a sys- tem, the coffee task considered above would likely start by invoking a semantic navigation module to find the mug and a grasping module to pick it up. Some prior work has been embodied AI benchmarks suggesting that more modular mod- els can outperform monolithic models (Min et al., 2021; et al., 2022: Zheng et al., 2022: Min et al.. 2022). However, these do not evaluate and explore the limitations of individual modules.
Embodied task completion is a challenge where an agent a simulated environment must pre- dict environment actions complete tasks based on natural language instructions and ego- centric visual observations. We propose vari- ant of this problem where the agent predicts actions at a higher level of abstraction called a plan, which helps make agent actions more interpretable and can be obtained from the ap- propriate prompting of large language models. We show that multimodal transformer mod- els can outperform language-only models for this problem but fall significantly short of ora- cle plans. Since collecting human-human dia- logues for emhodied environments is expensive time-consuming, we propose method to synthetically generate such dialogues, which we then use as training data for plan predic- tion. We demonstrate that multimodal trans- former models can attain strong zero-shot per- formance from our synthetic data, outperform- ing language-only models trained human- human data.
In this work, we formulate and explore the prob- lem of task planning for embodied task completion. We improve upon existing plan prediction models and demonstrate the potential for improvement by comparing them with human plans. We adapt the Execution from Dialogue History (EDH) bench- mark from the TEACh dataset (Padmakumar al., 2022) to evaluate models at the level of a plan - a sequence of object interaction actions paired with the type of object on which the action needs to be executed which are evaluated using task success upon execution with the aid of a heuristic plan exccution module. Plan prediction is more challenging in TEACh compared other embod- ied Al datasets, tasks can be hierarchical and parameterized, environments are cluttered, and ob- jects may be hidden inside closed receptacles. We evaluate variants of the multimodal Episodic Trans- former (E.T.) model, previously used to directly predict low-level actions in embodied task com- pletion (Shridhar et al., 2020; Padmakumar et al.. 2022) and find that these outperform finetuned language-only baseline.
1 Introduction
Embodied task completion (Shridhar et al., 2020; Padmakumar al., 2022) a challenge where an agent a simulated environment (Kolve et al., 2017: Savva et al., 2019; Chang et al., 2017) is given natural language context in the form of in- structions or dialogue and needs to take actions in the environment to complete desired task, addi- tionally making of egocentric visual observa- tions. This typically requires the agent to predict actions directly executable in the simulated envi- ronment. For example, an action sequence to make coffee could start with actions to move forward two steps, turn left, and pick up mug identified by a semantic segmentation mask. In contrast, physi- cal robot systems tend to be more modular with a
Data collection for embodied Al tasks involving natural language is expensive and time-consuming to collect (Padmakumar et al., 2022), motivating the need for methods that require less human- human data. We develop the first framework for ex-
Contributions from Mert inan and Dilek Hakkani-Tur were provided when they were employed at Amazon.

---

Dialogue History
Low-Level Actions
(Move, Forward), (Tum, Left), (Piciup Potato), (Place, CounterTop), (Pickup, Cupl. (Place. CounterTop). (Pidup, (Move Right), (Move, Right), (Move Right, Right), Right), [ToggleOft Faucet), (Move Rightt). (Place CounterTop)
What shall today?
(Pickup. Polsto), (Plaos, CounterTop), (Pickup, Cupl, (Place, CounterTop), (Piceup, Pot), (TagglaOOt Faucet), (Place, CounterTop!
Plan
Pleese boil potato
there another pot somewheree?
(PickUp, Cup), (Move Right), Right), (Move Right), (Tum Right), (Tum Right), (Plaos, Sinig, (ToggleOn, Faucet), (ToggleOti. Faucet), (PickUp Cup). (Pour. Pot), (Place, Sink), (PickUp. Potl, (Tum Left), (Tum Left), (Move Left), (Place, StoveBurner, (ToggleOn, SloveBumer), (PickUp, Potato), (Place, Pot)
You could byy filing the cup with water and emptying into
(PickUp, (Place, Sink), (ToggleOn, (PickUp Cupl, (Pour, Pot), (Place, Sink), (PickUp. Poti, (Place, StoveBumer), (ioggleOn, StoveBurter), (PickUp, Potato), (Place, Pat)
(ToggleOM
Good thinking! Thank you suggestion.
All done!
Figure I:
This figure depicts example EDH instance from the TEACh dataset with modifications converting action sequence to plan. Models are trained to predict plans instead of low-level action sequences, and a plan execution module identifies navigation and other adjustment steps to ensure effective execution of plan actions.
panding agenda-based dialogue simulation (Schatz- mann and Young, 2009) to multimodal embodied agent setup by augmenting agenda-based dialogue act prediction with rule-based module for identi- fying action sequences in the environment to plete tasks. We demonstrate that the E.T. based models trained only on synthetic data can achieve same models trained on real data, obtaining a zero-shot success rate of 17.09%, which outperforms the full shot suc- cess rate of plans generated by the language-on only baseline at 10.27%.
actions between human annotators role-playing a Commander and Follower collaborating in a sim- ulated home environment to complete household tasks. Only the Commander has access to task information, and only the Follower tions the environment requiring them to com- municate complete the task. An effective Fol- lower must engage in dialogue with the Comman- der, obtain relevant information such as details of the task to be completed and locations of objects, and reason about environment actions that can ac- complish relevant state changes to make progress in the task. We focus the EDH benchmark from the TEACh dataset where given history, well as past actions and image observa- tions, the Follower must predict subsequent actions in the environment to make progress with the task. This is evaluated by comparing environmental state changes arising from gold and predicted action se- quences. We modify the expected prediction from model to be a plan, which we define as quence of object interaction actions paired with the object category of the object they are to be exe- cuted upon An example of the task of boiling a potato is included in Figure 1. Note that in plan prediction, the model nceds to reason about physi- cal state changes that the act of boiling requires
take
about 85% the performance
To summarize, contributions include:
We formulate the problem of plan predic- tion for the TEACh dataset and evaluate language-only baseline, variants of multi- modal transformer model T.), and estab- lish oracle performance on this problem.
dialogue
We are the first to design framework for synthesizing embodied dialogues involving both utterances and environment actions to complete task.
We demonstrate that the synthetic data gener- ated by our framework results in competitive zero-shot performance in our problem.
2
Task Setup
While it is possible to specify more abstract plans, we choose this level of abstraction as the training data can he generated automatically from the TEACh EDH instances.
The TEACh dataset (Padmakumar et al., 2022) is an embodied dialogue dataset consisting of inter-

---

Naxt Task Step Ideritiiicttinn
Taak
Make Coffee
Simulator State
Synthesized Session:
[("objeat I "Mugi (00ggl +00.77 1-00000110000000000
Task
He Hello, what can T help you with today?
LiDirrty 1 -
Prepaze coffee in clean mug.
Next Task Step
I"target"; "Mugl-00 671 10000000000000000000000001000000 84", "prop": ail rty", "desired red valu
Assisted Plan Execution (Execute env actions Simulator State Updates)
Khere can find mug?
can find mug on the countertop near the
Agenda Based Dialog Simulator
Plan
((Piakup, Mag), (Place, Sink)
Nove Forward, Move Forward, Turn Right, Pickup Mug, Place 8 nink Togg on Paucet
Next Dialog
ExscuteStep
InfoCrob Detall
Plan Synthesis
Templated Utterance
Generation
Templated Utterance
You find mug countertop near the aink
Figure 2: Control Flow for Simulating Synthetic Embodied Dialogues with example of making coffee.
placing the potato in a container filled with water, which is then heated using a stove in the example. Other aspects of execution, such as navigating to required objects and fine-grained position adjust- ments, can be carried out by a separate execution module, which, in our case, is heuristic-based but also be learned. This problem is non-trivial in datasets such as TEACh where tasks are param- eterized, and hence highly variable, and diverse initial states can add or remove task steps. This is borne out our experiments by the significant gap between model generated and human plans.
Plan Prediction Models
We adapt the Episodic Transformer (E.T.) model (Pashevich et al., 2021) for plan prediction. This is a multimodal transformer model which, in our case, receives the EDH dialogue history as lan- guage input and egocentric image observations visual input and predicts plan steps consisting of an object interaction action and the type of object the action is taken on?. We obtain training data by filtering EDH action sequences to contain only object interaction actions and train the model as in Padmakumar et al. (2022), where the model receives images and plan steps from the EDH ac- tion history as input and predicts the entire output plan once. At inference time, the last plan step predicted is exccuted, and the input for the next time step updated with an image observation af- ter executing this plan step. We use three variants of this model. :. model as described above.
During inference, at each time step. plan pre- diction model is expected to predict interaction action and the category of the object on which it is be executed. This then executed by one of two possible plan execution modules described section 4. Execution terminates ei- ther when a model predicts special Stop action, reaches a limit of 100 plan steps, or results in 30 simulator execution failures. A plan step may fail execution for variety of reasons. may be infeasi- ble (e.g., trying to pick up cabinet), prerequisite step may not be completed (e.g., the Slice action is only feasible the agent holding knife), or the agent may be poorly positioned (c.g., g., too close to the fridge to open it).
object
E. Hierarchical: T. is modified pass output from the action prediction head input to the object prediction head.
E.T. + Mask: Uses predefined constraints determine whether the predicted action is
2Sce Appendix D for more detailed explanation and a figure.

---

We do this by building an agenda- based dia- logue simulator (Schatzmann and Young, 2009) over dialogue acts relevant to the TEACh dataset combined with the ExecuteStep action. We use subset of the dialogue acts annotated for the TEACh dataset in Gella et al. (2022), focusing feasible to execute on the predicted object, and if not, backs up the action with the highest probability that is feasible.
5.1 Next Task Step Identification
Our dialogue simulation process involves break- ing down a task into task steps corresponding to a single desired object state change, around which dialogue utterances and environment actions are constructed. Tasks in the TEACh dataset are de- fined sets of object properties that must be satis- fied for the task to be complete (Padmakumar al., 2022). The public TEACh simulation wrapper also includes a task checker that, when given a task definition and the current state of the simulator, can provide pending object state changes that need to be accomplished for the task to be considered complete. For example, Figure 2 demonstrates the synthesis of dialogue session related making coffee, which requires a mug in the environment be clean and filled with coffee. As the agent acts in the environment, the task checker examines mugs in the environment and identifies the one closest to completion. The task checker can indicate to the agent the object state changes that still need to be accomplished on this mug; for example, in Figure 2, it identifies a face that needs to be cleaned. This is single Task Step that can be used as a focus for dialogue utterances and environment actions. Once this Task Step is completed, the simulation process proceeds to the next Task Step of filling coffee, after which the task checker will indicate task completion, ending the dialogue simulation.
4 Plan Execution
While we can compare predicted plans with a ground truth plan using surface metrics such as edit distance, we believe a stronger test of predicted plans is executing them in the environment and measuring task success. To do this, pair models with heuristic plan execution modules:
Direct Plan Execution: Given a predicted object interaction action and object type, we use object coordinates from the simulator to identify the closest object of the type use the navigation graph to navigate to it and attempt to execute the action.
Assisted Plan Execution: Direct plan execu- tion can fail for various reasons. For exam- ple, if the sink is full and something needs to be placed in it, the sink needs to be cleared first. Although we train our plan prediction model with data that should enable it to pre- dict such intermediate steps, we wish to ex- plore whether models perform better if they abstract out such details. To do this, ana- lyze common execution failure cases and im- plement set of heuristics, detailed in Ap- pendix C, to increase their success.
In future work, we hope replace these with mod- els for plan executions (Zheng al., 2022).
5.2 Agenda Based Dialogue Simulation
Given the next Task Step, we build a semantic outline for snippet of synthetic dialogue related to this Task Step that includes dialogue acts ex- changing information related to this Task Step and predicting special action ExecuteStep that indicates a transition to predicting environment ac- tions that accomplish this Task Step
5 Synthetic Dialogue Generation
Collecting embodied dialogue examples is ex- pensive and time-consuming (Padmakumar et al., 2022). However, it would be desirable for embod- ied agents to adapt to new tasks without requiring human interaction data. Ideally, given task defini- tion, would like be able to bootstrap model for the task, which can then be further refined using techniques such as reinforcement learning. This work proposes a method to generate synthetic em- bodied dialogues to train an initial model without human interaction data. Our process for simulating synthetic embodied dialogues is outlined in Figure 2 and in the following sections. Additional details are included in Appendix G. We plan to release synthetic data for future research.
on requesting and receiving instructions related the task and how complete it, as well as the lo- cations of objects. Our agenda-based simulator consists of dialogue states, each computed as
"https: / www.:cch.baaaaaaeeeemmmmmmmmmaae com/alexa/tess teach
"Note that using the closest object is a heuristic und can fail. In our experiments, we evaluate two oracle plan prediction methods to quantify the limitations of this.
'Note that human-human dialogue in TEACh is much more free-form and we hope to achieve more versatility in future work.

---

boolean function of 8 binary dialogue features. We pre-define probabilities for sampling dialogue acts, ExecuteStep, and DoNothing actions each state. We then generate a dialogue session by sam- pling a sequence of actions, expanding each dia- logue act into an utterance using templates filled in with task and simulator information, and each ExecuteStep into sequence of environment ac- tions as described in section 5.3. Appendix G in- cludes more details of this process.
Oracle: As an upper bound the success rate obtainable with each of our plan exccution meth- ods, we obtain oracle plans using the ground truth actions in the EDH instance. We filter these action sequences, retaining only object interaction steps and converting object IDs to object types to match the plan representation used by our models.
Oracle with Object IDs (CorefOracle): To fur- ther test the limitations of our plan representation combined with the heuristic of selecting the closest object of a particular type, we produce plans from human action sequences containing object IDs in- stead of object types to avoid ambiguity during plan exccution.
5.3 Plan Synthesis
When we predict that the session should transition from dialogue to environment actions, we use a rule-based system to identify an action sequence in the environment that is likely to accomplish the next Task Step. We hard code plans for each pos- sible object state change, detailed in Appendix G. For example, if the object state change requires an object to be cleaned, the plan will involve mov- ing the object to the sink, turning on and off the tap, picking it up, and pouring out water accumu- lated from cleaning. These hard-coded plans do not account for handling difficulties arising from the current state of the environment, for example, clearing out the sink if it is too full to place the object to be cleaned. Hence, we execute these syn- thesized plans using Assisted Execution (Section 4) to improve success rate at completing task steps using these hard-coded plans.
We additionally include our best zero-shot model and our best model trained on both real and syn- thetic data. These models use synthetic data gen- erated according to the method outlined in Sec- tion using the initial states corresponding to the TEACh train set. The zero-shot model is trained only on synthetic data, and the data-augmented model is trained on combination of real and syn- thetic data.
We evaluate models based on the success rate (SR) and goal condition success rate (GC) as de- fined in the original TEACh paper (Padmakumar et al., 2022). Success rate, which measures the frac- tion of EDH instances for which predicted plans produced all expected object state changes, and GC, which measures the fraction of such object state changes across all instances, were calculated. Since the TEACh test set is not public, we follow the stan- dard protocol proposed in the TEACh codebase of using a standardized division of the original vali- dation sets into validation and test sets called the di- vided validation and divided test sets, each of which is further divided into Seen and Unseen splits.
6 Experiments
6.1 Plan Prediction Models
We evaluate our proposed plan prediction models on the EDH task of the TEACh dataset (Padmaku- mar et al., 2022). We experiment with each of the models in section 3 with each execution method in section 4. Additionally, we evaluate the following baseline and oracle conditions 6.
We present our results in Table 1. For sub- set of these conditions, ference with three random seeds and perform 2- sided Welch t-tests. Allowing for Bonferroni cor- rections over four tests, we find that T. + Mask is trending to be significantly better than the base- line with = 0.0381 on the split and p = 0.0164 on the divided tesoddd_tttteeeeeeeee split. We did not find any statistically significant difference between the T. Hierarchical cal and E.T. Mask models 8.
train and perform in-
Baseline: Our baseline is a language-only BART model (Lewis et al., 2020), finetuned to take in the EDH dialogue history and predicts the entire plan as sequence of tokens that are post-processed for validity and executed as in (Gella et al., 2022).
"We do not compare to TEACh EDH models in this pa- per (Padmakumar et al., 2022) as our execution methods ac- cess information that the TEACh baseline models cannot ac- cess. We do this to ensure that we are evaluating only the process of plan prediction without additional complications arising from navigation and simulator behavior
2 Ptttps: //gitbb.. thub. wttptcoo.ccocc..eccooooootc.tttt aaeaaaleaaaaaaeeaa teach "We did not perform statistical comparisons across all pairs of conditions as it is expensive and time-consuming to run an

---

EDH Plan Divided Val Split
EDH Plan Divided Test Split
Seen
Unseen
Seen
Unseen
Model
Execution
SR
GC
SR
GC
SR
GC
SR
GC
Direct
11.26
13.67
7.51
11.03
7.19
9.62
8.87
9.54
Baseline
Assisted
11.92
17.27
8.91
12.19
9.80
12.30
10.27
12.31
Direct
12.91
16.32
15.58
16.20
15.03
19.52
16.62
15.61
E.T.
Assisted
15.89
20.57
18.74
22.36
16.67
19.96
19.98
27.13
Direct
14.24
15.67
16.23
17.27
14.71
17.97
17.27
20.30
E.T. Hierarchical
Assisted
18.21
20.45
18.09
24.53
17.97
23.67
19.70
25.82
Direct
15.23
22.51
17.81
18.29
16.34
23.84
17.46
18.96
E.T. + Mask
Assisted
18.87
28.99
19.57
27.64
18.95
26.35
20.07
28.33
Best zero shot
Assisted
18.87
17.52
16.23
19.30
15.36
17.17
17.09
16.82
Best Augmented
Assisted
18.87
26.90
19.48
30.41
17.32
26.52
22.32
34.30
Direct
61.92
63.64
55.57
58.48
54.58
53.58
56.77
58.01
Oracle
Assisted
68.87
72.13
61.97
63.07
61.44
62.49
63.21
64.87
Direct
77.81
83.03
70.87
71.87
75.82
79.50
71.90
74.34
CorefOracle
Assisted
80.13
84.50
74.58
77.31
78.43
80.92
76.94
78.30
Table 1: Success rate (SR) and Goal Condition Success Rate (GC) of different models combined with different execution methods the TEACh EDH task. Oracle performances separated as upper bounds the task. Best performance results are bolded for each metric and split in the specific execution method.
Since oracle plans do not obtain 100% success rate, we observe the limitations of our plan exe- cution method, which can only handle 78.43% of unseen test instances even with object coreference resolved (CorefOracle).. We believe this is due to the difficulty in obtaining perfect positioning and placement even with ground truth simulator information, further supported by the gap between direct and assisted execution of oracle plans. We additionally that there is considerable scope for improvement from resolving umbiguity related to which object instance to manipulate, which ac- counts for an improvement of about 17% between Oracle and CorefOracle.
tasks such watering a plant, for which detailed steps are rarely provided in the dialogue, and iden- tifying how much of the task has already been com- pleted. Failures of the E.T. models mainly arise from not learning when to stop, which is a limi- tation of the current inference procedure. Other causes of failure include performing unrelated ma- nipulations easily visible objects or ignoring small objects in favour of larger ones.
On comparing models trained on real data with synthetic data, we find that the zero-shot models perform surprisingly well, outperforming the base- line trained real data and approaching the perfor- mance of the models that have the same architec- ture but are trained on real data. This suggests that when generalizing to new tasks for this application. might be reasonable to train a model purely on synthetic data and expect reasonable performance from interaction with human users.
We observe that while the E.T., E.T. Hierarchical and E. Mask models substan- tially improve over the baseline, there is also a considerable gap between them and Oracle which uses the same plan representation, which demon- strates that there is considerable scope for improve- ment in understanding the details of the task to be completed from dialogue, and reasoning about actions to take to achieve the corresponding state changes. Qualitatively, we find that multimodal input's main benefits are breaking down complex
6.2 Zero Shot Training Ablation
We perform further ablations to identify how zero- shot model performance varies according to data size in Table 2. While see a trend towards improvement in performance with increasing data size, and the best results are obtained at higher data sizes, the improvement is not perfectly con-
inference with enough random seeds to allow for Bonferroni corrections as the number of tests grows.

---

EDH Plan Divided Val Split
EDH Plan Divided Test Split
Seen
Unseen
Seen
Unseen
Train
Model
Set
SR
GC
SR
GC
SR
GC
SR
GC
Size
E.T.
Ix
12.25
9.50
13.17
9.33
14.05
11.04
13.82
9.36
E.T.
2x
15.89
13.95
14.47
13.58
14.38
12.03
15.97
11.85
E.T.
3x
17.88
11.94
14.84
10.61
13.07
12.14
16.06
11.22
E.T.
4x
16.89
11.28
13.45
13.07
14.05
11.26
14.94
13.54
E.T. Hierarchical
1x
15.56
10.15
13.36
10.77
11.76
9.73
12.32
9.74
E.T. Hierarchical
2x
13.89
11.16
14.19
16.68
11.44
13.50
14.47
18.53
E.T. Hierarchical
3x
18.87
17.52
16.23
19.30
15.36
17.17
17.09
16.82
E.T. Hierarchical
4x
18.21
10.45
14.75
10.14
14.05
11.97
15.69
11.21
E.T. Mask
1x
13.58
9.98
13.82
10.90
13.73
12.30
13.35
9.83
E.T. Mask 2x
14.24
16.39
15.49
16.08
14.38
12.41
15.59
14.94
E.T T. Mask
3x
16.89
12.05
14.10
12.45
12.09
11.21
15.03
12.09
E. T. Mask
4x
15.23
14.37
14.66
15.20
14.38
13.45
14.29
14.98
We explore how zero shot model performance varies with the size of the synthetic training set (as a proportion of the size of the human-human training data). This table reports success rate (SR) and goal condition success rate (GC) on the EDH divided test split with assisted execution.
Table 2:
EDH Plan Divided Val Split Seen
EDH Plan Divided Test Split Seen
Unseen
Unseen
Syn-
thetic
Model
SR
GC
SR
GC
SR
GC
SR
GC
Data
Size
T.T.
Ix
18.21
17.52
17.90
29.12
17.65
23.67
21.38
31.63
E.T.
2x
17.55
25.00
19.02
24.99
18.63
23.51
19.70
24.60
T.
4x
17.22
26.84
18.37
26.12
17.97
25.26
19.79
26.01
T.T. Hierarchical
Ix
16.89
24.70
18.74
25.73
18.30
23.46
20.45
25.65
E.T. Hierarchical
2x
18.54
19.48
18.74
23.55
20.92
28.05
19.61
27.91
E.T. Hierarchical
14.90
20.31
17.90
24.22
17.97
26.95
20.17
24.30
E.T. Mask
1x 18.87
26.90
19.48
30.41
17.32
26.52
22.32
34.30
T.T. Mask
2x
16.89
32.84
19.02
30.66
18.63
27.99
20.45
29.98
E. Mask
4x
18.21
27.38
18.65
28.01
20.26
27.88
19.79
27.28
Table 3: We explore how synthetic data size impacts model performance. Models here trained combination the cuuman-humant TEACh training data and synthetic data of the size indicated in column Synthetic Data Size (as proportion of the size of the human-human training data). This table reports rate (SR) and goal condition success rate (GC) on the EDH divided test split with assisted execution.
sistent with data size. We believe this sub-linear scaling with the increase in data due to a combi- nation of limitations in the range TEACh initial states in which our synthetic plan execution method can generate valid action sequence successfully.
limitations of the diversity in synthetic templates and limitations in the ability of the E. model model the TEACh plan prediction task effectively. We believe that through engineering improvements or a learned Reinforcement Learning policy, we can

---

improve the range of initial states covered; with the assistance of better LLMs, we can produce more diverse synthetic dialogue, and using neural SLAM models, we can overcome the limitations of what a particular model can learn. We plan to explore these directions in future work.
Zheng et al., 2022; Zhang al., 2022), which we compare to baseline.
User simulation in the dia- logue community originally consisted of rule-based systems designed using linguistic knowledge to en- able finetuning dialogue systems individual user preferences and subsequently evolved into trainable probabilistic models that can be used to bootstrap a dialogue system in the initial development phase and further finetune it through reinforcement learn- ing (Schatzmann et al., 2006; Young et al., 2013). 3). A common method for building user simulators is agenda- basdd simulation (Schatzmann and Young, 2009), which uses predefined set of transition probabilities between dialogue acts in combina- tion with goal information to sample subsequent dialogue acts. This has been used to bootstrap a range of dialogue models ranging from probabilis- tic POMDP models (Schatzmann et al., 2007) and text-to-SQL models (Liu et al., 2022) to hierar- chical deep reinforcement learning methods (Peng et al., 2017). In this work, we augment a standard grgenda-basede simulator with an additional intent to determine transitions to acting in the environment to generate situated dialogues. Another popular paradigm for dialogue simulation is to develop two models one for the user and one for the agent side and train them simultaneously using reinforcement learning (Liu and Lane. 2017: Shah et al., 2018 8). This has also been used for some multimodal dia- logue domains (Das et al., 2017), but choose not to adapt it in domain as the time to generate single dialogue is higher in situated applications due the latency from executing environment ac- tions in the simulator.
Dialogue Simulation:
Note that we did not perform other hyperpa- rameter tuning for models in Table besides the changes in training data. The best condition, T.T. Hierarchical with a synthetic training set of size 3x as large as the human-human training set from Table 2 has been reported in Table 1 for comparison with models trained on human-humen data. We find that this model trained purely on synthetic data obtains success rate of 17.09% the divided unseen test split, outperforming the language-only baseline trained on real data at 8,87% and approaches close to the performance E. Hierarchical trained on human-human data at 19.70%.
6.3 Data Augmentation Training Ablation
In Table 3, we ablate different sizes of synthetic data when used in data augmentation. We find that when both real and synthetic data are included, larger sizes of the synthetic training set are less beneficial than when trained only on synthetic data, The best condition E. T. + Mask with a combina- tion of fuuman-human and synthetic data of equal size has been included in Table as "Best Aug- mented." We find that at 22.32% on the divided unseen test split, this slightly outperforms the same model condition trained on 20.07%, and is much stronger than the best condi- tion using only synthetic data at 17.09%.
data at
7 Related Work
Conclusions and Future Work
Interactive systems physi- cal robots typically have a modular structure in which task plamning plays a significant role (Chen et al., 2010; Khandelwal et al., 2017: Peshkin et al., 2001). In simulated environments, Logeswaran et al. (2022) propose a language-only finetuned GPT-2 model for task planning on ALFRED Some end-to-end ALFRED models also have task plan- ning component (Min et al., 2021; Jia al., 2022; Blukis al., 2022). However, this is sim- pler dataset where task planning can be cast a 7-way classification problem. Prior work has also explored language-only task planning using fine- tuned BART models in TEACh (Gella et al., 2022;
Task Planning:
We develop a model for multi-mode plan predic- tion for the TEACh dataset using the Episodic Transformer architecture and evaluate end-to-end performance on the TEACh EDH task in conjunc- tion with heuristic plan execution modules. We additionally experiment with training this model us- ing only synthetic data generated using an agenda- based dialogue simulator combined with an en- vironment action generator that uses combina- tion of rules and simulator information to iden- tfy sequences of actions in the environment that can make progress with the task. We find that our E.T. plan prediction models outperform a BART baseline, even when BART is finetuned on human-

---

human embodied dialogue data but E.T. is fine- tuned only on synthetic data, suggesting that our dialogue simulation approach viable alterna- tive to expensive data collection for bootstrapping an embodied task completion model on new tasks. We also find a considerable performance gap be- tween models and humans in plan prediction that cannot be easily closed by techniques such as data augmentation.
representations.
In this work, we describe a method to generate synthetic dialogues for embodied task completion by augmenting an agenda-based dialogue simula- tor with modules that break up an embodied Al task into individual object state changes, and rule- based methods identify actions that complete them. Additionally, we currently manually define transition probabilities between dialogue acts for simulation, which would also need to be modified for new dataset. While our approach is effec- tive at bootstrapping plan prediction models with- out any human-human interaction data, it requires simulator-specific enginecring, particularly when defining heuristics for assisted plan execution. Ad- ditionally, for both plan prediction inference and dialogue simulation, every action must be executed in the simulator. This results in considerable com- pute time, as discussed in the appendix. While dialogue simulation does not require the use of a GPU, the plan prediction models do both for train- ing and inference. We have additionally found that the Episodic Transformer models used in the paper show noticeable variance in performance when trained with the same hyperparameters but with dif- ferent random seeds. We attempt to strengthen our conclusions by training models with multiple seeds for statistical analysis where it is beneficial, but we would also like to highlight the development of models with less variance as an important direction for future research.
The recent success of large language models in a wide variety of structured prediction tasks, in- cluding robotic planning tasks similar to ours, is a promising future direction for our work. Some pre- liminary attempts prompting large language models indicate that it is non-trivial design an appro- priate representation of the dialogue history and plans for LLMs to guess at the remaining objects required for subsequent actions effectively. Addi- tionally, while it is likely that recent long context LLMs can likely take in a significant amount of state information of the environment, further work is required to determine the best way to provide this in environments such as A12-THOR, especially if a proposed method much scale up more realistic home environments with larger numbers of objects. It is likely also possible to improve the diversity of the generated simulated data by paraphrasing generated utterances using LLMs.
9 Limitations
In this paper, we explore the problem of plan pre- diction for embodied task completion, and conduct our experiments in the TEACh dataset, which is set in the AI2-THOR simulator. While we hypoth- esize that models developed for plan prediction will transfer better to physical robots as they align better with levels of abstraction at which robotics systems currently implemented, further experi- mentation is needed to evaluate such transferability, In the short term, such experiments will likely need to work on problems with a simpler action space as some of the Follower actions supported A12- THOR, such as slicing, are not supported in most robots available currently. Additionally, it would be beneficial to test plan prediction models and our dialogue simulation method similar tasks set in other simulators. This is a direction we plan to pursue in future work. We believe this beyond the scope of this publication due to the consider- able engincering effort involved in adapting models across different embodied AI simulators and task
Finally, our work currently limited to English as we are not familiar with datasets in other lan- guages that provide language instructions for tasks that require complex reasoning over multi-step ac- tion sequences.
10 Ethics Statement
This work is part of a broader tradition of build- ing natural language interfaces to control vari- ous devices. Natural language interfaces such as language- -aaged search and intelligent personal as- sistants provide convenience and have the potential to make multiple forms of technology ranging from mobile phones and computers, as well as robots or other machines such as ATMs self-checkout counters more accessible and less intimidating users who are unfamiliar or uncomfortable with other interfaces on such devices such as command shells, nutton-basedd interfaces or changing visual user interfaces. Spoken language interfaces can

---

also be used make such devices more accessi- ble for the visually impaired or users who have difficulty with fine motor control.
Sugato Bagchi, Gautam Biswas, and Kazuhiko Kawa- mura. 1996. Interactive task planning under uncer- tainty and goal changes. Robotics and Autonomous Systems, 181-2))157-167
User trust in such interfaces is essential. De- pending on the circumstances, therefore, some con- siderations to keep in mind are: (1) whether the collection of personal data benefits the user: (2) whether the collection of personal data is transpar- ent the user; (3) whether the user understands whether and how they can control the collection of personal data; and (4) whether the user has the ability to elect whether to use such interfaces, opt into or out of the collection of certain personal data, access and update certain personal data, and delete certain personal data. Additionally, safeguards may be required as embodied agents become capable of interacting with arbitrary objects in the world reduce the likelihood of accidents or malicious misuse.
Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Mano- lis Savva, Alexander Toshev, and Erik Wijmans. 2020. Objectnav revisited: On evaluation of em- bodied agents navigating to objects. arXiv preprint arXiv:2006.132 13171.
Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. 2022. A persistent spatial semantic representation for high-level natural language instruc- tion execution. In Conference on Robot Learning. pages 706-717. PMLR.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Mat- terport3d: Learning from rgb-d data in indoor envi- ronments. International Conference on Vision (3DV).
In this work, we also experiment using simu- lated embodied dialogue data to train models. On one hand, the use of simulated data can limit the collection of personal data. On the other hand, sim- ulators may not be designed to represent the full range of user behavior and may perform better for some users than others.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. Touchdown: Nat- ural language navigation and spatial reasoning in visual street environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 22538-12547,
Xiaoping Chen, Jianmin Ji. Jiehui Jiang, Guoqiang Jin, Feng Wang, and Jiongkun Xie. 2010. Developing high-level cognitive functions for service robots. In AAMAS, volume 10, pages 989-996.
References
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. 2022. Do as can, not as i say: Ground- ing language in robotic affordances. arXiv preprint arXiv:2204.01691.
Abhishek Das, Satwik Kottur, José MF Moura. Stefan Lee, and Dhruv Batra. 2017. Learning cooperative visual dialog agents with deep reinforcement learning. In Proceedings of the IEEE international conference on computer vision, pages 2951-29600
Richard E Fikes and Nils J Nilsson. 1971. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence, 283-4):189- 208.
Ahmed Al-Moadhen, Renxi Qiu, Michael Packianather, Ze Ji, and Rossi Setchi. 2013. Integrating robot task planner with common-sente sense knowledge base to im- prove the efficiency of planning. Procedia Computer Science, 22:211 1-220. 17th International Conference Knowledge Based and Intelligent Information and Engineering Systems- KES2013.
Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf. James Traer, Julian De Freitas, Jonas Ku- bilius. Abhishek Bhandwaldar, Nick Haber, Megumi Sano, et al. 2020. Threedworld: A platform for interactive multi-modal physical simulation. arXiv preprint arXiv:2007.0449)
Peter Anderson, Angel Chang. Devendra Singh Chap- lot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mot- taghi, Manolis Savva, et al. 2018a. On evalua- tion of embodied navigation agents. arXiv preprint arXiv:1807.066 1807.0677.
Michael Gelfond and Yulia Kahl. 2014. Knowledge representation, reasoning, and the design of intelli- gent agents: The answer-set programming approach. Cambridge University Press.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018b. Vision- and-Language Navigation: Interpreting visually- grounded navigation instructions in real environ- ments. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR).
Spandana Gella, Aishwarya Padmakumar, Patrick Lange, and Dilek Hakkani-Tur 2022. Dialog Acts for Task-Driven Embodied Agents. In Proceedings of the 23nd Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDial), pages 111-123.

---

N GGoalann Rosen, GD Konidaris, and S Tellex, 2020. Simultaneously learning transferable symbols and language groundings from perceptual data for instruc- tion following. Robotics: Science and Systems XVI.
the Association for Computational Linguistics, pages 7871-7880.
Nir Lipovetzky. 2014. Structure and inference in classi- cal planning. Lulu. com.
Peter E Hart, Nils J Nilsson, and Bertram Raphael. 1968. A formal basis for the heuristic determination of min- imum cost paths. IEEE transactions on Systems Sci- ence and Cybernetics, 4(2): 100-0700
Bing Liu and Ian Lane. 2017. Iterative policy learning in end-to-end trainable task-oriente, neural dialog models. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 482-
Malte Helmert. 2004, A planning heuristic based on causal graph analysis. In ICAPS, volume 16, pages 161-170.
489. IEEE.
Qi Liu, Zihuiwen Ye, Tao Yu, Phil Blunsom, and Linfeng Song. 2022. Augmenting multi-turn text- to-sql datasets with self-play. arXiv preprint arXiv:2210.1200
Zhiwei Jia, Kaixiang Lin, Yizhou Zhao, Qiaozi Gao, Govind Thattai, and Gaurav Sukhatme. 2022. Lear- ing to act with affordance-ane aarerre multimodal neural slam. arXiv preprint (rxii:2201.09800
Lajanugen Logeswaran, Yao Fu, Moontae Lee, and Honglak Lee. 2022. Few-shot subgoal planning with language models. arXiv preprint 1424
Yugian Jiang, Nick Walker, Justin Hart, and Peter Stone. 2019. Open-world reasoning for service robots. In Proceedings of the International Conference on Au- tomated Planning and Scheduling, volume 29, pages 723-733.
Drew V McDermott. 1996. A heuristic estimator for means-ends analysis in planning. In AIPS, vol- ume 96, pages 142-149.
So Yeon Min, Devendra Singh Chaplot, Pradeep Kumar Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov. 2021. Film: modular methods. In International Conference on Learning Reprexentations.
Yash Kant, Arun Ramachandran, Sriram Yenamandra, Igor Gilitschenski, Dhruv Batra, Andrew Szot, and Harsh Agrawal. 2022. Housekeep: Tidying virtual households using commonsense reasoning. arXiv preprint arXiv:2205.1071 10712.
Following instructions in language with
So Yeon Min, Hao Zhu, Ruslan Salakhutdinov, and Yonatan Bisk. 2022. Don't copy the teacher: Data and model challenges in embodied dialogue. arXiv preprint 10.004433
Piyush Khandelwal, Shiqi Zhang, Jivko Sinapov, Mat- teo Leonetti, Jesse Thomason, Fangkai Yang. Ilaria Gori, Maxwell Svetlik, Priyanka Khante, Vladimir Lifschitz, et al. 2017. Bwibots: A platform for bridg- ing the gap between ai and human-robot interaction research. The International Journal of Robotics Re- search, 36(5-7):635-65)
Anjali Narayan- Chen, Prashant Jayannavar, and Ju- lia Hockenmaier. 2019. Collaborative dialogue in minecraft. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 5405-5415.
Hyounghun Kim, Abhaysinh Zala, Graham Burri, Hao Tan, and Mohit Bansal. 2020. Arramon: joint navigation aavigation-assembly instruction interpretation task in dynamic environments. In Findings of EMNLP.
Aishwarya Padmakumar, Jesse Thomason, Ayush Shri- vastava, Patrick Lange, Anjali Narayan-Chen, Span- dana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani- Tur. 2022. Teach: Task-driven em- bodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36. pages 2017-2025.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli Van- derBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. 2017, Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv: 1712.05444.
Alexander Pashevich, Cordelia Schmid, and Chen Sun. 2021. Episodic transformer for vision-and-language navigation. In Proceedings of the IEEE/CVF Interna- tional Conference Computer Vision, pages 15942- 15952.
George Konidaris, Leslie Pack Kaelbling, and Tomas Mozano-Perez. 2018 From skills to symbols: Learn- ing symbolic representations for abstract high-level planning. J. Artif. Int. Res., 61(1):215-28)
Séverin Lemaignan, Mathieu Warnier, E Akin Sisbot, Aurélie Clodic, and Rachid Alami. 2017. Artificial cognition for social human-robot interaction: An implementation. Artificial Intelligence, 247:45-69,
Baolin Peng. Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, and Kam-Fai Wong. 2017. Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning. arXiv preprint arXiv:/704 1704.03084.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sgquence-t--squeeee ural language generation. translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of
Michael A Peshkin, J Edward Colgate. Wit Wanna- suphoprasit, Carl A Moore, R Brent Gillespie, and Prasad Akella. 2001. Cobot architecture. IEEE Transactions on Robotics and Automation, 17(4))377- 390.
pre-training for nat-

---

Caroline Ponzoni Carvalho Chanel, Alexandre Albore, Jorrit T'hooft, Charles Lesire, and Florent Teichteil- Königsbuch. 2019. Ample: an anytime planning and execution framework for dynamic and uncertain problems in robotics. Autonomous Robots, 43(1):37- 62.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2020. Vision-and-dialle and-dialog navi- gation. In Conference on Robot Learning, pages 394-406. PMLR.
Mare Toussaint and Christian Goerick. 2007. Proba- bilistie inference for structured planning in robotics. In 2007 IEEE/RSJ International Conference on Intel- ligent Robots and Systems, pages 3073- IEEE
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. 2018. Virtualhome: Simulating houschold activitics via programs, In Proceedings of the IEEE Confer- ence Computer Vision and Pultern Recognition, pages 8494-8502.
Claudia Yan, Dipendra Misra, Andrew Bennnett, Aaron Walsman, Yonatan Bisk, and Yoav Artzi 2018, Chalet: Cornell house agent learning environment.
Silvia Richter and Matthias Westphal. 2010. The lama planner: Guiding cost-based anytime planning with landmarks. Journal of Artificial Intelligence Re- search, 39:127-177.
Steve Young, Milica Gašić, Blaise Thomson, and Ja- son D Williams. 2013. Pomdp-based statistical spo- ken dialog systems: A review. Proceedings of the IEEE, 101(5):1100-117
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub. Jia Liu, Vladlen Koltun, Jitendra Malik, et al. 2019. Habitat: A platform for embodied research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9339-9337.
Yichi Zhang, Jianing Yang, Jiayi Pan, Shane Storks, Nikhil Devraj, Ziqiao Ma. Keunwoo Peter Yu, Yuwei Bao, and Joyce Chai. 2022. Danli: Deliberative agent for following natural language instructions. arXiv preprint arXiv:2210.1244
Kaizhi Zheng, Kaiwen Zhou, Jing Gu, Yue Fan, Jialu Wang, Zonglin Li, Xuchai He, and Xin Eric Wang. 2022. Jarvis: A neuro-symbolic commonsense reasoning framework for conversational embodied agents. arXiv preprint arXiv:2208. 13266.
Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda- beaedd user simulation for bootstrapping a POMDP dialogue sys- tem. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Compan- Rochester,
A Licensing and Responsible Use
In this section, we discuss the licensing and usage of existing scientific artifacts in this paper:
ion Volume, Short Papers, pages New York. Association for Computational Linguis- tics.
TEACh dataset: The TEACh dataset re- leased under a CDLA-Sharing V 1.0 license, with images released under Apache 2.0 and code under an MIT license. We believe our us- age does not violate any of these license terms. We do not redistribute any of these as part of our work as they are publicly available. The TEACh dataset was created to study models for translating natural language instructions combined with egocentric visual observations into action sequences. Our use case is a sub- task of this intended use case. We also believe using the TEACh code to generate synthetic dialogue sessions is consistent with this goal.
Jost Schatzmann, Karl Weilhammer, Matt Stuttle, and Steve Young. 2006. A survey of statistical user simu- lation techniques for reinfoncement-leaming of dia- logue management strategies. The knowledge engi- neering review, 21(2):77-126
Jost Schatzmann and Steve Young. 2009. The hidden agenda user simulation model. IEEÉ transactions on audio, speech, and language processing, 17(4):733- 747.
Pararth Shah, Dilek Hakkani- -ürr Gokhan Tür, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry Heck. 2018. Building a conversational agent ovenight with dialogue self- play. arXiv preprint arXiv: 180.0487...
Mohit Shridbar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zeulemoyer, and Dieter Fox. 2020. Alfred: A bench- mark for interpreting grounded instructions for ev- eryday tasks. In Proceedings of the IEEE/CVF con- ference computer vision and pattern recognition, pages 10740-10749.
A12-THOR simulator: The A12-THOR simu- lator (Kolve et al., 2017) is necessary to use the TEACh dataset and generate synthetic di- alogue sessions. We believe
usage of A12- THOR does not violate the license. We plan release our data under the CDLA-Sharing V 1.0 license, with images released under Apache 2.0, which we believe will be con- sistent with AI2-THOR and TEACh license
Alane Suhr, Claudia Yan. Jacob Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi 2019. Executing instructions in situated collaborative interactions. arXiv preprint arXiv:1910.03655.

---

terms. We believe our work is consistent with the intended usage of A12-THOR, which is to further embodied Al research the household domain.
representations used in planners (Fikes and Nils- son, 1971; McDermott, 1996; Gelfond and Kahl, 2014; Konidaris et al., 2018; Gopalan et al., 2020), search algorithms (Hart et al., 1968; Helmert, 2004; Richter and Westphal, 2010) and handling uncer- tainty via probabilistic methods (Toussaint and Goerick, 2007; Bagchi et al., 1996; Ponzoni Car- valho Chanel et al., 2019). More recent work has focused on expanding beyond fully defined world representations by expanding to use com- mon sense (AI-Moadhen et al., 2013) and open worlds (Jiang et al., 2019). Some interesting efforts in this direction use Large Language Models to perform planning (Ahn et al., 2022).
BART model: We finetune BART (Lewis et al., 2020) as baseline for our task, which is released under the Apache 2.0 license. We believe our usage is consistent with the license and do not intend to redistribute it. We believe that our use is consistent with the intended usage of BART as a general purpose sequence to sequence language model.
Episodic Transformer Model: We use the Episodic Transformer model (Pashevich et al., 2021), which released under an MIT Li- cense, for plan prediction with some architec- tural modifications. We believe our usage is consistent with the intent of this model, which was designed for very similar embodied AT application in similar dataset.
Embodied AI Tasks in Simulation Simulated environments (Kolve et al., 2017; Savva et al., 2019; Puig et al., 2018; Chang et al., 2017; Yan et al., 2018) have been used over recent years plore the efficacy of deep learning methods that di- rectly use egocentric visual observations instead of data from expensive sensors. While there a chal- lenge in transferring results from simulated to real environments, simulated environments are more accessible, less expensive, and allow for the testing of technologies that may not be sufficiently safe for use in the real world (Savva et al., 2019). Addition- ally, while simulated environments can be used for tasks that do require the use of language (An- derson al., 2018 8a; Batra et al., 2020; Gan et al., 2020; Kant et al., 2022), they play a particularly valuable role in developing language understand- ing and reasoning capabilities over actions that are currently difficult for physical robots to com- plete, but we hope it will become a reality in the future (Kolve et al., 2017). Much of the work in lan- guage understanding for embodied AI happens us- ing vision and language navigation, where an agent must learn to navigate through previously un- seen environment purely based on natural language route instructions (Anderson et al., 2018b; Chen et al., 2019; Thomason et al., 2020). Embodied task completion additionally requires performing object manipulation actions (Shridhar et al., 2020; Pad- makumar et al., 2022: Suhr al., 2019; Kim al., 2020; Narayan- -hhen al., 2019). In this work. we specifically focus on the TEACh dataset (Pad- makumar et al., 2022) as it involves more complex tasks that require non-trivial task planning.
The TEACh dataset was manually inspected by the original authors to remove identifying informa- tion and offensive utterances (Padmakumar et al., 2022). Since our generated data is templated, we do not believe this can contain personally identifying information or offensive utterances. Our synthetic dialogues are set in the TEACh environment and only cover the tasks listed in the TEACh dataset. The dialogues are in English and are mainly in- tended to cover requesting and informing of task steps (in the form of object state changes) and loca- tions of objects. They support a limited breakage of strict turn-taking by allowing an agent to not generate an utterance at some time steps in the generation procedure.
B Additional Related Work
Task Planning on Physical Robots Task plan- ning has long been a standard component of physi- cal robot architectures (Chen et al., 2010), particu- larly with general purpose service robots (Khandel- wal et al., 2017; Peshkin et al., 2001). Classical task planners include a symbolic representation of the state of the world, a goal, and skills the robot can execute. They are expected find sequence of skills that, when executed, will transform the world into the goal state, typically using heuristic search algorithms (Lipovetzky, 2014). Over the years, research in planning has improved the symbolic

---

C Assisted Plan Execution
Place
Place
This section outlines the full set of heuristics in- volved in assisted plan execution:
AcBiT
For all actions, if the target object property change is already complete, do nothing to avoid execution failure.
Tamporal Encocing
Pickup: If the object is inside receptacle (container), open the receptacle. After pickup. if a receptacle was opened, close it.
Translo Layere
FC Layer
Lookup Thble
Ploue:
Plaze
Pootanal Enooding
2xx Connolttion Laper
Lookup Taole
Restin to Bacesore
Place: If the target receptacle is recep- tacle, take out and place it on the counter first (for example, if we need to place some- thing on a plate that is inside a drawer). the target receptacle needs to be opened, open it and close it after placement (for example, drawer microwave needs to be opened to place something inside). If placement at- tempt fails, try removing the existing contents of the receptacle one by one make more space.
p3000
Dislogus History
Image Obsorvatios
Figure 3: This depicts the architecture for the T.. based models. The basic E.T. model does not have connection hetween action and object heads, but E.T. Hierarchical does. There three main in- put processing components (dialogue history, image observation, and previous plan actions). These are then input into the positional and temporal encodings and transformer layers to develop the next feasible set of high-level plan actions.
Open, Close: Toggle the target object relevant (for example, microwaves need to be turned off to open them).
a dropout of 0.3 to obtain feature maps of 512 x These are then fed into two convolutional layers with 256 and 64 filters of size 1 x 1, respectively, and mapped using a fully connected layer to size 768.
ToggleOn, ToggleOff: If the target is open, close it first (for example, microwaves need to be closed to turn them on).
The language input is tokenized using revtok and encoded using two transformer layers with 12 attention heads and an embedding size of 768, which are trained from scratch. The language and visual encodings are concatenated and passed through two multimodal transformer layers with 12 attention heads, which are also trained from scratch in our experiments.
Slice: If the target is in receptacle, move it to the counter first.
Additionally, we also attempt position adjustments increase the chance of success.
E.T. model
This section discusses the Episodic Transformer (E.T.) Model (Pashevich et al., 2021) along with our modifications in greater detail. For conve- nience, we include a diagram of the model in Fig- ure 3. The T. model receives language (in our case, EDH dialogue history) and egocentric image observations of size 900 x 900 as input. Visual observations are first resized to 224 x 244 and then encoded using visual encoder that is based Faster R-CNN model trained on 325K frames of expert demonstrations from the ALFRED train fold (which comprises seen splits of TEACh ) and not finetuned any of our experiments. This encoder average-pood pools ResNet features four times and adds
To predict actions and objects, the final embed- ding corresponding to each image from the in- put is projected using fully connected layers and then passed to two independent sofumax prediction heads over the action and object space, respectively. At inference time, the last action, which is not a padding token, is identified and used for prediction, along with the object, at the same time step. The E.T. cterarchical model connects the two pre- diction heads by passing the output of the action head as additional input to the object head. The E.T. + Mask model examines the action-object
Phttpss//bppp.or//ppp//ttt//ee

---

pair predicted inference time, and if they are found to be incompatible, replaces the action with the action of highest probability with the predicted object. The object is assumed to be the more re- liable prediction as more likely to be directly visible.
GPU, The training was done using gradient ac- cumulation across 4 GPUs of p3.8xlarge AWS EC2 instance. We use the A optimizer with 81 - 0.9, $23 - 0.99, c - 1e08 and weight de- cay of 0.01. We use learning rate of 5e05 with a linear warmup over 500 steps. The BART model can be finetuned in under an hour using all GPUs of a p3.8xlarge AWS EC2 instance. We first per- formed inference the BART model and saved the predicted plans to file before separately exe- cuting them in the A12-THOR simulator. This process can also be completed in under an hour. Executing stored plans either in the case of the BART model or the oracle conditions took about 2.5 hours using 3 GPUs of a p3.8xlarge AWS EC2 instance for the combined divided_val_sse and divided_ tes___ests_e__e_e splits and about 8 hours for the combined di divided_val_unee and divided_test_urgg splits.
E Plan Prediction Model Hyperparameters
For training the E.T., E.T. + Mask and E.T. Hierarchical methods, we retained hyperparam- eters from the original TEACh paper (Padmakumar al., 2022), except the batch size, without further hyperparameter tuning, and used the largest batch size that could fit in a single GPU of p3.8xlarge AWS EC2 instance. Note that the following hyper- parameters were kept constant for all experiments reported in this paper, and results in different tables arise only from changes in training data, model choice (between T. T.T. Hierarchical and T. Mask), and plan execution method. We used the AdamW optimizer with 0.33 weight decay with learning rate of le for the first 10 epochs and le 5 for the last 10 epochs. We trained all mod- els for 20 epochs with a batch size of and reported results from the final epoch. We replace sampling with rotation permutations of our training dataset per epoch, ensuring that every training example is seen exactly once in our dataset. For the language decoder in the transformer, we use a dropout of 0.1, and for the encoder, we use a dropout of 0. The different T. models required hours for preprocessing (extracting image features using the ResNet-50 backbone) and about hours per model for training using 4 GPUs of a p3.8xlarge AWS EC2 instance. At inference time, we could use maximum of GPUs for inference required by the simulator. When using 3 GPUs of a p3.8xlarge AWS EC2 instance, E.T. models took about 11 hours to complete inference jointly on the divided_ sateeedvvl__seee val_seen and splits and about 35 hours to complete infer- ence jointly on the divided_val. _add__aa__nneen and divided_ tel__ttstt_sssee is due to the size of the various splits.
Data Statistics
The number of games and EDH instances in the TEACh data splits and batches of synthetic data used in this paper are included in Table
Number of EDH instances
Number of games
Split
TEACh train
3121
3895
TEACh
83
302
divided_val val_seen
TEACh
309
1078
divided_var Lac__val__useee unseen
TEACh
306
divided_test_seen
TEACh
GPU was
303
1071
divided_test rest_unseen
Synthetic 1x
1587
2034
Synthetic 2x
3272
3875
Synthetic 3x
4952
7201
seatedd_est_seen
Synthetic 4x
6360
9284
Table 4: Data statistics in TEACh and synthetic data splits.
splits. The time difference
For the baseline BART model, we retain hy- perparameters from the model presented in (Gella al., 2022). We take the pretrained BART-base model from the Huggingface library and finc- tune for 20 epochs using batch size of 2 per
G Dialogue Simulation Details
This section provides a more detailed description of the dialogue simulation process. Dialogue sim- ulation begins by sampling which agent starts the interaction the Commander or the Follower, each
10 "hatss//hugginggfacc.od

---

with 50% While is possible create new initial states, we iterate over each initial state in the train split of the TEACh dataset, each of which is as- sociated with a task to be completed in that state. For each agent, the Commander and the Follower we maintain state for the agent that is factored into binary state features. We then use predefined probabilities for sampling different dialogue acts in each state and alternate taking turns between the two agents. In this implementation, we use the following dialogue acts defined in Gella al. (2022):
The state features used are:
di dialogue_stadl sttuuessttttdd: Agents start with this feature set to False, indicating that the agent is in the initial state before any dialogue has taken place and must initiate dialogue. It gets set to True once an initial utterance has been exchanged.
goal_ _wc.ommiccateeeee This feature is False initially and to True after an Instruction utterance has been sent from the Commander the Follower communicating the high level task.
RequestForInstruction
cur jur_step_requestte This feature is False initially, gets set to True when the Follower sends RequestStep action to the Comman- der and reset to False when environment ac- tions are exccuted.
Instruction
Acknowledge
cur cur_step_sent: This feature is False ini- tially, gets set to True when the Commander sends the current step the Follower through a Step dialogue act and reset False when environment actions are executed.
FeedbackPositive
FeedbackNegative
However, we use both FeedbackPositive and FeedbackNegativv only to end the dialogue cither as a success or failure respectively. We addition- ally divide the Instruction dialogue act into two sub-types for convenience:
This feature is False initially, gets set to True when the Fol- lower requests the location of object and reset False when environment actions are executed.
tt__step__oj_rrquustteee
Instruction: For communicating the task and its parameters
cur jrr_step_obj__ene This feature is False initially, gets set True when the Comman- der sends the location of an object and reset False when environment actions are exe- cuted.
Step: For communicating a single desired ob- ject state change that would result in progress towards completion of the task, for example, cleaning mug to fill it with coffee eventually.
We correspondingly also create a special RequestStep action. Besides this, we have two non-dialogue actions that agent can also choose
This feature is False ini- tially and gets set to True when the task is completed successfully.
task_complete:
perform:
follower_stuck: This feature is used to identify failed dialogues. It is False initially and set to true if the Follower attempts envi- ronment actions but is unable to accomplish the intended object state change. When this is identified, the dialogue is terminated early.
ExecuteStep: This is cue transition to actions in the environment. This is only per- formed by the Follower which identifies rule-based plan to accomplish the desired state change and executes with assisted plan exe- cution described in section 4.
Given the value of the state features, the next state of the agent computed boolean functions over state features included in Table Given the dialogue acts we then sample dialogue acts accord- ing to Table 6 for the Commander and Table for the Follower.
DoNothing: This allows an agent skip turn and hence avoids rigid turn-taking in the resultant dialogue and introduces variability in the amount of information communicated in the dialogue to mimic real dialogues better.

---

dialogue_not__tarteed
dialogue_started
dialeguestarrre///poocccmmiiiii A-
Lask_.coppeeeee-=ffoolwoorrsssss A-
goal_or_step_start
daalgguggstarreeeleeoooooooooonn A=
makk.complette/-ffoollllerrttttt A-
step_start
cur.sstep_eepustteeee/rrcrssssse
step_requested
A
Shalogug_starrree//tttttsccc.ooo A-
step_sent
rbj_eoo_eequested
A
obj_loc_sent
task_complete
follower_stuck
dialogue_started A task A follower_stur
Table 5: Boolean functions over dialogue state features to compute current dialogue state (- represents NOT and A represents AND).
Instruction
0.8
RequestFor
1.0
dialogue_not_star
lettep
0.1
dialogue_not_star htt
eanstruction
DoNothing
0.1
RequestStep
0.8
Instruction
0.8
goal_or_step_stare
ReqForObjLoc
0.1
goal_or_step_star
Step
0.1
AndOD
DoNothing
0.1
ExecuteStep
0.1
Step
0.9
RequestStep
0.8
step_reqeested
DoNothing
0.1
step_start
ReqForObjLoc
0.1
AndOD
Step
0.9
step_start
ExecuteStep
0.1
DoNothing
0.1
step_requested
ExecuteStep
1.0
InfoOnObject
0.9
obj_loc_
Details
ReqForObjLoc
0.9
DoNothing
0.1
AndOD
step_sent
ExecuteStep
0.1
Feedback Negative
follower_stuck
1.0
ExecuteStep
1.0
bbj_loc_requesteee
Feedback
task_complete
1.0
Positiu tive
obj _oj_loc_sent
ExecuteStep
1.0
Table 6: Prohabilities for sampling various dialogue acts for the Commander given the dialogue state.
Table 7: Probabilities for sampling various dialogue acts for the Follower given the dialogue state.

---

Given a dialogue act, we use the following tem- plates to get utterances:
Desired
Out-
Plan
RequestForInstruction Hello, what can help you with today?
come
(Pickup, (TARGET)), (Place,
Clean- ing
Sink), (ToggleOn, Faucet), (ToggleOff, Faucet), (Pickup, (TARGET)), (Pour, Sink)
Instruction: This is filled in with the description field from the task definition, for example Make coffee or Put all Forks in any Sink.
(Pickup, (TARGET)), (Place, CoffeeMachine); (ToggleOn, CoffeeMachine)
Making Coffee
RequestStep: Done. what should do next?
(Pickup, Knife), (Slice, (TARGET)), (Place, CounterTop)
Slicing
Step: This filled it from the condition_faillue_e. _esc field from the TEACh task definition. For example, The Mug does not have coffee. or The Fork must be placed in Sink.
(Pickup, (TARGET)), (Place, Toaster), (ToggleOn, Toaster), (Pickup, (TARGET)), (Place, CounterTop)
Toast- ing Bread
seggueestoooiieeeeaatiooonnntter Where can I find a/an (object)? where (object) is identified from the task step.
(Pickup, (TARGET)), (ToggleOff, Microwave), (Open, Microwave), (Place, Microwave), (Close, Microwave), (ToggleOn, Microwave), (Toggleoff, Microwave), (Open, Microwave), (Pickup, (TARGET)), (Close, Microwave), (Place, CounterTop)
InfoonObjectDetail:: You can find a/ and (object) in/on a/an (object) near a/an (object) where the reference objects are identified using location information in the simulator.
Cook- ing
FeedbackPositive: Great! We' re all done.
(Pickup, (TARGET)), (Place, (DESTRED_VALUE))
Placing
If an ExecuteStep action is sampled, we then synthesize the plan for the property to be changed part of the current task step using Table 8. This is then executed using assisted exccution (Section 4, Appendix C) to obtain the final sequence en- vironment actions. If this produces the desired object state change, dialogue simulation contin- ues; otherwise. is terminated by entering the state. Our final implementation of dialogue simulation successfully simulates dia- logue that completes the task in 35.4% of the initial states the TEACh dataset.
(Pickup, (TARGET)), (Place, Bowl), (Pickup, Bowl), (Place, Sink), (ToggleOn, Faucet), (ToggleOff, Faucet), (Pickup, Bowl), (ToggleOff, Microwave), (Open, Microwave), (Place, Microwave), (Close, Microwave), (ToggleOn, Microwave), (ToggleOff, Microwave), (Open, Microwave), (Pickup, (TARGET)), (Close, Microwave), (Place, CounterTop)
Boiling
(Pickup, Cup), (Place, Sink), (ToggleOn, Faucet), (ToggleOff, Faucet), (Pickup, Cup), (Pour, (TARGET))
Fill
With
Water
Turn On
(ToggleOn, (TARGET))
Open
(Open, (TARGET))
Table 8: Plans to accomplish object state changes


Reviews:
[{"review_id": "49Serw6KgM", "reviewer": "Reviewer_YKo6", "summary": "", "questions": "Q1: The paper would benefit from a bit more discussion on why the \"data scaling law\" analyses in Table 2 and 3 don't follow a strictly increasing pattern. Could it be stemmed from the lack of diversity in the dialog template used to generate the data?\n\nQ2: I suggest the authors refer to the EDH used in this paper as \"EDH-Plan-Only\" since the EDH has been simplified in this paper (see L197-198) by giving the agent the ground-truth coordinate of the object thus removing the need to learn navigation. Using the term EDH might mislead some readers to draw a direct comparison between the results in this paper and other papers that uses the full EDH setting. Note that this is a suggestion and it does not negatively influence my rating of this paper.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time and insights, particularly the acknowledgement of the engineering effort that goes into works such as this. \n\nTo answer the reviewer’s questions:\n - Q1: We believe the sub-linear scaling with the increase in data is due to a combination of limitations in the range of TEACh initial states in which our synthetic plan execution method is able to successfully generate a valid action sequence, limitations of the diversity in synthetic templates and limitations in the ability of the ET model to effectively model the TEACh plan prediction task. We believe that through engineering improvements, or a learned Reinforcement Learning policy, we can improve the range of initial states covered, with the assistance of better LLMs we can produce more diverse synthetic dialogue and using neural SLAM models, we can overcome the limitations of what a particular model can learn. We plan to explore these directions in future work and hope that the reviewer will consider the current results sufficient for an initial publication.\n - Q2: We will make the suggested change to EDH-Plan_Only to avoid confusion with the original TEACh EDH task.\n\nWe will also include the additional suggested references in our related work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1: Important topic. Data scarcity has been one of the major bottlenecks in embodied AI research. The reliance on humans to collect such dialog data, especially in an embodied setting, is expensive and not scalable. This paper sheds some initial light on how machine-generated data could help alleviate this problem.\n\nS2: Solid engineering work. The author described the process of generating the dialog and trajectory data concisely in the paper, but having worked in this domain myself, I know the tremendous amount of engineering work that must have gone into this. Good job!\n\nS3: Thorough experiments, convincing results. It is encouraging to see that this type of human-generated data shows some initial signs of life. The \"data scaling law\" analysis in Table 2 and 3 is also helpful, although I think it could use a bit more analysis and discussion (see Q1 below).", "reasons_to_reject": "No strong reason to reject.", "questions_to_authors": "Q1: The paper would benefit from a bit more discussion on why the \"data scaling law\" analyses in Table 2 and 3 don't follow a strictly increasing pattern. Could it be stemmed from the lack of diversity in the dialog template used to generate the data?\n\nQ2: I suggest the authors refer to the EDH used in this paper as \"EDH-Plan-Only\" since the EDH has been simplified in this paper (see L197-198) by giving the agent the ground-truth coordinate of the object thus removing the need to learn navigation. Using the term EDH might mislead some readers to draw a direct comparison between the results in this paper and other papers that uses the full EDH setting. Note that this is a suggestion and it does not negatively influence my rating of this paper.", "paper_topic_and_main_contributions": "This paper studies the prospect of using machine-generated embodied dialog and trajectory to replace/augment human-generated dialog/trajectory for embodied task learning. Specifically, the authors use the ground-truth task progress information provided by the simulator (TEACh in AI2THOR), paired with a template-based embodied dialog generation pipeline, and a rule-based action planner to obtain purely machine-generated dialog exchanges and action trajectories. The authors evaluate the effectiveness of this generated data on a simplified version of the TEACh EDH task (where navigation is removed from the task, the agent only needs to predict object manipulation actions and object class). The model used for comparison of human-generated data (original TEACh) and the machine-generated data is an end-to-end model (Episodic Transformer, E.T.) that takes in visual observation, dialog history, and previous actions and predicts the next plan step. Encouragingly, the author finds that models trained with machine-generated data only perform on par with models trained with human-generated data, and mixing machine- and human-generated data together could outperform using human-only data. This finding suggests the effectiveness of using machine-generated data for embodied dialog.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Modular solution on ALFRED:\n- A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution (https://arxiv.org/abs/2107.05612)\n\nModular solution on TEACh:\n- DANLI: Deliberative Agent for Following Natural Language Instructions (https://arxiv.org/abs/2210.12485)"}}, {"review_id": "oY2lbcQOnw", "reviewer": "Reviewer_zTkK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insights. We apologize that our paper was difficult to a reader less experienced in this field and would appreciate constructive suggestions on how to revise the text to make it more broadly accessible if the paper is accepted. \nAdditionally, while we show improvements for only the TEACh EDH plan prediction task, we do show in Table 1 that this in itself is a fairly challenging task with the TEACh baseline having about 20% test unseen success rate but human plans having about 63% success rate.\nWe additionally believe that our synthetic data generation method is a valuable resource beyond its use in TEACh tasks, particularly in the exploration of other embodied AI problems for which no manually collected action sequence data exist.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper addresses the issue of low availability of training data for a complex multimodal task.\nThe author show how properly generated synthetically training data can help improve the performance of multimodal transformer over model trained on real data.", "reasons_to_reject": "1. Paper was a relatively hard reading for me and I feel would not be easily accessible to readers who do not experience in this field.\n2. The task variant is a simplification of the actual TEACh task, as such performance of the proposed method is not yet proven for the actual task that better reflects the actual challenges of the task.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper deals with prediction of high level plan (sequence of actions) for a given dialogue history, past actions and image observations. They present a synthetic dialogue generation method, which they use to then train multimodal transformers.\nThey show that multimodal models trained on this synthetically generated data outperforms text-based baseline models that are trained on real data.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WIzFDBdmXI", "reviewer": "Reviewer_MiCJ", "summary": "", "questions": "In addition to the above comments in the reason to reject section, would it be possible to generate similar synthetic data by prompting LLM? Comparing with the synthetic dialogue generated by the proposed method, what would be the advantages of the proposed method in generating synthetic dialogue?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insights. We would like to highlight that we do mention plan prediction as being an important subtask for existing methods such as JARVIS and FILM. However, these works do not explicitly evaluate this subproblem or quantify its difficulty for the TEACh or ALFRED datasets. In ALFRED, plan prediction can be reduced to a simple 7-way classification, but in TEACh, this is a significantly more challenging problem due to the increased complexity of the tasks and the range of task parameters (which we found to be very difficult to predict accurately from the TEACh dialogs). \nAdditionally, we had explored generating the dialogue entirely, as well as paraphrasing synthetic dialogue using OPT models, but we found that this often led to inaccuracies in details of the task such as the exact object to be used, resulting in worse downstream performance. We can include these negative results in our appendix if accepted and can also repeat the experiments with more recent open source LLMs. \nWe do think that this would be a good testbed for prompting LLMs, although it is a good question to decide how much ground truth information about the environment setup should be made visible to the LLM. Some preliminary experiments we performed in this direction with OPT models did not lead to any successful plans but we hope to iterate on this prompting approach in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper formulates the problem of plan prediction for the TEACh dataset, and generates synthetic dialogues for the task. The experiment shows that the model trained on the synthetic data can achieve good zero-shot performance. The way of generating synthetic dialogues could be generalized to help solving other tasks.", "reasons_to_reject": "The proposed plan prediction task is not novel where it is already adopted in prior works on TEACh dataset, eg Film: Following instructions in language with modular methods and Jarvis: A neuro-symbolic commonsense reasoning framework for conversational embodied agents, where they predict subgoals before generate low-level actions. \n\nAlso, the evaluation misses large language models where they are likely to achieve good performance on the plan prediction task with simple in-context learning.", "questions_to_authors": "In addition to the above comments in the reason to reject section, would it be possible to generate similar synthetic data by prompting LLM? Comparing with the synthetic dialogue generated by the proposed method, what would be the advantages of the proposed method in generating synthetic dialogue?", "paper_topic_and_main_contributions": "The paper addresses the problem of embodied task completion, where an artificial agent in a simulated environment predicts actions to complete tasks based on natural language instructions and egocentric visual observations. The authors propose a variant of this problem where the agent makes predictions at a higher level of abstraction, which they refer to as a \"plan\". The paper presents results showing that multimodal transformer models trained on synthetic data can achieve strong zero-shot performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the use of machine-generated dialog-trajectory pairs for training symbolic plan prediction models in embodied trajectory-from-dialog tasks. The paper proposes a method to generate this data from the simulator, and finds that a plan-prediction transformer model trained on this data is on par with human-generated data.

Some strengths cited by individual reviewers:
“The paper solves an important problem of data scarcity in embodied AI research”. 
“Solid engineering work.”
The synthetic data generation pipeline is general with some modification
“Thorough experiments and convincing results”

Outstanding concerns after rebuttal:
- Only looked at training transformers, did not try LLM prompting. My opinion: although LLM prompting is a very popular approach and deserves being looked at, it is also its own moving target that this paper did not aim to study.
- Section describing data-generation was hard to read for reviewer zTkK, who is not in the area. Constructive feedback was discussed

Overall, there is consensus that the paper is sound and has addressed the research question it sought to study. Reviewer YKo6 gave a soundness score of 4 with the highest confidence, while the other reviewers scored it as 3.

Excitement scores are 4 from reviewer YKo6 (the most confident and well-matched reviewer), and 3 from everyone else.
Decision: Accept-Main