tagE: Enabling an Embodied Agent to Understand Human Instructions
Authors: Chayan Sarkar
Avik Mitra
Pradip Pramanick
Tapas Nayak
Venue: EMNLP 2023 Findings (2023)
Keywords: human-robot interaction; NLP for robotics; task and argument extraction; task and argument grounding;
TL;DR: A system for embodied agent for extracting tasks and the corresponding arguments from a natural language instruction and grounding them

Abstract: Natural language serves as the primary mode of communication when an intelligent agent with a physical presence engages with human beings. While a plethora of research focuses on natural language understanding (NLU), encompassing endeavors such as sentiment analysis, intent prediction, question answering, and summarization, the scope of NLU directed at situations necessitating tangible actions by an embodied agent remains limited. The inherent ambiguity and incompleteness inherent in natural language present challenges for intelligent agents striving to decipher human intention. To tackle this predicament head-on, we introduce a novel system known as task and argument grounding for Embodied agents (tagE). At its core, our system employs an inventive neural network model designed to extract a series of tasks from complex task instructions expressed in natural language. Our proposed model adopts an encoder-decoder framework enriched with nested decoding to effectively extract tasks and their corresponding arguments from these intricate instructions. These extracted tasks are then mapped (or grounded) to the robot's established collection of skills, while the arguments find grounding in objects present within the environment. To facilitate the training and evaluation of our system, we have curated a dataset featuring complex instructions. The results of our experiments underscore the prowess of our approach, as it outperforms robust baseline models.

tagE: Enabling an Embodied Agent to Understand Human Instructions
Chayan Sarkar and Avik Mitra and Pradip Pramanick and Tapas Nayak TCS Research, India
(sarkar (sarkar.chayan, mitra.avikl, pradip. pramanick, nayak. tapas) @tcs.com
Abstract
2020). However, large vocabulary and variation of word/phrrssewientences of any natural language (e.g., English) makes it very difficult for a robot to understand human intention and perform the task (Pramanick et al., 2019b).
Natural language serves the primary mode of communication when intelligent agent with a physical presence engages with human beings. While a plethora of research focuses on natural language understanding (NLU), encompassing endeavors such as sentiment analysis, intent prediction, question answering. and summariza- tion, the scope of NLU directed at situations necessitating tangible actions by an embodied agent remains limited. The inherent ambigu- ity and incompleteness inherent in natural lan- guage present challenges for intelligent agents striving to decipher human intention. To tackle this predicament head-on, we introduce novel system known as fask and argument grounding for Embodied agents (tagE). At its core, our system employs an inventive neural network model designed to extract series of tasks from complex task instructions expressed in natural language. Our proposed model adopts encoder-decoder framework enriched with nested decoding to effectively extract tasks and their corresponding arguments from these intri- cate instructions. These extracted tasks are then mapped (or grounded) to the robot's 's established collection of skills, while the arguments find grounding in objects present within the environ- ment. To facilitate the training and evaluation of our system, have curated dataset fea- turing complex instructions. The results of our experiments underscore the prowess of our ap- proach, as it outperforms robust baseline mod- els.
Recent advances in natural language processing (NLP), in particular, the rise of large-scale neural language models have simplified NLP tasks with high accuracy (Devlin et al., 2019; Brown et al., 2020). However, in order for robot perform task, the task intention has to be mapped to a known set of skills of the robot (task grounding) so that some action can be taken in the physical world. Additionally, the objects associated with task (task arguments) should be mapped to the objects within the environment (argument ground- ing). Most robots use object detector, which fixed vocabulary. A human may not be aware of that vocabulary or may not remember it. As a result, a different word/phrase can be used refer the same object. Therefore, argument ground- ing becomes equally important in order to perform physical action in the environment.
Existing works on intent prediction map the in- tended task to the robot's capability (Brawer et al.. 2018). However, they can neither extract the argu- ments associated with a task nor handle complex instructions with multiple tasks. Works relation extraction generally find the relevant triplets in a natural language text, where the triplets have the form of head, relation, and tail < h, r, L >> (Nayak and Ng, 2019). One can use such a method to find a task-argument triplet (Figure 1a). But, mul- tiple arguments may be associated with the same tasks. Extracting such triplets where the head/tail is common in many such triplcts is not straight- forward (Figure 1b, Ic). Moreover, the existing methods neither ground the task nor ground the arguments. For example, in Figure 1b, the word 'keep' signifies the 'PLACING' task according to the robot's 's capability, and the word 'fridge' needs
1 Introduction
Robots in our daily surroundings are often engage with human beings for various purposes. As nat- ural language interaction capability increases the acceptability and usability of robot, many stud- ies have focused on natural language interaction with a robot (Williams et al., 2015). This can be particularly useful if we can provide task in- struction in natural language (Pramanick et al.,

---

goal
theme
MOUROE
scunce
theme
to the tble and pick
black on the dining table pick it and keep inside the fridge
pick up the mag and the brown bond from the wemden table
theme
gral
gpi gool, 'aabl""
piok PICKING, theme; "black cup', CUP
pick up', PICKING, theme, "ma"" CUP
piic", theme, 'cup"
pick', PICKING, source. 'dining
pick up PICKING, source. 'woeden table". TABLE
kkepp' PLACING, theme, "black cup"
"bown bowl'. BOWL pick up', PICKING, souroe, "worden table", TABLE
pick up' PICKING,
Keep', PLACTNG, godl "Frlge",, REFRIGERATOR
SoTA haseling- simple
sub-iasks single irguments each
(1) ligft- two utbsttakss with mulliple argumants commen argummn,, which is extracted using an inberent coroference resulver
(c) tugF- sub--aaks the commun baok plrnss and cummon argument
Figure/ I: Some natural language task instruction (a) example where existing triplet extractor can be employed for task and argument pair extraction, (b) example of why it is necessary to ground task and argument along with co-reference resolution (pentuple extraction), (c) example of why generative approach for task and argument extraction is required.
be mapped to 'REFRIGERATOR' according the object detector's vocabulary. Similarly, the same task phrase 'pick up' as well as the same ar- gument phrase 'wooden table' are shared by two tasks as shown in Figure lc. This can only be possible if an NLU system can generate as many triplets as possible from the given natural language instruction without any limitation.
Related Work
There are three different areas that are relevant this work - Secquence-t--sequeee tured information extraction, and natural language understanding of instructions for robots.
learning, struc-
Sequence-to-sequence learning: Encoder- decoder models are popular for the sequence- to-sequence (S2S) learning. There are different types of S2S tasks for which encoder-decoder ar- chitecture is used, e.g., neural machine transla- tion (Sutskever al., 2014; Bahdanau et al., 2015; Luong et al., 2015). joint entity and relation extrac- tion (Zeng et al., 2018; Nayak and Ng, 2020), cross- lingual open information extraction (Zhang et al.. 7), joint extraction of aspect-opinion -eeiiio-sseiiiiment triplets (Mukherjee et al., 2021), etc.
We have developed a novel deep neural net- work architecture, called tagE (task and argument grounding for Embodied agents) that extract the set tasks and their relevant arguments from a natural language complex instruction. The contributions of this article are following.
We proposed a novel mncoder-decoder archi- tecture for natural language understanding that employs layered decoding. Unlike ex- isting triplet extractors, tagE' finds pentuple < h, hgg r, fg >, where hg and tg repre- sent the grounded values of the task (h) and argument (t), respectively.
Structured Information Extraction: Struc- tured information extraction from unstructured text is an important task in natural language processing. Entity- -eeity-Relation extraction (Miwa and Bansal, 201 2016; Shen and Huang, 2016; Vashishth et al., 2018; Nayak and Ng, 2019), aspect-sentiment triplets ex- traction (Xu et al., 2020; Jian et al., 2021; Mukher- jee et al., 2021), causality extraction (Li al., 2021), event extraction (Liu al., 2018; Sha al., 2018), attribute- extraction (Roy et al., 2021,
tagE employs shallow auto-regressive de- coder, which enables it to extracts many grounded task-argument pairs possible, re- solve co-reference, and handle the shared task and/or argument phrase.
2022) are such important tasks. 'BIO" tag-based se- quence labeling models are popular to extract struc- ture information from text. In recent times, pointer networks are explored for such tasks (Vinyals et al., 2015). Seo et al. (2017); Kundu and Ng (2018) used pointer networks for the machine-reading comprehension task identify the answer span in the passage. Nayak and Ng (2020) used a simi- lar network to identify the entities in sentences for joint entity and relation extraction tasks. Similarly, Becquin (2020) used such networks for identifying
We have curated new dataset that enables grounded task-argument extraction. The model can ground the argument based on ob- ject detector vocabulary. The task-argument relation data and argument grounding data are detached in way that the object detec- tor as well as the class of objects can be ex- panded/modified !
'nyy resource related to this work will be made availahle at https: //gi thwb. com/ wwt////rkaa/////////oooccc// tsar.ar/tagE.

---

casual spans in the text. (Mukherjee al., 2021) used pointer networks for joint extraction of aspect- opinion-sentiment triplets from online reviews.
et al., 2020: Min al., 2021). Though these models are particularly effective for following detailed in- structions with explicitly mentioned tasks in known environments, they often generate incorrect plan for high-level instructions with implicit sub-goals that require long- oong-horizon task planning (Min et al., 2021). Corona et al. (2021) proposed segment- ing the instruction into a sequence of task types and training separate seq2seq models for different task types. Subsequently, several works have pro- posed modular methods that decouple multi-modaa seq2seq learning into separate language understand- ing, perception and controller components. For example, Nguyen et al. (2021) proposed a model with separate decoders for predicting task predi- cates of navigation and manipulation parts of the instruction. Similarly, other works (Jansen, 2020; Huang et al., 2022) explored plan generation us- ing large pre-trained language models. Liu et al. (2022) trained two separate models for action and object prediction, and Ri et al. (2022) studied plan generation solely from language input by training CTC-based model. However, all of these mod- els directly predict low-level action sequence or grounded goal predicates; thus still learns agent and environment specific representations.
Natural Language Understanding for Robots: Natural language understanding for robots mostly involves exccuting human- provvddd directives given in natural language. Significant progress has been made by i) restricting the action space of the robot to navigation, i.e., posing it as a Vision and Language Navigation (VLN) problem, and ii) providing detailed step-by-step instructions that either reduce or remove the burden of planning from high-level task goals (Anderson et al., 2018 8; Blukis et al., 2019; Shah et al., 2022). A few works have attempted to include manipulation in VLN, but still allowing step-by-step instructions and lim- ited to a single or a constrained set of manipulation actions (Misra et al., 2018; Kim et al., 2020; Pa- shevich al., 2021). We focus on more general problem that assumes the arbitrary action capabil- ities of robot that includes both navigation and manipulation. Thus the problem can be defined as generating an action sequence (plan) for high- level natural language instruction that contains one or more tasks. Several approaches have been pro- posed to solve this. Predominant methods exploit the embodied nature of robotic agent to infer and refine the plan by primarily using multi-modal input that includes visual feedback and action pri- ors (Paxton et al., 2019; Shridhar et al., 2020; Singh et al., 2021; Zhang and Chai, 2021; Ahn et al., 2022). Thus natural language understanding in these systems is simplified by obtaining a latent representation of the language input to bias the in- ference using attention modeling. Embodied agents that can answer natural language queries have also been developed by following similar approach of planning by biasing the agent's 's exploration using latent linguistic representations (Das al., 2018).
Existing approaches of structured prediction from instructions follow a sequence labeling ap- proach which has two major limitations i) it can't classify tokens multiple times and/or with separate class label, and ii) it can't handle co-reference in complex instructions. In contrast, we propose novel generative model that has significant advan- tages over sequence labeling.
3 Proposed Framework
In this section, we formally define the task before describing the neural network architecture. Given natural language instruction X = (21, )n) with n tokens, the goal to extract a set of tasks, j=1 where Ly is the j'th
Several works proposed models that perform vi- sual grounding of referring expressions an em- bodied agent's ego-view, following a similar ap- proach for encoding language input (Qi et al., 2020; Rufus et al., 2021; Roh et al., 2022; Pramanick et al., 2022). A major limitation of this approach of end-to-end learning with multi-modal input is that the models are often heavily biased towards specific dataset, simulator, and agent with specific capabilities. Thus, they often exhibit poor general- ization to unseen environments and fails to gener- ate plan for unseen composition of tasks (Shridhar
T [1,4, (tz, 15, d)))"" task. [T] is the number of tasks, l and i5 represents the positions of the start and end tokens of the task description span, and i represents the type of the task (grounded task). Additionally, we extract the set of arguments for each task. Specifically, for task
Ij. we extract A₁- [ajxl ags = (afte afk= 111,) where as& is the k'th argument, |A,l is the number of arguments for the task, aze and afe represents the positions of the start and end tokens of the argument description span, and dfs represents the

---

index
15
token
the
black
1s on the
dining table pick
it
and
keep
inside
the
fridge
Being
Theme, 2 Theme, Source:
annotation
8 Picking,
11 Placing.
2 Theme, 15 15 Goal, 15 15 Container object
tag
cup
cup
Ltable
reffigerator
Figure? 2: Example of instruction annotation - the annotation for grounded task types and the corresponding argument types are shown row 3, and the BIO tags for argument (object) grounding shown row
grounded objotts
grounded taok dooodkrr
inser degunion type decollr
(engsmmenn) E. (urgmanrg)
1- E- (argnesst
- (eggnangg)))
ENN
Fneader
(tranafommr lsyen)
hr
N
Attention
Alletionn
Task Insraction
Figure 3: Encoder-decoder architecture of our tagE model.
type of the argument. The number of arguments for the tasks can be different. We added the example of such positional sequence in Figure 2. Additionally, we do the grounding of the arguments, i.e., map the span of an argument an object it is physical object.
task extraction, argument extraction, and argument grounding together.
3.1 Instruction Encoder
We use pre-trained BERT tokenizer and BERT model (Devlin al., 2019) obtain the contex- tual encoder representation of the input tokens, h E Rd,, where Rd the vector of real numbers of dimension d4.
In Figure 3, we give an overview our proposed model. We use an encoder-decoder model inspired from Nayak and Ng (2020); Mukherjee et al. (2021) for this task. The encoder is used to encode the given natural language instruction and decoder is used to generate the task and argument sequence. However, Nayak and Ng (2020) proposed linear decoding process for joint entity and relation ex- traction task and Mukherjee et al. (2021) proposed an aspect-sentiment triplets extraction task. But such linear decoding scheme is not suitable for this task. We need to extract a list of tasks and for each task, we need to extract list of arguments. Thus, we propose a novel nested decoding approach for task and argument extraction. The outer decoder is responsible to generate the task sequence and inner decoder is responsible generate the argument se- quence for each task. For argument grounding, we use a 'BIO' sequence labeling based approuch. We jointly train our model for these three objectives of
3.2 Object Grounding
The arguments that refers to physical object in the environment needs to be grounded, i.e.. mapped to the object detector's vocabulary that the agent is using. We use 'BIO' sequence la- beling approach for object grounding. The en- coder vectors h are passed to a feed-forward layer with softmax activation for classifying span to grounded objects. From this sequence label- ing, we obtain the list of object spans in the in- struction along with the grounded object class. Each argument span extracted by the inner argu- ment decoder assigned to the class type from this list. In our 'B1O' tagging scheme total num- ber tags, K x #grounded_objeen Hygrounded_objeee is the number of objects the vocabulary of the object detector of the the
1.

---

robot. For each token in the instruction, we get K dimensional softmax output vector. We use this vectors in our argument extraction network to im- prove the extraction process. Please note that for object (argument) grounding, sequence labeling is sufficient as even if an argument is shared in mul- tiple tasks, it is detected as the same object by the object detector.
Algorithm 1 Proposed nested decoding algorithm
TD = citt_tas_ddecodee
AD unit_aruument [in_aarummnt_decodes _decoderrdrrrrrreeeeeee
for in range(mar farnaarttssk_aatt _taak_cskntt do
TD decodes task tj for k in Lange(mar_arg.com do
AD decodes an argument azk for task ty
end for end for
3.3 Task Sequence Generation
We use LSTT--based ruto-regressive decoder for generating the task sequence. At every time step of the task decoding, decoder LSTM re- ceives an overall representation encoder context C dR and representation of the previously gen- erated tasks 1/:0 bj) as input. The encoder context representation e, is obtained using an at- tention mechanism on encoder hidden states h used in Bahdanau et al. (2015) for neural ma- chine translation. At the first decoding step, bo is taken a vector, and the subsequent task vectors bj's is are obtained after each decoding step (described in Section 3.5). Task decoder LSTM outputs the task hidden vector hy c IR.,,, This hid- den representation is concatenated with the hidden representations h. of the encoder and passed to a span detection module to mark the start and end token of the task description. This span detection module is described later in detail. After one task is extracted, the inner decoder (argument decoder) receives the task hidden vector hz and starts gener- ating the argument sequence for this task. Once all the arguments of this task are generated, this outer task decoder moves on to generate the next task.
these two modules is similar in nature. They have BiLSTM layer with two feed-forward layers with softmax activation. We concatenate the task hid- den vector h2 or argument hidden vector hy with the encoder vectors h and pass it to the BiLSTM layer. The output of the BiLSTM layer is passed to a feed-forward layer to convert each hidden rep- resentation to a scalar score. We get scalar score corresponding to each token in the instruction. We apply softmax activation across these scalar scores to normalize them. The corresponding token with the highest normalized score is marked as the start token of span. Similarly, another feed-forward layer with softmax activation is used to mark the end token of the span. We obtain the vector rep- resentations of these spans (uj or Pjk) using the normalized scores and BiLSTM outputs as done in Nayak and Ng (2020).
Next, Uj and h are concatenated and passed to a feed-forward layer with softmax classify the task type (vy). We use Vj and task type em- bedding (Ex) layer to get the task type vector tj. Further, the vector representation of the task by is obtained by concatenating uy and ty. Similarly, we classify the argument type (ayx) and obtain the vector representation of the argument Cjs as shown in Figure 3.
3.4 Argument Sequence Generation
Like task decoding, we use similar LSTM-hased auto-regressive decoder for extracting the argu- ments. For the extracted task tj. at every time step k of the argument decoding process, the argu- ment decoder LSTM receives the task representa- tion vector hf. encoder context ezt vectors, and all the previously generated arguments for this task K= Cjk).
3.6 Training and Inference
We train proposed model in mini-batches of size B for multiple epochs and update the model parameters using the negative log-likelihood loss and gradient descent-based optimizer A (Loshchilov and Hutter, 2019). Following is the loss function for the task extraction decoder.
3.5 Span Identification
B
We identify the task and argument description spans in the instruction using their start and end tokens. We use two different span detection mod- ules (shown in blue color in Figure 3) for task and argument span identification. But the design of
L. = B
In( (B)))) + In(e,) + In(c))]
where s and e are the softmax outputs of the corre- sponding gold-label start and end positional index

---

of the task span and is softmax output of the gold-label task type. Similarly, the following the loss function for the argument extraction decoder.
4.1 Dataset
We annotate a new dataset for our experiments. We build the dataset by extending the natural language instruction sentences from two robotic commands datasets (Shridhar al., 2020; Vanzo et al., 2020). The current of task types included in Table 8. The task types are also adapted from robotic task datasets. The first two tasks are not real task types, but often appear in robotic task instruction and need to be processed for understanding the other tasks. Though our datasct is adapted from the existing ones, the annotation is significantly different to suit the purpose. This consists of the following features (i) the token span of the set of sub-tasks in the instruction along with their grounded types, (ii) the of arguments associated with each task along with the token span and the argument type, (iii) provision of sharing an argument by multiple tasks, (iv) provision of classifying a token multiple times, (v) resolution of co-reference of objects, and (vi) argument grounding considering the object classes of the underlying object detector.
B
C2
La
In(ssk) + In(es))
CCJ
+ +In(rz+)
where and are the softmax outputs of the corre- sponding gold-label start and end positional index of the argument span and is softmax output of the gold-label argument type.
Following is the loss for the argument grounding.
n
Lg =
In(g:)
where g is softmax output of the gold-label tag for the i-th token in the instruction. The final loss for a mini-batch of size M of the tagE model is calculated as follows.
L M
1 M
C. 16
La Log
Figure 2 depicts our annotation schema with an example. As any task type argument type can be represented by multiple tokens in the input, we annotate each type with start and end token. For example, the 'Source' of the cup is denoted by two tokens 'dining table', which marked by the start and end token index and 7, respectively. There are three sub-tasks in the example- - Becingloocaten 'Picking', and placing. Each sub-task and its cor- responding arguments form a substructure, which is separated by vertical line as shown in the ex- ample. Within each substructure, the first element always denotes the sub-task, which is followed by the arguments and separated by semicolons. Addi- tionally, each element has three components start token index, end token index, and type of this token span (task/argument type).
During training, we include the end of sequence 'EOS' task and 'EOS' argument at the end of their sequences and let the model learn to stop extrac- tion of tasks/arguments. At the inference time, we run both decoders for maximum number of steps but ignore any tasks/arguments extracted after the 'EOS', During inference, we follow greedy ap- proach to select the start and end tokens of the task or argument spans. We select the start and end where the product of the two probabilities is maxi- mum, and the end token does not appear before the start token. We include the parameter settings used to train our tagE model in Table 1.
Batch size Optimizer
16
AdamW
Since there is separate substructure for each sub-task, a particular token span indicating a partic- ular argument can be shared by multiple sub-tasks. For Figure 2, the 'theme' argument is shared by all three sub-tasks, and the 'Source' argument is shared by two sub-tasks. Additionally, this annota- tion scheme supports shared argument to be clas- sified differently for different sub-tasks. Similarly, token or token span can be classified multiple times as different classes, e.g., token 15 is classi- fied as 'Goal' and 'Containing object" is sub-tast 'Placing.. The idea behind multiple classifications
Learning rate
0.0001
#Epochs
100
Early stop count
20
Table Parameter settings used to train the tagE model.
4 Experiments
In this section, we describe the dataset, parameter settings, and evaluation metrics that are used to
train and evaluate our model.

---

is to provide additional information the task plan- ner.
than 60% overall in the input compared the train and development set.
We have done a separate annotation for argu- ment grounding from an object detector's s point of view. the object classes are changed (in partic- ular the vocabulary set), this annotation has to be changed accordingly. But separate annotation en- sures that the annotation for the task and argument type remains fixed even if the argument ground- ing annotation changed. Since an object detec- tor would always recognize an object by the same being treated as a different
4.2 Evaluation Metric
We define the performance of our system using a strict FI score, which is the harmonic mean of the precision and recall of our classifier as the met- ric definition. To that effect, we consider miss- ing subtask or attribute label to be negative which means that in the case of detection in the case of the wrong classification, the metric takes the prediction as 'wrong'. Conversely, only on a correct label match, it takes the metric as correct. Every hit and every miss recorded in a confusion matrix where it is populated for all types of tasks and all attributes with 'None' being the record for the misses of a given task attribute. Thereafter the confusion matrices are used to calculate the precision and the recall for the particular task at- tribute classes. Additionally, a combined Fl score is also generated as an overall metric for the base- line comparisons that take into account both the task and attribute combinations together.
name, irrespective argument type for tasks, one-time prediction is suf- ficient. Thus, we annotate for argument grounding sequence labeling job using the BIO tagging scheme. The BIO tag for the tokens is shown in Figure 2.
Train
Dev
Test
#lnstruction
1,180
182
580
#single task instruction
755
145
417
#multi task instruction
425
37
163
Table 2: Statisties of the instructions in our dataset.
Task type
Train
Dev
Test
Results
being_located located
64
16
being_in_catesn
49
16
To evaluate our model, we have defined a num- ber of baseline systems. Pramanick et al. (2019a) proposed the first baseline where they have used a CRF-based model for task and argument extraction (Featurre CRF in Table 4. Essentially, the model works sequence labeling job. Apart from lower accuracy in task and argument prediction, such model (sequence labeling) cannot (i) classify the shared task/argument, (ii) reclassify token(s), and (iii) resolve coreference. The next baselinc that we used is a pre-trained BERT model and a fully connected layer as classification head on top. The performance of Feature CRF and BERT base- line is very similar. Then, combine these two approaches, i.e., instead of using pre-trained to- ken embedding (like Feature CRF), used BERT as the encoder layer and the CRF as the classifi- cation layer. Though this BERT CRF performs better as compared to the Feature CRF or BERT model, it again emits the behavior of sequence labeling. Also, we have used two sequence-to sequence models as baseline by finetuning them, namely BART (Lewis et al., 2019) and T5 (Raffel et al., 2020).
bringing
160
17
changing oper. state
45
12
checking state
22
14
cutting
14
12
following
52
12
13
giving
34
13
inspecting
10
10
motion
465
42
227
opening
60
17
picking
259
28
128
placing
226
105
pushing
12
14
rotation
380
228
scarching
#total
1,894
Table 3: Statistics of the different task types in our
dataset.
Once prepared, the data was first proportionately divided to the training, development, and testing sets with 1,180, 182, and 580 inputs, respectively (See Table 2). We include the statisties about dif- ferent task types in Table 3. Though the dataset is not balanced in terms of the number of task types, while splitting, we ensured that the distribution of the task types is similar in each of the splits. Also, there a balance between instructions with single- task and multiple-tasks (Table 2). Additionally, we carefully selected the test set such that there less
As mentioned earlier, tagE follows a generative approach to tackle the limitations of the existing

---

Inference
Without arg grounding
With arg grounding
Model
time
Prec.
Rec.
F1
Prec.
Rec.
F1
Feature-CRF
28.9 ms
0.60
0.58
0.59
0.48
0.47
0.47
BERT
91.2 ms
0.65
0.60
0.62
0.52
0.48
0.50
BERT-CRF
102,1 ms
0.68
0.61
0.64
0.53
0.50
0.51
BART (with beam width
341.8 ms
0.68
0.68
0.68
0.59
0.60
0.60
BART (with beam width 10)
754.79
0.68
0.70
0.69
0.60
0.62
0.61
T5 (with beam width 1)
626.72 ms
0.73
0.74
0.73
0.66
0.67
0.66
T5 (with beam width 10)
2029.17 ms
0.74
0.74
0.74
0.67
0.67
0.67
tagE
108.2
0.85
0.80
0.82
0.72
0.67
0.69
Table Performance of tagE with respect to various baseline methods.
BERT Encoder
Number of parameters
Training time/epoch
Inference time
Fi without arg grounding
with arg grounding
mini
17.5 M
7.9
77.1 ms
0.74
0.62
small
44.8 M
8.5 s
79.7 ms
0.76
0.64
medium
57.5 M
9.6
90.1 ms
0.78
0.66
base
139.2 M
13.3 s
108.2 ms
0.82
0.69
large
382.4 M
23.0 s
135.6 ms
0.80
0.68
Table 5: tagE is trained with different-sized BERT encoders, resulting in different parameters and Fl scores.
Task type
Prec.
F1
Argument type
Prec.
Rec.
F1
being located
1.00
0.97
agent
0.85
0.80
0.82
being in _ategooy
1.00
1.00
area
0.54
0.34
0.42
bringing
0.94
0.78
category
0.17
0,20
0.18
laanging_opcratiinn state
1.00
1.00
1.00
container portal
0.76
0.81
0.79
checking_state
1.00
0.93
0.97
containing object
0.70
0.76
0.73
cutting
0.92
1.00
0.96
eotheme
0.77
0.91
0.83
following
0.85
1.00
0.92
degree
0.52
0.89
0.65
giving
L.00
1.00
1.00
desired state
0.43
0.46
0.44
inspecting
0.20
1.00
0.33
device
0.77
0.83
0.80
motion
0.94
0.93
0.93
goal
0.85
0.85
0.85
opening
0.94
1.00
0.97
cogoal
0.74
0.81
0.78
picking
0.92
0.96
0.94
manner
0.89
0.89
0.89
placing
0.99
0.97
0.98
operational_stte _tate
0.67
0.67
0.67
pushing
0.50
1,00
0.67
recipient
1.00
1.00
1.00
rotation
0.99
0.95
0.97
source
0.71
0.74
0.72
searching
0.92
0.92
0.92
cosource
0.76
0.81
0.79
theme
0.88
0.87
0.87
Table 6: Task-wise performance of tagE.
Table Argument performance tagE.
methods. The performance gain is evident from Table 4. We have calculated a combined Fl score considering all the sub-tasks and the corresponding arguments in an input. As the argument grounding is separately annotated, this training separately. We have done two sets of training one with the argument grounding training a sepa- rate job, and another with a joint job of argument grounding and task & argument type prediction. The experiment shows that joint training performs
much better than separate training.
As ablation, we have experimented with differ- ent types of BERT encoders. Though all types of BERT models use number of stacked trans- former layers get vector representation of nat- ural language input, the variation stems from the number of transformer layers and dimension of the output vectors. We have experimented with different variations (i) mini with 4 layers and
be done

---

256-dimensional vector, (ii) small with 4 layers and 122-dimensional vector, (iii) medium with 8 layers and 512-dimensional vector, (iv) base uncased with 12 layers and 936-dimensional vector, and (v) large uncased with 24 layers and 1024-dimensional vec- tor. The pre-trained models for the variations of BERT is provided by Ture et al. (2019). A small number of layers and vector dimensions leads to less number of parameters, smaller model size, and smaller training & inference time. However, this impacts the accuracy of the system as evident in Table 5. With a larger encoder network, the per- formance of tagE keeps increasing with some satu- ration point. The large BERT model, even though it has a much larger network, is unable to outper- form the BERT base model. Thus, we fixated on tagE with the BERT base model the encoder. We provide further accuracy statistics of each tasks and arguments for the test split of the dataset for tagE. The details are summarized Table and 7.
tagl' is trained to extract the sub-tasks in the same sequence as they appear in the instructions. In certain situations, the robot may need to perform the sub-tasks in a different order. For example, in the instruction "bring a can of beer, if you can find chilled one", even though the Bringing sub- task appears earlier than the Check-state sub-task, the robot has to perform them in the reverse order. However, these problems are typically solved by task planning.
Ethics Statements
There is no ethical concerns about this work.
References
Hyemin Ahn, Obin Kwon, Kyungdo Kim, Jaeyeon Jeong, Howoong Jun, Hongjung Lee, Dongheui Lee. and Songhwai Oh. 2022. Visually grounding lan- guage instruction for history- manipulation. In 2022 International Conference on Robotics and Automation (ICRA).
6 Conclusions
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton Van Den Hengel. 2018. Vision- and-language navigation: Interpreting visually- grounded navigation instructions in real environ- ments. In Proceedings of the IEFE conference on computer vision and pattern recognition.
Instructing a robot in natural language certainly im- proves the usability and acceptability of the robot. However, understanding the instructions given in unstructured text can be challenging for a robot. A robot needs to perform multiple jobs to com- pletely understand natural language instruction such extracting the (sub)tasks, extracting their associated arguments, and grounding of the argu- ments. Previously, researchers have tried to solve these jobs independently, thus misses the interac- tion among them in the given instruction. To over- comes this limitation, we propose neural network model tagE, which can solve these three tasks to- gether in end-to-end fashion. We also annotate suitable dataset for experiments and our proposed approach outperforms some strong baselines on this dataset. In future, we would integrate plan- ner module that would generate an executable plan from the output of tagE.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning align and translate. In ICLR.
Guillaume Becquin. 2020. Gbe at fincausal 2020, task 2: Span-based causality extraction for financial doc- uments. Proceedings of the 1st Joint Workshop on Financial Narrative Processing and MultiLing Financial Summarisation.
Valts Blukis, Yannick Terme, Eyvind Niklasson, Ross A Knepper, and Yoav Artzi. 2019. Learning to map natural language instructions physical quad- copter control using simulated flight. arXiv preprint
Jake Brawer, Olivier Mangin, Alessandro Roncone. Sarah Widder, and Brian Scassellati. 2018 Situ- ated human-robot collaboration: predicting intent from grounded natural language. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
7 Limitations
In our experiments, we observe that while the model is mostly accurate on the test dataset, it fails certain long natural language inputs. In particular, the sub-tasks and their arguments are not often not predicted for very long input with more than sub-tasks. However, such long in- put with as many sub-tasks is not very common in typical human-robot interactive scenarios. Also,
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems.

---

Rodolfo Corona, Daniel Fried, Coline Devin, Dan Klein, and Trevor Darrell. 2021. Modular networks for com- positional instruction following. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention- based neural machine translation. In EMNIP.
So Yeon Min, Devendra Singh Chaplot, Pradeep Kumar Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov. 2021. Film: Following instructions in language with modular methods. In International Conference on Learning Representations.
Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018. Neural modular con- trol for embodied question answering. In Conference on Robot Learning.
Dipendra Kumar Misra, Andrew Bennett, Valts Blukis, Eyyind Niklasson, Max Shatkhin, and Yoav Artzi. 2018. Mapping instructions to actions in 3d environ- ments with visual goal prediction. In EMNLP.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In NAACL.
Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using LSTMs on sequences and tree structures. In ACL.
Wenlong Huang. Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot Extracting actionable knowledge for em- bodied agents. arXiv preprini 220002201.07207.
Rajdeep Mukherjee, Tapas Nayak, Yash Butala, Sourangshu Bhattacharya, and Pawan Goyal. 2021. Paste: A tagging-fire decoding framework using pointer networks for aspect sentiment triplet extrac- tion. In EMNLP.
planners:
Peter Jansen. 2020. Visually- -rouully-rounded planning with-
out vision: Language models infer detailed plans from high-level instructions. In Findings of the As- sociation for Computational Linguistics: EMNLP 2020.
Tapas Nayak and Hwee Tou Ng. 2019. Effective tention modeling for neural relation extraction. In CoNLL.
Samson Yu Bai Jian, Tapas Nayak, Navonil Majumder, and Soujanya Poria. 2021. Aspect sentiment triplet extraction using reinforcement learning. In CIKM.
Tapas Nayak and Hwee Tou Ng. 2020. Effective mod- eling of encoder-decoder architecture for joint entity and relation extraction. In AAAI.
Hyounghun Kim, Abhaysinh Zala, Graham Burri, Hao Tan, and Mohit Bansal. 2020. Arramon: A joint navigation-assenn instruction interpretation task in dynamic environments. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2020.
Van-Quang Nguyen, Masanori Suganuma, and Takayuki Okatani. 2021. Look wide and interpret twice: Improving performance on interactive instruction- following tasks. In I/CAI.
Alexander Pashevich, Cordelia Schmid, and Chen Sun. 2021. Episodic transformer for vision-and- lad--uugeg navigation. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision
Souvik Kundu and Hwee Tou Ng. 2018. A question- focused fulti-factor attention network for question answering. In AAAI.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising
Chris Paxton, Yonatan Bisk, Jesse Thomason, Arunk- umar Byravan, and Dieter Foxl. 2019. Prospection: Interpretable plans from language by predicting the future. In 2079 International Conference on Robotics and Automation (ICRA).
pre-training for natural language generation, translation, and comprehension. arXiv preprint 19Xiv: 1366.1341.
Pradip Pramanick, Hrishav Bakul Barua, and Chayan Sarkar. 2020. Decomplex: Task planning from com- plex natural instructions by collocating robot. 2020 JEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS).
Zhaoning Li, Qi Li, Xiaotian Zou, and Jiangtao Ren. 2021. Causality extraction based on self-attentive bilstm-crf with transferred embeddings. Neurocom- puting.
Haoyu Liu, Yang Liu, Hongkai He, and Hangfang Yang. 2022. Lebp-languag expectation & binding pol- icy: A two-stream framework for embodied vision- and-language interaction task learning agents. arXiv preprint 2XXx:2203.04637
Pradip Pramanick, Chayan Sarkar, P Balamuralidhar. Ajay Kattepur, Indrajit Bhattacharya, and Arpan Pal. 2019a. Enabling human-like task identification from natural conversation. In 2019 JEFF/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS).
Xiao Liu, Zhunchen Luo, and Heyan Huang. 2018. Jointly multiple events extraction via attention-based graph information aggregation. In EMNLP.
Pradip Pramanick, Chayan Sarkar, Snehasis Banerjee, and Brojeshwar Bhowmick, 2022. Talk-to-resolve: Combining scene understanding and spatial dialogue to resolve granular task ambiguity for collocated robot. Robotics and Autonomous Systems.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.

---

Pradip Pramanick, Chayan Sarkar, and Indrajit Bhat- tacharya. 2019b. Your instruction may be crisp, but not clear to me! In 2019 28th IEFE International Conference on Robot and Human Interactive Com- munication (RO-MAN).
Zettlemoyer. and Dieter Fox. 2020. Alfred: bench- mark for interpreting grounded instructions for every- day tasks. In Proceedings of the IEFF/CVE confer- ence on computer vision and pattern recognition.
Kunal Pratap Singh, Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi, and Jonghyun Choi. 2021. Factorizing perception and policy for interactive in- struction fallowing. Proceedings the IEEF/CVF International Conference on Computer Vision
Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang. Chunhua Shen, and Anton van den Hengel. 2020. Reverie: Remote embodied visual referring expression in real indoor environ- ments. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition.
Ilya Sutskever, Oriol Vinyals, and Quoe V. Le. 2014. Sequence to sequence learning with neural networks. In NeurlPS.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research.
Iulia Turc, Ming- Biig-WWii Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint / 1906.08962v.
Ryokan Ri, Yufang Hou, Radu Marineseu, and Akihiro Kishimoto. 2022. Finding sub-task structure with natural language instruction. In ACL Workshop on Learning with Natural Language Supervision.
Andrea Vanzo, Danilo Croce, Emanuele Bastianelli, Roberto Basili, and Daniele Nardi. 2020. Grounded language interpretation of robotic commands through structured learning, Artificial Intelligence, 278:103181
Junha Roh, Karthik Desingh, Ali Farhadi, and Dieter Fox. 2022. Languagerefer: Spatial-language model for 3d visual grounding. In Conference on Robot Learning.
Shikhar Vashishth, Rishabh Joshi, Sai Suman Prayaga, Chiranjib Bhattacharyya, and Partha Talukdar. 201 8. Reside: Improving distantly-uuerrvised neural rela- tion extraction using side information. In EMNLP.
Kalyani Roy, Pawan Goyal, and Manish Pandey. 2021. Attribute value generation from product title using language models. In Proceedings of The 4th Work- shop on e C-Comnerce and NLP.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. Advances in neural infor- mation processing systems, 28.
Kalyani Roy, Tapas Nayak, and Pawan Goyal. 2022. Exploring generative models for joint attribute value extraction from product titles. arXiv preprint arXiv:2208.07130.
Tom Williams, Gordon Briggs, Bradley Oosterveld, and Matthias Scheutz. 2015. Going beyond literal command- -basdd instructions: Extending robotic nat- ural language interaction capabilities. In Proceedings of the AAAI Conference on Artificial Intelligence.
Nivedita Rufus, Kanishk Jain, Unni Krishnan R Nair, Vi- neet Gandhi, and K Madhava Krishna. 2021. Ground- ing linguistic commands to navigable regions. In 2021 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS).
Lu Xu, Hao Li, Wei Lu, and Lidong Bing. 2020. Position-aware tagging for aspect sentiment triplet extraction. In EMNLP.
Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2018. Extracting relational facts by an end-to-end neural model with copy mechanism. In ACI.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannanch Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. ICLR.
Lei Sha, Feng Qian, Baobao Chang, and Zhifang Sui. 2018. Jointly extracting event triggers and arguments by weeeeeddncy-bbeeeeeeeeeeeeeee
Sheng Zhang, Kevin Duh, and Benjamin Van Durme. 2017. MT/IE: Cross-lingual open information extrac- tion with neural sequence-to-sege -e-queence models. In EACL.
rnn and censor-based argument interaction. In AAAI.
Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. 2022. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. arXiv preprint 2rX
Yichi Zhang and Joyce Chai. 2021. Hierarchical task learning from language instructions with unified transformers and self mofitooni.., In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021.
Yatian Shen and Xuanjing Huang. 2016. Attention- based convolutional neural network for semantic re- lation extraction. In COLING.
Appendix
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke
We include the example of different task types, ar- gument types, and instructions in Table 8. Each

---

Task type
Argument types
Example instruction
being located (is)
source (table)
the cup is on
theme (living room),
this is a living room with green curtains
2. being in calegory (is)
category (with green curtains) recipient (me), theme (cup), source(table)
bringing (hring)
bring cup from the table
operational_staee (on).
opprraainngaa state (tum)
turn on television
device (television)
checking_stee state (check)
theme (stereo), desired_stad state (on)
please check if the stereo is
cutting (cut)
theme (apple), source (dining table)
cut the apple the dining table
following (follow)
cotheme (person), goal (kitchen)
follow the person to kitchen
theme (plate), agent (robot), recipient (me)
giving (pass)
robot you pass plate
9. inspecting (look)
manner (down), source (finor)
look down on the floor
10. motion (go)
goal (window)
go near the window
11. opening (open)
container portal (cabinet)
open the cabinet
take the bottle from the
12. picking (take)
theme (bottle), source (bedside table)
bedside table
placing (put)
theme (hottle), goal (trash)
trash can you push the on the table
put the bottle
theme (box). agent (you),
14. pushing (push)
source (table)
15. rotation (tum)
manner (your left), agent (robot)
robot turn to your left
16. searching (find)
recipient (me), theme (red shirt) find me the red shirt
Table Task and argument types in our dataset with example instruction the first two tasks are not real tasks. The phrase that represents the task and argument in each instruction are shown inside the bracket.
task and argument corresponds one/multiple words in the instruction. The corresponding words for the task and arguments are also marked (within the parenthesis).


Reviews:
[{"review_id": "Tcw39y7rGZ", "reviewer": "Reviewer_vT9w", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we thank the reviewer for the invaluable comments and suggestions. Please find our responses in the following.\n \n**Q1**: Missing References: Wu, Zhenyu, et al. \"Embodied Task Planning with Large Language Models.\" arXiv preprint arXiv:2307.01848 (2023). https://arxiv.org/pdf/2307.01848 might be worth taking into account in \"Related Work\". \n\n**Response**: Thank you for pointing to this related work. As this was published in Arxiv after the submission deadline, we were not aware of this work. We shall include this as part of the related work.\n\n**Q2**: Typos Grammar Style And Presentation Improvements\n\n**Response**: We are sorry for not paying greater attention to the proof reading and thankful to the reviewer for pointing these. We will address all the typos, grammar styles, presentation limitation, etc. in the final version.\n\n**Q3**: briefly explain what B, I and O stands for\n\n**Response**: In the BIO tagging scheme, B, I, O stands for beginning, inside, and outside respectively. We shall include this in the final version of the article with a pointer to the example given in Figure 2.\n\n**Q4**: “Each subtask … example.” Unclear what it refers to. \n\n**Response**: We have incorrectly used the term subtask. It is only tasks that we are referring to. There can be complex and compound instruction, and the instruction can have multiple tasks as listed Figure 4. We shall replace all the mention of subtask with task.\n\n**Q5**: “In our … inputs.” is a duplication of line 603-605. Delete\n\n**Response**: Thank you for pointing it out. We shall do it.\n\n**Q6**: Table 3: does “score without arg grounding” mean that task- and argument training is done separately? If yes, there is an inconsistency with what is said in line 554-555. Please, clarify.\n\n**Response**: There are three parts of the network – argument (object) grounding, task prediction, argument prediction. Training for task and argument prediction are always done together. If you look at Figure 2, the “annotation” row includes task and argument prediction along with their span is always predicted together. Now, for the “Placing” task, there is “Goal” argument. The token “fridge” is predicted as the “goal”. “Score without arg grounding” includes accuracy for the task, corresponding args and the span is matching with the ground truth. This does not include the accuracy of predicting (grounding) this “fridge” as “Refrigerator”. This argument grounding is necessary as we may refer the objects in different way. But, the object detector that the robot uses should be able to map the objects to the vision algorithm’s vocabulary. This is done with BIO tagging and later at post-processing matching the span. \n\nWe shall clarify this in the final version of the article.\n\n**Q7**: Section 4.1: When describing the dataset, add information on the distribution of single task and multi task instances relative to task types. Are the instances available per type approximately balanced?\n\n**Response**: We have provided an overall details in Table 2. We will include further break-ups of the dataset in the final version.\n\n**Q8**: As regards the evaluation, it is unclear whether the F1 scores relate to single train-test runs or to n-folds. In any case, report results from n-folds. \n\n**Response**: The current results depicts the F1 score of a single test score run as depicted in table 3 & 4. We have done train-test with 5 different seeds. The resulting F1 scores are – “without arg grounding”, it is 0.78 with 0.03 standard deviation (it was reported 0.79 in the paper);  “with arg grounding”, it is  0.66 with standard deviation 0.028 (it was reported 0.67 in the paper).\nWe shall replace all the results with average on multiple train-test runs.\n\n**Q9**: Also do an error analysis for the GATE model separating results for single and multiple tasks and task types. \n\n**Response**: We will get one extra page in the final version and use it for such analysis.\n\n**Q10**: Note: There are a few singular/plural and subject-verb incongruencies throughout the text. \n\n**Response**: As mentioned earlier, we shall do a thorough proof reading.\n\n**Q11**: Reconsider your ethics statement: just to give an example, when the model is used with real robots in real-world scenarios, performance quality matters. Therefore, users should be well aware of system limitations.\n\n**Response**: We agree with this. We shall add an ethics statement after deliberation with our ethical committee.\n\n**Q12**: The contribution depends on data that are simply not available outside the author's institution or consortium; not enough details are provided.\n\n**Response**: We shall definitely release the dataset along with all the necessary details. We shall also add a few more details of the dataset, e.g., task distribution. We may not be able to release the full code due to our organization’s policy. However, we shall help any researchers in reproducing the code and result. We shall add a line in the article regarding this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a neural approach and respective dataset to extracting subtasks and their arguments from complex NL instructions for robots. This is quite helpful for natural human-robot interaction. \nThe dataset should be published.", "reasons_to_reject": "The presentation of the evaluation results lacks some detail. See “Presentation Improvements”.", "questions_to_authors": "", "paper_topic_and_main_contributions": "An encoder-decoder architecture is presented that learns to extract subtasks and related arguments from compound natural language task instructions. The extracted tasks are mapped to the known skillset of a robot, the arguments to objects in the environment. Apart form the proposed neural model, a dataset for task-argument extraction is presented. However, it is unclear whether the authors intend to publish the dataset.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Wu, Zhenyu, et al. \"Embodied Task Planning with Large Language Models.\" arXiv preprint arXiv:2307.01848 (2023). \nhttps://arxiv.org/pdf/2307.01848\nmight be worth taking into account in \"Related Work\"."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 140: briefly explain what B, I and O stands for\nLine 448: typo “locatede”\nLine 449-452: “Each subtask … example.” Unclear what it refers to. Seems to be a remnant from a previous version of the paper.\nLine 610-613: “In our … inputs.” is a duplication of line 603-605. Delete.\nTable 3: does “score without arg grounding” mean that task- and argument training is done separately? If yes, there is an inconsistency with what is said in line 554-555. Please, clarify.\nSection 4.1: When describing the dataset, add information on the distribution of single task and multi task instances relative to task types. Are the instances available per type approximately balanced?\nAs regards the evaluation, it is unclear whether the F1 scores relate to single train-test runs or to n-folds. In any case, report results from n-folds. Also do an error analysis for the GATE model separating results for single and multiple tasks and task types.\nNote: There are a few singular/plural and subject-verb incongruencies throughout the text.\nNote that GATE in NLP is a well known acronym, see https://gate.ac.uk.\nReconsider your ethics statement: just to give an example, when the model is used with real robots in real-world scenarios, performance quality matters. Therefore, users should be well aware of system limitations."}}, {"review_id": "BwO0dcXZXW", "reviewer": "Reviewer_Tmep", "summary": "", "questions": "Same as Reasons To Reject", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s insightful feedback and recommendations. Our replies are given below.\n\n**Q1**: The dataset contains less than 2k examples, which may be too small. \n\n**Response**: Even though we have a relatively smaller dataset, it contains quite a lot of natural variation and combinations of multiple tasks. We have noticed that now we can add more instructions that would increase the absolute count, but not the variety in the instruction. \n\n**Q2**: In the experiment, the baseline models only contains CRF, BERT and T5. However, in order to demonstrate the effectiveness of the proposed model, it is beneficial to request results from ChatGPT, by constructing appropriate prompts. \n\n**Response**: We believe that BERT-CRF, BART, and T5 are strong baselines considering the application premise. We have to run this module along with a few more neural network models, like ASR, Vision, TTS for a robotics application, where network connectivity is not always available. These models have to run on edge device (or on-board). Thus, memory and processing power is a concern. Also, the overall system output has a time budget (real-timeness). Therefore, utilizing chatGPT is not feasible for our application.\n\n**Q3**: Could reproduce the results with some difficulty. The settings of parameters are underspecified or subjectively determined; the training/evaluation data are not widely available. \n\n**Response**: We shall definitely release the dataset along with all the necessary details. We shall also add a few more details of the dataset, e.g., task distribution. We may not be able to release the full code due to our organization’s policy. However, we shall help any researchers in reproducing the code and result. We shall add a line in the article regarding this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper curates a new dataset that enables grounded task-argument extraction.\n\nIn this scenario of Embodied Agents, this paper proposes to ground on the argument and the task simultaneously, and designs an encoder-decoded architecture that employs layered decoding. ", "reasons_to_reject": "The dataset contains less than 2k examples, which may be too small.\n\nIn the experiment, the baseline models only contains CRF, BERT and T5. However, in order to demonstrate the effectiveness of the proposed model, it is beneficial to request results from ChatGPT, by constructing appropriate prompts.", "questions_to_authors": "Same as Reasons To Reject", "paper_topic_and_main_contributions": "This paper curates a new dataset that enables grounded task-argument extraction, and proposes an encoder-decoder based NLU for Embodied Agents.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "YKk1zXPsyd", "reviewer": "Reviewer_MBSB", "summary": "", "questions": "- What is K in your dataset? Why is K = 2*(#objects) +2, instead of +1 (the O label)?\n- Were the two robotic commands datasets combined? How are they related?\n- If the paper is accepted, will the code and data be available?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer’s constructive comments and suggestions. We have addressed them in the following.\n\n**Q1**: Not detailed information on the annotation process in included (was it done programmatically? if yes, how? if not, who did it? how many annotators?). \n\n**Response**: The annotation is done in two stages. First, a group of junior annotators created the dataset including the instruction and annotation. A common schema is defined for the annotation. In the next stage, a senior annotator checked all the annotations carefully. The instructions were carefully chosen so that there is sufficient natural variation and does not does vary in one/two tokens.\n\n**Q2**: Some of the basic statistics that would be useful to understand the data are not reported (number of objects, number of instances per task type, whether there is overlap between the two datasets). \n\n**Response**: We will include such statistics about our dataset in the final version and will also release it for future research.\n\n**Q3**: The text explaining how the evaluation metric is computed is not very clear. The results+discussion section is too short and compressed. Error analysis is minimal, and the ablation only consists in varying the BERT encoder architecture. It would be useful to separately report the performance of each component (the object labeling, the task classification, the task span, the argument classification, the argument span), on each subtask, on each dataset. \n\n**Response**: Yes, we should have done better space management to include all these  results. For example, \nTask-wise accuracy looks like this – \n\n[Being_located - 0.968,  Being_in_category - 0.769, Bringing - 0.97, Changing_operational_state - 1.0, Checking_state - 1.0,  Closure - 0.903, Cutting - 1.0, Following - 1.0, Giving - 1.0, Inspecting - 0.182, Motion - 0.964, Placing - 1.0, Picking - 0.972, Pushing - 0.981, Rotation - 0.959, Searching 0.737]\n\nIn the final version, there is a provision of one extra page, which we shall utilize to describe the dataset and detailed results.\n\n**Q4**: The paper needs proofreading (see some examples of typos, unclear passages and incomplete references below). The limitation section text is duplicated. \n\n**Response**: We thank the reviewer for pointing these out. We will address all the typos, grammar, clarity and citation issues in the final version of the paper. We have fixed the text duplication in the limitation section.\n\n\n**Q5**: What is K in your dataset? Why is K = 2*(#objects) +2, instead of +1 (the O label)? \n\n**Response**: K is the total number of labels for BIO tagging. Hence, K should be 2*(#objects) + 1. \n\n**Q6**: Were the two robotic commands datasets combined? How are they related? \n\n**Response**: Yes, we have combined the two datasets. However, we have selected a subset of the text instructions from both the dataset setting the constraint that each instruction should contain task types from the task-set mentioned in the paper. \nBoth the datasets are for evaluation of a robotic agent to perform tasks given in natural language instructions. \n\nPlease note, we have added around 40% instructions on top of these combined dataset and annotated the entire set.\n\n**Q7**: If the paper is accepted, will the code and data be available? \n\n**Response**: We shall definitely release the dataset along with all the necessary details. We shall also add a few more details of the dataset, e.g., task distribution. We may not be able to release the full code due to our organization’s policy. However, we shall help any researchers in reproducing the code and result. We shall add a line in the article regarding this.\n\n**Q8**: Typos Grammar Style And Presentation Improvements: Clarifications and unclear or imprecise passages [046] I don't think that 'simplified' the tasks is a suitable verb here The first two subsections in the related work section are not very informative, as they contain lists of citations without further discussion or how exactly this paper fits in the current context. [l451] Which vertical lines? [l496] Overall what? [l525] Job does not sound as a proper term in this section. [538] What does it mean to emit the behaviour of sequence labelling? [l606] What does it mean 'not often not predicted'? Typos and Grammar: [l032] are often engage? [l061] actions [l102] extract [l201, l206] [l276] the inner [l277] responsible for generating [l288] refer [l448] located [l523] closing parenthesis missing [l553] & not appropriate here [l592] who misses? [l599] \n\n**Response**: \n\n[046] Agreed. We have changed the text segment to “...large scale language models have shown significant improvements in many NLP tasks (Devlin et al., 2019; Brown et al., 2020).”\n\nWe will make the first two subsections of related work more clear by discussing the relevance of the cited papers with our work. \n\n[l451] We overlooked this mistake in description. To clarify, we earlier visualized the different tasks in the example in Figure 2 with vertical lines, which we later removed but did not correct the text accordingly. \n\n[l496] This was a typo. We meant to say overlap instead of overall. We calculate the overlap in terms of word-level lexical similarity. \n\n[l525] Agreed, we used the term to avoid confusion with the term ‘task’. We have changed the text segment to “.. the model performs sequence labeling”\n\n[538] We meant to say that this models also has the same limitations of a sequence labeling model. We’ll rephrase this text in the final version. \n\n[l606] We mean to say that the decoder sometimes predicts lesser number of tasks and the corresponding arguments compared to the reference annotation, for very long inputs.\n\n[l032] - [l599] We shall fix the typos and grammar mistakes.\n\n\n**Q9**: In the future Style For the paragraph titles in Section 2, I suggest using the command \\paragraph{}. \n\n**Response**: We’ll fix this in the final version of the paper.\n\n**Q10**: When the citation is a part of the text, use \\citet instead of \\citep (for example, lines 144, 147, 150). Do not type the citation right before the citation itself (e.g. in l209, l216, l222, l224). Suggestion: use {} for the set notation. \n\n**Response**: We will do the needful.\n\n**Q11**: Algorithm 1 has no reference in the text. Please check the APA guideliens for table formatting. In particular, horizontal and vertical lines should be avoided. For instance, horizontal lines are only needed at the top and bottom and after the header. \n\n**Response**: We will take care of it.\n\n**Q12**: Some references are incomplete (e.g. Bahdanau 2015, Devlin 2019, Kundu 2018). Besides, as per ACL guideliens, arXiv versions should not be cited when the paper has been published in a venue (e.g. Blukis 2019, Huang 2022) \n\n**Response**: Noted; we shall add proper citation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes an end-to-end model that integrates multiple subtasks that ground NLU to the robot's environment representation (sequence labelling for objects, task type and span prediction, argument type and span prediction), with a detailed description of the implemented architecture.\n- Two existing datasets are adapted into a new format that allows joint training of these tasks.\n- Five baseline models are used for comparison of the performance of the new model.", "reasons_to_reject": "- Not detailed information on the annotation process in included (was it done programatically? if yes, how? if not, who did it? how many annotators?). Some of the basic statistics that would be useful to understand the data are not reported (number of objects, number of instances per task type, whether there is overlap between the two datasets). The text explaining how the evaluation metric is computed is not very clear.\n- The results+discussion section is too short and compressed. Error analysis is minimal, and the ablation only consists in varying the BERT encoder architecture. It would be useful to separately report the performance of each component (the object labelling, the task classification, the task span, the argument classification, the argument span), on each subtask, on each dataset.\n- The paper needs proofreading (see some examples of typos, unclear passages and incomplete references below). The limitation section text is duplicated.", "questions_to_authors": "- What is K in your dataset? Why is K = 2*(#objects) +2, instead of +1 (the O label)?\n- Were the two robotic commands datasets combined? How are they related?\n- If the paper is accepted, will the code and data be available?", "paper_topic_and_main_contributions": "The paper proposes a model to improve the alignment between the NLU component and the physical environment of a robot. This is realised as grounding tasks (intents) and arguments (objects) extracted from natural language instructions to the representations of the robot's environment.\n\nThe main contributions are a neural network-based architecture called GATE to extract intents and arguments from instructions and the adaptation of existing datasets to instantiate the proposed framing for the task.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Clarifications and unclear or imprecise passages\n- [046] I don't think that 'simplified' the tasks is a suitable verb here\n- The first two subsections in the related work section are not very informative, as they contain lists of citations without further discussion or how exactly this paper fits in the current context.\n- [l451] Which vertical lines?\n- [l496] Overall what?\n- [l525] Job does not sound as a proper term in this section.\n- [538] What does it mean to emit the behaviour of sequence labelling?\n- [l606] What does it mean 'not often not predicted'?\n\n\nTypos and Grammar:\n- [l032] are often engage?\n- [l061] actions\n- [l102] extract\n- [l201, l206]\n- [l276] the inner\n- [l277] responsible for generating\n- [l288] refer\n- [l448] located\n- [l523] closing parenthesis missing\n- [l553] & not appropriate here\n- [l592] who misses?\n- [l599] In the future\n\nStyle\n\n- For the paragraph titles in Section 2, I suggest using the command \\paragraph{}.\n- When the citation is a part of the text, use \\citet instead of \\citep (for example, lines 144, 147, 150).\n- Do not type the citation right before the citation itself (e.g. in l209, l216, l222, l224).\n- Suggestion: use {} for the set notation.\n- Algorithm 1 has no reference in the text.\n- Please check the APA guideliens for table formatting. In particular, horizontal and vertical lines should be avoided. For instance, horizontal lines are only needed at the top and bottom and after the header.\n- Some references are incomplete (e.g. Bahdanau 2015, Devlin 2019, Kundu 2018). Besides, as per ACL guideliens, arXiv versions should not be cited when the paper has been published in a venue (e.g. Blukis 2019, Huang 2022)"}}]

Author rebuttal:

Meta-review: This paper proposes a system for resolving and following compound (multi-task) instructions in a robot instruction following task. The approach also maps the extracted tasks and arguments to known robot skills and the surrounding environment. The paper proposes a formal annotation scheme to represent subtasks and referenced objects. Performance is quantified using label prediction accuracy (rather than downstream instruction following performance), as well as inference time as the end goal is to run inference on-device. Details on the annotation process, dataset statistics, and more analysis of model errors should be added to the next revision of the paper.
Decision: Accept-Findings