Dialogue Structure Parsing on Multi-Floor Dialogue
Based on Multi-Task Learning
Seiya Kawano1 , Koichiro Yoshino1,2 , David Traum3 , Satoshi Nakamura1,2
1
Graduate School of Science and Technology, Nara Institute of Science and Technology, Nara, Japan.
2
RIKEN, Center for Advanced Intelligence Project AIP, Tokyo, Japan.
3
USC Institute for Creative Technologies, Los Angeles, CA, USA.
{kawano.seiya.kj0, koichiro}@is.naist.jp, traum@ict.usc.edu, s-nakamura@is.naist.jp
Abstract
A multi-ﬂoor dialogue consists of multiple setsof dialogue participants, each conversing within
their own ﬂoor, but also at least one multicommunicating member who is a participant ofmultiple ﬂoors and coordinating each to achieve ashared dialogue goal. The structure of such dialogues can be complex, involving intentional structure and relations that are within or across ﬂoors.
In this study, we propose a neural dialogue structure
parser based on multi-task learning and an attention
mechanism on multi-ﬂoor dialogues in a collaborative robot navigation domain. Our experimental
results show that our proposed model improved the
dialogue structure parsing performance more thanthose of single models, which are trained on eachdialogue structure parsing task in multi-ﬂoor dialogues.

1

Identifying aspects of multi-ﬂoor dialogue structure can be
critical for building cooperative applications that have to participate in multi-ﬂoor dialogues, for example collaborative
navigation robots [Lukin et al., 2018; Bonial et al., 2018].
However, most existing studies on dialogue structure parsing addressed only single-ﬂoor dialogues. There are standard
annotation schemes for both dialogue acts [Bunt et al., 2012]and discourse relations [Prasad and Bunt, 2015] in singleﬂoor dialogues. Some proposed models have parsed the dialogue structure. However, these schemes do not fully address
the issues of dialogue structure in multi-ﬂoor dialogues. A
previous work proposed an annotation scheme of dialogue
structure on multi-ﬂoor dialogues [Traum et al., 2018]. This
scheme is based on two important aspects of dialogue structure: transaction units and the relations between utterances.
A transaction unit clusters utterances from multiple participants and ﬂoors that contribute to achieving the initiating participant’s intention. Relations link utterances to antecedents
within the unit. However, there is no previous work on automatic dialogue structure parsing for multi-ﬂoor dialogue.

Introduction

In single-ﬂoor dialogues, each participant can access any ofthe dialogue contents. For example, two people talking face
to face, or an online conference involving participants fromdifferent places is a single-ﬂoor dialogue because each participant can access all of the dialogue contents. By contrast, amulti-ﬂoor dialogue consists of multiple sets of dialogue participants, each conversing within their own ﬂoor, but also atleast one multi-communicating member who is a participantof multiple ﬂoors and coordinating each to achieve a shareddialogue goal. For example, in a restaurant, a server communicates with customers to take their orders in the dining
room (one ﬂoor) and talks with other workers in the kitchen(another ﬂoor) who prepare the customer’s food. All the participants work toward the joint goal of providing the customer
with their desired meals, however in this case, only the server
participates in both ﬂoors, conveying orders from customer
to kitchen and perhaps information about item availability or
speed from kitchen back to customers. Another example is in
military units, where soldiers follow their commander’s orders, which are decided at headquarters. Such situations arequite common in the real world, where we have different dialogue ﬂoors for decision-making and actions based on decisions.

In this paper, we propose a ﬁrst neural dialogue structure
parser for multi-ﬂoor dialogue structure. Our proposed parser
has an attention mechanism to predict structure across different ﬂoors. In the following sections, we describe the dialogue
structure parsing task on multi-ﬂoor dialogue, an annotation
scheme, and our target domain (Section 2). We describe our
proposed system based on the end-to-end approach, which
automatically identiﬁes the dialogue structure of multi-ﬂoor
dialogues by recurrent neural networks. The deﬁnitions oftransaction units, antecedents, and relation-types are closely
related to each other. We applied the attention mechanismand multi-task learning to improve the overall performanceof the dialogue structure parser considering their characteristics (Section 3). We experimentally evaluated the dialogue structure parsing performance of our model using automatic metrics that focus on micro- and meso-level structures[Traum and Nakatani, 1999] in dialogues (Section 4). Our
proposed model using multi-task learning improved the overall performance compared to models trained on single-task
settings (Section 5). Finally, we conclude by describing theperformance of our proposed model and discussing possible
future directions (Section 6).

Table 1: Dialogue example of multi-ﬂoor dialogue

#
1
2

3

Left Floor
Commander
DM → Commander
move to where you
see the ﬁrst cone
I m not sure which
object you are referring to. Can you describe it in anotherway, using color or
its location?
move to the cone on
the right a red coneon the right

4
5
6
7
8
9
10
11

2

DM → RN

Right Floor
RN

move to face thecone on the rightexecuting...
take another picture
done
done
image
image sent
sent

Dialogue Structure in Multi-ﬂoor Dialogue

For our initial investigations, we use a dataset of multiﬂoor dialogue structure, created as part of a long-term
project to develop an autonomous robot [Marge et al., 2016;
Lukin et al., 2018; Gervits et al., 2019], which is commanded by remote human participants. The robot is in an
unfamiliar physical environment, where it performs object
searches through natural language interaction. The dataset
consists of “Wizard of Oz” dialogues where two wizards control the robot and communicate with the human commander.
The dialogue manager wizard (DM) communicates directlywith the commander in natural language and handles clariﬁcations or misconceptions that might not be applicable given
the environment and robot capabilities. A robot navigator
wizard (RN) controls the robot with a joystick controller, but
communicates only with the DM. There are thus two separate ﬂoors - one between commander and “robot” (actually
the DM), and one between the two wizards. These ﬂoors arecalled “left” and “right”, for convenience. Table 1 shows anexample of an actual dialogue excerpt, including two ﬂoors
and four distinct message streams. The commander gives its
intention to the DM on their dialogue ﬂoor (left ﬂoor). The
DM talks with the commander (when necessary) to clarify
the commander’s intention. After completely understanding
the commander’s intention, the DM moves to another dialogue ﬂoor (right ﬂoor) to transfer the commander’s intention
to the RN, which operates the robot based on the given intention and reports the result to the DM. The DM returns to theﬁrst ﬂoor to feedback the result to the commander. Note thatthe DM can communicate with any participants by moving
among several dialogue ﬂoors to transfer the information asa multi-communicator [Reinsch Jr et al., 2008]; but the RN

TU
1Annotations
Ant Rel1

1

requestclariﬁcation

1

2

clariﬁcationrepair

13translation-r

1
2
1
1
2
2
2

3

ack-doing

4
7
6
9
10

ack-donetranslation-ltranslation-r
ack-donetranslation-l

and the commander cannot directly communicate.
Previous work deﬁned an annotation scheme for such
multi-ﬂoor dialogues to specify their characteristics[Traum et al., 2018]. To capture the information updateprocess of the dialogue participants, this scheme focusedon the intentional structure [Grosz and Sidner, 1986], which
consists of units of multiple consecutive utterances, and therelations between pairs of utterances within the unit. Theydeﬁned an annotation scheme for (1) transaction units, (2)antecedents, and (3) relation-types, and the dataset includes
human-annotated data. In this study, we explore a model that
automatically identiﬁes these structures. Below we describe
the annotation scheme in [Traum et al., 2018].

2.1 Transaction Unit
A transaction unit (TU) is a basic unit of intentional structure in a multi-ﬂoor interaction. It consists of the initial utterance that expresses the intention of the speakers and every
subsequent utterance across all the ﬂoors to achieve the original speaker’s intention. Each utterance belongs to a transaction unit, which is deﬁned by a set of utterances. The “TU”
column of Table 1 shows an numerical identiﬁer for the unit
which is the same for all utterances that are part of the TU.
In some cases, multiple transactions are “active” at thesame time, in that they have been initiated but not terminated. For example, Table 1 shows a case where two transaction units are included in the dialogue: TU1 is about moving
somewhere, while TU2 is about taking a picture. TU2 is initiated in utterance #6, before TU1 is completed in utterance#8. Both transactions are thus running in parallel during thispart of the dialogue.

2.2 Antecedent and Relation-Type

3

In [Traum et al., 2018], relations are annotated between utterances in the same TU, using antecedents and relation-types.
Any utterances after the ﬁrst utterance in the transaction unit
have antecedents, shown in the “Ant” column of Table 1, as
the utterance ID of the antecedent utterance. Relation types
are summarized in Table 2. These relations are categorizedﬁrst as to whether they are from the same participant (expansions), from different participants in the same ﬂoor, or acrossﬂoors. Each of these categories has a set of speciﬁc relations
and in some cases sub-types. Relation types are indicated in
the “Rel” column in Table 1.

In this section, we introduce a neural dialogue structure parser
for the annotation scheme proposed by [Traum et al., 2018].
A dialogue structure parser based on end-to-end neural networks improved the parsing performance more than legacy
models using hand-crafted features [Afantenos et al., 2015;
Shi and Huang, 2019]. Thus, we built an end-to-end neuraldialogue structure parser model with available data and explored its limitations.
In our dialogue structure parsing task on multi-ﬂoor dialogues, three tasks are closely related: transaction units, antecedents, and relation-type identiﬁcations. We expect thatmulti-task learning will improve the overall parsing performance more than single models. The attention mechanism
can explicitly represent their relations. Thus, our model isbased on a recurrent neural network that has both soft andhard attention mechanism with multi-task learning.
Our proposed model (Fig. 1) mainly includes four networks:• Hierarchical encoder has utterance and context encoders for encoding each dialogue context in different
dialogue levels.
• The antecedent predictor estimates the antecedent that
corresponds to each utterance.
• The transaction-unit predictor estimates the type oftransaction boundaries of each utterance.
• The relation-type predictor estimates the relation-typeof each utterance and its antecedent.
The transaction-unit and relation-type predictors share theprediction results of the antecedent predictor as attention
weights, because their prediction results are related to thepotential tree structures decided by the antecedent predictor
model. Such a two-stage approach, which predicts the dependency structure of the utterances and its relation-types,
resembles previous work [Shi and Huang, 2019]. However,that model targets single-ﬂoor dialogue structure parsing, andour model predicts the dialogue structure of multi-ﬂoor dialogues and clusters the utterances in different ﬂoors as one
transaction unit.

The set of relations within a transaction deﬁne a tree structure, where the ﬁrst utterance is the root node, which has no
relation-type or antecedent annotations. In the example in
Table 1, #1 and #6 are the root nodes of the two transaction
units.
Table 2: Relation-types in a multi-ﬂoor dialogue

Type
Expansions

Responses

Translations

Sub-types
relate utterances that are produced by thesame participant within the same ﬂoor.
continuelink-next
correction
summarization
relate utterances by different participants
within the same ﬂoor.
acknowledgment
done
doing
wilco
understandtry
unsurecan’tclariﬁcation
req-clarclar-repair
missing info
nackrepeatprocessing
question-responseanswernon-answerother
3rd turn feedback
reciprocal response
relate utterances in different ﬂoors.
transalation-ltransalation-r
comment
quotation

Neural dialogue structure parser for
Multi-Floor Dialogue

3.1 Hierarchical Encoder
Our hierarchical encoder consists of utterance and context encoders. The utterance encoder receives a word at each time
step using forward and backward GRUs [Cho et al., 2014] toencode each utterance into a ﬁxed-length vector:−→ −−→ −−−→
ht,i = GRUutt (ht,i−1 , Embedding(wt,i )),(1)←− ←−− ←−−−
(2)ht,i = GRUutt (ht,i+1 , Embedding(wt,i )),−→ ←−ht,i = [ht,i ; ht,i ],
(3)
|Ut |
1 ∑
ht =
ht,i .
|Ut | i=1(4)Here t is the utterance numbers in the dialogue context and iis the word order in the utterance. ht,i is the hidden vector calculated from each word wt,i and the hidden vector in previous

Figure 1: Overview of proposed neural dialogue structure parser

time-step ht,i−1 in utterance Ut = {wt,1 , wt,2 , · · · , wt,N }.
Each word wt,i is converted to a ﬁxed-length vector using
an embedding layer before calculating the hidden vector. In
each utterance, we added a special symbol, which indicates
the types of ﬂoors, to preﬁxes and sufﬁxes of utterance andtrained the embedding rule as done with words.
In the context encoder, utterance vectors are input to encode the dialogue history to get context-level vector representation h′t for each utterance in the dialogue contexts:−→′ −−→ −−→
ht = GRUhist (h′t−1 , ht )),(5)←−′ ←−− ←′−−
(6)ht = GRUhist (ht+1 , ht ),−→←−h′t = [ h′t ; h′t ].
(7)Weintroduceasoft-attention
mechanism[Luong et al., 2015] for dialogue contexts to computecontextual representation h̄attn
t for each utterance Ut :′attention(h′t−j , h′t ) = h′T
t−j Want ht ,exp(attention(h′t−j , h′t ))αj = ∑k′′j=1 exp(attention(ht−j , ht )h̄attn=tk∑

αj · h′t−j .

,

(8)
(9)

(10)

j=1

Here k is the number of previous utterances considered in
the calculation of attention, Wattn is a trainable weight-matrix,and αj ∈ [0, 1]k .
In addition, we introduce a hard-attention mechanism for
explicitly considering the antecedent, which corresponds toeach turn t:
h̄ant
t =

k
∑

βj · h′t−j

(11)

j=1

Here βj takes 1 if utterance Ut−j is the antecedent of utterance Ut and 0 in other cases (βj ∈ {0, 1}k ).

Attention vectors h̄attn
and h̄anttt , which are calculated on
the basis of the hard and soft-attention mechanisms, are combined:
attn
ant
′
ĥfc
(12)
t = tanh(Linearattn ([h̄t ; h̄t ; ht ]).
Here Linearattn is a linear transformation layer, which includes a bias term. ĥfc
t is a shared vector for predicting
the transaction units and relation-types. Note that gold antecedent β is used in training; however, in the inference, themodel uses predicted distribution of h̄antt by the antecedent
predictor.

3.2 Antecedent Predictor
As shown in Table 1, each utterance has an annotation of
the corresponding antecedent as its utterance ID (#). To predict the antecedents for each utterance Ut , we calculated thescores between each utterance and the contextual utterances:′(13)
antecedent(h′t−j , h′t ) = h′T
t−j Want ht ,exp(antecedent(h′t−j , h′t ))βˆj = ∑k.
′′j=1 exp(antecedent(ht−j , ht )(14)Here, k is the number of preceding utterances that can be
the antecedent, Want is a trainable weight-matrix, and βˆj ∈[0, 1]k . By calculating the position of antecedent from theweights of attention, we can carry this knowledge forward toother predictions in the rater step: transaction-unit prediction
and relation-types prediction.
We set the cross-entropy loss between predicted distribution β̂ and actual antecedent label β as a loss function thatenforces that the contextual utterance has the highest scorewhen it is the antecedent of Ut :k∑
Lt,ant = −(15)
βj log(βˆj ).
j=1

Note that we also calculate the attention weight corresponding to the case where the utterance does not have anyantecedent using the trainable vector and the hidden vectorh′t .

3.3 Transaction-Unit Predictor

4

We formulate the problem of transaction unit prediction asa sentence classiﬁcation problem that determines the boundaries of the transaction units in dialogues. The transactionunit predictor classiﬁes each utterance into the following
three classes:• Start: the utterance is the beginning of a transactionunit.
• Continue: the utterance belongs to the same transaction
unit as the previous utterance.
• Other: the utterance cannot be categorized into either of
the above classes.
Other indicates that the utterance belongs to an already opentransaction that is different from the one the previous utterance belongs to, such as utterance #7 and #9 in Table 1. We
predict transaction boundaries using ĥfc
t , derived from the calculation results of soft and hard-attentions to the context:
fc
p̂tu
(16)
t = softmax(Lineartu pred (ĥt )).
Here Lineartu pred is a linear transformation layer that includes
a bias term, and p̂tu
t is the predicted distribution of the transaction boundaries.
We used the cross-entropy loss as the loss function:

In our experiment, we evaluated the dialogue structure parsing performance of our proposed model. In this section, we
describe the dataset for the training and evaluation, the settingof the model training, and the evaluation metrics.

|pt |
∑
tu

Lt,tu = −tu
ptu
j log(p̂t,j ).

(17)

j=1

Here ptu
t is a three-dimensional vector corresponding to thetype of target transaction boundaries.

3.4 Relation-Type Predictor
We used ĥfc
t as well as the transaction-unit predictor to predictthe relation-type of each utterance with its antecedent:
fc
p̂rel
(18)
t = softmax(Linearrel pred (ĥt )).
Here Linearrel pred is a linear transformation layer that includes the bias term and p̂relt is the predicted distribution of
the relation-types.
We used the cross-entropy loss for the training:
|prel
j |

Lt,rel = −

∑

rel
prel
j log(p̂t,j ).

(19)

j=1

Here prelt is a vector, whose dimensions correspond to a relation label deﬁned in Table 2.

3.5 Objective Function
We have to optimize the above three models not only to a single model but also to the other two models because these tasks
are closely related. In this study, we introduce a multi-task
loss, which combines each prediction loss of the antecedent,
the transaction-unit, and the relation-type predictor. In multitask learning, we interpolate the loss functions of three tasks:N
1 ∑L=(γant Lt,ant + γtu Lt,tu + γrel Lt,rel ).
(20)
N t=1
Here N is the dialogue length. γant , γtu , and γrel are theweights for adjusting the importance of each predictor in theloss calculation.

Experimental Settings

4.1 Dataset
We used a dataset [Traum et al., 2018] that contains Exp. 1
and Exp. 2 data1 . The dialogues were annotated based on apreviously described scheme [Traum et al., 2018], which was
speciﬁcally designed to handle multiple conversational ﬂoors.
As shown in Table 3, these dialogue data consist of 48 dialogues (1829 transactions) executed by several different commanders.
Table 3: Numbers of dialogues, utterances, and transactions

Exp. 1
Exp. 2

Dialogues
24
24

Utterances
4527
6994

Transactions
780
1049

To evaluate the parsing performance of the proposedmodel, we randomly divided all of the dialogues in Exp. 1 andExp. 2 into six subsets and applied double cross-validation[Mosier, 1951]. We used a single subset for validation and atest-set for each, and the remaining subset was used as training data. We evaluated every possible combination of training, validation, and test-set and the ﬁnal performance by amajority vote on the prediction results of the models, whichshare the same test-set.

4.2 Model Settings
We evaluated the dialogue structure parsing performance ofthe proposed model in multi-ﬂoor dialogues by comparing
the dialogue structure parsing performances of the proposedmodel with the multi-task loss (Multi) and the models individually trained for each task (Single). We also compared thecases based on both the Ofﬂine and Online models. The proposed model described in Section 3 uses bi-directional GRUs
in the context encoder to make predictions for each utterance
Ut ; this means the model cannot start parsing during the dialogue. We call this setting Ofﬂine. In contrast, we also considered a model that only uses previous contexts without subsequent contexts in the prediction for each utterance Ut . We
call this setting Online. The online model is important for
real-time dialogue robot processing, which can only use theobserved information based on the interaction sequence. We
built the online model only using forward-GRUs instead ofbidirectional-GRUs in the context encoder.
We used the same hyper-parameter settings in each model.
The vocabulary size was 500, the word embedding size was100, and the hidden size was 300. We used byte pair encoding (BPE) for tokenization [Sennrich et al., 2016]. In training, we used a mini-batch size of 64 and an Adam optimizer [Kingma and Ba, 2014] with a learning rate of 1e-4.
We set γant , γtu , and γrel to 1. In the relation-type prediction, we integrated the ‘acknowledgement,” “clariﬁcation,”1

The annotation for Exp. 3 is still in progress.

Table 4: Prediction performances of transaction units, antecedents, and relation-types

Models

Prec.

Rec.

TU
F1

TuAcc

Prec.

Rec.

F1

TreeAcc

Prec.

Rec.

Rel
F1

TreeAcc w/ rel

Majority
Single-Online
Multi-Online

55.71
95.43
95.99
(96.26)95.31
96.05(96.30)

74.64
95.46
95.99
(96.27)95.35
96.07(96.31)

63.80
95.44
95.99
(96.26)95.33
96.06(96.30)

81.19
84.25
(85.34)
81.46
84.52(85.51)

23.59
93.92
93.93
94.26
94.58
-

48.57
90.89
90.84
90.70
91.95
-

31.76
92.34
92.33
92.40
93.21
-

68.12
70.09
68.83
71.35
-

8.54
92.38
93.69
(94.75)94.86
93.75(94.68)

29.22
92.84
94.11
(94.94)93.22
94.26(94.95)

13.21
92.53
93.80
(94.77)92.91
93.90(94.69)

63.80
66.81
(67.74)
64.62
69.05(69.92)Single-Ofﬂine
Multi-Ofﬂine

Antand “question-response” sub-types into these classes becausesome sub-types rarely appeared in the dataset. In addition,
we deﬁned a label where utterance has no antecedent, as well
as relation-types (#1 and #6 in Table 1).

4.3 Evaluation Metrics
We deﬁned the micro and meso-level evaluation metrics forour dialogue structure parsing task. For the micro-level evaluation, we deﬁned the label prediction performances of the antecedents, the transaction units, and the relation-types by precision (Prec.), recall (Rec.), and F1. Note that we took the relative position of each utterance from its antecedent as a label
to compute the metrics when evaluating the antecedent prediction performance. In other words, we compared the difference between the position of predicted antecedents and actual
antecedents. We also introduced metrics for the meso-levelstructure [Traum and Nakatani, 1999] in dialogues to evaluate the consistency of the parsing results. We used the following three metrics:• TuAcc is the ratio of the transaction units that perfectlypredicted the transaction boundaries for each utterance
within the transaction unit.
• TreeAcc is the ratio of the transaction units that perfectlypredicted the antecedents for each utterance within thetransaction unit.
• TreeAcc w/ rel is the ratio of the transaction units
that perfectly predicted the antecedents and the relationtypes for each utterance within the transaction unit.
Note that the meso-level metrics are stricter than the microlevel metrics, which judge the prediction result of each utterance.

5

Experimental Results

Table 4 shows the performances of each dialogue structure
parser. Here Single denotes a case where the transaction unit,
the antecedent, and the relation-type predictors were individually trained. Multi denotes a case where these models weretrained with multi-task learning loss. In addition, Ofﬂine indicates that the model used the dialogue entirely for parsing,and Online indicates that the model used only the preceding
context of each utterance. Majority denotes a case where thealways predicts the most frequent label. Prec., Rec., and F1

are the weighted averages2 of the precision, recall, and F1scores of the predicted labels. The brackets are the prediction
results where the oracle antecedent was fed into the model.
The result shows that the dialogue structure parsing performance (transaction unit prediction, antecedent prediction, andrelation-types prediction) of the Ofﬂine models have slightly
improved from the Online models. This result indicates that
subsequent contexts are useful for predicting labels for eachutterance, but we have enough prediction accuracy without
using the subsequent context. When the Multi models use
oracle antecedents to predict the transaction units and therelation-type, we can further improve the performance of dialogue structure parsing.
Table 5: Transaction unit prediction performance of Multi-Ofﬂine
model
Tu-LabelStart
Continue
Other
Weighted-Avg

Precision
0.9466
0.97500.86940.9605

Recall
0.93060.98000.85910.9607F1
0.93850.97750.86420.9606

Count
1829
8599
1093
11521

Table 5 shows the results of the transaction unit predictionin Multi-Ofﬂine model, which was the best model in our experiment. We conﬁrmed that the model achieved over 85%
F1 for all labels (types of transaction boundary). However,there is still a problem in the predicting “Other” (when the utterance belongs to a different transaction than the previous utterance, but it is not the start of a new transaction). Our model
decided labels with the highest prediction probability for eachutterance; however, we did not take into account the consistency of the prediction results in the sequence of dialogue. To
solve this problem, we can introduce a model that takes into
account information about the entire prediction results, suchas Conditional Random Field (CRF) [Lafferty et al., 2001]
for further improvements.
Table 6 shows the results of the antecedent prediction in
the Multi-Ofﬂine model. Here each label indicates the relative position from each utterance to its antecedent. Note that
this table only shows the prediction results by considering amaximum of 10 previous utterances. We excluded a few cases
2
We calculated the weighted averages based on the label frequencies.

Table 6: Antecedent prediction performance of Multi-Ofﬂine model
Position
-10
-9
-8
-7
-6
-5
-4
-3
-2
-1
Weighted-Avg

Precision
58.33
73.08
82.50
80.77
90.48
81.31
91.87
94.98
94.84
95.99
94.58

Recall
66.67
57.58
63.46
74.12
85.39
64.44
85.32
87.00
93.06
95.47
91.15

F1
62.22
64.41
71.74
77.30
87.86
71.90
88.48
90.82
93.94
95.73
93.21

Count
21
33
52
85
178
135
477
1023
2509
4262
8775

when the antecedent of the utterance is not included in the ten
previous utterances, and the utterance has no antecedent. Ourmodel can predict antecedents with high performance when
the relative position was not distant. On the other hand, theprediction performance was below 80% when the relative positions were distant (greater than ﬁve in absolute). This result
suggests the difﬁculty of addressing long-term dependency indialogues. In addition, our model ignores the consistency of
the tree structure associated with the predicted antecedents.
The search for dialogue structures using dynamic programming probably has the potential to improve the performanceof our model.
Table 7: Relation-type prediction performance of Multi-Ofﬂine
model
Relation-Type
Expansions-continue-link-next-correction-summarization
Responses-acknowledgement-clariﬁcation-processing-question-answer-other-3rd-turn-feedback-reciprocal-response
Translations-l-r-comment
No-antecedent
Weighted-Avg

Precision

Recall

F1

Count0.90900.99370.40000.00

0.88900.99690.11110.00

0.89890.99530.1739
0.00

955
318
36
13

0.97290.79511.00000.57050.33330.50000.00

0.97060.85000.99570.51740.06060.04000.00

0.97170.8216
0.99780.54270.10260.0741
0.00

3366
420
233
172
33
25
5

0.95930.98300.40000.91850.93750.98140.98400.38100.94630.94260.97030.98350.39020.93220.9390

1563
1942
21
2419
11521

Table 7 shows the results of the relation-type predictionsin the Multi-Ofﬂine model. Our model showed higher F1scores in frequent relation-types, including when the utterance has no antecedent. There is still a challenge in predicting low-frequent relation-types due to the lack of training
data. Ongoing annotation work [Traum et al., 2018] with additional data may remedy this problem. We also need to lookat ways to deal with these unbalanced labels.

Finally, in Table 8, we show an example of dialogue structure parsing on a fragment of multi-ﬂoor dialogue. Note that
we displayed the correct labels in brackets when the label wasincorrectly predicted, and “#” corresponds to cases where theutterance does not have the antecedent. The ﬁrst example
shows that the model accurately predicts all the transaction
boundaries, antecedents, and relation-types, even if transactions were interleaved. However, the second example is including error predictions of transaction boundaries. In thisexample, there are only two TUs, but the model has determined that the utterance has three TUs. Note that, even if we
assume the prediction of TU at #8 is correct, the prediction
at #11 is still not correct. When such confusion occurs, theerror extends beyond one utterance to multiple utterances. In
many cases, delays in communication and differences in thequality of annotations between Exp.1 and 2 often confuse predictions.

6

Conclusion

We built a neural dialogue structure parser with an attention mechanism that applies multi-task learning to automatically identify the dialogue structure of multi-ﬂoor dialogues.
The experimental results showed that our proposed model
improved the identiﬁcation performance on all tasks compared to the model trained on single task settings. However, problems remain with the performance of the dialogue
structure identiﬁcation due to the lack of training data, especially for rare labels. To prevent this problem, we willconsider pre-training and the transfer learning of models using existing dialogue corpora and discourse-relation datasets.
We also explore the possibility of introducing powerful models of similar tasks related for predicting tree-structure ina document, such as a dependency parsing [Nivre, 2010]
and discourse parsing based on rhetorical structure theory[Webber et al., 2012; Mann and Thompson, 1988].
This study has developed the ﬁrst baseline model for automatic identiﬁcation of dialogue structure on multi-ﬂoor dialogues. It has a potential for applying to the automatic annotation of dialogue structure on multi-ﬂoor dialogues, andencourage the development of a dialogue manager and robot
navigator on multi-ﬂoor settings.

Acknowledgments
This research and development work was supported by Grantin-Aid for JSPS Fellows No. 20J14823. We would like tothank United States Army Research Laboratory for sharing
the dataset for this research. Author Traum was supported inpart by the Army Research Ofﬁce under Cooperative Agreement Number W911NF-20-2-0053. The views and conclusions contained in this document are those of the authors andshould not be interpreted as representing the ofﬁcial policies,
either expressed or implied, of the Army Research Ofﬁce or
the U.S. Government. The U.S. Government is authorized
to reproduce and distribute reprints for Government purposes
notwithstanding any copyright notation herein.

Table 8: Examples of the dialogue structure parsing on multi-ﬂoor dialogue#
1

Left Floor
Commander
DM → Commander
turn
right
twenty degrees

2
3
4
5
6
7
8

turn right 20
image
done
sent

image

go forward ﬁfteen feetsent
and go throughdoor on rightmove forward
about 15 feet ,going throughdoor on right ,image
executing ...
take a picture
image
image sent
sent
turn left ninety
degrees

6
7
8

TU
Start

executing ...

9

10
1
2
3
4
5

Right Floor
DM → RN
RN

turn left 90
executing ...
take a picture
after each command

9

done

10

take pic after
each command

11

image

12
13

image sent
sent

References[Afantenos et al., 2015] Stergos Afantenos, Eric Kow,
Nicholas Asher, and Jérémy Perret. Discourse parsing for
multi-party chat dialogues. In Proceedings of the 2015
Conference on Empirical Methods in Natural Language
Processing, pages 928–937, Lisbon, Portugal, September
2015. Association for Computational Linguistics.
[Bonial et al., 2018] Claire Bonial, Stephanie M Lukin, Ashley Foots, Cassidy Henry, Matthew Marge, Kimberly A
Pollard, Ron Artstein, David Traum, and Clare R Voss.
Human-robot dialogue and collaboration in search andnavigation. In Proceedings of the Annotation, Recognition
and Evaluation of Actions (AREA) Workshop of the 2018
Language Resources and Evaluation Conference (LREC),
2018.
[Bunt et al., 2012] Harry Bunt, Jan Alexandersson, Jae-

Prediction
Ant
#

Rel
#

Continue
Continue
Continue
Continue

1
1
1
4translation-r
response-ack.
translation-r
response-ack.

Start

#

#

Other
Other

5
6

translation-l
expansion-cont.

Continue

8translation-r

Continue
Start
Continue
Continue
Continue
Start

8
#
1
2
3
#

response-ack.
#translation-r
response-ack.
translation-l#

Continue
Continue
Start(Continue)

5
5
#
(5)

translation-r
response-ack.
#(expansion-cont.)

6

response-ack.

8translation-r

8translation-r

1112

response-ack.
translation-l

Other(Continue)
Other(Continue)
Other(Continue)
Continue
Continue

Woong Choe, Alex Chengyu Fang, Koiti Hasida, Volha
Petukhova, Andrei Popescu-Belis, and David R Traum.
ISO 24617-2: A semantically-based standard for dialogueannotation. In Proc. of Conference on International Language Resources and Evaluation, pages 430–437, 2012.
[Cho et al., 2014] Kyunghyun Cho, Bart van Merrienboer,
Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares,
Holger Schwenk, and Yoshua Bengio. Learning phrase
representations using rnn encoder–decoder for statisticalmachine translation. In Proc. of EMNLP, pages 1724–
1734, 2014.
[Gervits et al., 2019] Felix Gervits, Anton Leuski, Claire
Bonial, Carla Gordon, and David Traum. A ClassiﬁcationBased Approach to Automating Human-Robot Dialogue.
In Proceedings of International Workshop on Spoken Dialog System Technology (IWSDS), page 12, Siracusa, Italy,
April 2019.

[Grosz and Sidner, 1986] Barbara J Grosz and Candace L
Sidner. Attention, intentions, and the structure of discourse. Computational linguistics, 12(3):175–204, 1986.
[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba.
Adam: A method for stochastic optimization. arXiv
preprint arXiv:1412.6980, 2014.
[Lafferty et al., 2001] John Lafferty, Andrew McCallum,and Fernando CN Pereira. Conditional random ﬁelds:
Probabilistic models for segmenting and labeling sequence
data. 2001.
[Lukin et al., 2018] Stephanie Lukin, Felix Gervits, Cory
Hayes, Pooja Moolchandani, Anton Leuski, John G
Rogers III, Carlos Sanchez Amaro, Matthew Marge, ClareVoss, and David Traum. Scoutbot: A dialogue systemfor collaborative navigation. In Proceedings of ACL 2018,
System Demonstrations, 2018.
[Luong et al., 2015] Minh-Thang Luong, Hieu Pham, andChristopher D Manning.
Effective approaches toattention-based neural machine translation. arXiv preprint
arXiv:1508.04025, 2015.
[Mann and Thompson, 1988] William C Mann and Sandra A
Thompson. Rhetorical structure theory: Toward a functional theory of text organization. Text, 8(3):243–281,
1988.
[Marge et al., 2016] Matthew Marge, Claire Bonial, Kimberly A Pollard, Ron Artstein, Brendan Byrne, Susan G
Hill, Clare Voss, and David Traum. Assessing agreementin human-robot dialogue strategies: A tale of two wizards.
In International Conference on Intelligent Virtual Agents,
pages 484–488. Springer, 2016.
[Mosier, 1951] Charles I Mosier. I. problems and designs ofcross-validation 1. Educational and Psychological Measurement, 11(1):5–11, 1951.
[Nivre, 2010] Joakim Nivre. Dependency parsing. Language
and Linguistics Compass, 4(3):138–152, 2010.
[Prasad and Bunt, 2015] Rashmi Prasad and Harry Bunt. Semantic relations in discourse: The current state of iso24617-8. In Proceedings of the 11th Joint ACL-ISO
Workshop on Interoperable Semantic Annotation (ISA-11),2015.
[Reinsch Jr et al., 2008] N Lamar Reinsch Jr, Jeanine Warisse Turner, and Catherine H Tinsley. Multicommunicating: A practice whose time has come?
Academy of Management Review, 33(2):391–403, 2008.
[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, andAlexandra Birch. Improving neural machine translation
models with monolingual data. In Proc. ACL, volume 1,
pages 86–96, 2016.
[Shi and Huang, 2019] Zhouxing Shi and Minlie Huang. A
deep sequential model for discourse parsing on multi-party
dialogues. In Proceedings of the AAAI Conference on Artiﬁcial Intelligence, volume 33, pages 7007–7014, 2019.
[Traum and Nakatani, 1999] David Traum and Christine H
Nakatani. A two-level approach to coding dialogue for discourse structure: activities of the 1998 dri working groupon higher-level structures. In Towards Standards and Tools
for Discourse Tagging, 1999.
[Traum et al., 2018] David Traum,
Cassidy Henry,
Stephanie Lukin, Ron Artstein, Felix Gervits, Kimberly Pollard, Claire Bonial, Su Lei, Clare Voss, Matthew
Marge, et al. Dialogue structure annotation for multi-ﬂoor
interaction. In Proceedings of the Eleventh International
Conference on Language Resources and Evaluation(LREC 2018), 2018.
[Webber et al., 2012] Bonnie Webber, Markus Egg, andValia Kordoni. Discourse structure and language technology. Natural Language Engineering, 18(4):437, 2012.