Who is Speaking? Speaker-Aware Multiparty Dialogue Act Classification
Authors: Ayesha Qamar
Adarsh Pyarelal
Ruihong Huang
Venue: EMNLP 2023 Findings (2023)
Keywords: Dialogue acts
multiparty dialogues
speaker modeling
TL;DR: Explicitly modeling speakers in multiparty dialogues helps with dialogue act classification.

Abstract: Utterances do not occur in isolation in dialogues; it is essential to have the information of who the speaker of an utterance is to be able to recover the speaker’s intention with respect to the surrounding context. Beyond simply capturing speaker switches, identifying how speakers interact with each other in a dialogue is crucial to understanding conversational flow. This becomes increasingly important and simultaneously difficult to model when more than two interlocutors take part in a conversation. To overcome this challenge, we propose to explicitly add speaker awareness to each utterance representation. To that end, we use a graph neural network to model how each speaker is behaving within the local context of a conversation. The speaker representations learned this way are then used to update their respective utterance representations. We experiment with both multiparticipant and dyadic conversations on the MRDA and SwDA datasets and show the effectiveness of our approach.

Who is Speaking? Speaker-Aware Multiparty Dialogue Act Classification
Ayesha Qamar', Adarsh Pyarelal and Ruihong Huang"
*Texas A&M University, *University of Arizona (ayesha, huangrh ) @ @tamu.edu, adarsh arizona.edu
Abstract
Speaker
Utterance
Label
Utterances do not occur in isolation in dia- logues; is essential to have information of who the speaker of an utterance is be able to recover the speaker's ss intention with respect to the surrounding context. Beyond simply captur- ing speaker switches, identifying how speakers interact with each other in a dialogue cru- cial understanding conversational flow. This becomes increasingly important and simultane- ously difficult to model when more than two interlocutors take part in conversation. To overcome this challenge, we propose to explic- itly add speaker awareness to each utterance representation. To that end, neural network to model how each speaker is behaving within the local context a conversa- tion. The speaker representations learned this way are then used to update their respective utterance representations. We experiment with both multiparticipant and dyadic conversations on the MRDA and SwDA datasets and show the effectiveness of our approach.
A
and i've been sharing them with u folks also.
Statement
B
i'm sorry you'l 've been what?
Understanding check
C
okay.
Backchannel
B
showing them?
Understanding check
A
sharing them with the w folks,
Correct Misspeaking
B
sharing them.
Mimic
B
okay.
Acknowledgement
B
okay.
Backchannel
C
okay.
Floor Grabber
graph
Table 1: A short excerpt from dialogue involving multiple speakers from the MRDA corpus.
One example of this is the broader category of the Question DA, which, along with seeking in- formation, also implicitly gives the floor to the addressee. Therefore, interlocutor information is needed to instill situational awareness of speakers in a conversation. In a dialogue, speakers influence each other which in turn dictates how they behave and convey their intentions. Different speakers can also fulfill different roles, for example, within the local context of conversation. a speaker can be leading the conversation by providing information while others could be active listeners and signal they are following along by means of backchannels as shown in Table
1 Introduction
Dialogue acts (DAs) describe the intention behind a speaker's s utterance in a conversational context (Searle, 1969). Recognizing DAs is helpful for sev- eral downstream tasks such as question-answar answering (Goo and Chen, 201 8a), summarization (Goo and Chen, 2018b; Oya and Carenini, 2014), and conver- sational AI (Xu al., 2018; Wang al., 2020). Utterances are the building blocks of a conversa- tion, but they do not occur in isolation. To recover the communicative function behind an utterance, we need to consider its context. The context here refers not just to previous utterances, but also to dialogue structure, speaker behavior, etc. Utter- ances are multifunctional in the sense that they encode many roles-es e.g. turn management, relay- ing communicative intentions, etc. (Bunt, 2006).
There has not been much prior work on incor- porating speaker awareness for DA classification, particularly in the case of multiparty conversations (i.e., conversations involving more than two inter- locutors). While there is some research modeling speakers for DA classification, it is mostly focused on two-party conversations. Chi al. (2017) use a separate BiL.STM model each speaker in a dyadic setting. Although this one way to capture speaker behavior, does not scale to the case of multiparty
'Code available at https://ggittbb..mm//ysssaass Speaker-gras geiieerr-rapp-aae model Dors--------D--LLL.

---

dialogues and real-world situations where the num- ber of speakers can vary at lest time. Recently, He et al. (2021) introduced a way to capture speaker turns by learning turn embeddings. For this to work in a multiparty setting, the dialogue needs to be reduced to dyadic one. This is not ideal since doing this would take away crucial information on different participants' contributions.
utterances. To capture context, hierarchical RNN with some variation used. DA label sequences can help learn associations between tags that occur together or follow a pattern; conditional random fields (CRFs) have extensively been used for this purpose. Additionally, Raheja and Tetreault (2019) use attention to leverage context more effectively. To make utterances speaker turn-aware, He al. (2021) learn two turn embeddings to capture turns. The embedding vector based on the speaker label is added to the utterance vector and then passed to another RNN to capture context. Whereas Shang et al. (2020) use CRF layer to model turn-taking, While this is an effective way to inject speaker infor- mation, multiparty dialogues need to be converted to dyadic dialogues for these approaches to work.
In the related field of emotion recognition in dialogues (ERC), some work has been done to encode speakers (Lee and Choi, 2021; Ghosal et al.. 2019; Zhang et al., 2019; Song et al., 2023). These works primarily use graph neural networks (GNNs) to enhance the representation of utterances. Speaker information is injected either through edges or by adding speaker nodes the graph. Since the main objective of these methods is to encode utterances better, often different types of edges are introduced to better capture the different relationships between utterances. For example, different types of edges for past and future connections (Ishiwatari et al., 2020) or learning dedicated edge types for each individual speaker (Ghosal et al., 2019). This translates to learning a bigger model with more parameters as well as denser graph with many edges. Due to the large memory requirements of such models, training them can be hard in a resource-scarce situation. We show in our work that instead of using the speaker as a means to capture utterance context better, directly learning the speaker representations is more efficient. This results in simpler graph with fewer edges.
Chi et al. (2017) work in the setting of conversa- tional agents and aim to capture the speaker roles of the agent and the user using a separate BiLSTM for each. This technique cannot be applied to natural conversations where the number of participants is not fixed.
Colombo et al. (2020) use three RNN encoders: word-level, speaker-level, and utterance-level. The speaker-level RNN is fed utterances grouped by speaker to capture speaker personas. The repre- sentations from this encoder are then fed to the utterance-level encoder for wider context modeling. Their work differs from ours in the way that they model each speaker. While they encode utterance context at the speaker level, we learn an explicit speaker representation.
In this work, we present a GNN-based frame- work that takes contextual utterance vectors as input and encodes conversation dynamics by connecting the node of each speaker with their utterances. The learned speaker representations are then con- catenated with the utterance representations get speaker-enriched utterance representations which are then used for DA classification. We conduct extensive analysis on the ICSI Meeting Recorder Dialog Act (MRDA) corpus (Shriberg et al., 2004) to understand how the fine-grained DA classes are affected by incorporating speaker representations.
Graph-based methods to encode speakers in di- alogues We discuss some of the work here that uses GNNs (Scarselli et al., 2008) for emotion recognition in conversations. Ghosal et al. (2019) build graph with only utterances nodes and incorporate context at the speaker level using dif- ferent edges. There is separate directed edge type from each speaker every other speaker while also differentiating the directionality of past and future. Thus, there are 2M edge types in total, where M is the number of unique speakers. This technique does not scale in cases where there are more speakers in dialogue at test time than the maximum number of speakers seen during training. Shen al. (2021b) and Sheng et al. (2020) con- struct a dialogue graph by only using utterance nodes. Two different edge types are used to indicate if two connected utterances share the same speaker or not. In some ways, this is akin to converting a
2 Related Work
DA classification Most existing works on DA clas- sification (Kumar et al., 2018; Chen et al., 2018; Raheja and Tetreault, 2019; Bothe et al., 2018; Khanpour et al., 2016) use recurrent neural net- works (RNNs) as backbone models to represent

---

multiparty dialogue to a two-party one, since the utterances only have binary information of speakers, Some other works construct the graph with both speaker and utterance nodes. This includes Lee and Choi (2021), who treat ERC as relation extraction task through a GNN. While Zhang et al. (2019) share speaker nodes across dialogues, we are more interested in capturing the dynamics of speaker influence within a local context and assume that any new speaker can be seen during test time. Liang al. (2021) learn fixed emheddings for each unique speaker and use them to initialize the speaker nodes. One thing common to all these papers is that they explicitly learn utterance node representations through the graph (Sun et al., 2021; Song et al., 2023)-i.e., they build the graph representations as means get better utterance representations. The speaker nodes serve global nodes for effective message passing between utterances. This not only results a very dense graph but also increases other complexities by means of introducing dif- ferent edge types based on speaker associations. In contrast, we are interested in learning speaker representations and how using those can enrich the utterances. Doing only requires connecting the speaker to its utterance nodes. We present simple graph construction scheme and show that modeling speakers directly not only efficient but effective as well. Our proposed approach can be used on lop of any pre- traitrained utterance encoding system.
al. (2019) model used to derive a representation e(u,) for each utterance. We use RoBERTa-base with 12 layers and fine-tune the final layer. To get the contextual representations, the utterances are passed to a bidirectional GRU (Cho et al., 2014). These final utterance representations are defined by
[ho. h1, h1] GRU ([e(uo), ((()) (1)
where h; Rd. Here, d is the dimension of each utterance vector.
4.2 Speaker Turn Indicating Tokens
Participants speak intermittently in dialogue. If single speaker has had the floor for the past few utterances and the current utterance also by the same speaker, then the chances of certain DA la- bels such as backchannel, mimic, and collaborative completion decrease, whereas the chances for other DA labels such as repeat, self-correct misspeak- ing increase. This example shows why modcling speaker turns is crucial.
To capture this behavior of turn-taking the utter- ance level, a turn indication token can used (Ze- lasko et al., 2021). Each utterance prepended with a special token to get the updated utterance u' as given by (2). These updated utterances are then fed to the encoder defined in $ 4.1. The upside of encoding speaker turns this way is that no new parameters are introduced into the model.
3 Task
"1 = (same) ui if S(u)) = S(u;-1)
(2)
A dialogue D with IPI participants can be defined collection of utterances D (ug. ....... M ii......)))
(switch) + uus if S(u;) # 5(4:1)
That is, |D] Each utterance is associated with a speaker given by S(u;) s; and DA label yr- The aim of DA classification is to assign DA label from a set of labels to each utterance u, in D.
4.3 Graph Speaker Modeling
The addition of the turn token can still only capture the binary speaker transitions. While this might be enough in the case of dyadic conversation, it reduces multiparty conversation to two-party one. To overcome this, on top of the turn tokens, we also learn a graph-based representation for each speaker and use it to inform each utterance its speaker.
4 Model
In this section, we describe the components of model, overview of which is presented in Figure 1.
Graph Structure We define the graph as G (V, where the nodes E V can be one of two types: utterance vil or a speaker v/. Labeled edges eij & denote edges between vi and vy. Finally, r R is the type of relation an edge represents.
4.1 Utterance Encoder
The utterance encoder is the same baseline as pre- sented He al. (2021).2 The sentence separator <<> token from the last layer of the RoBERTa> Liu
"e include the base model without the speaker turn embeddings. 3ht hhtg::/gggggggggeaeeee ace. wttee///o/o//tts/orrrreeeeeee noder_doc/roberta
We introduce a single type of relation R the graph that an edge eij connecting two nodes v;

---

Randomly inttiaiized
speaker vectors
8
2
RGAT
ho
hg
ho
h4
hgg
h
h1
h3
h
h3
h2
hg
h2
Classify
hg
h4
Contextual utterance
enriched
uttarannees
Figure 1: Example graph for dialogue with five utterances and three unique speakers. Dotted lines denote edges of type (v", i's )sm))
and Vy can take. We denote it by (v%, "stu)) ( represents an undirected edge between an utterance and its speaker. We show in $ 6.3 that compared to more complex graph structures, this one type of edge is enough to model speakers effectively.
Dialogues
Utterances
Dataset
|C|
[P]
Train
Val
Test
Train
Val
Test
MRDA
50
3-9
50
11
11
75k
15.3k
15k
SwDA
43 2
1003
112
19
193.3k
20.2k
4.5k
Graph Construction Given dialogue D with IPI speakers, utterance nodes are initialized using (I), following Zhang et al. (2019), and speaker nodes are randomly initialized. Therefore, the total number of nodes is [V] = [ |P].
Table 2: Number of classes (ICI), participants per dia- logue (IPI), and number of dialogues and utterances in each split.
eppeaker-enricced is passed to feed-forwer network to get the predicted label:
Speaker Learning The utterance encoder in all speaker graph experiments initialized with the trained weights of the encoder given in $ 5.3. The Whaseline layer is removed, and the encoder weights are kept fixed during training. We use Relational Graph Attention Network (RGAT) (Bushridge et al., 2019) model the speakers with respect to their utterances. The graph constructed and initialized as detailed above and passed to RGAT to get updated representations the vertices given by:
WSpp-Crraph (1 (#secalerr-eurccce)))
(5)
Experiments
Dataset
We experiment with two publicly available datasets for DA classification. For both the datasets, we use the dataset split used by Lee and Dernoncourt (2016). The dataset overview and distribution of utterances are given in Table
V RGGA(VV,,, R.
MRDA The first dataset is the MRDA corpus, which consists of around 72 hours of natural conver- sations, with an average dialogue length of 1445.41 utterances. There are 11 general and 39 specific tags, along with three types of disruptions and non-speech label. The MRDA corpus follows hi- erarchical annotation scheme, where each utterance labeled with compulsory general and need-based zero more specific tags.
DA Classification The updated speaker repre- sentations P. vV from the graph extracted and concatenated with their respective utterance representation h from (1). The final utterance representation is given by:
h speakerennicheed [hg s
(4)
Here, Ts S(()) is the graph node representation of ith utterance's s speaker, enpecherenrrcche E 2d Finally,
Over the years, many grouping schemes have been devised to consolidate the MRDA tags into

---

MRDA
SWDA
70000
30000
10000
30000
samples in dala
20006
40002
20000
2010
20002
10000
29
Classes
Figure 2: Both the datasets follow long-tailed distribution where few frequent classes have very large number of samples and there are many infrequent classes with only few samples.
higher-level categories. One of the most widely used such grouping the basic-tags introduced in Ang et al. (2005)4. While the majority of the existing works using MRDA focus on these five coarse-grained categories (Kumar et al., 2018; 8; Chen et al., 2018; Raheja and Tetreault, 2019; Bothe et al.. 2018; Khanpour et al., 2016; He et al., 2021), focus the fine-grained labels.
length is 192.3 utterances.
5.2 Evaluation Metric
The distributions of DAs in both the MRDA and the SwDA datasets are highly skewed, with the five most frequent classes making up 66.7% and 78.1% of the data in MRDA and SwDA respectively. Deep learning models have been known to be biased toward the classes with the most number of samples in the data (Johnson and Khoshgoftaar, 2019). As shown in Figure 2, both datasets follow long-tailed distribution, with few majority classes accounting for most of the data. Previous works primarily report the accuracy of their models.
In particular, an utterance is always assigned the first (in case of more than one) specific tag if one is present. If the DA label consists of only a general tag, then that label is used.5 We made three changes the labels: (i) we dropped rising lone since it is not DA (Dhillon al., 2004), (ii) we dropped declarative question because captures the syntactic structure of the utterance, is used when a question framed as a statement, and (iii) when a pipe symbol ()) is used to annotate the floor mechanism at the start of an utterance, we take the label assigned to the later part of the utterance instead of the floor mechanism. Floor not
However, relying solely on accuracy to judge model on highly imbalanced dataset can be prob- lematic (Gu al., 2009; Chawla, 2009; Bekkar al., 2013). Model performance can be overestimated by accuracy even it only performs well on a few frequent classes (Kotsiantis et al., 2006). Building a system that can also perform well on minority classes for DAs is important for many downstream tasks. For example, infrequent DAs such rep- etition request and partial reject can be effective indicators of misunderstanding for conversational agents (Aberdeen and Ferro, 2003). DAs such command and suggestion that together make up less than 4% data in MRDA, are especially important for conversational Al assistants to recognize.
mechanisms the start of an utterance the most informative tag when only choosing one label. Examples of annotated utterances are shown Table 1.
SwDA The second dataset we work with is the Switchboard Dialog Act corpus (Jurafsky, 1997), collection of telephone conversations between two people on a pre-specified topic. The SwDA corpus contains 43 DA types, and the average dialogue
The Fl score is the harmonic mean of precision and recall and it serves as balanced assessment of both these measures (Buckland and Gey, 1994). With the end goal of building a system that can be useful for several downstream tasks, we report the macro Fl, precision, and recall scores to give equal
"These are Statement, Question, Floorgrabber, Backchan- nel. and Disruption
5W use the full labels presented here: https://githul com/Na than an SMwp//MDA--A MMRDA-Corpus
<https: / thub. com/ cgpottss/wdaa

---

weight to each class. Although accuracy is the most appropriate metric to use in this setting, we still report it for a sense of comparison with prior work.
accuracy only. Table 3 shows the results fine- grained classes for the presented speaker graph model along with several related systems. Our results differ from previous work in that we report the accuracy from the epoch with the best FI score on the validation set instead of the best accuracy.
5.3 Baseline
As a baseline, we take the utterance representations from the encoder presented in $ 4.1 and pass them to ferd-forward layer to get class probabilities:
Most papers on DA classification do not have their code publicly available (with the exception of He et al. (2021)), making comparisons with them difficult. We present the results for three systems from DA classification.
Dasselin = Wbaseline (h).
(6)
BiLSTM+CRF (Kumar et al., 2018) a com- monly used model in DA classification. This is a hierarchical BiLSTM model with a CRF layer on top.
We use fixed chunk size of 128 for all the baselines. Based on the results in He et al. (2021), this is the best chunk size for SwDA. The effect of chunk size DA classification accuracy is negligible for the MRDA dataset. We compare the speaker graph model with two baselines. The other, turn- aware baseline, has augmented turn-aware input as presented in $ 4.2.
BiLSTM SelfAttCCRF (Raheja and Tetreault, 2019) is similar to BiLSTM+CRF, but they also introduce an attention component to ef- fectively incorporate contextual utterance rep- resentation from previous timestamp I; and word representations from 4, to compute utterance representation for f;-
5.4 Experimental Setup
The implementation is done in PyTorch (Paszke et al., 2019) version 1.12.1+cul 13 and PyTorch- geometric (Fey and Lenssen, 2019) version 2.1.0 was used for the graph experiments. The Hugging- face transformers library was used for RoBERTa. The AdamW (Loshchilov and Hutter, 2019) op- timizer used with weight decay of 5e 4. The LR is set to le and le - for the baseline and the graph models respectively. The test scores are taken from the epoch where the model gets the best validation score. All models are trained using the cross-entropy loss. Because of memory constraints, we segment each dialogue into non- overlapping chunks of size 128 in all experiments unless otherwise stated. Further details be found in Appendix A.
Turn Modeling (He et al., 2021) learns two speaker switch embeddings that are added to utterance embeddings. The utterance encoder is the same as our baseline.
We also compare with related works from emo- tion recognition,
DialogueGCN (Ghosal et al., 2019) leverages self and inter-speaker dependency to model the entire conversational context using a graph neural network.
DialogueRNN (Majumder et al., 2019) keeps track of the individual party states throughout the conversation using speaker-specific RNN. DI.logXI. (Shen et al., 2021 a) uses XL.Net Yang et al. (2019) as the base model and incorporates dialogue and speaker-level self- attention.
6 Results and Discussion
DAG-ERC (Shen et al., 202 treats the con- versation as a directed acyelic graph with only utterance node. Two different edge types indi- cate if two connected utterances belong to the same speaker or not.
We compare the speaker graph model against the stronger of the two baselines and consequently also use the turn-aware input for the graph model. All the results are average of 5 runs to account for the fluctuation introduced by randomness. The speaker graph model's performance statistically significant with p<0.0001 using Student's 's t-test.
DialogCRN (Hu et al., 2021) uses separate LSTMs model speaker and context depen- dencies.
SUNET (Song al., 2023) construct graph with both speaker and utterance nodes. They use a GRU to update speaker nodes based on their representations from previous layers.
6.1 Prior Work Baselines
As mentioned previously, prior work has focused on the high-level categories of DA and they report

---

MRDA
SwDA
Model
Precision
Recall
F1
Accuracy
Precision
Recall
F1
Accuracy
RoBeRTaaase
37.14
39.58
36.33
64.24
47.12
51.24
46.65
72.22
BiLSTM+CRF&
36.09
32.87
32.69
65.38
59.11
53.69
54.91
79.1
BiLSTM CeRo
34.32
32.19
31.21
63.66
54.41
51.14
$1.07
74.44
Turn Modeling
43.52
38.92
38.77
67.0
62.48
56.9
57.96
81.0
GaaloguGGCNRoBERTT
40.18
35.37
34.81
61.18
59.55
54.11
55.34
79.71
DialogueRN RaBBgue
37.58
38.9
37.12
65.04
56.04
61.87
57.21
81.2
D
40.67
37,99
37.97
63.79
59.46
$7.7
56.80
79.68
DAG-ERCA
43.64
38.46
39.34
67.01
59.24
54.6
55.77
78.62
DialogCRNRoBERTI
40.43
37.78
37.58
65.05
58.18
54.51
54.91
77.04
SUNET
37.66
40.72
38.11
64.36
51.64
58.6
53.19
80.58
Baseline
43.58
36.92
37.38
63.65
62.0
54.99
56.93
80.83
Turn Aware Baseline
43.08
38.5
38.53
66.96
64.62
57.31
58.99
81.58
Turn Aware Speaker Graph
44.53
39.11
40.06
66.32
63.72
57.36
58.81
80.86
Table 3: Fl score of the speaker-enriched model compared to prior work. The input to both the last two rows is augmented with turn tokens. are the results from rerunning the systems on our data and 9 are our reimplementation.
Since some of these works use older, non- contextualized word embeddings such as GloVe (Pennington et al., 2014), we swap them with RoBERTa (Liu et al., 2019) to be comparable with our work and to make sure the gains in our approach aren't tt due to the use of a better language model. For a fair comparison across models, we also chunk the dialogues into same size. All results are average of random seeds. Details on changes made to any baselines make them compatible with our data can be found in Appendix B.
models. We also present the accuracy in Table 3 for the sake of comparison with prior work.
The turn-aware graph model gives the best performance on MRDA. When we compare the speaker graph model with the baseline where the input to both also contains the turn tokens, we see an improvement of 1.53 F1 score. For the rest of the discussion, focus MRDA, since interested in multiparty dialogue setting.
MRDA
6.2 Classwise Results
SwDA Adding graph-learned speaker represen- tation does not help and brings the performance down slightly in terms of both Fl and accuracy. The speaker graph model 0.18 F1 worse than the best-performing turn-aware baseline. SwDA a two-party dataset and we postulate that adding speaker information the form of a token to the input is better able to make use of the speaker turns. This may be because of inducing turn awareness at earlier stage in the utterance before passing it to RNN to capture sequential context.
In this section, we analyze how the addition of speaker representations affects individual DA types. The detailed results can be found in Table 8. Perfor- mance on floor mechanisms improves with the ad- dition of speaker representations. Floor mechanism tags all share very similar vocabulary (Dhillon et al., 2004). In hold, speaker is passed the floor while in floor grabber, the speaker tries gain the floor. In order to disambiguate between these tags, it is vital have information about who is speaking.
the other hand, modeling speakers using a graph has the downside of losing this sequential information (Ishiwatari et al., 2020). This could deteriorate the performance in dyadic dialogues where speaker switch indicating token is enough to instill interlocutor information.
Without looking at prior turns, tags like affirma- tive answer and negative response can easily be confused with statements (Dhillon et al., 2004). These types of responses are hard catch because disambiguating them requires analyzing the utter- ance light of the dialogue context. The turn-aware speaker graph model gets a boost of 2.4 Fl points on affirmative answer and 5.4 F1 points on negative
As mentioned the carlier section, we mainly report and study the macro scores to evaluate the

---

spk uut
utt-+utt
utt-
spke> spk
F1
[D]
39.65
[D] |P]
39,57
2|
40.06
(2 + 2w)
39.72
2|20|(1 + w)
39.89
212 w) 2/P1
39.43
Table 4: The effect of introducing different edge connections on the macro Fl score. E is the approximate number of total edges in the model. The results are sorted by increasing density of the graph ascending order.
answer. This highlights the ability of the graph model capture the situated speaker behavior.
all our experiments, w is set to 10.
It clear from Table that the most impor- tant edge type is the one connecting every speaker with its own utterances. Capturing context at the graph level through the utt ult edge is not needed to model speakers well and hurts the perfor- mance slightly. One possible explanation for this is that the utterance nodes are already initialized with contextualized utterance vectors. Furthermore, connecting speakers with each other (spk + spk) hurts the performance in all cases. This shows that giving the speaker nodes direct access to only their own utterances is important and adding more edges can potentially introduce noise. The simplest graph with just single type of edge (v"l ((s(u)) denoted by spk - utf results in an Fl score that is still point better than the turn-award baseline.
Performance on minority classes 35 of the 50 classes in MRDA comprise less than 1% each of the data. Due to this class imbalance, it is diffi- cult to build a system that performs well on these classes. The speaker graph model is able to make improvements on many infrequent classes. Some examples include mimic, about-task, self-correct misspeaking, follow me, downplayer.
The graph model can capture the subtle nuances of speaker behavior. Tags such as mimic, down- player, joke see an improvement of 6.6, 3.8, 2.2 Fl score respectively. mimic utterances are those where the speaker repeats another speaker. They often serve form of acknowledgment from lis- teners (Dhillon et al., 2004). This shows certain intentions cannot be recovered from their semantic content alone.
The best-performing model also has edges be- tween the utterances and their respective speak- ers, i.e., the (v",, v v edges becomes undirected. Any (v".v") edges introduced to this setting hurts the performance. Capturing context at the graph level becomes computationally expensive and often impossible as the dialogue size increases. One com- mon workaround is to connect utterances falling within window size of each other. Even in such setting, the memory requirements incurred are often large and inhibit the value of be too big. We show in our work that this is not only redundant but also hurts the performance. Instead, a simple ob- jective learning speaker representations directly is not only compute efficient but performs better as well. The best performing model has 2| edges (row three of Table 4), whereas any graph with utt utt edges introduces additional 2|2/w w edges. While both settings have the number of edges linear function of the dialogue length, in practice the former can more effective low- resource settings such as under GPU memory and speed constraints.
6.3 Ablation Study on the Graph
In this section study how introducing different edges affects the model's ability to capture speakers. First, reiterate the edges used in our experiments along with introducing two new edge types.
(v" ) A directed edge between an ut- terance and its speaker.
A directed edge from one utterance to another.
(v%,v): An undirected edge between two speakers.
To also capture context at the graph level, we connect utterances with each other. Having an edge from every utterance to every other is not fcasible due GPU memory limitations. Following Ghosal al. (2019), every utterance node connected with its immediate past and future utterances falling within a window w. That is, vi has edges of type (v%, v*) to nodes [V" (i-wwr "., (-p vi, yie vire J. In 1+17

---

Chunk Size
Max
Avg/P1
F1
Graph Node
Macro FI
Full
6.13
Speaker
40.06
128
8
5
40.06
Utterance
39.46
96
8
4,8
39.96
Both speaker and ufterance
39.47
64
4.5
39.82
32
4
39.63
Table 6: The results MRDA by concatenating graph- learned speaker, utterance, and both speaker and utter- ance nodes with the utterance representations from the sequential encoder.
3,3
39.62
Table 5: Segmenting MRDA dialogues into smaller chunks various sizes results in fewer active speakers in that chunk. Here [P] is the numher of participants in the dialogue.
concatenated performs the best. Our assumption is that since the model contains two GNN layers, the utterance nodes are also able to implicitly learn more context through the edge that connects them with their speaker. Context in this case is the utter- ances by the same speaker because utterances from different speakers do not interact the graph level.
6.4 Effect of Chunk Size
Modeling long conversations in one go is not fea- sible due computational resource limitations. Therefore, we segment the dialogues into smaller, more managcable chunks. The maximum and av- erage number of speakers corresponding to each chunk size along with the macro Fl scores of the graph model are shown in Table 5. It is possible that this segmentation can strip away useful information in cases where utterance later stage of the dialogue relies on an earlier utterance resolve context issues.
Conclusion
We propose a graph-based approach to learn speaker-informed utterance representations for DA classification. We show that directly learning speaker representations with a simple graph is both effective and efficient. Instilling speaker informa- tion this way helps disambiguate DA labels in the case of multiparty dialogues. The learned speaker representations can be used on top of any utterance encoding scheme to include speaker information. Future work can also look into incorporating audio features to encode speaker since prosody plays an important part in signaling intention.
Our experiments show there not much differ- ence in performance by choosing a smaller chunk size. A smaller chunk size of 16 still gives macro Fl score improvement over the turn-aware baseline that had access to a larger context of 128. Fur- thermore, observed performance gains when only speakers on average involved in multiparty setting, as opposed the purely dyadic case of SwDA where no gains were observed. This highlights the usefulness of speaker modeling in multi multiipprtiicpantt dialogues. The results on the validation set are presented in Table 7.
Acknowledgements
We would like to thank the anonymous reviewers for their valuable feedback and input. Research sponsored by the Army Research Office and accomplished under Grant Number W911NF-20-1- 0002. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Gov- ernment is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
6.5 Utterance Graph Nodes
In this section we present the results of the model af- ter using the utterance nodes from the graph instead of the speaker nodes to augment the utterance rep- spenker-enriched resentations from Equation 1. The n becomes:
I sppeaker-nnichhed
[hy v/)
Limitations
Similarly, we also include the results of concate- nating both speaker and utterance nodes. Table shows that although all three systems perform better than the baseline, the model with speaker nodes
Training graph neural network with longer dia- logues and dense edges can be infcasible because GPU memory restrictions. For example, the choice

---

of type of edge connections to have along with other hyperparameters such as the window size can be restrictive because of compute resource limitations.
Zheqian Chen, Rongqin Yang. Zhou Zhao, Deng Cai, and Xiaofei He. 2018. Dialogue act recognition via crf-attentive structured network. In The s international acm sigir conference on research & development in information retrieval, pages 225-234.
Ethics Statement
Ta-Chung Chi, Po-Chun Chen, Shang-Yu Su, and Yun- Nung Chen. 2017. Speaker role contextual modeling for language understanding and dialogue policy learn- ing. In Proceedings the Eighth International Joint Conference on Natural Language Processing ( (Vol- ume 2: Short Pupers), pages Asian Federation of Natural Language Processing.
One potential use of dialogue act recognition is in the domain of conversational agents and chatbots. Al assistants can be used carry out commands for users. If the underlying DA recognition systems used by such assistants are brittle, they can nega- tively impact the end users. Since the performance of DA systems on finer- ener-grained labels is not robust, caution should be used while integrating them with real-world applications.
Taipei, Taiwan.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bah- danau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder ap- proaches. arXiv preprint arXiv::409 1259.
Pierre Colombo, Emile Chapuis, Matteo Manica, Em- manuel Vignon, Giovanna Varni, and Chloe Clavel. 2020. Guiding attention in seuuennc-to-sqquenee models for dialogue act prediction. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7594-7601.
References
John Aberdeen and Lisa Ferro. 2003. Dialogue patterns and misunderstandinns, In Proceedings of the ISCA Workshop on Error Handling in Spoken Dialogue Systems, pages 17-23.
Rajdip Dhillon, Sonali Bhagat, Hannah Carvey, and Elizabeth Shriberg. 2004, Meeting recorder project: Dialog act labeling guide. Technical report, INTER- NATIONAL COMPUTER SCIENCE INST BERKE- LEY CA.
Jeremy Ang, Yang Liu, and Elizabeth Shriberg. 2005. Automatic dialog act segmentation and classification in multiparty meetings. In Sicceeddigss 005. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005., volume 1, pages I-
1061. IEEE
Matthias Fey and Jan Eric Lenssen. 2019. Fast graph representation learning with pytorch geometric. arXiv preprint
Mohamed Bekkar, Hassiba Kheliouane Djemaa, and Taklit Akrouf Alitouche. 2013. Evaluation measures for models assessment over imbalanced data sets. J Inf Eng Appl, 3(10).
Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. 2019. Dia- loguegen: A graph convolutional neural network for emotion recognition in conversation. arXiv preprint arXiv: 17540.
Chandrakant Bothe, Cornelius Weber, Sven Magg, and Stefan Wermter. 2018. A context- based approach for dialogue act recognition using simple recurrent neural networks. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Chih-Wen Goo and Yun-Nung Chen. 2018a. Abstractive dialogue summarization with sentence-gated mod- cling optimized by dialogue acts. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 735-742. IEEE
Michael Buckland and Fredric Gey. 1994. The relation- ship between recall and precision. Journal of the American society for information science, 45(1):12- 19.
Chih-Wes Goo and Yun-Nung Chen. 2018b. Abstractive dialogue summarization with sentence-gated mod- eling optimized by dialogue acts. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 735-742.
Harry Bunt. 2006. Dimensions in dialogue act anno- tation. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06).
Qiong Gu, Li Zhu, and Zhihua Cai. 2009, Evalua- tion measures of the classification performance of imbalanced data sets. volume 51, pages 461-471,
Dan Busbridge, Dane Sherburn, Pictro Cavallo, and Nils Y Hammerla. 2019. Relational graph attention networks. arXiv preprint 1rXiv:1904.055
Zihao He, Leili Tavabi, Kristina Lerman, and Moham- mad Soleymani. 2021. Speaker turn modeling for dialogue act classification. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2021, pages 2150-2157, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Nitesh V Chawla. 2009, Data mining for imbalanced datasets: An overview. Data mining and knowledge discovery handbook, pages 875-886.

---

Dou Hu, Lingwei Wei, and Xiaoyong Huai. 2021. Dia- logueCRN: Contextual reasoning networks for emo- tion recognition in conversations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 7042-7052, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1rXiv:197. 1907.1109999
Ilya Loshchilov and Frank Hutter. 2019, Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Taichi Ishiwatari, Yuki Yasuda, Taro Miyazaki, and Jun Goto. 2020. Relation-aware graph attention net- works with relational position encodings for emotion recognition conversations. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7360-7370, Online. Association for Computational Linguisties.
Navonil Majumder, Soujanya Poria, Devamanyu Haz- arika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. Dialoguerno: An attentive rnn for emotion detection in conversations. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 33, pages
Justin M Johnson and Taghi M Khoshgoftaar. 2019. Sur- vey on deep learning with class imbalance. Journal of Big Data, 6(1):1-54,
Tatsuro Oya and Giuseppe Carenini. 2014. Extrac- tive summarization and dialogue act modeling on email threads: An integrated probabilistic approach. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 133-140.
Dan Jurafsky. 1997. Switchboard swbd-damsl shallow- discourse-functtor annotation coders manual. Insti- tute of Cognitive Science Technical Report.
Hamed Khanpour, Nishitha Guntakandla, and Rodney Nielsen. 2016. Dialogue act classification in domain- independent conversations using deep recurrent neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Lin- guistics: Technical Popers, pages 2012-2022 Osaka, Japan. The COLING 2016 Organizing Committee.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high- pelformffomcce deep learning library. Advances in neural information processing systems, 32.
Sotiris Kotsiantis, Dimitris Kanellopoulos, Panayiotis Pintelas, et al. 2006. Handling imbalanced datasets: A review. GESTS international transactions com- puter science and engineering, 30(1):25-36.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pages 1532-1543.
Harshit Kumar, Arvind Agarwal, Riddhiman Dasgupta, and Sachindra Joshi. 2018. Dialogue act sequence labeling using hierarchical encoder with erf. In Pro- ceedings of the aaai conference on artificial intelli- gence, volume 32.
Vipul Raheja and Joel Tetreault. 2019. Dialogue Act Classification with Context-Awarr Self-Attention In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3727-3733, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Bongseok Lee and Yong Suk Choi. 2021. Graph based network with contextualized representations of turns in dialogue. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 443-455, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008 The graph neural network model. IEEE transactions neural networks, 20(1)61-88.
Ji Young Lee and Franck Dernoncourt. 2016. Sequential short-text classification with recurrent and convolu- tional neural networks. In Proceedings of the 2016 Conference of the North American Chapler of the Association for Computational Linguisties: Human Language Technologies, pages 515-520, San Diego, California. Association for Computational Linguis- tics.
John Rogers Searle. 1969. Speech acts: An essay in the philosophy of language, volume 626. Cambridge university press.
Guokan Shang, Antoine Tixier, Michalis Vazirgiannis, and Jean- Pierre Lorré 2020. Speaker-change aware CRF for dialogue act classification. In Proceedings of the 28th International Conference on Computa- tional Linguistics, pages 450-664, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Yunlong Liang, Fandong Meng, Ying Zhang. Yufeng Chen, Jinan Xu, and Jie Zhou, 2021. Infusing multi- source knowledge with heterogeneous graph neural network for emotional conversation generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13343-13352.

---

Weizhou Shen, Junqing Chen, Xiaojun Quan, and Zhix- ian Xie. 2021a. Dialogxl: All-in-one xlnet for multi- party conversation emotion recognition. In Proceed- ings the AAAI Conference on Artificial Intelligence volume 35, pages 13789-13797.
of the Association for Computaaiional Linguistics. 9:1179-1195.
Dong Zhang, Liangqing Wu, Changlong Sun, Shoushan Li. Qiaoming Zhu, and Guodong Zhou. 2019, Mod- eling both context-and speaker-sensitive dependence for emotion detection multi-speaker conversations. In L/CAI, pages
Weizhou Shen, Siyue Wu, Yunyi Yang, and Xiaojun Quan. 2021b. Directed acyclic graph network for conversational emotion recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1551-1560, Online. Association for Computational Linguistics.
A Experimental Setup
A learning rate (LR) scheduler reduces the LR by 0.1 after 4 epochs of no improvement on the validation set. We train the models, including all the baselines, for a maximum of 100 epochs with early stopping after no improvement on the macro Fl on the validation set for 10 epochs.
Dongming Sheng, Dong Wang. Ying Shen, Haitao Zheng, and Haozhuang Liu. 2020. Summarize before aggregate: A global-to-local heterogeneous graph inference network for conversational emotion recogni- tion. In Proceedings of the 28th International Confer- ence on Computational Linguistics, pages 4153-4163, Barcelona, Spain (Online). International Committee on Computational Linguistics.
For training the baseline encoder, we keep most of the hyperparameters the same as He et al. (2021). The GRU hidden size used for the encoder is 200, since this is the largest encoder model we could fit on the GPU due to memory constraints when using the output from the encoder input to the graph model.
Elizaheth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang. and Hannah Carvey, 2004. The ICSI meet- ing recorder dialog act (MRDA) corpus. In Pro- ceedings of the 5th SIGdial Workshop on Discourse and Dialogue HLT--NACL 2004, pages 97-100, Cambridge, Massachusetts, USA. Association for Computational Linguisties.
The window hyperparameter was chosen from search space of 50, 10). There are two layers of RGAT with attention computed "across-relation" using sadditive-sell-atttettoo attention heads for both layers is set to two.
The number of
Rui Song. Fausto Giunchiglia, Lida Shi, Qiang Shen, and Hao Xu. 2023. Sunet: Speakcr-utterance interaction graph neural network for emotion recognition in conversations. Engineering Applications of Artificial Intelligence, 123:106315.
The graph model on top of the encoder has around 154M parameters. All the experiments on MRDA are done single NVIDIA A100 GPU, and the model converges in about half an hour. The SwDA experiments are all done on a single NVIDIA RTX A6000, with a single run taking approximately an hour.
Yang Sun, Nan Yu, and Guohong Fu. 2021. Integrating rich utterance features for emotion recognition in multi-party conversations. International Confer- ence on Neural Information Processing, pages 51 1-62. Springer.
B Baseline Models
Kai Wang, Junfeng Tian, Rui Wang, Xiaojun Quan, and Jianxing Yu. 2020. Multi-domain dialogue acts and response co- generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7125-7134, Online. Association for Computational Linguisties.
All the systems included (except DialogXI.) use RoBERTa-bass as the pre-trained language model. For all the graph models, the hyperparameter is set 10-the same as in our graph model.
We make two changes to SUNET (Song et al., 2023) in the results we report after replicating their system. First, They use the utterance rep- resentations of all the utterances of a speaker from the training set to initialize that speaker node. We randomly initialize speakers. The utterance nodes are initilized using trained encoder from $ 5.3. This was because we couldn't fit the model on GPU by directly us- ing RoBERTa representations since those have a large dimension compared to the encoder (768 vs 400).
Can Xu, Wei Wu, and Yu Wu. 2018. Towards explainable and controllable open domain dialogue generation with dialogue acts. arXiv preprint arXiv: 1807.07255
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information pro- cessing systems, 32.
Piotr 7elasko, Raghavendra Pappagari, and Najim Dehak. 2021. What helps transformers recognize conversa- tional structure? importance of context, punctuation, and labels in dialog act recognition. Transactions

---

Chunk Size
Fl
Full
128
38.58
96
38.63
64
38.62
32
38.59
16
38.29
Table 7: Effect of chunk size on the validation set of MRDA for the speaker graph model.
For Turn Modeling (He et al., 2021) system, we do not include Topic embeddings for the SwDA dataset.
C Results on Validation set
We present the results on the validation here. All the results are an average runs. Even though the model with chunk size 96 performs slightly better on the validations set, We picked the chunk size to be 128. This makes for fair comparison with the baseline that uses 128-sized chunks.
D Classwise Results
The test set does not have any sample of the welcome class.

---

% Distribution
Label
Description
Turn-aware Baseline
Baseline
Turn-aware
Speaker Graph
34.73
Statement
73.2
76.8
76.2
1.29
qw
Wh-Question
81.2
81.2
80.6
0.24
gb Rhetorical Question
9.6
9.0
14.6
0.35
qu Or Clause After Y/N Question
87.6
88.8
90.0
0.15
qr
Or Question
63.8
66.2
64.2
0.11
qo
Open-ended Question
18.0
13.0
16,20
1.97
Y/N Question
67.8
68.8
68.2
0.79
g
Tag Question
72.0
72.8
72.2
5.47
aa Accept
48.0
50.6
51.4
0.23
aap Partial Accept
5.8
5.6
1.08
na Affirmative Answer Reject
14.4
15.2
17.6
0.87
78.4
78.6
78.8
0.14
arp Partial Reject
0.0
0.0
2.0
0.48
nd
Dispreferred Answer
16.0
15.8
16.4
0.34
ng Negative Answer
2.4
2.2
7.8
14.02
b Backchannel
68.2
78.6
79.0
6.74
bk Acknowledgement
56.8
59.4
$8.2
2.12
ba Assessment/Apprees Appreciation
58.2
58.2
58.2
0.14
bh Rhetorical Question Backehannel
52.0
57.2
65.2
0.24
h Hold
36.4
50.8
53.6
5.77
fh Floor Holder
80.0
81.8
82.0
1.3
fg
Floor Grabber
7.8
11.8
20.0
0.38
cc Commitment
37.8
35.4
33.0
2.65
cs
Suggestion
46.8
46.0
45.8
0.7
co
Command
$3.0
$4.2
$1.0
0.33
am
Maybe
44.4
46.4
45.6
0.82
No Knowlcdge
68.0
65.6
67.0
0.27
Mimic
2.4
2.4
9.0
0.2
Repeat
0.0
0.0
0,0
0.14
bs Summary
0.0
0.0
0.0
0.59
f Follow Me
39.8
41.0
47.0
0.22
br Repetition Request
61.0
58.6
60.6
1.99
bu Understanding Check
46.4
49.8
48.0
3.55
df
Defending/Explanttion
48.6
49.4
46.4
3.06
Flaboration
31.2
36.6
36.4
0.78
2
Collaborative Completion
13.0
25.8
25.2
0.05
bc
Correct Misspeaking
0.0
0.0
0.0
0.14
bsc Self-Corred Misspeaking
1.8
0.0
14.2
0.38
bd Downplayer
4.4
6.6
10.4
0.01
by Sympathy
0.0
0.24
Apology
84.6
85.8
88.4
38.0
Thanks
86.8
84.8
86.4
0.01
Welcome
0.28
fo Exclamation
81.0
80.4
80.6
0.4
About-Task
9.6
6.4
12.2
0.29
Topic Change
5.2
5.0
7.2
0.44
Joke
4.8
0.6
7.0
0.21
t1
Self Talk
9.0
8.2
5.4
0.17
13
Third Party Talk
0.0
0.0
0.0
3.03
%
Disruption
48.4
57.0
59.4
Table 8: Per class macro Fl scores the test for all models MRDA. Distribution is the percentage of labels of a class in the whole data. Scores are the average of random runs.


Reviews:
[{"review_id": "66RXuk3pBt", "reviewer": "Reviewer_U5G3", "summary": "", "questions": "No. ", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for bringing the related work to our attention.\n\n>“Insufficient innovation”  \n\nWe briefly summarize and compare the approaches in [1-5] to our work.   \n[1] This is a very recent work (published < 3 months ago) that also incorporates speaker information through GNNs. They update the speaker representation through a different update function than ours and they also include regularization loss to discourage similarity between different speaker representations. They construct the graph using three types of edges. One that connects every utterance to its speaker, one that connects an utterance with the utterances by the same speaker, and the last edge connects utterances with utterances of other speakers. We build the graph a little differently-- utterances are connected with a single type of edge.  \n[2] build a directed acyclic graph (DAG) where the nodes are the utterances. They argue that a DAG is more suitable for dialogue modeling because information cannot flow backward from future utterances to the past. They incorporate the speaker information by distinguishing the edge types one utterance can have with another: one for when the utterances are by the same speaker and another edge type if they are spoken by different speakers. There are multiple differences between this work and ours. The main difference is the way they construct the graph---there is no explicit learning of speaker representations. They only use two different edge types to indicate if two connected utterances share the same speaker or not. In some ways, this is akin to converting a multiparty dialogue to a two-party because the utterances only have binary information of speakers now. This is not an appropriate way to handle multiparty dialogues (a conclusion also mentioned in [2]).   \n[3] Use BERT (or its variants) to encode utterances, followed by a turn attention module and an RNN to inject context and sequential information respectively. Then a dialogue graph is constructed. They treat ERC as a link prediction task between the speaker and the utterance node. Speaker information is encoded in two ways. (1) A speaker token id is summed with each token’s representation from a BERT/BERT-like model. This requires learning a fixed number of speaker embeddings. (2) In the dialogue graph, each utterance of the same speaker is connected. Again, doing so does not explicitly learn speaker representations for each dialogue.   \n[4] Present a way to incorporate emotion and speaker traits for emotional conversation generation. They construct a graph with utterance, speaker, emotion, and facial expression nodes. The speaker nodes are learned through a matrix of fixed size (the number of unique speakers in data). They experimented with a dataset derived from a TV show with a fixed number of dominant speakers. On the contrary, we target the setting of natural meetings where we assume that at test time we could be given new speakers not seen during training.  \n[5] Incorporate sequential information in relational graph attention network (RGAT) [7] through positional embeddings. The nodes in their graph are only utterances and they model speaker dependency by four different types of edges between utterances: whether utterances are from the same speaker or not combined with directionality of past and future. This technique suffers the same pitfalls of not scaling to multiparty conversations. In contrast, we add speaker nodes to the graph.  \n\n---\n\n>Too weak baselines.\n\nBelow we show results after rerunning the system from [2]. The results are on MRDA dataset which is the primary dataset we experiment with. We conducted an extensive hyperparameters search and report the final results from the epoch with best validation F1. We used learning rate from [ 5e-2, 1e-3, 2e-3, 3e-3, 4e-3,  5e-3, 9e-4, 5e-4, 1e-4, 5e-5 ], batch size from [64, 128, 256] and trained for a maximum of 300 epochs.\n\n\n| Model      | Macro F1 | \n| :---        |    :----:   | \n| Our baseline        |    38.69  | \n| Our approach        |    39.89  | \n| DialogueGCN [8]      | 34.67       | \n| DAG-ERC [2]   |  25.15        | \n\n\nMultiparty DA classification under the fine-grained setting is a challenging task. DAG-ERC performs poorly on MRDA. Our assumption is that DAG-ERC is more suitable for two-party dialogues. Other systems we have compared with in the paper from ERC [8] also perform poorly than our baseline.\n\n\n\n---\n\nReferences:\n\n[7] Busbridge, D., Sherburn, D., Cavallo, P., & Hammerla, N. Y. (2019). Relational graph attention networks. arXiv preprint arXiv:1904.05811.  \n[8] Ghosal, D., Majumder, N., Poria, S., Chhaya, N., & Gelbukh, A. (2019). Dialoguegcn: A graph convolutional neural network for emotion recognition in conversation. arXiv preprint arXiv:1908.11540.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Very well written and logical.\n2. Detailed parameter exploration and rich experimental details.\n3. Sufficient experimental datasets.", "reasons_to_reject": "1. Insufficient innovation. Although the application scenario of this paper is DA classification, the authors still need to explore and compare the related studies on Emotion Recognition in Conversations (ERC) because the two fields are too similar. As a point of comparison, the authors' main technical details, both the construction of heterogeneous graphs and the introduction of speakers' information into the dialog, have been heavily discussed in [1, 2, 3, 4]. In addition, the use of RGAT is also not new [5]. Therefore, the authors cannot substantiate the innovativeness of the proposed method without discussing the above work.\n2. Too weak baselines. As articulated in the previous point, the authors need to make a more comprehensive experimental comparison of similar methods, but the manuscript only shows DialogueGCN, DialogueRNN and appraoch in [6]. I don't doubt for a second that recent work in the field can achieve results similar to or better than the authors' proposed method.\n\n[1] Song, Rui, et al. \"SUNET: Speaker-utterance interaction Graph Neural Network for Emotion Recognition in Conversations.\" Engineering Applications of Artificial Intelligence 123 (2023): 106315.\n[2] Shen, Weizhou, et al. \"Directed Acyclic Graph Network for Conversational Emotion Recognition.\" Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.\n[3] Lee, Bongseok, and Yong Suk Choi. \"Graph Based Network with Contextualized Representations of Turns in Dialogue.\" 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021). ASSOC COMPUTATIONAL LINGUISTICS-ACL, 2021.\n[4] Liang, Yunlong, et al. \"Infusing multi-source knowledge with heterogeneous graph neural network for emotional conversation generation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 15. 2021.\n[5] Ishiwatari, Taichi, et al. \"Relation-aware graph attention networks with relational position encodings for emotion recognition in conversations.\" Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.\n[6] He, Zihao, et al. \"Speaker Turn Modeling for Dialogue Act Classification.\" Findings of the Association for Computational Linguistics: EMNLP 2021. 2021.", "questions_to_authors": "No. ", "paper_topic_and_main_contributions": "In this paper, the authors consider introducing speakers' features in the process of Dialogue acts (DAs) recognition and modeling multi-round conversations involving multiple speakers by constructing heterogeneous graphs and learning conversations containing speakers' features using RGAT to facilitate classification. The authors emphasize that there has been little previous work on incorporating speaker awareness into DA classification. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Song, Rui, et al. \"SUNET: Speaker-utterance interaction Graph Neural Network for Emotion Recognition in Conversations.\" Engineering Applications of Artificial Intelligence 123 (2023): 106315.\n[2] Shen, Weizhou, et al. \"Directed Acyclic Graph Network for Conversational Emotion Recognition.\" Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.\n[3] Lee, Bongseok, and Yong Suk Choi. \"Graph Based Network with Contextualized Representations of Turns in Dialogue.\" 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021). ASSOC COMPUTATIONAL LINGUISTICS-ACL, 2021.\n[4] Liang, Yunlong, et al. \"Infusing multi-source knowledge with heterogeneous graph neural network for emotional conversation generation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 15. 2021.\n[5] Ishiwatari, Taichi, et al. \"Relation-aware graph attention networks with relational position encodings for emotion recognition in conversations.\" Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020."}}, {"review_id": "VDRZFA8f0x", "reviewer": "Reviewer_JZjf", "summary": "", "questions": "Question A: Is the improvement in F1 on MRDA significant?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and suggestions.\n\n> It does not appear that the code has been made open-source.\n\nWe will open-source our code with the camera-ready version.\n\n---\n\n> similar approaches mentioned in the related work (Chi, Colombo, Ghosal, Zhang, or Sun) are not included\n\n All these papers do not have their code publicly available (except for Ghosal, those results are included in the paper), which makes comparing with them difficult. Here we present an additional relevant system from emotion recognition on MRDA dataset. Results are average on 3 random runs after an extensive hyperparameter search. \n\n| Model      | Macro F1 | \n| :---        |    :----:   | \n| Our baseline        |    38.69  | \n| Our approach        |    39.89  | \n| DAG-ERC [1]   |  25.15        | \n\n[1] build a directed acyclic graph (DAG) with utterances as the only nodes. They incorporate the speaker information by distinguishing the edge types one utterance can have with another: one for when the utterances are by the same speaker and another edge type if they are spoken by different speakers. There is no explicit learning of speaker representations. They only use two different edge types to indicate if two connected utterances share the same speaker or not. This technique does not work well for multiparty dialogues as it essentially converts it to a two-party conversation.\n\n\n---\n\n> Question A: Is the improvement in F1 on MRDA significant?\n\nYes. We conducted Student’s t-test and our model’s performance is statistically significant with p<0.0001.\n\n---\n\n> In footnote 6, you mention that the hidden size is 200. Why not use 384?\n\nThe output from the utterance encoder is fed to the graph neural network. Setting the hidden size to 384 resulted in utterance representations of size 768 since these utterance representations were the input to the graph, we ran into a cuda out of memory error. So we had to restrict the hidden size to 200 to fit the graph on gpu.\n\n---\n\nReferences\n\n[1] Shen, W., Wu, S., Yang, Y., & Quan, X. (2021). Directed acyclic graph network for conversational emotion recognition. arXiv preprint arXiv:2105.12907.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper itself is clear and well-written, with a thorough literature review that makes it easy for the reader to see how their work differs from prior work. The explanation in the analysis section for why the graph helps with multi-party dialogues seems reasonable. The GNN approach for DA classification is interesting and novel.\n\n", "reasons_to_reject": "It does not appear that the code has been made open-source. The results look good, although they are not completely convincing since most of the similar approaches mentioned in the related work (Chi, Colombo, Ghosal, Zhang, or Sun) are not included, it is unclear whether the improvement in F1 score on MRDA is significant since the increase is small, and the graph does not help for SwDA.", "questions_to_authors": "Question A: Is the improvement in F1 on MRDA significant?", "paper_topic_and_main_contributions": "This paper presents a new approach to dialogue act (DA) classification using a graph neural network (GNN) and turn-indicating tokens. They demonstrate on the fine-grained DA prediction task from the MRDA dataset that speaker representations learned from the GNN outperform other baselines, specifically in multi-party dialogues. The paper itself is clear and well-written, with a thorough literature review that makes it easy for the reader to see how their work differs from prior work. The explanation in the analysis section for why the graph helps with multi-party dialogues seems reasonable. The GNN approach for DA classification is interesting and novel.\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "•\tIt’s confusing why the graph structure subsection in 4.3 is included when some of those relations do not match the edges in Figure 1. It seems like the graph construction subsection is clearer and sufficient. OR explicitly state which are not in the figure and why.\n•\tExplain a bit about what the RGAT function actually does.\n•\tTable 2 appears much later than when it’s first referenced in the text.\n•\tWhen Table 3 is referenced, accuracy is mentioned, so you should reference Table 4 too.\n•\tIn footnote 6, you mention that the hidden size is 200. Why not use 384?\n•\t“into same size” -> “into the same size”\n•\t“are average” -> “are the average”\n•\tIn the column headings in Table 5, what is the italicized P?\n"}}, {"review_id": "bLCbnLPFJE", "reviewer": "Reviewer_pYXY", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. \n\n> Very loose papers with many unnecessary details.\n\nWe believe that it is in the interest of reproducibility to err on the side of providing too much detail, rather than too little. However, we are open to any constructive feedback you may have on which details are not necessary.\n\n---\n\n> Idea and approach not novel.\n\nWe argue in our paper that explicitly modeling speakers for DA classification is important. While prior work for DA has, to a limited extent, tried to model speakers, the approaches are limited to two-party conversations and also don’t scale well to unseen speakers at test time. A thorough discussion can also be found in the paper L052 - L069 and L102-L131. The approach of using graph neural networks to encode a dialogue and inject speakers has been presented before, building on that work we create a simple GNN with utterance and speaker nodes. We explicitly optimize for learning good speaker representations in contrast to using GNNs to encode more utterances’ context, as is done in most prior work (L161-L169). For a more detailed discussion please also refer to the section “Graph-based methods to encode speakers in dialogues”.\n\n---\n\n> Insufficient comparative systems used\n\nWe have run another system from the field of emotion recognition that incorporates speaker information. The results on MRDA are shown below. Results are average on 3 random runs after an extensive hyperparameter search. \n\n| Model      | Macro F1 | \n| :---        |    :----:   | \n| Our baseline        |    38.69  | \n| Our approach        |    39.89  | \n| DAG-ERC [1]   |  25.15        | \n\n[1] build a directed acyclic graph (DAG) with utterances as the only nodes. They incorporate the speaker information by distinguishing the edge types one utterance can have with another: one for when the utterances are by the same speaker and another edge type if they are spoken by different speakers. There is no explicit learning of speaker representations. They only use two different edge types to indicate if two connected utterances share the same speaker or not. This technique does not work well for multiparty dialogues as it essentially converts it to a two-party conversation.\n\n---\n\nReferences\n\n[1] Shen, W., Wu, S., Yang, Y., & Quan, X. (2021). Directed acyclic graph network for conversational emotion recognition. arXiv preprint arXiv:2105.12907.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Relevant approach to DA classification.\n- Relevant use of graphs to monitor speakers in dialogue.\n- Modular approach provided which can be used with other utterance encoders as well.", "reasons_to_reject": "- Very loose papers with many unnecessary details.\n- Idea and approach not novel.\n- Insufficient comparative systems used.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper aims to use explicit speaker representations to improve the performance of dialogue act classification. They propose a GNN based network to capture speaker specific nuances and Transformer based model to encode utterances.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vNMYjh4MPY", "reviewer": "Reviewer_tXBt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and your suggestions.\n\n> RR 1: Limited effect of the proposal.\n\nWe have worked under the fine-grained setting of DA labels as opposed to only the top level that prior work has focused on. Fine-grained DA labels are important for many downstream applications but they are also more difficult to gain performance boosts on because of the extremely skewed data distribution and the inherent difficulty of the task. Therefore, gaining improvements on this task is not easy.\n\n---\n\n> RR 2. \n\nWe conducted Student’s t-test and our model’s performance is statistically significant with p<0.0001.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "RA 1: It proposed to use GNN to encode speaker information.", "reasons_to_reject": "RR 1: Limited effect of the proposal.\n\nRR 2: The minority class analysis in Section 6.2 does not consider sample size. For example, MRDA data has 15K test utterances, *mimic* and *repeat* consisting of 15Kx0.27%=41 and 15Kx0.2%=30 samples, respectively. The improved 4.8 and 8.6 F1 scores will **roughly** correspond to 41*4.8%=2 and 30*8.6%=2.6 data samples. I suggest the authors provide significance test results to support their conclusions.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposed a GNN-based approach to dialog act recognition.\n\nThe new model mainly consists of three components\n- A speaker turn token is prepended to each utterance following past works.\n- An utterance encoder that uses RoBERTa outputs as utterance embeddings.\n- A graph neural network that maintains speaker nodes and utterance nodes. utterance-utterance and speaker-utterance edges are constructed to propagate information.\nAnd the updated GNN node embeddings are used for the final classification.\n\nExperimental evaluation on MRDA and SwDA shows that the speaker turn token contributes to the most improvement. The GNN component is complex but has very limited effect. In some cases adding the GNN even degrades the performance.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Important results in the main body should use support from the main body, but at L466 the authors refer to results in the Appendix."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the role of speaker information in the task of dialog act classification. It proposes to use speaker-information using graph neural networks to improve the utterance representations. Experiments on both dyadic and multi-participant conversations demonstrate the utility of the proposed model. 

Overall, the paper, while well-written, suffers some concerns with respect to novelty. In particular, discussions amongst reviewers and authors revealed the existence of many related approaches that share the same motivation – using/encoding speaker information using some form of graph network. Given minor differences, strong justification for novelty was missing. 

Another valid suggestion revolves around space optimization where space could be used to perform a better comparison to graph-based baselines and also non-GAT-based speaker-encoding baselines.
Decision: Accept-Findings