Hierarchical Fusion for Online Multimodal Dialog Act Classification
Authors: Md Messal Monem Miah
Adarsh Pyarelal
Ruihong Huang
Venue: EMNLP 2023 Findings (2023)
Keywords: Dialog Act Classification
Multimodality
Early Fusion
Online Inference
TL;DR: We propose a hierarchical fusion network that predicts dialog acts based on audio and ASR generated transcripts.

Abstract: We propose a framework for online multimodal dialog act (DA) classification based on raw audio and ASR-generated transcriptions of current and past utterances. Existing multimodal DA classification approaches are limited by ineffective audio modeling and late-stage fusion. We showcase significant improvements in multimodal DA classification by integrating modalities at a more granular level and incorporating recent advancements in large language and audio models for audio feature extraction. We further investigate the effectiveness of self-attention and cross-attention mechanisms in modeling utterances and dialogs for DA classification. We achieve a substantial increase of 3 percentage points in the F1 score relative to current state-of-the-art models on two prominent DA classification datasets, MRDA and EMOTyDA.

Hierarchical Fusion for Online Multimodal Dialog Act Classification
Md Messal Monem Miah Adarsh Pyarelal Ruihong Huang*
*Texas A&M University, 'University of Arizona
[messal. huangrh ttamu.edu, {adarsh} @ arizona.edu
Abstract
Speaker
Utterance
Dialog act
We propose framework for online multimodal dialog act (DA) classification based on raw audio and ASg-gennrated transcriptions of cur- rent and past utterances. Existing multimodal DA classification approaches are limited by ineffective audio modeling and late-stage fu- sion. We showcase signifieant improvements in multimodal DA classification by integrating modalities at more granular level and incor- porating recent advancements in large language and audio models for audio feature extraction. We further investigate the effectiveness of self- attention and cross -tossntttenioon mechanisms in modeling utterances dialogs DA classi- fication. We achieve substantial increase of 3 percentage points in the Fl score relative to current state-of-the-art models on two promi- nent DA classification datasets, MRDA and EMOTyDA.
mc018
guess when Sunil gets here he can do his last
Suggestion
something.
me013
Yeah.
Acknowledgement
mc013
So probably should wait him to be- fore do his.
Command
mc006 me018
Okay
Rackchannel Appreciation
That's good
Table I: Short segment of a dialog from MRDA corpus.
depend automatic speech recognition (ASR) for transcriptions, which are often subject noise and inaccuracies. Ortega et al. (2019) demonstrated that DA classification models trained on oracle tran- scriptions exhibit subpar performance when applicd to noisy transcriptions.
Second, audio signals contain important acoustic and prosodic cues, which crucial for DA classi- fication (Jurafsky et al., 1998), but overlooked when focusing solely on text transcriptions. An example of the importance of acoustic information DA modeling can found in the brief dialog excerpt presented in Table 1. The utterances la- beled as suggestion and command are lexically very similar, making disambiguation based solely on textual content difficult.
1 Introduction
Dialog Acts (DAs), as described by Searle (1969), are the minimal units linguistic communication that represent the speaker's intention behind an utterance in a conversation. Successful DA clas- sification facilitates a range of downstream tasks including task-oriented dialog systems (Blache al., 2020; Wang et al., 2020b), conversational agents (Ahmadvand et al., 2019; Wood et al., 2020), and di- alog summarization (Oya and Carenini, 2014; Goo and Chen, 2018). Since DA classification is eru- cial for understanding spontaneous dialog (Stolcke et al., 2000), a significant amount of effort has been put into modeling DAs computationally, However, most of the research efforts on DA classification heavily rely on oracle transcriptions (Ortega and Vu, 2017; li et al., 2018; Li al., 2019; Colombo al., 2020; He al., 2021), resulting in several limitations.
In this research, explore real-time DA classifi- cation, utilizing only the raw audio signals as input. However, the presence of noise in the audio sig- nals and potential inaccuracies in the transcriptions generated by Automatic Speech Recognition (ASR) systems make the setting particularly challenging. This motivates to investigate a multimodal frame- work that leverages the complementary nature of audio and textual representations, enabling reliable DA classification despite the limitations of noisy audio signals and imperfect ASR transcriptions. few previous research efforts explore the setup but are limited by either poor dialog modeling or ineffective use of audio. He et al. (2018) make use
First, oracle transcriptions are typically inacces- sible during real-tine time dialog processing, such as in conversational agent systems. These systems

---

of ASR generated transcriptions and audio features but do not consider contextual information while modeling the utterances, resulting into drammat- ically lower performance in comparison to using oracle text.
utterance vector for further processing by RNN to capture contextual information. Zelasko et al. (2021) explore the effects of broader context, capi- talization and punctuation on DA classification and report strong result using XeN and Longformer.
We propose novel multimodal approach that uses a hierarchieal fusion technique to combine the two modalities etrly-stage, resulting in su- perior performance compared to utilizing individ- ual modalities for online DA classification. Our proposed approach leverages multimodal fusion strategies to effectively model both the utterances and their context. Additionally, we propose state- of of-thh-art audio feature extraction process that hances audio-based modeling of dialogs, leading improved performance in DA classification. We release the codes for feature extraction and experi- ments in this repository.
Multimodality for Dialog Modeling Multi- modality is widely explored in multimodal timent analysis (MSA) and emotion recognition in conversation (ERC) tasks. Hazarika et al. (2018) propose Conversational Memory Network (CMN) for ERC that uses GRU to model the utterances from fused feature representations and use atten- tion mechanism to implement the memory network. Poria et al. (2017) present (i) CAT-LSTM, con- textual attention-based LSTM network for model- ing utterance relationships and (ii) AT-Fusion, attention-based fusion mechanism that enhances multimodal fusion for MSA. Tsai et (2019) intro- duce the cross- attention based Multimodal Trans- former (MulT), an end-to-end model that expands upon the standard transformer network (Vaswani et al., 2017) to enable learning representations di- rectly from unaligned multimodal streams for ERC. However, unlike ERC and MSA, multimodality is less explored for DA classification. He et al. (2018) combine CNN for audio feature augmenta- tion and RNN to model utterances from individual modalities and fuse them through concatenation. While Ortega and Thang Vu (2018) utilize CNN to capture intra-utterance context for both audio and textual modalities, they only consider dialog- level context for modeling textual representation. Saha et al. (2020) propose Triplet Attention Sub- network, incorporating self and cross attention jointly model emotion and DA in dialogs.
2 Related Work
Dialog Act Classification Several studies have explored different approaches to facilitate DA clas- sification, with focus employing various neural network architectures and attention mechanisms for leveraging contextual information. Ortega and Vu (2017) use CNNs to acquire utterance repre- sentations and further investigate various forms of attention mechanisms to infuse context aware- ness into the utterance representations. Bothe et al. (2018) propose using a pre-trained character-level language model to generate sentence representa- tions and RNN to learn dialogue act from the current utterance representation and the context of previous utterances. li et al. (2018) introduce CRF-attentive structured network with integrated structured attention to simultaneously model con- textual utterances and their corresponding DAs. Li et al. (2019) jointly model DAs and topics using a shared bidirectional GRU-based utterance encoder and task-specific attention mechanisms.
3 Proposed Method
In this section, we will describe the components of the proposed model in detail (see Figure 1)
3.1 Task Deseription
Raheja and Tetreault (2019) propose a hierarchi- cal RNN with context-aware self attention to model various levels of utterance and dialog act seman- tics. Wang et al. (2020a) propose Heterogeneous User History (HUH) graph convolution network, complemented by denoising mechanisms in order effectively integrate user's historical answers grouped by DA labels improve DA classification. He et al. (2021) propose speaker turn-aware ap- proach where the turn embedding vector, learned based on the speaker label combined with the
A dialog consists n utterances (U) accompa- nied by their respective labels (Y), arranged in chronological order. Each utterance comprises a text component (x²) and an audio component (xς²). Mathematically, dialog with utterances can be represented as:
(U,Y) Y) - (x/. [1,n]]- (1)
Here, x; denotes the 7t utterance, where x rep- resents the text component and x represents the

---

audio component. A DA label y; from a predefined set of labels (C), is assigned to each utterance. Our proposed network takes this input data and aims to accurately classify the DA associated with each utterance. Since, we are interested in online DA classification, we only consider the raw speech sig- nals as the input. We use Whisper (Radford et al., 2022) to obtain transcriptions.
3.3 Model Architecture
The proposed model consists of three primary com- ponents: (1) an early-stage fusion encoder that fuses the two modalities at the utterance level and incorporates attention-based dialog modeling using the fused utterances, (2) unimodal encoders that generate context-aware representations from each modality individually, and (3) a late-stage fusion classifier that combines the context-aware repre- sentations from both the early-stage fusion encoder and the unimodal encoders and applies linear layers to perform the final DA classification.
3.2 Multimodal Feature Extraction
Textual Features We use a pre-trained RoBERTa (Liu et al., 2019) model to extract textual features. The textual representation from each utterance x is passed through the pre-trained RoBERTa model, and the representations from the last hidden layer are extracted as the token embeddings. Mathematically x can be represented as follows:
3.3.1 Early-stage Fusion Encoder
In this branch of the network, every utterance (xf..()) within the dialog is processed by two Bi- LSTM networks. Specifically, Bi-LSTM, is used to model the textual utterance representation extracted fromx/,, while Bi-LSTM, used to model the audio utterance representation extracted from x². At any time f, the forward LSTM, computes the forward hidden vector h: based on the previous hidden vector - and the input token embedding, (w/), while the backward LSTM, computes the backward hidden vector h based on the opposite previous hidden vector / and the input token embedding (w/). Subsequently, the forward and the backward hidden vectors are concatenated into the final hidden vector, h; of the Bi-LSTM model. Here, h is a 2d,, dimenninnll vector where d, is the hidden dimension of the LSTMs. Mathematically,
x {w/, w?............ .... w/} E BR
(2)
Here, w/ represents the jth token of the 7th utterance, k the total number of tokens in an utterance, and d, the embedding dimension.
Audio Features We use a pre-trained Whisper model to extract audio features. The audio signal corresponding with each utterance x is passed through pre- mre-trained Whisper model and the hidden states from the last encoder layer are extracted as the frame embeddings. Mathematically x², can be represented as follows:
x/2 = (1 STM)) mxdd
(3)
Here, f represents the jth frame of the 1hh utterance, m the total number of frames in an utterance, and da the embedding dimension.
7.14 LSTM (W/, Ti!-L
(1)
The choice of Whisper as the audio feature ex- tractor is motivated by two key factors. First, given the real-time nature of online DA classification, the usage of Whisper, which is already used for tran- scriptions, eliminates the necessity for separate feature extraction model. This not only reduces the time requirements but also simplifies the inference process by leveraging the same pre-trained model. Second, we observe that using features extracted by Whisper leads to better DA classification re- sults than using other popularly used features for similar tasks such as MFCC, eGeMAPS (Eyben al., 2016), and even transformer-bases features like Wav2Vec2 (Baevski et al., 2020) embeddings. We provide a detailed comparison of the model performance using these features in $ 5.
7/1' L.TM,, (w// T..t1.1)
(2)
1/ = concat( T. ...5 E RR.. 2d.
(3)
We use the self self-attention mechanism to calculate the utterance embedding sf from the token embeddings as a weighted sum of all the hidden states from the Bi-LSTM network. To achieve this, the hidden vector h/4" f is passed through simple multi-layer perceptron (MLP) with non-lincarity function (tanh) to generate new hidden representation. This, in turn, is passed through another single- neuron MLP followed by a softmax to determine the attention weight for each token, a in the utterance. This process can be represented by the following

---

Cornect DwaEE
ulterance orjemantttion
Libar Garcut
Stage
We ietakoo size maodeang
Contest Bi-LSVNG
Fusion
Atsationn
Audio Uditooal
stage Fusern
Testual Uninedal
ullssance rppomomitation
uesernnce cejresttaaao
Goncstatation
Mueus mat--ooling
Ulteoasee
Cane
Estraction
Dernction
Uninodal Inatones
Unimaabé
W ST
Raw Audio
ASR
(a) Unimodal Encoders
(b) Overall Model Architecture
(c) Barly-stage Pusion Eacoder
Figure The proposed model architecture.
set of equations:
token or frame representations.
p tanh( Whh! bi)))
(4)
pf tanh Wyh b3)
(8)
af softmax, (W (W
(5)
Pj ja tanh W4 Wah ba)
(9)
$ Σ a N/. RR2d,
(6)
Pairwise dot-products between intermediate token and frame representations are computed using ma- trix multiplication. Considering, Pf e &x2dd and pa mm22dd are the two matrices of the interme- diate token and frame representations of all the utterances, The following matrix multiplication is employed compute the attention weights.
where Wiz12 and by represent the weights and bias of the MLP layers respectively.
The audio utterance representation set is com- puted by the Bi-LSTM, followed self-attention mechanism in an identical manner. Finally the two representations are concatenated to compute the multimodal utterance representation sy:
S P/ x (P9) RR kxm
(10)
Si concat(s(.. Add
(7)
Here, & and i represents the number tokens each utterance and the number of frames in each utterance, respectively, and 2d,, is the embedding dimension. Another MLP transformation is applied the audio frame embedding matrix and finally multiplied with the with the attention weights calculate the attended token embeddings,
As an alternative design for early-stage fusion encoding, we investigate the integration cross- attention between token and frame representations when generating the multimodal utterance repre- sentation. The objective of incorporating cross- attention is to learn soft alignment score that captures the relationship between the textual and audio modalities within the same utterance. While calculating cross attention, we use either the text modality as source and audio as target or vice versa. Two cross-attention based utterance embedding are computed, one is s/.0 where, the attention weights are calculated using the dot product of the two modality representations, A!" and ky."" and based the attention weights the token embeddings aggregated
Pf S MLP((P)) kx2dd
(11)
Finally, the average of all the cross-attended token embeddings per utterance is computed to gener- ate the utterance embedding, si-a. Another cross- attended utterance embedding siu. is also calculated in a similar manner. The final utterance represen- tation the concatenation of the self and cross attended utterance representations.
utterance embedding. The s/ is calculated in a similar manner. To achieve this,
the token embeddings and frame embeddings are passed through individual simple MLP layers fol- lowed by tanh activation to generate intermediate
S; = (s st.0 s(it) E 8dh
(12)

---

This utterance representation is then passed through single-layer MLP that projects it lower- dimensional space for further processing.
from token vectors can be summarized as follows:
4 = maxpoll(CNN((w)))
w w w......,, A.d
Once have generated the multimodal utter- ance representations, incorporate contextual information from the previous utterances into the utterance representation. To achieve this, we use a window width w that includes the current ut- terance representation and w-/ previous utterance representations for modeling the context-aware ut- terance representation. The context modeling pro- cess follows similar approach to the utterance modeling. The context encoder consists of Bi- LSTM and attention mechanism. The Bi-LSTM encodes the utterance representations into hidden states 8t. while the attention mechanism calculates the attention weight B for each of the hidden states. The final context-aware utterance representation is calculated as follows:
(14)
c Σ 7:LSTM(u)) E E
(15)
Similarly, we compute a contextualized audio rep- resentation c for each utterance and pass both representations to the final late-stage fusion classi- fier module.
While the early-stage fusion encoder is structured with Bi-LSTM and attention-basee design, the uni- modal encoder is designed using convolution and maxpooling to model utterances based on token frame embeddings. The 1D convolution essentially functions temporal convolution, facilitating the integration of local context when generating utterance embeddings. Our experimental findings have demonstrated that the alternative encoder de- sign, featuring convolutional filters, outperforms the use of similar LSTM-based architecture in the cemmodal-encoder branches. Our hypothesis is that the convolutional filters employed in the unimodal branches capture unique utterance representations compared to the LSTMs used in early-stage fusion encoders, resulting in this performance improve- ment.
Σ Bigi
Ci
(13)
The context-aware utterance representation is finally passed through single-layer MLP that projects to a lower-dimensional space.
3.3.2 Unimodal Encoders
3.3.3 Late-stage Fusion Classifier
This module integrates the multimodal and uni- modal contextualized representations from the pre- ceding layers by concatenating them, thereby pro- ducing a combined representation, which is subse- quently fed into two-layer MLP to perform DA classification.
While the early-stage fusion encoder focuses on fusing modalities effectively, our approach also incorporates modality-specific encoders that model context-aware utterance representations from text and audio individually. To develop these uni- modal encoders, we augment the model proposed by Ortega and Thang Vu (2018).
z' concat(ci,cf.coo
(16)
First, we use ID CNN followed by max-pooling effectively model utterance representations from the token or frame vectors of an utterance. The kernel size and number filters for CNN are two hyperparameters that we tune for the best perfor- mance of the model. Finally, we contextualize the utterance representations in a similar manner as described in the context modeling for carly- fu- sion. To achieve this, attention mechanism, which generates contextu- alized textual utterance representations by attending to past utterance representations in given window. The process of generating utterance representation
z! ReLUU be))
(17)
8 argmax(softmax bea))
(18)
Here, (ct..c)), and (ct.e2) represent the weights and biases of the MLP layers in the classifier, re- spectively. The final MLP layer projects the joint representation to C-dimensional embedding space, where C' is the number of classes. Softmax acti- vation is applied calculate the probability of the utterance belonging to each class. Finally, the argmax operation is applied to predict the DA cor- responding to the utterance.
LSTTM model with

---

Dialogs
Utterances
Dataset
|C]
|/]
Train
Val
Test
Train
Val
Test
MRDA
52
1442.5
12
12
75K
16.4K
16.7K
EMOTyDA
12
9.6
727
104
208
6.9K
IK
2.1K
EMOTyDA The EMOTYDA dataset contains DA annotations for two popular datasets for emotion recognition in conversations: MELD (Poria et al., 2019) and IEMOCAP (Busso al., 2008). For consistency with the choice of our first dataset, we choose the multiparty split of the EMOTyDA data which is the DA adaptation of the MELD corpus. We refer to this split as EMOTyDA in the rest of the Table 2: Number of classes |C], utterances per dialog [/1, and number of dialogs and utterances in each split.
4
Experimental Setup
paper. The dataset consists of 1039 short dialogs with an average length of 9.6 utterances. The 12 most commonly occurring DA tags out of the 42 WWBD-DAMSS (Jurafsky and Shriberg, 1997) tags used to annotate utterances of the EMOTyDA dataset. Saha et al. (2020) argue that, given the short nature of the dialogs, 12 tags are enough capture the granular roles the utterances the dialogs, and use 831 dialogs for training and 208 dialogs for testing. We further split the original training set into 727 training and 104 validation dialogs and kept the original test set intact for a fair comparison.
4.1
Datasets
We conduct experiments and present our findings based on two publicly available benchmark datasets that contain audio recordings of multi-party conver- sations: the Meeting Recorder Dialog Act (MRDA) corpus (Shriberg et al., 2004) and the EMOTyDA dataset (Saha et al., 2020). We provide the statistics the datasets in Table 2.
The MRDA corpus consists 75 multi- party meetings, each of which is considered sin- gle dialog. The dialogs are on average of 1442.5 utterances in length. The dataset provides both oracle transcriptions and corresponding audio sig- nals, enabling online DA classification. We split the meetings into 51 train, 12 validation and 12 test meetings. The MRDA corpus adopts DA labeling scheme comprised of 52 distinct DA tags (Dhillon et al., 2004) that can be further clustered at diffcrent levels of granularity.
MRDA
4.2 Evaluation Metric
Both the MRDA and the EMOTyDA datasets tain a high degree of class imbalance, evi- dent from the label distributions presented in Ap- pendix A. The problem is more severe in the MRDA dataset, where the five most frequent la- bels, namely Statement (s), Backchannel (b), Floor Holder (fh). Acknowledgement (bk), and Accept (aa), collectively account for approximately 65% of the total utterances. In contrast, the least frequent 12 tags constitute just 1% of the dataset. Simi- larly, in the EMOTyDA dataset, the top classes, namely Statement-NonOpinian (s), Question (q), Answer (ans), Stutement-Opinion (o), and Others (oth), make up approximately 80% of the data.
Specifically, these dialog acts are grouped into a general set of 12 tags and basic set of 5 tags. The "basic-tags" grouping schemne introduced by Ang al. (2005) has been widely adopted by subse- quent studies, including Ortega and Vu 2)1 Li al. (2019); Raheja and Tetreault (2019); He et al. (2021). However, we contend that relying solely on the 5 basic tags limits our ability to capture the nu- anced roles of utterances in the dialog. Hence, we use the full set of DA tags for a more fine-grained analysis.'
Thus, argue that while previous works have predominantly used accuracy metric for report- ing model performance, is not the most suitable metric for evaluating the performance of models trained on these imbalanced datasets. An accuracy- based evaluation can be misleading, as a model that performs well on the majority classes may achieve an overall higher accuracy while neglecting the mi- nority classes. To address this limitation, use the macro Fl score as our primary evaluation metric. This metric provides a more balanced assessment of model performance in this highly imbalanced setting. We also report the accuracy of the models to consistent with prior work.
Diffferent gramularities of DA lahels are available here: https://sst //tpthuggttttttt// cww Sarprrann///AAAAAA Corpus

---

4.3 Implementation Details
(2021) is the most recent baseline for DA classifica- tion We adapt it for online DA classification by training with only ASR generated transcriptions.
We employ Huggingface to utilize pre-trained RoBERTa and Whisper models for our feature ex- traction modules. The proposed models are trained using Categorical Cross-entropy loss function and optimized using the Adam optimizer. Appendix provides detailed descriptions on hyperparameters, and Appendix E provides descriptions on compute hardware used and time requirements.
5.2 Audio Features
Various types of acoustic features have been ex- plored in the literature for different audio process- ing tasks such as emotion recognition, sentiment analysis and DA classification. Mel Spectrograms and MFCC (Mel-Frequency Cepstral Coefficients) are most prominently used (Ortega and Thang Vu, 2018; He et al., 2018; Saha et al., 2020; Rejaibi ct al.,
5 Results and Analysis
5.1 Baseline and Model Variants
2022: Chudasama et al., 2022). In recent years, the extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS), an expert knowledge-based compi- lation of 88 acoustic features (Li et al., 2020; Meng et al., 2022; Haider et al., 2021) and transformer- based acoustic feature extraction method using wav2vec2 (Pepino et al., 2021; Sharma, 2022) have gained attention for modeling emotions from audio signals. For our proposed approach, we extract the last hidden states of Whisper encoder as our acoustic features for input utterances.
We have evaluated our proposed Hierarchical Fu- sion (HF) model for online DA classification by comparing it against three strong related baselines and four variants of our own model.
5.1.1 Model variants
EF Model consists solely of the Etrly-stage Fu- sion Encoder component of our proposed model. Audio-Only Model is variant of our model that exclusively utilizes audio as input. In this config- uration, the textual utterance encoder from the EF Model is removed. Text-Only Model is the text counterpart of the Audio-Only Model, using only text as input. LF Model is a late-stage fusion vari- ant that leverages the Audio-Only and Text-Only models to generate contextualized unimodal repre- sentations. In this variant of the model, modality fusion occurs after contextualization while in the EF Model, the contextualization occurs after modality fusion. Finally the HF Model comprises of both the Early-stage Fusion Encoder and convolution based Unimodal Encoders.
Audio Feature
Accuracy
Macro Fl [
MFCC
49.55
8.42
eGeMAPs
48,95
8.31
Wav2Nec2
52.52
18.92
Whisper Encoder
55.63
25.44
Table 3: Performance of the Audio-Only model using various audio features on MRDA dataset.
We have evaluated the performance our Audio- Only model on the MRDA dataset, employing the aforementioned feature sets. As shown in Table 3, audio features extracted from the Whisper encoder yield the highest performance in terms of both accuracy and F1. An improvement of ~ percentage points is achieved over the nearest feature.
5.1.2 Related baselines
Related baselines we compare our model with are:
Lexico-acoustic Model: Ortega and Thang Vu (2018) is one of the very few related works that per- forms multimodal DA classification which creates an even ground for comparison with our proposed multimodal DA classification approach. MulT: Tsai et al. (2019) is originally proposed for multi- modal emotion recognition which uses transformer architecture and cross- modal attention. We adapt the model for online DA classification due to its reproducibility and popularity relevant baseline in recent works. Speaker Turn Modeling: He et al.
5.3 Online DA classification
Given that we are exploring a novel setup of online DA classification, which involves a fine-graine ine-grained analysis of MRDA, we lack directly comparable results from existing systems. Therefore, we adapt three relevant baselines to suit the requirements of online DA classification. To ensure fair com- parison, we use Whisper-generated transcripts and Whisper encoder features for training and evalu- ating all the baseline models on the two datasets. The results in Table 4 indicate the promising per- formance improvements achieved by the proposed
Phttps: hhttps:/huuginggface

---

MRDA
EMOTyDA
Model
AccuracyT
Macro FI1
Accuracy!
Macro FIÇ
Lexico-acoustic Model (Ortega and Thang Vu, 2018)
59.71
25.52
54.88
38.72
MulT (Tsai et al., 2019) Speaker Turn Modeling (He et al., 2021)
58,18
24.06
47.22
36.68
54.46
25.31
50.24
38.93
HP Model
59,86
29.11
55.58
41.95
Table 4: Performance comparison of the proposed and baseline models on MRDA and EMOTYDA datasets.
MRDA
EMOTyDA
Model
Accuracy1
Macro F11
Accuracy?
Macro F11
Audio-Oaly Model
55.63
25.44
49.06
36.71
Text-Only Model
54.39
23.72
52.07
36.86
LF Model
55.66
25.78
50.82
37.44
EF Model
57.69
28.73
53.03
EF Model + Cross Attention
57.9
28.63
52.28
37.96
HF Model
59.86 60.26
29.11
55.58
41.95
HF Model Cross Attention
28.9
55.25
41.4
Table 5: Ablation study of the proposed model.
model in online DA classification. Compared to the relevant baselines, our model demonstrates an increase in FI score of ~3.5 percentage points the MRDA dataset and ~3 percentage points on the EMOTyDA dataset.
ture of the MRDA audio signals, which introduces inherent challenges in accurate transcription by the ASR system. Furthermore, the LF model demon- strated minor improvement over the individual modalities in terms of Fl score for both datasets.
Notably, while the other baseline models incor- porate both audio signals and noisy transcriptions, the Speaker Turn Model stands out by achieving a higher Fl score for the EMOTyDA dataset, despite being trained only on the noisy transcriptions. This be attributed to the fact that the ASR generated transcriptions EMOTyDA
The most significant performance improvement was achieved by using early stage fusion (EF). The EF model exhibited almost 2 percentage points of improvement in Fl score over the LF model for both the datasets. This component addresses the major limitation of the previous DA classification research efforts which used multimodality of dialog modeling. Finally, by incorporating the unimodal encoders into the EF model, we observe an increased overall accuracy the model by ap- proximately percentage points without sacrificing the Fl score.
cleaner. Audio signals the MRDA dataset are inherently noisy, susceptible to inter-channel interferences, leading to even noisier ASR transcriptions.
later stage
Overall, the similar performance gain on both the datasets using the proposed model indicates the robustness of our approach.
We do not observe any notable improvement of the model performance by incorporating cross- attention while utterance encoding. This observa- tion is aligned with the findings of Rajan et al. (2022) for similar task. Overall, we obtain approximately 5.5 Fl points and 4.5 Fl points improvement over the textual model for the MRDA and EMOTyDA datasets respectively.
5.4 Ablation Study
To further analyze the impact of the individual components of our proposed model we perform a comprehensive ablation analysis (see Table 5). We compare the results of the complete HF model with the other variants described in subsubsection 5.1.1. First, examine the performance of the modality-specifie models on both datasets. The Audio-Only model outperforms the Text-Only model on the MRDA dataset, while the reverse scenario is observed for the EMOTyDA dataset. This discrepancy can attributed to the noisy na-
5.5 Class-wise Performance
To understand the effectiveness of multimodal HF model, We compare the performance of our HF model and Text-only model on individual classes. Out 52 MRDA classes, We achieve a performance

---

Compar son of Scores
Model
Multimodal
Accuracy
1/0
Ortega and Thang Vu (2018)
Y
84.7
Ravi and Kozareva (2018)
N
86.7
Raheja and Tetreault (2019)
N
91.1
He al. (2021)
N
91.4
Colombo al. (2020)
N
91.6
li (2018)
N
91.7
al. (2019)
N
92.2
Chapuis et al. (2020)
N
92.4
10
5.20
HF Model (online)
82.6
Model (oracle)
91.8
Table 6: Comparison with state-of-the-art models on
Figure 2: Impact of HF model on individual classes,
MRDA dataset trained with basic tags.
Model
Accuracy
Score
gain for 32 classes. Among the 7 question classes, the HF model experiences performance gain on question classes with 24 F1 points improvement for Or Question (qr). We can see the performance boost on a selected set of classes in Figure 2. For some classes like, Negative Answer (ng), Third-party Talk (13) the Text-Only model fails to correctly classify any utterance, while HF model achieves an Fl score of 8 and 69 for these classes. Similar trends have been observed for the EMOTyDA dataset, where the HF model achieves performance gain on 10 out of classes.
Saha al. (2020)
DA only (Text Audio)
40.30
41.16
DA ER (Text Audio)
49.42
41.69
DA ER (Text Video)
51.00
44.52
HF Model (online)
55.58
41.95
Model (oracle)
63.42
50.3
Table Comparison with statc-of-the-art models on EMOTyDA dataset
only trained on DA classification dataset. We con- duct similar experiments for the EMOTYDA dataset. The only ttate-of-the-art model for this dataset is Saha et al. (2020), which jointly models DA and emotion (ER). Our proposed model outperforms their best model, jointly trained on DA and ER with text and video, by 12% in terms of accuracy and 6 Fl points.
5.6 Comparison with SOTA Models
While our research primarily focuses on designing efficient DA classification model for online setup, also measure the performance of our model using oracle transcriptions. This allows us assess the effectiveness and robustness of our proposed approach in comparison to the statc-of- the-art DA classification models. From Table 6, we can observe that, our proposed HF model out- performs all but two existing models when trained with oracle transcriptions. Notably, et al. (2019) achieves higher performance by jointly modeling DAs and topics, leveraging the complementary information from both tasks. From the ablation studies of the corresponding paper, it is evident that without employing the topic modeling jointly with DA classification, the model performance de- creases by 2% and falls below the performance of our proposed model. Additionally, Chapuis et al. (2020) is large model pre-trained on OpenSubti- tles, a massive corpus of spoken conversations and fine-tuned relevant datasets for DA classifica- tion. The superior performance can be attributed to the pre-training on relevant corpus and hence performs slightly better than our approach which is
Conclusion
We have put forward Hierarchical Fusion Model for online Dialogue Act (DA) classification. The early-stage fusion encoder component of the pro- posed model generates multimodal utterance rep- resentations more effectively. We also propose a novel audio feature extraction method using Whis- per. Our detailed examination of different parts of our model confirms their relative importance and explains the effectiveness of our appraoch. The proposed Hierarchical Fusion Model yields markable 3% improvement in the F1 score the related bascline, and noteworthy 5% improvement over using a text based model for online DA classi- fication. Inspired by our exciting results from early fusion, in future we want to investigate even more granular fusion of the modalities at word level from unaligned data.

---

Limitations
either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Gov- ernment authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
One of the major limitations that we face is the GPU memory restrictions. We employ two large language models RoBERTa and Whisper for feature extraction. However, we have not tuned the param- eters of these models while extracting features. We believe an end-to-end training by incorporating these feature extraction models in the training will produce better features fordialog modeling. How- ever, having limited access to GPUs restrict us from doing
References
Ali Ahmadvand, Jason Ingyu Choi, and Eugene Agichtein. 2019. Contextual dialogue act classifi- cation for open-domain conversational agents. In Proceedings of the 42nd International ACM SIGIR Conference Research and Development in Informa- tion Retrieval, SIGIR' 19, page 1273-1276, New York, NY, USA. Association for Computing Machinery.
Another limitation that we have faced while working with the MRDA corpus the inherent alignment issues of the dataset. For some of the meeting recordings, the audio signal was not per- fectly aligned with the provided timestamps for utterances.
Jeremy Ang. Yang Liu, and Elizabeth Shriberg. 2005. Automatic dialog act segmentation and classification in multiparty meetings. Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005., 1L/1061-1/106 Vol. 1.
Ethics Statement
Online DA classification plays a vital role in con- versational agents and chatbots. The accurate iden- tification and categorization of Dialogue Acts are vital for understanding the intentions, queries, and responses within conversation. If the Dialogue Act recognition systems falter or generate unreli- able outcomes, it can result in misunderstandings, misinterpretations, and flawed responses from the conversational agents. Hence, it is crucial to exer- cise caution when integrating Dialogue Act clas- sification systems into real-world applications. A comprehensive evaluation of their performance, de- pendability, and resilience becomes essential before widespread deployment. Rigorous testing, valida- tion, and ongoing improvement efforts should be undertaken to enhance their accuracy, adaptabil- ity, and applicability in real-world conversational contexts. By ensuring the quality and reliability of Dialogue Act recognition systems, we can mitigate the potential negative impacts on end users and elevate the overall user experience in interactions with conversational agents.
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vee 2.0: A framework for self superviedd learning of speech representations.
Philippe Blache, Massina Abderrahmane, Stéphane Rauzy, Magalie Ochs, and Houda Oufaida. 2020. Two-level classification for dialogue act recognition in task-oriented dialogues. In Proceedings of the 28th International Conference on Computational Linguis- tics, pages 4915-4925, Barcelona, Spain (Online). International Committee on Computational Linguis- tics.
Chandrakant Bothe, Cornelius Weber, Sven Magg, and Stefan Wermter. 2018. A context-based approach for dialogue act recognition using simple recurrent neural networks. In Proceedings of the Fleventh International Conference Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Carlos Busso, Murtaza Bulut, Chi-Chur Lee, Abe Kazemzadeh, Emily Mower Provost, Samuel Kim, Jeannette Chang. Sungbok Lee, and Shrikanth Narayanan. 2008 lemocap: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42:335-359.
Emile Chapuis, Pierre Colombo, Mattco Manica, Matthieu Labeau, and Chloé Clavel. 2020. Hier- archical pre-training for sequence labelling in spoken dialog. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 2636-2648, Online. Association for Computational Linguistics.
Acknowledgements
We would like to thank the anonymous reviewers for their valuable feedback and input. Research was sponsored by the Army Research Office and was accomplished under Grant Number W911NE-20--- 0002. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies,
Vishal Chudasama, Purbayan Kar, Ashish Gudmalwar, Nirmesh Shah, Pankaj Wasnik, and Naoyuki Onoe. 2022. M2fnet: Multi-modal fusion network for emo- tion recognition in conversation.

---

Pierre Colombo, Emile Chapuis, Matteo Manica, Em- manuel Vignon, Giovanna Varni, and Chloe Clavel. 2020. Guiding attention in sequence-to-sequenee models for dialogue act prediction. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7594-761.
Jeng-Lin Li, Tzu-Yun Huang, Chun-Min Chang, and Chi- Chun Lee. 2020. A ceveform-feature dual branch acoustic embedding network for emotion recognition. Frontiers in Computer Science, 2.
Ruizhe Li, Chenghua Lin, Matthew Collinson, Xiao Li, and Guanyi Chen. 2019. A dual- -ttenntttnioon hierarchical recurrent neural network for dialogue act classifica- tion. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 383-392 Hong Kong. China. Association for Computational Linguistics.
Rajdip Dhillon, Sonali Bhagat, Hannah Carvey, and Eliiaabtth Shriberg. 2004. Meeting recorder project: Dialog act labeling guide.
Florian Eyben, Klaus R. Scherer, Björn W. Schuller, Johan Sundberg, Elisabeth André, Carlos Busso, Laurence Y. Devillers, Julien Epps. Petri Laukka. Shrikanth S. Narayanan, and Khiet P. Truong. 2016. The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective comput- ing. IEEE Transactions on Affective Computing, 1(2):190-202.
Zhegian li, Ronggin Yang. Zhou Zhao, Deng Cai, and Xiaofei He. 2018. Dialogue act recognition via crf- attentive structured network. In The 41st Interna- rional ACM SIGIR Conference Research & De- velopment in Information Retrieval, SIGIR '18, page 225- 234, New York, NY, USA. Association for Com- puting Machinery.
Chih-Wen Goo and Yun-Nung (Vivian) Chen. 2018. Abstractive dialogue summarization with sentence- gated modeling optimized by dialogue acts. 2018 IEEE Spoken Language Technology Workshop (SLT),
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy. Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
pages 735-742.
Liyu Meng, Yuchen Liu, Xiaolong Liu, Zhaopei Huang. Yuan Cheng, Meng Wang, Chuanhe Liu, and Qin Jin. 2022. Multi-modal emotion estimation for in-the- wild videos.
Fasih Haider, Senja Pollak, Pierre Albert, and Saturnino Luz. 2021. Emotion recognition in low-resource settings: An evaluation of automatic feature selection methods. Computer Speech Language, 65: 15:10110 101119) 19.
Daniel Ortega, Chia-Yul.i Gisela Vallejo, Pavel Denisov, and Ngoc Thang Vu. 2019. Context- aware neural- based dialog act classification on automatically gen- erated transcriptions. In ICASSP 2019 2019 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 7265-7269.
Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zim- mermann. 2018. Conversational memory network for emotion recognition in dyadic dialogue videos. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long Papers), pages 2122-2132, New Orleans, Louisiana. Association for Computational Linguistics.
Daniel Ortega and Ngoc Thang Vu. 2018. Lexico- acoustic neural-basee based models for dialog act classifi- cation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 6194-6198.
Xuanli He, Quan Hung Tran, William Havard, Laurent Besacier, Ingrid Zukerman, and Gholamreza Haffari. 2018. Exploring textual and speech information in dialogue act classification with speaker domain adaptation.
Daniel Ortega and Ngoc Thang Vu. 2017. Neural- based context representation learning for dialog act classification. In Proceedings of the 18th Annual SIGdial Meeting Discourse and Dialogue, pages 247-252, Saarbrücken, Germany. Association for Computational Linguistics.
Zihao He, Leili Tavabi, Kristina Lerman, and Moham- mad Soleymani. 2021. Speaker turn modeling for dialogue act classification. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2021, pages 50522157, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tatsuro Oya and Giuseppe Carenini. 2014. Extrac- tive summarization and dialogue act modeling on email threads: An integrated probabilistic approach. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 133-140, Philadelphia, PA, U.S.A. Association for Computational Linguistics.
Dan Jurafsky and Elizabeth Shriberg. 1997. Switchboard swbd-damsl thallow-discours--fincee function annotation coders manual.
Leonardo Pepino, Pablo Riera, and Luciana Ferrer. 2021. Emotion recognition from speech using wav2vec 2.0 embeddings.
Daniel Jurafsky, Elizabeth Shriberg, Barbara Fox, and Traci Curl. 1998. Lexical, prosodic, and syntactic cues for dialog acts. In Discourse Relations and Discourse Markers.
Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Mazumder, Amir Zadeh, and couis-Philippe Morency. 2017. Multi-level multiple attentions for

---

contextual multimodal sentiment analysis. In 2017 IEEE International Conference on Data Mining (ICDM), pages 1033-1038.
Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor. Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339-33 -339344...
Soujanya Poria, Devamanyu Hazarika, Navonil Ma- jumder, Gautam Naik, Erik Cambria, and Rada Mihal- cea. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceed- ings the 57th Annual Meeting of the Association for Computational Linguistics, pages 527-536, Florence. Italy. Association for Computational Linguisties.
Yao-Hung Hubert Tsai, Shaojic Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Rus- lan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6558- 6569, Florence, Italy. Association for Computational Linguistics.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision.
Vipul Raheja and Joel Tetreault. 2019. Dialogue Act Classification with Context-Aware Self- Attentinnnnon. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers). pages 3727-3733, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Dong Wang. Ziran Li, Haitao Zheng, and Ying Shen. 2020a. Integrating user history into heterogeneous graph for dialogue act recognition. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4211-4221, Barcelona, Spain (On- line). International Committee on Computational Lin- guistics.
Vandana Rajan, Alessio Brutti, and Andrea Cavallaro. 2022. Is cross-attention preferable to self-attention for multi-modal emotion recognition?
Sujith Ravi and Zornitsa Kozareva. 2018. Self- seff-govenning neural networks for on-device short text classifica- tion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 887-893, Brussels, Belgium. Association for Computational Linguistics.
Kai Wang. Junfeng Tian, Rui Wang. Xiaojun Quan, and Jianxing Yu. 2020b. Multi-domain dialogue acts and response CO co-ggenrration. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7125-7134, Online. Association for Computational Linguistics.
Emna Rejaibi, Ali Komaty, Fabrice Meriaudeau, Said Agrebi, and Alice Othmani. 2022. Mfcc-based recur- rent neural network for automatic clinical depression recognition and assessment from speech. Biomedical Signal Processing and Control, 71:103107.
Andrew Wood, Zachary Eberhart, and Collin McMillan. 2020. Dialogue act classification for virtual agents for software engineers during debugging. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW" page 462-469, New York, NY, USA. Association for Computing Machinery.
Tulika Saha, Aditya Patra, Sripamn Saha, and Pushpak Bhattacharyya. 2020. Towards emotion-aided multi- modal dialogue act classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1361-43722 Online. Association for Computational I. Linguistics.
Piotr Zelasko, Raghavendra Pappagari, and Najim Dehak. 2021. What helps transformers recognize conversa- tional structure? importance of context, punctuation, and labels in dialog act recognition. Transactions of the Association for Computational Linguistics. 9:1163-1179 1179.
John R. Searle. 1969. Speech Acts: An Essay the Phi- losophy of Language. Cambridge University Press.
Wenjie Zheng, Jianfei Yu, Rui Xia, and Shijin Wang. 2023. A facial expression-ave aware multimodal multi- task learning framework for emotion recognition in multi pariippptyy conversations. In Proceedings of the s1 Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 15445-15453 Toronto, Canada. Association for Com- putational Linguistics.
Mayank Sharma. 2022. Multi lingual multi-task speech emotion recognition using wav2vee 2.0. ICASSP 2022 - 2022 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 6907-6911.
Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang. and Hannah Carvey, 2004. The ICSI meet- ing recorder dialog act (MRDA) corpus. In Pro- ceedings of the 5th SIGdial Workshop on Discourse and Dialogue HLT-NAACL 2004, pages 97-100, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
A Label Distribution for MRDA and EMOTyDA
Figure shows the label distribution for the MRDA and EMOTYDA datasets.

---

200
11:00
MRDA
EMOTyDA
Figure 3: Label distribution of the two datasets.
Feature
Macro F1
Accuracy
Name
Best Value
Last hidden state
29.11 28.42
59.86
Hidden dimensions of encoders
128
Average of last hidden states
59.34
Kernel size
5
No of channels for layers
256
Table Comparison of different configurations of RoBERTa features
Window size for context modeling
3
Table Choice of hyperparameters
B Configuration of Textual Feature Extraction
choice of window size and stride for Whisper is consistent with that of other audio feature extraction methods across the literature. The final output from the whisper encoder is 1500 frames per utterance with 512 dimensional embedding per frame.
We use pre-trained RoBERTa as feature extractor to extract token embeddings from the textual input. The choice of this feature extractor is consistent with contemporary works on DA classification (He al., 2021) and other NLP tasks (Zheng al., 2023; Chudasama et al., 2022) in the literature. RoBERTa is also more recent and proven to be better than BERT on several NLP benchmarks and hence better choice of feature extractor.
D Hyperparameters
To control the learning rate, we implemented a linear LR scheduler that gradually decreased the rate from the initial value of 1e-3 after 5 epochs. For the MRDA dataset, our experiments are conducted over 40 epochs, while for the EMOTyDA dataset, the experiments were carried out for 20 epochs. To prevent overfitting, used carly stopping, ceasing training if improvement in validation macro Fl score was observed for consecutive epochs.
We experimented with two variants of RoBERTa features, one where the token embeddings the average of last hidden states and the other where the token embeddings are solely from last hidden state. As shown in Table 8, we did not find any sig- nificant improvement using the intermediate layers and hence continued with the last hidden states for feature extraction for all the experimentation.
The best model for inference on the test set was selected based on macro Fl score on the validation set. To ensure the reliability of the results, each experiment was repeated 5 times with different random seeds, and the reported results represent the average performance across these runs.
Configuration of Audio Feature Extraction
A list of hyperparameters and the chosen values upon experimentation shown in Table
For extracting frame embeddings from the audio, we used the pre-trained model, Whisper and followed the same setup that used in original Whisper implementation for pre-processing audio inputs. The input audio utterance padded or clipped to 30s and 25ms window with 10 ms stride is employed to extract the mel-spectrograms. The
E Training and Inference
All experiments were performed on a single NVIDIA RTX A6000 GPU. Training an epoch

---

the MRDA dataset took approximately 40 min- utes, with an additional 12 minutes each for valida- tion and testing. Training time can be reduced by caching the audio features but in turns increase the memory requirement on the CPU. Training an epoch on the EA dataset took approximately 7 minutes, with an additional 2 minutes each for validation and testing. The latency of the model at inference time is 0.16 seconds to infer dialog act for each utterance.


Reviews:
[{"review_id": "t2XnE5ECwj", "reviewer": "Reviewer_7qHe", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback and suggestions! We feel encouraged by your appreciation towards our Dialog Act (DA) classification model, which is capable of working with automatic error-prone transcriptions. We\nrespond to some of your feedback and comments below.\n\n### Clarifying the results in the oracle setup\n\n> When comparing the results with those of previous approaches on the MRDA corpus, the proposed approach is still outperformed by an unimodal approach, even when relying on oracle transcriptions. This suggests that the parts of the system that deal with textual information can be improved.\n\nIn the oracle setup where we have used clean transcriptions to train and evaluate the model, our proposed model outperforms all but two of the related baselines. These results are justified in the *section 5.6, lines 582--589* of the paper. To elaborate, one of the two unimodal approaches are Li et al. [1], which joint models DA and topic to enhance the performance of DA classification. From the ablation studies of the corresponding paper, it is evident that without employing the topic modeling task jointly with DA classification, the model performance decreases by 2% and falls below the performance of our proposed model. The second one, Chapius et al. [2] is essentially a large model pre-trained on OpenSubtitles, a massive corpus of spoken conversations and fine-tuned on relevant datasets for DA classification. The superior performance can be attributed to the pre-training on relevant corpus and hence performs slightly better than our approach which is only trained on DA classification dataset.  \n\n### Missing References\n\n> The related work on dialog act recognition in the paper starts in 2017. Considering that starting date, the authors miss several papers related to the task.\n\nWe want to thank you for drawing our attention to all the relevant papers. Bothe et al. (2018), Chen et al. (2018), Kumar et al. (2018), Wang et al. (2021), and Zelasko et al. (2021) were actually explored during the literature review phases of the work but later missed in the paper. We plan to cite these and some of the other suggested papers in the final version of the paper.\n\n- [1] Li et al. 2019.  [A dual-attention hierarchical recurrent neural network for dialogue act classification](https://aclanthology.org/K19-1036/). \n\n- [2] Chapius et al. 2021. [Hierarchical pre-training for sequence labelling in spoken dialog](https://aclanthology.org/2020.findings-emnlp.239/).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "When considering real-time analysis of spoken interaction, the ability to work with automatic error-prone transcriptions is particularly important. This paper addresses that problem in the context of the dialog act recognition task, which is an important step in the understanding process.", "reasons_to_reject": "When comparing the results with those of previous approaches on the MRDA corpus, the proposed approach is still outperformed by an unimodal approach, even when relying on oracle transcriptions. This suggests that the parts of the system that deal with textual information can be improved. Furthermore, the related work on dialog act recognition is missing several references.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a multimodal approach to dialog act recognition that combines audio and textual information and avoids the need for oracle transcriptions. The results on two datasets (MRDA and EMOTyDA) reveal competitive performance with state-of-the-art approaches.\n\nNOTE: This is an emergency review, so I'm sorry for not going into much detail.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "(573) Something is missing after \"for an online\" "}, "extra_text": {"Missing_References": "The related work on dialog act recognition in the paper starts in 2017. Considering that starting date, the authors miss several papers related to the task. For instance, and non-exhaustively:\n\nBothe et al. (2018) A Context-based Approach for Dialogue Act Recognition using Simple Recurrent Neural Networks\nChen et al. (2018) Dialogue Act Recognition via CRF-Attentive Structured Network\nKumar et al. (2018) Dialogue Act Sequence Labeling Using Hierarchical Encoder With CRF\nWan et al. (2018) Improved Dynamic Memory Network for Dialogue Act Classification with Adversarial Training\nRibeiro et al. (2019) Deep Dialog Act Recognition using Multiple Token, Segment, and Context Information Representations\nRen et al. (2020) Intention Detection Based on Siamese Neural Network With Triplet Loss\nWang et al. (2020) Integrating User History into Heterogeneous Graph for Dialogue Act Recognition\nWang et al. (2021) Balance the Labels: Hierarchical Label Structured Network for Dialogue Act Recognition\nZelasko et al. (2021) What Helps Transformers Recognize Conversational Structure? Importance of Context, Punctuation, and Labels in Dialog Act Recognition\n\nI'm not saying that all of these papers should be cited, but the authors should at least make it clear that the papers cited in the related work are just some examples of existing approaches to the task. However, I think that these papers should be checked, as they may provide insight for future work by the authors.\n\nFurthermore, when possible, peer-reviewed publications should be cited instead of their pre-print counterparts. For instance, (Colombo, 2020) was published in the proceedings of AAAI."}}, {"review_id": "ittiPBZugf", "reviewer": "Reviewer_YaT8", "summary": "", "questions": "Q.A: Any specific reason to use RoBERTa as the feature extractor for text rather than other foundation models?\n\nQ.B: A lot of previous research showed using the intermediate layer outputs may lead better generalization compared to the last layer. Did author compare the features extracted by other layers?\n\nQ.C: For the early stage fusion, can we also consider the positional encoding + Transformer as the fusion architecture? Is there any specific reason to use Bi-LSTM rather than other architectures?\n\nQ.D: Typically, 1D Convolution model are used to extract the features from raw data such as word-level vectors and raw audio wav. Considering the proposal system have already used pre-trained model to generate the feature-level representations, why should we still use Conv models for the uni-modality encoders?\n\nQ.E: Since Whisper also targets for the ASR task, just wondering if we can directly use the Whisper ASR transcripts for DA task? What about the performance compared to the proposed multimodal design?\n\nQ.F:  Do we also need to consider the latency for online DA? Since it will be a real-time system, I think latency should be an important evaluation metric. ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and suggestions. We are glad that you perceive our experiments as thorough and comprehensive, and find our results compelling. We\nrespond to your feedback and comments below.\n\n### Comments\n\n> The novelty of hierarchical fusion is relatively limited, as the idea of combining uni-modality and multi-modality fusion is not new. Similar concepts have been previously introduced in dialog-based multimodal systems, and specific architectural designs require further explanation.\n\nWe agree with the reviewer that the term 'Hierarchical Fusion' is used across the literature for multimodal fusion. However, in most of the recent literatures, the term hierarchical fusion is used in a different perspective compared to ours. For instance, in the papers Huang et al. [1], Majumder et al. [2] and Wang et al. [3], hierarchical fusion is used to represent a common idea where the features from two out three available modalities (audio, text and video) are concatenated first to synthesize bi-modal features. These bi-modal features are in turns concatenated to synthesize the tri-modal features which are eventually used for classification. On the contrary, in our proposed approach, we only have two features (text and audio). In **early-stage fusion**, we first extract the utterance embeddings individually from the modalities to combine them in a single utterance representation and then we incorporate contextual information. Whereas, in the unimodal encoder branches, we contextualize the encodings from the individual modalities separately. Finally we combine the three contextualized encodings, which is referred to as **late-stage fusion**. Essentially, in the context of our work, hierarchical fusion (HF) refers to fusing the two modalities before and after contextualization while the existing multimodal approaches at least for DA classification only fuse after contextualizing the features. From ablation studies presented in table 5, we observe that, we only get <1% improvement in F1 score if we fuse the modality representations after contextualization (LF model), while if we first combine and then contextualize (EF model), we get 3% improvement in F1 score. If we choose to combine at both levels (HF model), we get further improvement in F1 score and 2% improvement in accuracy. \n\n### Questions\n\n> Q.A: Any specific reason to use RoBERTa as the feature extractor for text rather than other foundation models?\n\n**Response:** We use RoBERTa in order to be consistent with contemporary works on DA classification (He et al. [4]) and other NLP tasks (Zheng et al. [5], Chudasama et al. [6] etc.) in the literature. RoBERTa is also more recent and proven to be better than BERT on several NLP benchmarks and hence a better choice of feature extractor. In any case, the choice of the textual feature extractor is not the main contribution of this paper contribution, and thus we are fairly agnostic to it.\n\n> Q.B: A lot of previous research showed using the intermediate layer outputs may lead better generalization compared to the last layer. Did author compare the features extracted by other layers?\n\n**Response:** In the early stages of our experiment we employed both variants of RoBERTa features, one where the token embeddings are the average of last 4 hidden states and the other where the token embeddings are from last hidden state. As shown in the following table, we did not find any significant imrovement using the intermediate layers and hence continued with the last hidden states for feature extraction. \n\n|   Feature   | Macro F1|    Accuracy  |\n| :---        |    :----:   |          ---: |\n| Last hidden state      | **29.11**       |  **59.86** |\n| Average of last 4 hidden states  | 28.42       | 59.34      |\n\n> Q.C: For the early stage fusion, can we also consider the positional encoding + Transformer as the fusion architecture? Is there any specific reason to use Bi-LSTM rather than other architectures?\n\n**Response:** In the proposed model, LSTM + attention architechture is used for two different functionalities. The first one is modelling utterances from token (text) or frame (audio) embeddings and the second one is context modelling. \n\nFor modelling utterances from token or frame embeddings, we are already extracting features from RoBERTa and Whisper models which use Transformer architecture to compute features. Essentially, we are using the LSTM+attention model on top of the features to aggregate the token embeddings into utterance embedding. We have also tried to experiment with fully self-attention based architecture for modelling utterances but that did not outperform the LSTM based encoder. We have experimented with different combinations of up-to 3 layers and 6 attention heads due to computational resource constraints. \n\nOn the other hand, to model contexts, we do not need infinitely long context window because for modelling context-aware utterance embeddings, only previous few utterances are important and a simple LSTM model is sufficient to achieve that.\n\nInfact, one of the related works (Multimodal Transformer, MulT) that we have mentioned in the paper, uses end-to-end Transformer architecture and could not outperform our model, which indicates the importance of separate utternace and dialog modelling units we employ in the proposed model.\n\n> Q.D: Typically, 1D Convolution model are used to extract the features from raw data such as word-level vectors and raw audio wav. Considering the proposal system have already used pre-trained model to generate the feature-level representations, why should we still use Conv models for the uni-modality encoders?\n\n**Response:** Indeed, the 1d conv+maxpooling architecture is employed in the unimodal encoders to model the utterances from token or frame embeddings. 1D convolution essentially works as a temporal convolution that helps to incorporate local context while computing utterance embedding from token embeddings. We have used LSTM+attention for utterance modelling in the early-stage fusion encoder already and our experiments show that using an alternative design of encoder with conv filters provide better results than using the similar LSTM based architecture in the unimodal encoder branches. Essentially, this was a design choice based on empirical results. Our hypothesis is, the conv filters in the unimodal branches learn different representations of utterances than LSTMs employed in the early-stage fusion encoders and hence contribute to the better results. \n\n> Q.E: Since Whisper also targets for the ASR task, just wondering if we can directly use the Whisper ASR transcripts for DA task? What about the performance compared to the proposed multimodal design?\n\n**Response:** This is already implemented in the paper and in-fact we use Whisper to generate transcriptions for all the text-based experiments in the online setup of our work. The comparison of the Whisper generated text-only model with the HF model can be found in table 4 of the paper. \n\n> Q.F: Do we also need to consider the latency for online DA? Since it will be a real-time system, I think latency should be an important evaluation metric.\n\n**Response:** We agree with the reviewer that latency is a very important factor for online model. However, since our proposed model is the first of the kind, there is no baseline to compare the latency with. We have computed the latency of the model and it is 0.16 seconds to infer dialog act for one utterance, which will serve as a comparable baseline for future online DA classification works.\n\n- [1] Huang et al. 2023.  [Multimodal Sentiment Analysis in Realistic Environments Based on Cross-Modal Hierarchical Fusion Network](https://www.mdpi.com/2079-9292/12/16/3504). \n\n- [2] Majumder et al. 2018. [Multimodal sentiment analysis using hierarchical fusion with context modeling](sciencedirect.com/science/article/abs/pii/S0950705118303897).\n\n- [3] Wang et al. 2021.  [Hierarchical Multimodal Fusion Network with Dynamic Multi-task Learning](https://ieeexplore.ieee.org/document/9599141). \n\n- [4] He et al. 2021. [Speaker Turn Modeling for Dialogue Act Classification](https://aclanthology.org/2021.findings-emnlp.185/).\n\n- [5] Zheng et al. 2013.  [A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party Conversations](https://aclanthology.org/2023.acl-long.861/). \n\n- [6] Chudasama et al. 2022. [M2FNet: Multi-modal Fusion Network for Emotion Recognition in Conversation](https://ieeexplore.ieee.org/document/9857351).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a thorough investigation with comprehensive experiments and quantitative analysis, comparing both baseline models and ablations.\n\n2. The results indicate that the introduced system outperforms state-of-the-art baselines, while also highlighting the effectiveness of the proposed hierarchical fusion design.", "reasons_to_reject": "1. The novelty of hierarchical fusion is relatively limited, as the idea of combining uni-modality and multi-modality fusion is not new. Similar concepts have been previously introduced in dialog-based multimodal systems, and specific architectural designs require further explanation.", "questions_to_authors": "Q.A: Any specific reason to use RoBERTa as the feature extractor for text rather than other foundation models?\n\nQ.B: A lot of previous research showed using the intermediate layer outputs may lead better generalization compared to the last layer. Did author compare the features extracted by other layers?\n\nQ.C: For the early stage fusion, can we also consider the positional encoding + Transformer as the fusion architecture? Is there any specific reason to use Bi-LSTM rather than other architectures?\n\nQ.D: Typically, 1D Convolution model are used to extract the features from raw data such as word-level vectors and raw audio wav. Considering the proposal system have already used pre-trained model to generate the feature-level representations, why should we still use Conv models for the uni-modality encoders?\n\nQ.E: Since Whisper also targets for the ASR task, just wondering if we can directly use the Whisper ASR transcripts for DA task? What about the performance compared to the proposed multimodal design?\n\nQ.F:  Do we also need to consider the latency for online DA? Since it will be a real-time system, I think latency should be an important evaluation metric. ", "paper_topic_and_main_contributions": "This paper present an multimodal approach that employs a hierarchical fusion technique to early-stage combine two modalities, yielding remarkable performance surpassing that of utilizing individual modalities for online DA classification. The proposed approach adeptly incorporates multimodal fusion strategies, effectively capturing both utterances and their context. The author also introduced a audio feature extraction process by using the pre-trained Whisper model to enhance the modeling of dialogs based on audio, resulting in improved DA classification performance. The result showed the proposed system achieved 3% improvement in the F1 score compared to the related baseline and an impressive 5% improvement over using a text-based model for online DA classification on MRDA and EMOTyDA dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The author proposed \"Hierarchical Fusion\" as the title of this paper. However, this concept only appeared in section introduction and result. I recommend to put several description sentence about this concept in proposed method section to better explain it. \n\n2. Typo, page 4 line 606, \"appraoch\"."}}, {"review_id": "rVXqqt4JUu", "reviewer": "Reviewer_ybDR", "summary": "", "questions": "Q1: Why not compare with SoTA model from Section 5.6 in the online setting experiments (Section 5.3)?\n\nQ2: The utterance encoder designs in the unimodal encoder and early-stage fusion encoder are different. The former is conv + max pooling, and the latter is BiLSTM + attention. What is the reason for making such decision? Due to the existence of alternative unimodal utterance encoders in the early-stage fusion encoder, we can conclude that **the early-stage fusion encoder** contributes the most, but it is questionable how much **modality fusing** contributes.\n\nQ3: Frame embedding sequences are usually very long and hard to encode without compressing into short sequences. The early-stage fusion encoder uses utterance BiLSTM to encode audio embeddings at a frame level. Will not it be very memory consuming and difficult to train? What is the average length of frame embedding sequences in your experiment?\n\nQ4: In the last paragraph of Section 3.3.1, the authors say cross attention is applied to textual and audio representations, but details of the cross attention are neither presented in Figure 1 nor by any equations. Can the authors provide more details?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks you for your valuable feedback and suggestions. We respond to your feedback and comments below.\n\n### Comparison with SoTA\n\n> In the oracle settings (that uses oracle texts as textual inputs), the proposed model slightly lags behind a SoTA unimodal baseline.\n\n**Response:** In the oracle setup where we have used clean transcriptions to train and evaluate the model, our proposed model outperforms all but two of the related baselines. These results are justified in the *section 5.6, lines 582--589* of the paper. To elaborate, one of the two unimodal approaches are Li et al. [1], which joint models DA and topic to enhance the performance of DA classification. From the ablation studies of the corresponding paper, it is evident that without employing the topic modeling task jointly with DA classification, the model performance decreases by 2% and falls below the performance of our proposed model. The second one, Chapius et al. [2] is essentially a large model pre-trained on OpenSubtitles, a massive corpus of spoken conversations and fine-tuned on relevant datasets for DA classification. The superior performance can be attributed to the pre-training on relevant corpus and hence performs slightly better than our approach which is only trained on DA classification dataset. \n\n\n### Doubts regarding baselines\n> In the customized online setting (that uses ASR transcripts from Whisper as textual inputs), the proposed model outperforms several relatively weak baselines.\n\n> The evaluation results are not particularly strong in comparison with unimodal models from before 2021 (see Q1 below).\n\n> Q1. Why not compare with SoTA model from Section 5.6 in the online setting experiments (Section 5.3)?\n\n**Response:** We respectfully differ with the reviewers notion that the baselines are weak. We have chosen 3 baselines and we have justified the choice of our baselines in the paper (lines 450 - 460). The first paper we have chosen as the baseline is the **only** related work that performs multimodal DA classification which creates an even ground for comparison with our multimodal DA classification approach. The second baseline is a Multimodal Transformer based approach which is one of the most cited (765) papers in the field of multimodal sentiment analysis/emotion recognition and cited as a relevant baseline in the recent papers. We have chosen this paper as the code is publicly available and hence it is easily reproducible. The third baseline is the one of the most recent works published on DA classification. For EMOTyDA, we only have one baseline paper and our paper outperforms it by a large margin. \n\nThe SoTA model, Chapius et al. 2021 [2] in the table 5.6 is a large model pre-trained on dialog specific corpus, OpenSubtitles and fine-tuned on relevant datasets for DA classification. The model is not publicly available and considering the scale of its training and development, implementing the work and comparing the results with our work is beyond our capacity. However, we have compared our work with 2 of the other relevant works (which were publicly available) presented in table 5.6, Ortega & Thang Vu (2018) [3] and He et al. (2021) [4], as relevant baselines in the online setup. They are referred to as **Lexico-acoustic Model** and **Speaker Turn Modeling** in the table 5.3 respectively and from the performance comparison, it is evident that our proposed model outperforms the two models by quite a good margin.\n\n### Role of modality fusion\n\n> It is not very clear whether modality fusion plays an important role in the proposed model (see Q2 below).\n> Q2: The utterance encoder designs in the unimodal encoder and early-stage fusion encoder are different. The former is conv + max pooling, and the latter is BiLSTM + attention. What is the reason for making such decision? Due to the existence of alternative unimodal utterance encoders in the early-stage fusion encoder, we can conclude that the early-stage fusion encoder contributes the most, but it is questionable how much modality fusing contributes.\n\n**Response:** \"Indeed, the two architectures used for unimodal feature extraction in the early-stage fusion encoder and the unimodal encoders are different. The early-stage fusion encoder has Bi-LSTM+attention based design which is a more straightforward choice based on the literature to model the utterances from token embeddings. The conv+maxpooling architecture is also employed in the unimodal encoders to model the utterances from token or frame embeddings. 1D convolution essentially works as a temporal convolution that helps to incorporate local context while computing utterance embedding from token embeddings. Our experiments show that using an alternative design of encoder with conv filters in the unimodal-encoder branches provide better results than using the similar LSTM based architecture. Essentially, this was a design choice based on empirical results. Our hypothesis is, the conv filters in the unimodal branches learn different representations of utterances than LSTMs employed in the early-stage fusion encoders and hence contribute to the better results. \n\nWe are unclear on what is meant by the sentence 'Due to the existence of alternative unimodal utterance encoders in the early-stage fusion encoder, we can conclude that the early-stage fusion encoder contributes the most, but it is questionable how much modality fusing contributes,' since earlier in the review, the reviewer seems to equate early-stage fusion with modality fusing: `Modality-fusing encoding (called early-stage fusion)`. It would be helpful to get a bit more clarification on this point in order to write a response but we are trying to explain the confusion to the best of our understanding of the comment. \n\nWe are under the impression that, the reviewer is convinced by our early-stage fusion encoder component but not entirely confident about how much the early fusion of the modalities is contributing to the performance gain and the overall performance gain could be attributed to the alternative designs of encoders rather than fusion. To validate the importance of fusing the modalities early, such as in the early-stage fusion encoder, we employ an alternative design, where we have the 4 unimodal utterance encoders (2 text based and 2 audio based) and we concatenate the representations from the 4 encoders to further use them in the classifier. \n\n|  Experiment   | Macro F1|    Accuracy  |\n| :---        |    :----:   |          ---: |\n| Original HF model    | **29.11**       |  **59.86** |\n| Alternative design without early fusion  | 27.29      | 56.05      |\n\nWe can observe that, the alternative design yields almost 2% less F1 score and almost 4% less accuracy compared to our HF model. This results reinforce the importance of modality fusion at the early stage of the model and context modelling. We not only have designed utterance encoders to extract good embeddings, we have carefully designed the model with fusion components that contribute to the overall results.\n\n### Clarification regarding frame embedding sequences\n> Q3: Frame embedding sequences are usually very long and hard to encode without compressing into short sequences. The early-stage fusion encoder uses utterance BiLSTM to encode audio embeddings at a frame level. Will not it be very memory consuming and difficult to train? What is the average length of frame embedding sequences in your experiment?\n\n**Response:** For extracting frame embeddings, we used the pre-trained model, Whisper and followed the same setup that is used in Whisper. The input audio utterance is padded or clipped to 30s and a 25ms window with 10 ms stride is employed to extract the mel-spectrograms. The choice of window size and stride for Whisper is consistent with that of other audio feature extraction methods across the literature. The final output from the whisper encoder is 1500 frames per utterance with 512 dimensional embedding per frame. We have successfully trained our model using these features and details of training and GPU requirement is provided in appendix B and C of our paper.  \n\n### Clarification regarding cross-attention\n> Q4: In the last paragraph of Section 3.3.1, the authors say cross attention is applied to textual and audio representations, but details of the cross attention are neither presented in Figure 1 nor by any equations. Can the authors provide more details?\n\n**Response:** We have provided the details about implementation of cross-attention in lines 281-292. However, as we did not find notable performance improvement using cross-attention as mentioned in the paper (*section 5.4, lines 546-550*), we chose to not include the cross-attention component in the figure. The details of cross-attention implementation is provided below and we also plan to incorporate these details in the final paper.\n\nFollowing the conventions of the paper, we use equation 3 from the paper to compute the token/frame representations from Bi-LSTM, $ h_{i}^{j, t} $ and $ h_{i}^{j, a} $ from text and audio modalities. Now while calculating cross attention, we use either the text modality as source and audio as target or vice versa. We want to compute two cross-attention based utterance embedding, one is $s_{i}^{t,a}$ where, the attention weights are calculated using the dot product of the two modality representations and based on the attention weights the token embeddings are aggregated to an utterance embedding and on the other hand, another is $s_{i}^{a,t}$ where, the attention weights are calculated using the dot product of the two modality representations and based on the attention weights the frame embeddings are aggregated to an utterance embedding. To achieve this, we first pass the token embeddings and frame embeddings through a simple MLP layer followed by a tanh activation to generate intermediate token/frame representations.\n\n$$ p_{i}^{j, t} = \\tanh(W_1 h_{i}^{j, t} + b_1)$$\n$$ p_{i}^{j, a} = \\tanh(W_2 h_{i}^{j, a} + b_2)$$\n\nNow we use, matrix multiplication to compute pairwise dot-products between intermediate token and frame representations. If $P_{i}^{t}$ and  $P_{i}^{a}$ are the two matrices of all the intermediate token and frame representations in the utterance, we compute the following matrix multiplication to compute the attention weights. Here the dimension of  $P_{i}^{t}$ is  (num_tokens $ \\times $ embedding_dim) and the dimension of $P_{i}^{a}$ is (num_frames $\\times$ embedding_dim).\n\n$$ S = P_{i}^{t} \\times (P_{i}^{a})^{T} $$\n\nThe dimension of this matrix is (num_tokens $\\times$ num_frames). Then we use softmax to normalize the attention weights. Now we apply another MLP transformation on the audio frame embedding matrix and finally multiply it with the attention weights to get the attended token embeddings,\n\n$$ P_{i}^{t'} = S \\times MLP(P_{i}^{a})$$\n\nThe dimension of this matrix is (num_tokens $\\times$ embedding_dim). Finally, we average all the attended-token embeddings to generate the utterance embedding, $s_{i}^{t,a}$. In a similar way we can compute the  $s_{i}^{a,t}$. The final utterance embedding will be the concatenation of the self and cross attended utterance embeddings.\n\n$$ s_{i} = concat(s_{i}^{t}, s_{i}^{a}, s_{i}^{t,a}, s_{i}^{a,t})$$\n\nThe rest of the model remains unchanged.\n\n- [1] Li et al. 2019.  [A dual-attention hierarchical recurrent neural network for dialogue act classification](https://aclanthology.org/K19-1036/). \n\n- [2] Chapius et al. 2021. [Hierarchical pre-training for sequence labelling in spoken dialog](https://aclanthology.org/2020.findings-emnlp.239/).\n\n- [3] Ortega and Thang Vu 2018. [Lexico-Acoustic Neural-Based Models for Dialog Act Classification](https://ieeexplore.ieee.org/document/8461371).\n\n- [4] He et al. 2021. [Speaker Turn Modeling for Dialogue Act Classification](https://aclanthology.org/2021.findings-emnlp.185/).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "RA1: An effective method of combining textual and audio features for dialog act recognition (or similar tasks).\n\nRA2: Confirmed performance improvement by incorporating the proposed unimodal and modalitiy-fusing components.", "reasons_to_reject": "RR1: The experiment design and model architecture design prevent readers from concluding what factor is the most important for improving task performance (see Q1, Q2 below). \n\nRR2: Important technical details are missing (see Q4 below).", "questions_to_authors": "Q1: Why not compare with SoTA model from Section 5.6 in the online setting experiments (Section 5.3)?\n\nQ2: The utterance encoder designs in the unimodal encoder and early-stage fusion encoder are different. The former is conv + max pooling, and the latter is BiLSTM + attention. What is the reason for making such decision? Due to the existence of alternative unimodal utterance encoders in the early-stage fusion encoder, we can conclude that **the early-stage fusion encoder** contributes the most, but it is questionable how much **modality fusing** contributes.\n\nQ3: Frame embedding sequences are usually very long and hard to encode without compressing into short sequences. The early-stage fusion encoder uses utterance BiLSTM to encode audio embeddings at a frame level. Will not it be very memory consuming and difficult to train? What is the average length of frame embedding sequences in your experiment?\n\nQ4: In the last paragraph of Section 3.3.1, the authors say cross attention is applied to textual and audio representations, but details of the cross attention are neither presented in Figure 1 nor by any equations. Can the authors provide more details?", "paper_topic_and_main_contributions": "This paper invesitages approach to dialog act recognition from textual and audio inputs.\n\nThe main contribution is the proposed modality-fusing architecture:\n- Feature extraction\n  - Extract textual token embeddings and audio frame embeddings from pretrained RoBERTa and Whisper models, respecitvely.\n- Unimodal encoding\n  - Compute unimodal utterance embeddings using convolution + max pooling.\n  - Compute unimodal dialog embedding from contextualized unimodal utternace embeddings using LSTM + attention.\n- Modality-fusing encoding (called early-stage fusion)\n  - Compute unimodal utterance embedding using BiLSTM + attention.\n  - Obtain multimodal utterance embedding by concatenating unimodal embeddings.\n  - Compute multimodal dialog embedding from contexutalized multimodal utterance embeddings using BiLSTM + attention.\n- Classification (called late-stage fusion)\n  - Combine the unimodal dialog embeddings and multimodal embedding to compute the final output.\n\nThe proposed model is evaluated on the MRDA and EMOTyDA datasets.\n- In the customized online setting (that uses ASR transcripts from Whisper as textual inputs), the proposed model outperforms several relatively weak baselines.\n- In the oracle settings (that uses oracle texts as textual inputs), the proposed model slightly lags behind a SoTA unimodal baseline.\n- The ablation study shows the existence of the modality-fusing component contributes the most to its performance.\n\nIn general,\n- the paper proposed an effective method of fusing multimodal features for dialog act recognition;\n- the evaluation results are not particularly strong in comparison with unimodal models from before 2021 (see Q1 below);\n- it is not very clear whether **modality fusion** plays an important role in the proposed model (see Q2 below).", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It is better to put Section 5.2 after the main results (i.e. at least after the current Section 5.4)."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the task of multimodal dialog act classification in an online setting – raw audio and ASR text. 

Reviewers appreciated the alignment of assumptions towards real-world applications. They also found the experimentation to be comprehensive and sound.

From a technical perspective, the excitement was average as the components proposed in the paper are fairly standard and known.

In addition, some of the choices and assumptions in the architecture were found to be confusing – which warrants improvements in the writing.
Decision: Accept-Findings