Dialogue Act-Aided Backchannel Prediction Using Multi-Task Learning
Authors: Wencke Liermann
Yo-Han Park
Yong-Seok Choi
Kong Joo Lee
Venue: EMNLP 2023 Findings (2023)
Keywords: backchannel prediction
multi-task learning
dialogue act
pre-trained audio encoder
voice activity projection

Abstract: Produced in the form of small injections such as "Yeah!" or "Uh-Huh" by listeners in a conversation, supportive verbal feedback (i.e., backchanneling) is essential for natural dialogue. Highlighting its tight relation to speaker intent and utterance type, we propose a multi-task learning approach that learns textual representations for the task of backchannel prediction in tandem with dialogue act classification. We demonstrate the effectiveness of our approach by improving the prediction of specific backchannels like "Yeah" or "Really?" by up to 2.0\% in F1. Additionally, whereas previous models relied on well-established methods to extract audio features, we further pre-train the audio encoder in a self-supervised fashion using voice activity projection. This leads to additional gains of 1.4\% in weighted F1.

Dialogue Act-Aided Backchannel Prediction Using Multi-Task Learning
Wencke Liermann', Yo-Han Park?, Yong-Seok Choi?, Kong Joo Lee
Department of Computer Engineering, Chungnam National University
Department of Radio and Information Communications Engincering. Chungnam National University
wliermann@o. cnu. kc.kr (happy1 11501, yseokchoi, kjoolee)@cnu.ac... ac.
Abstract
is the task of identifying time points in an on- going conversation which competent listener would respond with a certain backchannel type. Since the advent of ChatGPT, modern dialogue sys- tems exhibit answer quality levels on par with hu- mans in various professions (OpenAI, 2023). Once equipped with human-like attentive listening capa- bilities like backchanneling, their application could be extended social areas such as counseling and elderly care, not to replace humans but help where the demand exceeds the supply.
Produced in the form of small injections such "Yeah!" or "Uh-Huh" by listeners in conversation, supportive verbal feedback (i.e., backchanneling) is essential for natural dia- logue. Highlighting its tight relation to speaker intent and utterance type, we propose a multi- task learning approach that learns textual rep- resentations for the task of backchannel predic- tion in tandem with dialogue act classification. We demonstrate the effectiveness of our ap- proach by improving the prediction of specific backchannels like "Yeah" or "Really?" by up to 2.0% in F1. Additionally, whereas previ- ous models relied on well-established methods extract audio features, further pre-train the audio encoder in a self--upe fashion using voice activity projection. This leads to additional gains of 1.4% in weighted FI.
The phenomenon of backchanneling is tightly linked to dynamics of turn-taking, i.c., a funda- mental human ability to orderly manage speaking opportunities. Listeners do not intend to claim the turn when producing backchannel. On the con- trary, they use them to signal their continued will- ingness remain in the listener position (Iwasaki, 1997). This willingness certainly depends on what the speaker has said. If the previous statement is descriptive or a personal narrative that the listener is just learning about, they might be more likely to encourage the speaker to continue through general backchannels, e.g., -uh" In comparison, if the previous statement holds an opinion or some alleged general fact, the listener could agree using specific backchannel, e.g.. "That's right!", but they might as well have reason dispute and may want to present their own view. First evidence that such a relation between dialogue acts and backchannel- ing exists was provided by Morikawa et al. (2022) in a preliminary theoretical investigation.
1 Introduction
The development of a topic constitutes an integral part of any conversation. At any time, some par- ticipant, the speaker, is said to be in control of the topic flow. Simultaneously, other participants attend the talk with various degrees of verbal feed- back. Such feedback may take the form of simple vocalic sounds with little to no meaning, c.g., "Uh- Huh", or be extended to more stereotypical phrases, e.g., "Wow!", "Seriiusly?" (Iwasaki, 1997). In the absence of such so-called backchannels, speak- ers have been found to tell qualitatively worse sto- ries that they cut off abruptly and half-heartedly (Bavelas et al., 2000). Not only do backchannels shape the narrative, but they also help build feel- ing of understanding and trust between conversa- tion partners (Huang et al., 2011; Barange et al., 2022). Therefore, is unsurprising that contin- uous research efforts are put into equipping dia- logue agents with the all-so-importan ability of backchannel prediction. Backchannel prediction
Building on the theoretical observations above, we propose a multi-task model, BCDA, to simulta- neously learn the main task of Backchannel (BC) prediction and the sub-task of Dialogue Act (DA) classification. A model trained on the task of DA classification quickly learns to identify utterance types. We allow the BC prediction model to "eaves- drop on" this knowledge and use as additional guidance to extract more relevant textual features.
"Corresponding author.

---

We demonstrate the effectiveness of this approach and compare to the previously suggested sub-task of sentiment classification. Lastly, we verify that BC prediction models benefit from self-supervised Voice Activity Projection pre-training (Ekstedt and Skantze, 2022) even in the presence of large quan- tities of labeled task-specific backchannel data.
d
Prediction
clasification
Danal
X
2 Related Work
Regarding the task of backchannel prediction, has been a long-standing observation that audio fea- tures are superior to lexical features (Ruede et al.. 2017; Ortega et al., 2020), with some early models avoiding the inclusion of the latter entirely (Hara et al., 2018). In the process, Mel Frequency Cep- stral Coefficients (MFCCs) have established them- selves simple yet powerful measure of wave- form representation (Adiba et al., 2021; Jang et al., 2021) that only slowly superseded by hidden representations from pre-trained speech recogni- tion models (Ishii et al., 2021). When later models started to incorporate lexical features, they first needed to handle issues of automatic speech recog- nition quality (Ortega et al., 2020) and the ram- ifications of possible time delays in fast-paced real-time dialogues (Adiba et al., 2021). Gradu- ally, additional gains obtainable through lexical features convinced several authors to incorporate naive word embeddings (Ruede et al., 2017; Or- tega et al., 2020) or, more recently, pre-trained language models (Ishii et al., 2021: Jang et al., 2021). Reflecting the multi-faceted nature of dia- logue, backchannel prediction has been performed successfully in tandem with other tasks, such as sentiment classification (Jang et al., 2021) or turn- taking prediction (Hara et al., 2018; Ishii et al., 2021). While results from Hara et al. (2018 8) sug- gest that a model trained jointly on the tasks of turn-taking, filler, and backchannel prediction out- performs its single-task variant, Ishii et al. (2021) find that it is not necessarily turn-taking cues itself that are useful, but instead, the participants' turn- management willingness (e.g., willingness to listen, willingness to speak). So far, no multi-tasking ap- proach employed dialogue acts directly.
BERT
History Hiski] kx
History H[t:a]
Figure Dialogue Act-Aided Backchannel Prediction (BCDA) model architecture.
taken without alterations from Ekstedt and Skantze (2022). They employ frame-wise encoder to extract audio features 2t, E IR2266 at 100 feature vectors (frames) per second. The chosen encoder combines three input types. First, presentations from a frozen speech recognition model (i.e., CPC) (Riviere et al., 2020) are used to encode waveform information. Then, the participants' voice activity. i.e., the notion of whether someone is speaking (1) or not (0) is encoded in binary vector. Finally, past voice activity is aggregated into history vec- tor. For two speakers, this aggregation takes the activity ratio of speaker A over speaker for se- lected past time intervals. All three feature vectors are mapped to a shared space and added before being pushed through a decoder-only transformer.
As the original authors proposed, we train this architecture using the self-supervised task of Voice Activity Projection (VAP) (Ekstedt and Skantze, 2022). Given an 8s input chunk, the model has to predict a 2s future window for each frame, which is modeled as four bins per speaker of sizes 200, 400, 600, and 800ms. If a speaker is speaking for more than half of the bin, it is marked active (1), else inactive (0). The resulting binary configuration is translated to its decimal counterpart and encoded as a one-hot gold prediction vector. A model pre- trained using VAP has been successfully applied predict turn turn-taking events, including BC, in zero- shot fashion (Ekstedt and Skantze, 2022). But, it
3 Model
3.1 Architecture
We apply a multi-moda approach that exploits au- dio features in combination with lexical features (see Figure 1). The audio-processiie component is

---

has not been verified whether VAP can also be useful initialization step prior to task-specific finetuning. Note that we use pooled speaker listener audio during pre-training while finetuning and evaluation are performed using speaker-only audio. In this way. we want to ensure that the model is unable to cheat (i.e.. pick up on backchannel in the audio) even if BC timestamps are slightly off.
like
the
beach, what
Yeah.
Uh-huh
can why.
Figure 2: Segmentation of utterances into history seg- ments (h (h6) given the backchannel "yeah". hl also called the current segment.
For the text- processing component, we employ an approach inspired by Malhotra et al. (2022). Given a history of past speech segments, encoded using BERT (Devlin et al., 2019), the segment that directly precedes the current timestamp projected as the query vector, while the full history serves as keys and values. To reflect the relative importance of segment in relation to its distance from the current timestamp, the dot product between query and key is scaled down by monotonically decreas- ing function of time (i.e., Time- Aware Attention) before the resulting scalars are used to produce a history enriched representation of the current seg- ment. Intuitively, while not all prior speech seg- ments may be directly related to the current one, some may prove valuable to supplement omitted or abbreviated content, i.e., pronoun resolution.
like a BC and were speaker's only voice activity in fixed interval. To obtain an equal number of negative examples, they defined the timestamp 2s before each BC to be instance of NO-BC. Later, Ortega et al. (2020) divided their list into generic forms, e.g., "Uh-Huu", and specific forms, e.g.. "Wow!" which they assigned the labels Continuer and Assessment, respectively. The result are 122k data instances: 50% NO-BC, 22.5% Continuer, and 27. 5% Assessment.
While DA annotations for half the conversations in the corpus are theoretically available (Jurafsky et al., 1997). this annotation was performed on an early version of the transcript before was reseg- mented, corrected, and enriched with word-level start- and end aod-timestamps (Deshmukh et al., 1998). We align the two transcript versions using naive Levenshtein distance- based approach. Despite an imperfect alignment and the fact that only 48.2% of all conversations have DA annotations, be- gin with, we manage to obtain labels for 41.9% of all utterances preceding a BC (or NO-BC), Out a total of 42 DA tags, the most frequent ones are suaemeenn-oonopppooooooonniiiuut (68.5%) and staeemenn-opinnoo/ddipppubblllle (24.2%).
Finally, the representations of both modalities are concatenated and fed through a fully connected softmax layer, resulting in a probability distribution across BC classes. Weight updates are obtained using a cross-entropy (CE) loss.
3.2 Multi-task Learning (MTL) Setting
the MTI. setting, the lexical representation is further passed to another fully connected softmax layer to predict the DA the current segment (if an annotation is available). A probability for each DA is returned and the CE loss is calculated. The total loss for training the multi-task model is linear combination of the previously mentioned main task loss and this new sub-task loss: (1- main loss + X* sub loss. Through an exhaustive search, À was set to 0.3
4.2 Preprocessing
Each BC instance is linked to timestamp, and the 1.5s of audio preceding it will be used as input the model. Additionally, we need to represent tex- tual transcriptions as history of speech segments. To handle utterance overlap and interruptions, we propose the following segmentation. Given the timestamp of an instance, we collect all the pre- ceding words, sorting them by their start time. We then aggregate neighboring words into one segment if they originally belonged to the same utterance (this implies them being pronounced by the same speaker). An example given in Figure 2. The utterance can see why" is split into two segments to accommodate the interjection "what", which it-
4 Experiment
4.1 Backchannel Data
In our experiments, we employ the Switchboard Corpus, corpus of about 260 hours of dyadic small-talk telephone conversations (Godfrey and Holliman, 1993). Based on an aggregated list of surface forms, Ruede et al. (2017) automatically assigned utterances the label BC if they looked

---

Trainbl. Params. Audio
Model
Text
NO-BC
Continuer
Assessment
Weighted F1
Baselines ( reimplementation)
Oriega (Ortega et al., 2020)
2.3K
24.0K
72.4
41.6
47.0
58.4
BPM ST (Jang et al., 2021)
146.0K
109.0M
79.6
41.5
50.3
63.0
BPM_MT* (Jang et al., 2021)
146.0K
109.05M
79.7 (+0.1)
41.8 (+0.3)
50.1 (-0.2)
63.0 (+0.0)
Ours
SC
2.6M
111.0M
82,5
46.8
49.4
65.4
BCDA_MT
2.6M
111.03M
82.8 (+0.3)
46.7 (-0.1)
51.4 (+2.0)
66.0 (+0.6)
Table Backchannel prediction results. With the exception of Ortega al. (2020) shown values are averages across 10 random seeds. Numbers in brackets indicate differences between single-task (ST) and multi-task (MT) models.
A
PT
MT
NO-BC
Continuer
Assessment
Weighted F1
79.77 (#00.05)
43.62 (+1 1.6))
42.40 (+ 1.55)
61,36 (+00255)
80.80 (110.6)
44.03 (10))
45.11 (1.)
62.71 (1:0.15)
72.79 (+0.47)
40.52 (+0.822
45.46 (+0.66)
58.00 (+0.37)
+
73.15 0+20))
40.60 (++0,97)
46.52 (+00.70)
58.49 (+00.16)*
82.52 (10.16)
46.77 (1.0.99)
49.38 (10.80)
65.35 (10.16)
82.81 (=10.6))*
46.70 (#0.66)
51.35 (=:055
66.02 (+-0.1)**
Table 2: Ablation study results as averaged across 10 random seeds. Numbers in brackets show one standard deviation, Asterisks mark values that are significantly better than other values in the same block (paired t-test, p < 0.01). (A) Audio encoder. (PT) VAP pre-training, (T) Text encoder. (MT) Multi-task learning using DA prediction.
self becomes one segment. The average number of words in the current segment is 13.4, aligning with related work that used 5 to 20 words as input.
served no significant gains when the sub-task of sentiment classification was introduced. Notably, our model outperforms all baselines by at least a margin of 3% in weighted F1. This performance increase comes at the cost of a comparably small increase in model size of about 4.5M parameters compared to BPM. In comparison, BPM replaced the traditional word2vec embedding employed by Ortega with the powerful BERT model. Despite an additional 109M parameters, they outperformed Ortega by no more than 4.6% in weighted FI.
4.3 Training Setting
We upply train-test-valid split suggested by Ruede et al. (2017). The model is trained using an Adam optimizer with learning rate (Ir) of 2c-5 for BERT and the audio component and an Ir of le-4 for ev- erything else. The former uses warmup scheduler that gradually increases the Ir over 10% of train- ing time before decreasing it again zero. The latter uses a linear decreasing Ir scheduler. During loss computation, we assign each class a rescaling weight (NO-BC: 0.8, Continuer: 1.3, Assessment: 1.1) to counteract class imbalance. The chosen his- tory size is 3, batch size 16, and the number of training epochs 10. The best model is saved based the weighted F1 of the validation set. Each model configuration was trained on ten different randomly chosen seed values and test set perfor- mance values are averaged across those.
To provide insight into the origins of our im- provements, we perform extensive ablation study (see Table 2). As illustrated, an audio- only model outperforms the much heavier text- only model by more than 3.3% in weighted FI. However, both components have complementary strengths. While the audio component excels at predicting CONTINUER, the text component shows better results in predicting ASSESSMENT, Their combination outperforms both single modal- ity models by least 4.0% in weighted F1. This shows the value of including text information.
5 Results
Additionally, we performed significance tests for each pair of rows in Table 2, evaluating differences between naive audio-only model vs. pre-trained audio-only model, text-only single task model vs. a text-only multi-task model, and full single task model vs. a full multi-task model. First of all, VAP pre-training leads to a significant perfor- mance increase across two out of three categories,
We compare our results to the models of Ortega et al. (2020) and Jang et al. (2021). Jang et al. (2021) perform backchannel prediction in tandem with sentiment classification. Initially proposed for private Korean counseling data, we reimple- ment their approach using labels from a pre-trained English sentiment classifier. Surprisingly, we ob-

---

with an overall increase in weighted F1 of around 1.4%. This proves the effectiveness of this form of pre-training, even in the presence of large volumes labeled data. Finally, it can be observed that the proposed sub-task of dialogue act prediction benefits mainly and to large extent the performance on ASSESSMENT (+2.0%), with only minor but significant gains on NO-BC (+0.3%) and no sig- nificant effect on CONTINUER. This might be explained by the fact that ASSESSMENT holds specific backchannel forms, e-g., "Wow.", which are very much content-dependent. Therefore, a bet- ter encoding of the context through content- based multi-task learning can help in correctly predict- ing them. In contrast, CONTINUER are generic backchannel, e.g.. "Uh", which can often be in- serted urbitrarily and independent of content.
useful hints for the prediction of certain types of backchannels.
Conclusion
We proposed a multi-task learning scheme that effectively incorporates dialogue act annotations to enhance backchannel prediction without estab- lishing any requirement for such annotations to be available at inference time. We demonstrated its superiority over the previously suggested sub- task of sentiment classification in casual settings void of strong emotions. Especially, the prediction of specific backchannels could be improved by as much as 2%. Moreover, we have shown that self- supervised pre-training using voice activity pro- jection benefits the performance of backchannel prediction models. Overall, outperform previ- ous baselines by 3%. All experiment code is made publicly available'.
6 Discussion
When applied to counseling data, the sub-task of sentiment classification has been previously shown yield considerable improvements of as much as 3% in weighted Fl (Jang al., 2021). The dis- crepancy between those findings and ours could be explained by a difference in the nature of the data. While counseling data is emotion-loaded and the counselor committed to validating those, casual conversations between strangers are quite the opposite. In our data, more than 72.6% of all backchannels have been produced in response completely neutral utterance. The proposed sub-task of dialogue act classification constitutes a promising extension and alternative, especially in such casual conversations where the sub-task of sentiment classification fails. Our dialogue act- aided multi-task model achieves accuracy of 75.12% at the task of dialogue act classification. It is especially good at the differentiation between statement-opinion (sv) and statement-non-oping (sd). Jurafsky et al. (1997) describe sv as view- points that the listener could conceivably have their own (possibly differing) opinion about, while sd are non-disputable descriptives or narratives. They further state that this distinction was introduced to reflect "the different kinds of responses [they] opinions (which often countered or dis- agreed with via further opinions) and statements (which more often get [...] backchannels)" (Juraf- sky et al., 1997, Section 1)). While they express doubts about whether or not this distinction actu- ally fruitful, we have shown that it provides indeed
Limitations
In comparison to simple intuitive tasks like senti- ment analysis that can be performed by any layper- son, dialogue act annotation is commonly per- formed by linguists, researchers, or other experts in discourse analysis. While it is possible to train layperson to perform this task with proper guide- lines and instructions, the quality of the resulting annotation likely varies. Experts, on the other hand, are often expensive to hire. As for any other task, automatic classifiers exist, but they are still insufficiently accurate, c.g., 83.2% accuracy on the Switchboard Dialog Act Corpus (He et al., 2021). Moreover, researchers have proposed dialogue act label sets of varying granularity and focus. Each such set might be less more beneficial when used in tandem with backchannel prediction. The label set applied this study might not be representative of every possible label set, just as it might not be among the most effective ones.
Acknowledgements
This work was supported by Institute of Informa- tion & communications Technology Planning & Evaluation (UTTP) grant funded by the Korea gov- (MnMen((MSTT) N0.RS-20022-0155555 Artificial Intelligence Convergence Innovation Human Re- sources Development (Chungnam National Univer- sity))

---

References
prediction of turn-changing and turn- mun-aamgement willingness improve backchannel modeling? In Pro- ceedings of the s1 ACM International Conference on Intelligent Virtual Agents, pages 131-138, Associ- ation for Computing Machinery.
Amalia Istiqlali Adiba, Takeshi Homma, and Toshinori Miyoshi. 2021. Towards immediate backchannel gen- eration using ireettion-bsedd early prediction model. In ICASSP 2021- 2021 IEEE International Confer- ence Acoustics, Speech and Signal Processing (ICASSP), pages 7408-7412, IEEE
Shoichi Iwasaki. 1997. The northridge earthquake con- versations: The floor structure and the 'loop' se- quence in japanese conversation. Journal of Prag- matics, 28(6):661-693.
Mukesh Barange, Sandratra Rasendrasoa, Maël Bouab- delli, Julien Saunier, and Alexandre Pauchet. 2022. Impact of adaptive multimodal empathic behavior on the user interaction. In Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents, pages 1-8, Association for Computing Ma- chinery.
Jin Yea Jang, San Kim, Minyoung Jung, Saim Shin, and Gahgene Gweon. 2021. BPM_MT Enhanced backchannel prediction model using multi-task learn ing. In Proceedings of the 2021 Conference on Em- pirical Methods in Natural Language Processing, pages 3447-3452, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Janet B Bavelas, Linda Coates, and Trudy Johnson. 2000. Listeners as co- narrators. Journal of Personal- ity and Social Psychology, 79(6)941-995
Dan Jurafsky. Elizabeth Shriberg, and Debra Bi- asca, 1997. Switchboard SWBD-DAMSL shallow- discourse-function annotation coders manual. Tech- nical Report Draft 13, University of Colorado, Insti- tute Cognitive Science.
Necraj Deshmukh, Aravind Ganapathiraju, Andi Glee- son, Jonathan Hamaker, and Joseph Picone. 1998. Resegmentation of SWITCHBOARD. In 5th Inter- national Conference on Spoken Language Processing (ICSLP 1998). ISCA.
Ganeshan Malhotra, Abdul Waheed, Aseem Srivastava, Md Shad Akhtar, and Tanmoy Chakraborty, 2022. Speaker and time-aware joint contextual learning for dialogue-act classification in counselling conversa- tions. In Proceedings of the 15th ACM International Conference on Web Search and Data Mining, pages 735-745. Association for Computing Machinery.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Akira Morikawa, Ryo Ishii, Hajime Noto, Atsushi Fukayama, and Takao Nakamura. 2022. Determining most suitable listener backchannel type for speaker's s utterance. In Proceedings of the 22nd ACM Inter- national Conference on Intelligent Virtual Agents, pages 3. Association for Computing Machinery.
Erik Ekstedt and Gabriel Skantze, 2022. Voice ac- tivity projection; Self-supervised learning of turn- taking events. In Interspeech 2022, pages 5190-5194, ISCA.
OpenAl. 2023. Gpt-4 technical report. Computing Research Repository, arXiv:2303.08771 087774.. Version 3.
John Godfrey and Edward Holliman. 1993. Switchboard-1 Release 2 LDC97S62. Linguistic Data Consortium.
Daniel Ortega, Chia-Ye Yu Li, and Ngoc Thang Vu. 2020. OH, JEEZ! or UH-HUH? a listener-aware backchan- nel predictor on ASR transcriptions. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8064- 8068. IEEE
Kohei Hara, Koji Inoue, Katsuya Takanashi, and Tatsuya Kawahara. 2018. Prediction of turn-taking using mul- titask learning with prediction of backehannels and fillers. In Interspeech 2018, pages 991-995. ISCA.
Zihao He, Leili Tavabi, Kristina Lerman, and Moham- mad Soleymani. 2021. Speaker turn modeling for dialogue act classification. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2021, pages 2150-2157, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Morgane Riviere, Armand Joulin, Mazaré, and Emmanuel Dupoux. 2020. Unsuper- vised pretraining transfers well across languages. In ICASSP 2020 2020 IEEF International Confer- ence on Acoustics. Speech and Signal Processing (ICASSP), pages 7414-7418. IEEE.
Lixing Huang. Louis Phiis-Philippe Morency, and Jonathan Gratch. 2011. Virtual rapport 2.0. In Intelligent Virtual Agents, pages 68-79. Springer Berlin Heidel- berg.
Robin Ruede, Markus Müller, Sebastian Stüker, and Alex Waibel. 2017. Enhancing Backchannel Predic- tion Using Word Embeddings. In Interspeech 2017, pages 879-883. ISCA.
Ryo Ishii, Xutong Ren, Michal Muszynski, and Louis- Philippe Morency. 2021. Multimodal and multitask approach to listener's 'ss backchannel prediction: Can

---

A Baselines
This section provides a more in-depth explanation of the chosen baselines. We selected two baselines from previous literature for comparison.
Ortega et al. (2020) employed CNNs in tandem with word2vec embeddings, MFCC features and listener embedding. They the same data as is applied in our study. While they do not directly report Fi values, they provide a confusion matrix based on which those could be calculated.
Jang et al. (2021) employed pre-trained BERT language model together with an LSTM on top of MFCC features. As no public implementation is available, reimplemented their model based on the information available in the paper. The only parameter missing from their explanation was the hidden size of the LSTM module. After some ex- perimentation, we set this parameter 128. their multi-task approach, the original authors used Ko- rean sentiment dictionary that assigned each word in a collection one out of five sentiments (very neg- ative, negative, neutral, positive, very positive). To the best of our knowledge, fine-grained dictionary such this one is not available in English. There- fore, we experimented with some alternative ap- proaches to include sentiment classification. Given a pre-trained sentiment classifier for English', we obtain a probability distribution across the senti- ments: negative, neutral, and positive. We can now either predict this distribution directly, using KI. divergence for loss propagation encode the cat- egory with the highest probability as a hard label, using cross-entropy for loss propagation. The latter approach yielded better performance values, so we used our experiments. Input to the model first segment joined with any following segment until the speaker changes or the history size (of 3) is exceeded.
Sttp://tngnngaacccccaddi/tttttte sentiment-latest


Reviews:
[{"review_id": "ELfyu9i87V", "reviewer": "Reviewer_kE2m", "summary": "", "questions": "See reasons to reject. ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, thank you for taking the time to review our submission.\n\n### Model Complexity\nAs pointed out, the given short paper did not include a model complexity analysis. To make up for that, we accumulated relevant information on the type and number of all parameters of each compared model in the table below. (as obtained by pytorch_lightning)\n\n| | Audio | Text | Other | Multitask | Total |\n|:---|---|---|---|---| ---: |\n| Ortega (2020) | CNN + Lin. L.: 2.3K | CNN + Lin. L.: 24.0K | Lin. L.: 2.7K | / | 29.0K |\n| BPM (2021)  | LSTM: 146.0K | BERT: 109.0M | Lin. L.: 114.0K | Lin. L.: 49.2K | 109.3M |\n| BCDA (Ours)  | Transformer: 2.6M | BERT + Attention: 111.0M | Lin. L.: 3.1K | Lin. L.: 31.5K | 113.6M |\n\n**Comparison to Related Work Baselines**\nWhile our model does indeed have the largest number of parameters, it adds only an additional 4.3M compared to the BPM (2021) baseline. This is an approx. 4% increase in model size through which we were able to outperform BPM (2021) by a margin of 3% in F1. In comparison, BPM (2021) replaced the traditional word2vec text embedding employed by Ortega (2020) with the powerful BERT model. This caused the model size to increase by more than 100M parameters, and still they were able to only obtain an increase in F1 of 4.7%. As can be seen, our obtained increase in performance came at the cost of a comparably small increase in model size.\n\n**Comparison Single Task and Multi Task**\nBPM (2021) as well as our suggested model BCDA use a sequence of linear layers on top of the text encoding to implement multitask training. Those linear layers are only active during training and can be entirely removed during inference. Therefore, the multitask approach does not affect inference performance or inference time in any way. However, it is true that during training an additional loss has to be computed and propagated. But the propagation through shallow linear layers is fast and did not noticeable affect training time for either model.\n\n\n### Significance Test\nGiven the quirks of our setup not many significance tests can be safely applied. We do not perform cross validation, and instead use a fixed test set to make our results directly comparable to previous research (Ortega, 2020). While some other authors with similar settings bootstrap the test set to obtain several performance values which could then be used in a t-test or a wilcoxon rank sum test, we deem this as bad practice. This is because, it creates dependent samples and therefore violates the indepence assumption of above tests.\n\nTo the best of our knowledge, the only significance test applicable in our situation is the McNemar-Test. However, this test cannot measure whether a difference in F1 is significant, but only whether the error counts of two models are significantly different. Given a matrix of error counts:\n\n| | BCDA\\_MT Correct | BCDA\\_MT Wrong |\n| --- | --- | --- |\n| BCDA_ST Correct | 7077 | 471 |\n| BCDA_ST Wrong | 546 | 3276 |\n  \nWe perform the McNemar-Test in R and obtain:   \nMcNemar’s chi-squared = 5.531, df = 1, p-value = **0.01868 (< 0.05 => significant)**\n\nTo conclude, our Multitask Version is significantly more often correct when the Single Task Version is wrong, than the other way around.\n\nWe hope we were able to answer all your questions. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a novel approach to backchannel prediction by incorporating dialogue act annotations through multi-task learning. \n2. Extensive experiments and ablation studies are conducted. \n3. Well-structured and clearly explains the methodology and experimental setup. ", "reasons_to_reject": "1. Lack of model complexity analysis: Considering the marginal performance improvement compared to the baseline and the additional parameters and computational burden introduced by multi-task learning, it is essential for the authors to conduct a model complexity analysis. This analysis will help clarify the potential applications and prospects of the proposed method.\n\n2. The improvement of MT over ST is very limited. The authors should perform significance tests to ensure the reliability of the experimental results.", "questions_to_authors": "See reasons to reject. ", "paper_topic_and_main_contributions": "The authors propose to adopt a multi-task learning paradigm for the backchannel prediction task, with its sub-task being dialogue act classification. This design is motivated by the observation that backchanneling is closely connected to dialogue act, which in turn plays a crucial role in turn-taking prediction. Through their study, the authors effectively demonstrate the efficacy of their approach and provide a comparison to the previously suggested sub-task of sentiment classification.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2xLDO8amSa", "reviewer": "Reviewer_fpPn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind and thorough review.\nWe would hereby like to address your qualms regarding our submission.\n\n**Regarding our choice of data:** Backchannel Prediction is most often researched in the context of counselling conversations. Due to valid privacy concerns any data collected in this context is rarely made public. To the best of our knowledge, the Switchboard Corpus is the largest public corpus of its kind. While we do have access to a smaller private dataset, we agreed that our experiments would be most useful to fellow researchers if performed on a public data set where any results could be readily reproduced.\n\n**Performance:** The overall improvement in weighted F1 might seem small, but it is worth pointing out that the performance in the category *Assessment* increased by a whole 1%. Indeed the trend we observed during hyperparameter was that this category benefited most from the Multitasking with improvements of up to +2.9% (with an overall increase in weighted F1 of 0.8%) in certain configurations. Reason might be that Backchannels of the Category *Assessment* cannot be produced in arbitrary places and are more dependent on the context including the previous dialogue acts.\nNonetheless, we understand your concerns. To verify whether obtained results are likely to have occurred by chance, we perform a significance test.\n\nTo the best of our knowledge, the only significance test applicable in our situation is the McNemar-Test. However, this test cannot measure whether a difference in F1 is significant, but only whether the error counts of two models are significantly different. Given a matrix of error counts:\n\n| | BCDA\\_MT Correct | BCDA\\_MT Wrong |\n| --- | --- | --- |\n| BCDA_ST Correct | 7077 | 471 |\n| BCDA_ST Wrong | 546 | 3276 |\n(Test set size: 11370)\n\nWe perform the McNemar-Test in R and obtain:  \nMcNemar’s chi-squared = 5.531, df = 1, p-value = **0.01868 (< 0.05 => significant)**\n\nThis leads us to believe, that our Multitask Version is significantly more often correct and the observed improvement is very likely not a byprodukt of chance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed approach is creative; it goes against the classic thinking that encoding audio is strictly superior to encoding text for backchannel prediction. Additionally, it is linguistically motivated and is the first paper to tie turn-taking and dialogue act recognition with backchannel prediction using multi-task learning.\n\n2. The experimental results are competitive, improving upon existing state-of-the-art performance. There are also thorough ablations, exploring both unimodal and multimodal networks. ", "reasons_to_reject": "1. A large focus of the paper is on multi-task learning, and the relation between dialogue act classification and backchannel prediction. However, the actual improvements from single-task to multi-task learning are somewhat slim: 65.5 to 66.1. It is not clear whether these improvements are by chance (e.g., would improvements hold across multiple seeds, multiple trials, etc.)\n\n2. The task is somewhat narrow, and the paper only focuses on one corpus, Switchboard, from 1993. However, this narrow focus may still be sufficient for a short paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a multi-modal approach to backchannel prediction, trained using multi-task learning. They posit that the task of backchannel prediction is closely related to dialogue act classification (and broadly, turn-taking) and propose a multi-task loss to learn both tasks. They improve upon the existing state-of-the-art performance on backchannel prediction and demonstrate that dialogue actions/intents are relevant to backchannel prediction.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mX0NwNCjol", "reviewer": "Reviewer_ww4h", "summary": "", "questions": "1. From Table 2, adding the proposed multi-task learning increases merely by 0.6% accuracy (65.5% —> 66.1%); a significant test is needed to prove the effectiveness, such as a single-tail t-test. Most of the improvement comes from using the existing pre-trained audio encoder.\n2. In the abstract: \"we further pre-train the audio encoder in a self-supervised,\" the authors actually use the audio pre-trained model to initialize the audio encoder. It seems that the paper does not involve a pre-training stage.\n3. Please provide the detailed distribution of the 42 DA classification. There is only 40+1 DA (plus one empty) in the training code, and the statement-non-opinion (68.5%) and statement-opinion (24.2%) is not match the distribution in SwDA [1].\n4. Turn-taking is just a subset of dialog act, where in this work, the author frames it as a three-way classification (no-BC, BC-Continuer, BC-Assessment). The author may compare the traditional DA classification method with BCDA_MT. For example, [2] could achieve 83.2% ACC in the 42 DA classification.\n\n[1] https://compprag.christopherpotts.net/swda.html#tags\n[2] He, Zihao, et al. \"Speaker Turn Modeling for Dialogue Act Classification.\" Findings of the Association for Computational Linguistics: EMNLP 2021. 2021.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we want to thank you for your in-depth feedback and all helpful suggestions.  \nHereby, we want to provide the requested information.\n\n**Question 1: Is the improvement significant?**  \nIn order to answer this question we perform two significance tests. First, we perform a paired t-test. As we do not perform cross-validation and have only one fixed test set at our disposal, we apply bootstraping to the test set to obtain a collection of performance values. For this purpose we choose 1000 items from the test set with replacement, we repeat this process 10 times to obtain a total of 10 test sets. We then calculate the weighted F1 for each set and use the obtained paired values (one value for the single task model, one value for the multi task model) to perform a paired t-test. We repeat the whole process 1000 times and average the p-values across those runs. The obtained average p-value is 0.000967 which is smaller than 0.05 and therefore indicates that our results are significant.\n\n**BUT** we do not believe in the truthfulness of this result. One requirement of the t-test and most other significance tests is that the performance values must be independent of each other. As we draw all samples from the same set, samples will overlap and are therefore definitely dependent. A violation of the independence assumption causes a large Type I error, that is, it becomes more likely to claim something as significant which might in fact just be a random difference.\n\nTo the best of our knowledge, the only significance test really applicable in our situation is the McNemar-Test. However, this test cannot measure whether a difference in F1 is significant, but only whether the error counts of two models are significantly different. Given a matrix of error counts:\n\n| | BCDA\\_MT Correct | BCDA\\_MT Wrong |\n| --- | --- | --- |\n| BCDA_ST Correct | 7077 | 471 |\n| BCDA_ST Wrong | 546 | 3276 |\n(Test set size: 11,370)\n\nWe perform the McNemar-Test in R and obtain:  \nMcNemar’s chi-squared = 5.531, df = 1, p-value = **0.01868 (< 0.05 => significant)**\n\nThis shows that the multitask model is significantly more often correct than its single task variant.\n\n**Question 2: Does our paper involve a pretraining stage?**  \nIndeed it does. The audio encoder first encodes the audio waves into feature vectors. For this purpose we apply an already pretrained speech recognition model. This model remains always frozen and will not be further finetuned. On top of this feature encoder sits a decoder-only transformer which is randomly initialised. We pre-train this transformer using the task of Voice Activity Projection by Ekstedt (2022). It is true that Ekstedt proposed their pretraining for the Switchboard Corpus but we did not take their pretrained model, but instead performed the pre-training ourself. Why? Because Ekstedt uses cross validation, while we have a fixed split that is different from their division into train-valid-test.  \n*So what is our contribution?* Ekstedt evaluated their pre-training only with regard to the model's zero-shot performance on the task of backchannel prediction. They never evaluated whether the proposed pre-training objective would bring any additional gains in the presence of labeled task-specific backchannel data that a model can be trained on directly.\n\n**Question 3: Why does the stated distribution of dialogue acts not match the official documentation of the SWDA?**  \nWe do not let our model predict the dialogue act for any arbitrary utterance. Instead, each item in our backchannel data has a history of past utterances, the most recent one called *current utterance*. Our model predicts only the dialogue act of this current utterance. Therefore, we obtain a different distribution of dialogue acts. All backchannel data items and their history, including the dialogue act of the current utterance can be found in the submitted code at: `data/swb/utterance_is_backchannel_with_da_with_context.csv`\n\nHere is our detailed distribution (across all splits):  \nsd: 34998 (68.4879%)  \nsv: 12357 (24.1815%)  \nqy: 721 (1.4109%)  \n%: 437 (0.8552%)  \n^q: 291 (0.5695%)  \nqy^d: 258 (0.5049%)  \nad: 209 (0.409%)  \nqh: 201 (0.3933%)  \nbf: 197 (0.3855%)  \naa: 183 (0.3581%)  \nba: 156 (0.3053%)  \nqw: 155 (0.3033%)  \nna: 121 (0.2368%)  \nh: 94 (0.1839%)  \nfc: 91 (0.1781%)  \nnd: 83 (0.1624%)  \nng: 80 (0.1566%)  \nqo: 58 (0.1135%)  \nb: 45 (0.0881%)  \n^h: 45 (0.0881%)  \nno: 43 (0.0841%)  \n^2: 41 (0.0802%)  \ncc: 29 (0.0568%)  \nbc: 28 (0.0548%)  \nbh: 25 (0.0489%)  \nqrr: 23 (0.045%)  \nbk: 19 (0.0372%)  \nt1: 17 (0.0333%)  \nar: 16 (0.0313%)  \nam: 12 (0.0235%)  \nqw^d: 11 (0.0215%)  \nny: 10 (0.0196%)  \nbr: 10 (0.0196%)  \nb^m: 8 (0.0157%)  \nfa: 8 (0.0157%)  \nnn: 6 (0.0117%)  \nt3: 5 (0.0098%)  \nfp: 4 (0.0078%)  \nbd: 4 (0.0078%)  \n^g: 1 (0.002%)  \nft: 1 (0.002%)  \n  \n**Question 4: Is our task just a subset of dialogue act classification?**  \nNo. Backchannel identification is a subset of dialogue act classification, but not our task of backchannel prediction. The difference is that former has the backchannel utterance as well as its surrounding (previous and following) context at its disposal to perform a prediction, while we only have the previous context available.\n\n**Question 5: How good is our model BCDA_MT at the task of dialogue act classification?**  \nOur model achieves an accuracy of 75.12% at this task. It is especially good at the differentiation between statement-non-opinion and statement-opinion, as well as the differentiation between several question types.\nWhile this falls short of the 83.2% achieved by Zihao et al. (2021), readers should not forget that we weighted the dialogue act classification loss by a lambda of 0.3, giving it less importance.\n\nWe hope all your questions were answered. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Proposed a multi-tasking method that jointly models 3 way classification of and 42-way classification of DA in a speech-text multimodal approach\n2. Achieve improvement compared with previous methods.", "reasons_to_reject": "1. Innovations are some how limited, a combination of existing work. Experimental settings/datasets from Ortega, Multi-task from BPM_MT, and pre-trained audio encoder from Ekstedt (VAT). \n2. The improvement from the proposed idea is incremental (<1%), and a significant test is needed to prove its effectiveness.\n3. Lack of in-depth analysis and insights.", "questions_to_authors": "1. From Table 2, adding the proposed multi-task learning increases merely by 0.6% accuracy (65.5% —> 66.1%); a significant test is needed to prove the effectiveness, such as a single-tail t-test. Most of the improvement comes from using the existing pre-trained audio encoder.\n2. In the abstract: \"we further pre-train the audio encoder in a self-supervised,\" the authors actually use the audio pre-trained model to initialize the audio encoder. It seems that the paper does not involve a pre-training stage.\n3. Please provide the detailed distribution of the 42 DA classification. There is only 40+1 DA (plus one empty) in the training code, and the statement-non-opinion (68.5%) and statement-opinion (24.2%) is not match the distribution in SwDA [1].\n4. Turn-taking is just a subset of dialog act, where in this work, the author frames it as a three-way classification (no-BC, BC-Continuer, BC-Assessment). The author may compare the traditional DA classification method with BCDA_MT. For example, [2] could achieve 83.2% ACC in the 42 DA classification.\n\n[1] https://compprag.christopherpotts.net/swda.html#tags\n[2] He, Zihao, et al. \"Speaker Turn Modeling for Dialogue Act Classification.\" Findings of the Association for Computational Linguistics: EMNLP 2021. 2021.", "paper_topic_and_main_contributions": "This paper introduces a multi-task learning approach for backchannel prediction in dialogues, enhancing textual representations and utilizing self-supervised pre-training for audio encoders. Outperforming previous models by 3% in WF1, improving computational understanding of back-channeling for applications like conversational AI and speech recognition.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The manuscript is more of a technical report than a research paper. The author should focus on the original part rather than the known technique."}, "extra_text": {"Missing_References": "He, Zihao, et al. \"Speaker Turn Modeling for Dialogue Act Classification.\" Findings of the Association for Computational Linguistics: EMNLP 2021. 2021.\nSkantze, Gabriel. \"Turn-taking in conversational systems and human-robot interaction: a review.\" Computer Speech & Language 67 (2021): 101178."}}]

Author rebuttal:

Meta-review: This paper proposes an approach to predict backchannels via multi-task training and viewing the problem similarly to dialogue act prediction. The reviewers generally like the novelty of the approach and agree that there are improvements over SOTA, but also comment that those improvements are marginal. The authors reply with additional experiments demonstrating the statistical significance of their results. For these reasons I believe this would be a good candidate for Findings.
Decision: Accept-Findings