SlowGuess/ACL_2020_OCR huggingface dataset:
To look in more detail at how the system models the offset distribution we can investigate the generated distributions of labelled response dialogue acts in our test set. Fig. 7 shows plots of backchannels vs. statements (Fig. 7a), and yes vs. no (Fig.7b) responses. In the second rows, we can see that the full model is able to accurately capture the differences in the contours of the true distributions. For example, in the no dialogue acts, the full model accurately generates a mode that is delayed (relative to yes dialogue acts).
Encoder Ablation The performance of the response encoder was analysed in an ablation study, with results in rows 3 through 5 of Table 1. Without the response encoder, there is a large decrease in performance, relative to the full model. From looking at the encoders with only acoustic and linguistic modalities, we can see that the results benefit more from the acoustic modality than the linguistic modality. If we consider the impact of the encoder in more detail, we would expect that the network would not be able to model distributional differences between different types of DA responses without an encoder. This is confirmed in the fourth rows of Fig. 7, where we show the generated distributions without the encoder. We can see that without the encoder, the distributions of the all of the dialogue act offsets are almost exactly the same.
Inference Network Ablation In rows 6 and 7 of Table 1 we present an ablation of the inference network. We can see that removing either the acoustic or linguistic features from the user's features is detrimental to the results. An interesting irregular
(a) Only Acoustic
(b) Only Linguistic
Figure 8: Generated offset distributions for the inference network ablation.
(a)  $w_{\mathrm{KL}} = 0.0$
Figure 9: T-SNE plots of  $z$  for four different dialogue acts using two different  $w_{\mathrm{KL}}$  settings.
(b)  $w_{\mathrm{KL}} = 10^{-3}$
ity is observed in the results for the model that uses only acoustic features (row 6): the MAE is unusually high, relative to the  $L_{\mathrm{BCE}}$ . In all other rows, lower  $L_{\mathrm{BCE}}$  corresponds to lower MAE. However, row 6 has the second lowest  $L_{\mathrm{BCE}}$ , while also having the second highest MAE.
In order to examine this irregularity in more detail, we look at the generated distributions from the inference ablation, shown in Fig. 8. We observe that the linguistic features are better for predicting the mode of the distribution whereas the acoustic features are better at modelling the  $-100\mathrm{ms}$  to  $+150\mathrm{ms}$  region directly preceding the mode. Since word embeddings are triggered  $100\mathrm{ms}$  after the end of the word, the linguistic features can be used to generate modal offsets in the  $150\mathrm{ms}$  to  $200\mathrm{ms}$  bin. We propose that, in the absence of linguistic features, there is more uncertainty about when the user's turn-end has occurred. Since the majority of all ground-truth offsets occur after the user has finished speaking, the unusually high MAE in row 6 could be attributed to this uncertainty in whether the user has finished speaking.
3.2 RTNet-VAE Discussion
RTNet-VAE Performance In rows 8 through 12 of Table 1 we show the results of our experiments with RTNet-VAE with different settings of  $w_{\mathrm{KL}}$ . As  $w_{\mathrm{KL}}$  is increased, the  $L_{\mathrm{BCE}}$  loss increases while the  $L_{\mathrm{KL}}$  loss decreases. Examining some example distributions of dialogue acts generated by RTNet-VAE using  $w_{\mathrm{KL}} = 10^{-4}$  (shown in the fifth rows of Fig. 7) we can see that RTNet-VAE is capa
Figure 10: Interpolated distributions
ble of generating distributions that are of a similar quality to those generated by RTNet (shown in the second row). We also observe that RTNet-VAE using  $w_{\mathrm{KL}} = 10^{-4}$  produces competitive results, in comparison to the full model. These observations suggest that the inclusion of the VAE in pipeline does not severely impact the overall performance.
In Fig. 9 we show the latent variable  $z$  generated using RTNet-VAE and plotted using t-SNE (van der Maaten and Hinton, 2008). To show the benefits of imposing the Gaussian prior, we show plots for with  $w_{\mathrm{KL}} = 0.0$  and  $w_{\mathrm{KL}} = 10^{-3}$ . The plots show the two-dimensional projection of four different types of dialogue act responses: statements (sd), no (nn), yes (ny), and backchannels (b). We can observe that for both settings, the latent space is able to organize the responses by dialogue act type, even though it is never explicitly trained on dialogue act labels. For example, in both cases, statements (shown in blue) are clustered at the opposite side of the distribution from backchannels (shown in red). However, in the case of  $w_{\mathrm{KL}} = 0.0$  there are "holes" in the latent space. For practical applications such as interpolation of vector representations of dialogue acts (discussed in the next paragraph), we would like a space that does not contain any of these holes since they are less likely to have semantically meaningful interpretations. When the Gaussian prior is enforced (Fig. 9b) we can see that the space is smooth and the distinctions between dialogue acts is still maintained.
Latent Space Applications As mentioned in Section 2.2, part of the appeal in using the VAE in our model is that it enables us to discard the response encoding stage. We can exploit the smoothness of the latent space to skip the encoding stage by sampling directly from the trained latent space.
We can approximate the distribution of latent variables for individual dialogue act response types using isotropic Gaussians. This enables us to efficiently represent the dialogue acts using mean and standard-deviation vectors, a pair for each dialogue act. Fig. 7 shows examples of distributions generated using Gaussian approximations of the latent space distributions in the final rows. We can see that the generated outputs have similar properties to the true distributions.
We can use the same parameterized vector representations to interpolate between different dialogue act parameters to achieve intermediate distributions. This dimensional approach is flexible in that we give the dialogue manager (DM) more control over the details of the distribution. For example, if the objective of the SDS was to generate an agree dialogue act, we could control the degree of agreement by interpolating between disagree and agree vectors. Figure 10 shows an example of a generated interpolated distribution. We can see that the properties of the interpolated distribution (e.g. mode, kurtosis) are perceptually "in between" the reject and accept distributions.
4 Listening Tests
It has shown that response timings vary based on the semantic content of dialogue responses and the preceding turn (Levinson and Torreira, 2015), and that listeners are sensitive to these fluctuations in timing (Bögels and Levinson, 2017). However, the question of whether certain response timings within different contexts are considered more realistic than others has not been fully investigated. We design an online listening test to answer two questions: (1) Given a preceding turn and a response, are some response timings considered by listeners to be more realistic than others? (2) In cases where listeners are sensitive to the response timing, is our model more likely to generate responses that are considered realistic than a system that generates a modal response time?
Participants were asked to make A/B choices between two versions of a turn pair, where each version had a different response offset. Participants were asked: "Which response timing sounds like it was produced in the real conversation?" The turn pairs were drawn from our dataset and were limited to pairs where the response was either dispreferred or a backchannel. We limited the chosen pairs to those with ground truth offsets that were either clas
(a) Early, modal, and late regions.
(b) Generated distributions for six turn pairs. The highlighted regions indicate the region that was preferred by listeners. The red line indicates the ground truth offset.
Figure 11: Listening test experiments
sified as early or late. We classified offsets as early, modal, or late by segmenting the distribution of all of the offsets in our dataset into three partitions as shown in Fig. 11a. The cutoff points for the early and late offsets were estimated using a heuristic where we split the offsets in our dataset into two groups at the mode of the distribution (157 ms) and then used the median values of the upper (+367 ms) and lower (-72 ms) groups as the cutoff points. We selected eight examples of each dialogue act (four early and four late). We generated three different versions of each turn pair: true, modal, and opposite. If the true offset was late, the opposite offset was the mean of the early offsets (-316 ms). If the true offset was early, the opposite offset was the mean of the late offsets (+760 ms).
We had 25 participants (15 female, 10 male) who all wore headphones. We performed binomial tests for the significance of a given choice in each question. For the questions in the first half of the test, in which we compared true vs. opposite offsets, 10 of the 16 comparisons were found to be statistically significant  $(p < 0.05)$ . In all of the significant cases the true offset was considered more realistic than the opposite. In reference to our first research question, this result supports the conclusion that some responses are indeed considered to be more realistic than others. For the questions in the second half of the test, in which we compared true vs. modal offsets, six out of the 16 comparisons were found to be statistically significant. Of the six significant preferences, three were a preference for the true offset, and three were a preference for the modal offset. To investigate our second research question, we looked at the offset distributions generated by our model for each of
the six significant preferences, shown in Fig. 11b. For the turn pairs where listeners preferred non-modal offsets (top row), the distributions generated by our system deviate from the mode into the preferred area (highlighted in yellow). In pairs where listeners preferred modal offsets (bottom row) the generated distributions tend to have a mode near the overall dataset mode (shown in the green line). We can conclude, in reference to our second question, that in instances where listeners are sensitive to response timings it is likely that our system will generate response timings that are more realistic than a system that simply generates the mode of the dataset.
In this paper, we have presented models that can be used to generate the turn switch offset distributions of SDS system responses. It has been shown in prior studies (e.g. (Bögels et al., 2019)) that humans are sensitive to these timings and that they can impact how responses are perceived by a listener. We would argue that they are an important element of producing naturalistic interactions that is often overlooked. With the advent of commercial SDS systems that attempt to engage users over extended multi-turn interactions (e.g. (Zhou et al., 2018)) generating realistic response behaviors is a potentially desirable addition to the overall experience.
The ADAPT Centre for Digital Content Technology is funded under the SFI Research Centres Programme (Grant 13/RC/2106) and is co-funded under the European Regional Development Fund.
Sara Bogels, Robin H. Kendrick, and Stephen C. Levinson. 2019. Conversational expectations get revised as response latencies unfold. Language, Cognition and Neuroscience, pages 1-14.
Sara Bogels and Stephen C. Levinson. 2017. The Brain Behind the Response: Insights Into Turn-taking in Conversation From Neuroimaging. Research on Language and Social Interaction, 50(1):71-89.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating Sentences from a Continuous Space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-21, Berlin, Germany. Association for Computational Linguistics.
Sasha Calhoun, Jean Carletta, Jason M. Brenier, Neil Mayo, Dan Jurafsky, Mark Steedman, and David Beaver. 2010. The NXT-format Switchboard Corpus: A rich resource for investigating the syntax, semantics, pragmatics and prosody of dialogue. Language Resources and Evaluation, 44(4):387-419.
Nina Dethlefs, Helen Hastie, Verena Rieser, and Oliver Lemon. 2012. Optimising incremental dialogue decisions using information density for interactive systems. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 82-93. Association for Computational Linguistics.
David DeVault, Kenji Sagae, and David Traum. 2011. Incremental interpretation and prediction of utterance meaning for interactive dialogue. Dialogue & Discourse, 2(1):143-170.
Florian Eyben, Klaus R. Scherer, Bjorn W. Schuller, Johan Sundberg, Elisabeth Andre, Carlos Busso, Laurence Y. Devillers, Julien Epps, Petri Laukka, Shrikanth S. Narayanan, and Khiet P. Truong. 2016. The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing. IEEE Transactions on Affective Computing, 7(2):190-202.
John J Godfrey and Edward Holliman. 1997. Switchboard-1 release 2. Linguistic Data Consortium, Philadelphia, 926:927.
David Ha and Douglas Eck. 2018. A neural representation of sketch drawings. In International Conference on Learning Representations.
Peter A. Heeman and Rebecca Lunsford. 2017. Turn-taking offsets and dialogue context. In Proc. Interspeech 2017, pages 1671-1675.
Kobin H. Kendrick and Francisco Torreira. 2015. The timing and construction of preference: A quantitative study. Discourse Processes, 52(4):255-289.
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In 2nd International Conference on Learning Representations.
Divesh Lala, Pierrick Milhorat, Koji Inoue, Masanari Ishida, Katsuya Takanashi, and Tatsuya Kawahara. 2017. Attentive listening system with backchanneling, response generation and flexible turn-taking. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 127-136, Saarbrücken, Germany. Association for Computational Linguistics.
Divesh Lala, Shizuka Nakamura, and Tatsuya Kawahara. 2019. Analysis of Effect and Timing of Fillers in Natural Turn-Taking. In *Interspeech* 2019, pages 4175-4179. ISCA.
Stephen C. Levinson and Francisco Torreira. 2015. Timing in turn-taking and its implications for processing models of language. Frontiers in Psychology, 6.
Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A Persona-Based Neural Conversation Model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 994-1003, Berlin, Germany. Association for Computational Linguistics.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research, 9(Nov):2579-2605.
Raveesh Meena, Gabriel Skantze, and Joakim Gustafson. 2014. Data-driven models for timing feedback responses in a Map Task dialogue system. Computer Speech & Language, 28(4):903-922.
Louis-Philippe Morency, Iwan de Kok, and Jonathan Gratch. 2010. A probabilistic multimodal approach for predicting listener backchannels. Autonomous Agents and Multi-Agent Systems, 20(1):70-84.
Ryosuke Nakanishi, Koji Inoue, Shizuka Nakamura, Katsuya Takanashi, and Tatsuya Kawahara. 2018. Generating Fillers based on Dialog Act Pairs for Smooth Turn-Taking by Humanoid Robot. IWSDS, page 11.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Antoine Raux and Maxine Eskenazi. 2009. A finite-state turn-taking model for spoken dialog systems. In HLT-NAACL, pages 629-637. ACL.
Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck. 2018. A hierarchical latent vector model for learning long-term structure in music. In ICML, pages 4361-4370.
Matthew Roddy, Gabriel Skantze, and Naomi Harte. 2018. Multimodal Continuous Turn-Taking Prediction Using Multiscale RNNs. In Proceedings of the 2018 on International Conference on Multimodal Interaction - ICMI '18, pages 186-190, Boulder, CO, USA. ACM Press.
Harvey Sacks, Emanuel A. Schegloff, and Gail Jefferson. 1974. A Simplest Systematics for the Organization of Turn-Taking for Conversation. Language, 50(4):696.
David Schlangen and Gabriel Skantze. 2011. A general, abstract model of incremental dialogue processing. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, pages 710-718.
Gabriel Skantze. 2017. Towards a General, Continuous Model of Turn-taking in Spoken Dialogue using LSTM Recurrent Neural Networks. In Proceedings of SigDial, Saarbrucken, Germany.
RJ Skerry-Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron J Weiss, Rob Clark, and Rif A Saurous. 2018. Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron. Proceedings of the 35th International Conference on Machine Learning, page 10.
Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteor. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339-373.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 654-664, Vancouver, Canada. Association for Computational Linguistics.
Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2018. The Design and Implementation of XiaoIce, an Empathetic Social Chatbot. arXiv preprint arXiv:1812.08989.