SlowGuess/ACL_2020_OCR huggingface dataset:
Matthew Roddy and Naomi Harte  
ADAPT Centre, School of Engineering  
Trinity College Dublin, Ireland  
{roddym, nharte}@tcd.ie
The timings of spoken response offsets in human dialogue have been shown to vary based on contextual elements of the dialogue. We propose neural models that simulate the distributions of these response offsets, taking into account the response turn as well as the preceding turn. The models are designed to be integrated into the pipeline of an incremental spoken dialogue system (SDS). We evaluate our models using offline experiments as well as human listening tests. We show that human listeners consider certain response timings to be more natural based on the dialogue context. The introduction of these models into SDS pipelines could increase the perceived naturalness of interactions.
The components needed for the design of spoken dialogue systems (SDSs) that can communicate in a realistic human fashion have seen rapid advancements in recent years (e.g. Li et al. (2016); Zhou et al. (2018); Skerry-Ryan et al. (2018)). However, an element of natural spoken conversation that is often overlooked in SDS design is the timing of system responses. Many turn-taking components for SDSs are designed with the objective of avoiding interrupting the user while keeping the lengths of gaps and overlaps as low as possible e.g. Raux and Eskenazi (2009). This approach does not emulate naturalistic response offsets, since in human-human conversation the distributions of response timing offsets have been shown to differ based on the context of the first speaker's turn and the context of the addressee's response (Sacks et al., 1974; Levinson and Torreira, 2015; Heeman and Lunsford, 2017). It has also been shown that listeners have different anticipations about upcoming
Figure 1: Overview of how our model generates the distribution of turn-switch offset timings using an encoding of a dialogue system response  $h_z$ , and features extracted from the user's speech  $x_n$ .
responses based on the length of a silence before a response (Bögels et al., 2019). If we wish to realistically generate offsets distributions in SDSs, we need to design response timing models that take into account the context of the user's speech and the upcoming system response. For example, offsets where the first speaker's turn is a backchannel occur in overlap more frequently (Levinson and Torreira, 2015). It has also been observed that dispreferred responses (responses that are not in line with the suggested action in the prior turn) are associated with longer delays (Kendrick and Torreira, 2015; Bögels et al., 2019).
1 Our code is available at https://github.com/mattroddy/RTNets.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2442-2452 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Overview We propose a neural model for generating these response timings in SDSs (shown in Fig. 1). The response timing network (RTNet) operates using both acoustic and linguistic features extracted from user and system turns. The two main components are an encoder, which encodes the system response  $h_{z}$ , and an inference network, which takes a concatenation of user features  $(x_{n})$  and  $h_{z}$ . RTNet operates within an incremental SDS framework (Schlangen and Skantze, 2011) where information about upcoming system responses may be available before the user has finished speaking. RTNet also functions independently of higher-level turn-taking decisions that are traditionally made by the dialogue manager (DM) component. Typically, the DM decides when the system should take a turn and also supplies the natural language generation (NLG) component with a semantic representation of the system response (e.g. intents, dialogue acts, or an equivalent neural representation). Any of the system response representations that are downstream from the DM's output representation (e.g. lexical or acoustic features) can potentially be used to generate the response encoding. Therefore, we assume that the decision for the system to take a turn has already been made by the DM and our objective is to predict (on a frame-by-frame basis) the appropriate time to trigger the system turn.
It may be impractical in an incremental framework to generate a full system response and then re-encode it using the response encoder of RTNet. To address this issue, we propose an extension of RTNet that uses a variational autoencoder (VAE) (Kingma and Welling, 2014) to train an interpretable latent space which can be used to bypass the encoding process at inference-time. This extension (RTNet-VAE) allows the benefit of having a data-driven neural representation of response encodings that can be manipulated without the overhead of the encoding process. This representation can be manipulated using vector algebra in a flexible manner by the DM to generate appropriate timings for a given response.
Our model's architecture is similar to VAEs with recurrent encoders and decoders proposed in Bowman et al. (2016); Ha and Eck (2018); Roberts et al. (2018). Our use of a VAE to cluster dialogue acts is similar to the approach used in Zhao et al. (2017). Our vector-based representation of dialogue acts takes inspiration from the 'attribute vectors' used in Roberts et al. (2018) for learning musical struc
ture representations. Our model is also related to continuous turn-taking systems (Skantze, 2017) in that our model is trained to predict future speech behavior on a frame-by-frame basis. The encoder uses a multiscale RNN architecture similar to the one proposed in Roddy et al. (2018) to fuse information across modalities. Models that intentionally generate responsive overlap have been proposed in DeVault et al. (2011); Dethlefs et al. (2012). While other models have also been proposed that generate appropriate response timings for fillers (Nakanishi et al., 2018; Lala et al., 2019) and backchannels (Morency et al., 2010; Meena et al., 2014; Lala et al., 2017).
This paper is structured as follows: First, we present how our dataset is structured and our training objective. Then, in sections 2.1 and 2.2 we present details of our two models, RTNet and RTNet-VAE. Section 2.3 presents our input feature representations. In section 2.4 we discuss our training and testing procedures. In sections 3.1 and 3.2 we analyze the performance of both RTNet and RTNet-VAE. Finally, in section 4 we present the results of a human listener test.
Dataset Our dataset is extracted from the Switchboard-1 Release 2 corpus (Godfrey and Holliman, 1997). Switchboard has 2438 dyadic telephone conversations with a total length of approximately 260 hours. The dataset consists of pairs of adjacent turns by different speakers which we refer to as turn pairs (shown in Fig. 2). Turn pairs are automatically extracted from orthographic annotations using the following procedure: We extract frame-based speech-activity labels for each speaker using a frame step-size of  $50\mathrm{ms}$ . The frame-based representation is used to partition each person's speech signal into interpausal units (IPUs). We define IPUs as segments of speech by a person that are separated by pauses of  $200\mathrm{ms}$  or greater. IPUs are then used to automatically extract turns, which we define as consecutive IPUs by a speaker in which there is no speech by the other speaker in the silence between the IPUs. A turn pair is then defined as being any two adjacent turns by different speakers. The earlier of the two turns in a pair is considered to be the user turn and the second is considered to be the system turn.
Training Objective Our training objective is to predict the start of the system turn one frame ahead
Figure 2: Segmentation of data into turn pairs, and how the inference LSTM makes predictions.
of the ground truth start time. The target labels in each turn pair are derived from the ground truth speech activity labels as shown in Fig. 2. Each 50 ms frame has a label  $y \in \{0,1\}$ , which consists of the ground truth voice activity shifted to the left by one frame. As shown in the figure, we only include frames in the span  $R$  in our training loss. We define the span  $R$  as the frames from the beginning of the last IPU in the user turn to the frame immediately prior to the start of the system turn.
We do not predict at earlier frames since we assume that at these mid-turn-pauses the DM has not decided to take a turn yet, either because it expects the user to continue, or it has not formulated one yet. As mentioned previously in section 1, we design RTNet to be abstracted from the turn-taking decisions themselves. If we were to include pauses prior to the turn-final silence, our response generation system would be additionally burdened with making turn-taking decisions, namely, classifying between mid-turn-pauses and end-of-turn silences. We therefore make the modelling assumption that the system's response is formulated at some point during the user's turn-final IPU. To simulate this assumption we sample an index  $R_{\mathrm{START}}$  from the span of  $R$  using a uniform distribution. We then use the reduced set of frames from  $R_{\mathrm{START}}$  to  $R_{\mathrm{END}}$  in the calculation of our loss.
2.1 Response Timing Network (RTNet)
Encoder The encoder of RTNet (shown in Fig. 3) fuses the acoustic and linguistic modalities from a system response using three bi-directional LSTMs. Each modality is processed at independent timescales and then fused in a master Bi-LSTM which operates at the linguistic temporal rate. The output of the master Bi-LSTM is a sequence of encodings  $h_0, h_1, \ldots, h_I$ , where each encoding is a concatenation of the forward and backward hidden states of the master Bi-LSTM at each word index.
The linguistic Bi-LSTM takes as input the sequence of 300-dimensional embeddings of the tokenized system response. We use three special tokens: SIL, WAIT, and NONE. The SIL token is used whenever there is a gap between words that is greater than the frame-size (50ms). The WAIT and NONE tokens are inserted as the first and last tokens of the system response sequence respectively. The concatenation  $[h_0; h_1; h_I]$  is passed as input to a RELU layer (we refer to this layer as the reduction layer) which outputs the  $h_z$  encoding. The  $h_z$  encoding is used (along with user features) in the concatenated input to the inference network. Since the WAIT embedding corresponds to the  $h_0$  output of the master Bi-LSTM and the NONE embedding corresponds to  $h_I$ , the two embeddings serve as "triggering" symbols that allow the linguistic and master Bi-LSTM to output relevant information accumulated in their cell states.
The acoustic Bi-LSTM takes as input the sequence of acoustic features and outputs a sequence of hidden states at every 50ms frame. As shown in Fig. 3, we select the acoustic hidden states that correspond to the starting frame of each linguistic token and concatenate them with the linguistic hidden states. Since there are no acoustic features available for the WAIT and NONE tokens, we train two embeddings to replace these acoustic LSTM states (shown in purple in Fig. 3). The use of acoustic embeddings results in there being no connection between the WAIT acoustic embedding and the first acoustic hidden state. For this reason we include  $h_1$  in the  $[h_0; h_1; h_I]$  concatenation, in order to make it easier for information captured by the the acoustic bi-LSTM to be passed through to the final concatenation.
Inference Network The aim of our inference network is to predict a sequence of output probabilities  $Y = [y_{R_{\mathrm{START}}}, y_{R_{\mathrm{START}} + 1}, \dots, y_N]$  using
Figure 3: The encoder is three stacked Bi-LSTMs. We use special embeddings (shown in purple) to represent the acoustic states corresponding to the first and last tokens (WAIT and NONE) of the system's turn.
Figure 4: VAE
a response encoding  $h_z$ , and a sequence of user features  $X = [x_0, x_1, \dots, x_N]$ . We use a single-layer LSTM (shown in Fig. 2) which is followed by a sigmoid layer to produce the output probabilities:
$$
\left[ h _ {n}; c _ {n} \right] = \operatorname {L S T M} _ {\inf } \left(\left[ x _ {n}; h _ {z} \right], \left[ h _ {n - 1}; c _ {n - 1} \right]\right)
$$
$$
y _ {n} = \sigma \left(\boldsymbol {W} _ {h} h _ {n} + \boldsymbol {b} _ {h}\right)
$$
Since there are only two possible output values in a generated sequence  $\{0,1\}$ , and the sequence ends once we predict 1, the inference network can be considered an autoregressive model where 0 is passed implicitly to the subsequent time-step. To generate an output sequence, we can sample from the distribution  $p(y_{n} = 1|y_{R_{\mathrm{START}}} = 0,y_{R_{\mathrm{START}} + 1} = 0,\dots ,y_{n - 1} = 0,X_{0:n},h_{z})$  using a Bernoulli random trial at each time-step. For frames prior to  $R_{\mathrm{START}}$  the output probability is fixed to 0, since  $R_{\mathrm{START}}$  is the point where the DM has formulated the response. During training we minimize the binary cross entropy loss ( $L_{\mathrm{BCE}}$ ) between our ground truth objective and our output predictions  $Y$ .
2.2 RTNet-VAE
Motivation A limitation of RTNet is that it may be impractical to encode system turns before triggering a response. For example, if we wish to apply RTNet using generated system responses, at run-time the RTNet component would have to wait for the full response to be generated by the NLG, which would result in a computational bottleneck.
If the NLG system is incremental, it may also be desirable for the system to start speaking before the entirety of the system response has been generated.
VAE To address this, we bypass the encoding stage by directly using the semantic representation output from the DM to control the response timing encodings. We do this by replacing the reduction layer with a VAE (Fig. 4). To train the VAE, we use the same concatenation of encoder hidden states as in the RTNet reduction layer  $\left([h_0; h_1; h_I]\right)$ . We use a dimensionality reduction RELU layer to calculate  $h_{\mathrm{reduce}}$ , which is then split into  $\mu$  and  $\hat{\sigma}$  components via two more RELU layers.  $\hat{\sigma}$  is passed through an exponential function to produce  $\sigma$ , a non-negative standard deviation parameter. We sample the latent variable  $z$  with the standard VAE method using  $\mu$ ,  $\sigma$ , and a random vector from the standard normal distribution  $\mathcal{N}(0, \mathbf{I})$ . A dimensionality expansion RELU layer is used to transform  $z$  into the response encoding  $h_z$ , which is the same dimensionality as the output of the encoder:
$$
h _ {\text {r e d u c e}} = \operatorname {R E L U} \left(W _ {\text {r e d u c e}} \left[ h _ {0}; h _ {1}; h _ {I} \right] + b _ {\text {r e d u c e}}\right)
$$
$$
\mu = \operatorname {R E L U} \left(W _ {\mu} h _ {\text {r e d u c e}} + b _ {\mu}\right)
$$
$$
\hat {\sigma} = \operatorname {R E L U} \left(W _ {\sigma} h _ {\text {r e d u c e}} + b _ {\sigma}\right)
$$
$$
\sigma = \exp (\frac {\hat {\sigma}}{2})
$$
$$
z = \mu + \sigma \odot \mathcal {N} (0, \mathbf {I})
$$
$$
h _ {z} = \operatorname {R E L U} \left(W _ {\text {e x p a n d}} z + b _ {\text {e x p a n d}}\right)
$$
We impose a Gaussian prior over the latent space using a Kullback-Liebler (KL) divergence loss term:
$$
L _ {\mathrm {K L}} = - \frac {1}{2 N _ {z}} (1 + \hat {\sigma} - \mu^ {2} - \exp (\hat {\sigma}))
$$
The  $L_{\mathrm{KL}}$  loss measures the distance of the generated distribution from a Gaussian with zero mean
and unit variance.  $L_{\mathrm{KL}}$  is combined with  $L_{\mathrm{BCE}}$  using a weighted sum:
$$
L = L _ {\mathrm {B C E}} + w _ {\mathrm {K L}} L _ {\mathrm {K L}}
$$
As we increase the value of  $w_{\mathrm{KL}}$  we increasingly enforce the Gaussian prior on the latent space. In doing so our aim is to learn a smooth latent space in which similar types of responses are organized in similar areas of the space.
Latent Space During inference we can skip the encoding stage of RTNet-VAE and sample  $z$  directly from the latent space on the basis of the input semantic representation from the dialogue manager. Our sampling approach is to approximate the distribution of latent variables for a given response-type using Gaussians. For example, if we have a collection of labelled backchannel responses (and their corresponding  $z$  encodings) we can approximate the distribution of  $p(z|\text{label} = \text{backchannel})$  using an isotropic Gaussian by simply calculating  $\mu_{\text{backchannel}}$  and  $\sigma_{\text{backchannel}}$ , the maximum likelihood mean and standard deviations of each of the  $z$  dimensions. These vectors can also be used to calculate directions in the latent space with different semantic characteristics and then interpolate between them.
2.3 Input Feature Representations
Linguistic Features We use the word annotations from the ms-state transcriptions as linguistic features. These annotations give us the timing for the starts and ends of all words in the corpus. As our feature representation, we use 300 dimensional word embeddings that are initialized with GloVe vectors (Pennington et al., 2014) and then jointly optimized with the rest of the network. In total there are 30080 unique words in the annotations. We reduced the embedding number down to 10000 by merging embeddings that had low word counts with the closest neighbouring embedding (calculated using cosine distance).
We also introduce four additional tokens that are specific to our task: SIL, WAIT, NONE, and UNSPEC. SIL is used whenever there is a silence. WAIT and NONE are used at the start and end of all the system encodings, respectively. The use of UNSPEC (unspecified) is shown in Fig. 5. UNSPEC was introduced to represent temporal information in the linguistic embeddings. We approximate the processing delay in ASR by delaying the annotation by  $100\mathrm{ms}$  after the ground truth frame where
the user's word ended. This 100 ms delay was proposed in Skantze (2017) as a necessary assumption to modelling linguistic features in offline continuous systems. However, since voice activity detection (VAD) can supply an estimate of when a word has started, we propose that we can use this information to supply the network with the UNSPEC embedding 100ms after the word has started.
Acoustic Features We combine 40 log-mel filterbanks, and 17 features from the GeMAPs feature set (Eyben et al., 2016). The GeMAPs features are the complete set excluding the MFCCs (e.g. pitch, intensity, spectral flux, jitter, etc.). Acoustic features were extracted using a 50ms framestep.
2.4 Experimental Settings
Training and Testing Procedures The training, validation, and test sets consist of 1646, 150, 642 conversations respectively with 151595, 13910, and 58783 turn pairs. The test set includes all of the conversations from the NXT-format annotations (Calhoun et al., 2010), which include references to the Switchboard Dialog Act Corpus (SWDA) (Stolcke et al., 2000) annotations. We include the entirety of the NXT annotations in our test set so that we have enough labelled dialogue act samples to analyse the distributions.
We used the following hyperparameter settings in our experiments: The inference, acoustic, linguistic, and master LSTMs each had hidden sizes of 1024, 256, 256, and 512 (respectively). We used a latent variable size of 4, a batch size of 128, and L2 regularization of 1e-05. We used the Adam optimizer with an initial learning rate of 5e-04. We trained each model for 15000 iterations, with learning rate reductions by a factor of 0.1 after 9000, 11000, 13000, and 14000 iterations.
While we found that randomizing  $R_{\mathrm{START}}$  during training was important for the reasons given in Section 2, it presented issues for the stability and reproducibility of our evaluation and test results for  $L_{\mathrm{BCE}}$  and  $L_{\mathrm{KL}}$ . We therefore randomize during training and sampling, but when calculating the test losses (reported in Table 1) we fix  $R_{\mathrm{START}}$  to be the first frame of the user's turn-final IPU.
We also calculate the mean absolute error (MAE), given in seconds, from the ground truth response offsets to the generated output offsets. When sampling for the calculation of MAE, it is necessary to increase the length of the turn pair since the response time may be triggered by the
Figure 5: The user's linguistic feature representation scheme. The embedding for each word is triggered  $100\mathrm{ms}$  after the ground truth end of the word, to simulate ASR delay. The UNSPEC embedding begins  $100\mathrm{ms}$  after a word's start frame and holds information about whether a word is being spoken (before it has been recognized) and the length of each word.
sampling process after the ground truth time. We therefore pad the user's features with 80 extra frames in which we simulate silence artificially using acoustic features. During sampling, we use the same  $R_{\mathrm{START}}$  randomization process that was used during training, rather than fixing it to the start of the user's turn-final IPU. For each model we perform the sampling procedure on the test set three times and report the mean error in Table 1.
Best Fixed Probability To the best of our knowledge, there aren't any other published models that we can directly compare ours to. However, we can calculate the best performance that can be achieved using a fixed value for  $y$ . The best possible fixed  $y$  for a given turn pair is:  $y_{\mathrm{tp}} = \frac{1}{(R_{\mathrm{END}} - R_{\mathrm{START}}) / \mathrm{FrameLength}}$ . The best fixed  $y$  for a set of turn pairs is given by the expected value of  $y_{\mathrm{tp}}$  in that set:  $y_{\mathrm{fixed}} = \mathbb{E}[y_{\mathrm{tp}}]$ . This represents the best performance that we could achieve if we did not have access to any user or system features. We can use the fixed probability model to put the performance of the rest of our models into context.
(a) Full Model
(b) Fixed Probability
3.1 RTNet Discussion
RTNet Performance The offset distribution for the full RTNet model is shown in Fig. 6a. This
#	Model	L BCE	L KL	MAE	Details
1	Full Model	0.1094	-	0.4539	No VAE
2	Fixed Probability	0.1295	-	1.4546	Fixed Probability
3	No Encoder	0.1183	-	0.4934	Encoder Ablation
4	Only Acoustic	0.1114	-	0.4627
5	Only Linguistic	0.1144	-	0.4817
6	Only Acoustic	0.1112	-	0.5053	Inference Ablation
7	Only Linguistic	0.1167	-	0.4923
8	wKL=0.0	0.1114	3.3879	0.4601	Inclusion of VAE
9	wKL=10-4	0.1122	1.5057	0.4689
10	wKL=10-3	0.1125	0.8015	0.4697
11	wKL=10-2	0.1181	0.0000	0.5035
12	wKL=10-1	0.1189	0.0000	0.5052
Table 1: Experimental results on our test set. Lower is better in all cases. Best results shown in bold.
Full Model
No Encoder
Figure 6: Generated offset distributions for the test set using the full model and the fixed probability (random) model.
VAE
Vector Representation
(a) BC/Statement
Figure 7: Generated offset distributions for selected response dialogue acts using different model conditions.
Vector Representation
(b) Yes/No
baseline RTNet model is better able to replicate many of the features of the true distribution in comparison with predicted offsets using the best possible fixed probability shown in Fig. 6b. The differences between the baseline and the fixed probability distributions are reflected in the results of rows 1 and 2 in Table 1. In Fig. 6a, the model has the most trouble reproducing the distribution of offsets between  $-500\mathrm{ms}$  and  $0\mathrm{ms}$ . This part of the distribution is the most demanding because it requires that the model anticipate the user's turn-ending. From the plots it is clear that our model is able to do that to a large degree. We observe that after the user has stopped speaking (from 0 seconds onward) the generated distribution follows the true distribution closely.