Modeling Real-Time Interactive Conversations as Timed Diarized Transcripts

## Abstract

Abstract Chatbots built upon language models have exploded in popularity, but they have largely been limited to synchronous, turn-by-turn dialogues. In this paper we present a simple yet general method to simulate real-time interactive conversations using pretrained text-only language models, by modeling timed diarized transcripts and decoding them with causal rejection sampling . We demonstrate the promise of this method with two case studies: instant messenger dialogues and spoken conversations, which require generation at about 30 tok/s and 20 tok/s respectively to maintain real-time interactivity. These capabilities can be added into language models using relatively little data and run on commodity hardware.

## 1 Introduction

Chatbots built upon language models have exploded in popularity, but their interaction model is extremely limited: the user and the system take turns writing messages, where the system waits until the user finishes their message to respond then responds instantly and uninterruptibly. Extensions to support audio have used speech to text and text to speech to eliminate the need for typing and reading the screen (OpenAI, ), but the constraints of the interaction model have remained the same.

In this paper we present a simple yet general method to simulate real-time interactive conversations using pretrained text-only language models, namely: model timed diarized transcripts—i.e., sequences of [timestamp, speaker id, message]—at the desired granularity, and then decode these transcripts with causal rejection sampling—i.e., sample a continuation that will be finalized at the predicted timestamp, and if there is intervening user input before the timestamp, reject the planned continuation (to the extent that its probability under the model has changed) and resample a new one. This method is naturally sparse over time and number of speakers, scaling computation with the amount of content being actively produced at each moment in time.

We demonstrate the promise of this method with case studies in two domains. First, we use the instant messenger chat history between the first authors to train a real-time interactive asynchronous text dialogue model. Second, we use public speech datasets with diarized transcripts to train a real-time spoken conversation model, cascaded through word-level speech to text and text to speech models. Here there is an additional complication in that real-time streaming speech to text systems are unstable, i.e., predictions may change in light of future context. We address this with retconning, i.e., revising the user’s input history but keeping any already finalized system outputs.

We evaluate these embodiments of our method with respect to performance (properties of the control token format and of our proof-of-concept implementation) and quality (test perplexity, offline human ratings, and online human ratings)—across finetuned models from 160M to 12B parameters. For the offline human rating setting only, we also use long in-context learning to test larger pretrained models available by API. In order to maintain real-time interactivity, generation needs to be about 28 tokens per second for the instant messenger use case and 22 tok/s for spoken conversations, which are easy to achieve on a single A100 at our model scales. We find that, predictably, better pretrained models lead to better results, though there is still obvious room for improvement with dataset/model scale.

We publicly release our code (and some demo videos) at \repolink. We hope that these proofs of concept spark the imagination and show that language models can easily be adapted to new real-time interaction modes.

## 2 Method

We model timed diarized transcripts using causally masked (decoder-only) language models. Given a sequence of events $e_{i}$, where each event $e_{i}$ consists of a timestamp $t_{i}$ (timed), a speaker id $s_{i}$ (diarized), and a message $m_{i}$ (transcript), we model $p(e_{i}|e_{1},...,e_{i-1})$. In practice, this function decomposes into $p(t_{i}|e_{1},...,e_{i-1})$, $p(s_{i}|e_{1},...,e_{i-1},t_{i})$, and $p(m_{i}|e_{1},...,e_{i-1},t_{i},s_{i})$, or even more granular distributions if these components are represented as multiple tokens. By modeling events sparsely over time, we are able to sample transcripts with computation proportional to the number/complexity of the events, rather than the time duration.

In order to make this model interactive, we use causal rejection sampling. We pick a particular speaker id $S$ to represent the user and sample candidates $\hat{e}_{i}\sim p(e_{i}|e_{1},...,e_{i-1})$, where we interpret the timestamps $t$ within these events with respect to the current real time. If an input from the user $(S,T,M)$ interrupts before the timestamp $\hat{t}_{i}$ is reached, we reject the candidate $\hat{e}_{i}$ and sample a new candidate $\hat{e}_{i+1}\sim p(e_{i+1}|e_{1},...,e_{i-1},e_{i}=(S,T,M))$. If no such interruption occurs before $\hat{t}_{i}$, there are two possibilities: If the speaker id $\hat{s}_{i}$ within $\hat{e}_{i}$ is not $S$, we accept the message candidate $\hat{m_{i}}$, emit it to the user, then sample $\hat{e}_{i+1}$, etc. If $\hat{s}_{i}$ is $S$, then we resample $\hat{e}^{\prime}_{i}\sim p(e_{i}|e_{1},...,e_{i-1},t_{i}\geq\hat{t}_{i})$.

Because it takes some amount of time $t_{latency}$ (varying with message length) to execute the model and sample from $p(e_{i}|...)$, if the user repeatedly provides input less than $t_{latency}$ before the predicted timestamps $\hat{t}_{i}$, the model will be starved and unable to generate any acceptable events. We provide two modifications to mitigate recomputation from user interruption:

First, we enforce a hard lower bound on the model’s generation bandwidth by stipulating that if the user input comes within $t_{react}$ of $\hat{t}_{i}$, we accept $\hat{e}_{i}$ as a candidate for $\hat{e}_{i+1}$. The relationship between $t_{latency}$ and $t_{react}$ determines whether the model can maintain real-time interactivity in the worst case. We do not expect moderate $t_{react}$ to harm generation quality too much because a human reaction time of approximately 150-200 ms (; ) should be reflected in the underlying causal structure of human training data.

Second, we reduce the average amount of recomputation by integrating speculative decoding (Leviathan et al., ; Chen et al., ). Rather than discard the candidate $\hat{e}_{i}$ unconditionally upon user interruption, we treat it as a draft for the new generation, rejecting and resampling based on the closeness of $p(e_{i}=\hat{e}_{i}|e_{1},...,e_{i-1},t_{i}\geq T)$ and $p(e_{i+1}=\hat{e}_{i}|e_{1},...,e_{i-1},e_{i}=(T,S,M))$. Note that this is different from traditional speculative decoding, where a smaller model for the same distribution drafts a candidate;(^1^11The traditional kind of speculative decoding could also be used to speed up the initial autoregressive candidate generation; we omit this for simplicity.) the use of different prompts under the same model resembles classifier-free guidance (Ho & Salimans, ; Sanchez et al., ). Like with $t_{react}$, we expect this to work to the extent that there is a looseness in the causal dependencies of nearby messages from different parties.(^2^22You can also trade off between potentially wasted computation and interactivity by sampling the timestamp first and waiting until it approaches to generate the rest of the message, vs. sampling multiple sequential event candidates ahead of time.)

See Algorithm 1 for a formal description of causal rejection sampling (speculative decoding omitted for clarity; see Appendix A for the full version), or see our code at [this link](https://github.com/gahdritz/rtic).

Figure: Algorithm 1 Causal rejection sampling (without speculative decoding)

We now present two case studies demonstrating how this method can be applied to different domains: instant messenger dialogues and spoken conversations.

### 2.1 Instant Messenger Dialogues

The method as described above can be applied to instant messenger dialogues with minimal modifications. We use as our domain 9 years of instant messenger history between the first authors. This means we are not just modeling the evolution of synchronous conversations where both participants are actively engaged, but asynchronous conversations where participants may be offline and where the date/time may influence the content of the conversation. Instant messenger conversations can be highly multimodal, in particular with audio, images, and hyperlinks; we consider only text and leave multimodality to future work.

In the notation from above, we instantiate $t$ with the message’s calendar date/time (down to decisecond granularity), $s$ with an id representing the message sender (one of the two authors), and $m$ with the message plaintext (terminated by an “end of message” token). As a sequence length optimization, when prefixes of the timestamp are repeated in consecutive messages, we omit them. We design the control format to be prefix-free so that it can be interpreted without lookahead while decoding; this means that control tokens can decoded in a structured way (including that time only flows forward) by appropriately filtering and renormalizing the next token vocabulary. See Figure 1(a) for a specification of the format and Figure 1(b) for an example of what preprocessed data looks like.

Figure: (a) Control token format. “?” denotes an optional element. In brief: the format consists of the speaker id (omitted when matching the previous message), then the timestamp (prefixes omitted when matching the previous message), then the message itself. We use distinct separators (’+’, ’;’, ’.’) between digit fields to distinguish them while decoding without lookahead, while remaining relatively tokenizer-agnostic. This format could be further optimized given a fixed vocabulary.

### 2.2 Spoken Conversations

We also apply our general method to timed diarized word-level automatic speech recognition (ASR) transcripts. By cascading input through speech-to-text and output through text-to-speech, we can simulate spoken conversations. Note that—like cascaded approaches in general—this has the obvious limitation that it bottlenecks the input and output through text, stripping away aspects of speech like tone and introducing errors from intermediate models. While there exist off-the-shelf streaming speech-to-text models that output word-level timestamps, we are not aware of any text-to-speech models (streaming or otherwise) that accept them as input: the closest is incremental text-to-speech (Ma et al., ). This limits our ability to generate natural-sounding speech; we use word-level text to speech invoked at the specified timestamps and consider this out of scope.

There is an additional complication due to the use of streaming speech-to-text models: these models are able to achieve low latency because they output preliminary transcriptions that may change in light of future input and are only finalized some time later. This means that not only can the user’s input interrupt the model’s candidate generation, but the input can retroactively change after a candidate has been generated, accepted, and spoken out.

We address this with retconning, i.e., when the speech-to-text model’s prediction for the input changes, we replace the old prediction with the new one in the transcript prefix, without changing any model generations that were accepted after that point. More formally, if we have sampled $\hat{e}_{j}\sim p(e_{j}|e_{1},...,e_{i},...,e_{j-1})$ and the user interrupts with a revision $e^{\prime}_{i}$, we reject $\hat{e}_{j}$ (subject to the $t_{react}$ window and speculation described above) and resample $\hat{e}^{\prime}_{j}\sim p(e_{j}|e_{1},...,e^{\prime}_{i},...,e_{j-1})$. This should not have a significant impact on either performance or quality, since processing $n$ tokens in parallel is much faster than $n$ tokens sequentially, and because humans also reinterpret what they’ve already heard in light of new speech (which should be reflected in ground truth causal structure). See Appendix B for a more formal description of causal rejection sampling with retconning, or see our code at [this link](https://github.com/gahdritz/rtic).

We use as our dataset 1000 hours of oral arguments before the U.S. Supreme Court (; Boyle, ). Court oral arguments are an interesting domain because they have many participants ($\sim$10 per transcript) and are information dense, though they have longer conversation turns and fewer interruptions than typical conversations.

In the formal language from Section 2, we instantiate $t$ with the word’s start timestamp modulo 10 seconds(^3^33This compromise reduces the number of tokens, at the expense of being able to model more than 10 seconds of silence.) (down to centisecond granularity), $s$ with an opaque identifier representing the speaker, and $m$ with the word plaintext (terminated by an “end of message” token).
We omit the speaker id in repeated spans. See Figure 2(a) for a more complete description of the format and Figure 2(b) for an example of what preprocessed data looks like.(^4^44Note that for generality, the duration of each word should probably also be modelled. We omit it here because it would be discarded in our word-level text to speech step anyway.)

Figure: (a) Control token format. “?” denotes an optional element. In brief: the format consists of the speaker id (omitted when matching the previous message), then the timestamp (prefixes omitted when matching the previous message), then the message itself.

## 3 Evaluation

For both case studies we evaluate performance and quality. We finetune the following models: Pythia 160M, 1.4B, & 12B (Biderman et al., ), Gemma 2B (Team et al., ), and Llama 2 7B (Touvron et al., ); see Appendix C for details.
Where possible, we also compare with in-context learning using state-of-the-art commercial language models: Claude 3 Sonnet () and GPT-4 Turbo (). See Appendix D for details.

For performance, we report:

- •
generation bandwidth in tokens/second required to maintain real-time interactivity, scored on historical data
- •
control token overhead ratio, scored on historical data
- •
speculation acceptance rate as an average number and fraction of draft tokens, scored on historical data
- •
performance properties for the proof of concept implementation

For quality, we report:

- •
document-level negative log likelihood (NLL) on the held out test set (rather than token-level perplexity, to make comparisons meaningful across tokenizers)
- •
offline human ratings, i.e., a human ranks conversations that were generated by continuing a prefix from the test set noninteractively
- •
online human ratings, i.e., a human interacts with each model given a conversation prefix from the test set, and then ranks them
- •
statistics about the distribution of predicted time gaps, compared to historical data

For human rating settings, we use the same prefixes of 64 messages ($\sim$1024 tokens) across all models. For the offline ratings, we also compare with the ground truth continuation.
Note that while context lengths have recently made massive strides (128K for GPT-4 Turbo (), $>$1M for Claude 3 , and $>$10M for Gemini 1.5 (Reid et al., )), they are still not long enough to fit our training sets (20.2M tokens of messenger history and 40.3M tokens of oral arguments) and usage is subject to rate limits. We therefore use only the most recent 16K tokens of history as context.

One of the first authors prepared the test harness; the other served as the rater. The human evaluation scores range from 0 to 6, where 0 is nonsensical and 6 is indistinguishable from real. These scores should only be used to judge relative quality and not quality in absolute.

### 3.1 Instant Messenger Dialogues

As our dataset we use 9 years of instant messenger conversation history between the first authors, totaling 37,649,697 characters across 1,393,508 text-based messages (we exclude messages from other modalities).
We use the first 95% of the messages as the train set, the next 2.5% as a validation set, and the last 2.5% as a test set.

#### 3.1.1 Performance

Figure: Figure 3: Statistics about the overhead of our control formats for instant messenger dialogues (top) and spoken conversations (bottom), and the requirements to maintain real-time interactivity. Left: Lengths (in Llama 2 tokens) of plaintext messages vs. control tokens for examples in the training set. Right: Fractions of the messages in the ground-truth dataset, including control tokens, that could be generated in real time for a given minimum generation rate, in tokens per second (again using the Llama 2 tokenizer). A message $m$ can be generated in real time if it can be generated in the time between the latest message outside of a short reaction window ($t_{react}=200$ms) immediately before $m$, and $m$ itself. (We assume that for small $n$, the increase in cost for passing $n$ tokens through the network in parallel vs. 1 token is negligible, i.e. we are primarily modeling the cost of generating system responses, not ingesting user inputs.) For spoken conversations, we include performance figures for an optimized tokenizer which treats uses a single token for 3-digit timestamps.

See Figure 3 for details on the performance properties of our instant messenger control format. The highlights are: With $t_{react}$ = 200ms, the 99th percentile generation bandwidth required to maintain real-time interactivity is 28 tok/s, and the 99.9th percentile is 75 tok/s. This range is largely pathological cases like long pasted text. On average, the control-formatted token length is 3.2x the plaintext length (median 2.4x); speculative sampling saves an additional 11.02 draft tokens (69.5% of tokens) per interruption in Llama 2.

In terms of our prototype: We interact with an A100 40GB server executing unquantized off-the-shelf model inference over ssh; this is more than sufficient to maintain real-time interactivity with all of our finetuned models. Communication latency is negligible, and the model checks for interruptions after generating each token (i.e., $\frac{1}{\text{\# tok/s}}$ latency).

#### 3.1.2 Quality

See Table 1 for instant messenger quality results across models; see Appendix F for qualitative examples. The trends are unsurprising: better pretrained models achieve better perplexity and better human ratings, though still substantially worse than the ground truth. One exception is that API-based models with in-context learning mimic style worse than finetuned models, and sometimes fail completely due to refusals. We now describe some qualitative observations:

**Table 1: Instant messenger (top) and spoken conversation (bottom) quality scores. ft = finetuned and icl = in-context learning. We compute negative log likelihood per document rather than averaged per token, so that it is comparable across vocabularies. Human ratings range from 0 (worst) to 6 (best). When relevant, we provide scores in parentheses with refusals filtered out. We rate consistency (how coherent the conversation is generally) and fidelity (how well the model mimics the authors specifically) for instant messenger, and content vs. timing for speech. See Appendix E for more details and experiments comparing the ground truth and predicted timestamp distributions.**
|  | NLL ($\downarrow$) | Offline Human Ratings ($\uparrow$) | Online Human Ratings ($\uparrow$) |  |  |
| --- | --- | --- | --- | --- | --- |
| Instant messenger |  | Consistency | Fidelity | Consistency | Fidelity |
| Pythia 160M (ft) | 3181 | 1.45 | 3.00 | 1.4 | 2.6 |
| Pythia 1.4B (ft) | 2397 | 2.55 | 3.65 | 3.4 | 4.8 |
| Pythia 12B (ft) | 2305 | 2.90 | 3.70 | 3.0 | 3.0 |
| Gemma 2B (ft) | 2376 | 2.95 | 3.65 | 2.8 | 3.2 |
| Llama 2 7B (ft) | 2179 | 3.90 | 4.40 | 3.8 | 4.2 |
| Claude 3 Sonnet (icl) | - | 1.85 (5.29) | 1.25 (3.57) | - | - |
| GPT-4 Turbo (icl) | - | 5.30 | 1.80 | - | - |
| ground truth | - | 5.95 | 6.00 | - | - |

##### Overpowering tone

API-based models are tuned to have a particular voice, which bleeds through into the generated messages. So while the conversations are more coherent, they are usually easy to distinguish from the ground truth based on style cues alone. Claude 3 often refuses to perform the task when the chat history discusses politics.

##### Speaker consistency

The finetuned models sometimes struggle to maintain consistent identities for the speakers, mostly across conversations (e.g., one speaker talks about having a sister, when it is only the other speaker who has a sister) but sometimes also within conversations (i.e., a speaker appears to respond to itself).

##### Promise as an evaluation for long context LLMs

Instant messenger history continuation is a promising task for human evaluation of long in-context learning. Each message history is highly distinct, yet private and therefore guaranteed to be unleaked. While it is prohibitively time-consuming for a human rater to read extremely long prompts in general, if they are instead a participant in the original conversation, they are already deeply familiar with the content and can easily spot errors without additional effort.

### 3.2 Spoken Conversations

As our training dataset, we use a random 1000-hour subset of cases argued before the U.S. Supreme Court, totaling 33,640,559
characters. We sample other cases into a $\sim$350-hour val set and $\sim$295-hour test set. We preprocess the data with WhisperX (Radford et al., ; Bain et al., ), which supports timed diarized word-level ASR. Note that pseudolabeled diarized speech data tends to undercapture timestamp overlap across speakers (Liesenfeld et al., ), so this data may not reflect fine-grained turn-taking behavior. We lowercase and strip punctuation from the data to make the formatting consistent with streaming ASR.

#### 3.2.1 Performance

See Figure 3 for more details on the performance properties of our spoken conversation control format.
The highlights are: With $t_{react}$ = 200ms, the 99th percentile is 36 tok/s and 99.9th is 45 tok/s. On average, the control-formatted token length is 4.3x the plaintext length (median 5x). Note that this ratio is heavily dependent on the way the tokenizer handles digits; many modern tokenizers force individual digits to be separate tokens to improve arithmetic, but in this case, given enough data, 000-999 could reasonably be single tokens. We calculate the rates for this “optimized tokenizer”: the 99th percentile is 22 tok/s and 99.9th is 30. On average, the control-formatted token length is 1.8x plaintext length (median 2.0x).

In terms of our proof of concept implementation: we use Google Cloud streaming [Speech-To-Text](https://cloud.google.com/speech-to-text?hl=en) and [Text-To-Speech](https://cloud.google.com/text-to-speech?hl=en) APIs on the client, piped through an ssh tty as text to an A100 40GB server. We measure the end-to-end latency of the former at about 500 ms (from word end to model input) and the latter at about 80 ms; on-device cascade and base models would likely have even lower latency.

#### 3.2.2 Quality

See Table 1 for spoken conversation quality results across models; see Appendix F for qualitative examples. It is prohibitively time-consuming to read the entire context or each case, and the rater has some legal knowledge but is not an expert, so there may be more of a gap in content quality than is reflected by the scores. In the offline human rating setting, we play the transcripts aloud to judge timing, though with word-level text to speech it is difficult to judge the finer points. Like for instant messanger dialogues, better pretrained models tend to achieve better results. Llama 2 7B (ft) responds remarkably well to turn-taking in the online setting, though there is still obvious room for improvement in all regards.

## 4 Related Work

We survey related work in three areas: text dialogues, spoken dialogues, and use of language models to model time broadly.

### 4.1 Text Dialogue Modeling

Modeling text dialogues is perhaps the founding problem of artificial intelligence: Turing’s imitation game poses the challenge of distinguishing man from machine through turn-by-turn text dialogue (Turing, ). While timing is mentioned here (a model that responds too quickly could be distinguished from a human), the interaction model is limited. Since then there has been a wealth of work on dialogue systems (Ni et al., ), initially with complex rule-based methods (Weizenbaum, ) but shifting over time towards unified deep learning methods, culminating in Meena & LaMDA (Adiwardana et al., ; Thoppilan et al., ), the Blenderbot series (Roller et al., ; Komeili et al., ; Shuster et al., ), and of course the recent wave of chatbots such as ChatGPT (Schulman et al., ), Gemini (Google, ), Copilot (), Claude (Anthropic, ), Pi (), Coral (), HuggingChat (), etc. These chatbot works have primarily focused on basic, goal-directed conversational capabilities in the desired domains, which until recently has been very challenging, and less on the interaction model. Replika () and certain modes in Character.AI () do allow multiple messages per conversation turn, but with undisclosed methods and unclear limitations.

CICERO (Bakhtin et al., ) studies Diplomacy, a political strategy game that involves instant messaging with other players in real time. The primary focus is on using dialogue paired with actions to achieve certain goals in the game, which implies the ability to imitate natural timing to avoid raising suspicion with human players. CICERO uses a chain of encoder-decoder models and heuristics to perform tasks such as predicting the next message time vs. content independently, and not all context is available to all models. Messages are rejected/resampled when user input causally intervenes on planned messages. Our work uses a simpler approach with a single transcript in a decoder-only model, which minimizes recomputation and makes all information available for all decisions; we further improve performance by using a reaction time window and causal speculative decoding.

The task of imitating specific people based on their digital footprint (for better or worse) has captured the popular imagination, featuring in shows like [Silicon Valley](https://en.wikipedia.org/wiki/Silicon_Valley_(season_6)#:~:text=%22Artificial%20Lack%20of%20Intelligence%22), [Black Mirror](https://en.wikipedia.org/wiki/Be_Right_Back) and [Westworld](https://en.wikipedia.org/wiki/The_Riddle_of_the_Sphinx_(Westworld)) and described with names like generative clones or ghosts in academic literature (Morris & Brubaker, ). Blog posts about finetuning LMs on personal chat histories are relatively common, but they either model timed transcipts noninteractively, or synchronous turn by turn conversations interactively (as a traditional chatbot). We are not aware of prior work that turns models of timed transcripts into interactive applications.

### 4.2 Spoken Dialogue Modeling

To go beyond manually crafted turn-taking heuristics for what is in generality an extremely complex task (Skantze, ), the main approach for generating spoken conversations has been direct audio modeling. dGSLM (Nguyen et al., ), AudioLM (Borsos et al., ), and SpiRit-LM (Nguyen et al., ) do this by modeling learned discrete tokens with autoregressive language models; the former models two streams of audio (dialogues), while the latter two model one. While the token modeling is causal, the tokenization is not, so these methods cannot yet be used in a streaming fashion.

Discrete audio tokenization is generally performed at a fixed rate of $\sim$40-50 tok/s for a single audio stream, vs. $\sim$20 tok/s for our approach supporting arbitrary numbers of speakers.(^5^55With that said, as with sparse vs. dense approaches generally, under extreme load the bandwidth required for sparse indexing over time may be higher than dense tokenization without indexing. And because our approach is sparse over time, it is more difficult to batch and has inconsistent load, which may be disadvantageous for bulk serving.) This fits into the general pattern of cascaded vs. end-to-end models: cascaded models are generally more performant/require less data and therefore can be developed sooner using fewer resources, but they are eventually superseded by end-to-end models which can provide the optimal quality given sufficient resources.

Though not exactly dialogue, simultaneous translation often operates through a cascade of ASR and TTS, though timing information (besides the relative ordering of words in the source and target streams) is stripped away (Ren et al., ; Ma et al., ).

### 4.3 Time-Aware Language Models

There are many works that make language models aware of time in one sense or another. Even without special effort, language models learn latent representations of time to the extent that it helps explain the training distribution (Gurnee & Tegmark, ). The language model CTRL (Keskar et al., ) is conditioned on metadata about each document, which may include the publication date. Whisper (Radford et al., ) and some other speech-to-text models predict timestamps as text. Park et al. () lets loose generative agents in a virtual town environment, where they act on schedules in accordance with the virtual time. Language models have been used as the backbone for time series forecasting, whether pretrained (Das et al., ), finetuned (Jin et al., ), or zero-shot (Gruver et al., ), though here time is usually dense (proceeds at a fixed rate). We are not aware of works that model timestamps as text and interpret those timestamps as an input/output stream with respect to the real-world time.

## 5 Conclusion

In this paper, we presented a simple yet general method for simulating real-time interactive conversations using pretrained language models—modeling timed diarized transcripts and decoding with causal rejection sampling—situated in two use cases: instant messenger dialogues and spoken conversations. It is easy to imagine extensions such as multiple simultaneous conversations with one simulated individual (by adding conversation ids in addition to speaker ids) or modeling multimodal conversations (images, actions, etc.), though this may require more capable language models. While we demonstrated the promise of this method using interactive conversations, it can be applied to turn language models into interactive models for any kind of event sequence, i.e., sparse-over-time world models. We hope that this method will facilitate more flexible interaction with the underlying capabilities of language models and enable new applications in fields such as gaming and entertainment.

## Ethical Considerations

While work improving the ability to simulate real-time interactive conversations can make language models more useful or delightful, it also poses risks for fraud and manipulation. In order to mitigate these risks, we limit our work to simulating natural conversations in text, a medium which is perceived as less trustworthy than audio or video. (While we simulate the timing aspects of spoken conversation, our generations are still easily distinguished from real speech due to the unnatural sound of word-level text to speech, and custom text-to-speech models would be required to move beyond this.) We provide only proofs of concept with small datasets, and do not scale up to sizes where these capabilities would become more refined. We also do not study goal-directed methods which could be used to steer a model to execute fraud.

We believe that it is valuable to expose this capability overhang so that the community can respond with appropriate measures. For example, a better understanding of the amount of data needed to impersonate someone with a generative clone could affect how much conversational data users are comfortable sharing publicly on social media, or motivate end-to-end encryption/disappearing messages to prevent private data leakage in the event of hacking. Developing interfaces for language models that are not immediately distinguishable from humans could also help to evaluate extreme risks like deception and persuasion in frontier models (Shevlane et al., ), to the extent that people react differently to communication that they perceive to be from a model vs. another person. Bad actors are already capable of sophisticated [deepfake scams](https://www.cnn.com/2024/02/04/asia/deepfake-cfo-scam-hong-kong-intl-hnk/index.html) and aren’t exactly forthcoming about their methods.

There are also ethical considerations when simulating real people or fictional characters absent ill intent, such as privacy and the effects of parasocial relationships; these tend to be general concerns that are not strictly related to real-time interactivity. See Morris & Brubaker () for an in-depth discussion of these factors. In terms of the specific datasets we used in this paper: We used our own instant messenger history with the consent and active involvement of both participants, and do not release the data/model for privacy reasons.
The U.S. Supreme Court’s oral arguments are inherently public and the conversation is in a specialized legal domain rather than anything that would encourage parasocial relationships.
We model only text transcripts and use generic text to speech voices (i.e., we do not contribute methods to impersonate any of the speakers).

## Reproducibility Statement

We publicly release the code for our case studies at \repolink. We do not release our own personal instant messenger history for reasons of privacy, but you can reproduce the instant messenger case study by bringing your own data. The data for the spoken conversation case study is public and can be reproduced.

## Acknowledgments

GA is supported by a fellowship from the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University.

LMK is supported by the Rhodes Trust.

## Appendix A Causal Rejection Sampling Algorithm with Speculative Decoding

In Algorithm 2, we present a description of causal rejection sampling including speculative decoding. For simplicity, we describe the speculative rejection sampling as if it rejects or accepts an entire event, but in implementations where events are composed of multiple tokens, the acceptance/rejection acts in finer granularity on tokens (so a prefix in a speculated event can be accepted, and only the rest has to be resampled).

Note that in order to maintain the validity of the rejection sampling, we must condition the draft distribution on $t_{i}\geq T$, because if $\hat{t}_{i}$ had been $<T$ it would have already been finalized and we would not be considering it for rejection sampling. Renormalizing this correctly in timestamps consisting of multiple tokens requires some finesse. For our instant messenger case, this is further complicated by the fact that as interruptions come in, the same timestamp in a draft message may change format. For example, after a message planned for :02;17.8, a generation may plan for :03;52.0, but after an interruption at :03;24.7, the draft must be reinterpreted to have a timestamp of ;52.0). We expect that speculation is not worth the implementation burden unless you are using a simple custom vocabulary for timestamps or pressing up against performance limits. In our experiments performance was adequate without it.

Figure: Algorithm 2 Causal rejection sampling with speculation

## Appendix B Causal Rejection Sampling Algorithm with Retconning (without Speculative Decoding)

In Algorithm 3, we present a description of causal rejection sampling with retconning, which supports streaming word-level ASR input where previous inputs may change retroactively given new context. For simplicity, we do not include speculative decoding, though it could also be applied upon retconning.

Figure: Algorithm 3 Causal rejection sampling with retconning (without speculative decoding)

## Appendix C Finetuning Details

We finetune Pythia 160M, Pythia 1.2B, Pythia 12B, Gemma 2B, and Llama 2 7B. We finetune for several epochs with learning rate $10^{-5}$ and batch size 512. We use early stopping on validation loss (computed once per epoch) with a minimum delta of 0.01 and a patience of 3.

## Appendix D In-Context Learning Details

For our ICL experiments using GPT and Claude, we use the following system prompts and decode with default sampling parameters.

Instant messenger dialogues:

Spoken conversation:

## Appendix E More Quantitative Results

For the instant messenger human ratings: offline ratings are averaged over 20 examples per model and online ratings are averaged over 5. For the spoken conversations human ratings: offline ratings are averaged over 10 examples per model and online ratings are averaged over 5.

See Figure 4 for experiments comparing the distribution of predicted timestamps to the ground truth distribution.

Figure: Figure 4: Conversations generated by fine-tuned language models exhibit realistic message timings. Top: Log-binned histogram of the delays (in seconds) between successive messages in 512 independent 1000-token conversations generated unconditionally by fine-tuned Llama 2 7B (temperature 1, top-p=0.95 (Holtzman et al., )), compared to delays in a corresponding chunk of consecutive ground-truth messages of the same size sampled at random from the same month and year as the simulated ones. Mean conversation length is 73 messages. The empirical distributions are very similar (25-bin Kullback–Leibler divergence = 0.005), attributable to nucleus sampling. Bottom: Consecutive message delays for continuations of three randomly selected message history prefixes, ground truth (dotted) vs. predicted (solid). We do not expect these to perfectly match due to irreducible entropy, but the resemblance in trajectory shows that the model is not just learning first-order statistics.

## Appendix F Qualitative Examples

See below for examples of ground truth and generated conversations, for both the instant messenger dialogue and spoken conversation case studies.

See Figure 5 for a ground truth example, Figure 6 for a Pythia 160M example, Figure 7 for a Pythia 1.4B example, Figure 8 for a Pythia 12B example, Figure 9 for a Gemma 2B example,
Figure 10 for a Llama 2 7B example,
Figure 11 for a Claude 3 Sonnet example, and Figure 12 for a GPT-4 Turbo example.

See Figure 13 for a ground truth example, Figure 14 for a Pythia 160M example, Figure 15 for a Pythia 1.4B example, Figure 16 for a Pythia 12B example, Figure 17 for a Gemma 2B example,
Figure 18 for a Llama 2 7B example,
Figure 19 for a Claude 3 Sonnet example, and Figure 20 for a GPT-4 Turbo example.

Figure: Figure 5: Ground truth instant messenger example.

Figure: Figure 6: Pythia 160M instant messenger example.

Figure: Figure 7: Pythia 1.4B instant messenger example.

Figure: Figure 8: Pythia 12B instant messenger example.

Figure: Figure 9: Gemma 2B instant messenger example.

Figure: Figure 10: Llama 2 7B instant messenger example.

Figure: Figure 11: Claude 3 Sonnet instant messenger example.

Figure: Figure 12: GPT-4 Turbo instant messenger example.

Figure: Figure 13: Ground truth spoken conversation example.

Figure: Figure 14: Pythia 160M spoken conversation example.

Figure: Figure 15: Pythia 1.4B spoken conversation example.

Figure: Figure 16: Pythia 12B spoken conversation example.

Figure: Figure 17: Gemma 2B spoken conversation example.

Figure: Figure 18: Llama 2 7B spoken conversation example.

Figure: Figure 19: Claude 3 Sonnet spoken conversation example.

Figure: Figure 20: GPT-4 Turbo spoken conversation example.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: See weaknesses above
soundness: 3
strengths: The paper deals with interactive online generation of dialogue, in both message-by-message and word-by-word scenarios; this is a really important task from the point of view of building genuinely interactive conversational agents, and one that really hasn't seen much attention recently. The approach taken is quite intuitive to understand and is explained quite clearly. The evaluation shows that the approach can be effective, not only in generating coherent dialogue turns but including some important dialogue phenomena such as self-repair and turn-taking phrases (as shown by the examples in the appendix).
summary: The paper describes a method for applying a standard LLM to the task of generating dialogues incrementally, either turn-by-turn or word-by-word, by considering an output as a triple of (utterance time, speaker, message) and predicting on that basis. Experiments are given with turn-by-turn text-based instant messaging, and word-by-word text-based dialogue (starting with spoken dialogue, but processing with ASR to a text transcript and then experimenting on that). Experiments show that some plausible outputs can be generated, that generally bigger LLMs outperform smaller ones, and that the better models can receive good ratings from human evaluators.
weaknesses: There is a fairly large amount of pre-LLM work on the details of incremental word-by-word dialogue modelling, including agents that can listen and generate on a word-by-word basis: the paper cites Skantze (2021)'s review of turn-taking treatment but would be stronger if it gave more comparison to work in that area.

The evaluation gives comparisons between LLMs on general metrics, but could be stronger if it looked at more details of coherence, turn-taking etc. It would be really nice if there was some discussion (quantitative or qualitative) of the linguistic phenomena that do/don't seem to be captured by this approach: the paper mentions coherence and consistency, but word-by-word and turn-by-turn modelling mean that other issues become important, e.g. realistic turn-taking patterns, interruption and overlap. The appendix transcripts show that some cases display good examples of some of these, some less so.

The evaluation uses data with a fairly low level of interactivity compared to many dialogue datasets. Instant messaging is an asynchronous, turn-by-turn medium; the spoken dialogues must be treated more incrementally, and are modelled word-by-word, but come from court transcripts in which the level of formality is high, and thus speaker changes, overlaps, interruptions etc are likely to be rare compared to more everyday, informal conversation. It would be helpful to know more about the average turn length, turn duration, number of speaker changes etc in this data compared to some of the more standard conversational datasets used in dialogue system development.

The model is one of generating a transcript as a whole, including all participants, whereas a practical conversational agent would have to react to other agents' contributions, online, and continue to adapt as their contributions come in (possibly interrupting, overlapping, leading to conversational directions that are not in the agent's interest, etc) - so some discussion of what would be involved in fitting this approach into those kind of constraints would be very helpful. 

Relatedly, the model's basic assumption that p(e|context) can be decomposed into p(t|context) x p(s|context,t) x [...] seems to mean that it's predicting an utterance event time, and then predicting the speaker of that utterance event. A more realistic agent might predict speaker (or know that it wants to speak) first, and then need to predict when to speak - would that fit with this approach?

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. What considerations were made regarding the potential for user fatigue in interactions involving high rejection rates in real-time conversations?
2. Have you considered building an end-to-end model? Integrating ASR and TTS into the LLM as well to achieve faster response times?
soundness: 3
strengths: 1. Introduces a feasible approach for real-time interaction modeling using timed diarized transcripts and causal rejection sampling, which can be integrated into standard pre-trained models.
2. Demonstrates applicability with two distinct real-world cases: asynchronous instant messaging and real-time spoken conversations, adding diversity to potential model interactions.
summary: The paper presents a method for simulating real-time interactive conversations by modeling diarized, timed transcripts, combined with causal rejection sampling. This method enables pre-trained, text-only language models to handle asynchronous and synchronous dialogues, as demonstrated with case studies involving instant messaging and spoken conversation simulations. The approach aims to maintain interactivity with minimal hardware requirements, while retaining a natural flow of dialogue.
weaknesses: 1. There is little comparison with already existing work in this area (i.e. https://arxiv.org/abs/2405.19487) 
2. The model is trained on narrow datasets (instant messaging and court transcripts), raising doubts about its generalization to diverse, real-world conversational scenarios.
3. Reliance on ASR and TTS systems may introduce errors and disrupt natural flow in spoken dialogues. An end-to-end audio model could reduce these issues by handling speech inputs and outputs directly.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Q1: Could you clarify what is meant by the "first 95%"? Is this based on chronological order? If so, wouldn’t this temporal split risk introducing shifts in topics or conversational styles over time, potentially affecting the robustness and validity of the evaluation?
> We use the first 95% of the messages as the train set, the next 2.5% as a validation set, and the last 2.5% as a test set.

Q2: In Figure 4, timing is evaluated based on the delays between successive messages. Could you clarify why only delays are measured rather than both delays and any potential early responses? Additionally, how is the alignment between generated and ground-truth messages determined? If the generated messages differ significantly in content from the ground-truth, what is the meaning or value of comparing the timing between two sets of messages that may not correspond closely in their content?

Q3: Why were different evaluation metrics used for human assessments in the instant messaging and spoken conversation settings? Could you explain the reasoning behind this choice and how it aligns with the specific goals of each domain?
soundness: 2
strengths: - Adapting text-only LLMs to model real-time interactive conversations is an important yet underexplored problem with the potential to unlock new applications without the need for costly retraining. The proposed method is straightforward but effective, demonstrated across a variety of LLMs from different model families and scales, and applicable to both fine-tuning and in-context learning.
- The paper includes both offline and online human evaluations in addition to automatic metrics, which are particularly valuable for assessing the performance of LLMs.
summary: The paper introduces a method for simulating real-time interactive conversations using pretrained text-only language models, incorporating two key modifications. First, it employs timed diarized transcripts to represent each timestamped event, with each entry consisting of a timestamp, speaker ID, and message content. The model is tasked with predicting the probability of the next event based on event history, and sampling can proceed token by token, similar to standard causal language model text generation. During inference, a technique termed causal rejection sampling enables real-time interaction by discarding and resampling responses when interrupted by the user, thus adapting to dynamic input. To improve response speed during user interruptions, the authors also introduce two enhancements: (1) accounting for model generation latency and user reaction time and (2) applying a modified speculative decoding method to reuse partially generated responses before interruption.

The method is evaluated in two domains: instant messaging and spoken conversation. For instant messaging, the authors processed a 9-year message history between the first authors, resulting in 37 million characters in the diarized transcript format. For spoken conversations, they utilized a 1000-hour subset of U.S. Supreme Court oral arguments, totaling 33 million characters. This spoken data was converted into word-level transcripts with precise timing using an ASR engine. Experiments involved both open-source LLMs (Pythia, Gemma, LLaMA) ranging from 160M to 7B parameters, and state-of-the-art proprietary LLMs. Evaluation metrics included document-level perplexity, offline human evaluations ranking generated continuations, and online human evaluations involving direct interaction with the model. Key findings indicate that (1) the method meets real-time interactivity constraints on feasible hardware (e.g., a 40GB A100 GPU for a 7B LLaMA 2 model), and (2) larger, high-quality LLMs generally perform better in real-time interactive conversation modeling.
weaknesses: - A critical aspect of real-time conversation is that the user can interrupt at any moment, requiring the model to discard its current response and handle the new input immediately. However, while the proposed method has the potential to address this, it is not thoroughly tested in the experiments. There is not even a qualitative example showing such behavior from the trained models.
- Using instant messaging dialogues to evaluate real-time interactions may not be ideal, as people’s response habits vary depending on availability, typing speed, and other factors. Modeling response timing directly may be less meaningful in this setting, and the turn-based nature of messaging lacks the fluidity and interruption dynamics of real-time conversations.
- The human evaluation lacks details, such as the number of conversations evaluated per method, the evaluation rubric, and consistency across ratings. It is unclear how the conversations are compared or if a consistent standard was applied, which is particularly important given the challenge of evaluating long texts.
- The control tokens introduce significant overhead in the spoken conversation domain without an optimized tokenizer that treats numbers from 0 to 999 as single tokens. Since existing models may not support such tokens, this limits the method’s practicality when used with standard tokenizers.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: NA
soundness: 2
strengths: The proposed dialogue system supports multi-speaker, simultaneous conversations, marking a significant advancement over traditional turn-by-turn dialogue models. Additionally, the method’s design allows it to be easily integrated into various models, demonstrating its versatility and broad applicability.
summary: This paper presents a novel approach for achieving real-time response in dialogue systems by modeling timed, diarized transcripts and decoding through causal rejection sampling. The model is tested on both instant messenger dialogues and spoken conversations, with results demonstrating its ability to sustain real-time interactivity.
weaknesses: 1. Additional Evaluation Metrics to Consider
- While the model demonstrates high generation bandwidth, how does your approach ensure the model engages at the right time rather than speaking continuously? In other words, how does your method balance high generation rates with appropriate turn-taking? 
- An ablation study comparing models without rejection sampling and those fine-tuned for next-turn speaker and response prediction could help validate the model design.
2. Positioning and Contribution

This paper proposes a system design for real-time conversation. However, its novelty from a learning and evaluation perspective isn’t fully highlighted. I’d be open to discussing potential ways to position this paper’s contributions for ICLR.

Meta-review: This paper presents an interesting method for creating transcripts, however reviewers felt that the work was not well positioned with respect to other existing work in the area (references have been given by individual reviewers) and that the method was only tested on very narrow datasets: conversations between the two first authors and a subset of court cases of the US Supreme Court.  The evaluation was seen as lacking in detail, described as: "the number of conversations evaluated per method, the evaluation rubric, and consistency across ratings."  There were also questions raised as to whether or not the method could handle interruptions in a timely manner.

Reviewers think that the paper has promise but that a re-write with better positioning and more extensive evaluation are needed (including evaluating how the system performs under interruption).

Decision: Reject