Venue: NAACL 2021

# When does text prediction benefit from additional context?

An exploration of contextual signals for chat and email messages

 Stojan Trajanovski

Microsoft &Chad Atalla

Microsoft &Kunho Kim

Microsoft

Vipul Agarwal

Microsoft &Milad Shokouhi

Microsoft &Chris Quirk

Microsoft

###### Abstract

Email and chat communication tools are increasingly important for completing daily tasks. Accurate real-time phrase completion can save time and bolster productivity. Modern text prediction algorithms are based on large language models which typically rely on the prior words in a message to predict a completion. We examine how additional contextual signals (from previous messages, time, and subject) affect the performance of a commercial text prediction model. We compare contextual text prediction in chat and email messages from two of the largest commercial platforms Microsoft Teams and Outlook, finding that contextual signals contribute to performance differently between these scenarios. On emails, time context is most beneficial with small relative gains of 2% over baseline. Whereas, in chat scenarios, using a tailored set of previous messages as context yields relative improvements over the baseline between 9.3% and 18.6% across various critical service-oriented text prediction metrics.

+
Footnote †: _Proceedings of NAACL HLT 2021: IndustryTrack Papers_, pages 1–9 June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Email and chat communication tools are increasingly important for completing daily professional and personal tasks. Given the recent pandemic and shift to remote work, this usage has surged. The number of daily active users in Microsoft Teams, the largest business communication and chat platform, has increased from 20 million (2019, pre-pandemic) to more than 115 million in October (2020). On the other hand, email continues to be the crucial driver for formal communication showing ever increasing usage. Providing real-time suggestions for word or phrase auto-completions is known as text prediction. The efficiency of these communications is enhanced by suggesting highly accurate text predictions with low latency. Text prediction services have been deployed across popular communication tools and platforms such as [MierosoftTextPredictions, 2020] or GMail Smart Compose (Chen et al., 2019).

Modern text prediction algorithms are based on large language models and generally rely on the prefix of a message (characters typed until cursor position) to create predictions. We study to what extent _additional contextual signals_ improve text predictions in chat and email messages in two of the largest commercial communication platforms: Microsoft Teams and Outlook. Our contributions are summarized as:

* We demonstrate that prior-message context provides the greatest lift in the Teams (chat) scenario. A 5 minute window of prior messages from both senders works the best, with relative gains from 9.3% up to 18.6% across key metrics (total match and estimated characters accepted). This 5 minute window of prior messages from both senders outperforms the corresponding 2 and 10 minute scenarios.
* We find that context about message composition time provides the largest gains for the Outlook (email) scenario, while adding the subject as context only marginally helps. These relative gains are moderate (2-3% across various metrics).
* We conclude that the different characteristics of chat and email messages impede domain transfer. The best contextual text prediction models are custom trained for each scenario, using the most impactful subset of contextual signals.

The remainder of the paper is organized as follows. We give an overview of state-of-the-art related research in Section 2. More details on the signals used for contextualization are provided in Section 3. Section 4 provides information on the language model, performance metrics, and statistical details

about the data. Experiment results and comparisons are presented in Section 5. We conclude in Section 6. Ethical considerations on the data and processes are discussed in Section 7.

## 2 Related work

Text prediction services have been applied for various applications, including text editor Darragh et al. (1990), query autocompletion on search engine Bast and Weber (2006); Bar-Yossef and Kraus (2011), mobile virtual keyboard Hard et al. (2018). Recently prediction service is applied on communication tools for composing email and chat messages to improve user writing productivity Kannan et al. (2016); Deb et al. (2019); Chen et al. (2019); Microsoft Text Predictions (2020).

To predict correct text continuation, such applications leverage efficient lookups with pre-generated candidates, using most popular candidates (MPC) Bar-Yossef and Kraus (2011), or using large-scale language models Bengio et al. (2003). State-of-the-art language models Jozefowicz et al. (2016); Mnih and Hinton (2009); Melis et al. (2018) rely on the most recent deep learning architectures, including large LSTMs Hochreiter and Schmidhuber (1997) or transformers Vaswani et al. (2017), while prior approaches involve n-gram modeling Kneser and Ney (1995); James (2000); Bickel et al. (2005).

In this work, we focus on the application of text prediction on production-level online communication tools, to help users compose emails Chen et al. (2019); Microsoft Text Predictions (2020), and in addition chat messages. In particular, we focus on examining useful contextual signals to give more accurate predicted text, using time, subject, and prior messages. Various contextualization techniques (e.g., hierarchical RNNs) have been applied to add useful additional signals such as preceding web interaction, linking pages, similar search queries or visitor interests of a page White et al. (2009); previous sequence of utterances Park et al. (2018); Zhang et al. (2018); Yoo et al. (2020) or related text snippets Ke et al. (2018).

## 3 Contextualization concepts

We examine several signals accompanying the main message text: **message compose time**, **subject**, and **previous messages**. We combine these signals with the message body into a single "contextualized" string, using _special tokens_ to separate signals, as shown in Figure 0(a). This approach is inspired by Chen et al. (2019), as they showed that concatenating contextual signals into a single input string gave a comparable result to more complex methods encoding these signals separately1. The remainder of this section explains details about each contextual signal we use.

Footnote 1: They also use subject and previous email as contexts.

TimeComposition time is a contextual signal which can provide added value for text prediction, enabling suggestions with relevant date-time words, like "weekend", "tonight". We encode local date and time, as shown in Figure 0(a), and use \(<\)BOT\(>\) and \(<\)EOT\(>\) to separate from other signals.

SubjectMessage subjects often contain the purpose or summarized information of a message. In the email scenario, we use _subject_ as context. In the chat scenario, subject is not available, so we use the _chat window name_ as a proxy for subject (can be auto-generated or manually set by users). In both cases, the subject context is wrapped with \(<\)BOU\(>\) and \(<\)EOU\(>\) special tokens.

Previous email messagesPrevious messages can provide valuable background information which influences the text of the current message being composed. In the email case, we create pairs of messages and replies. These pairs are concatenated with a \(<\)COT\(>\) special token to create a single contextual string. In cases where the email was the first in a thread, the prior email context is left blank.

Previous chat messagesPrior message contextualization for chat scenario is much more complex. Chat conversations typically consist of many small messages sent in quick succession. Given the email and chat message length statistics in Section 4, we expect chat messages to be about 10\(\times\) smaller than emails. So, we limit chat histories to 20 messages, which is roughly equivalent to an email-reply pair in length. Among these prior messages, any number and any order could be from the current sender, or the other participant.

We segment chat histories by _message blocks_ and _time windows_. A series of uninterrupted messages sent by one sender is considered as a single _message block_. Messages sent within the past \(N\) minutes are within a _time window_, which enforces recency as a proxy for relevance.

We define three prior message context aggregation modes in the chat scenario (visualized in 

Figure 0(b)), mimicking prior email context:

1. [label=()]
2. _Ignore-Blocks_: chat messages from the _current sender_, in the past \(N\) minutes, ignoring any message block boundaries.
3. _Respect-Blocks_: chat messages from the _current sender_, in the past \(N\) minutes, confined to the _most recent message block_.
4. _Both-Senders_: chat messages from _both senders_, in the past \(N\) minutes. When the sender turn changes, strings are separated by a space or a special token <COT>.

For each mode, we consider time windows of \(N=\{2,5,10\}\) minutes.

## 4 Data and Language Model

### Data

Our model training depends on real messages from two of the largest commercial communication platforms Microsoft Teams and Outlook; this involves a multi-pronged system for ensuring our customers' privacy. We work within rigorous privacy rules, using tools with privacy features built in, and preprocessing all data through multiple privacy precautions before it is digested by our models. User data from our communication platforms is never visible to humans for analysis, in any raw or preprocessed format. We run this data through our pipelines and are only able to view resulting text prediction metrics. Section 7 contains more details about these privacy precautions.

Chat messagesWe sample Teams data from more than 3.8 billion curated one-on-one chat messages that span 6 months (say May - October 2020), followed by privacy precautions and noise filters.

The data is sorted by time and split into train, validation, and test sets in non-overlapping time periods. We use over 90% of the data for training, holding out 75,000 samples for validation and 25,000 samples for testing. Each message is recorded in its respective dataset along with all associated context. In a statistical analysis of the chat message lengths (see Figure 2, blue box) we find that mean tokens number is 9.15 (length in characters is 48), while median tokens number is 6 (with character length 31). Therefore, when iterating character-by-character through the messages, as done in inference for text predictions, the test set has over 1M evaluation points (resampled periodically, see Section 7.1).

Email messagesIn email experiments, we use approximately 150 million Outlook commercial emails from a period of 6 months, which go through the same privacy precautions mentioned above and in Section 7. The emails are then sorted, filtered for noise, and cut into train, validation, and test sets by their date ranges. A statistical analysis of email lengths (see Figure 2, green box) reveals that mean number of tokens is 94 (with length in char

Figure 1: Examples of (a) context encoding pipeline and (b) chat prior message aggregation modes.

Figure 2: Box-plot statistics: number of tokens in a context-aggregated message from Microsoft Teams and Outlook. Green diamond markers represent the mean, bold red lines are the medians, margins of the boxes are lower and upper quartiles while whiskers end-points are the minimums and maximums.

 

acters being 561), while the median is 53 tokens (and 316 characters). This is roughly 10\(\times\) larger than chat messages. When splitting train, test, and validation sets, over 90% of the data is allocated to the training set. The test set is subsampled to 3,000 emails (unlike the 25,000 messages for the chat test set) since this roughly leads to final contextualized datasets of the same size. Each resulting test set contains just over 1 million evaluation points, as in the chat setting.

Additionally, we use the Avocado dataset as a publicly available dataset, which consists of emails from 279 accounts of a defunct IT company referred to as "Avocado" see details in Oard et al. (2015), for debugging and validation, allowing us to directly view data and outputs. This dataset is split into validation and test sets, each with roughly 3,000 emails for evaluation.

### Prior-message aggregation statistics

When applying the chat-specific prior-message grouping modes defined in Section 3, the number of prior messages fetched as context varies. Table 1 presents details on how many messages the different aggregation modes end up grouping. Both single-sender modes introduce smaller volumes of context than the _Both-Senders_ mode. For example, the amount of prior messages grouped in the 5 minutes _Ignore-Blocks_ mode is similar to the 2 minutes _Both-Senders_ mode; where 2.5 chat messages are combined on average, and 56-59% of chat messages have at least one message as context. For emails, only around 50% have prior email context.

The number of tokens per contextualized message (including current and aggregated prior messages) varies between the email scenario and various aggregation modes in the chat scenario. Figure 2 provides statistics on these aggregated message lengths. In the chat case, the _Both-Senders_ mode with a 10 minute time window results in the largest aggregate length, with a median around 27 tokens, and mean above 40 tokens. The _Respect-Blocks_ mode does not show significant length increases as the time window grows, due to the message block boundary limits. For emails, the median total tokens remains similar regardless of including the previous message. This is because half of emails are not part of an email-reply pair.

### Language model

Once the message data is preprocessed and jointly encoded with contextual signals, it is passed as an input to the Language Model. The production system uses a two-layer (550, 550) LSTM (with 6000 sampled softmax size loss) which is optimized to maximize the Estimated Character's Accepted metric (described in Section 5.1). All contextualization experiments use the production model architecture as the baseline. Both baseline and contextual models are trained on 16 GPUs.

We have conducted experiments with more complex language models (e.g., transformers, deeper LSTMs), but we use the production model in this paper as (i) its simpler architecture enables large-scale low-latency text prediction serving and (ii) the goal of this work is to explore how different contextual signals add to the baseline performance.

## 5 Experiments and results

We conduct experiments for both email and chat messages with individual contextual signals (time, subject, prior messages) and combinations of those.

### Performance Metrics

In all experiments, we level the **S**uggestion **R**ate (SR) (number of suggestions per message), then evaluate model variant performance against the following text prediction metrics:

* **MR**: **M**atch **R**ate is the ratio of the number of matched suggestions and the total number of generated suggestions.
* **ChM / sugg**: **C**haracter's **M**atched per **sugg**gestion is the average number of matched characters per given suggestion
* **Est. ChS / sugg**: **E**stimated **C**haracter's **S**aved per **sugg**estion is the estimated number of characters that the user is saved from typing, per suggestion. (Based on observed acceptance probabilities from real users.)

\begin{table}
\begin{tabular}{l c r} \hline  & \multicolumn{1}{c}{**\% msgs**} & \multicolumn{1}{c}{**mean mssg**} \\
**Configuration** & **with context** & **as context** \\ \hline
2 min Respect-Blocks & 30.76\% & 1.44 \\
5 min Respect-Blocks & 34.19\% & 1.54 \\
10 min Respect-Blocks & 35.54\% & 1.59 \\ \hline
2 min Ignore-Blocks & 43.31\% & 1.76 \\
5 min Ignore-Blocks & 55.94\% & 2.51 \\
10 min Ignore-Blocks & 63.24\% & 3.23 \\ \hline
2 min Both-Senders & 58.90\% & 2.51 \\
5 min Both-Senders & 70.20\% & 3.99 \\
10 min Both-Senders & 76.10\% & 5.40 \\ \hline \end{tabular}
\end{table}
Table 1: Microsoft Teams chat message statistics -amount of aggregated context per message.

 

* **TM**: \({\bf T}\)otal \({\bf M}\)atches is the number of suggestions which match the upcoming text.
* **ChM**: \({\bf Characters}~{}{\bf Matched}\) is the number of matched characters from all suggestions.
* **Est. ChA**: \({\bf Estimated}~{}{\bf Characters}~{}{\bf Accepted}~{}{\bf is the estimated2 total number of suggested characters accepted by users.

Footnote 2: Based on observed acceptance probabilities on large-scale production traffic, users tend to accept longer suggestions.

### Experiments with chat messages

The performance results for chat messages from Microsoft Teams compared to the non-contextual baseline model are shown in Table 2. For comparability, we train the model's confidence threshold to level each model's suggestion rate (SR) at 0.5 suggestions / message.

Contextualization with just the chat window name (subject) yields moderate gains, possibly because the typically short chat messages are so sparse on context that a chat topic name, or participant names from a chat header, provides a starting foothold for relevance. In contrast, from the last table rows, we see that the benefits from subject context diminish once prior messages are used as a context, suggesting that the subject proxy is much weaker than prior message context. Table 2 also shows that compose-time can act as a confounding context signal for chat messages, especially in experiments with no prior messages as a context. This is possibly due to the numerically-heavy time encoding confusing the model in contrast to the short text of chat messages. The experiments also show that the benefits of these contextual signals are not additive.

All three prior message aggregation modes (_Ignore-Blocks_, _Respect-Blocks_, and _Both-Senders_) show gains across all performance metrics, with all time window sizes. _Both-Senders_ mode achieves the most significant relative gains: above 9.3% for Match Rate and the Total Matches; more than 11.7% for the character match and character match per suggestion; and more than 18.6% for the characters saved per suggestion and character acceptance. This indicates that messages from the other sender provide significant value, when used with a well-tuned time window. It provides relevant conversation context from all senders, eliminating confusing gaps between messages, and enables suggestions in response to questions posed by the other sender. In particular, the _Ignore-Blocks_ mode does worse than _Both-Senders_, since _Ignore-Blocks_ can violate conversation continuity, including messages \([k,k+2]\) from the current sender, and skipping message \(k+1\) from the other sender.

For the single-sender modes, _Respect-Blocks_ generally performs slightly worse as it utilizes only part of the messages taken by the _Ignore-Blocks_ mode. This indicates that seeing a longer prefix of the current message block (more similar to writing a long email) makes an impact on text prediction in chat messages. Lastly, we observe that a 5 minute time window works better than 2 and 10 minute time windows. Shorter time windows seem to miss important prior context while a larger windows lead

\begin{table}
\begin{tabular}{l c c c c c c} \hline
**Configuration / context mode** & **MR** & **ChM / suggg** & **Est. ChS / suggg** & **TM** & **ChM** & **Est. ChA** \\ \hline Chat name & +5.38\(\uparrow\) & +6.05\(\%\) & +7.83\(\%\)\(\uparrow\) & +5.22\(\uparrow\) & +5.99\(\%\)\(\uparrow\) & +7.86\(\%\)\(\uparrow\) \\ Time & -3.49\(\downarrow\) & -4.28\(\downarrow\) & -6.33\(\downarrow\) & -3.48\(\downarrow\) & -4.25\(\downarrow\) & -6.36\(\downarrow\) \\ Time+Chat name & -13.98\(\downarrow\) & -14.72\(\downarrow\) & -16.57\(\downarrow\) & -13.96\(\uparrow\) & -14.67\(\downarrow\) & -16.57\(\downarrow\) \\ \hline
2 min Respect-Blocks & +5.91\(\uparrow\) & +7.65\(\%\) & +12.65\(\%\) & +5.95\(\%\) & +7.72\(\uparrow\) & +12.62\(\%\) \\
2 min Ignore-Blocks & +5.91\(\%\)\(\uparrow\) & +7.68\(\%\) & +12.95\(\%\) & +5.78\(\%\) & +7.62\(\uparrow\) & +12.84\(\%\)\(\uparrow\) \\
2 min Both-Senders & +5.91\(\%\)\(\uparrow\) & +8.77\(\%\) & +16.87\(\uparrow\) & +6.01\(\uparrow\) & +8.77\(\%\) & † & +17.01\(\%\)\(\uparrow\) \\ \hline
5 min Respect-Blocks & +5.65\(\%\)\(\uparrow\) & +7.79\(\%\) & † & +13.86\(\%\) & † & +5.56\(\%\)\(\uparrow\) & +7.74\(\%\)† & † & +13.72\(\%\)† \\
5 min Ignore-Blocks & +6.72\(\%\)\(\uparrow\) & +9.01\(\%\) & † & +15.66\(\%\)\(\uparrow\) & † & +6.59\(\%\)† & † & +15.48\(\%\)† \\
**5 min Both-Senders** & **+9.41\(\%\)\(\uparrow\)** & **+11.76\(\%\)\(\uparrow\)** & **+18.67\(\%\)** & **+9.30\(\%\)** & † & **+11.72\(\%\)� & † & **+18.67\(\%\)†** \\ \hline
10 min Respect-Blocks & +5.65\(\%\)\(\uparrow\) & +7.79\(\%\)† & † & +13.55\(\%\) & † & +6.63\(\%\)† & † & +13.53\(\%\)† \\
10 min Ignore-Blocks & +6.99\(\%\) & † & +9.32\(\%\) & † & +15.66\(\%\) & † & +9.24\(\%\)� & † & +15.56\(\%\)† \\
10 min Both-Senders & +8.06\(\%\)\(\uparrow\) & +10.57\(\%\) & † & +17.77\(\%\) & † & +7.86\(\%\) & † & +10.51\(\%\)† & † & +17.55\(\%\)† \\ \hline Time+5 min Respect-Blocks & +3.76\(\%\)\(\uparrow\) & +5.51\(\%\) & † & +10.24\(\uparrow\) & † & +3.84\(\uparrow\) & † & +5.51\(\%\)� & +10.22\(\%\)† \\ Chat name+5 min Respect-Blocks & +5.11\(\%\)\(\uparrow\) & +6.36\(\%\) & † & +9.34\(\%\) & † & +5.13\(\%\)� & † & +6.35\(\%\)� & † & +9.40\(\%\)† \\ \hline Time+Chat name+5 min Respect-Blocks & +5.38\(\%\)\(\uparrow\) & +7.79\(\%\)† & † & +14.16\(\%\) & † & +5.43\(\%\)† & † & +7.82\(\%\)� & † & +14.26\(\%\)† \\ Time+Chat name+5 min Ignore-Blocks & +5.11\(\%\)\(\uparrow\) & +6.97\(\%\) & † & +12.05\(\%\) & † & +5.15\(\%\)� & † & +6.99\(\%\)� & † & +12.00\(\%\)† \\ Time+Chat name+5 min Both-Senders & +8.87\(\%\)\(\uparrow\) & +11.53\(\%\) & † & +18.37\(\%\) & † & +8.91\(\%\)� & † & +11.52\(\%\)� & † & +18.36\(\%\)† \\ \hline \end{tabular}
\end{table}
Table 2: Microsoft Teams chat messages experiment results with various contextualization modes. First column is the experiment configuration, other columns are **relative gains**, over the noncontextual baseline, of the performance metrics (Section 5.1) with a leveled suggestion rate of 0.5.

 

to over-saturation of irrelevant information.

### Experiments with email messages

The gains from the contextualization in email messages are more moderate compared to those from chat messages. The comparison of the contextualized models with the baseline on commercial Microsoft Outlook emails and Avocado dataset are given in Table 3 and 4 respectively. For emails, the results suggest that time as a context (or time+subject in the Avocado dataset) offers most promising relative gains of 2-3%. This contrasts the observed trend from chat messages. Time is more important for emails since emails are often longer, contain greetings, farewells, and meeting requests with time-related keywords (e.g., "tomorrow", "last night", "after the weekend"). Additionally, numerical tokens from the time context are less likely to outnumber the message content tokens, since emails are about 10\(\times\)longer than chat messages.

With the chosen architecture, neither subject nor prior message context signals provide value in the email scenario. Subjects may introduce keywords, but the implemented method of encoding context and body into a single string did not demonstrate an ability to pull out those key words for suggestions. Likewise, prior message context did not benefit the email scenario. As Figure 2 shows, emails with prior messages are significantly longer than any of the chat context aggregations. Prior emails may have critical information steering the direction of an email thread, but our production-oriented metric are not significantly affected. The implemented architecture may not be strong enough to isolate and make use of those cues, instead becoming confounded by the vast influx of tokens from another sender. This emphasizes that the email and chat scenarios require different context signals, and may benefit from different underlying architectures.

Qualitative analysis with the Avocado setGiven our commercial data-visibility constraints due to the privacy considerations, we perform a qualitative analysis on the public Avocado dataset Oard et al. (2015). Using this public data, we evaluate text predictions from one of the promising email context modes: time context. As shown in Table 5, we use diff tools to identify when the time context model and baseline model create (i) correct suggestions, (ii) wrong suggestions, and (iii) no suggestions. We see that the time context model improves on all three columns. When directly examining cases where the time-context model renders a new correct suggestion, compared to the baseline, we observe a trend of time-related n-grams. Words like "tomorrow", "available", "September" are seen more frequently in correct suggestions (see Figure 3). The same trend is also observed in the Time+Subject model.

## 6 Conclusions

We study the role of context in text prediction for chat and email platforms. Testing with previous messages, subject, time as additional contextual

\begin{table}
\begin{tabular}{l c c c c c c} \hline
**Configuration / context mode** & **MR** & **ChM / sugg** & **Est. ChS / sugg** & **TM** & **ChM** & **Est. Cha** \\ \hline Subject & -1.46\%\(\downarrow\) & -0.21\(\downarrow\)\(\downarrow\) & +1.77\%\(\uparrow\) & -1.58\%\(\downarrow\) & -0.22\%\(\downarrow\) & +1.80\%\(\uparrow\) \\
**Time** & +0.24\%\(\uparrow\) & +1.59\%\(\uparrow\) & **+4.87\%\(\uparrow\)** & +0.20\%\(\uparrow\) & +1.55\%\(\uparrow\) & **+4.75\%\(\uparrow\)** \\ Previous Email & -3.89\%\(\downarrow\) & -3.50\%\(\downarrow\) & -2.43\%\(\downarrow\) & -3.85\%\(\downarrow\) & -3.42\%\(\downarrow\) & -2.43\%\(\downarrow\) \\
**Time+Subject** & **+1.70\%\(\uparrow\)** & **+2.32\%\(\uparrow\)** & +3.32\%\(\uparrow\) & **+1.75\%\(\uparrow\)** & **+2.34\%\(\uparrow\)** & +3.41\%\(\uparrow\) \\ \hline \end{tabular}
\end{table}
Table 4: Avocado test set Oard et al. (2015) messages experiment results for various contextualization modes. First column is experiment configuration, other columns are **relative gains**, over the noncontextual baseline, of performance metrics (Section 5.1) with a leveled suggestion rate of 2.5.

\begin{table}
\begin{tabular}{l c c c c c c} \hline
**Configuration / context mode** & **MR** & **ChM / sugg** & **Est. ChS / sugg** & **TM** & **ChM** & **Est. Cha** \\ \hline Subject & -0.81\%\(\downarrow\) & -0.36\%\(\downarrow\) & +0.76\%\(\uparrow\) & -0.74\%\(\downarrow\) & -0.35\%\(\downarrow\) & +0.76\%\(\uparrow\) \\
**Time** & **+2.02\%\(\uparrow\)** & **+2.25\%\(\uparrow\)** & **+2.88\%\(\uparrow\)** & **+2.01\%\(\uparrow\)** & **+2.22\%\(\uparrow\)** & **+2.83\%\(\uparrow\)** \\ Previous Email & -9.72\%\(\downarrow\) & -10.56\%\(\downarrow\) & -13.05\%\(\downarrow\) & -9.65\%\(\downarrow\) & -10.56\%\(\downarrow\) & -13.12\%\(\downarrow\) \\ Time+Subject & +0.20\%\(\uparrow\) & +0.47\%\(\uparrow\) & +1.06\%\(\uparrow\) & +0.23\%\(\uparrow\) & +0.49\%\(\uparrow\) & +1.11\%\(\uparrow\) \\ \hline \end{tabular}
\end{table}
Table 3: Microsoft Outlook email messages experiment results with various contextualization modes. First column is experiment configuration, other columns are **relative gains**, over the noncontextual baseline, of the performance metrics (Section 5.1) with a leveled suggestion rate of 3.8.

\begin{table}
\begin{tabular}{c c c c} \hline  & \multicolumn{3}{c}{**Time as context / Baseline in Avocado test set**} \\
**cases** & **correct / w wrong** & **correct / no sugg** & **no sugg / wrong** \\ \hline
**context win** & **256** & **1494** & **2825** \\
**context loss** & 239 & 1400 & 2553 \\ \hline \end{tabular}
\end{table}
Table 5: Comparing text predictions of time-context model vs baselines. ”Context win” row holds counts of cases where contextual model suggestions beat baseline suggestions.

 

signals, we find that the different characteristics of emails and chat messages influence the selection of contextual signals to use. Previous message contextualization leads to significant gains for chat messages from Microsoft Teams, when using an appropriate message aggregation strategy. By using a 5 minute time window and messages from both senders, we see a 9.4% relative increase in the match rate, and an 18.6% relative gain on estimated characters accepted. Chat messages are often short and can lack context about a train of thought; previous messages can bring necessary semantics to the model to provide a correct prediction. Benefits are comparatively insignificant for subject and compose time as contextual signals in chat messages.

In the email scenario based on Microsoft Outlook, we find that time as a contextual signal yields the largest boost with a 2.02% relative increase on the match rate, while subject only helps in conjunction with time, and prior messages yields no improvement. More complex models may be needed to reap subject and prior message gains for emails, but the current architecture was chosen for large-scale serving latency.

Future work involves exploring different encodings for contextual signals, such as utilizing hierarchical RNNs Park et al. (2018); Yoo et al. (2020) to better capture context, or using more advanced architectures such as transformers or GPT-3.

## 7 Ethical Considerations

When working with sensitive data and running a service which generates text predictions for the general public, we are responsible for preserving user privacy and serving fair and inclusive suggestions.

### Privacy considerations on user data

Our service framework follows the regulatory requirements of internal company-wise standards and General Data Protection Regulation GDPR (2018) to meet the user privacy regulations and customer premises. All customer chat and email data, from Teams and Outlook, used in this work are classified as customer content, which is not visible to humans for any purpose. Only system byproduct data, which is not linkable to specific users or groups, is obtained and viewed for quantitative evaluation. This includes internal service logs or numerical metrics (shown in Section 5.1). We also regularly re-sample training and test data due to our privacy and data retention policies, preserving similar data set sizes. We strictly use only publicly available data, such as the Avocado dataset Oard et al. (2015), for debugging and visible qualitative evaluation.

### Blocklisting

In pursuit of fair, respectful, and responsible suggestions, we employ a blocklist. This blocklist step in our text prediction system consists of a large dictionary containing denigrative, offensive, controversial, sensitive, and stereotype-prone words and phrases. Text from the message body and contextual signals serves as input to the blocklist. Then, if any word or phrase from the blocklist is found in the input, all further suggestions are suppressed for the message.

In the email scenario, the full body and context is used for blocklist checks, resulting in a blocklist trigger rate of 47.42%. This means that 47.42% of our data points contain a blocklisted term in their input text, and we avoid triggering suggestions on those points. Naturally, this rate of blocklist

Figure 4: Initial blocklist trigger rates for various contextualization merging modes in Microsoft Teams chat messages.

Figure 3: The 20 most common new suggestions triggered by the time-context model, on data points from the Avocado test set Oard et al. (2015) where the baseline renders zero suggestions.

 

triggering increases as more context is added to the pool of text being checked.

This phenomenon introduces an added complexity to the chat scenario. A noncontextual baseline chat model would fail to trigger the blocklist on a response to an offensive statement from two messages ago. Figure 4 shows how the blocklist trigger rate varies as larger windows of chat history are used as context. We ensure that all chat models check the past 5 messages against the blocklist, no matter how many prior messages are used for text prediction inference. With 5 prior messages fed to the blocklist in chat conversations, the blocklist trigger rate is 25.08%, instead of 5.89% with no added prior messages.

## Acknowledgements

We would like to thank the members of Microsoft Search, Assistant and Intelligence (MSAI) group for their useful comments and suggestions.

## References

* Bar-Yossef and Kraus (2011) Ziv Bar-Yossef and Naama Kraus. 2011. Context-sensitive query auto-completion. In _Proc. of the 20th Intl. Conf. on World Wide Web (WWW)_, pages 107-116.
* Bast and Weber (2006) Holger Bast and Ingmar Weber. 2006. Type less, find more: fast autocompletion search with a succinct index. In _Proc. of the 29th Annual Intl. ACM Conf. on Research and Development in Information Retrieval (SIGIR)_, pages 364-371.
* Bengio et al. (2003) Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. _Jour. of Machine Learning Research_, 3:1137-1155.
* Bickel et al. (2005) Steffen Bickel, Peter Haider, and Tobias Scheffer. 2005. Learning to complete sentences. In _European Conf. on Machine Learning (ECML)_, pages 497-504. Springer.
* Chen et al. (2019) Mia Xu Chen, Benjamin N Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M Dai, Zhifeng Chen, et al. 2019. Gamail Smart Compose: Real-time Assisted Writing. In _Proc. of the 25th ACM SIGKDD Intl. Conf. on Knowledge Discovery & Data Mining_, pages 2287-2295.
* Darragh et al. (1990) John J. Darragh, Ian H. Witten, and Mark L. James. 1990. The reactive keyboard: A predictive typing aid. _Computer_, 23(11):41-49.
* Deb et al. (2019) Budhaditya Deb, Peter Bailey, and Milad Shokouhi. 2019. Diversifying reply suggestions using a matching-conditional variational autoencoder. In _Proc. of Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_, pages 40-47. Association for Computational Linguistics.
* Commission (2018) European Commission. 2018. EU data protection rules. [https://ec.europa.eu/info/law/law-topi](https://ec.europa.eu/info/law/law-topi) c/data-protection/eu-data-protection-rules_en. Online; accessed 6 January 2021.
* Hard et al. (2018) Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Francoise Beaufays, Sean Augenstein, Hubert Eichner, Chloe Kiddon, and Daniel Ramage. 2018. Federated learning for mobile keyboard prediction. _arXiv:1811.03604_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_, 9(8):1735-1780.
* James (2000) Frankie James. 2000. Modified kneser-ney smoothing of n-gram models. Technical report, RIACS.
* Spataro (2019) Jared Spataro. 2019. 5 attributes of successful teams. [https://www.microsoft.com/en-us/microsoft-365/blog/2019/11/19/5-attributes-successful-teams/](https://www.microsoft.com/en-us/microsoft-365/blog/2019/11/19/5-attributes-successful-teams/). Online; accessed 6 January 2021.
* Spataro (2020) Jared Spataro. 2020. Microsoft Teams reaches 115 million DAU--plus, a new daily collaboration minutes metric for Microsoft 365. [https://www.microsoft.com/en-us/microsoft-365/blog/2020/10/28/microsoft-teams-reaches-115-million-dau-plus-a-new-daily-collaborati](https://www.microsoft.com/en-us/microsoft-365/blog/2020/10/28/microsoft-teams-reaches-115-million-dau-plus-a-new-daily-collaborati) on-minutes-metric-for-microsoft-365/. Online; accessed 6 January 2021.
* Jozefowicz et al. (2016) Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. _arXiv:1602.02410_.
* Kannan et al. (2016) Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufman, Balint Miklos, Greg Corrado, Andrew Tomkins, Laszlo Lukacs, Marina Ganea, Peter Young, and Vivek Ramavajilana. 2016. Smart reply: Automated response suggestion for email. In _Proc. of the ACM SIGKDD Conf. on Knowledge Discovery and Data Mining (KDD)_, page 955-964.
* Ke et al. (2018) Nan Rosemary Ke, Konrad Zolna, Alessandro Sordoni, Zhouhan Lin, Adam Trischler, Yoshua Bengio, Joelle Pineau, Laurent Charlin, and Christopher Pal. 2018. Focused hierarchical RNNs for conditional sequence processing. In _Proc. of the 35th Intl. Conf. on Machine Learning (ICML)_, volume 80, pages 2554-2563, Stockholm, Sweden.
* Kneser and Ney (1995) Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In _Intl. Conf. on Acoustics, Speech, and Signal Processing (ICASSP)_, volume 1, pages 181-184.
* Melis et al. (2018) Gabor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. In _6th Intl. Conf. on Learning Representations (ICLR)_, Vancouver, BC, Canada 

Microsoft Text Predictions. 2020. Write faster using text predictions in Word, Outlook. https://insi der.office.com/en-us/blog/text-predi ctions-in-word-outlook. Online; accessed 7 April 2021.
* Mnih and Hinton (2009) Andriy Mnih and Geoffrey E Hinton. 2009. A scalable hierarchical distributed language model. In _Advances in Neural Information Processing Systems (NeurIPS)_, pages 1081-1088.
* Oard et al. (2015) Douglas Oard, William Webber, David A. Kirsch, and Sergey Golitsynskiy. 2015. Avocado research email collection LDC2015T03. Philadelphia: Linguistic Data Consortium.
* Park et al. (2018) Yookoon Park, Jaemin Cho, and Gunhee Kim. 2018. A hierarchical latent structure for variational conversation modeling. In _Proc. of the Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_, pages 1792-1801, New Orleans, LA, USA. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems (NeurIPS)_, pages 5998-6008.
* White et al. (2009) Ryen W. White, P. Bailey, and Liwei Chen. 2009. Predicting user interests from contextual information. In _Proc. of the 32nd Intl. ACM Conf. on Research and development in information retrieval (SIGIR)_.
* Yoo et al. (2020) Kang Min Yoo, Hanbit Lee, Franck Dernoncourt, Trung Bui, W. Chang, and Sang-goo Lee. 2020. Variational hierarchical dialog autoencoder for dialogue state tracking data augmentation. In _Proc. of the Conf. on Empirical Methods in Natural Language Processing (EMNLP)_.
* Zhang et al. (2018) Zhuosheng Zhang, Jiangtong Li, Pengfei Zhu, Hai Zhao, and Gongshen Liu. 2018. Modeling multi-turn conversation with deep utterance aggregation. In _Proc. of the 27th Intl. Conf. on Computational Linguistics (COLING)_, pages 3740-3752, Santa Fe, New Mexico, USA. Association for Computational Linguistics.