SlowGuess/NAACL_2022_OCR huggingface dataset:
Shanqing Cai* Subhashini Venugopalan* Katrin Tomanek Ajit Narayanan Meredith Ringel Morris Michael P. Brenner
Google Research {cais, vsubhashini} @google.com
Motivated by the need for accelerating text entry in augmentative and alternative communication (AAC) for people with severe motor impairments, we propose a paradigm in which phrases are abbreviated aggressively as primarily word-initial letters. Our approach is to expand the abbreviations into full-phrase options by leveraging conversation context with the power of pretrained large language models (LLMs). Through zero-shot, few-shot, and fine-tuning experiments on four public conversation datasets, we show that for replies to the initial turn of a dialog, an LLM with 64B parameters is able to accurately expand over  $70\%$  of phrases with abbreviation length up to 10, leading to an effective keystroke saving rate of up to  $77\%$  on these expansions. Including a small amount of context in the form of a single conversation turn more than doubles abbreviation expansion accuracies compared to having no context, an effect that is more pronounced for longer phrases. Additionally, the robustness of the models against typo noise can be enhanced through fine-tuning on noisy data.
The prevalent paradigm of text entry on computing devices is sequential typing of characters. Word completion and prediction can theoretically save up to  $40 - 50\%$  keystrokes when 3-5 predictions are provided (Trnka and McCoy, 2008; Fowler et al., 2015). This reduces the motor and cognitive demand of entering text, especially on devices where typing is difficult, e.g., phones. In AAC use cases such as eye-gaze keyboards for severely motor-impaired individuals, the cost per keystroke is so high that there is a desire to save as many keystrokes as possible. Gaze-typing requires the user to precisely control the direction and timing of gaze for each keystroke, resulting in an extremely low text-entry speed of 8-10 words per minute and
Figure 1: Our approach to abbreviation expansion based on an LLM with context compared to one without. The conversation context (e.g., a previous turn of conversation) along with the abbreviation of the intended phrase form the LLM's input. Sampled continuations from the model are filtered to discard those that do not match the abbreviation. Top-5 options after sorting by frequency are presented.
severely limiting real-time communication (Waller, 2019). A text-entry paradigm with substantially higher keystroke saving rate (KSR) can reduce motor demand and thereby benefit AAC usage in real-time communication.
One potential paradigm is "SMS language", a spontaneously-evolved system for saving keystrokes in which each word is abbreviated as a single letter, such as in the well-known abbreviations sg for sounds good and ttyl for talk to you later (Anjaneyulu, 2013). SMS language features a high KSR (75-80%), but is limited by its small closed set of common phrases of mostly six words or shorter. Its abbreviation scheme is not applied to longer or less frequent phrases because such abbreviations would be hard for the recipient to decipher. For example, the abbreviation iipitb is highly ambiguous and may represent many possible phrases, e.g., it is pouring in the bay and it is pretty in the backyard (see Figure 1 for more examples). Some existing AAC systems support abbreviation expansion (e.g., Tobii), but are limited by hardcoded,
Human Language Technologies, pages 1261 - 1275
closed phrase sets.
The current study is based on the insight that although decoding open-set phrases from abbreviations is hard without context due to ambiguity, providing conversational context significantly constrains the space of likely phrases as shown by the example in Fig.1 (it is playing in the backyard). Hence we propose a high-KSR abbreviation scheme that focuses on conversational scenarios. We apply this scheme to three existing dialog datasets and create datasets for abbreviation expansion (AE).
This allows us to study whether LLMs, trained on web text including conversational data, can enable AE and benefit from added context. We take a 64B parameter LLM and compare zero-shot, few-shot, and fine-tuning performance on the AE task. Additionally, we simulate typing noise to study tolerance of the approach to typos. The main contributions of our work are:
1. Demonstrating the potential of abbreviation expansion using LLMs aided by conversational context for highly-abbreviated text entry, while measuring the effects of different amounts of context and different dialog turns.
2. Describing a high-KSR abbreviation scheme, a method for simulating typing noise, and conversation datasets based on these.
3. Comparing zero-shot, few-shot, and model fine-tuning approaches for the AE task and their tolerance to typo noise.
Abbreviation expansion for text entry. Previous research on aiding text entry through AE used abbreviation schemes such as using only content words (Demasco and McCoy, 1992), discarding certain vowels and consonants (Shieber and Nelken, 2007), and flexible letter saving schemes (Pini et al., 2010; Adhikary et al., 2021; Gorman et al., 2021). Spontaneous abbreviations schemes primarily omit vowels, repeating consonants, last characters, and spaces, and lead to modest KSR (e.g.,  $25 - 40\%$  in Willis et al. 2005, and  $21\%$  in Adhikary et al. 2021.) The low KSR of such schemes can be attributed to the implicit need for a human reader to decode the phrases without significant cognitive burden. N-gram models and neural language models (LMs) have been applied to expanding abbreviations for these relatively low-KSR schemes. By using LSTM models and context, Gorman et al.
(2021) achieve a word error rate of  $1.5\%$ .adhikary et al. (2021) report a  $24.2\%$  top-5 sentence error rate decoding abbreviations using an RNN to augment an n-gram LM. Our presented approach is a step towards using automation and context to expand abbreviations at a higher KSR that is close to that of SMS language.
Large language model prompting and fine-tuning. Our approach builds on prior work on LLMs including few-shot prompting, fine-tuning, and conversation models (Raffel et al., 2019; Brown et al., 2020; Adiwardana et al., 2020; Roller et al., 2020). We focus primarily on few-shot prompting (Brown et al., 2020) and fine-tuning (Ruder, 2021). Few-shot prompting uses a text description of a task along with a small number of examples for the task in the input text in order to elicit desired task responses from an LLM. In the zero-shot scenario, no examples are provided. Prompting involves no updates to the model parameters. Model fine-tuning requires more data compared to prompting, but often leads to higher task accuracy than prompt engineering (e.g., Austin et al. 2021; Lester et al. 2021). For our AE task, data for fine-tuning can be synthesized from existing conversation datasets based on an abbreviation scheme (Sec. 3). Thus, we explore both prompting and fine-tuning and compare their performance.
Assisting text entry with context. Textual contexts have been exploited to aid email writing (Kannan et al., 2016; Chen et al., 2019). For text entry in AAC, Wisenburn and Higginbotham (2008) demonstrated that providing noun phrases from a conversation partner's speech as selection options increases text-entry speed by  $36.7\%$ .adhikary et al. (2019) concluded that with currently-attainable accuracy of ASR, partner speech can be valuable in improving language modeling for AAC text entry. Shen et al. (2022) used a fine-tuned GPT-2 model (Radford et al., 2019) to expand bags of keywords into full phrases in conversational contexts based on the ConvAI2 dataset (Dinan et al., 2020) and reported a KSR of  $77\%$  at a word error rate threshold of 0.65. Our current study differs from the previous studies in the following aspects. First, we provide an abbreviation scheme to allow greater user control over the exact phrase structure and wording. Second, we performed detailed quantitative analysis of the combined predictive power of state-of-the-art LLMs and context awareness.
Abbreviation Scheme. Our abbreviation scheme differs from previous studies in that we optimize for KSR and do not expect a human reader to be able to easily decode the abbreviations. Additionally, it offers the benefit that each given phrase is mapped to a fixed abbreviation. The detailed rules for abbreviating phrases are:
1. Each word is abbreviated as its initial letter, unless the word contains an apostrophe (i.e., contraction), in which case the word is split at the apostrophe and the initial letters from the splits are taken (e.g., can't -> ct). This prevents abbreviations that are otherwise identical but semantically opposite (e.g., can vs. can't).
2. All letters in the abbreviation are lowercase.
3. Arabic numerals in a sentence are preserved (e.g., see you at 10 o'clock -> sya10oc).
4. Sentence-final punctuation are removed. Mid-sentence punctuation and special characters (e.g., # and $) are preserved to help constrain the structure of the sentence (e.g., OK, but be quick. -> o, bbq).
3.1 Datasets for context-aware AE
We study modified versions of existing dialog datasets, which we converted for the context-aware AE task. We also describe how we simulate typos.
Datasets. Table 1 summarizes the four datasets. We use their original train/dev/test splits in our experiments. The Turk Dialogues dataset (Vertanen, 2017) consists of crowd-sourced dialogs, each of which is exactly six turns in length. The dataset has typos and grammatical errors. We manually correct these and refer to the corrected dataset as Turk Dialogues Corrected (TDC).<sup>1</sup> We use three more datasets, DailyDialog (Li et al., 2017), a dataset of everyday conversations; the Cornell Movie Dialogues (CMD) (Danescu-Niculescu-Mizil and Lee, 2011) based on movie scripts, and the Turk AAC dataset (TAC) (Vertanen and Kristensson, 2011). For evaluation on out-of-domain dialogs, we use the TaskMaster-1 Self Dialogs (TMSD) dataset (Byrne et al., 2019), a corpus of dialogs written by crowdworkers for task-oriented scenarios such as ordering pizza. TMSD is used only for evaluation and not for training or validation of the models. For DailyDialog, we remove 228 dialogues from the
test split that are duplicate with conversations in the train split (see Appendix A), which leads to what we call the DailyDialog Corrected (DDC) dataset. No correction is applied to the other datasets. The TAC dataset contains only isolated phrases without any conversational-turn context. Hence we use it only for training. In all of our experiments, we combine data from the training splits of all four datasets when fine-tuning models. We perform evaluations on the TDC, DDC, CMD, and TMSD datasets. The TDC dataset is chosen as our primary benchmarks because of its strict six-turn dialog structure.
Modifications for the AE task. The abovementioned datasets are typically used to study dialog generation. For our scenario, we convert each turn of the conversation in these datasets into the following canonical format:
Context:	{Content of the contextual turn}
Shorthand:	{Abbreviation of next turn}
Full:	{Expanded content of next turn}
Context:	{Would you like to sit down?}
Shorthand:	{n,imfsu}
Full:	{No, I'm fine standing up}
For the AE task, the context consists of one or more previous dialog turns. When context is absent (e.g., for the opening turn), the context part is omitted. For a multi-turn dialog, the  $\mathfrak{n}^{th}$  (1-based) example contains the first (n - 1) dialog turns as the context as well as the shorthand and the full form of the  $\mathfrak{n}^{th}$  turn. Thus, a 6-turn conversation yields six examples for the AE task. When multiple sentences are present in a single turn, we use only the first sentence for expansion; when a turn is used as context, all available sentences are used. Table 2 shows examples generated from all six turns of a dialog from TDC. Each dialog in the TDC, DDC, and CMD datasets yields several examples covering different amount of context. We create only 0-context-turn examples for the TAC dataset since it contains only isolated phrases.
Text-entry noise in AE datasets. As with our AE scheme, the introduction of noise to the datasets is also motivated by the AAC text entry use case, and in particular eye-gaze typing, which is error prone (Feit et al., 2017). Here, misclicks occur frequently and must be taken into account when designing a gaze-driven text entry system. In order to simulate the noise, we model eye-gaze typing as uncorrelated 2D Gaussian distributions around the intended key (Azenkot and Zhai, 2012). To
<sup>1</sup>The corrected version is available in the file turk_DIALOGues.corrected.txt in Supplemental Data
Dataset	train	dev	test
#conv.	#examples	Avg. tokens	#conv.	#examples	Avg. tokens	#conv.	#examples	Avg. tokens
Turk Dialogues Corrected (TDC)	859	5,154	54.4 ± 24.0	280	1,680	54.5 ± 24.3	280	1,680	55.0 ± 24.7
Turk AAC (TAC)	5,019	5,019	20.5 ± 4.3	559	559	20.9 ± 4.4	565	565	20.1 ± 4.0
DailyDialog Corrected (DDC)	11,188	87,170	101.1 ± 77.0	823	6,498	98.9 ± 72.3	772	5,852	96.7 ± 69.2
Cornell Movie Dialog (CMD)	66,848	244,798	68.3 ± 71.8	8,645	31,272	65.5 ± 67.4	7,444	27,429	69.8 ± 76.2
Table 1: Summary of datasets with number of conversations (conv.), examples, and average tokens (mean ± 1 SD in number of SentencePiece tokens) used in our experiments for the context-aware AE task.
Original dialog	AE example	AE example (noise σ=0.3)
Would you like to sit down?
No, I'm fine standing up
Are you sure you don't
want to sit down?
Been sitting all day. Work was
just one meeting after another.
Oh, I'm sorry. I don't enjoy
work days like that.
It feels good to stretch
my legs a bit.	0-turn context: Shorthand: {wyltsd}.
Full: {Would you like to sit down?}
1-turn context: Context: {Would
you like to sit down?}. Shorthand:
{n,imfsu}. Full: {No, I'm fine stand-
ing up}
...
5-turn context: Context: {Would you
like to sit down?} {No, I'm fine stand-
ing up} {Are you sure you don't want to
sit down?} {Been sitting all day. Work
was just one meeting after another.}
{Oh, I'm sorry. I don't enjoy work
days like that.}. Shorthand: {ifgtsm-lab}. Full: {It feels good to stretch my
legs a bit.}	0-turn context: Shorthand: {wy!tsd}.
Full: {Would you like to sit down?}
1-turn context: Context: {Would
you like to sit down?}. Shorthand:
{n,ifsu}. Full: {No, I'm fine stand-
ing up}
...
5-turn context: Context: {Would you
like to sit down?} {No, I'm fine stand-
ing up} {Are you sure you don't want to
sit down?} {Been sitting all day. Work
was just one meeting after another.}
{Oh, I'm sorry. I don't enjoy work days
like that.}. Shorthand: {ifgtsmoab}.
Full: {It feels good to stretch my legs a
bit.}
Table 2: An example dialog and the generated AE examples without and with typo noise. The six-turn dialog is an excerpt from the train split of the TDC dataset. In the 3rd column, the typos in abbreviation are marked in red.
Figure 2: Keyboard layout for simulating noise in AE key presses. The circles on the  $f$  key show  $1\sigma$  around the mean for  $\sigma \in \{0.3, 0.5\}$  in the 2D Gaussian distributions used to model typing noise.
simulate noise in the abbreviation input, we use a simplified rectangular-grid qwerty keyboard layout with 30 keys arranged in three rows and 10 columns. The keys are  $1 \times 1$  squares with no gaps in between. The keystrokes for an intended key are drawn from 2D Gaussian distribution centered on the center of the intended key and standard deviations denoted  $\sigma$  equal in the two spatial dimensions. To model different levels of noise, we use three values of  $\sigma$ : 0.0 (i.e., no-typo baseline), 0.3, and 0.5, which corresponds to  $0\%$ ,  $13\%$ , and  $44\%$  character error rates, respectively. Examples with simulated typos are shown in Table 2.
3.2 Large Language Model
One of our goals is to test whether zero-shot and few-shot prompting of LLMs are effective at the AE task without the need for supervised fine-tuning. Prompting is the method of eliciting desired task-specific responses from an LLM by including a
natural-language description of the task and/or input-output examples of the task in the input string for an LLM, without altering the model's weights (Brown et al., 2020). Zero- and few-shot prompting differ in whether any examples are included in the prompt to the LLM. For this, we use a decoder-only Transformer language model (Vaswani et al., 2017) from the LaMDA (Thoppilan et al., 2022) family of models. Our experiments are based on the 64B parameter model, unless otherwise specified. This model has 32 Transformer layers, with  $d_{model} = 8192$ ,  $d_{ff} = 65536$ ,  $h = 128$ ,  $dk = dv = 128$ . The model was pre-trained on 2.97B public web documents, Wikipedia, and dialogs. The training data was tokenized with the SentencePiece vocabulary (Kudo and Richardson, 2018) of size 32K. We call this the BaseLLM.
We also developed fine-tuned versions of this model for the AE task. The fine-tuning uses examples in the format as shown in Table 2. Since the BaseLLM is a decoder only model, and we use both the context and abbreviation as triggers to the model during inference, we modify the loss to only be calculated on the tokens of the AE target, i.e. the full form to be predicted in the pair of curly brackets after "Full:". For both training and inference, we split the characters in the abbreviation with spaces to force SentencePiece to
use per-character IDs. We tune two models, FT-LLM on the combined AE datasets without typos, and FTnoise-LLM on the version with simulated typos. Both use early stopping on a dev set consisting of combined examples from the dev splits of TAC and TDC (Table 1).
Models. We use and compare the following models in our different experiment settings.
Look-Up Table (LUT). As a straight-forward, non-ML baseline, we compile a dictionary of 375,298 sentence-level abbreviations from the train splits of the datasets in Table 1. Each abbreviation maps to one or more phrases with their frequencies, leading to 447,249 unique abbreviation-sentence pairs. During evaluation, we map the query abbreviation to the top-5 expansion phrases (by frequency) by using the dictionary and breaking ties randomly.
BaseLLM (from Sec. 3.2). We study the BaseLLM in the zero-shot and few-shot (specifically 4-shot) settings<sup>3</sup>. The four examples are selected from the train split of the TDC dataset (see Appendix B). We quantify the variability of the model on a sets of 856 4-example sequences from the train split of the TDC dataset. The best performing one on the dev set is denoted BaseLLM*.
FTnoise-LLM tuned on simulated typos with noise level  $\sigma = 0.3$  (see Appendix C), and FT-LLM tuned on AE data without noise as described in Sec. 3.2 are additional models we compare to.
T5 encoder-decoder. For comparison with smaller models, we use the T5 encoder-decoder small (60M), large (770M), and 3B parameter models fine-tuned on AE data without noise, identical to FT-LLM.
We evaluate the fine-tuned models in the setting without any explicit natural language instructions (denoted "no instr.") unless mentioned otherwise. For all models, we perform random sampling with temperature  $= 1.0$  over the top_k = 40 candidates with the highest logits at each step. We decode 128 samples for each abbreviation unless otherwise specified. For each model and evaluation setting we report the standard deviations (SDs) of metrics over 3 repeated runs.
Abbv. length	TDC (dev)	TDC (test)	DDC (test)	CMD (test)	TMSD (test)
1-2	85 (5.1%)	105 (6.2%)	166 (21.5%)	2,003 (26.9%)	176 (22.9%)
3-4	324 (19.3%)	293 (17.4%)	168 (21.8%)	1,753 (23.6%)	109 (14.2%)
5-6	454 (27.0%)	439 (26.1%)	152 (19.7%)	1,396 (18.8%)	113 (14.7%)
7-8	339 (20.2%)	376 (22.4%)	118 (15.3%)	851 (11.4%)	129 (16.8%)
9-10	221 (13.2%)	218 (13.0%)	64 (8.3%)	528 (7.1%)	111 (14.4%)
1-10	1,423 (84.7%)	1,431 (85.2%)	668 (86.5%)	6,531 (87.8%)	638 (82.9%)
Table 3: Datasets used for evaluation sliced by abbreviation lengths. Number of dialog turns in each range and their percentage (in parentheses) as compared to the total are noted.
Studies. For the BaseLLM, we study the variance in performance based on the prompt selection. For all the models, we sample multiple responses for each query, hence we study the effect of number of responses sampled on AE accuracy and latency. We also compare the performance of the models with varying amounts of conversation context and with no context. To study the effect of typos, we compare the performance of the models on the noise induced AE dataset. To measure the impact of model size on accuracy and latency, we also finetune and evaluate performance of the decoder-only LaMDA models with fewer than 64B parameters, specifically 4B, 8B, and 27B parameters. All these models were trained on the same data, so that the model size constitutes the only difference.
Evaluation. We only evaluate on conversation queries with abbreviation length  $\leq 10$  characters. This encompasses the majority (85%) of the dialog turns from the original dataset (Table 3). Where applicable, we prepend the following natural-language instruction to the model input for the AE task: "Given previous turn(s) of conversation and acronym of reply, write the full phrase."
Before calculating performance metrics, we filter the model's responses: we remove sentence-final punctuation, standardize whitespace to one space, lower-case, de-duplicate, and filter for precise match of the abbreviation. The responses that pass the filtering are sorted by descending count. For evaluation with noise, we do filtering to allow matches to nearby characters on the keyboard.
Metrics. Accuracy measures whether any response expansion exactly matches the ground truth (with standardized letter-casing and whitespace, and discarded final punctuation). Additionally, we measure BLEU score (Papineni et al., 2002) using the SacreBLEU library (Post, 2018) as a more fine-grained metric for the similarity between AE options and the ground truth. For both metrics, we report performance in the top-5 responses after they are sorted based on frequency.
Key Stroke Savings (KSR) measures the num
$^{2}$ Appendix D and F provide details on fine-tuning and discuss the effect of character splitting.
3The prompts are prefixed with the natural language instruction "Given acronym, write the full phrase." when there's no context or "Given previous turn(s) of conversation and acronym of reply, write the full phrase." when there is context.
Model	TDC-test	TDC-test+noise (σ=0.3)	DDC-test	CMD-test	TMSD-test
Acc.@5	BLEU@5	Acc.@5	BLEU@5	Acc.@5	BLEU@5	Acc.@5	BLEU@5	Acc.@5	BLEU@5
Look-Up Table (LUT)	14.3 ± 0.2	23.6 ± 0.1	10.5 ± 0.0	15.8 ± 0.7	48.1 ± 0.2	55.4 ± 0.3	30.9 ± 0.1	39.2 ± 0.1	29.3 ± 0.1	34.7 ± 0.1
T5-small (60M)	42.7 ± 0.5	59.9 ± 0.1	21.2 ± 0.1	36.1 ± 0.3	69.1 ± 0.5	78.1 ± 0.6	38.7 ± 0.0	50.4 ± 0.1	50.7 ± 0.3	64.8 ± 0.5
T5-large (770M)	55.2 ± 0.6	68.6 ± 0.4	27.3 ± 0.6	40.9 ± 0.3	74.2 ± 0.1	81.7 ± 0.1	41.2 ± 0.0	52.6 ± 0.1	57.1 ± 0.1	70.1 ± 0.2
T5-3B (3B)	59.4 ± 0.4	72.8 ± 0.1	26.9 ± 0.8	41.9 ± 0.7	77.6 ± 0.5	83.9 ± 0.5	43.5 ± 0.1	54.8 ± 0.2	59.5 ± 0.2	72.5 ± 0.3
BaseLLM* 64B (best, 4shot)	43.7 ± 1.2	54.9 ± 0.5	38.1 ± 0.1	42.0 ± 0.5	38.4 ± 0.4	43.3 ± 0.6	22.5 ± 0.2	25.9 ± 0.1	32.0 ± 0.7	36.2 ± 0.3
FT-LLM 64B (no instr.)	74.4 ± 1.0	81.8 ± 0.8	44.5 ± 0.7	55.0 ± 0.3	75.1 ± 0.6	82.1 ± 0.6	48.1 ± 0.1	57.9 ± 0.2	62.0 ± 0.3	73.9 ± 0.2
FTnoise-LLM 64B (no instr.)	72.3 ± 0.9	81.1 ± 0.5	60.9 ± 0.3	71.4 ± 0.5	74.8 ± 0.4	82.1 ± 0.3	47.5 ± 0.1	57.3 ± 0.1	63.3 ± 0.1	74.4 ± 0.2
Table 4: Comparing models (from Sec. 4) on the AE task on turn-2 given turn-1 as context. We report accuracy and BLEU score at top-5, as percentages, std. dev. computed on 3 runs. Higher is better, values in bold are highest in each column.
ber of saved keystrokes compared to the full length of the phrase. Note, however, that AE succeeds only for a subset of the cases, while for others the top-5 options do not contain the intended phrase. Hence we compute two types of KSR:
$\mathbf{KSR}_{all}$ , computed on all phrases, is defined as