SlowGuess/NAACL_2022_OCR huggingface dataset:
Per Ola Kristensson and Keith Vertanen. 2012. The potential of dwell-free eye-typing for fast assistive gaze communication. In Proceedings of the symposium on eye tracking research and applications, pages 241-244.
Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957.
Päivi Majaranta and Kari-Jouko Räihä. 2007. Text entry by gaze: Utilizing eye-tracking. Text entry systems: Mobility, accessibility, universality, pages 175-187.
Martez E Mott, Shane Williams, Jacob O Wobbrock, and Meredith Ringel Morris. 2017. Improving dwell-based gaze typing with dynamic, cascading dwell times. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 2558-2570.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Stefano Pini, Sangmok Han, and David R Wallace. 2010. Text entry for mobile devices using ad-hoc abbreviation. In Proceedings of the International Conference on Advanced Visual Interfaces, pages 181-188.
Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, et al. 2020. Recipes for building an open-domain chatbot. arXiv preprint arXiv:2004.13637.
Sebastian Ruder. 2021. Recent Advances in Language Model Fine-tuning. http://ruder.io/ recent-advances-lm-fine-tuning.
Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596-4604. PMLR.
Junxiao Shen, Boyin Yang, John J Dudley, and Per Ola Kristensson. 2022. Kwickchat: A multi-turn dialogue system for aac using context-aware sentence generation by bag-of-keywords. In 27th International Conference on Intelligent User Interfaces, pages 853-867.
Stuart M Shieber and Rani Nelken. 2007. Abbreviated text input using language modeling. Natural Language Engineering, 13(2):165-183.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239.
Tobii. How to create an abbreviation expansion in compass. https://us.tobiidynavox.com/blogs/support-articles/how-to-create-an-abbreviation-\( \)expansion-in-compass\( . Accessed: 2022-04-22.
Keith Trnka, John McCaw, Debra Yarrington, Kathleen F McCoy, and Christopher Pennington. 2009. User interaction with word prediction: The effects of prediction quality. ACM Transactions on Accessible Computing (TACCESS), 1(3):1-34.
Keith Trnka and Kathleen F McCoy. 2008. Evaluating word prediction: framing keystroke savings. In Proceedings of ACL-08: HLT, Short Papers, pages 261-264.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Keith Vertanen. 2017. Towards improving predictive aac using crowdsourced dialogues and partner context. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, pages 347-348.
Keith Vertanen and Per Ola Kristensson. 2011. The imagination of crowds: conversational aac language modeling using crowdsourcing and large data sources. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 700-711.
Annalu Waller. 2019. Telling tales: unlocking the potential of aac technologies. International journal of language & communication disorders, 54(2):159-169.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2021. Ethical and social risks of harm from language models.
Tim Willis, Helen Pain, and Shari Trewin. 2005. A probabilistic flexible abbreviation expansion system for users with motor disabilities. In Accessible Design in the Digital World Conference 2005, pages 1-9.
Bruce Wisenburn and D Jeffery Higginbotham. 2008. An aac application using speaking partner speech recognition to automatically produce contextually relevant utterances: Objective results. *Augmentative and Alternative Communication*, 24(2):100-109.
A Removal of duplicate dialogs from the DailyDialog dataset
We observed that the DailyDialog dataset (Li et al., 2017) contains a significant number of dialogs in its dev (validation) and test splits that are identical or nearly identical to the dialogs found in its train split. We determined two dialogs to be duplicate by using the following criterion:
1. If both dialogs consist of the same number of turns and the corresponding turns are all identical (case-insensitive), or
2. If both dialogs consist of the same number of turns and there are three or more turns at which both dialogs contain identical text (case-insensitive).
See the file daily_DIALOG_deduplications.csv in Supplemental Data for a list of the 177 dialogs in the dev split and the 228 dialogs in the test splits that are found to be duplicates with the train split and hence are removed from our DailyDialog Corrected (DDC) dataset.
B 4-shot examples for BaseLLM*
We select four consecutive dialogues from the 859 examples from train split of the TDC dataset (Vertanen, 2017) while varying the starting conversation, which yields  $859 - 4 + 1 = 856$  different 4-shot prompt sets.
C Tuning on noisy data vs. accuracy
Preliminary experiments have shown that  $\sigma = 0.3$  is a good trade-off between accuracy gains on noisy data and losses on non-noisy data.
D Model fine-tuning details
Our model fine-tuning uses the AdaFactor optimizer (Shazeer and Stern, 2018). The nominal batch size 16 is made more efficient through example packing (Raffel et al., 2019), leading to an average effective batch size of approximately 200 examples under a maximum sequence length of 1024 tokens. We used TPUv3s (Jouppi et al., 2018) with a configuration of 4x8 for the LLM fine-tuning. Our fine-tuning recipe applies a constant, low learning rate of  $5 \times 10^{-5}$  and a dropout rate of 0.2, which helps to prevent early overfitting. Early stopping is based on a dev set consisting of combined examples
from the dev splits of the TAC and TDC datasets. We find the best checkpoint after 2100 and 1800 training steps for the FT-LLM and FTnoise-LLM models, respectively, which amounts to approximately 1-1.2 epochs of training. We ran a small set of hyperparameter tuning experiments, varying batch size, learning rate and dropout and chose the best setting based on the  $\mathrm{TAC + TDC}$  dev set.
E Computation cost
Fine-tuning of the 64B LLM uses TPU v3 with a 4x8 configuration, i.e., 32 TPUs. FT-LMM and FTnoise-LLM are each trained for approximately 2100 and 1800 steps, respectively. The training time is approximately 3 hours. This leads to a model fine-tuning budget of  $32 \times 3 = 96$  TPU * hour per model.
Evaluation and inference on the 64B LLM uses TPU v3 with a 4x4 configuration, i.e., 16 TPUs. Each example (batch size  $= 128$  samples) takes  $0.653\mathrm{s}$ . This leads to  $16 \times 0.568 / 128 = 0.071$  TPU  $\times$  second per sample.
F Splitting characters in abbreviations.
Pilot experiments showed the importance of programmatically inserting spaces between characters in the abbreviations. Since the vocabulary used by the LaMDA models is fairly large (32k entries), unless we enforce character-level splitting, subsequences of multiple characters in many abbreviations will be combined into spurious tokens, leading to slightly reduced AE accuracy.
G Recovery from failure - analysis
In the best scenario of replying to a question, the fine-tuned LLM is capable of predicting the correct phrase expansion approximately  $81\%$  of the times with top-5 options and sufficient sampling (Figure 5). Hence the model will fail to find the correct expansion at least  $19\%$  of the cases.
H Inference latencies of different LaMDA model sizes
In Figure H we compare the latencies during inference time for the decoder-only models of different sizes. Compared to the 4B model, the 27B model shows  $1.5\mathrm{x}$  latency, while the 64B model shows  $2.2\mathrm{x}$  latency. While the latency increase is quite significant, this analysis shows that we cannot substitute the 64B model with a smaller model (e.g.,
#	Context	Abbreviation	Ground truth	Non-matching expansion options
1	Awesome! My favorite weather!	swhottwp	Shall we head over to the water park?	shall we head out to the water park
2	Can we go out for a drive?	ygstc	Yeah go start the car	yes go start the car yes go straight to church yes go settle the children yeah get some tunes cranked yes go straight to chicago
3	i took a lot of courses, such as philosophy, logic, ethics, aesthetics, etc	wcdylb	which course did you like best	what courses do you like best what courses did you like best what course do you like best what course did you like best which courses did you like best
4	it's hard to be optimistic about things with the way the economy's headed... the trade deficit is getting larger, consumption's down, i really think we're headed for a recession	tehbsfawn	the economy has been stagnant for a while now	the economy has been slowing for a while now the economy has been sluggish for a while now the economy has been strong for a while now the economy has been slow for a while now the economy has been suffering for a while now
5	What is your name?	mnir	My name is Rey	my name is robert my name is rebecca my name is richard my name is rose my name is roy
6	hey, isabelle...	l	Logan	lisa linda look lillian liz
7	so, paula, where are you from	imfc,o	i'm from canada, originally	i'm from china, ok i'm from california, originally i'm from california, ok i'm from california, okay i'm from california, obviously
8	hey sandra, what's wrong? you look furious	ivhiwt	i've had it with Tim	i've had it with this i've had it with them i've heard it was true i've had it with that i've had it with these
Table 7: Examples of failed AE. Examples #1-4 show AE options that miss the ground-truth phrase closely. The cases highlighted in boldface have near identical meaning to the ground truth, but differ only in details of a single word. Examples #5-8 show AE options that match the ground truth except for the a proper noun.
Model	TDC-dev
Acc.@5	BLEU@5
Look-Up Table (LUT)	16.9 ± 0.2	25.2 ± 0.2
T5-small (60M)	37.8 ± 0.0	59.2 ± 0.5
T5-large (770M)	48.2 ± 0.0	69.1 ± 0.5
T5-3B (3B)	53.9 ± 0.0	72.3 ± 0.5
BaseLLM* (best, 4shot)	43.0 ± 1.0	52.0 ± 1.4
FT-LLM (no instr.)	76.7 ± 1.1	83.9 ± 0.5
FTnoise-LLM (no instr.)	75.8 ± 0.7	83.4 ± 0.2
Table 8: Comparing models (from Sec. 4) on the AE task on turn-2 given turn-1 as context. We report accuracy and BLEU score at top-5, as percentages, std. dev. computed on 3 runs. Higher is better, values in bold are highest in each column. The TDC-dev set was used for model selection before evaluation on test sets.
Figure 7: Inference latencies for different sizes of the LaMDA model (4B, 8B, 27B, and 64B.) The latencies are shown as box plots.
by increasing the number of samples) in a way that improves latency without significantly harming the AE accuracy (compare the AE accuracies in Figure 5.)