SlowGuess/ACL_2020_OCR huggingface dataset:
We trained two sets of dementia and control LSTM models. The first set was trained in order to replicate the findings of Fritsch et al. (2019), using the same RWTHLM package (Sundermeyer et al., 2014) and following their methods as closely as possible in accordance with the description provided in their paper. Each model's cross-entropy loss was optimized over 20 epochs with starting learning rate optimization performed on a heldout set of 10 transcripts. The second set was trained using the GluonNLP averaged stochastic gradient weight-dropped LSTM (standard-lstm-lm-200 architecture) model consisting of 2 LSTM layers with word embedding (tied at input and output) and hidden layers of 200 and 800 dimensions respectively (see Merity et al. (2017) for full details on model architecture). In training the GluonNLP models, the main departure from the methods used by Fritsch et al. (2019) involved not using a small heldout set of transcripts to optimize the learning rate because we observed that the GluonNLP models converged well prior to the 20th epoch with a starting learning rate of 20 which was used for all stochastically initialized models. With pre-trained models we used a lower starting learning rate of 5 to preserve information during subsequent training on DementiaBank. All GluonNLP models were trained using batch size of 20 and back propagation through time (BPTT) window size of 10. During testing, batch size was set to 1 and BPTT to the length of the transcript (tokens). Unseen transcript perplexity was calculated as  $e^{\text{loss}}$ .
3.4 Evaluation
As subjects in the DementiaBank dataset participated in multiple assessments, there are multiple transcripts for most of the subjects. In order to avoid biasing the models to individual subjects, we
followed the participant-level leave-one-out cross-validation (LOOCV) evaluation protocol of Fritsch et al. (2019) whereby all of the picture description transcripts for one participant are held out in turn for testing and the LMs are trained on the remaining transcripts. Perplexities of the LMs are then obtained on the heldout transcripts, resulting in two perplexity values per transcript, one from the LM trained on the dementia  $(P_{dem})$  and control  $(P_{con})$  transcripts. Held-out transcripts were scored using these perplexity values, as well as by the difference  $(P_{con} - P_{dem})$  between them.
3.5 Interrogation of models
For interrogation by perturbation, we estimated the perplexity of our models for each of the six synthetic narratives of Bird et al. (2000). We reasoned that an increase in  $P_{con}$  and a decrease in  $P_{dem}$  as words are replaced by higher-frequency alternatives to simulate progressive lexical retrieval deficits would indicate that these models were indeed capturing AD-related linguistic changes. For interrogation by interpolation, we extracted the parameters from all layers of paired LSTM LMs after training, and averaged these as  $\alpha LM_{dem} + (1 - \alpha)LM_{con}$  to create interpolated models. We hypothesized that a decrease in perplexity estimates for narratives emulating severe dementia would occur as  $\alpha$  (the proportional contribution of  $LM_{dem}$ ) increases.
The results of evaluating classification accuracy of the various language models are summarized in Table 1. The  $95\%$  confidence interval for GluonNLP models was calculated from perplexity means obtained across ten LOOCV iterations with random model weight initialization on each iteration. The RWTHLM package does not provide support for GPU acceleration and requires a long time to perform a single LOOCV iteration (approximately 10 days in our case). Since the purpose of using the RWTHLM package was to replicate the results previously reported by Fritsch et al. (2019) that were based on a single LOOCV iteration and we obtained the exact same AUC of 0.92 on our first LOOCV iteration with this approach, we did not pursue additional LOOCV iterations. However, we should note that we obtained an AUC of 0.92 for the difference between  $P_{con}$  and  $P_{dem}$  on two of the ten LOOCV iterations with the GluonNLP LSTM model. Thus, we believe that the GluonNLP
<sup>7</sup>Available at https://github.com/facebookresearch/fastText
Other hyperparameters per (Cohen and Widdows, 2018)
MODEL	CONTROL	DEMENTIA	CONTROL-DEMENTIA
AUC	95% CI	AUC	95% CI	AUC	95% CI
RWTHLM LSTM	0.80	-	0.64	-	0.92	-
GluonNLP LSTM	0.80	± 0.002	0.65	± 0.002	0.91	± 0.004
Table 1: Classification accuracy using individual models' perplexities and their difference for various models.
Figure 1: Relationship between log frequency bands used to replace words in synthetic Cookie Theft picture descriptions to simulate degrees of semantic dementia and perplexity of LSTM language models trained on picture descriptions by controls and dementia patients.
LSTM model has equivalent performance to the RWTHLM LSTM model.
Having replicated results of previously published studies and confirmed that using the difference in perplexities trained on narratives by controls and dementia patients is indeed the current state-of-the-art, we now turn to explaining why the difference between these LMs is much more successful than the individual models alone.
First, we used the six "Cookie Theft" narratives designed to simulate semantic dementia to examine the relationship between  $P_{con}$  and  $P_{dem}$  with GluonNLP LSTM LMs and log lexical frequency bands. The results of this analysis are illustrated in Figure 1 and show that  $P_{dem}$  is higher than  $P_{con}$  on narratives in the lower log frequency bands (less simulated impairment) and lower in the higher log frequency bands (more simulated impairment).
We confirmed these results by calculating mean log lexical frequency on all DementiaBank narratives and fitting a linear regression model to test for associations with perplexities of the two LMs. The regression model contained mean lexical frequency as the dependent variable and  $P_{dem}$  and  $P_{con}$  as independent variables, adjusted for age, education and the length of the picture description narrative. In order to avoid likely practice effects
across multiple transcripts, we only used the transcript obtained on the initial baseline visit; however, we did confirm these results by using all transcripts to fit mixed effects models with random slopes and intercepts in order to account for the correlation between transcripts from the same subject (mixed effects modeling results not shown).
The results demonstrate that the association between perplexity and lexical frequency is significant and positive for the control LM (coeff: 0.563,  $p < 0.001$ ) and negative for dementia LM (coeff: -0.543,  $p < 0.001$ ). Age, years of education, and length of the narrative were not significantly associated with lexical frequency in this model. These associations show that the control LM and dementia LM are more "surprised" by narratives containing words of higher lexical frequency and lower lexical frequency respectively. If the use of higher lexical frequency items on a picture description task portends a semantic deficit, then this particular pattern of results explains why it is the difference between the two models that is most sensitive to manifestations of dementia and suggests that there is a point at which the two models become equally "surprised" with a difference between their perplexities close to zero. In Figure 1, that point is between log lexical frequency bands of 2.0 and 2.5 corresponding to the mild to moderate degree of semantic impairment reported by Bird et al. (2000). Notably, in the clinical setting, the mild forms of dementia such as mild cognitive impairment and mild dementia are also particularly challenging and require integration of multiple sources of evidence for accurate diagnosis (Knopman and Petersen, 2014).
The results of our interpolation studies are shown in Figure 2. Each point in the figure shows the average difference between the perplexity estimate of a perturbed transcript  $(Px)$  and the perplexity estimate for the unperturbed  $(Po$ : frequency band 0) sample for this model<sup>9</sup>. While all models tend
We visualized this difference because perplexities at  $\alpha = 0.5$  were generally higher, irrespective of whether component models were initialized stochastically, or had pre-trained word embeddings in common. Perplexities of  $\alpha = 0.75$  models were slightly lower than those of their majority constituents.
RANDOM	PRETRAINED	RANDOM	PRETRAINED
Pcon-Pα	AUC	95% CI	AUC	95% CI	ACCeer	95% CI	ACCeer	95% CI
α = 0.25	0.842	± 0.008	0.838	± 0.015	0.689	± 0.036	0.724	± 0.034
α = 0.5	0.816	± 0.009	0.813	± 0.005	0.669	± 0.035	0.665	± 0.033
α = 0.75	0.931	± 0.003	0.941	± 0.006	0.854	± 0.031	0.872	± 0.010
α = 1.0	0.908	± 0.004	0.930	± 0.005	0.846	± 0.023	0.839	± 0.017
Table 2: Performance of randomly-instantiated and pre-trained (subword-based skipgram embeddings) interpolated "two perplexity" models across 10 repeated per-participant LOOCV runs.  $\alpha$  indicates the proportional contribution of the dementia model.  $ACC_{eer}$  gives the accuracy at equal error rate. Best results are in **boldface**, and results using the approach of Fritsch et al. (2019) are in *italics*.
to find the increasingly perturbed transcripts more perplexing than their minimally perturbed counterparts, this perplexity decreases with increasing contributions of the dementia LM. However, when only this model is used, relative perplexity of the perturbed transcripts increases. This indicates that the "pure" dementia LM may be responding to linguistic anomalies other than those reflecting lack of access to infrequently occurring terms. We reasoned that on account of this, the  $\alpha = 0.75$  model may provide a better representation of dementia-related linguistic changes. To evaluate this hypothesis, we assessed the effects on performance of replacing the dementia model with this interpolated model. The results of these experiments (Table 2) reveal improvements in performance with this approach, with best AUC (0.941) and accuracy at equal error rate (0.872) resulting from the combination of interpolation<sup>10</sup> with pre-trained word embeddings. That pre-trained embeddings further improve performance is consistent with the observation that the elevation in perplexity when transitioning from  $\alpha = 0.75$  to  $\alpha = 1.0$  is much less pronounced in these models (Figure 3). These results are significantly better than those reported by Fritsch et al (2019), and our reimplementation of their approach.
These improvements in performance appear to be attributable to a smoothing effect on the perplexity of the modified dementia models in response to unseen dementia cases. Over ten repeated LOOCV iterations, average perplexity on held-out dementia cases was significantly lower than that of the baseline 'dementia' model  $(51.1 \pm 0.81)$  for both the  $\alpha = 0.75$ $(47.3 \pm 0.32)$  and pre-trained embeddings  $(44.8 \pm 0.53)$  models. This trend is further accentuated with the severity of dementia - for transcripts corresponding to a mini-mental state
Figure 2: Stochastically initialized models. Elevation in perplexity over unperturbed transcript  $(Po)$  with the proportional contribution of a dementia model  $(\alpha)$  to an interpolated model. Each point is the mean of 268 (held-out participants) data points. Error bars are not shown as they do not exceed the bounds of the markers.
Figure 3: Pretrained word embeddings. Elevation in perplexity over unperturbed transcript  $(Po)$  with the proportional contribution of a dementia model  $(\alpha)$  to an interpolated model. Each point is the average of 268 data points, and error bars are not shown as they do not exceed the bounds of the markers.
10 Simply weighting the difference in model perplexities does not perform as well as interpolating model weights, with at best a 0.001 improvement in AUC over the baseline.
exam (MMSE)  $\leq 10$ $(n = 16)$ , average perplexities are  $148.29\pm 7.69$ ,  $105.01\pm 3.48$  and  $121.86\pm 7.67$  for baseline 'dementia',  $\alpha = 0.75$  and pre-trained embeddings models respectively. In both cases, average perplexity of the interpolated  $(\alpha = 0.75)$  pretrained embeddings model fell between those of the exclusively pre-trained (lowest overall) and exclusively interpolated (lowest in severe cases) models.
A practical issue for automated methods to detect dementia concerns establishing their accuracy at earlier stages of disease progression, where a readily disseminating screening tool would arguably have greatest clinical utility, especially in the presence of an effective disease-modifying therapy. To this end, Fritsch et al. (2019) defined a "screening scenario" in which evaluation was limited to participants with a last available MMSE of 21 or more, which corresponds to a range of severity encompassing mild, questionable or absent dementia (Perneczky et al., 2006). In this scenario, classification accuracy of the 'paired perplexity' LSTM based model was only slightly lower (AUC: 0.87) than the accuracy on the full range of cognitive impairment (AUC: 0.92). We found similar performance with our models. When limiting evaluation to those participants with a last-recorded MMSE  $\geq 21$ , average AUCs across 10 LOOCV iterations were  $0.836 \pm 0.014$ ,  $0.879 \pm 0.01$ ,  $0.893 \pm 0.004$ , and  $0.899 \pm 0.012$  for the baseline (Fritsch et al (2019)), pretrained embeddings, interpolated  $(\alpha = 0.75)$  and interpolated  $(\alpha = 0.75)$  with pretrained embeddings variants, respectively. These results support the notion that paired neural LMs can be used effectively to screen for possible dementia at earlier stages of cognitive impairment.
The contributions of our work can be summarized as follows. First, our results demonstrate that the relationship between LM perplexity and lexical frequency is consistent with the phenomenology of DAT and its deleterious effects on patients' vocabulary. We show that the "two perplexities" approach is successful at distinguishing between cases and controls in the DementiaBank corpus because of its ability to capture specifically linguistic manifestations of the disease. Second, we observe that interpolating between dementia and control LMs mitigates the tendency of dementia-based LMs to be "surprised" by transcripts indicating severe dementia, which is detrimental to performance when the difference between these LMs is used as a basis for classification. In addition, we find a similar
smoothing effect when using pre-trained word embeddings in place of a randomly instantiated word embedding layer. Finally, we develop a modification of Fritsch et al's "two perplexity" approach that is consistent with these observations - replacing the dementia model with an interpolated variant, and introducing pre-trained word embeddings at the embedding layer. Both modifications exhibit significant improvements in performance, with best results obtained by using them in tandem. Though not strictly comparable on account of differences in segmentation of the corpus amongst others, we note the performance obtained also exceeds that reported with models trained on text alone in prior research. Code to reproduce the results of our experiments is available on GitHub<sup>11</sup>.
While using transcript text directly is appealing in its simplicity, others have reported substantial improvements in performance when POS tags and paralinguistic features are incorporated, suggesting fruitful directions for future research. Furthermore, prior work on using acoustic features shows that they can contribute to discriminative models (Konig et al., 2015); however, Dementia Bank audio is challenging for acoustic analysis due to poor quality and background noise. Lastly, while our results do support the claim that classification occurs on the basis of dementia-specific linguistic anomalies, we also acknowledge that DementiaBank remains a relatively small corpus by machine learning standards, and that more robust validation would require additional datasets.
We offer an empirical explanation for the success of the difference between neural LM perplexities in discriminating between DAT patients and controls, involving lexical frequency effects. Interrogation of control- and dementia-based LMs using synthetic transcripts and interpolation of parameters reveals inconsistencies harmful to model performance that can be remediated by incorporating interpolated models and pre-trained embeddings, with significant performance improvements.
This research was supported by Administrative Supplement R01 LM011563 S1 from the National Library of Medicine.
<sup>11</sup>https://github.com/reversec/tale_of_two_perplexities
Daniel C Aguirre-Acevedo, Francisco Lopera, Eliana Henao, Victoria Tirado, Claudia Munoz, Margarita Giraldo, Shrikant I Bangdiwala, Eric M Reiman, Pierre N Tariot, Jessica B Langbaum, et al. 2016. Cognitive decline in a colombian kindred with autosomal dominant alzheimer disease: a retrospective cohort study. JAMA neurology, 73(4):431-438.
Amit Almor, Daniel Kempler, Maryellen C. MacDonald, Elaine S. Andersen, and Lorraine K. Tyler. 1999a. Why do Alzheimer patients have difficulty with pronouns? Working memory, semantics, and reference in comprehension and production in Alzheimer's disease. Brain and language, 67(3):202-227.
Amit Almor, Daniel Kempler, Maryellen C. MacDonald, Elaine S. Andersen, and Lorraine K. Tyler. 1999b. Why do alzheimer patients have difficulty with pronouns? working memory, semantics, and reference in comprehension and production in alzheimer's disease. Brain and Language, 67(3):202-227.
Lori JP Altmann and Jill S McClung. 2008. Effects of semantic impairment on language use in alzheimer's disease. In Seminars in Speech and Language, 01, pages 018-031. © Thieme Medical Publishers.
Alzheimer's Association. 2018. 2018 Alzheimer's disease facts and figures. Alzheimer's & Dementia, 14(3):367-429.
Arlene J. Astell and Trevor A. Harley. 1996. Tip-of-thetongue states and lexical access in dementia. Brain and Language, 54(2):196-215.
James T Becker, François Boiler, Oscar L Lopez, Judith Saxton, and Karen L McGonigle. 1994. The natural history of alzheimer's disease: description of study cohort and accuracy of diagnosis. Archives of Neurology, 51(6):585-594.
Shauna Berube, Jodi Nonnemacher, Cornelia Demsky, Shenly Glenn, Sadhvi Saxena, Amy Wright, Donna C Tippett, and Argye E Hillis. 2018. Stealing cookies in the twenty-first century: Measures of spoken narrative in healthy versus speakers with aphasia. American journal of speech-language pathology, 28(1S):321-329.
H Bird, MA Lambon Ralph, K Patterson, and JR Hodges. 2000. The rise and fall of frequency and imageability: how the progression of semantic dementia impacts on noun and verb production in the cookie theft description. *Brain and Language*, 73.:17 - 49.
Linda Boise, Richard Camicioli, David L Morgan, Julia H Rose, and Leslie Congleton. 1999. Diagnosing dementia: perspectives of primary care physicians. The Gerontologist, 39(4):457-464.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
John Bond, C Stave, A Sganga, O Vincenzino, B O'connell, and RL Stanley. 2005. Inequalities in dementia care across europe: key findings of the facing dementia survey. International Journal of Clinical Practice, 59:8-14.
Thorsten Brants. 2000. Tnt - a statistical part-of-speech tagger.
Marc Brysbaert and Boris New. 2009. Moving beyond kucera and francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for american english. Behavior Research Methods, 41(4):977-990.
Trevor Cohen and Dominic Widdows. 2018. Bringing order to neural word embeddings with embeddings augmented by random permutations (earp). In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 465-475.
Claudia Cooper, Paul Bebbington, James Lindesay, Howard Meltzer, Sally McManus, Rachel Jenkins, and Gill Livingston. 2011. The meaning of reporting forgetfulness: a cross-sectional study of adults in the English 2007 Adult Psychiatric Morbidity Survey. Age and ageing, 40(6):711-717.
Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179-211.
W. N. Francis and H. Kucera. 1979. Brown corpus manual. Technical report, Department of Linguistics, Brown University, Providence, Rhode Island, US.
Kathleen Fraser, Jed Meltzer, and Frank Rudzicz. 2015. Linguistic features identify alzheimer's disease in narrative speech. Journal of Alzheimer's disease : JAD, 49.
Kathleen C Fraser, Jed A Meltzer, and Frank Rudzicz. 2016. Linguistic features identify alzheimer's disease in narrative speech. Journal of Alzheimer's Disease, 49(2):407-422.
Julian Fritsch, Sebastian Wankerl, and Elmar Noth. 2019. Automatic diagnosis of alzheimer's disease using neural network language models. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5841-5845. IEEE.
Elaine Giles, Karalyn Patterson, and John R. Hodges. 1996. Performance on the Boston Cookie theft picture description task in patients with early dementia of the Alzheimer's type: Missing information. *Aphasiology*, 10(4):395-408.
Yoav Goldberg. 2016. A primer on neural network models for natural language processing. Journal of Artificial Intelligence Research, 57:345-420.
Harold Goodglass. 2000. Boston diagnostic aphasia examination: Short form record booklet. Lippincott Williams & Wilkins.
Daniel B. Hier, Karen Hagenlocker, and Andrea Gellin Shindler. 1985. Language disintegration in dementia: Effects of etiology and severity. *Brain and Language*, 25(1):117-133.
Sepp Hochreiter. 1998. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6(02):107-116.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Frederick Jelinek, Robert Mercer, L R Bahl, and J K Baker. 1977. Perplexity - a measure of the difficulty of speech recognition tasks. Journal of the Acoustical Society of America, 62:S63.
Jorg D. Jescheniak and Willem J. M. Levelt. 1994. Word frequency effects in speech production: Retrieval of syntactic information and of phonological form. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20(4):824-843.
Michael I Jordan. 1986. Serial order: A parallel distributed processing approach. Technical report, CALIFORNIA UNIV SAN DIEGO LA JOLLA INST FOR COGNITIVE SCIENCE.
Sweta Karlekar, Tong Niu, and Mohit Bansal. 2018. Detecting linguistic characteristics of alzheimer's dementia by interpreting neural models. arXiv preprint arXiv:1804.06440.
Philipp Klumpp, Julian Fritsch, and Elmar Nöth. 2018. Ann-based alzheimers disease classification from bag of words. In Speech Communication; 13th ITG-Symposium, pages 1-4. VDE.
David S. Knopman and Ronald C. Petersen. 2014. Mild cognitive impairment and mild dementia: A clinical perspective. Mayo Clinic Proceedings, 89(10):1452-1459.
Alexandra Konig, Aharon Satt, Alexander Sorin, Ron Hoory, Orith Toledo-Ronen, Alexandre Derreumaux, Valeria Manera, Frans Verhey, Pauline Aalten, Phillip H. Robert, and Renaud David. 2015. Automatic speech analysis for the assessment of patients with predementia and alzheimer's disease. Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 1(1):112-124.
Willem J. M. Levelt. 2001. Spoken word production: A theory of lexical access. Proceedings of the National Academy of Sciences, 98(23):13464-13471.