A quantitative evaluation of keyword measures for corpus-based discourse analysis

We evaluated the automatic identification of keywords in our target corpus of German press texts on multiresistant pathogens; a recurring and highly controversial topic in public discourse (1.3M tokens). In corpusbased discourse analysis, keywords have been described as “a quick and simple ‘way in’” to corpus comparison (Baker et al. 2013), while the details on how to best 1) calculate and 2) categorise them lacks consensus. There have been multiple attempts to answer question 1) (cf. Kilgarriff 2001, Paquot & Bestgen 2009, Lijffijt et al. 2016). However, these studies focus mostly on either the number of keywords or the mathematical adequacy, but do not evaluate against an authentic use-case specifically tailored to discourse analysis. Our approach draws on a set of previously determined qualitative linguistic categories and evaluates statistically generated keyword lists against them. The keywords were calculated using three measures: ● log-likelihood (G2) (Dunning 1993) ● log ratio (LR), an intuitive measure implemented in the CQPweb corpus analysis software (Hardie 2012) ● LRconf, a conservative estimate for the log ratio coefficient using the lower bound of a 99% confidence interval with Bonferroni correction and two different German national broadsheet newspapers as reference corpora (RC): ● years 2011–2014 of Süddeutsche Zeitung (SZ), a left-leaning daily newspaper (290M tokens) ● years 2011–2014 of Frankfurter Allgemeine Zeitung (FAZ), a right-leaning daily newspaper (150M tokens) All corpora were POS-tagged and lemmatized using the state-of-the-art morphological analyzer SMOR (Schmid et al. 2004). Applying a frequency threshold of f ≥ 5 in the RC, we obtained the 200 top-scoring keywords for each combination of keyness measure and RC. The various measures show substantial differences (Figure 1). The results suggest that LRconf is intermediate between the other two measures, but it is closer to LR than G2. The choice of the RC has more influence than expected – according to the literature, the keywords should be relatively stable for large RC (Scott & Tribble 2006). While the top-200 lists against SZ vs. FAZ have an overlap of 89.0% for G2, the overlap is only 78.0% for LRconf and 58.5% for LR. Especially for LR, which tends to select lower-frequency keywords, the differences are partly due to the frequency threshold on the RC; but even for keywords occurring f ≥ 5 times in both RC, the ranking differences are bigger than for G2. A crucial difference between the keyness measures is that G2 prefers keywords with high frequency in the target corpus, while LR is biased towards low-frequency keywords (Hardie 2014) . Figure 2 confirms this expectation, showing LRconf as a compromise between the two extremes. Our evaluation is based on a manual classification of the keywords into 33 categories previously identified in a detailed qualitative analysis (Peters 2017). A keyword is considered a true positive (TP) if it can be clearly assigned to a category or expresses positive or negative sentiment, otherwise it is considered a false positive (FP). All six lists were pooled for the manual annotation, resulting in a total of 455 distinct lemmas.