SlowGuess/ACL_2020_OCR huggingface dataset:
Hila Gonen $^{1}$  Ganesh Jawahar $^{2}$  Djamé Seddah $^{2}$  Yoav Goldberg $^{1,3}$
$^{1}$ Department of Computer Science, Bar-Ilan University  $^{2}$ Inria Paris
3Allen Institute for Artificial Intelligence
hilagnn@gmail.com ganeshjwhr@gmail.com
yoav.goldberg@gmail.com djame.seddah@inria.fr
The problem of comparing two bodies of text and searching for words that differ in their usage between them arises often in digital humanities and computational social science. This is commonly approached by training word embeddings on each corpus, aligning the vector spaces, and looking for words whose cosine distance in the aligned space is large. However, these methods often require extensive filtering of the vocabulary to perform well, and—as we show in this work—result in unstable, and hence less reliable, results. We propose an alternative approach that does not use vector space alignment, and instead considers the neighbors of each word. The method is simple, interpretable and stable. We demonstrate its effectiveness in 9 different setups, considering different corpus splitting criteria (age, gender and profession of tweet authors, time of tweet) and different languages (English, French and Hebrew).
Analyzing differences in corpora from different sources (different time periods, populations, geographic regions, news outlets, etc) is a central use case in digital humanities and computational social science. A particular methodology is to identify individual words that are used differently in the different corpora. This includes words that have their meaning changed over time periods (Kim et al., 2014; Kulkarni et al., 2015; Hamilton et al., 2016b; Kutuzov et al., 2018; Tahmasebi et al., 2018), and words that are used differently by different populations (Azarbonyad et al., 2017; Rudolph et al., 2017). It is thus desired to have an automatic, robust and simple method for detecting such potential changes in word usage and surfacing them for human analysis. In this work we present such a method.
A popular method for performing the task (§4) is to train word embeddings on each corpus and then to project one space to the other using a vector-space alignment algorithm. Then, distances between a word-form to itself in the aligned space are used as an estimation of word usage change (Hamilton et al., 2016b). We show that the common alignment-based approach is unstable, and hence less reliable for the usage change detection task (§3,§7). In addition, it is also sensitive to proper nouns and requires filtering them.
We propose a new and simple method for detecting usage change, that does not involve vector space alignment (§5). Instead of trying to align two different vector spaces, we propose to work directly in the shared vocabulary space: we take the neighbors of a word in a vector space to reflect its usage, and consider words that have drastically different neighbours in the spaces induced by the different corpora to be words subjected to usage change. The intuition behind this approach is that words that are used significantly differently across corpora are expected to have different contexts and thus to have only few neighboring words in common. In order to determine the extent of the usage change of a word, we simply consider its top-k neighbors in each of the two corpora, and compute the size of the intersection of the two lists. The smaller the intersection is, the bigger we expect the change to be. The words are ranked accordingly.
The advantages of our method are the following:
1. Simplicity: the method is extremely simple to implement and apply, with no need for space alignment, hyperparameter tuning, and vocabulary filtering, except for simple frequency cutoffs.
2. Stability: Our method is stable, outputting similar results across different word embed
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 538-555 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
dings trained on the same corpora, in contrast to the alignment-based approach.
3. Interpretability: The ranking produced by our method is very intuitive to analyze. Looking at the neighborhood of a word in the two corpora reveals both the meaning of the word in each, and the extent to which the word has changed.
4. Locality: The interpretability aspect is closely linked to the locality of the decision. In our approach, the score of each word is determined only by its own neighbours in each of the spaces. In contrast, in the projection based method the similarity of a pair of words after the projection depends on the projection process, which implicitly takes into account all the other words in both spaces and their relations to each other, as well as the projection lexicon itself, and the projection algorithm. This makes the algorithmic predictions of the projection-based methods opaque and practically impossible to reason about.
We demonstrate the applicability and robustness of the proposed method (§7) by performing a series of experiments in which we use it to identify word usage changes in a variety of corpus pairs, reflecting different data division criteria. We also demonstrate the cross-linguistic applicability of the method by successfully applying it to two additional languages beyond English: French (a Romance language) and Hebrew (a Semitic language).
We argue that future work on detecting word change should use our method as an alternative to the now dominant projection-based method. To this end, we provide a toolkit for detecting and visualizing word usage change across corpora.<sup>1</sup>
Our aim is to analyze differences between corpora by detecting words that are used differently across them. This task is often referred to as "detecting meaning change" (Azarbonyad et al., 2017; Del Tredici et al., 2019).
However, we find the name "meaning change" to be misleading. Words may have several meanings in the different corpora, but different dominant sense in each corpus, indicating different use of the
word. For this reason, we refer to this task as "detecting usage change".
We define our task as follows: given two corpora with substantial overlapping vocabularies, identify words that their predominant use is different in the two corpora. The algorithm should return a ranked list of words, from the candidate that is most likely to have undergone usage-change, to the least likely.
Since the primary use of such algorithm is corpus-based research, we expect a human to manually verify the results. To this end, while the method does not need to be completely accurate, it is desirable that most of the top returned words are indeed those that underwent change, and it is also desirable to provide explanations or interpretations as to the usage of the word in each corpus. Lastly, as humans are susceptible to be convinced by algorithms, we prefer algorithms that reflect real trends in the data and not accidental changes in environmental conditions.
A desired property of an analysis method is stability: when applied several times with slightly different conditions, we expect the method to return the same, or very similar, results. Insignificant changes in the initial conditions should result in insignificant changes in the output. This increases the likelihood that the uncovered effects are real and not just artifacts of the initial conditions.
Recent works question the stability of word embedding algorithms, demonstrating that different training runs produce different results, especially with small underlying datasets. Antoniak and Mimno (2018) focuses on the cosine-similarity between words in the learned embedding space, showing large variability upon minor manipulations on the corpus. Wendlandt et al. (2018) make a similar argument, showing that word embeddings are unstable by looking at the 10-nearest neighbors (NN) of a word across the different embeddings, and showing that larger lists of nearest neighbors are generally more stable.
In this work, we are concerned with the stability of usage-change detection algorithms, and present a metric for measuring this stability. A usage-change detection algorithm takes as input two corpora, and returns a ranked list  $r$  of candidate words, sorted from the most likely to have changed to the least likely. For a stable algorithm, we expect different runs to return similar lists. While we do not
1https://github.com/gonenhila/usage_ change
care about the exact position of a word within a list, we do care about the composition of words at the top of the list. We thus propose a measure we call intersection@  $k$ , measuring the percentage of shared words in the top-k predictions of both outputs:
$$
\operatorname {i n t e r s e c t i o n} @ k \left(r _ {1}, r _ {2}\right) = \frac {\left| r _ {1} ^ {k} \cap r _ {2} ^ {k} \right|}{k} \tag {1}
$$
where  $r_1$  and  $r_2$  are the two ranked lists, and  $r_i^k$  is the set of top  $k$  ranked words in ranking  $r_i$ .
A value of 0 in this measure means that there are no words in the intersection, which indicates high level of variability in the results, while a value of 1 means that all the words are in the intersection, indicating that the results are fully consistent. We expect to see higher intersection@  $k$  as  $k$  grows. This expectation is confirmed by our experiments in Section 7.2.
We measure the stability of the usage-change detection algorithms with respect to a change in the underlying word embeddings: we apply the intersection@k metric to two runs of the usage-change detection algorithm on the same corpus-pair, where each run is based on a different run of the underlying word embedding algorithm.
4 The Predominant Approach
The most prominent method for detecting usage change is that of Hamilton et al. (2016b), originally applied to detect shifts in dominant word senses across time. It is still the predominant approach in practice, with recent works building upon it (Yao et al., 2018; Rudolph and Blei, 2018). This method was also shown to be the best performing one among several others (Schlechtweg et al., 2019).
It works by training word embeddings on the two corpora, aligning the spaces, and then ranking the words by the cosine-distance between their representations in the two spaces, where large distance is expected to indicate significant change in meaning. We refer to this method as AlignCos.
The alignment is performed by finding an orthogonal linear transformation  $Q$  that, when given matrices  $X$  and  $Y$ , projects  $X$  to  $Y$  while minimizing the squared loss:
$$
Q = \arg \min  _ {Q} | | Q X - Y | | _ {2}, \quad \text {s . t .} Q \text {i s o r t h o g o n a l}
$$
The rows of  $X$  correspond to embeddings of words in space A, while the rows of  $Y$  are the corresponding embeddings in space B. This optimization is solved using the Orthogonal Procrustes (OP) method (Schönemann, 1966), that provides a closed form solution.
Vector space alignment methods are extensively studied also outside of the area of detecting word change, primarily for aligning embedding spaces across language pairs (Xing et al., 2015; Artetxe et al., 2018b; Lample et al., 2018a; Artetxe et al., 2018a). Also there, the Orthogonal Procrustes method is taken to be a top contender (Lample et al., 2018b; Kementchedjhieva et al., 2018).
4.1 Shortcomings of the alignment approach
Self-contradicting objective. Note that the optimization procedure in the (linear) alignment stage attempts to project each word to itself. This includes words that changed usage, and which therefore should not be near each other in the space. While one may hope that other words and the linearity constraints will intervene, the method may succeed, by mistake, to project words that did change usage next to each other, at the expense of projecting words that did not change usage further apart than they should be. This is an inherent problem with any alignment based method that attempts to project the entire vocabulary onto itself.
Requires non-trivial filtering to work well. In addition, the alignment-based method requires nontrivial vocabulary filtering to work well. For example, Hamilton et al. (2016b) extensively filter proper nouns. Indeed, without such filtering, proper-nouns dominate the top of the changed words list. This does not indicate real word usage change, but is an artifact of names being hard to map across embedding spaces. In that respect, it makes sense to filter proper nouns. However, some cases of word usage change do involve names. For example, the word "Harlem", which is used as either a name of a neighborhood in NY or as a name of a basketball team, was detected by our method as a word whose usage changed between tweets of celebrities with different occupations ( $\S 7.1$ ).
Not stable across runs. As we discuss in Section 3 and show in Section 7.2, the approach is not very stable with respect to different random seeds in the embeddings algorithm.
2This is also indicated by the large number of citations: 350 according to Google Scholar.
Age	Gender	Occupation	Day-of-week	Hebrew	French
young	older	male	female	creator	sports	performer	weekday	weekend	2014	2018	2014	2018
#words	58M	116M	293M	126M	87M	132M	126M	142M	114M	42M	155M	867M	1B
*tweets	5M	8M	23M	9M	6M	11M	10M	9M	7M	4M	13M	82M	104M
#vocab	42K	73K	114K	69K	63K	66K	69K	81K	72K	84K	187K	263K	350K
Table 1: Statistics of the different splits.
5 Nearest Neighbors as a Proxy for Meaning
Rather than attempting to project two embedding spaces into a shared space (which may not even map 1:1), we propose to work at the shared vocabulary space. The underlying intuition is that words whose usage changed are likely to be interchangeable with different sets of words, and thus to have different neighbors in the two embedding spaces. This gives rise to a simple and effective algorithm: we represent each word in a corpus as the set of its top  $k$  nearest neighbors (NN). We then compute the score for word usage change across corpora by considering the size of the intersection of the two sets (not to be confused with intersection@  $k$  defined in Section 3):
$$
\operatorname {s c o r e} ^ {k} (w) = - \left| N N _ {1} ^ {k} (w) \cap N N _ {2} ^ {k} (w) \right| \tag {2}
$$
where  $NN_{i}^{k}(w)$  is the set of k-nearest neighbors of word  $w$  in space  $i$ . Words with a smaller intersection are ranked higher as their meaning-change potential.
We only consider the words in the intersection of both vocabularies, as words that are rare in one of the corpora are easy to spot using the frequency in the two spaces, and do not neatly fit the definition of usage change.
Note that our method does not require extensive filtering of words - we only filter words based on their frequency in the corpus<sup>3</sup>.
We use a large value of  $k = 1000^4$  in practice, because large neighbor sets are more stable than small ones (Wendlandt et al., 2018), leading to improved stability for our algorithm as well.
Limitations Similar to previous methods, our method assumes high quality embeddings, and
hence also a relatively large corpus. Indeed, in many cases we can expect large quantities of data to be available to the user, especially when considering the fact that the data needed is raw text rather than labeled text. Using a limited amount of data results in lower quality embeddings, but also with smaller vocabulary size, which might affect our method. For high-quality embeddings with small vocabulary sizes, we believe that changing  $k$  accordingly should suffice. Naturally, results will likely degrade as embeddings quality deteriorate.
It is also important to note that, like previous approaches, our method does not attempt to provide any guarantees that the detected words have indeed undergone usage change. It is only intended to propose and highlight candidates for such words. These candidates are meant to later be verified by a user who needs to interpret the results in light of their hypothesis and familiarity with the domain. Unlike previous methods, as we discuss in Section 7.4, our method also provides intuitive means to aid in such an interpretation process.
We compare our proposed method (NN) to the method of Hamilton et al. (2016b) described in Section 4 (AlignCos), in which the vector spaces are first aligned using the OP algorithm, and then words are ranked according to the cosine-distance between the word representation in the two spaces. This method was shown to outperform all others that were compared to it by Schlechtweg et al. (2019).
We demonstrate our approach by using it to detect change in word usage in different scenarios. We use the following corpora, whose statistics are listed in Table 1.
We consider three demographics-based distinctions (age, gender, occupation), a day-of-week
<sup>5</sup>Some extensions may yield improved results (filtering out proper names, as done in Hamilton et al. (2016b), or jointly learning and aligning the spaces (Bamler and Mandt, 2017; Rudolph et al., 2017; Rudolph and Blei, 2018; Yao et al., 2018), but we stick to this setting as it is the most general out of this line of work, and the one most commonly used in practice, for which an open implementation is available.
3For English experiments we also filter stopwords according to the predefined list from NLTK. 4While this value may seem arbitrary, we tested several values in that range which yielded very similar results. However, the appropriate range may change when used with smaller corpora, or substantially different vocabulary sizes. We consider  $k$  to be the only hyperparameter of our method, and note that it is rather easy to set.
based distinction, and short-term (4y) diachronic distinctions. We also compare to the longer-term (90y) diachronic setup of Hamilton et al. (2016b), which is based on Google books.
Author Demographics The Celebrity Profiling corpus (Wiegmann et al., 2019) consists of tweets from celebrities along with their traits such as age, gender and occupation. Based on these labels, we create the following splits: (1) Age: Young (birthyear 1990-2009) vs. Older (birthyear 1950-1969); (2) Gender: Male vs. Female; (3) Occupation: pairwise splits with Performer, Sports and Creator.
Day-of-week Yang and Leskovec (2011) collect 580 million tweets in English from June 2009 to February 2010, along with their time-stamps. As this is a fairly large corpus, we consider the tweets of a single month (November 2009). We create a split based on the Day-of-Week: weekday (tweets created on Tuesday and Wednesday) vs. weekend (tweets created on Saturday and Sunday). We remove duplicated tweets, as preliminary experiments revealed odd behavior of the representations due to heavily duplicated spam tweets.
French Diachronic (4y, tweets) Abitbol et al. (2018) compile a collection of tweets in French between the years 2014 and 2018. The authors utilize several heuristics based on the users' spatial information to consider tweets from users based in French territory only. We use the 2014 and 2018 portions of the data, and create a split accordingly.
Hebrew Diachronic (4y, tweets) The Hebrew data we use is taken from a collection of Hebrew tweets we collected for several consecutive years, up to 2018. The collection was performed by using the streaming API and filtering for tweets containing at least one of the top 400 most frequent Hebrew words. We use the 2014 and 2018 portions of the data, and create a split accordingly.
English Diachronic (90y, books) For diachronic study on English corpora, we make use of the embeddings trained on Fiction from Google Books (Davies, 2015) provided by the authors of Hamilton et al. (2016b), specifically for the two years, 1900 and 1990. These embeddings are originally aligned using Orthogonal Procrustes and the words whose relative frequencies are above  $10^{-5}$  in both the time periods are ranked using cosine distance.
6.1 Implementation details
Tokenization and Word Embeddings We use 300 dimensions word2vec vectors with 4 words context window. Further details of embeddings algorithm and tokenization are available in the appendix.
Vocabulary and Filtering We perform frequency-based filtering of the vocabulary, removing stop words (the most frequent 200 words for each corpus, as well as English stop words as defined in nltk), as well as low frequency words (we discard the  $20\%$  least frequent words in each corpus, and require a minimum of 200 occurrences).
Notably, we do not perform any other form of filtering, and keep proper-nouns and person-names intact.
We consider neighbors having a raw frequency greater than 100 and identify 1000 such nearest neighbors ( $k = 1000$ ) to perform the intersection.
7.1 Qualitative Evaluation: Detected Words
We run our proposed method and AlignCos (Hamilton et al., 2016b) on the different scenarios described in Section 6, and manually inspect the results. While somewhat subjective, we believe that the consistent success on a broad setting, much larger than explored in any earlier work, is convincing. We provide examples for two of the setups (English Diachronic and Performer vs. Sports), with the rest of the setups in the appendix. For each one, we list a few interesting words detected by the method, accompanied by a brief explanation (according to the neighbors in each corpus).
In addition, we depict the top-10 words our method yields for the Age split (Table 2), accompanied by the nearest neighbors in each corpus (excluding words in the intersection), to better understand the context. For comparison, we also mention the top-10 words according to the AlignCos method. Similar tables for the other splits are provided in the Appendix.
Across all splits, our method is able to detect high quality words as words that undergo usage change, most of them easily explained by their neighboring words in the two corpora. As expected, we see that the AlignCos method (Hamilton et al.,
AGE (YOUNG VS. OLDER)
NN	neighbors in each corpus
dem	dese, yuh, them, nuh, dey, ayye, dats, tha, betta, fuk repub, democrats, centrist, manchin, primaries, party's, alp, dfl, gopers, repubs
dam	damm, mannnnn, mannnn, mane, huh, ahh, oo, buggin, koo, mannn dams, basin, river, dredging, reservoir, drainage, wastewater, sewerage, refinery, canal
rep	reppin, wear, allegiance, all-american, wildcat, alumni, tryout, hoosier, recruit, ua sen., congresswoman, chairwoman, co-chairs, gazelka, salazar, amb, comptroller, staffer, cong
assist	points, shutout, scoresheet, scored, pts, hattrick, sheet, nil, sacks, assisting, contact, coordinate, locating, coordinating, administer, equip, consular, deploy, locate
pr	cameron, -pr, erik, lap, sargeant, laps, tundra, teamjev, caution, restart stunt, puerto, promotional, rico, creative, ploy, hire, spin, freelance, fema
fr	frfr, forreal, foreal, lmaooo, madd, tho, bck, bruhh, lmao, fwm pavone, incl, from, wrk, ger, joseph, covey, env, w, ans
joint	jawn, fusion, scorpion, sumn, spot, db, cb, joints, mgmt, fye high-level, convened, minsk, two-day, bilateral, counter-terrorism, delegations, asean, convene, liaison
mega	, fantastic, simulator, macau, lotus, fuji, bmw, awesome, mclaren, fab gujarat, becos, multi-billion, gta, rupees, dollar, maharashtra, major, crores, multi-million
flow	beard, vibin, jeezy, drizzy, lite, mohawk, dreads, sauna, boomin, vibe illicit, influx, accumulation, moisture, absorb, overwhelm, heart's, drains, curtail, diverting
icymi	superintendent, bureau, commissioner, spokesman, exec, state's, prosecutor, reuters, montgomery, conway re-upping, reichert, newsmakers, sherrod, column, arizona's, otl, holcomb, rundown, wrap-up
AlignCos top-10	leo, whip, savage, nd, cole, pb, ace, carter, fr, bb
Table 2: Top-10 detected words from our method (NN) vs. AlignCos method (last row), for corpus split according to the age of the tweet-author. Each word from our method is accompanied by its top-10 neighbors in each of the two corpora (Young vs. Older).
2016b) is highly sensitive to names, featuring many in the top-10 lists across the different splits. As opposed to AlignCos, our method is robust to global changes in the embedding space, since it looks at many neighbors. As a result, it is not sensitive to groups of words that "move together" in the embedding space (which might be the case with names).