SlowGuess/ACL_2020_OCR huggingface dataset:
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237.
Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1267-1273, Minneapolis, Minnesota. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-training. Technical report, OpenAI.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. Technical report, OpenAI.
Miguel A Ré and Rajeev K Azad. 2014. Generalization of Entropy Based Divergence Measures for Symbolic Sequence Analysis. PloS one, 9(4):e93532.
Alex Rosenfeld and Katrin Erk. 2018. Deep Neural Models of Semantic Shift. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 474-484.
Peter J. Rousseeuw. 1987. Silhouettes: A Graphical Aid to the Interpretation and Validation of Cluster Analysis. Journal of Computational and Applied Mathematics, 20:53-65.
Maja Rudolph and David Blei. 2018. Dynamic embeddings for Language evolution. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, pages 1003-1011. International World Wide Web Conferences Steering Committee.
Dominik Schlechtweg, Sabine Schulte im Walde, and Stefanie Eckmann. 2018. Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), volume 2, pages 169-174.
Hinrich Schütze. 1998. Automatic Word Sense Discrimination. Computational Linguistics, 24(1):97-123.
Xuri Tang. 2018. A State-of-the-Art of Semantic Change computation. Natural Language Engineering, 24(5):649-676.
Xuri Tang, Weiguang Qu, and Xiaohe Chen. 2016. Semantic Change Computation: A Successive Approach. World Wide Web, 19(3):375-415.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. 2019. Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings. In Proceedings of the 15th Conference on Natural Language Processing, KONVENS 2019, Erlangen, Germany.
Derry Tanti Wijaya and Reyyan Yeniterzi. 2011. Understanding Semantic Change of Words over Centuries. In Proceedings of the 2011 International Workshop on Detecting and Exploiting Cultural Diversity on the Social Web, pages 35-40. ACM.
Yang Xu and Charles Kemp. 2015. A Computational Evaluation of Two Laws of Semantic Change. In CogSci.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In Proceedings of the IEEE International Conference on Computer Vision, pages 19-27.
This appendix includes supplementary materials related to Section 5.1.
A.1 New Dataset of Similarity Judgements
Obtaining usage pairs For each of our 16 target words, we sample five usages from each of their usage types, one for every 20-year period in the last century of COHA. When a usage type does not occur in a time interval, we uniformly sample an interval from those that do contain occurrences of that usage type. All possible pairwise combinations (without replacement) are generated for each target word, resulting in a total of 3,285 usage pairs.
Crowdsourced annotation We use the crowdsourcing platform Figure Eight (since then acquired by Appen<sup>9</sup>) to collect five similarity judgements for each of these usage pairs. To control the quality of the similarity judgements, we select Figure Eight workers from the pool of most experienced contributors, we require them to be native English
speakers and to have completed a test quiz consisting of 10 similarity judgements. For this purpose, 170 usage pairs were manually annotated by the first author with 1 to 3 acceptable labels. The compensation scheme for the raters is based on an average wage of 10 USD per hour.
Figures 4 and 5 (on the next pages) show the full instructions given to the annotators and Figure 3 illustrates a single annotation item.
Please read carefully the following two sentences where the word [[federal]] occurs:
- robert m. hitchcock, who prosecuted the amerasia case in 1945, testified today that he had been gravely handicapped because the government's best evidence had been produced by illegal seizures by [[federal]] agents. the prosecution, he asserted, was in fact fortunate under the circumstances to have done as well as it did.
- there should be such a fire every saturday afternoon at the same time with the same actual damage. this time it was the records and documents of the [[federal]] trade commission, said to be " priceless." "also the reels of official motion pictures of historical or technical value.
How similar are the two occurrences of [[federal]]? (required)
1. Unrelated
$\bigcirc$  2. Distantly related
3. Closely related
$\bigcirc$  4. Identical
Cannot decide (please use this option as little as possible)
Figure 3: An annotation item on the Figure Eight crowdsourcing platform.
A.2 Correlation Results
We measure Spearman's rank correlation between human- and machine-generated usage similarity matrices using the Mantel test and observe a significant positive correlation for 10 out of 16 words. Table 2 presents the correlation coefficients and  $p$ -values obtained for each word.
ρ	p
federal	0.131	0.001
spine	0.195	0.032
optical	0.227	0.003
compact	0.229	0.002
signal	0.233	0.008
leaf	0.252	0.001
net	0.361	0.001
coach	0.433	0.007
sphere	0.446	0.002
mirror	0.454	0.027
card	0.358	0.055
virus	0.271	0.159
disk	0.183	0.211
brick	0.203	0.263
virtual	-0.085	0.561
energy	0.002	0.990
Table 2: Correlation results per word.
$^{9}$ https://appen.com
Each question includes two sentences. Both sentences contain a target word between double brackets, as in: [[target]]. Your task is to rank the similarity of the two usages of the target word according to the following scale:
1. unrelated
2. distantly related
3. closely related
4. identical
IMPORTANT: your task is to evaluate the similarity of the two usages of the same word, not the similarity of the two sentences in general.
If you are unable to choose a label because you do not understand the sentences, select the option "cannot decide". Please try to use this option as little as possible!
You will see two sentences. Both contain the target word marked by double brackets; in this example it's the word [[current]].
Read the sentences carefully:
- in any case, it's not a question of electrocution. we can arrange a relay which will break the [[current]] at the instant of application of weight. if the robot should place his weight on it, he wo n't see.
already, while it was still a blueprint, they were proud of their idea, of its simple clean lines and undeniable originality -- it owed nothing in its conception to any of the [current] models of revolutionary strategy . the japanese red army comrades , whadi haddad and his pflp contingent , even the matchless " carlos " could only admire .
And then select how similar the two usages of the word [[current]] are:
1. unrelated
2. distantly related
3. closely related
4. identical
5. (cannot decide)
You can choose only one label. Please try to use the option "cannot decide" as little as possible.
Why do texts look weird?
The sentences you'll read don't look like they were taken from a book. This is because they have gone through some text processing. You should not be concerned nor influenced in your decisions by the fact that:
- all words are lowercase (written in small letters), even proper names or the pronoun "I"
- whitespaces may appear where you don't expect them (e.g. before a comma) and may sometimes not appear where you'd expect them (e.g. between words)
strange characters and words occasionally appear
- some words are misspelled
a few words are missing
- the target word may appear multiple times (but your judgement should be about the occurrence signalled by the[[ ] marker)
Please simply ignore these aspects while labelling!
Figure 4: Annotation instructions (part 1).
What do the similarity labels mean? More examples
Let's now look at examples for all four labels. Remember that you are evaluating the similarity of two word usages—not the overall similarity of the two sentences!
1. How similar are these two usages of [[current]]?
- prices of the leading issues . considering past earnings records , are apparently on a conservative basis measured by [[current]] market valuations in other groups . on the other hand there is no particular speculative incentive for operations in this group , with all signs pointing to a lower volume of sales in the last half of the year .
one of the weirdest was the disappearance of anchovies off the coast of peru . why this happened is still unclear . one theory is that the cause was the 1972 - 73 invasion of a warm-water [ [current] ] called el nino , which upset the ecology of the coldwater humboldt current , drastically reducing the supply of 119 economics in plain english plankton and other nutrients on which anchovies ( as well as whales ) feed .
UNRELATED.
In the first sentence, "current" means being most recent or occurring at the present time. In the second sentence, "current" refers to a flow of water within a lake or an ocean. These two meanings have no properties in common; it is not possible to explain one usage in terms of the other.
2. How similar are these two usages of [[current]]?
- it is quite possible to arrive at the right conclusions for the wrong reasons, just as it is possible to ignore history but not to repeat it. thus, the summers book represents an important [[current]] of thought in the u.s. military, which rightly argues that the vietnam defeat was not the fault of the military; never again should young americans be sent into battle without public backing and a clear definition of the goals of the military engagement.
get busy with that I-tube ! If you do n't have it apart , cleaned , and together again before the day is out , i 'll coagulate your brains with alternating [[current]] . " not a robot moved !
DISTANTLY RELATED.
In the first sentence, "current" refers to a feeling or idea that exists within a group of people. In the second sentence, "current" refers to a flow of electricity. Although these two meanings of "current" are different, they are related as they do share some properties: for example, both currents of thought and electronic current can flow, and they are both often the result of an interplay of forces. This is why connecting the two meanings in a sentence results in a perhaps sophisticated but understandable statement: (A) "a new current of thought is flowing through the circuits of parliament".
3. How similar are these two usages of [[current]]?
one of the weirdest was the disappearance of anchovies off the coast of peru . why this happened is still unclear . one theory is that the cause was the 1972 - 73 invasion of a warm-water [ [current] ] called el nino , which upset the ecology of the coldwater humboldt current , drastically reducing the supply of 119 economics in plain english plankton and other nutrients on which anchovies ( as well as whales ) feed
he had survived by managing in a stupor to drag himself into a windowless shed behind his house. col. michael wiener , an israeli army doctor , said many of the survivors in the valley below the lake , such as in souboum , may simply have been in an air [ [current] ] that did not have any poison , while someone standing only a few yards away may have been killed. dr. weiner , the head of a 17-member rescue unit that came to camoer team to nkamba , about 100 miles northeast of bamenda .
CLOSELY RELATED.
In the first sentence, "current" refers to a flow of water within a lake or an ocean. In the second sentence, "current" refers to a steady flowing movement of air. These two meanings are closely related: both refer to a steady and continuous flowing movement of some physical element. As in the previously encountered example (A), we can construct a sentence that relates the two usages: (B) "I can't remember whether El Nino is the name of an ocean or a wind current". Note, however, that the meanings are still ultimately different: a flow of water is not a flow of air.
4. How similar are these two usages of [[current]]?
dell's shares, on the other hand, go for 26 times projected 2004 earnings-but its business is three times as profitable as apple 's. The company 's supporters say [ [current] ] profits matter little because jobs has proved time and time again that he can create new products and trailblaze markets. that may be so, but as transamerica portfolio manager chris bonavico , who does n't own apple ' s stock , notes , " Apple will remain a company that is neat from a product and consumer standpoint but crap from an investor standpoint .
- prices of the leading issues . considering past earnings records , are apparently on a conservative basis measured by [[current]] market valuations in other groups . on the other hand there is no particular speculative incentive for operations in this group , with all signs pointing to a lower volume of sales in the last half of the year .
IDENTICAL.
In both sentences, "current" means being most recent, up-to-date, or occurring at the present time. The two meanings are identical because they share virtually all properties, as can be seen from the following example. In the sentence (C) "I can't remember whether Donald Trump is the current president or vice-president of the United States", the meanings of "current" are ultimately the same regardless of whether Trump is president or vice-president. (Note the difference with respect to the constructed sentence (B) above.)
Now you're ready to start!
Figure 5: Annotation instructions (part 2).