SlowGuess/ACL_2020_OCR huggingface dataset:
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In NAACL-HLT.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019a. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR.
Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Jason Phang, Edouard Grave, Haokun Liu, Najoung Kim, Phu Mon Htut, Thibault Févry, Berlin Chen, Nikita Nangia, Anhad Mohananey, Katharina Kann, Shikha Bordia, Nicolas Patry, David Benton, Ellie Pavlick, and Samuel R. Bowman. 2019b. jiant 1.2: A software toolkit for research on general-purpose text understanding models. http://jiant.info/.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Felix X. Yu, Aditya Bhaskara, Sanjiv Kumar, Yunchao Gong, and Shih-Fu Chang. 2017. On binary embedding using circulant matrices. JMLR, 18(1):5507-5536.
Felix X Yu, Sanjiv Kumar, Henry Rowley, and Shih-Fu Chang. 2015. Compact nonlinear maps and circulant extensions. arXiv preprint arXiv:1503.03893.
We now describe the embeddings (Appendix A.1), tasks (Appendix A.2), and models (Appendix A.3) we use in our experiments in more detail.
A.1 Embeddings
We compare the performance of BERT contextual embeddings with GloVe embeddings and random embeddings. We specifically use 768-dimensional  $\mathrm{BERT}_{\mathrm{BASE}}$  WordPiece embeddings, 300-dimensional GloVe embeddings, and 800-dimensional random embeddings. We freeze each set of embeddings prior to training, and do not fine-tune the embeddings during training. The random embeddings are normalized to have the same Frobenius norm as the GloVe embeddings. We now describe how we use circulant matrices to reduce the memory requirement for the random embeddings.
Circulant Random Embeddings To store a random  $n$ -by- $d$  matrix in  $O(n)$  memory instead of  $O(nd)$ , we use random circulant matrices (Yu et al., 2017). Specifically, we split the  $n$ -by- $d$  matrix into  $\frac{n}{d}$  disjoint  $d$ -by- $d$  sub-matrices (assuming for simplicity that  $d$  divides  $n$  evenly), where each submatrix is equal to  $CD$ , where  $C = \operatorname{circ}(c) \in \mathbb{R}^{d \times d}$  is a circulant matrix based on a random Gaussian vector  $c \in \mathbb{R}^d$ , and  $D = \operatorname{diag}(r) \in \mathbb{R}^{d \times d}$  is a diagonal matrix based on a random Radamacher vector  $r \in \{-1, +1\}^d$ . Note that a circulant matrix  $\operatorname{circ}(c)$  is defined as follows:
$$
\operatorname {c i r c} (c) := \left( \begin{array}{c c c c c} c _ {0} & c _ {d} & \ldots & c _ {2} & c _ {1} \\ c _ {1} & c _ {0} & \ldots & c _ {3} & c _ {2} \\ \ldots & \ldots & \ddots & \ldots & \ldots \\ c _ {d - 1} & c _ {d - 2} & \ldots & c _ {0} & c _ {d} \\ c _ {d} & c _ {d - 1} & \ldots & c _ {1} & c _ {0} \end{array} \right).
$$
Random circulant embeddings have been used in the kernel literature to make kernel approximation methods more efficient (Yu et al., 2015). For downstream training and inference, one can simply store the  $d$ -dimensional  $c$  and  $r$  vectors for each of the  $\frac{n}{d}$  disjoint  $d$ -by- $d$  sub-matrices, taking a total of  $O(n)$  memory. Alternatively, one can simply store a single random seed  $(O(1)$  memory), and these  $c, r$  vectors can be regenerated on the fly each time a row of the embedding matrix is accessed. Note that in addition to being very memory efficient, random embeddings avoid the expensive pretraining process over a large language corpus.
We perform evaluations on three types of standard downstream NLP tasks: named entity recognition (NER), sentiment analysis, and natural language understanding. NER involves classifying each token in the input text as an entity or a non-entity, and further classifying the entity type for identified entities. We evaluate on the CoNLL-2003 benchmark dataset, in which each token is assigned a label of "O" (non-entity), "PER" (person), "ORG" (organization), "LOC" (location), or "MISC" (miscellaneous). Sentiment analysis involves assigning a classification label at the sentence level corresponding to the sentiment of the sentence. We evaluate on five binary sentiment analysis benchmark datasets including MR, MPQA, CR, SST, and SUBJ. We also evaluate on the benchmark TREC dataset, which assigns one of six labels to each input example. For natural language understanding, we use the standard GLUE benchmark tasks, and the GLUE diagnostic task.
A.3 Downstream Task Models
We use the following models and training protocols for the NER, sentiment analysis, and GLUE tasks:
NER: We use a BiLSTM task model with a CRF decoding layer, and we use the default hyperparameters from the flair (Akbik et al., 2019) repository:[8] 256 hidden units, 32 batch size, 150 max epochs, and a stop-condition when the learning rate decreases below 0.0001 with a decay constant of 0.5 and patience of 4. In our evaluation, we report micro-average F1-scores for this task.
Sentiment analysis: We use the architecture and training protocol from Kim (2014), using a CNN with 1 convolutional layer, 3 kernel sizes in  $\{3,4,5\}$ , 100 kernels, 32 batch size, 100 max epochs, and a constant learning rate. We report the validation error rates in evaluations of each task.
GLUE: We use the Jiant (Wang et al., 2019b) implementation of a BiLSTM with 1024 hidden dimensions, 2 layers, 32 batch size, and a stop-condition when the learning rate decreases below 0.000001 with a decay constant of 0.5 and patience of 5. We consider the following task-specific performance metrics: Matthews correlation for CoLA, MNLI, and the diagnostic task, validation F1-score for MRPC and QQP, and validation accuracy for QNLI and RTE.
https://github.com/zalandoresearch/flair.
B Impact of Training Data Volume
We now provide additional details regarding our experiments on the impact of training set size on performance (Appendix B.1), our complete set of empirical results from these experiments (Appendix B.2), as well as theoretical support for the strong performance of random embedding models in these experiments, when trained with sufficient downstream data (Appendix B.3).
B.1 Additional Experiment Details
For each task, we evaluate performance using five fractions of the full training dataset, to understand how the amount of training data affects performance:  $\{\frac{1}{4^4},\frac{1}{4^3},\frac{1}{4^2},\frac{1}{4^1},1\}$ . For each fraction  $c$ , we randomly select a subset of the training set of the corresponding size, and replicate this data  $1 / c$  times; we then train models using this redundant dataset, using the model architectures and training protocols described in Appendix A.3. In downstream training we perform a separate hyperparameter sweep of the learning rate at each fraction of the training data, and select the best learning rate for each embedding type. We use the following lists of learning rates for the different tasks:
- NER:  $\{.003, .01, .03, .1, .3, 1, 3\}$ .
- Sentiment analysis:  $\{1\mathrm{e} - 5,3\mathrm{e} - 5,1\mathrm{e} - 4,3\mathrm{e} - 4,$ $1\mathrm{e} - 3,3\mathrm{e} - 2,1\mathrm{e} - 2\}$
- GLUE:  $\{1\mathrm{e} - 6,3\mathrm{e} - 6,1\mathrm{e} - 5,3\mathrm{e} - 5,1\mathrm{e} - 4,3\mathrm{e} - 4,$ $1\mathrm{e} - 3\}$
B.2 Extended Results
In Figures 2 and 3, we show the performance of random, GloVe, and BERT embeddings on all the NER, sentiment analysis, and GLUE tasks, as we vary the amount of training data. We can see that across most of these results:
- Non-contextual embedding performance improves quickly as the amount of training data is increased.
- The gap between contextual and noncontextual embeddings often shrinks as the amount of training data is increased.
- There are many tasks for which random and GloVe embeddings perform relatively similarly to one another.
B.3 Theoretical Support for Random Embedding Performance
To provide theoretical support for why, given sufficient training data, a model trained with random embeddings might match the performance of one trained with pretrained embeddings, we consider the simple setting of Gaussian process (GP) regression (Rasmussen and Williams, 2006). In particular, we assume that the prior covariance function for the GP is determined by the pretrained embeddings, and show that as the number of observed samples from this GP grows, the posterior distribution gives diminishing weight to the prior covariance function, and eventually depends solely on the observed samples. Thus, if we were to calculate the posterior distribution using an inaccurate prior covariance function determined by random embeddings, this posterior would approach the true posterior as the number of observed samples grew.
More formally, for a fixed set of words  $\{w_{1},\ldots ,w_{n}\}$  with pretrained embeddings  $\{x_{1},\dots,x_{n}\} \subset \mathbb{R}^{d}$ , we assume that the "true" regression label vector  $y^{*}\in \mathbb{R}^{n}$  for these words is sampled from a zero-mean multivariate Gaussian distribution  $y^{*}\sim \mathcal{N}(0,K)$ , where the entries  $K_{ij}\coloneqq k(x_i,x_j)$  of the covariance matrix  $K$  are determined based on the similarity  $k(x_{i},x_{j})$  between the pretrained embeddings  $x_{i},x_{j}\in \mathbb{R}^{d}$  for words  $i$  and  $j$ . We then assume that we observe  $m$  noisy samples  $(y_{1},\ldots ,y_{m})$  of the "true" label vector  $y^{*}$ , where each  $y_{i}\in \mathbb{R}^{n}$  is an independent sample from  $\mathcal{N}(y^*,\sigma^2 I)$ . To summarize:
$$
y ^ {*} \sim \mathcal {N} (0, K),
$$
$$
y _ {1}, \ldots , y _ {m} \sim \mathcal {N} (y ^ {*}, \sigma^ {2} I).
$$
The question then becomes, what is the posterior distribution for  $y^{*}$  after observing  $(y_{1},\ldots ,y_{m})$ ? The closed form solution for this posterior is as follows:
$$
p \left(y ^ {*} \mid y _ {1}, \dots , y _ {m}\right) = \mathcal {N} (\bar {y} _ {m}, \bar {K} _ {m}), \text {w h e r e}
$$
$$
\bar {y} _ {m} = K \left(K + \frac {\sigma^ {2}}{m} I\right) ^ {- 1} \left(\frac {1}{m} \sum_ {i = 1} ^ {m} y _ {i}\right),
$$
$$
{\bar {K} _ {m}} = {K \left(K + \frac {\sigma^ {2}}{m} I\right) ^ {- 1} \frac {\sigma^ {2}}{m} I.}
$$
Importantly, we observe that as  $m\to \infty$  ,that  $\bar{y}_m\rightarrow y^*$  (because  $K\big(K + \frac{\sigma^2}{m} I\big)^{-1}\rightarrow I$  and
As an example, we could have  $k(x_{i},x_{j}) \coloneqq \exp \left(-\| x_{i} - x_{j}\|^{2} / (2\sigma^{2})\right)$  be the Gaussian kernel.
Figure 2: Performance of random, GloVe, and BERT embeddings on the NER (top row) and sentiment analysis (bottom three rows) tasks as we vary the amount of training data.
Figure 3: Performance of random, GloVe, and BERT embeddings on GLUE tasks as we vary the amount of training data.
$\frac{1}{m}\sum_{i = 1}^{m}y_{i}\rightarrow y^{*})$  , and  $\bar{K}_m\to 0$  . Thus, if we were to compute the posterior distribution for this GP using an uninformative prior covariance function  $K^{\prime}$  determined by random embeddings  $\{x_1',\ldots ,x_n'\}$ $(K_{ij}^{\prime} = k(x_i^{\prime},x_j^{\prime}))$  , this posterior would approach the posterior computed from the "true" prior covariance function  $K$  as the number of observations  $m\to \infty$  . Thus, GP regression with an informative prior derived from the pretrained embeddings performs the same as GP regression with an uninformative prior derived from random embeddings, as the number of observed samples approaches infinity.
C Study of Linguistic Properties
We now describe in more detail how we define our metrics for the three linguistic properties for both NER and sentiment analysis tasks (Appendix C.1), as well as provide extended empirical results from our linguistic studies (Appendix C.2).
C.1 Linguistic Properties: Detailed Definitions
We define the metrics in detail below for our three linguistic properties: complexity of text structure (Appendix C.1.1), ambiguity in word usage (Appendix C.1.2), and prevalence of unseen words (Appendix C.1.3). To provide further intuition for these metrics, in Figure 4 we present actual examples from the CoNLL-2003 NER task and the CR sentiment analysis task for each of the metrics, along with the errors made by each embedding type on these examples.
C.1.1 Complexity of Text Structure
We define the following metrics for NER and sentiment analysis to measure the structural complexity of an entity or sentence, respectively:
NER: For NER, we measure the linguistic complexity of an entity in terms of the number of tokens in the entity (e.g., "George Washington" spans 2 tokens), as correctly labeling a longer entity requires understanding the relationships between the different tokens in the entity name.
Sentiment analysis: For sentiment analysis, we need a sentence-level proxy for structural complexity; toward this end, we leverage the dependency parse tree for each sentence in the dataset. In particular, we characterize a sentence as more structurally complex if the average distance between
dependent words is higher. We consider this definition because long-range dependencies generally require more contextual information to understand. To avoid diluting the average dependency length, we do not include dependencies where either the head or the tail of the dependency is a punctuation or a stop word.
As an example, consider the sentence "George Washington, who was the first president of the United States, was born in 1732". In this sentence, there is a dependence between "George" and "born" of length 14, because there are 13 intervening words or punctuations. This is a relatively large gap between dependent words, and would increase the average dependency length the sentence.
C.1.2 Ambiguity in Word Usage
The next linguistic property we consider is the degree of ambiguity in word usage within a task. To measure the degree of ambiguity in the language, we define the following metrics in the context of NER and sentiment analysis:
NER: For NER as a word-level classification task, we consider the number of labels (person, location, organization, miscellaneous, other) a token appeared with in the training set as a measure of its ambiguity (e.g., "Washington" appears as a person, location, and organization in the CoNLL-2003 training set). For each token in the validation set, we enumerate the number of tags it appears with in the training set.
Sentiment analysis: For sentiment analysis, we measure the ambiguity of a sentence by considering whether the words in the sentence generally appear in positive or negative sentences in the training data. For the binary case, we take the average over words in the sentence of the unigram probability that a word is positive, and then compute the entropy of a coin flip with this probability of being "heads". More specifically, to compute the unigram probability  $p(+1|w)$  for a word  $w$ , we measure the fraction of training sentences containing  $w$  which are positive. Our ambiguity metric is then defined for a sentence  $S$  as
$$
H \left(\frac {1}{| S |} \sum_ {w \in S} p (+ 1 \mid w)\right),
$$
where  $H(p) = -p\log_2(p) - (1 - p)\log_2(1 - p)$  is the entropy of a coin flip with probability  $p$ . Intuitively, sentences with generally positive (or negative) words will have low entropy, and be easy to classify even with non-contextual embeddings.
<sup>10</sup>We use the StanfordNLP dependency parser for our metric: https://pypi.org/project/stanfordnlp/.
Metric	NER CoNLL-2003 Task Validation Set Example
Complexity	4-token entity
‘The FOMC (Federal Open Market Committee) has been forecasting a slowing in economic activity and moderating household demand will have a large impact on overall economic growth,’ Naroff said in a written comment.
Unseen	unseen tokens
| Buddy Groom pitched a perfect ninth inning.
Ambiguity	0 tags
| The Texan blasted in two aces to hold serve at 5-2 and then converted his eighth match point for victory when Washington found the net with another backhand from 40-0.
| 3 tags {Location, Person, Organization}
Metric	Sentiment Analysis CR Task Validation Set Example
Complexity	3 5 7 A huge beautiful color screen which can not been scratched at all even with the rough handling I give it. Shows dependency lengths between non-stop-words (highlighted)
Unseen	Anyhow some demerits of this phone, the phone has a slow processor and the ear volume is lower than 6610 but then the 6610 has the best ear volume among most nokia variants. Words unseen at train time (highlighted)
Ambiguity	0.29 0.89 0.70 0.50 0.64 1.00 5) Lack of good accessories - this is more of an oversight of creative 's marketing. Shows proportion with which non stop-words (highlighted) occur in positive train example
Figure 4: Examples from the CoNLL-2003 NER task (above) and the CR sentiment analysis task (below) validation sets, to provide further intuition for the three linguistic properties. All of the examples above fall in the validation set slices that have metric values above the median, and are thus considered relatively difficult examples according to these linguistic metrics. For example, in the case of NER, (1) the "Federal Open Market Committee" is a relatively long, 4-token entity, (2) "Buddy" and "Groom" are both tokens that were not seen during training, and (3) "Washington" was seen in the training set with three different entity type labels (location, person, organization). In the case of the sentiment analysis examples, (1) the complexity metric sentence has several long dependences (lengths 3, 5, and 7) because it has numerous adjective, adverb, and noun modifiers, (2) the unseen metric sentence has four words that were not seen during training ("anyhow", "demerits", "processor", "variants"), and (3) the ambiguity metric sentence has words that were mainly positive during training ("good", "creative"), as well as words which were mainly negative during training ("lack"). We use empty vs. filled-in squares of different colors to show whether a given embedding type got an example correct vs. incorrect, respectively (see legend).
Category	BERT	Random	GloVe
LS	0.19	0.14	0.13
PAS	0.33	0.20	0.20
L	0.12	0.15	0.13
KCS	0.10	0.17	0.13
Overall	0.500	0.475	0.465
Table 3: The performance (Matthews correlation coefficients) of BERT, random, and GloVe embeddings across the four linguistic categories defined by the GLUE diagnostic task: lexical semantics (LS), predicate-argument structure (PAS), logic (L), and knowledge and common sense (KCS). We also include the overall diagnostic performance.
For non-binary sentiment tasks with  $C$ -labels (e.g.,  $C = 6$  for the TREC dataset), we consider the entropy of the average label distribution  $\frac{1}{|S|} \sum_{w \in S} p(y \mid w) \in \mathbb{R}^C$  over the words in the sentence. Here,  $p(y \mid w)$  is defined as the fraction of the sentences in the training set containing the word  $w$  which had the label  $y$ . Note that for stop words and punctuation, we always consider  $p(y \mid w)$  as the uniform distribution over the set of possible labels  $y$  (for both binary and non-binary classification tasks).
C.1.3 Prevalence of Unseen Words
We define the following metrics for the prevalence of unseen words for NER and sentiment analysis tasks:
NER: For a word in the NER validation set, we consider as our metric the inverse of the number of times the word appeared in the training data (letting  $1/0 \coloneqq \infty$ ). We consider the inverse of the number of training set appearances because intuitively, if a word appears fewer times in the training set, we expect it to be harder to correctly classify this word at test time—especially for non-contextual or random embeddings.
Sentiment analysis: For sentiment analysis, given a sentence, we consider as our metric the fraction of words in the sentence that were never seen during training. More specifically, we count the number of unseen words (that are not stop words), and divide by the total number of words in the sentence. Intuitively, sentences with many unseen words will attain high values for this metric, and will be difficult to classify correctly without prior (i.e., pretrained) knowledge about these unseen words.
C.2 Extended Results
We present the detailed results from our evaluation of the different embedding types on the GLUE diagnostic dataset (Appendix C.2.1), and extended validation of the linguistic properties we define in Section 3.3 (Appendix C.2.2).
C.2.1 GLUE Diagnostic Results
The GLUE diagnostic task facilitates a fine-grained analysis of a model's strengths and weaknesses in terms of how well the model handles different linguistic phenomena. The task consists of 550 sentence pairs which are classified as entailment, contradiction, or neutral. The GLUE team curated the sentence pairs to represent over 20 linguistic phenomena, which are grouped in four top-level categories: lexical semantics (LS), predicate-argument structure (PAS), logic (L), and knowledge and common sense (KCS). We follow the standard procedure and use the model trained on the MNLI dataset (using the random, GloVe, or BERT embeddings) to evaluate performance on the diagnostic task. We report the Matthews correlation coefficient (MCC) performance of the different embedding types on the four top-level categories in Table 3.
Our two key observations are: (1) the noncontextual embeddings (random and GloVe) perform similarly to one another across all four top-level categories; (2) the performance difference between contextual and non-contextual embeddings is most stark for the predicateargument (PAS) category, which includes phenomena that require understanding the interactions between the different subphrases in a sentence. Within PAS, the BERT embeddings attain a  $10+$  point improvement in MCC over random embeddings for sentences reflecting the following phenomena: Relative Clauses/Restrictivity, Datives, Nominalization, Core Arguments, Core Arguments/Anaphora/Coreference, and Prepositional Phrases.
C.2.2 GloVe vs. BERT Results
In Table 4, we replicate the results from Table 2, but instead of comparing BERT embeddings to random embeddings, we compare them to GloVe embeddings. We can see that for 11 out of 14 cases for the complexity and ambiguity metrics, the gap between contextual (BERT) and non-contextual (GloVe) performance is larger for the validation slices above the median than below; this aligns with our results comparing random and BERT embeddings.
Task	Complexity	Ambiguity	Unseen
Abs.	Rel.	Abs.	Rel.	Abs.	Rel.
NER (CoNLL)	+6.7	4.0	+5.9	3.3	-1.4	0.8
Sent. (MR)	-0.6	0.9	+6.5	2.5	-1.0	0.9
Sent. (SUBJ)	-1.8	0.6	+4.4	6.0	-1.3	0.6
Sent. (CR)	+1.2	1.5	-2.4	0.4	0.0	1.0
Sent. (SST)	+7.8	5.3	+6.0	3.2	-2.8	0.6
Sent. (TREC)	+2.2	1.4	+8.1	4.1	+3.7	1.8
Sent. (MPQA)	+6.6	-3.2	+2.9	-1.8	+0.4	3.0
Table 4: For our complexity, ambiguity, and unseen prevalence metrics, we slice the validation set using the median metric value, and compute the average error rates for GloVe and BERT on each slice. We show that the gap between GloVe and BERT errors is larger above than below the median in 11 out of 14 of the complexity and ambiguity results both in absolute (Abs.) and relative (Rel.) terms; however, on the unseen metrics, this only holds for 2 out of 7 cases, which suggests that GloVe embeddings are able to relatively effectively deal with unseen words.
Interestingly, this is only the case for 2 out of 7 of the cases for the unseen metrics. This is likely because both GloVe and BERT embeddings are able to leverage pretrained semantic information about unseen words to make accurate predictions for them, and thus perform relatively similarly to one another on unseen words.
2663