SlowGuess/ACL_2020_OCR huggingface dataset:
Simran Arora*, Avner May*, Jian Zhang, Christopher Ré
{simarora, avnermay, zjian, chrismre}@cs.stanford.edu
We study the settings for which deep contextual embeddings (e.g., BERT) give large improvements in performance relative to classic pretrained embeddings (e.g., GloVe), and an even simpler baseline—random word embeddings—focusing on the impact of the training set size and the linguistic properties of the task. Surprisingly, we find that both of these simpler baselines can match contextual embeddings on industry-scale data, and often perform within 5 to  $10\%$  accuracy (absolute) on benchmark tasks. Furthermore, we identify properties of data for which contextual embeddings give particularly large gains: language containing complex structure, ambiguous word usage, and words unseen in training.
In recent years, rich contextual embeddings such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) have enabled rapid progress on benchmarks like GLUE (Wang et al., 2019a) and have seen widespread industrial use (Pandu Nayak, 2019). However, these methods require significant computational resources (memory, time) during pretraining, and during downstream task training and inference. Thus, an important research problem is to understand when these contextual embeddings add significant value vs. when it is possible to use more efficient representations without significant degradation in performance.
As a first step, we empirically compare the performance of contextual embeddings with classic embeddings like word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014). To further understand what performance gains are attributable to improved embeddings vs. the powerful downstream models that leverage them, we also compare with a simple baseline—fully random embed
dings—which encode no semantic or contextual information whatsoever. Surprisingly, we find that in highly optimized production tasks at a major technology company, both classic and random embeddings have competitive (or even slightly better!) performance than the contextual embeddings. $^{1,2}$
To better understand these results, we study the properties of NLP tasks for which contextual embeddings give large gains relative to non-contextual embeddings. In particular, we study how the amount of training data, and the linguistic properties of the data, impact the relative performance of the embedding methods, with the intuition that contextual embeddings should give limited gains on data-rich, linguistically simple tasks.
In our study on the impact of training set size, we find in experiments across a range of tasks that the performance of the non-contextual embeddings (GloVe, random) improves rapidly as we increase the amount of training data, often attaining within 5 to  $10\%$  accuracy of BERT embeddings when the full training set is used. This suggests that for many tasks these embeddings could likely match BERT given sufficient data, which is precisely what we observe in our experiments with industry-scale data. Given the computational overhead of contextual embeddings, this exposes important trade-offs between the computational resources required by the embeddings, the expense of labeling training data, and the accuracy of the downstream model.
To better understand when contextual embeddings give large boosts in performance, we identify three linguistic properties of NLP tasks which help explain when these embeddings will provide gains:
- Complexity of sentence structure: How interdependent are different words in a sentence?
This aligns with recent observations from experiments with classic word embeddings at Apple (Ré et al., 2020).
2These tasks are proprietary, so we share these results anecdotally as motivation for our study.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2650-2663 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
- Ambiguity in word usage: Are words likely to appear with multiple labels during training?
- Prevalence of unseen words: How likely is encountering a word never seen during training?
Intuitively, these properties distinguish between NLP tasks involving simple and formulaic text (e.g., assistant commands) vs. more unstructured and lexically diverse text (e.g., literary novels). We show on both sentiment analysis and NER tasks that contextual embeddings perform significantly better on more complex, ambiguous, and unseen language, according to proxies for these properties. Thus, contextual embeddings are likely to give large gains in performance on tasks with a high prevalence of this type of language.
We discuss the different types of word embeddings we compare in our study: contextual pretrained embeddings, non-contextual pretrained embeddings, and random embeddings; we also discuss the relative efficiency of these embedding methods, both in terms of computation time and memory (Sec. 2.1).
Pretrained contextual embeddings Recent contextual word embeddings, such as BERT (Devlin et al., 2018) and XLNet (Yang et al., 2019), consist of multiple layers of transformers which use self-attention (Vaswani et al., 2017). Given a sentence, these models encode each token into a feature vector which incorporates information from the token's context in the sentence.
Pretrained non-contextual embeddings Noncontextual word embeddings such as GloVe (Pennington et al., 2014), word2vec (Mikolov et al., 2013), and fastText (Mikolov et al., 2018) encode each word in a vocabulary as a vector; intuitively, this vector is meant to encode semantic information about a word, such that similar words (e.g., synonyms) have similar embedding vectors. These embeddings are pretrained from large language corpora, typically using word co-occurrence statistics.
Random embeddings In our study, we consider random embeddings (e.g., as in Limsopatham and Collier (2016)) as a simple and efficient baseline that requires no pretraining. Viewing word embeddings as  $n$ -by-  $d$  matrices ( $n$ : vocabulary size,  $d$ : embedding dimension), we consider embedding matrices composed entirely of random values. To reduce the memory overhead of storing these  $n \cdot d$
random values to  $O(n)$ , we use circulant random matrices (Yu et al., 2017) as a simple and efficient approach (for more details, see Appendix A.1).<sup>3,4</sup>
2.1 System Efficiency of Embeddings
We discuss the computational and memory requirements of the different embedding methods, focusing on downstream task training and inference.
Computation time For deep contextual embeddings, extracting the word embeddings for tokens in a sentence requires running inference through the full network, which takes on the order of 10 ms on a GPU. Non-contextual embeddings (e.g., GloVe, random) require negligible time  $(O(d))$  to extract an embedding vector.
Memory Using contextual embeddings for downstream training and inference requires storing all the model parameters, as well as the model activations during training if the embeddings are being fine-tuned (e.g., 440 MB to store BERT<sub>BASE</sub> parameters, and on the order of 5-10 GB to store activations). Pretrained non-contextual embeddings (e.g., GloVe) require  $O(nd)$  to store a  $n$ -by-  $d$  embedding matrix (e.g., 480 MB to store a 400k by 300 GloVe embedding matrix). Random embeddings take  $O(1)$  memory if only the random seed is stored, or  $O(n)$  if circulant random matrices are used (e.g., 1.6 MB if  $n = 400k$ ).
We provide an overview of our experimental protocols (Section 3.1), the results from our study on the impact of training set size (Section 3.2), and the results from our linguistic study (Section 3.3). We show that the gap between contextual and noncontextual embeddings often shrinks as the amount of data increases, and is smaller on language that is simpler based on linguistic criteria we identify.
To study the settings in which contextual embeddings give large improvements, we compare
Note that one could also simply store the random seed, though this requires regenerating the embedding matrix every time it is accessed.
4We provide an efficient implementation of circulant random embedding matrices here: https://github.com/HazyResearch/random_embedding.
Pretrained contextual and non-contextual embeddings also require significant computational resources during pretraining. For example training  $\mathrm{BERT}_{\mathrm{BASE}}$  takes 4 days on 16 TPU chips.
Figure 1: NER (CoNLL-2003; left), and sentiment analysis (SST; right) performance, as a function of the fraction of the training set used. As the amount of training data increases, the non-contextual embedding performance improves quickly, generally narrowing the gap with the contextual embeddings.
them to GloVe and random embeddings across a range of named entity recognition (NER) (Tjong Kim Sang and De Meulder, 2003), sentiment analysis (Kim, 2014), and natural language understanding (Wang et al., 2019a) tasks. We choose these lexically diverse tasks as examples of word, sentence, and sentence-pair classification tasks, respectively. For our embeddings, we consider 768-dimensional pretrained  $\mathrm{BERT}_{\mathrm{BASE}}$  word embeddings, 300-dimensional publicly available GloVe embeddings, and 800-dimensional random circulant embeddings. We keep the embedding parameters fixed during training for all embedding types (no fine-tuning), to isolate the benefits of pretraining from the benefits of task training. We use a CNN model (Kim, 2014) for sentiment analysis and a BiLSTM (Akbik et al., 2018; Wang et al., 2019a) for the NER and General Language Understanding Evaluation (GLUE) tasks. For more details on the tasks, models, and training protocols, please see Appendix A.
3.2 Impact of Training Data Volume
We show that the amount of downstream training data is a critical factor in determining the relative performance of contextual vs. non-contextual embeddings. In particular, we show in representative tasks in Figure 1 that the performance of the non-contextual embedding models improves quickly as the amount of training data is increased (plots for all tasks in Appendix B). As a result of this improvement, we show in Table 1 that across tasks when the full training set is used, the non-contextual embeddings can often (1) perform within  $10\%$  absolute accuracy of the contextual
Task	Performance gap	Sample complexity ratio
B	R-B	G-B	R/B	G/B
NER	CoNLL	94.8	-9.1	-2.7	16	4
Sent.	TREC	95.8	-10.3	-6.0	4	4
MPQA	89.6	-4.7	-0.9	16	1
CR	88.5	-5.0	-3.5	16	4
SUBJ	97.7	-8.7	-3.3	256	16
SST	91.6	-11.2	-6.4	256	64
MR	85.9	-13.4	-6.9	256	16
GLUE	RTE	61.0	-1.8	-2.5	1	64
MRPC	84.8	-8.8	-6.9	16	4
QQP	86.5	-9.2	-6.8	16	16
CoLA	51.7	-34.6	-40.1	64	256
STS-B	85.6	-29.1	-19.8	64	16
QNLI	84.6	-15.8	-9.5	64	16
MNLI	78.8	-17.1	-12.0	64	16
SST	91.3	-15.9	-12.8	256	64
Table 1: Performance and sample complexity of random (R) and GloVe (G) relative to BERT (B) for NER, sentiment analysis (Sent.), and language understanding (GLUE) tasks. Second column shows BERT accuracy; third/fourth columns show the accuracy gap between BERT and random/GloVe; fifth/sixth columns show sample complexity ratios, the largest  $n \in \{1,4,16,64,256\}$  for which BERT outperforms random/GloVe when trained on  $n$ -times less data. We observe that non-contextual embeddings can often (1) perform within  $10\%$  absolute accuracy of the contextual embeddings, and (2) match the performance of contextual embeddings which are trained on  $1x - 16x$  less data. This sheds light on a tradeoff between the upfront cost of labeling training data and the inference-time computational cost of the embeddings.
embeddings, and (2) match the performance of the contextual embeddings trained on 1x-16x less data, while also being orders of magnitude more computationally efficient. In light of this, ML practitioners may find that for certain real-world tasks the large gains in efficiency are well worth the cost of labeling more data.
Specifically, in this table we show for each task the difference between the accuracies attained by BERT vs. GloVe and random (note that random sometimes beats GloVe!), as well as the largest integer  $n \in \{1, 4, 16, 64, 256\}$  such that BERT trained on  $\frac{1}{n}$  of the training set still outperforms non-contextual embeddings trained on the full set.
3.3 Study of Linguistic Properties
In this section, we aim to identify properties of the language in a dataset for which contextual embeddings perform particularly well relative to noncontextual approaches. Identifying such properties would allow us to determine whether a new task is
<sup>6</sup>We provide theoretical support for why random embeddings perform strongly given sufficient data in Appendix B.3.
likely to benefit from contextual embeddings.
As a first step in our analysis, we evaluate the different embedding types on the GLUE Diagnostic Dataset (Wang et al., 2019a). This task defines four categories of linguistic properties; we observe that the contextual embeddings performed similarly to the non-contextual embeddings for three categories, and significantly better for the predicate-argument structure category (Matthews correlation coefficients of .33, .20, and .20 for BERT, GloVe, and random, respectively. See Appendix C.2.1 for more detailed results). This category requires understanding how sentence subphrases are composed together (e.g., prepositional phrase attachment, and identifying a verb's subject and object). Motivated by the observation that contextual embeddings are systematically better on specific types of linguistic phenomena, we work to identify simple and quantifiable properties of a downstream task's language which correlate with large boosts in performance from contextual embeddings.
In the context of both word-level (NER) and sentence-level (sentiment analysis) classification tasks, we define metrics that measure (1) the complexity of text structure, (2) the ambiguity in word usage, and (3) the prevalence of unseen words (Section 3.3.1), and then show that contextual embeddings attain significantly higher accuracy than noncontextual embeddings on inputs with high metric values (Section 3.3.2, Table 2).
3.3.1 Metric Definitions
We now present our metric definitions for NER and sentiment analysis, organized by the above three properties (detailed definitions in Appendix C).
Complexity of text structure We hypothesize that language with more complex internal structure will be harder for non-contextual embeddings. We define the metrics as follows:
- NER: We consider the number of tokens spanned by an entity as its complexity metric (e.g., "George Washington" spans 2 tokens), as correctly labeling a longer entity requires understanding the relationships between the different tokens in the entity name.
- Sentiment analysis: We consider the average distance between pairs of dependent tokens in a sentence's dependency parse as a measure of the sentence's complexity, as long-range dependencies are typically a challenge for NLP systems.
Ambiguity in word usage We hypothesize that non-contextual embeddings will perform poorly in disambiguating words that are used in multiple different ways in the training set. We define the metrics as follows:
- NER: We consider the number of labels (person, location, organization, miscellaneous, other) a token appears with in the training set as a measure of its ambiguity (e.g., "Washington" appears as a person, location, and organization in CoNLL-2003).
- Sentiment analysis: As a measure of a sentence's ambiguity, we take the average over the words in the sentence of the probability that the word is positive in the training set, and compute the entropy of a coin flip with this probability.
Prevalence of unseen words We hypothesize that contextual embeddings will perform significantly better than non-contextual embeddings on words which do not appear at all in the training set for the task. We define the following metrics:
- NER: For a token in the NER input, we consider the inverse of the number of times it was seen in the training set (letting  $1/0 \coloneqq \infty$ ).
- Sentiment analysis: Given a sentence, we consider as our metric the fraction of words in the sentence that were never seen during training.
3.3.2 Empirical validation of metrics
In Table 2 we show that for each of the metrics defined above, the accuracy gap between BERT and random embeddings is larger on inputs for which the metrics are large. In particular, we split each of the task validation sets into two halves, with points with metric values below the median in one half, and above the median in the other. We see that in 19 out of 21 cases, the accuracy gap between BERT and random embeddings is larger on the slice of the validation set corresponding to large metric values, validating our hypothesis that contextual embeddings provide important boosts in accuracy on these points.
In Appendix C.2.2, we present a similar table comparing the performance of BERT and GloVe embeddings. We see that the gap between GloVe and BERT errors is larger above the median than below it in 11 out of 14 of the complexity and am
For sentiment tasks with  $C$ -labels ( $C = 6$  for the TREC dataset), we consider the entropy of the average label distribution  $\frac{1}{n} \sum_{i=1}^{n} p(y|w_i) \in \mathbb{R}^C$  over the sentence words  $w_i$ .
Task	Complexity	Ambiguity	Unseen
Abs.	Rel.	Abs.	Rel.	Abs.	Rel.
NER (CoNLL)	+4.6	1.4	+7.7	2.0	+5.0	1.4
Sent. (MR)	-5.4	0.7	+3.3	1.3	+1.2	1.1
Sent. (SUBJ)	-1.8	0.8	+6.7	2.3	+0.9	1.1
Sent. (CR)	+0.6	1.1	+3.0	1.8	+4.1	2.4
Sent. (SST)	+7.4	2.1	+8.7	2.4	+2.3	1.2
Sent. (TREC)	+5.1	1.7	+5.9	1.8	+4.4	1.5
Sent. (MPQA)	+7.9	13.5	+7.1	12.4	+1.3	1.4
Table 2: For our complexity, ambiguity, and unseen prevalence metrics, we slice the validation set using the median metric value, and compute the average error rates for BERT and random on each slice. We show that the gap between BERT and random errors is larger on the slice above the median than below it in 19 out of 21 cases, in absolute (Abs.) and relative (Rel.) terms.
biguity results, which is consistent with our hypothesis that context is helpful for structurally complex and ambiguous language. However, we observe that GloVe and BERT embeddings—which can both leverage pretrained knowledge about unseen words—perform relatively similarly to one another above and below the median for the unseen metrics.
The original work on ELMo embeddings (Peters et al., 2018) showed that the gap between contextual and non-contextual embeddings narrowed as the amount of training data increased. Our work builds on these results by additionally comparing with random embeddings, and by studying the linguistic properties of tasks for which the contextual embeddings give large gains.
Our work is not the first to study the downstream performance of embeddings which do not require any pretraining. For example, in the context of neural machine translation (NMT) it is well-known that randomly-initialized embeddings can attain strong performance (Wu et al., 2016; Vaswani et al., 2017); the work of Qi et al. (2018) empirically compares the performance of pretrained and randomly-initialized embeddings across numerous languages and dataset sizes on NMT tasks, showing for example that the pretrained embeddings typically perform better on similar language pairs, and when the amount of training data is small (but not too small). Furthermore, as mentioned in Section 2, random embeddings were considered as a baseline by Limsopatham and Collier (2016), to better understand the gains from using generic vs. domain-specific word embeddings for text classification tasks. In contrast, our goal for using random embeddings in
our study was to help clarify when and why pretraining gives gains, and to expose an additional operating point in the trade-off space between computational cost, data-labeling cost, and downstream model accuracy.
We compared the performance of contextual embeddings with non-contextual pretrained embeddings and with an even simpler baseline—random embeddings. We showed that these non-contextual embeddings perform surprisingly well relative to the contextual embeddings on tasks with plentiful labeled data and simple language. While much recent and impressive effort in academia and industry has focused on improving state-of-the-art performance through more sophisticated, and thus increasingly expensive, embedding methods, this work offers an alternative perspective focused on realizing the trade-offs involved when choosing or designing embedding methods. We hope this work inspires future research on better understanding the differences between embedding methods, and on designing simpler and more efficient models.
We gratefully acknowledge the support of DARPA under Nos. FA87501720095 (D3M), FA86501827865 (SDH), and FA86501827882 (ASED); NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); ONR under No. N000141712266 (Unifying Weak Supervision); the Moore Foundation, NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, the Okawa Foundation, American Family Insurance, Google Cloud, Swiss Re, the Stanford Graduate Fellowship in Science and Engineering, and members of the Stanford DAWN project: Teradata, Facebook, Google, Ant Financial, NEC, VMWare, andInfosys. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of DARPA, NIH, ONR, or the U.S. Government.
Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: an easy-to-use framework for state-of-the-art NLP. In *NAACL-HLT (Demonstrations)*, pages 54-59. Association for Computational Linguistics.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In COLING.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP.
Nut Limsopatham and Nigel Collier. 2016. Modelling the combination of generic and target domain embeddings in a convolutional neural network for sentence classification. In *BioNLP@ACL*, pages 136-140. Association for Computational Linguistics.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In LREC.
Pandu Nayak. 2019. Understanding searches better than ever before. [Online; published 25-Oct-2019; accessed 6-Dec-2019].
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In EMNLP.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL-HLT.
Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In NAACL-HLT (2), pages 529-535. Association for Computational Linguistics.
Carl Edward Rasmussen and Christopher K. I. Williams. 2006. Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press.
Christopher Ré, Feng Niu, Pallavi Gudipati, and Charles Srisuwananukorn. 2020. Overton: A data system for monitoring and improving machinelearned products. In CIDR.