Meaning Variation and Data Quality in the Corpus of Founding Era American English
Authors: Dallas Card
Accepted at: acl25

Meaning Variation and Data Quality in the Corpus of Founding Era American English

Dallas Card
University of Michigan School of Information*
dalc@umich.edu

Abstract
Legal scholars are increasingly using corpus based methods for assessing historical meaning. Among work focused on the so-called founding era (mid to late 18th century), the majority of such studies use the Corpus of Founding Era American English (COFEA) and rely on methods such as word counting and manual coding. Here, we demonstrate what can be inferred about meaning change and variation using more advanced NLP methods, focusing on terms in the U.S. Constitution. We also carry out a data quality assessment of COFEA, pointing out issues with OCR quality and metadata, compare diachronic change to synchronic variation, and discuss limitations when using NLP methods for studying historical meaning.

1 Introduction
Alongside dictionaries and other reference material, legal scholars are increasingly turning towards historical text corpora in order to make arguments about the historical meanings of terms, in so far as they are relevant to modern legal questions. While much of this work involves manually inspecting usages of individual terms, here we demonstrate the use of more recent NLP methods for assessing meaning change and variation, relying on masked language models (MLMs) and focusing on terms in the U.S. Constitution.

The main corpus used in investigating historical legal meanings from the so-called founding era (roughly the second half of the 18th century), is the Corpus of Founding Era American English (COFEA; Hashimoto, 2023), which brings together a series of text collections, including broadsides, legal statutes, debates, and letters. Because COFEA has become so central to arguments about legal meaning, it is essential to investigate both the limitations of the corpus, and what we can learn from it with using state of the art NLP techniques.

Using a full-text copy of the corpus, we first investigate and describe its contents, noting issues with unreliable metadata and the quality of the optical character recognition (OCR). In contrast to the in-depth study of individual words, as is more commonly done, we then present a broader automated investigation of meaning, along with an extensive discussion of the limitations of this approach.

The structure of this paper is as follows: we first describe and investigate COFEA (§3), noting several issues and limitations, especially with respect to the OCR and metadata (§4). Using MLMs, we then measure change and variation in the meaning of terms in the U.S. Constitution (§5), both across time and across collections, finding the former to be more extensive than the latter. Finally, we consider the inferred meanings of individual terms in the specific context of the Constitution, finding suggestive evidence of an overall bias towards more formal, as opposed to popular, meanings (§5.3), and end with a discussion of limitations. Data processing and analysis code, as well as interactive versions of all figures, are available online.1

2 Background
The U.S. Constitution was drafted in the summer of 1787, and quickly embraced as the fundamental law of the United States. As described in Giennapp (2018), disagreements began almost immediately as to how it should be interpreted, and continue to this day. In recent decades, much of this debate has been with respect to the theory of originalism, which is broadly the idea that the original meaning of the Constitution should remain in force.2

Independent of fundamental legal theories and questions, most scholars recognize the importance and difficulty of knowing and characterizing the

1 https://dallascard.github.io/cofea
2 There is an enormous literature on originalism which we do not attempt to survey here, but note there are a range of positions which all go under this name; see Solum (2019).
meaning of language. Among other tools used by legal scholars for this purpose is that of corpus linguistics—using a corpus of texts to attempt to understand the meaning of particular terms in context (Mouritsen, 2010). Since COFEA was introduced, in part to help answer questions about “ordinary meaning” in the founding era (Hashimoto, 2023), it has quickly become the main reference corpus for this type of investigation.3

Indeed, a popular genre of legal scholarship focuses on the meanings of specific terms or phrases, such as “citizens” (Stout et al., 2020), “carry” (Mouritsen, 2010), or “bear arms” (Goldfarb, 2019). Most of these papers tend to use relatively straightforward methods, such as collocations, keyword-in-context, or manual annotations (e.g., Barclay et al., 2019; Stout et al., 2020), but increasingly we are seeing legal scholars experiment with more advanced NLP techniques (e.g., Nyarko and Sanga, 2022; Livermore et al., 2024).

In this paper, we draw particular inspiration from Lee and Phillips (2019), who focus on handful of terms, like commerce and domestic violence, using collocations and manual coding, to examine both how the meanings of these terms have changed since the founding era, and the extent of variation within COFEA. Specifically, we leverage the capability of MLMs to assess meaning, considering all terms in the Constitution. While remaining agnostic with respect to debates about originalism, our purpose is both to illustrate the power of such methods, but also to demonstrate and discuss the limitations that these data and methods entail.

3 Data and Preprocessing

We first acquire a full text copy of COFEA from its creators. Although the corpus is made up of six collections, the vast majority of words come from three of these: books, pamphlets, and broadsides (EVANS), letters of the U.S. “Founding Fathers” (FOUNDERS), and a collection of mostly legal documents provided by HeinOnline (HEIN). Smaller collections include laws and resolutions enacted by Congress (STATUTES), records of the 1787 Constitutional Convention (FARRANDS), and a collection of the corresponding debates in state conventions (ELLIOTS). To narrow our focus, we limit our analysis to the period from 1760–1800, which is the period of greatest overlap across sources, as shown in Figure 1. For additional information about COFEA, please refer to Appendix A.

To augment COFEA, we also include all newspaper articles from those years published in The Pennsylvania Gazette (TPG), which has previously been used for investigating historical legal language (Strang, 2018). For a modern reference corpus, we use the Corpus of Contemporary American English (COCA; Davies, 2008). For the U.S. Constitution, we use the text provided by the National Archives, including the Bill of Rights. After some initial filtering and preprocessing of COFEA, we tokenize all documents using BERT (Devlin et al., 2019), do spelling normalization, and curate a list of bigrams based on normalized pointwise mutual information, which we augment with the bigrams from Lee and Phillips (2019). Additional details on sources and preprocessing are given in Appendices A and B.

![Number of tokens per year in the six collections that comprise COFEA. Grey bands indicate the period studied in this paper, and the dotted line shows the year in which the U.S. Constitution was written. Note that years associated with HEIN documents may in some cases by unreliable (see §4).](page_1012_153_388_388.png)

Figure 1: Number of tokens per year in the six collections that comprise COFEA. Grey bands indicate the period studied in this paper, and the dotted line shows the year in which the U.S. Constitution was written. Note that years associated with HEIN documents may in some cases by unreliable (see §4).

4 Data Quality Assessment

Before investigating meaning change, we first assess the quality of the source documents in COFEA. During initial filtering, we note the presence of a number of documents that contain modern text, such as editorial notes, which we exclude (see Appendix B.1). In addition, the years associated with some documents seem to be incorrect or imprecise. This is especially the case for documents in HEIN, some of which collect together
texts written over multiple years. For our purposes, this makes little difference, as most texts in HEIN appear to date from roughly 1760–1800. However, this could be a more serious issue for investigations which consider precise timing, such as before and after the Constitution was written, or documents from a particular year (see Appendix B.2).

To measure the quality of the OCR for each corpus, we use a simple but commonly used technique of checking for coverage using a dictionary (Springmann et al., 2014; Neudecker et al., 2021).4 After lemmatizing all documents, we measure the proportion of lemmas that appear in the Webster’s 1913 dictionary (Porter, 1913), enhanced with additional terms such as titles, names, places, and abbreviations (see Appendix C.1 for details).5

Figure 2 shows a plot of the OCR quality of each corpus based on this measure, where each dot represents one document, sorted by score, and evenly distributed across the x-axis, on a log scale. As can be seen, some of the worst documents are in EVANS and FOUNDERS, but overall HEIN has the worst quality over most of the range. Inspecting the data, the most common tokens in HEIN that are not in the dictionary appear to be poor OCR renderings of terms like “shall” (e.g., “thall”) and “justice” (e.g., “juifice”), as well as subwords from improper splitting, like “tion”. This means that many occurrences of some terms in the HEIN collection will effectively be missed by a simple keyword search, which is the primary way that scholars access COFEA. For alternative methods of assessing OCR quality, with broadly similar results, please refer to Appendix C.

![Line graph showing OCR quality across corpora as measured by coverage in the augmented Webster’s 1913 dictionary. Each point represents one document, showing the proportion of words in that document found in the dictionary, ranked and distributed across the x-axis.](page_1012_120_377_377.png)

Figure 2: OCR quality across corpora as measured by coverage in the augmented Webster’s 1913 dictionary. Each point represents one document, showing the proportion of words in that document found in the dictionary, ranked and distributed across the x-axis.

and inspecting individual term mentions.6

For all terms that occur in the Constitution, we sample up to 4000 instances from each of COFEA and COCA, along with samples of additional random words, and pass each word, masked and in context, through the model. Following Eyal et al. (2022), we save the top-\( k \) most probable substitutes (with \( k = 10 \)), excluding stopwords (details in Appendix D). For measuring changes in word meaning, we take the total counts of the top-\( k \) substitutes for all instances of the word in a corpus, to get a distribution over the vocabulary, and then compute the Jensen-Shannon divergence (JSD) between the distributions for each corpus.

5 Change and Variation in Meaning

5.1 Measuring changes in meaning

To investigate both change and variation in word meaning, we make use of masked language models, specifically bert-large-uncased. In particular, we borrow the technique from Card (2023), which uses changes in the most probable substitutes for terms as informative of word meaning, as this approach allows for both aggregate comparisons,

4According to Hashimoto (2023), documents in COFEA are being evaluated for OCR accuracy, but these scores are not yet available, to the best of our knowledge.
5Although this dictionary is from more than a century after most of the COFEA documents, dictionaries are known to lag behind linguistic change, and preserve some historical meanings, making this a reasonable choice.
6For comparison to a simpler method of measuring change in meaning, we also include a parallel analysis using the approach of Hamilton et al. (2016) in Appendix D.
<table>
  <tr>
    <th>Term</th>
    <th>Founding era</th>
    <th>Modern era</th>
    <th>JSD</th>
  </tr>
  <tr>
    <td>captures</td>
    <td>captures prizes capture seizures prize invasion insurrection violence invasions</td>
    <td>captures reflects shows represents describes</td>
    <td>0.91</td>
  </tr>
  <tr>
    <td>domestic violence</td>
    <td>invasion truce protection war commission</td>
    <td>violence abuse rape crime assault</td>
    <td>0.91</td>
  </tr>
  <tr>
    <td>marque</td>
    <td>mar truce protection war commission</td>
    <td>porte salle junta grange crescent</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>capitation</td>
    <td>direct poll land general state</td>
    <td>medicare payment insurance compensation</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>quartered</td>
    <td>stationed posted kept placed lodged</td>
    <td>sliced chopped peeled seeded trimmed</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>affirmation</td>
    <td>declaration oath certificate deposition</td>
    <td>expression recognition acceptance assertion</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>training</td>
    <td>training raising bringing exercising trained</td>
    <td>training education instruction practice</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td>piracies</td>
    <td>crimes offenses piracy murders treason</td>
    <td>crimes crime abuses offenses acts</td>
    <td>0.85</td>
  </tr>
</table>

Table 1: Constitutional terms with the largest meaning change from the founding to modern era, with most common substitutes, and corresponding JSD values. For an expanded list, see Table 7 in the Appendix.

senals). In addition, compared to a random set of background terms, we find that the constitutional terms have changed slightly but significantly more than other terms since the founding era, even after correcting for frequency (see analysis in Appendix D). As a sanity check, we also find that the terms showing the least change in meaning tend to be numbers and names of months or days, as was previously exploited by past work (Nyarko and Sanga, 2022). In this case, all but one of the top 20 lowest JSD scores correspond to numbers or temporal terms (e.g., days, years).

5.3 Specialized vs. Popular Meanings

An enduring debate about the Constitution is the extent to which it was written using a specialized legal vocabulary, as opposed to an accessible popular vernacular (McGinnis and Rappaport, 2018; Gienapp, 2018). Looking first at word counts, we find that, on average, constitutional terms appear more frequently in legal corpora than others. Figure 3 shows the relative frequencies of all terms in the Constitution in Popular, Legal, and Founders documents, projected onto the simplex.7 As can be seen, most of the terms in the Constitution are relatively evenly represented across these three types of sources. However, on average, there are more terms that appear more commonly in legal sources, rather than popular sources or Founders’ papers.

Frequency alone, however, does necessarily tell us about meaning. To that end, we carry out a brief comparison to provide suggestive evidence on this question. To do so, we use the same technique as above for studying change in meaning, but here apply it to variation across parts of the COFEA corpus representing legal vs. popular documents.

Overall, we find the synchronic variation between legal and popular sources is much less than the diachronic change in meaning from the founding to the modern era: the mean JSD across all constitutional terms is 0.45 for the former and 0.62 for the latter. Nevertheless, there are many terms which show clear differences in terms of primary meaning across sources. The terms with the largest differences are shown in Table 2, excluding those due to obvious OCR errors.

The above analysis tells us about the most common meanings, but in principle, MLMs also allow us to investigate the specific usage of terms in the Constitution itself. To get at this, we repeat the process described above to embed each specific token in the Constitution, and compare the top ten replacements suggested for the constitutional context to the most common replacements from each of the legal and popular subsets. For each mention, we count the token as leaning towards a specialized (legal) vs. popular usage if the set overlap is at least two greater for one than the other; otherwise, we count the mention as indeterminate. For words that

![A triangle plot showing the distribution of terms across Popular, Legal, and Founders, with contours overlaid.](page_1012_670_323_323.png)

Figure 3: Overall, constitutional terms are more common in legal than other sources. Each point represents one term, with relative frequencies across the three types of sources projected on to the simplex, with contours from kernel density estimation overlaid on top.

7For legal documents, we use all of STATUTES, FAR-RANDS, and ELLIOTS, and the parts of HeIN marked as “Legal”. For popular sources, we use EVANS and TPG.
<table>
  <tr>
    <th>Term</th>
    <th>Legal</th>
    <th>Popular</th>
    <th>JSD</th>
  </tr>
  <tr>
    <td>tender</td>
    <td>tender payment demand currency money</td>
    <td>tender kind soft generous great</td>
    <td>0.80</td>
  </tr>
  <tr>
    <td>dock</td>
    <td>dock ship navy naval docks</td>
    <td>dock market water street front</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>bankruptcies</td>
    <td>commerce trade religion slaves debts</td>
    <td>losses debts commerce trade failures</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>affecting</td>
    <td>affecting respecting touching concerning</td>
    <td>affecting interesting awful melancholy</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>repassed</td>
    <td>rejected amended approved repealed</td>
    <td>passed crossed entered left ascended</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>resignation</td>
    <td>resignation removal refusal appointment</td>
    <td>resignation submission patience obedience</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>sign</td>
    <td>sign receive make deliver take</td>
    <td>sign head tavern foot signs</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>searches</td>
    <td>search searches seizures arrests attacks</td>
    <td>searches search knows sees inquiries</td>
    <td>0.69</td>
  </tr>
</table>

Table 2: Constitutional terms with the largest difference in meaning between legal and popular sources in COFEA, (excluding those due to obvious OCR errors), shown with most common substitutes, and corresponding JSD values. All terms here lean towards typical legal meanings when used in the Constitution, according to the analysis described in section §5.3. See Table 10 in the Appendix for an expanded list.

occur more than once in the Constitution, we take the majority outcome, defaulting to indeterminate in the case of ties (see Appendix D.3 for details).

Inspecting the top 40 words with greatest difference between sources, we find only two that lean towards a popular meaning, and in both cases (aid and fix), these can be attributed to OCR errors. Of the remainder, 26 lean towards a specialized legal meaning, and the remaining 12 are indeterminate. Thus, for the terms in the Constitution that show the greatest variation in meaning across sources, most of these align more with how the terms were used in legal documents, and almost none with usage in more popular sources.

6 Discussion

Despite the popularity of corpus linguistics for assessing historical legal meanings, numerous criticisms have been directed at this approach in general, and COFEA in particular. With respect to COFEA itself, a major concern has to do with it being a non-representative sample of text. In particular, a vastly disproportionate amount of text in COFEA comes from a handful of people (the so-called “Founding Fathers”). As such, COFEA as a whole over-represents elite voices, and under-represents more popular sources.

With respect to the use of corpus linguistics in law, additional critiques have focused on biases in the sources included (Drakeman, 2020; Jennejohn et al., 2021), subjectivity in interpretation (Gries, 2020), placing too much weight on frequent usages (Herenstein, 2017), the historical or linguistic competence required in interpretation (Slocum and Gries, 2020), and lack of clear and reproducible methodology (Henderson et al., 2024). While our more computational approach has the advantage of being fully reproducible, other criticisms still apply, especially with respect to source bias.

Beyond the reasons for caution noted by others, we additionally demonstrate the presence of some modern text in COFEA, as well as some OCR errors and potentially unreliable dates, especially in HEIN. In addition, it is important to note that the methods we have used here primarily establish a single, relatively coarse sense for each time period or collection, and may miss more subtle nuances of meaning. While other methods could potentially exhibit greater sensitivity to such nuances, there is inherently only so much information in a single occurrence of a word. We can thus have much more confidence in aggregate comparisons, even though they lack some sensitivity to context.

Nevertheless, without implying any particular legal relevance, we can say that the extent of differences between specialized vs. popular meanings in the founding era is small compared to meaning changes over time, and that the constitutional terms that differ in meaning across source types seem to broadly lean towards legal senses. Confident judgements about any specific terms would require more input from legal and historical scholars, but overall our results highlight the importance of combining careful manual inspection with reproducible computational analyses on known data.

7 Conclusion

This paper provides the first external assessment of data quality in COFEA, and the first computational analysis of meaning change and variation for all terms in the U.S. Constitution. While noting limitations with respect to both methods and data, especially issues involving dates and OCR quality for the HEIN collection, we find that variation in meaning across sources is much less than change in meaning over time, with weak evidence in favor of the broadly specialized (legal) character of the language in the U.S. Constitution.
Limitations

Although the use of more sophisticated techniques than word counting or manual coding can help to shed some light on historical meanings, we should nevertheless note considerable limitations with the analyses presented here.

First, building on the limitations noted in the Discussion, COFEA itself is far from representative of all text produced during the founding era. Although Evans claims to be close to comprehensive of certain media (broadsides, books, and pamphlets), even the language used in these is still no doubt far from how people of the time typically spoke in conversation. Moreover, COFEA as a whole is missing newspaper articles, which were an important venue for public communication. Here, we have augmented COFEA with The Pennsylvania Gazette because it is easily accessibility, but other sources would be useful to consider.

In terms of the methods used here, our approach provides one dominant sense per time period (or per source), when in fact many terms may exhibit multiple meanings within any collection of documents. As such, our analyses are primarily getting at the most common meanings, and may miss less common senses. It is entirely possible, for example, that the more common sense of a term in the legal sources was also used in popular sources, but with less frequency than another sense. Although one could try to inspect individual usages in each corpus, inferences based on single usages will be much noisier than collection averages.

In addition, the meanings inferred via our approach are relatively coarse. For example, a notable legal case hinged on whether “carrying” a weapon should include transporting it in a vehicle (Mouritsen, 2010). Looking only at common replacements for carry in COFEA can help us to get a general sense of its meaning (e.g., bring, take, put), but this may tell us relatively little about the subtleties of implied meaning. As such, this approach is unlikely to be an adequate substitute for historical analysis; at best, computational methods can complement other approaches.

To compare with past work, we opted to use some common phrases, but how to best identify bigrams or other multiword expressions is open to debate. Although the method we use gives reasonable results, we note that the bigrams we included from Lee and Phillips (2019) are quite far from meeting our criteria. This raises the question of whether such phrases (e.g., public use) are best understood as distinct concepts or not, and illustrates the importance of having some criteria for deciding what to include. Because meaning is partly but not entirely compositional, studying the meaning of a particular word as opposed to a phrase across all mentions could lead to different results.

Regarding what can be inferred, it is also worth noting that the evidence for certain terms is extremely limited. For example, there are fewer than 100 mentions of the phrase domestic violence in COFEA. Of these, only a handful appear in documents that were published before the U.S. Constitution appeared. Thus, COFEA provides limited evidence on the meaning of the phrase as it existed prior to its usage in the Constitution.

While identifying broad changes in meaning is relatively easy, inferring the meaning of an individual mention of a word is inherently difficult, and sometimes underdetermined; our judgements as to the tendency of terms in the Constitution to reflect specialized (legal) as opposed to popular meanings should thus be interpreted cautiously. Ultimately, it is inherently difficult to know precisely what an author meant when they chose to use a word in a particular context. Although the combination of frequency differences and differences in inferred meanings suggest the presence of numerous terms in the Constitution that drew heavily on a specialized legal vocabulary, that does not necessarily apply to the Constitution as a whole.

Finally, and most importantly, we do not address broader questions about originalism, including how much it matters what a particular author meant, how a word would have been understood at a particular time, or whether such questions are even relevant to applying the law in a modern context. Independent of legal implications, we hope our work will add to the discussion of limitations associated with corpus linguistics generally, and the use of COFEA for this purpose in particular.

Acknowledgements

Many thanks to Ben Stone, Nicole Coleman, and Dave Armond for facilitating access to COFEA, and to Accessible Archives for providing access to The Pennsylvania Gazette. Additional thanks to Candice Laine Penelton for help with initial data exploration, and to Peter Henderson, Dan Jurafsky, and anonymous reviewers for thoughtful feedback on earlier drafts.
References

Stephanie H. Barclay, Brady Earley, and Annika Boone. 2019. Original meaning and the establishment clause: A corpus linguistics analysis. Ariz. L. Rev., 61:505–560.

Dallas Card. 2023. Substitution-based semantic change detection using contextual embeddings. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

Mark Davies. 2008. The Corpus of Contemporary American English (COCA). Available online at https://www.english-corpora.org/coca/.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Donald L. Drakeman. 2020. Is corpus linguistics better than flipping a coin? The Georgetown Law Journal, 109:81–101.

Matan Eyal, Shoval Sadde, Hillel Taub-Tabib, and Yoav Goldberg. 2022. Large scale substitution-based word sense induction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4738–4752, Dublin, Ireland. Association for Computational Linguistics.

Jonathan Gienapp. 2018. The Second Creation: Fixing the American Constitution in the Founding Era. Harvard University Press.

Neal Goldfarb. 2019. A (mostly corpus-based) linguistic reexamination of D.C. v. Heller and the second amendment. SSRN Electronic Journal.

Stefan Th. Gries. 2020. Corpora and legal interpretation: Corpus approaches to ordinary meaning in legal interpretation. In Malcolm Coulthard, Alison May, and Rui Sousa-Silva, editors, The Routledge Handbook of Forensic Linguistics, 2 edition, chapter 38, pages 628–643. Routledge.

William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489–1501, Berlin, Germany. Association for Computational Linguistics.

Brett Hashimoto. 2023. Corpus of Founding Era American English: designing a corpus for interpreting the United States Constitution. Corpora, 18(1):1–14.

Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland. Association for Computational Linguistics.

Peter Henderson, Daniel E. Ho, Andrea Vallebueno, and Cassandra Handan-Nader. 2024. Corpus enigmas and contradictory linguistics: Tensions between empirical semantic meaning and judicial interpretation. Minnesota Journal of Law, Science & Technology, 25(2).

Ethan J. Herenstein. 2017. The faulty frequency hypothesis: Difficulties in operationalizing ordinary meaning through corpus linguistics. Stan. L. Rev., 70.

Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.

Matthew Jennejohn, Samuel Nelson, and D. Carolina Núñez. 2021. Hidden bias in empirical textualism. Georgetown Law Journal, 109:767–811.

Thomas R. Lee and James C. Phillips. 2019. Data-driven originalism. University of Pennsylvania Law Review, 167(2):261–335.

Michael A. Livermore, Felix Herron, and Daniel N. Rockmore. 2024. Language model interpretability and empirical legal studies. Journal of Institutional and Theoretical Economics (JITE), 180(2):244–276.

Gregory E. Maggs. 2012. A concise guide to the records of the Federal Constitutional Convention of 1787 as a source of the original meaning of the Convention of 1787 as a source of the original meaning of the U.S. Constitution. Geo. Wash. L. Rev, 1707:1–43.

John O. McGinnis and Michael B. Rappaport. 2018. The constitution and the language of the law. William and Mary Law Review, 59(4):1321–1412.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.

Burt L. Monroe, Michael P. Colaresi, and Kevin M. Quinn. 2017. Fightin’ words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372–403.

Stephen C. Mouritsen. 2010. The dictionary is not a fortress: Definitional fallacies and a corpus-based approach to plain meaning. Brigham Young University Law Review, 2010(5):1915–1979.

Clemens Neudecker, Konstantin Baierer, Mike Gerber, Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. 2021. A survey of OCR evaluation tools and metrics. In Proceedings of the 6th International Workshop on Historical Document Imaging
and Processing, HIP ’21, page 13–18, New York, NY, USA. Association for Computing Machinery.

Julian Nyarko and Sarath Sanga. 2022. A statistical test for legal interpretation: Theory and applications. Journal of Law, Economics, and Organization, 38(2):539–569.

Noah Porter, editor. 1913. Webster’s Revised Unabridged Dictionary. C. & G. Merriam Co.

Brian G. Slocum and Stefan Th. Gries. 2020. Judging corpus linguistics. Southern California Law Review Postscript, 94:13–31.

Lawrence B. Solum. 2019. Originalism versus living constitutionalism: The conceptual structure of the great debate. Nw. U. L. Rev., 113(6):1243–1296.

Uwe Springmann, Dietmar Najock, Hermann Morgenroth, Helmut Schmid, Annette Gotscharek, and Florian Fink. 2014. OCR of historical printings of latin texts: Problems, prospects, progress. In Proceedings of the First International Conference on Digital Access to Textual Cultural Heritage, DATECH ’14, page 71–75, New York, NY, USA. Association for Computing Machinery.

Abigail Stout, Diana Coetzee, and Ute Römer. 2020. “We the citizens?”: A corpus linguistic inquiry into the use of “people” and “citizens” in the founding era. Georgia State University Law Review Georgia State University Law Review, 36(5):665–697.

Lee J. Strang. 2018. The original meaning of “religion” in the first amendment: A test case of originalism’s utilization of corpus linguistics. Brigham Young University Law Review, 2017(6):1683–1750.
Appendix

A  Details on Corpora and Sources

A.1 COFEA
COFEA comprises six subcorpora, each of which we briefly describe here. Token counts for each full corpus and the time period of interest are given in Table 3.

Evans: The Evans Early Imprint Series, created by the Text Creation Partnership, attempts to collect a nearly comprehensive set of books, pamphlets, and broadsides printed in America from the late 17th to early 19th century. As such, it best represents language of more general usage, compared to the other corpora in COFEA.

Founders: The National Archives Founders Papers Online project collects together the correspondence, personal papers, and private writings from John Adams, Benjamin Franklin, Alexander Hamilton, Thomas Jefferson, James Madison, and George Washington. Although these documents represent much more informal and personal language, compared to the legal documents in COFEA, they are nevertheless unusual for being written by some of the most powerful and privileged people in the United States at that time.

Hein: COFEA contains a series of collections from HeinOnline, including: the U.S. Treaties and Agreements Library, U.S. Congressional Documents, American Indian Law Collection, and Session Laws Library. As indicated by the titles, these are primarily statutes and legislative records.

Elliot: Compiled and printed by Jonathan Elliot, Elliot’s Debates contains the text of debates held at state conventions in relation to adopting the federal Constitution.

Statutes: Like Hein, the US Statutes at Large corpus, contains the text of laws enacted by Congress from the end of the 18th century.

Farrands: Farrand’s Records of the Federal Constitution of 1787 collects together documentary material related to the Constitutional Convention.

A.2 The Pennsylvania Gazette
The Pennsylvania Gazette (TPG) was a prominent U.S. newspaper throughout the 18th century (Strang, 2018). Although it is smaller than the largest COFEA corpora, the digitized version was hand-keyed, and thus avoids many of the problems that arise with OCR, which we leverage in Appendix C.2. We obtained a copy of The Pennsylvania Gazette from Accessible Archives.8 Token counts for TPG are also given in Table 3.

<table>
  <tr>
    <th>Subset</th>
    <th>Years</th>
    <th>Tokens</th>
    <th>Tokens (1760-1800)</th>
  </tr>
  <tr>
    <td>EVANS</td>
    <td>1640–1810</td>
    <td>117926408</td>
    <td>78327884</td>
  </tr>
  <tr>
    <td>FOUNDERS</td>
    <td>1706–1836</td>
    <td>67070491</td>
    <td>41131670</td>
  </tr>
  <tr>
    <td>HEIN</td>
    <td>1760–1834</td>
    <td>43957047</td>
    <td>39230854</td>
  </tr>
  <tr>
    <td>ELLIOTS</td>
    <td>1787</td>
    <td>1758037</td>
    <td>1758037</td>
  </tr>
  <tr>
    <td>STATUTES</td>
    <td>1789–1799</td>
    <td>531556</td>
    <td>531556</td>
  </tr>
  <tr>
    <td>FARRANDS</td>
    <td>1787</td>
    <td>835837</td>
    <td>835837</td>
  </tr>
  <tr>
    <td>TPG</td>
    <td>1725–1815</td>
    <td>28671182</td>
    <td>19148702</td>
  </tr>
</table>

Table 3: Token counts in each subcorpus

A.3 The U.S. Constitution
There are multiple versions of the text of the U.S. Constitution available online; here, we use the electronic copy provided by the National Archives.9,10 Specifically, we include the preamble and the seven articles, (but not the signatories), along with the Bill of Rights (Amendments I through X). We do not include Article 1 of the 1789 Joint Resolution of Congress (never ratified) or Article 2 (ratified in 1992). Each numbered section of each Article is processed as a separate document (or the entire Article in the case of Articles without numbered sections).

A.4 COCA
The Corpus of Contemporary American English (Davies, 2008) is a large and balanced dataset containing over 1 billion words of American English from multiple genres. Here, we use the text from the academic, fiction, magazine, and newspaper genres, for the years 1990–2017.

B Data Preprocessing

B.1 Initial filtering
Inspecting the documents in COFEA, it is immediately apparent that there are a number of errors, both in terms of metadata, and the text retrieved. For example, several documents in the FOUNDERS corpus consist only of the phrase “> We were unable to find any matches for your search.”, which

8https://www.accessible.com/accessible/
9https://www.archives.gov/founding-docs/constitution-transcript
10Interestingly, there are subtle variations in spelling in different versions of the Constitution provided by the websites for different branches of the U.S. government.
is presumably the result of a scraping error. We exclude these documents from our analysis, along with those with the title “Editorial Note”, and those with a body less than nine characters long.

To restrict our analysis to those documents that are primarily in English, we use fasttext to assess the primary language of each document, and discarded those classified as anything other than English.11

B.2 Metadata Assessment and Correction

All documents in COFEA have an associated year. In the case of EVANS, these are typically the year in which the document (such as a broadside) was published. For FARRANDS, by contrast—which provides a record of the Federal Convention of 1787—the documents are associated with the year in which the debates took place, even though this full collection was not published until 1911 (Maggs, 2012). Similarly, the letters in FOUNDERS are typically dated according to the date of the letter.

For the most part, these dates seem adequate to historical investigation. One collection, however, deserves additional scrutiny, namely HEIN. Although HEIN is one of the three large collections in COFEA by number of tokens, it consists of only 285 documents, in the version of COFEA we gained access to. Obviously many of these documents are quite long, including entire books. In this case, using the date of publication presents more of an issue, if the goal is to capture the language of a particular era. In particular, some documents in HEIN are historical texts, but include a modern preamble. In other cases, the documents collect together laws that were written over many years, such as “Laws of the State of New York”.

We briefly reviewed all of the documents in HEIN, and try to associate each document with the most recent text within it, to avoid contaminating historical text with modern usages. Fortunately, the given dates seem to be mostly reliable with respect to whether documents were written during the time period 1760–1800 or not. Thus, the issue with dates has very little effect on our selection of documents to study. However, a somewhat larger number of documents appear to have potentially unreliable dates relative to the year in which the Constitution was written (i.e., they may have been written before 1787, but dated later, or vice versa).

As such, one should be especially cautious when trying to look for usages which pre-date the writing of the Constitution itself.12

B.3 Spelling normalization

Because some terms appear with varied or archaic spellings (e.g., “choose” vs “chuse”), we begin by normalizing the spelling of all occurrences of terms that appear in the Constitution for which we can find a relevant alternate form. In more detail, we first use word2vec (Mikolov et al., 2013) to train static word vectors on documents from COFEA (from all years), after converting all text to lower case. For all terms in the Constitution, we find the top 100 most similar word vectors, in terms of cosine similarity, and compute the edit distance between the corresponding pair of words. We filter out those that have cosine similarity less than 0.5 or an edit distance of more than two. We then manually inspect all candidates for each term, and keep those that can be identified as legitimate alternate spellings. We then replace all version of the alternate spellings everywhere in our corpora with a standardized form. Ultimately, at most one alternate spelling was found for each term, mostly reflecting differences between British and American spellings. The full list of alternate spellings identified is given in Table 4.

B.4 Identifying bigrams

Because past work has focused on certain bigrams in the Constitution (Lee and Phillips, 2019), we curate a small list of multi-word expressions, to which we add past bigrams that have been considered. To construct our list, we compute the normalized pointwise mutual information (NPMI) for all bigrams that appear in COFEA (not just in the Constitution). We then keep noun-noun and adjective-noun bigrams with NPMI \( \geq 0.6 \) and at least 2000 mentions in COFEA. We also exclude bigrams with incorrectly tagged parts of speech, as well as names of people, quantities, streets, cities, counties, or bodies of water, and greetings, such as “obedient servant”. Finally, we add in the three bigrams from Lee and Phillips (2019) for comparison with past work, even though they would not meet our selection criteria for inclusion. The resulting bigrams are given in Table 5, along with the ones added from past work.

11https://fasttext.cc/docs/en/language-identification.html
12Note that the token counts per year showing in Figure 1 uses the uncorrected dates.
<table>
  <tr>
    <th>Word</th>
    <th>Alternate spelling</th>
    <th>Bigram</th>
    <th>Tags</th>
    <th>NPMI</th>
    <th>Count</th>
  </tr>
  <tr><td>ambassadors</td><td>embassadors</td><td>nova scotia</td><td>NN</td><td>0.99</td><td>3010</td></tr>
  <tr><td>among</td><td>amongst</td><td>united states</td><td>NN</td><td>0.93</td><td>145086</td></tr>
  <tr><td>authorized</td><td>authorised</td><td>rhode island</td><td>NN</td><td>0.88</td><td>14042</td></tr>
  <tr><td>behavior</td><td>behaviour</td><td>new york</td><td>NN</td><td>0.82</td><td>66155</td></tr>
  <tr><td>cannot</td><td>canot</td><td>south carolina</td><td>NN</td><td>0.82</td><td>15966</td></tr>
  <tr><td>choose</td><td>chuse</td><td>west indies</td><td>NN</td><td>0.82</td><td>8809</td></tr>
  <tr><td>choosing</td><td>chusing</td><td>head quarters</td><td>NN</td><td>0.78</td><td>12090</td></tr>
  <tr><td>compel</td><td>compell</td><td>reasonable charges</td><td>JN</td><td>0.78</td><td>8750</td></tr>
  <tr><td>control</td><td>controul</td><td>u. s.</td><td>NN</td><td>0.76</td><td>3466</td></tr>
  <tr><td>controversy</td><td>controversie</td><td>north carolina</td><td>NN</td><td>0.76</td><td>12787</td></tr>
  <tr><td>days</td><td>dayes</td><td>yellow fever</td><td>JN</td><td>0.75</td><td>2291</td></tr>
  <tr><td>defense</td><td>defence</td><td>small pox</td><td>JN</td><td>0.75</td><td>4602</td></tr>
  <tr><td>domestic</td><td>domestick</td><td>minister plenipotentiary</td><td>NN</td><td>0.75</td><td>2930</td></tr>
  <tr><td>increase</td><td>encrease</td><td>fellow citizens</td><td>JN</td><td>0.73</td><td>8278</td></tr>
  <tr><td>entered</td><td>entred</td><td>great britain</td><td>NN</td><td>0.72</td><td>35462</td></tr>
  <tr><td>expel</td><td>expell</td><td>common pleas</td><td>NN</td><td>0.71</td><td>4388</td></tr>
  <tr><td>favor</td><td>favour</td><td>west india</td><td>NN</td><td>0.70</td><td>3965</td></tr>
  <tr><td>guarantee</td><td>guaranty</td><td>new jersey</td><td>NN</td><td>0.70</td><td>16867</td></tr>
  <tr><td>habeas</td><td>habeus</td><td>new hampshire</td><td>NN</td><td>0.68</td><td>10289</td></tr>
  <tr><td>honor</td><td>honour</td><td>continental congress</td><td>NN</td><td>0.67</td><td>12059</td></tr>
  <tr><td>inferior</td><td>inferiour</td><td>east india</td><td>NN</td><td>0.67</td><td>2446</td></tr>
  <tr><td>judgment</td><td>judgement</td><td>lieutenant colonel</td><td>NN</td><td>0.66</td><td>2819</td></tr>
  <tr><td>labor</td><td>labour</td><td>court martial</td><td>NN</td><td>0.66</td><td>5150</td></tr>
  <tr><td>limited</td><td>limitted</td><td>grand jury</td><td>NN</td><td>0.65</td><td>2101</td></tr>
  <tr><td>massachusetts</td><td>massachusetts</td><td>military stores</td><td>JN</td><td>0.64</td><td>2656</td></tr>
  <tr><td>missdeameanors</td><td>missdemeanours</td><td>indian corn</td><td>NN</td><td>0.64</td><td>2149</td></tr>
  <tr><td>needful</td><td>needfull</td><td>dwelling house</td><td>NN</td><td>0.63</td><td>6707</td></tr>
  <tr><td>net</td><td>nett</td><td>human race</td><td>JN</td><td>0.63</td><td>2006</td></tr>
  <tr><td>offense</td><td>offence</td><td>fellow creatures</td><td>NN</td><td>0.62</td><td>2088</td></tr>
  <tr><td>offenses</td><td>offences</td><td>pounds sterling</td><td>NN</td><td>0.62</td><td>2236</td></tr>
  <tr><td>organizing</td><td>organising</td><td>quarter master</td><td>NN</td><td>0.61</td><td>4532</td></tr>
  <tr><td>payment</td><td>paiment</td><td>holy scriptures</td><td>NN</td><td>0.61</td><td>2669</td></tr>
  <tr><td>pennsylvannia</td><td>penslvania</td><td>foreign affairs</td><td>NN</td><td>0.60</td><td>4620</td></tr>
  <tr><td>piracies</td><td>pyracies</td><td>french republic</td><td>NN</td><td>0.60</td><td>3219</td></tr>
  <tr><td>privilege</td><td>priviledge</td><td>vice president</td><td>NN</td><td>0.60</td><td>3998</td></tr>
  <tr><td>privileged</td><td>privileged</td><td>domestic violence</td><td>JN</td><td>0.36</td><td>92</td></tr>
  <tr><td>privileges</td><td>priviledges</td><td>natural born</td><td>JN</td><td>0.35</td><td>368</td></tr>
  <tr><td>public</td><td>publick</td><td>public use</td><td>JN</td><td>0.20</td><td>738</td></tr>
  <tr><td>receive</td><td>recieve</td><td></td><td></td><td></td><td></td></tr>
  <tr><td>repel</td><td>repell</td><td></td><td></td><td></td><td></td></tr>
  <tr><td>rhode</td><td>rhood</td><td></td><td></td><td></td><td></td></tr>
  <tr><td>secrecy</td><td>secresy</td><td></td><td></td><td></td><td></td></tr>
  <tr><td>soldier</td><td>souldier</td><td></td><td></td><td></td><td></td></tr>
  <tr><td>supreme</td><td>supream</td><td></td><td></td><td></td><td></td></tr>
  <tr><td>swear</td><td>sware</td><td></td><td></td><td></td><td></td></tr>
  <tr><td>tranquility</td><td>tranquillity</td><td></td><td></td><td></td><td></td></tr>
  <tr><td>trial</td><td>tryal</td><td></td><td></td><td></td><td></td></tr>
  <tr><td>tried</td><td>tryed</td><td></td><td></td><td></td><td></td></tr>
  <tr><td>until</td><td>untill</td><td></td><td></td><td></td><td></td></tr>
  <tr><td>useful</td><td>usefull</td><td></td><td></td><td></td><td></td></tr>
  <tr><td>vessels</td><td>vessells</td><td></td><td></td><td></td><td></td></tr>
  <tr><td>welfare</td><td>wellfare</td><td></td><td></td><td></td><td></td></tr>
  <tr><td>writs</td><td>writts</td><td></td><td></td><td></td><td></td></tr>
</table>

Table 4: Alternate spellings identified for normalization

<table>
  <tr>
    <th>Bigram</th>
    <th>Tags</th>
    <th>NPMI</th>
    <th>Count</th>
  </tr>
  <tr><td>nova scotia</td><td>NN</td><td>0.99</td><td>3010</td></tr>
  <tr><td>united states</td><td>NN</td><td>0.93</td><td>145086</td></tr>
  <tr><td>rhode island</td><td>NN</td><td>0.88</td><td>14042</td></tr>
  <tr><td>new york</td><td>NN</td><td>0.82</td><td>66155</td></tr>
  <tr><td>south carolina</td><td>NN</td><td>0.82</td><td>15966</td></tr>
  <tr><td>west indies</td><td>NN</td><td>0.82</td><td>8809</td></tr>
  <tr><td>head quarters</td><td>NN</td><td>0.78</td><td>12090</td></tr>
  <tr><td>reasonable charges</td><td>JN</td><td>0.78</td><td>8750</td></tr>
  <tr><td>u. s.</td><td>NN</td><td>0.76</td><td>3466</td></tr>
  <tr><td>north carolina</td><td>NN</td><td>0.76</td><td>12787</td></tr>
  <tr><td>yellow fever</td><td>JN</td><td>0.75</td><td>2291</td></tr>
  <tr><td>small pox</td><td>JN</td><td>0.75</td><td>4602</td></tr>
  <tr><td>minister plenipotentiary</td><td>NN</td><td>0.75</td><td>2930</td></tr>
  <tr><td>fellow citizens</td><td>JN</td><td>0.73</td><td>8278</td></tr>
  <tr><td>great britain</td><td>NN</td><td>0.72</td><td>35462</td></tr>
  <tr><td>common pleas</td><td>NN</td><td>0.71</td><td>4388</td></tr>
  <tr><td>west india</td><td>NN</td><td>0.70</td><td>3965</td></tr>
  <tr><td>new jersey</td><td>NN</td><td>0.70</td><td>16867</td></tr>
  <tr><td>new hampshire</td><td>NN</td><td>0.68</td><td>10289</td></tr>
  <tr><td>continental congress</td><td>NN</td><td>0.67</td><td>12059</td></tr>
  <tr><td>east india</td><td>NN</td><td>0.67</td><td>2446</td></tr>
  <tr><td>lieutenant colonel</td><td>NN</td><td>0.66</td><td>2819</td></tr>
  <tr><td>court martial</td><td>NN</td><td>0.66</td><td>5150</td></tr>
  <tr><td>grand jury</td><td>NN</td><td>0.65</td><td>2101</td></tr>
  <tr><td>military stores</td><td>JN</td><td>0.64</td><td>2656</td></tr>
  <tr><td>indian corn</td><td>NN</td><td>0.64</td><td>2149</td></tr>
  <tr><td>dwelling house</td><td>NN</td><td>0.63</td><td>6707</td></tr>
  <tr><td>human race</td><td>JN</td><td>0.63</td><td>2006</td></tr>
  <tr><td>fellow creatures</td><td>NN</td><td>0.62</td><td>2088</td></tr>
  <tr><td>pounds sterling</td><td>NN</td><td>0.62</td><td>2236</td></tr>
  <tr><td>quarter master</td><td>NN</td><td>0.61</td><td>4532</td></tr>
  <tr><td>holy scriptures</td><td>NN</td><td>0.61</td><td>2669</td></tr>
  <tr><td>foreign affairs</td><td>NN</td><td>0.60</td><td>4620</td></tr>
  <tr><td>french republic</td><td>NN</td><td>0.60</td><td>3219</td></tr>
  <tr><td>vice president</td><td>NN</td><td>0.60</td><td>3998</td></tr>
  <tr><td>domestic violence</td><td>JN</td><td>0.36</td><td>92</td></tr>
  <tr><td>natural born</td><td>JN</td><td>0.35</td><td>368</td></tr>
  <tr><td>public use</td><td>JN</td><td>0.20</td><td>738</td></tr>
</table>

Table 5: Bigrams included in our analysis

available in tabular form online.\footnote{https://github.com/ahacop/websters-dict-1913-stardict} Because dictionaries typically do not include proper names, we augment the list of terms in this dictionary with the names of all countries, as well as common names of U.S. cities, states, and counties.\footnote{https://github.com/grammakov/USA-cities-and-states/} We additionally add in titles (e.g., “mr”, “mrs”, “esq”) and common person names, including names from the Old Testament,\footnote{https://github.com/hadley/data-baby-names/blob/master/old-testament.txt} and common baby names based on data from the U.S. Social Security Administration.\footnote{https://github.com/hadley/data-baby-names/blob/master/baby-names.csv}

We then preprocess each document in COFEA and TPG by using spaCy (Honnibal and Montani,

C OCR Quality Assessment

C.1 Dictionary-based OCR assessment

To assess the quality of documents according to coverage in an appropriate dictionary, we make use of the 1913 edition of Webster’s Revised Unabridged Dictionary (Porter, 1913), which is
2017) to convert all words to lemmas. From this, we drop punctuation, and convert tokens to lowercase. Based on inspecting the remaining tokens, we additionally add to the dictionary common abbreviations for months (e.g., “feb”), roman numerals (e.g., “xvi”), ordinals (e.g., “1st”), common abbreviations (e.g., “servt”), and obvious misspellings (e.g., “recieve”). Given these exclusions and adjustments, we finally compute the proportion of remaining words in each document that appear in the dictionary. In doing so, we further allow for certain suffixes that may have been missed by the lemmatizer (e.g., “ly”), as well as accommodating common British spellings (e.g., substituting “our” for “or” in a word). For full details, please refer to online replication code.

C.2 Language model OCR assessment

As an alternative way of assessing OCR quality, we also try using a character language model to estimate the perplexity of each document. Using kenlm (Heafield, 2011), we train a trigram character language model on the union of TPG and the similarly hand-keyed (Text Creation Partnership) portion of the Eighteenth Century Collections Online project (ECCO-TCP), both of which can be assumed to be of high quality.17 We then use this model to assess the perplexity of each document in each subcorpus in COFEA. Because kenlm is designed for word-level models, we convert all documents to sequences of characters, separated by spaces, with spaces converted to a <space> token. We also use the “discount fallback” option, because the distribution of characters differs dramatically from that of words.

The results of this assessment are show in Figure 4. As with the dictionary-based assessment, we find that the HEIN subcorpus again appears to be among the worst, in so far as the best documents (farthest to the right) in HEIN are worse than the best document in other collections. Surprisingly, FOUNDERS and FARRANDS appear to be worse than HEIN in the worst cases. In the case of FARRANDS, this appears to be due to the fact that some documents are very short, and contain a high number of capitals letters and punctuation (which are improbable given our training documents), such as “SATURDAY, JUNE 9, 1787”. More generally, the presence of abbreviations, symbols, and roman numerals referring to other sections also hurts perplexity.

The low quality documents in FOUNDERS can similarly be attributed to very short and terse documents (e.g., “July 31. 13. 25.lb brown sugar Th:J.”), as well as text that does not resemble normal paragraphs of text (e.g., lists of names or citations). The key takeaway here is that these documents do not seem to have issues with the transcription (such that keyword searches should not be affected), but that certain terms may occur in unusual contexts in these documents.

![OCR quality assessment made using a trigram character language model.](page_1016_186_388_388.png)

Figure 4: OCR quality assessment made using a trigram character language model.

17https://textcreationpartnership.org/tcp-texts/

C.3 Word frequencies across subcorpora

As a final way of identifying potential OCR problems, we examine raw frequency differences between collections in COFEA. Using the log-odd technique from Monroe et al. (2017), we identify the tokens that are most over-represented in each of the three largest collections (EVANS, FOUNDERS, and HEIN), relative to the others.

From this analysis, it is clear that there are differences between the COFEA collections in terms of formality, orthography, languages used, and OCR quality. For example, the most frequent unique terms in EVANS are distinctive punctuation (e.g., •, !, >, <, ?, ”), religious terms (e.g., God, Christ, church), and archaic forms (e.g., thy, thou, ye). Distinctive punctuation also appears in FOUNDERS (e.g., & - , “”, [ ], ), as well as terms of address (e.g., sir, excellency), and various short forms (e.g., obedt, servt, genl, yr).

For HEIN, many of the most distinctive terms are a mix of governmental terms (e.g., court, commit-
<table>
  <tr>
    <th>Term</th>
    <th>Count</th>
  </tr>
  <tr>
    <td>shall</td>
    <td>137392</td>
  </tr>
  <tr>
    <td>thall</td>
    <td>29932</td>
  </tr>
  <tr>
    <td>fiall</td>
    <td>12263</td>
  </tr>
  <tr>
    <td>ihall</td>
    <td>11966</td>
  </tr>
  <tr>
    <td>flall</td>
    <td>7633</td>
  </tr>
  <tr>
    <td>flhall</td>
    <td>5527</td>
  </tr>
  <tr>
    <td>lhall</td>
    <td>3111</td>
  </tr>
</table>

Table 6: Frequent misspellings of the term “shall” in HEIN, illustrating the prevalence of OCR errors.

tee, report), and apparent OCR mistakes, revealing differences in how these different datasets were digitized. For example, multiple of the most frequent distinctive terms appear to be misspellings of the word “shall”, such as “fhall” (see Table 6). This likely results in part from the use of the “long S”, which many OCR engines may misinterpret as an “f” or “t”. In other words, we should be aware that the documents from each subset may be partially identifiable based purely on orthographic features, independent of content.

D Measuring Change in Meaning

As described in the main paper, we measure meaning change by getting probable replacements for masked terms, and looking at how the distribution of these replacements differs between periods (or collections). As per Eyal et al. (2022), we first adapt a pretrained language model to the data, by doing continued masked language model pretraining, starting with bert-large-uncased. In particular, we continue training for five epochs, either on the union of COFEA and COCA, or just COFEA, for measuring change and variation, respectively. We then index the relevant corpora, to collect all occurrences of each term, and sample up to 4000 occurrences of each. We include all the terms in the Constitution (treating bigrams in Table 5 as single words), along with 10,000 random background terms, selected using stratified sampling, to over-represent more common tokens. In addition, for background terms, we limit ourselves to those with at least 50 occurrences, minimum length of 2, and exclude punctuation.

To get replacements, we mask the term of interest (replacing with it with a single mask token, even if it is composed of multiple tokens), and provide the mask, along with up to 50 tokens of context to either side of the fine-tuned model. We then collect the top 10 most likely replacement tokens, only counting whole word replacements with length greater than one, and excluding stop-words from the Snowball sampler. To measure meaning change between two corpora, we collect all replacements for each word from each corpora (e.g., COFEA vs. COCA), and measure the Jensen-Shannon divergence of the two distributions. All processing was run on one A6000 GPU.

Because there is no definitively best method for assessing changing in meaning, we also run an evaluation using the word vector-based approach from Hamilton et al. (2016). In particular, we use gensim to fit 100-dimensional word2vec vectors, with a window size of 5, for each corpus or collection to be compared. We then align word vectors using the Procrustes alignment, and use the cosine distance between vectors for a given word as an estimate of the difference in meaning.

D.1 Change in Meaning Over Time

As described in the main paper, we first measure change over time, by comparing COFEA to COCA using the method described above. A more complete list of the terms with the greatest change in meaning, is given in Table 7, along with corresponding JSD values.

To assess the amount of change in meaning experienced by terms in the Constitution, we compare these to the sampled set of random background terms mentioned above. Because of the inverse correlation between meaning change and term frequency noted in past work (Hamilton et al., 2016; Card, 2023), we attempt to measure the association with being in the Constitution while accounting for frequency. In particular, we model change in meaning using linear regression as a function of logged term frequency, and whether the term is in the Constitution. More precisely, we fit,

\[
\text{JSD}_t = \beta_0 + \beta_1 \cdot \log(\text{count}_t) + \beta_2 \cdot \mathbb{I}[t \in C] + \epsilon_t,
\]

using all terms in the Constitution, along with a large set of background terms. JSD$_t$ is the JSD for term $t$, count$_t$ is the count of term $t$ in both COFEA and COCA combined, $\epsilon_t$ is the error associated with term $t$, and $C$ is the set of terms in the Constitution. The output of this regression is given in Table 8, showing that the coefficient associated with constitutional terms is indeed significant, although the effect is small.

Figure 5 shows the measured change for constitutional terms (in orange) compared against this random sample of other terms (in blue), with select
<table>
  <tr>
    <th>Term</th>
    <th>Founding era</th>
    <th>Modern era</th>
    <th>JSD</th>
  </tr>
  <tr>
    <td>captures</td>
    <td>captures prizes capture seizures prize</td>
    <td>captures reflects shows represents describes</td>
    <td>0.91</td>
  </tr>
  <tr>
    <td>domestic violence</td>
    <td>invasion insurrection violence invasions</td>
    <td>violence abuse rape crime assault</td>
    <td>0.91</td>
  </tr>
  <tr>
    <td>marque</td>
    <td>mar truce protection war commission</td>
    <td>porte salle junta grange crescent</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>capitation</td>
    <td>direct poll land general state</td>
    <td>medicare payment insurance compensation</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>quartered</td>
    <td>stationed posted kept placed lodged</td>
    <td>sliced chopped peeled seeded trimmed</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>affirmation</td>
    <td>declaration oath certificate deposition</td>
    <td>expression recognition acceptance assertion</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>training</td>
    <td>training raising bringing exercising trained</td>
    <td>training education instruction practice</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td>piracies</td>
    <td>crimes offenses piracy murders treason</td>
    <td>crimes crime abuses offenses acts</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td>natural born</td>
    <td>natural native free good born</td>
    <td>serial cop born psycho professional</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td>arsenals</td>
    <td>magazines fortifications stores barracks</td>
    <td>weapons forces capabilities arsenal arms</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>emolument</td>
    <td>advantage profit benefit interest happiness</td>
    <td>finance money profit advantage brown</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>counterfeiting</td>
    <td>altering making printing destroying signing</td>
    <td>fraud theft crime smuggling terrorism</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>presentment</td>
    <td>complaint trial information indictment report</td>
    <td>consolidation verdict processing hearing</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>test</td>
    <td>test trial proof late bankrupt</td>
    <td>test tests testing exam assessment</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>reprisal</td>
    <td>commissions commission protection passports</td>
    <td>retaliation retribution violence punishment</td>
    <td>0.82</td>
  </tr>
</table>

Table 7: Constitutional terms with the largest meaning change from the founding to the modern era, shown with most common substitutes, and corresponding JSD values.

![Scatter plot showing change in meaning (JSD) vs. term count for constitutional and background terms](page_420_670_600_400.png)

Figure 5: Change in meaning between founding and modern eras vs. term counts in both corpora combined, with constitutional terms shown in orange, random background terms in blue, and select terms labeled.

<table>
  <tr>
    <th>Variable</th>
    <th>Coefficient</th>
    <th>Std. err.</th>
    <th>p-value</th>
  </tr>
  <tr>
    <td>Intercept</td>
    <td>0.9480</td>
    <td>0.003</td>
    <td>&lt; 0.001</td>
  </tr>
  <tr>
    <td>In Constitution</td>
    <td>0.0328</td>
    <td>0.004</td>
    <td>&lt; 0.001</td>
  </tr>
  <tr>
    <td>log(Count)</td>
    <td>-0.0384</td>
    <td>&lt; 0.001</td>
    <td>&lt; 0.001</td>
  </tr>
</table>

Table 8: Regression results modeling change in meaning (JSD) as a function of term frequency and whether or not the term is in the Constitution. As shown, constitutional terms have changed slightly but significantly more than random background terms since the Founding era.

terms labeled. It also shows the fitted regression model, giving an intuitive sense of the difference between constitutional and background terms with respect to change in meaning.

For comparison, Table 9 shows the 15 terms with the greatest change over time, according to the method of Hamilton et al. (2016). This list has many broad similarities with the results of our method, but also some differences. Four terms overlap with the top 15 from Table 7. Other terms here are similarly ranked by both methods (e.g., magazines is ranked 20th by the method in our paper). Surprising here is the inclusion of likewise (ranked in the bottom half of terms by the method in our paper), and the absence of domestic violence (which has clearly changed dramatically over time). Overall, the results from the two metrics have a correlation of 0.64.

<table>
  <tr>
    <th>Term</th>
    <th>Cosine Distance</th>
  </tr>
  <tr>
    <td>captures</td>
    <td>1.19</td>
  </tr>
  <tr>
    <td>respecting</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>whereof</td>
    <td>0.87</td>
  </tr>
  <tr>
    <td>likewise</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>vest</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>compact</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>natural born</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>writ</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>erection</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>emolument</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>magazines</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>capitation</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>compulsory</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>corpus</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>cannot</td>
    <td>0.76</td>
  </tr>
</table>

Table 9: Constitutional terms with the largest meaning change from the founding to the modern era, according to the method of Hamilton et al. (2016).

D.2 Specialized vs. Popular Meanings

As described in the main paper, we assess differences in meaning between legal vs. popular sources by using the method described above to compare legal sources in COFEA (FARRANDS, ELLIOTS, STATUTES, and the documents in HEIN marked as “Legal”) vs. popular sources (EVANS, and The Pennsylvania Gazette; TPG). The top 15 terms with the largest difference in meaning across sources are
<table>
  <tr>
    <th>Term</th>
    <th>Legal</th>
    <th>Popular</th>
    <th>JSD</th>
    <th>Lean</th>
  </tr>
  <tr>
    <td>aid*</td>
    <td>aid said laid id hid</td>
    <td>aid assistance support help protection</td>
    <td>0.80</td>
    <td>P</td>
  </tr>
  <tr>
    <td>tender</td>
    <td>tender payment demand currency money</td>
    <td>tender kind soft generous great</td>
    <td>0.80</td>
    <td>L</td>
  </tr>
  <tr>
    <td>fix*</td>
    <td>fix three four two five</td>
    <td>fix establish set determine settle</td>
    <td>0.78</td>
    <td>P</td>
  </tr>
  <tr>
    <td>dock</td>
    <td>dock ship navy naval docks</td>
    <td>dock market water street front</td>
    <td>0.75</td>
    <td>L</td>
  </tr>
  <tr>
    <td>fled*</td>
    <td>fled ed led ted fed</td>
    <td>fled retired escaped returned went</td>
    <td>0.72</td>
    <td>I</td>
  </tr>
  <tr>
    <td>bankruptcies</td>
    <td>commerce trade religion slaves debts</td>
    <td>losses debts commerce trade failures</td>
    <td>0.71</td>
    <td>L</td>
  </tr>
  <tr>
    <td>affecting</td>
    <td>affecting respecting touching concerning</td>
    <td>affecting interesting awful melancholy</td>
    <td>0.70</td>
    <td>L</td>
  </tr>
  <tr>
    <td>repassed</td>
    <td>rejected amended approved repealed</td>
    <td>passed crossed entered left ascended</td>
    <td>0.70</td>
    <td>L</td>
  </tr>
  <tr>
    <td>resignation</td>
    <td>resignation removal refusal appointment</td>
    <td>resignation submission patience obedience</td>
    <td>0.70</td>
    <td>L</td>
  </tr>
  <tr>
    <td>sign</td>
    <td>sign receive make deliver take</td>
    <td>sign head tavern foot signs</td>
    <td>0.69</td>
    <td>L</td>
  </tr>
  <tr>
    <td>searches</td>
    <td>search searches seizures arrests attacks</td>
    <td>searches search knows sees inquiries</td>
    <td>0.69</td>
    <td>L</td>
  </tr>
  <tr>
    <td>domestic</td>
    <td>domestic national public foreign internal</td>
    <td>domestic private public social family</td>
    <td>0.69</td>
    <td>I</td>
  </tr>
  <tr>
    <td>escaping</td>
    <td>escaping returning flying taken escape</td>
    <td>escaping escape avoiding returning escaped</td>
    <td>0.68</td>
    <td>I</td>
  </tr>
  <tr>
    <td>vest</td>
    <td>vest vest vested place leave</td>
    <td>vest coat jacket shirt hat</td>
    <td>0.67</td>
    <td>L</td>
  </tr>
  <tr>
    <td>training</td>
    <td>training muster organizing regulating exercising</td>
    <td>training raising bringing trained building</td>
    <td>0.67</td>
    <td>L</td>
  </tr>
</table>

Table 10: Constitutional terms with the largest difference in meaning between legal and popular sources in COFEA, shown with most common substitutes, and corresponding JSD values. Terms marked with an asterisk (*) appear to be primarily due to common OCR errors in Hein. The Lean column shows whether the specific mentions of the term in the Constitution appear to lean more towards the typical legal meaning (L), popular meaning (P), or are indeterminate (I).

shown in Table 10, with those which seem to be due to OCR error in HEIN marked with an asterisk. As can be seen, while some differences are related to different syntactic forms (e.g., sign as a verb as opposed to a noun), others show a subtle variation in meaning between legal and popular sources (e.g., resignation, searches, domestic, escaping).

For comparison, Table 11 shows the 15 terms with the greatest meaning variation across sources, according to the method of Hamilton et al. (2016). These results, compared against Table 10 are again broadly similar. This time, there are six terms in common among the top 15, but the correlation between scores is only 0.53. As with the results in the main paper, the lower values here again indicate less variation across sources within COFEA, relative to change over time.

<table>
  <tr>
    <th>Term</th>
    <th>Cosine Distance</th>
  </tr>
  <tr>
    <td>affecting</td>
    <td>0.87</td>
  </tr>
  <tr>
    <td>tender</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>bankruptcies</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>sign</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>discoveries</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td>training</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td>confession</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>sundays</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>aid</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>high</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>likewise</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>privileged</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>reexamined</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>comfort</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>nobility</td>
    <td>0.66</td>
  </tr>
</table>

Table 11: Constitutional terms with the largest meaning change from the founding to the modern era, according to the method of Hamilton et al. (2016).

D.3 Assessing meaning in the Constitution

Although inferring the meanings of individual mentions of terms is inherently difficult, we attempt to characterize the broad meanings of terms as they occur in the Constitution. To do so, we rely on the overall most common meanings inferred from comparing the legal to the popular sources.

For each mention of a term in the Constitution, we gather the k most common replacements, (with \( k = 10 \)), and then compare this set to the corresponding set of k overall most common replacements for each subset (legal vs. popular). That is, let \( R_c^w \) be the set of ten most common replacements for word w from corpus c, for \( c \in \{ \text{Legal} (L), \text{Popular} (P) \} \). For each mention of w in the Constitution, we use the same process to collect the ten most probable replacements for that specific mention, which we denote \( R_m^w \). We then compute the overlap between the set for that mention with corresponding replacements set for legal and popular sources. That is, let \( o_c^w = |R_m^w \cap R_c^w| \). If \( o_c^w > o_p^w + 1 \), then we count that mention of word w as a specialized legal usage. If \( o_p^w > o_c^w + 1 \), then we count it as a popular usage. If neither is satisfied, we count it as indeterminate.

This gives us an imperfect but still useful measure of meaning alignment. For example, consider the word tender. This term has numerous meanings, encompassing nouns, verbs, and adjectives. In the popular sources, the dominant meaning has to do with kindness (kind, soft, generous, great). In
legal sources, by contrast, the dominant meaning has to do with payment (payment, demand, currency, money, security). The word occurs once in the Constitution ("No State shall . . . make any Thing but gold and silver Coin a Tender in Payment of Debts"), from which we can infer that it is being used in the sense of payments. This too is captured by our metric; the top replacements suggested by the model (medium, fund, currency, standard) do not perfectly align with either usage, but are ultimately more similar to the financial sense, which is more common in legal documents, and thus \( o_L^{\text{tender}} > o_P^{\text{tender}} + 1 \). The same applies to terms like *faith* and *sign*, although many are less clear cut.

**D.4 Effect of Continued MLM Training**

Unsurprisingly, because BERT was trained primarily on contemporary text, it is better at predicting masked words from modern documents (i.e., COCA), rather than founding era documents. To mitigate this issue, we make use of continued masked language model training, to adapt the base model to our data, as described in the main paper. As a way of quantifying the impact of this, we compute the proportion of mentions included in our analyses in which the original (masked) word is included in the set of predicted terms, and then compare this to using the corresponding off-the-shelf model.

Using the vanilla *bert-large-uncased* model, (without continued training), we find that there is indeed an imbalance between the early and modern text. The proportion of mentions in which the masked term is included in the set of 10 most likely predictions is 60.0% in the modern documents and only 40.0% in the founding era documents. When using the model that has been adapted to these data (which is what we use for obtaining the results reported in this paper), the corresponding values are 64.6% and 60.0%, respectively, demonstrating that the continued MLM training has been effective in adapting the model to our data.

Reviewer paper summaries: The authors evaluate the feasibility of using masked language models with the Corpus of Founding Era American English (COFEA) for answering questions about semantic change in the meaning of terms in the US Constitution between its drafting and today, and in measuring variation in the meanings of those terms among genres of writing in the period 1760-1800.
Summary of strengths: The paper clearly addresses the composition of the COFEA corpus and the variation in OCR accuracy and date metadata resulting from their variation digitization histories. The authors clearly describe their preprocessing, measurement, and evaluation methods.
Summary of weaknesses: While the authors are not specifically focused on legal applications, it would be good to discuss how the inference methods described in the paper would be applied in examples of legal reasoning drawn from the literature they cite.

It would also be useful to address how syntactic change might be interacting with semantic change in the analysis they perform. For instance, in Table 1, which lists terms in the US Constitution that have changed between the COFEA and the contemporary Englush in COCA, "captures" substitutes for nouns in the 18th century but verbs in the 20th.  It might be useful to also measure change conditioned on part of speech.
Comments, suggestions and typos: It might be useful to check models' fit to the data by seeing how often they suggest the original masked word. This might be a way of evaluating the effectiveness of the continued pretraining of BERT on COFEA. (It might also be useful to compare predictions of the original BERT model.)

In the debates on originalism, "ordinary meaning" is sometimes expanded to "ordinary public meaning". It might be useful to connect this distinction to the legal vs. popular genres analysis.

Line 835: When you sampled up to 4000 occurrences here, I'm confused about how that relates to the 5000 occurrences sampled in line 178.
Overall assessment scores: 4
Metareview score: 4
Metareview: The paper studies semantic change comparing the COFEA corpus, a corpus of American English from the Founding Era to both modern English and English from a different domain in the same era. BERT-based representations are used to calculate similarities.
Summary of reasons to publish: - clear and useful presentation of a corpus that is potentially interesting for other researchers
- generally clearly described experiments that provide further insights in the corpus
- thorough description of limitations which can serve to caution non-expert users of these methods
- the paper provides an appropriate contribution in terms of substance for a short paper