Comparing Measures of Textual Innovation
Creators: name: Underwood, William E
name: Cohen, Rebecca
name: Griebel, Sarah
name: Park, Jaihyun
name: Jiayu Liu
name: Perkins, Jana M.
name: Li, Lucian
Publisher: name: OSF Registries
Year: 2023
Subjects: subject: Sociology of Culture
subject: Modern Literature
subject: Digital Humanities
subject: Arts and Humanities
subject: Library and Information Science
subject: Social and Behavioral Sciences
subject: Sociology
subject: FOS: Sociology
subject: cultural analytics
subject: science of science

description: Sarah Griebel, Becca Cohen, Lucian Li, Jiayu Liu, Jaihyun Park, Jana M. Perkins, Ted Underwood ## Research questions This preregistration applies to the first stage of a two-stage research project. Our goal in this first stage is to test different methods of assessing durable textual innovation: things writers do that depart from past practice, and also endure into the future. We will sometimes call durable innovation “precocity” for short—a word that usefully avoids any implication that a writer was “the first person to” do something, and claims simply that their behavior is more typical of a later point in time. Recent papers have found that measures of innovation correlate strongly with external evidence about the social impact of a text—e.g., evidence from citations (Zhang 2021) and reviews and sales (Underwood and So 2021). If this correlation is strong enough, measures of innovation might give us a way to inquire about change in domains where we don’t have external evidence of impact (which include most historical domains). If we trusted our textual metrics enough, we could also start to use the differences and lags between textual behavior and external evidence of impact to cast light on processes of change, and tease out cases of unacknowledged or delayed influence. For all of these reasons, we want to ensure that we have the best possible measure of textual innovation. In particular, if we plan to interpret discrepancies between textual behavior and social acknowledgment as meaningful lags, we may want to ensure that our textual metric tracks social impact as well as possible, and isn’t systematically biased in some way unrelated to the question. We therefore propose to test many different textual metrics of innovation, in order to identify the ones that match social impact most closely. We will use what we learn to inform the second stage of this project, which will ask about the importance of cohort succession in cultural change, with special attention to the lag (if any) between innovation and social recognition. ## Related research Two recent papers, Barron et al. (2018) and Vicinanza, Goldberg, and Srivastava (2022), have taken different approaches to measuring durable innovation. The former calculates KL divergence scores between topics discussed in speeches as compared to their past and to their future (using the term resonant to describe speeches that are more surprising when compared to the past than the future). The latter finetunes BERT models for specific slices of time to compare sentence word embeddings to their present and to their future, using the term prescience for those sentences that have a high probability of occurring in the future and a low probability in the present. These are both interesting measures of innovation, but they make significantly different assumptions. Since the Barron (2018) method relies on topic modeling, it measures only lexical distribution. The Vicinanza (2022) method measures word probability in context, but compares sentences only to the future: it produces a measure of prescience rather than durable innovation. We propose to test variations of both approaches, as well as an intermediate approach that uses neural document embeddings, but compares them to the future and the past (like Barron 2018). ##Data We have chosen to analyze a corpus from academic journals as well as a 19th-century fiction corpus. One reason to compare two domains is that they are documented with different kinds of social evidence. Reviews, prizes, and sales provide immediate feedback on fiction, while academic citations are more delayed. There are also some a priori reasons to imagine that innovation might work differently in artistic expression than in academic research; we hope to find a metric that works reasonably well for both. For our academic corpus, we selected influential journals in literary studies and economics, with a slight bias toward long-running journals, since that helps assure stability of the corpus across a century-long timespan. We gathered citation counts from Semantic Scholar as external evidence of impact. Our 19th-century fiction dataset comes from HathiTrust and is part of a dataset previously used in Underwood and So 2021. The primary external evidence of impact for this corpus are reviews in prestigious literary magazines. It’s a binary signal, where some books were found in a sample of reviews, and others were not. We may have some additional evidence re: sales and critical opinion, but most data about that will come from a later period in the second phase of our project. ## Methods We plan to compare three different measures of durable innovation. ### Topic modeling. For their comparisons, the Barron paper tracked “word-use patterns” over time within the speeches, relying on a topic model. We’ll take this method as our baseline, using a hyperparameter of 250 topics, and a chunk size of 500 words (or slightly more)—parameters that have already performed well in many similar problems. Novelty and transience will be measured as KL divergence. We’ll also test a method adopted by Underwood and So (2021), which assesses “novelty” (difference from the past) and “transience” (difference from the future) by comparing texts only to the 5% of most similar texts in each past or future year, across a 20-year window both forward and backward. The intuition for limiting the scope of comparison to the top 5% is that it’s necessary to address the highly-segmented character of the book market. (Mysteries innovate relative to other mysteries, not relative to YA paranormal fiction.) A similar intuition might extend to journal articles: we could limit our comparisons within subjects and sub-disciplines. This limited scope improved correlation with social evidence about fiction in Underwood and So (2021). But it might not apply to the more fluid subdivisions of an academic discipline. So we will test both approaches: comparing each chunk to all chunks in the past or future, and only to the 5% that are most similar. Finally, we’ll consider the possibility that significant innovation takes place only in a few parts of a text by testing two different aggregation strategies. Our baseline is to measure precocity for each 500-word chunk and then take mean precocity to produce a document-level score. But we’ll also consider scores produced by taking only the top 25% of chunks for a given document. So, for the document representation produced by topic models, we’ll actually be testing a 2x2 grid of different measurement scopes and aggregation strategies. ### Document embeddings. We’ll also strive to improve Barron’s method of past and future comparisons by applying it to a different document representation: pre-trained GTE embeddings by Li et al. (2023) which use contrastive learning for many different datasets/contexts. In order to use such GTE embeddings, a lot of pre-processing is required. First, we need to divide each file into pages and sentences. For pages where the sentence runs over onto the next page, we will write code that looks forward to the punctuation (or otherwise sentence delimiter) on the next page, and add the full sentence to the previous page where it started. Each page will be a list of sentences. Next we will create a front and back matter trimmer, to remove publication pages etc that are not the actual text of the story or journal itself that we are interested in. To do this, we will train a predictive model. Finally, we will cut the text into chunks of 512 tokens or less, with complete sentences. Thus this involves iterating over the text through the GTE embedder, cutting the text off at 512 tokens, and then cutting off any partial sentences at the end of the chunk (so that the chunk ends with the last complete sentence). We’ll compare document embeddings using cosine distance rather than KL divergence, since an embedding is not a probability distribution. Also, embeddings will use chunks slightly smaller than those used in topic models, because topic models don't perform optimally with short documents, but our embeddings are capped at 512 words. However, we will use the same 20-year time window for both representations, to help create an apples-to-apples comparison. We will also test embeddings on the same 2x2 grid of comparison scopes and aggregation methods described above for topic models. We hope that the ability of document embeddings to capture both semantic and stylistic factors (like word order) may improve on previous methods, producing a stronger correlation between textual precocity and external evidence of impact. But we remain open to the possibility that topic models will perform nearly as well as, or even better than, embeddings, since semantic factors often seem in practice to dominate even neural document representations. If the difference between methods is small, it might not rule out use of topic models, since they can use evidence that is more easily obtained. To avoid circularity, we will avoid comparing the chunks in paper X to any paper that shares an author with it — and to chunk y of a paper Y if: 1. Y cites X *and* 2. Either author names from X are mentioned in chunk y, or 3. A sequence of six or more words from X is quoted in chunk y. The goal here is not to completely exclude papers that cite X (they are after all a legitimate part of its influence on the future). We’re merely trying to exclude passages that would closely echo X because they actually quote language from X or paraphrase it. That form of textual similarity is so directly produced by the act of citation that it would seem odd to consider it as a separate variable; it will correlate with citation in an uninformatively circular way. (As a sanity check, we will also calculate precocity with these chunks included; we assume it will produce a stronger correlation with citation, but we don’t expect to consider that a meaningful advantage for a method.) In general, we will only consider our results meaningful if there is a positive relationship between textual similarity and external evidence of impact across the majority of methods described above. In the case of reviews, we'll do a t-test for difference of means; for citations of academic papers, Pearson's correlation with log-transformed number of citations. The average effect size can be small but should not be miniscule; let’s say Cohen’s d &gt; 0.02, and p &lt; 0.05. We will also only consider a difference between methods meaningful if it produces a significant difference in effect size after bootstrap resampling and a correction for multiple comparisons. ### Contextual improbability. Finally, we’ll consider a third method introduced by Vicinanza et al. (2022) which fine-tunes separate BERT models on different periods to estimate the contextual improbability of a text in each period. (They call this measure of improbability “contextual novelty,” although the phrase is slightly confusing since time is not a factor when only one period is being measured). When the improbability of a text decreases in a future period, they call that “prescience.” Specifically, Vicinanza et al. calculate the perplexity of each word in a sentence (the inverse of the probability the model assigned to the word), and then take the geometric mean of all perplexities in a sentence. At the document level, they average sentence perplexities arithmetically to produce a contextual novelty score. Then they compute prescience as the percentage decrease in contextual novelty: (CNt0 - CNt1) / CNt0 We’ll divide texts into four-year time slices, and train a separate model for each slice — including 4 years on either side to create a robust fine-tuning corpus (although the model will not be applied to those shoulder periods). For example, the model we use to predict the years 1960-1963 (inclusive) will also train on documents from 1956-1959 and 1964-1967 in order to reduce potential temporal bias. We’ll calculate prescience using a method close to Vicinanza: the percentage decrease between CNpresent and CN12yearsinfuture. In other words, documents from the period 1960-63 described above (characterized by a model trained on 1956-67) would be compared to improbabilities estimated by a model trained on the (non-overlapping) period 1968-1979. This allows us to cover the whole timeline with robust 12-year models while moving them forward only 4 years at a time. We’ll also try a variant related to the strategy in Barron et al. (2018): the percentage decrease from CN12yearsinpast to CN12yearsinfuture. This will favor documents that were becoming steadily less improbable across the whole 36-year window. The comparisons we’re making (across 24 or 36 years) are roughly comparable to the 20- and 40- year windows we’ll be using for topic models and document embeddings. We can’t duplicate the scope restriction to 5% of most-similar documents with this method (since it would require training far too many document-specific models). For the same reason, this model-based method will be unable to exclude cases of text-reuse or articles/books written by the same authors from the future corpus being compared to a present text. It will thus be inherently biased toward prolific authors and widely-quoted texts. For this reason, Vicinanza's method should be compared to a version of the other methods that operates on all texts, without the 5% restriction, or the text-reuse and authorship filters. On the other hand, we can test an aggregation strategy that calculates prescience at a sentence level and then describes documents by aggregating only the 25% of most-prescient sentences. This might hypothetically be more appropriate for academic articles than for fiction, but it’s an empirical question. Lacking previous experience with Vicinanza’s method, we are less confident that it will produce meaningful results. Failure to find a positive correlation with social evidence here would not vitiate the Barron method, which has already replicated several times. On the other hand, if it does produce a positive correlation, we'll need to ask how much of that is due to the method's inherent bias toward prolific authors. Running multiple versions of our other methods (with and without text-reuse &amp; authorship filters) will help us tease out an answer. ##References Alexander T. J. Barron, Jenny Huang, Rebecca L. Spang, and Simon DeDeo (2018). Individuals, institutions, and innovation in the debates of the French Revolution. http://doi.org/10.1073/pnas.1717729115 Devlin, J., Chang, M.-W., Lee, K., &amp; Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv. https://arxiv.org/abs/1810.04805 Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P., &amp; Zhang, M. (2023). Towards General Text Embeddings with Multi-stage Contrastive Learning. arXiv. https://arxiv.org/abs/2308.03281 Paul Vicinanza, Amir Goldberg, and Sameer B Srivastava (2023). A Deep-Learning Model of Prescient Ideas Demonstrates That They Emerge from the Periphery (https://doi.org/10.1093/pnasnexus/pgac275 ) Ted Underwood and Richard Jean So. 2021. “Can We Map Culture?” Journal of Cultural Analytics. https://culturalanalytics.org/article/24911 Zhang, X., Xie, Q., &amp; Song, M. (2021). Measuring the impact of novelty, bibliometric, and academic-network factors on citation count using a neural network. Journal of Informetrics, 15(2), 101140. doi:10.1016/j.joi.2021.101140