Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
Authors: Kent K. Chang
Mackenzie Hanh Cramer
Sandeep Soni
David Bamman
Venue: EMNLP 2023 Main (2023)
Keywords: membership inference
memorization
cultural analytics
large language models

Abstract: In this work, we carry out a data archaeology to infer books that are known to ChatGPT and GPT-4 using a name cloze membership inference query. We find that OpenAI models have memorized a wide collection of copyrighted materials, and that the degree of memorization is tied to the frequency with which passages of those books appear on the web. The ability of these models to memorize an unknown set of books complicates assessments of measurement validity for cultural analytics by contaminating test data; we show that models perform much better on memorized books than on non-memorized books for downstream tasks. We argue that this supports a case for open models whose training data is known.

Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT--
Kent K. Chang
Mackenzie Cramer
University of California, Berkeley kentkchangeberkelley. edu
University of California, Berkeley mackenzie. hanh@berkeley. edu
Sandeep Soni
David Bamman"
Emory University sandeep. soni coni@emory.co edu
University of California, Berkeley
dbamman@berkeley. edu
Abstract
Wow. sit down, fish the questions from my backpack, and go through them, inwardly curs- ing [MASK] for not providing me with a brief biography. I know nothing about this man I'm about to interview. He could be ninety or he could be thirty. + Kate (James, Fifty Shades of Grey).
In this work, we carry out a data archaeology to infer books that are known to ChatGPT and GPT-4 using name cloze membership infer- ence query. We find that OpenAl models have memorized a wide collection of copyrighted materials, and that the degree of memorization is tied to the frequency with which passages of those books appear on the web. The ability of these models to memorize an unknown set of books complicates assessments of measure- ment validity for cultural analytics by contami- nating test data; we show that models perform much better on memorized books than on non- memorized books for downstream tasks. We argue that this supports a case for open models whose training data known.
Some days later, when the land had been moist- ened by two or three heavy rains, [MASK] and his family went to the farm with baskets of seed- yams, their hoes and machetes, and the plant- ing began. + Okonkwo (Achebe, Things Fall Apart).
Figure 1: Name cloze examples. GPT-4 answers both of these correctly.
1 Introduction
Research in cultural analytics at the intersection of NLP and narrative is often focused on developing algorithmic devices to measure some phenomenon of interest in literary texts (Piper et al., 2021; Yo- der al., 2021: Coll Ardanuy al., 2020; Lvvans and Wilkens, 2018). The rise of harge-pretraince language models such as ChatGPT and GPT-4 has the potential to radically transform this space by both reducing the need for large-scale training data for new tasks and lowering the technical barrier to entry (Underwood, 2023).
what books a model has been trained on is critical to assess such sources of bias (Gebru et al., 2021), which can impact the validity of results in cultural analytics: if evaluation datasets contain memorized books, they provide false measure of future perfor- mance on non-memorized books; without knowing what books a model has been trained on, we are unable to construct evaluation benchmarks that can be sure to exclude them.
In this work, we carry out a data archaeology to infer books that are known to several of these large language models. This archaeology is member- ship inference query (Shokri et al., 2017) in which probe the degree of exact memorization (Tiru- mala al., 2022) for sample of passages from 571 works of fiction published between 1749-2020. This difficult name cloze task, illustrated in figure 1, has 0% human baseline performance.
At the same time, however, these models also present challenge for establishing the validity of results, since few details are known about the data used to train them. As others have shown, the accu- racy of such models is strongly dependent on the frequency with which a model has seen informa- tion in the training data, calling into question their ability to generalize (Razeghi et al., 2022; Kandpal et al., 2022a; Elazar et al., 2022); in addition, this phenomenon is exacerbated for larger models (Car- lini al., 2022; Biderman al., 2023). Knowing
This archaeology allows us to uncover a number of findings about the books known to OpenAl mod- els which can impact downstream work in cultural analytics:
Details of author contributions listed in the appendix.
1. OpenAl models, and GPT-4 in particular, have

---

memorized wide collection of in-copyright books.
Elam, 2023). In this light, our archaeology of books shares the Foucauldian impulse to "find a common structure that underlies both the scien- tific knowledge and the institutional practices of an age" (Gutting, 1989, p. 79; Schmidgen et al., 2023). to the inference tasks that involve them. Investi- gating data membership helps reflect how to best use LLMs for large-scale cultural analysis and evaluate the validity of its findings.
2. While others have shown that LLMs are able to reproduce some popular works (Henderson et al., 2023), we measure the systematic bi- ases in what books OpenAl models have and memorized, with the most strongly mem- orized books including science fiction/fantasy novels, popular works in the public domain, and bestsellers.
LLMs for cultural analysis. While they are more often the object of critique, LLMs are gaining prominence in large-scale cultural analysis part of the methodology. For example, some GPT models to tackle problems related to interpretation. that of real events (Hamilton and Piper, 2022), or of character roles (hero, villain, and victim, Stamm- bach et al., 2022). Others leverage LLMs on classic NLP tasks that can be used to shed light on literary and cultural phenomena, otherwise maintain an interest in those humanistic domains (Ziems et al., 2023). This work shows what tasks and what types of questions LLMs are more suitable to answer than others through our archaeology of data.
3. This bias aligns with that present in the gen- eral web, as reflected in search results from Google, Bing and C4, This confirms prior findings that duplication encourages memo- rization (Carlini et al., 2023), and provides a rough diagnostic for assessing knowledge about a book.
4. Disparity in memorization leads to disparity in downstream tasks. GPT models perform bet- ter on memorized books than non-memorized books at predicting the year of first publication for a work and the duration of narrative time for a passage, and are more likely to generate character names from books it has seen.
Documenting training data. Training on large text corpora such as BookCorpus (Zhu al., 2015) 5),, C4 (Raffel al., 2020) and the Pile (Gao et al., 2020) has been instrumental in extending the ca- pability of large language models. Yet, in contrast to smaller datasets, these large corpora are less carefully curated. Besides a few attempts at docu- menting large datasets (e.g.. C4; Dodge et al., 2021) or laying out diagnostic techniques for their qual- ity (Swayamdipta al., 2020), these corpora and their use in large models is less understood. Our focus on books in this work is an attempt to em- pirically map the information about books that is present in these models.
While our work is focused on ChatGPT and GPT- 4, we also uncover surprising findings about BERT as well: BookCorpus (Zhu et al., 2015), one of BERT's training sources, contains in-copyright ma- terials by published authors, including E.L. James' Fifty Shades of Grey, Diana Gabaldon's 's Outlander and Dan Brown's The Lost Symbol, and BERT has memorized this material as well.
As researchers cultural analytics are poised to use ChatGPT and GPT-4 for the empirical analy- sis of literature, our work both sheds light on the underlying knowledge in these models while also illustrating the threats to validity in using them.
Memorization. Large language models have shown impressive zero-shot or few-shot ability but they also suffer from memorization (Elangovan et al., 2021; Lewis et al., 2021). While memoriza- tion is shown in some cases to improve general- ization (Khandelwal et al., 2020), it has generally been shown to have negative consequences, includ- ing security and privacy risks (Carlini et al., 2021, 2023; Huang et al., 2022). Studies have quantified the level of memorization in large language mod- els (e.g., Carlini et al., 2022; Mireshghallah et al., 2022) and have highlighted the role of both ver- batim (e.g., Lee et al., 2022; Ippolito et al., 2022)
2 Related Work
Knowledge production and critical digital hu- manities. The archaeology of data that this work presents can be situated in the tradition of tool cri- tiques in critical digital humanities (Berry, 2011; Fitzpatrick, 2012; Hayles, 2012; Ramsay and Rock- well, 2012; Ruth et al., 2022), where we crit- ically approach the closedness and opacity of LLMs, which can pose significant issues they are used reason about literature (Elkins and Chun, 2020; Goodlad and Dimock, 2021; Henrickson and Prrono,P,,uel,,, 2022; Schmidgen et al., 2023;

---

You have seen the following passage in training data. What is the proper name that fills in the [MASK] token in it? This name exactly one word long, and is a proper name (not a pronoun or any other word). You must make guess, even if you are uncertain.
Example:
Input: Stay gold, [MASK], stay gold.
Output:
<hamesPonyhoy<cnames
Input: The door opened, and [MASK], dressed and hatted, entered with cup of tea.
Output: <hrt//eeety//nmmme>>
Input: My back's the window. expect stranger, but it's [MASK] who pushes open the door, flicks the light. can't place that, unless he's one of them. There was always that possibility. Output:
Figure 2: Sample name cloze prompt.
and subtler forms of memorization (Zhang et al., 2021). Henderson et al. (2023) in particular ex- amines the legal issues surrounding copyright in foundation models, focusing in particular the length of content extraction of copyrighted materi- als (including books as a case study) that poses the most risk to fair use determinations. Our analysis and experimental findings on the disparate sources of memorization and impact on research in cultural analytics add to this scholarship.
cloze, a version of exact memorization (Tirumala et al., 2022). Unlike other cloze tasks that focus on entity prediction for question answering/reading comprehension (Hill et al., 2015; Onishi al., 2016). no names all appear in the context inform the cloze fill. In the absence of information about each particular book, this name should be nearly impossible to predict from the context alone; it requires knowledge not of English, but rather about the work in question.
The name cloze setup is also particularly suit- able for data archaeology for ChatGPT//PT as opposed to perplexity-based measures (Gonen et al., 2022), since OpenAl does not the time of writing share word probabilities through their API.
Data contamination. A related issue noted upon critical scrutiny of uncurated large corpora is data contamination (e.g., Magar and Schwartz, 2022) raising questions about the zero-shot capability of these models (c.g.. Blevins and Zettlemoyer, 2022) and worries about security and privacy (Carlini al., 2021, 2023). For example, Dodge et al. (2021) find that text from NLP evaluation datasets is present in C4, attributing performance gain partly to the train-test leakage. Lee et al. (2022) show that C4 contains repeated long running sentences and near duplicates whose removal mitigates data contamination: similarly, deduplication has also been shown alleviate the privacy risks in models trained on large corpora (Kandpal et al., 2022b).
3.2 Evaluation
We construct this evaluation set by running BookNLP' over the dataset described below in $4, extracting all passages between 40 and 60 tokens with single proper person entity and no other named entities. Each passage contains complete sentences, and does cross sentence boundaries. We randomly sample 100 such passages per book, and exclude any books from our analyses with fewer than 100 such passages.
We pass each passage through the prompt listed in figure 2, which is designed to elicit a single word, proper name response wrapped in XML tags; two short input/output examples are provided illustrate the expected structure of the response.
3 Task
3.1
Name cloze
We formulate our task as cloze: given some con- text, predict single token that fills in mask. To account for different texts being more predictable than others, focus hard setting of predict- ing the identity of single name in passage of 40-60 tokens that contains no other named entities, Figure illustrates two such examples. In the following, we refer to this task as a name
In establishing baselines using the same evalua- tion set, predicting the most frequent name the dataset ("Mary") yields an accuracy of 0.6%. To assess human performance on this task, one of the authors of this paper followed the same instruc-
hhtpps://githhi https:///github.commboooknn
com

---

tions given ChatGPT/GPT-4, only using informa- tion present in the passage guess the masked name (i.e., without using any external sources of in- formation such as Google); the resulting accuracy is 0%, which suggests there is little information within the context that would provide a signal for what the true name should be.
Appendix presents the same for books published after 1928. Of particular interest in this list is the dominance of science fiction and fantasy works, including Harry Potter, 1984, Lord of the Rings, Hunger Games, Hitchhiker's 's Guide to the Galaxy, Fahrenheit 451, A Game of Thrones, and Dune- 12 of the top 20 most memorized books in copy- right fall in this category. Table 2 explores this in more detail by aggregating the performance by the top-level categories described above, including the specific genre for our subset of genre fiction.
4 Data
We evaluate 5 sources of English-languuge fiction:
91 novels from LitBank, published before 1923.
GPT-4 and ChatGPT are widely knowledge- able about texts in the public domain (included in pre-1923 LitBank); it knows little about works of Global Anglophone texts, works in the Black Book Interactive Project and Black Caucus Ameri- can Library Association award winners.
90 Pulitzer prize nominees from 1924-2020.
95 Bestsellers from the NY Times and Publishers Weekly from 1924-2020.
101 novels written by Black authors, either from the Black Book Interactive Project Black Cau- cus American Library Association award winners from 1928-2018
Source
GPT-4
ChatGPT
pre- pre-1923 LitBank
0.244
0.072
Genre: SF/Fantasy
0.235
0.108
95 works of Global Anglophone fiction (outside the U.S. and U.K.) from 1935-2020.
Genre: Horror Bestsellers
0.054
0.028
0.033
0.016
Genre: Action/Spy
0.032
0.007
99 works of genre fiction, containing science fiction/fantasy, horror, mystery/crime, romance and action/spy novels from 1928-2011
Genre: Mystery/Crime
0.029
0.014
Genre: Romance
0.029
0.011
Pulitzer
0.026
0.01
Global
0.020
0.009
BBIP/BCALA
0.017
0.011
Pre-1923 LitBank texts are born digital Project Gutenberg and are in the public domain in the United States; all other sources were created by purchasing physical books, scanning them and OCR ing them with Abbyy FineReader. As of the time of writing, books published after 1928 are generally in copyright in the U.S.
Table 2: Name cloze performance by book category.
5.2 BERT
For comparison, we also generate predictions for the masked token using BERT (only passing the passage through the model and not the prefaced instructions) to provide baseline for how of- ten model would guess the correct name when simply functioning as a language model (uncon- strained to generate proper names). As table illustrates, BERT's performance is near for all books-excepp for Fifty Shades of Grey, for which it guesses the correct name 13% of the time, sug- gesting that this book was known to BERT during training. Devlin et al. (2019) notes that BERT trained Wikipedia and the BookCorpus, which Zhu et al. (2015) describe as "free books written by yet unpublished authors." 74 Manual inspection of the Book BookCorpus hosted by huggingface confirms
5 Results
5.1 ChatGPT/GPT-4
We pass all passages with the same prompt through both ChatGPT sopt-3.5-turbol and GPT- 4 (gpt-4), using the OpenAI API. The total cost of this experiment with current OpenAl pricing #S0.002/thousand tokens for ChatGPT; $0.03/thousand tokens for GPT-4) is approximately $400. We measure the name cloze accuracy for book as the fraction of 100 samples from it where the model being tested predicts the masked name correctly.
Table presents the top 20 books with the high- est GPT-4 name cloze accuracy. While works in the public domain dominate this list, table in the
For complete results on all books, see https: / thub. com/ Com/bamman- group/gor books-books..
"Fifty Shades of Grey was originally self-published on fanfiction cctioonne nnet ca. 2009 before being published by Vintage Books in 2012.
Phttp://bbip. ku. www.ee/oooo-oolleetttiiiee
hht http://hguggggaceeeeee/ co/ toattttsstss/ooooooooo

---

GPT4
CharGPT
BERT
Author
Title
0.98
0.82
0.00
1865
Lewis Carroll
Alice's Adventures in Wonderland
0.76
0.43
0.00
1997
J.K. Rowling
Harry Potter and the Sorcerer's Stone The Scorlet Letter
0.74
0.29
0.00
1850
Nathaniel Hawthorne
0.72
0.11
0.00
1892
Arthur Conan Doyle
The Adventures of Sherlock Holmes
0.70
0.10
0.00
1815
Jane Austen
Enma
0.65
0.19
0.00
1823
Mary W. Shelley
Frankenstein
0.62
0.13
0.00
1813
Jane Austen
Pride and Prejudice
0.61
0.35
0.00
1884
Mark Twain
Adventures of Huckleberry Finn
0.61
0.30
0.00
1853
Herman Melville
Bartleby, the Scrivener
0.61
0.08
0.00
1897
Bram Stoker
Dracula
0.61
0.18
0.00
1838
Charles Dickens
Oliver Twist
0.59
0.13
0.00
1902
Arthur Conan Doyle
The Hound of the Baskervilles
0.59
0.22
0.00
1851
Herman Melville
Moby Dick; Or, The Whale
0.58
0.35
0.00
1876
Mark Twain
The Adventures of Tom Sawyer 1984
0.57
0.30
0.00
1949
George Orwell
0.54
0.10
0.00
1908
L. M. Montgomery
Anne of Green Gables The Fellowship of the Ring
0.51
0.20
0.01
1954
J.R.R. Tolkien
0.49
0.16
0.13
2012
E.L. James
Fifty Shades of Grey
0.49
0.24
0.01
1911
Frances H. Burnett
The Secret Garden
0.49
0.12
0.00
1883
Robert L. Stevenson
Treasure Island
0.49
0.16
0.00
1847
Charlotte Brontë
Jane Eyne: An Autobiography
0.49
0.22
0.00
1903
Jack London
The Call of the Wild
Table 1: Top 20 books by GPT-4 name cloze accuracy.
that Fifty Shades of Grey present within it, along with several other published works, including Di- ana Gabaldon's 's Outlander and Dan Brown's The Lost Symbol.
from H:
Pr (ē E Slerror)) Pr (2 E Ulerror)
log
where is the predicted character. To test for sta- tistical significance, perform randomization test (Dror et al., 2018), where the observed statistic is compared to a distribution of the same statistic calculated by randomly shuffling the names be- tween and U.
6 Analysis
6.1 Error analysis
We analyze examples on which ChatGPT and GPT- make errors assess the impact of memorization. Specifically, we test the following question: when a model makes a name cloze error, is it more likely to offer a name from a memorized book than a non-memorized one?
We find that for both ChatGPT (o = 1.34, <
0.0001) and GPT-4 (o 1.37, < 0.0001), the null hypothesis can be rejected, indicating that both models are more likely to predict character name from book they had memorized than character from a book they have not. This has important consequences: these models do not simply perform better on a set memorized books, but the in- formation from those books bleeds out into other narrative contexts.
To test this, we construct sets of seen (S) and unseen (U) character names by the models. To do this, we divide all books into three categories: M books the model has memorized (top decile by GPT-4 name cloze accuracy). -M as books the model has not memorized (bottom decile), and H as books held out to test the hypothesis. We identify the true masked names that are most associated with the books in M by calculating the positive pointwise mutual information between name and book pair- obtain set S, and the masked names most associated with books in -M to obtain set U. We also ensure that S and U are of the same size and have no overlap. Next, we calculate the observed statistic as the log-odds ratio on examples
6.2 Extrinsic analysis
Why do the GPT models know about some books more than others? As others have shown, dupli- cated content in training is strongly correlated with memorization (Carlini et al., 2023) and foundation models are able to reproduce popular texts more than random ones (Henderson al., 2023). While the training data for ChatGPT and GPT-4 un- known, it likely involves data scraped from the web, as with prior models' use of WebText and C4.

---

To what degree is a model's performance for book in our name cloze task correlated with the number of copies of that book on the open web? We assess this using four sources: Google and Bing search engine results, C4, and the Pile.
them both: popularity. Moby Dick, for example. widely read: many copies of it appear in univer- sity libraries, multiple copies of it may appear large-scale training datasets (such as Books3, part of Pile), quotations of it may appear in unindexed academic journals, and as we see here, it appears in multiple places on the public web (both in snip- pets and in the form of full text). All four of these factors are likely correlated with each other and commonly explained by the overall popularity of the text itself (coneretely, by an unobservable quan- tity such as the number of times it has been read worldwide). The correlation we see between web prevalence and memorization may be an indirect reflection of this latent causal graph.
For each book in our dataset, we sample 10 pas- sages at random from our evaluation set and select a 10-gram from it; we then query each platform to find the number of search results that match that exact string. We use the custom search API for Google, the Bing Web Search API? and indexes to C4 and the Pile by A12 (Dodge et al., 2021). 8 Table lists the results of this analysis, display- ing the correlation (Spearman p) between GPT-4 name cloze accuracy for book and the average number of search results for all query passages from it.
Domain
Hits
archive. org
337
Date
Google
Bing
C4
Pile
academia edu
257
pre-1928
0.74
0.70
0.71
0.84
goodreads. com
234
19st-1928
0.37
0.41
0.36
0.21
coursehero.. com
197
quizlet.
181
litcharts.
148
Table 3:
Correlation (Spearman p) between GPT-4 name cloze accuracy and number of search results in Google, Bing, C4 and the Pile.
fliphtm15. com
124
genius.com 109
118
amazon. com
issuu,
98
For works in the public domain (published be- fore 1928), 0.001) correlation between GPT-4 name cloze ac- curacy and the number of search results across all sources. Google, for instance, contains an average 2,590 results for 10-grams from Alice in Wonder- land, 1, 100 results from Huckleberry Finn and 279 results from Anne of Green Gables. Works in copy- right (published after 1928) show up frequently on the web well. While the correlation is not as strong as public domain texts (in part reflecting the smaller number of copies), it is strongly significant as well (p < 0.001). Google again has an average of 3,074 search results across the ten 10-ngrams we query for Harry Potter and the Sorcerer's Stone, 92 results for The Hunger Games and 41 results for A Game of Thrones.
Table 4: Sources for copyrighted material.
strong and significant (p <
Table 4 lists the most popular sources for copy- righted material. Notably, this list does not only in- clude sources where the full text is freely available for download as single document, but also sources where smaller snippets of the texts appear in re- views (on Goodreads and Amazon) and in study resources (such as notes and quizzes on Course- Hero, Quizlet and LitCharts). Since our querying process samples ten random passages from a book (not filtered according to notability or popularity in any way), this suggests that there exists sig- nificant portion of copyrighted literary works in the form of short quotes or excerpts on the open web. On Goodreads, for example, book can have a dedicated page of quotes from it added by users of the website (in addition to quotes highlighted in their reviews; ef. Walsh and Antoniak, 2021); on LitCharts, the study guide of a book often in- cludes detailed chapter-by-chapter summary that includes quotations.
While this does not indicate direct causation be- tween web prevalence and memorization, spec- ulate that there is hidden confounder that explains
v1/overview
"nttps::/wwwnncrrosfttt.t www.iiiccooooooo//o.ooooo com//en-us/bing/ppis/
bing-web-search-app
In addition, this also suggests that books that are reviewed and maintain an online presence or as- signed as course readings are more likely to appear on the open web, despite copyright restrictions.
"https://c4-sarchh apps.allenaie .org
'cf. "scemed to know where he was going, he was abvi- ously"; "He hent and pulled the ring of the trapdnor, which"; "I got few extra books for background reading, and"

---

this light, observations from literary scholars and sociologies remain pertinent: Bourdieu's (1987) concerns, that literary taste reflects and reinforces social inequalities, could also be relevant in the context of LLMs. Guillory's 's (1993)'s 's critique on canon formation, which highlights the role of ed- ucational institutions, can still serve a potent reminder of the impact of literary syllabi: since they have influence over what books should have flashcards and study guides created for them, they also indirectly sway which books might become more prominent on the open web, and likely in the training data for LLMs.
top 10% and bottom 10% by GPT-4 name cloze accuracy.
As table 5 shows, we see strong disparities in performance result of model's familiarity with book. ChatGPT refrains from making a predic- tion in only 4% of books knows well, but 9% in books it does not (GPT-4 makes valid prediction in all cases). When ChatGPT makes a valid predic- tion, the mean absolute error is much greater for the books it has not memorized (29.8 years) than for books it has (3.3 years); and the same holds for GPT-4 as well (14.5 years for non-memorized books; 0.6 years for memorized ones). Over all books with predictions, increasing GPT name cloze accuracy is linked decreasing error rates at year prediction (Spearman = -0.39 for ChatGPT, 0.37 for GPT-4, < 0.001).
7 Effect on downstream tasks
The varying degrees to which ChatGPT and GPT-4 have seen books their training data has the po- tential to bias the accuracy of downstream analyses. To test the degree to which this is true, we carry out two predictive experiments using GPT-4 in few-shot scenario: predicting the year of first publi- cation of a work from 250-word passage, and pre- dicting the amount of time that has passed within it. Note that here we are interested in the effects disparity in memorization on downstream tasks, not any individual model's 'ss performance on them, and for this reason, we focus on a few-shot setup without any fine-tuning.
Predicting the date of first publication is typi- cally cast as textual inference problem, estimat- ing a date from linguistic features of the text alone (e.g., mentions of airplanes may date a book to after the turn of the 20th century; thou and thee offer evidence of composition before the 1 18th cen- tury, etc.). While it is reasonable to believe that memorized access to an entire book may lead to improved performance when predicting the date from a passage within it, another explanation is simply that GPT models are accessing encyclope- dic knowledge about those works works-i.e.. identifying a passage as Pride and Prejudice and then access- ing learned fact about that work. For tasks that can be expressed as facts about text (e.g., time of publication, genre, authorship), this presents significant risk of contamination.
7.1 Predicting year of first publication
Our first experiment predicts the year of first publi- cation of book from random 250-word passage of text within it, a task widely used in the digital hu- manities when the date of composition for a work is unknown or contested (Jong et al., 2005; Kumar et al., 2011; ;; Bamman et al., 2017; Karsdorp and Fonteyn, 2019).
7.2 Predicting narrative time
Our second experiment predicts the amount of time that has passed in the same 250-word passage sam- pled above, using the conceptual framework of Un- derwood (2018), explored in the context of GPT-4 in Underwood (2023). Unlike the year prediction task above, which can potentially draw on encyclo- pedic knowledge that GPT-4 has memorized, this task requires reasoning directly about the informa- tion in the passage, as each passage has its own duration. Memorized information about com- plete book could inform the prediction about all passages within it (e.g., through awareness of the general time scales present in a work, or the amount of dialogue within it), in addition to having exact knowledge of the passage itself.
For each of the books in our dataset, sam- ple a random 250-word passage (with complete sentences) and pass through the prompt shown in figure in the Appendix to solicit a four-digit date as a prediction for the year of first publica- tion. We measure two quantities: whether a model is able make a valid year prediction at all (or re- frains, noting the year is "uncertain""); if it make a valid prediction, we calculate the prediction error for each book as the absolute difference between the predicted year and the true year (ý - v/). In order to assess any disparities in performance as a function of a model's knowledge about a book, we focus on the differential between the books in the

---

Model
missing, top 10%
missing. bot. 10%
p
MAE, top 10%
MAE, bot. 10%
ChatGPT 0.04
0.09
-0.39
3.3 [0.1-1233]
29.8 (20.2-41.2)
GPT-4
0.00
0.00
-0.37
0.6 [0.1-1.5]
14.5 [9.5-20.8]
Table Date of first publication performance, with 95% bootstrap confidence intervals around MAE. Spearman p across all data and all differences between top and bottom 10% within model are significant (p < 0.05).
As above, our core question asks: do we see a difference in performance between books that GPT-4 has memorized and those that it has not? To assess this, we again identify the top and bot- tom decile of books by their GPT-4 name cloze accuracy. We manually annotate the amount of time passing in 250-word passage from that book using the criteria outlined by Underwood (2018), including the examples provided GPT-4 Un- derwood (2023), blinding annotators to the identity of the book and which experimental condition it be- longs to. We then pass those same passages through GPT-4 using the prompt in figure 4, adapted from Underwood (2023). We calculate accuracy as the Spearman p between the predicted times and true times and calculate a 95% confidence interval over that measure using the bootstrap.
The virtues open models. This archaeology is required only because ChatGPT and GPT-4 are closed systems, and the underlying data on which they have been trained is unknown. While our work sheds light on the memorization behavior of these models, the true training data remains fundamen- tally unknowable outside of OpenAL. As others have pointed out with respect to these systems in particular, there are deep ethical and reproducibil- ity concerns about the use of such closed mod- els for scholarly research (Spirling, 2023). A pre- ferred alternative is to embrace the development of more open, transparent models, including LLaMA (Touvron et al., 2023), OPT (Zhang et al., 2022) and BLOOM (Scao al., 2022). While open- models will not alleviate the disparities in perfor- mance we see by virtue of their openness (LLaMA is trained on books in the Pile, and all use infor- mation from the general web, which each allocate attention to popular works), they address the issue of data contamination since the works in the train- ing data will be known; works in any evaluation data would then be able to be queried for member- ship (Marone and Van Durme, 2023).
We find a large but not statistically significant difference between T0P (p = 0.50 (.72) and B0 (p = 0.27 -0.02-00.5)) suggesting that GPT-4 may perform better on this task for books it has memorized, but not conclusive evi- dence that this is so. Expanding this same process to the top and bottom quintile sees no effect (TOP20 (p 0.47 (0.29-0.66) 0.63]; BOT (p 0.46 [0.25- 0.64)); if an effect exists, is only among the most memorized books. This suggests caution; more research is required to assess these disparities.
Popular texts are likely not good barometers of model performance. A core concern about closed models with unknown training data is test contamination: data in an evaluation benchmark (providing an assessment of measurement valid- ity) may be present in the training data, leading assessment to be overconfident in model's abili- ties. Our work here has shown that OpenAl models know about books in proportion their popularity on the web, and that their performance on down- stream tasks is tied to that popularity. When bench- marking the performance of these models on a new downstream task, it is risky to draw expectations of generalization from its performance on Alice in Wonderland, Harry Potter, Pride and Prejudice, and on-it simply knows much more about these works than the long tail of literature (both in the public domain and in copyright). In this light, we hope this work can serve as the starting point for future work on the generalization ability of LLM
8 Discussion
We carry out this work in anticipation of the ap- peal of ChatGPT and GPT-4 for both posing and answering questions in cultural analytics. We find that these models have memorized books, both in the public domain and in copyright, and the ca- pacity for memorization is tied to a book's 's overall popularity on the web. This differential in mem- orization leads to differential in performance for downstream tasks, with better performance on pop- ular books than on those not seen on the web. As we consider the use of these models for questions in cultural analytics, we see a number of issucs worth considering.

---

in the context of cultural analytics.
carries out probabilistic inference measure the familiarity of these models with a set of books, but the question of whether they truly exist within the training data of these models is not answerable. Mere familiarity, however, is enough to contami- nate test data.
Whose narratives? At the core of questions sur- rounding the data used train large language mod- els is one of representation: whose language, and whose lived experiences mediated through that lan- guage, is captured in their knowledge of the world? While previous work has investigated this in terms of what varieties of English pass GPT-inspired content filters (Gururangan et al., 2022), we can ask the same question about the narrative experi- ences present in books: whose narratives inform the implicit knowledge of these models? Our work here has not only confirmed that ChatGPT and GPT-4 have memorized popular works, but also what kinds of narratives count as "popular" particular, works in the public domain published before 1928, along with contemporary science fic- tion and fantasy. Our work here does not inves- tigate how this potential source of bias might be- come realized- e.g. how generated narratives are pushed to resemble text it has been trained on (Lucy and Bamman, 2021); I); how the values of the memo- rized texts are encoded to influence other behaviors, and so any discussion about it can only be speculative, but future work in this space can take the disparities we find in memorization starting point for this important work.
Our study has some limitations that future work could address. First, we focus on case study of English, but the analytical method we employ is transferable to other languages (given digitized books written in those languages). We leave to future work to quantify the extent of memorization in non-English literary content by large language models.
Second, the name cloze task is one operational- ization to assess memorization of large language models. As shown by previous research (e.g.. Lee et al., 2022), memorization can be much more acute with longer textual spans during training copied verbatim. We leave quantifying the full extent of memorization and data contamination of opaque models such as ChatGPT and GPT-4 for future work.
Ethical considerations
Our study probes the degree to which ChatGPT and GPT-4 have memorized books, and what impact that memorization has on downstream research in cultural analytics. This work uses the OpenAl API to perform the name cloze task described above, and at no point do we access, or attempt to access, the true training data behind these models, or any underlying components of the systems.
9 Conclusion
As research in cultural analytics is poised to be transformed by the affordances of large language models, we carry out this work to uncover the works of fiction that two popular models have mem- orized, in order to assess the risks of using closed models for research. In our finding that memoriza- tion affects performance on downstream tasks in disparate ways, closed data raises uncertainty about the conditions when a model will perform well and when it will fail; while our work tying memoriza- tion to popularity on the web offers rough guide assess the likelihood of memorization, it does not solve that fundamental challenge. Only open models with known training sources would do so. Code and data to support this work can be found at https: //gs:///iii thub. wommbaaman-ggruu gpt4-books.
Acknowledgments
The research reported in this article was supported by funding from the National Science Foundation (IS-1142591) and the National Endowment for the Humanities (HAA-271654-20),
References
David Bamman, Michelle Carney, Jon Gillick, Cody Hennesy, and Vijitha Sridhar. 2017. Estimating the date of first publication in large-scale digital library. In 2017 ACM/IEEE Joint Conference on Digital Li- braries (JCDL), pages 1-10. IEEE
Limitations
David M. Berry. 201 The computational turn: Think- ing about the digital humanities. Culture Machine. 12.
The data behind ChatGPT and GPT-4 is fundamen- tally unknowable outside of OpenAL. Our work

---

Stella Biderman, Hailey Schoelkopf. Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashamth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint 234113330.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing sta- tistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1383-1392, Melbourne, Aus- tralia. Association for Computational Linguistics.
Terra Blevins and Luke Zettlemoyer. 2022. Language contamination helps explains the cross-Tingual capa- bilities of English pretrained models. In Proceedings of the 2022 Conference on Empirical Methods in Nat-
M Elam. 2023. Poetry Will Not Optimize, or What Is Literature to AI? American literature; a journal of literary history, criticism and bibliography.
urul Language Processing, pages 3563-3574, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguisties.
Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021. Memorization vs. generalization Quantify- ing data leakage in NLP performance evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 1325-1335, Online. Association for Computational Linguistics.
Pierre Bourdieu. 1987. Distinction: social critique of the judgement of taste. Harvard university press.
Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag. Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. 2023. Ex- tracting training data from diffusion models. arXiv preprint arXiv:2301.1 13188.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, and Yoav Gold- berg. 2022. Measuring causal effects of data statis- tics on language model's 'factual' predictions. arXiv preprint arXiv: 22251
Nicholas Carlini, Daphne Ippolito. Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural lan- guage models. arXiv preprint arXiv:2202.0766
Katherine Elkins and Jon Chun. 2020. Can GPT-3 pass writer's Turing test? Journal of cultural analyties. 5(2).
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, UI- far Erlingsson, et al. 2021. Extracting training data from large language models. In USENIX Security Symposium, volume
Elizabeth F. Evans and Matthew Wilkens, 2018. Nation, ethnicity, and the geography of British fiction, 1880- 1940. Cultural Analytics.
Kathleen Fitzpatrick. 2012. The humanities, done digi- tally. Debates in the digital humanities, pages 12-15.
Mariona Coll Ardanuy, Federico Nanni, Kaspar Beelen, Kasra Hosseini, Ruth Ahnert, Jon Lawrence, Kather- ine McDonough. Giorgia Tolfo, Daniel CS Wilson, and Barbara McGillivray. 2020. Living machines: study of atypical animacy. In Proceedings of the 28th International Conference on Computational Linguis- tics, pages 4544, Barcelona, Spain (Online). International Committee on Computational Linguis- tics.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing. Travis Hoppe. Charles Foster, Jason Phang. Ho- race He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint 3XXx:2201100002
Timnit Gebru, Jamie Morgenstern, Briana Vec- chione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. 2021. Datashects for datasets. Communications of the ACM, 64(12):86- 92.
Jacob Devlin, Ming-Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. 2022. Demystifying prompts in language models via perplexity estimation.
Lauren M E Goodlad and Wai Chee Dimock. 2021. AI and the Human. PMLA, 136(2):317-319
John Guillory. 1993. Cultural capital the problem of lit- erary canon formation. University of Chicago Press, Chicago.
Jesse Dodge, Maarten Sap. Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colos- sal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286-1305, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Suchin Gururangan, Dallas Card, Sarah K Drier, Emily K Gade, Leroy Wang. Zeyu Wang, Luke Zettlemoyer, and Noah A Smith. 2022. Whose lan- guage counts as high quality? Measuring language ideologies in text data selection. arXiv preprint arXiv:2201.1044

---

Gary Gutting. 1989. Michel Foucault's Archaeology of Scientific Reason: Science and the History of Reason. Cambridge University Press.
models. In International Conference on Learning Representations.
Abhimanu Kumar, Matthew Lease, and Jason Baldridge. 201 Supervised language modeling for temporal resolution of texts. In Proceedings of the 20th ACM international conference on Information and knowl- edge management, pages 2069-2072.
Sil Hamilton and Andrew Piper. 2022. The COVID that wasn't: Counterfactual journalism using GPT. In Proceedings of the 6th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 83-93, Gyeongju, Republic of Korea. International Conference Computational Linguistics.
Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Kallison-Burchhe and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8424-8444 8445, Dublin, Ireland. Association for Computational Linguisties.
N. Katherine Hayles. 2012. How We Think: Transform- ing Power and Digital Technologies. In David M Berry, editor, Understanding Digital Humanities, pages 42 66. Palgrave Macmillan UK, London.
Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Perey Liang. 2023, Foundation models and fair use. arXiv preprint
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel, 2021. Question and answer test-train overlap in open- domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 1000-1008, Online. Association for Computational Linguistics.
155755.
Leah Henrickson and Albert Meroñoo-Peñulla. 2022. The Hermeneutics of Computer-Gennrated Texts. Configurations, 30(2):115-139
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The Goldilocks principle: Reading children's 's books with explicit memory representa- tions. ICIR.
Li Lucy and David Bamman. 2021. Gender and rep- resentation bias in GPT-3 generated stories. In Pro- ceedings of the Third Workshop on Narrative Un- derstanding, pages 48-55, Virtual. Association for Computational Linguistics.
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022. Are large pre-trained language models leak- ing your personal information? arXiv preprint arXiv:2205.12628
Inbal Magar and Roy Schwartz. 2022. Data contamina- tion: From memorization exploitation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Pa- pers), pages 157-165, Dublin, Ireland. Association for Computational Linguistics.
Daphne Ippolito. Florian Tramèr. Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christo- pher A Choquette-Choe and Nicholas Carlini. 2022. Preventing verbatim memorization in language mod- els gives a false sense of privacy. arXiv preprint arXiv:2210.173 17546.
Mare Marone and Benjamin Van Durme. 2023. Data portraits: Recording foundation model training data.
Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Berg- Kirk-airka 2022. An empirical analysis of memorization in fine- tuned autoregressive language models. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1816-1826, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
FM Jong, Henning Rode, and Djoerd Hiemstra. 2005. Temporal language models for the disclosure of his- torical text. Proceedings of the XVIth International Conference of the Association for History and Com- puting (AHC 2005).
Nikhil Kandpal, Haikang Deng. Adam Roberts, Eric Wallace, and Colin Raffel. 2022a. Large language models struggle to learn long-tail knowledge. arXiv preprint
Takeshi Onishi, Hai Wang. Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large- scale person-centered cloze dataset. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 2230-2235, Austin, Texas, Association for Computational Linguisties.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022b. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pages 10697-10707. PMLR.
Folgert Karsdorp and Lauren Fonteyn. 2019, Cultural entrenchment of folktales is encoded in language. Palgrave Communications, 5(1):25.
Andrew Piper, Richard Jean So, and David Bamman. 2021. Narrative theory for computational narrative understanding. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing, pages 298-311, I, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language

---

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):4485-5555
Kushal Tirumala, Aram Markosyan, Luke Zeytlemoyer, and Armen Aghajanyan. 2022. Memorization with- out overfitting: Analyzing the training dynamics of large language models. Advances in Neural Informa- tion Processing Systems, 35:38274-38290
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint urXiv:2302.13971.
Stephen Ramsay and Geoffre Rockwell. 2012. De- veloping things:
Notes toward an epistemology of building in the digital humanities. Debates in the digital humanities, pages 75-84.
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 840-854, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguisties.
Ted Underwood. 2018. Why literary time is measured in minutes. ELH, 85(2).
Ted Underwood. 2023.
Using GPT-4 measure the passage of time in fiction.
http://dddeeoooooooo///2///////1 ept-4-to-measure the tew-ttteeepppaeee passage- offssage-o--ime--ee -i--ee------ioo.
Nicolas Ruth, Andreas Niekler, and Manuel Burghardt. 2022. Peeking Inside the DH Toolbox Detection and Classification of Software Tools DH Publica- tions. Proceedings of the Computational Humanities Research Conference 2022, 1613:0073.
Melanie Walsh and Maria Antoniak. 2021. The Goodreads "Classics": A computational study of readers, Amazon, and crowdsoureed amateur crit- icism. Journal of Cultural Analyties, 6(2).
Michael Yoder, Sopan Khosla, Qinlan Shen, Aakanksha Naik, Huiming Jin, Hariharan Muralidharan, and Car- olyn Rosé. 2021. FanfictionNI.P. A text processing pipeline for fanfiction. In Proceedings of the Third Workshop on Narrative Understanding, pages 13-23, Virtual. Association for Computational Linguistics.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ili,. Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100.
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Car- lini. 2021. Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938.
Henning Schmidgen, Bernhard Dotzler, and Benno Stein. 2023. From the archive the computer: Michel Foucault and the digital humanities. Jour- nal of cultural analytics, 7(4).
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint 220.::2205.001 2206.01068.
Reza Shokri, Marco Stronati, Congzheng Song, and Vi- taly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE sym- posium on security and privacy (SP), pages 3-18. IEEE.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE in- ternational conference on computer vision, pages 19-27.
Arthur Spirling. 2023. Why open-souree generative Al models are an ethical way forward for science. Nature, 616(7957):413-4 413. 413.
Dominik Stammbach, Maria Antoniak, and Elliott Ash. 2022. Heroes, villains, and victims, and GPT-3: Au- tomated extraction of character roles without train- ing data. In Proceedings of the 4th Workshop of Narrative Understanding (WNU2022), pages 47-56, Seattle, United States. Association for Computational Linguistics.
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can large lan- guage models transform computational social sci- ence? arXiv submission 4840038.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannanch Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275-9293, Online, Association for Computa- tional Linguistics.
Appendix
Author contributions
Kent Chang: performed extrinsic analysis; an- notated narrative time duration data; wrote paper.

---

Mackenzie Cramer: performed human name cloze experiment: annotated narrative time du- ration data; researched self pufl-uublisiing history of Fifty Shades of Grey.
Sandeep Soni: performed the statistical anal- ysis of predicted names; annotated narrative time duration data; wrote paper.
David Bamman: conducted experiments, an- notated narrative time duration data; wrote paper.

---

What was the year in which the following passage was written? You must make a prediction of year, even if you are uncertain.
Example:
He felt his face. 'I miss sneering something love. How used to love gather in the checker-tiled kitchen in front of the old boxy cathode-ray Sony whose reception was sensitive to airplanes and sneer at the commercial vapidity of broadcast stuff."
Input:
Output: 1996</ya96/
The multiplicity of its appeals the perpetual surprise of its contrasts and resemblances! All these tricks and turns of the show were upon him with spring as he descended the Casino steps and paused the pavement at its doors. He had not been abroad for seven years and what changes the renewed contact produced! If the central depths pin-point surface remained the same. And this the very place bring the completeness of the renewal. The sublimities, the perpetuities, might have left him he but this tent pitched for day's 'sy' revelry spread roof of oblivion between himself and his fixed sky.
Input:
untouched, hardly
Output: <ycar> 105e/yer0s
Input: He did not speak to Jack G.H., nor they to him. He a cigarette with shaking fingers and watched the spinning billiard balls roll and gleam and clack over the green stretch of doth, dropping into holes after bounding cushions. He felt impelled to say something to ease the swelling and, with twin eddies of blue smoke jutting from black nostrils, shouted hoarsely. "Jack. betcha two bits you 't make it!" Jack did not answer; late now." Bigger said. "You wouldn't bet, so you lost." He spoke without looking. His entire body hungered something exciting and violent to relieve the tautness, It was now ten minutes three and Gus had not come. Gus stayed away much longer, it would too late. And Gus knew that. If they were going to do anything, it certainly ought to be done before folks started coming into the streets to buy their fond for supper, and while the cop was down the other end of block.
fro from the rubber a spittoon
chest. Hurriedly, he flicked his cigarette
ball shot straight across table and vanished into a side pocket. "You would've ve lost." " Jack said. "Too
keen sensation,
Output:
Figure 3: Sample prompt for year of first publication prediction.

---

GPT-4
ChatGPT
BERT
Date
Author
Title
0.76
0.43
0.00
1997
J.K. Rowling
Harry Potter and the Sorcerer's Stone
0.57
0.30
0.00
1949
George Orwell
1984
0.51
0.20
0.01
1954
J.R.R. Tolkien
The Fellowship the Ring
0.49
0.16
0.13
2012
E.L. James
Fifty Shades of Grey The Hunger Games
0.48
0.14
0.00
2008
Suzanne Collins
0.43
0.27
0.00
1954
William Golding
Lord of the Flies
0.43
0.17
0.00
1979
Douglas Adams
The Hitchhiker's Guide the Galaxy
0.30
0.16
0.00
1959
Chinua Achebe
Things Fall Apart
0.28
0.12
0.00
1977
J. R. Tolkien and Christopher Tolkien
The Sibnarillion
0.27
0.13
0.00
1953
Ray Bradhury
Fahrenheit 451
0.27
0.13
0.00
1996
George R.R. Martin
Game Thrones
0.26
0.05
0.01
2003
Brown
The Vinci Code
0.26
0.08
0.00
1965
Frank Herbert
Dane
0.25
0,20
0.01
1937
Zora Neale Hurston
Their Eyes Were Watching God
0.14
0.00
1961
Harper
To a Mockinghird
0,03
0.03
1953
Ian Fleming
Casino Royale
0.13
0.00
1984
William Gibson
Neuromancer
0.20
0.10
0.00
1985
Orson Scott Card
Ender's s Game
0.19
0.12
0.00
1932
Aldous Huxley
Brave New World
0.18
0.07
0.00
1937
Margaret Mitchell
Gone with the Wind
D.17
0.05
0.00
1968
Philip K. Dick
Do Andrvids Dream of Electric Sheep?
0.16
0.06
0.05
2009
Dan Brown
The Lost Symbol
0.15
0.04
0.04
2013
Dan Brown
Inferno
0.15
0.08
0.01
2014
Veronica Roth
Divergent
0.15
0.04
0.00
1940
John Steinheck
The Grapes of Wrath
0.13
0.05
0.00
1983
James Kahn
Return of the Jedi
0.13
0.02
0.01
1928
D. H. Lawrence
Lady Chatterley 's Lover Roots
0.13
0.03
0.00
1977
Alex Haley
0.11
0.11
0.00
1961
Irving Stone
The Agony and the Eastasy
0.11
0.01
0.03
1957
Ian Fleming
From Russia with Love A Wrinkle in Time
0.11
0.04
0.00
1962
Madeleine L'Engle
0.11
0.04
0.01
1939
Marjorie Kinnan Rawlings
The Yearling
0.10
0.05
0.00
1975
E. L. Doctorow
Ragtime
0.10
0.05
0.00
1929
Dashiell Hammett
The Maltese Falcon
0.10
0.08
0.07
1991
Diana Gabaldon
Outlander
0.10
0.02
0.00
1989
Kazuo Ishiguro
The Remains of the Day
0.10
0.01
0.00
1983
Alice Walker
The Color Purple
0.09
0.02
0.00
1934
Dorothy I.. Sayers
The Nine Tailors
0.09
0.03
0.00
1985
Margaret Atwood
The Handmaid's Tale Beloved
0.09
0.04
0.00
1988
Toni Morrison
0.08
0.07
0.00
1982
Joe Nazel
Every Goodbye Ain't dn't Gone
0.08
0.06
0.01
1984
Helen Hooven Santmyer
An. Ladies of the Club
0.08
0.02
0.00
2006
Max Brooks
World War Z.
0.08
0.02
0.00
1993
Irvine Welsh
Trainspotting the King Men
0.08
0.03
0.00
1947
Rohert Penn Warren
0.02
0.00
1952
Ralph Ellison
Invisible Man
0.07
0.01
0.00
1951
Isaac Asimov
Foundation Interview with the Vampire The Shining Bridget Jones's 's Diary
0.00
1976
Anne Rice
0.04
0.02
1977
Stephen King
0.01
0.00
1996
Helen Fielding
Table
Top 50 books in copyright (published after 1928) by GPT-4 name cloze accuracy.

---

Read the following passage of fiction. Then do five things.
I: Briefly summarize passage,
2: Reason step by step to decide how much time described in the passage. the passage doesn't include any explicit reference to time. you can guess how much time the events described would have taken. Even description can imply the passage of time by mentioning the carlier history of people or buildings. But characters' references to the past or future in spoken dialogue should not count as time that passed in the scene, Report the time using units of years, weeks, days, hours, or minutes. Do not say zero or N/A,
3: If you described a range of possible times in step 2 take the midpoint of the range. Then multiply to convert the units into minutes.
4: Report only the number of minutes elapsed, which should match the number in step 3. Do not reply N/A.
5: Given the amount of speculation required in step 2, describe your certainty about the estimate-cither high, moderate, or low.
Input:
TWENTY-FIV It was raining again the next morning, slanting gray rain like swung curtain of crystal beads. I got feeling sluggish and tired and stood looking out of the windows, with dark harsh taste of Sternwoods still in mouth. I was empty of life as scarecrow's 's pockets. I went out to the kitchenette and drank two cups of black coffee. You can have hangover from other things than alcohol. I had one from women. Women made me sick. I shaved and showered and dressed and got my raincoat out and went downstairs and looked out of the front door. Across the street, a hundred feet up. gray Plymouth sedan was parked. It was the same one that had tried to trail me around the day before, the sume one that I had asked Eddic Mars about. There might he cop in it, if cop had that much time on his hands and wanted to waste it following me around. Or it might he smoothic in the detective business trying to get noseful of somebody else's s case in order to chisel way into it. Or it might be the Bishop of Bermuda disapproving of my night life.
Output:
I: A detective wakes up 'the next morning,' looks window for undefined time, drinks (and presumably needs make) two cups of coffee, then shaves and showers and gets dressed before stepping out his front door and seeing a car.
2: Making coffee, showering. and getting dressed take least hour. There's some ambiguity about whether count the implicit reference to yesterday (since this "the next morming) as time elapsed in the passage, but let's say no, since yesterday is not actually described. So, hour 90 minutes.
3: 1.25 hours have elapsed. Multiplying by 60 minutes an hour that's minutes.
4: 75 minutes.
5: Low confidence, because ambiguity about reference the previous day.
Input:
CHAPTER "TOM!" No answer. "TOM!" No answer. "What's gone with that boy. wonder? You TOM!" No answer. The old lady pulled her spectacles down and looked over them about the room; then she put them up and looked out under them. She seldom or never looked through them for so small a thing as a boy; they were her state pair, the pride of her heart, and were built for "style," not service-she could have seen through a pair of stove-lids just as well. She looked perplexed for a moment, and then said, not fiercely. but still loud enough for the furniture to hear: "Well, I lay if get hold of you I'll-" She did not finish, for by this time she was bending down and punching under the hed with the broom, and so she needed breath to punctuate the punches with. She resurrected nothing but the cat. "I never did see the beat of that boy!" She went to the open door and stood in it and looked out among the tomato vines and "jimpson" weeds that constituted the garden. No Tom. So she lifted up her voice at an angle calculated for distance and shouted: "Y-o-u-u TOM!" There was a slight noise behind her and she turned just in time to seize a small bay by the slack of his roundabout and arrest his flight.
Output:
I: An old lady calls for boy named Tom, checks for him under the bed, goes to the open door and calls for him then finally catches him."
2: The lady's 's actions are described minutely and seem rapid: they probably took two to four minutes. The lady also alludes to
Tom's past bad behavior. but references to the past in dialogue should not count as time passing in the scene.
3: Three minutes have elapsed.
4: 3 minutes.
5: High confidence.
Input: [text]
Output:
Figure 4: Sample prompt for narrative time prediction, from Underwood (2023).


Reviews:
[{"review_id": "94nUn6OH2G", "reviewer": "Reviewer_2wkf", "summary": "", "questions": "1. is there any way to evaluate the perplexity or uniqueness of the segment itself, and its impact? “Stately, plump [MASK] came from the stairhead, bearing a bowl of lather on which a mirror and a razor lay crossed.” would itself quite an unusual construction, even without the mask.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and kind words! \n\n> Is there any way to evaluate the perplexity or uniqueness of the segment itself, and its impact?\n\nThe short answer is no; calculating the perplexity of the prompt for the model requires the model to share the assigned probabilities to words or sequences. Unfortunately, ChatGPT and GPT-4 models currently don’t share them through their API. \n\nWe agree this perplexity angle would be interesting. The perplexity of the prompt, as some research (e.g., Gonen et al. 2022) [1] has found, is linked to the model’s ability to generalize. The impact of having seen “unusual constructions” on perplexity is worth further investigation as well. As Liu et al. 2023 [2] noted, OpenAI has recently modified their model behavior to discourage reproducing copyrighted material, and it’s unclear whether this would figure in the calculation of perplexity for unique sentences without the mask. \n\nWe will note this in the camera-ready version of our paper. \n\n[1] Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, Luke Zettlemoyer. “Demystifying Prompts in Language Models via Perplexity Estimation.” https://arxiv.org/abs/2212.04037\n\n[2] Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, Hang Li, “Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment.” https://arxiv.org/abs/2308.05374", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main topics of this work are timely, and hit on both fundamental questions of analysis (as seen in the discussion of cited works about the criticism of 'canon') with black-box tools, and the technical question of memorisation in LLMs. The paper has relevance beyond the core application area, and contributes significantly to the discussion of the state of the art. The methodology is sound, applies its statistical approach appropriately, and draws conclusions from the evidence.", "reasons_to_reject": "I cannot think of any substantive reason to reject. ", "questions_to_authors": "1. is there any way to evaluate the perplexity or uniqueness of the segment itself, and its impact? “Stately, plump [MASK] came from the stairhead, bearing a bowl of lather on which a mirror and a razor lay crossed.” would itself quite an unusual construction, even without the mask.\n", "paper_topic_and_main_contributions": "This paper addresses the use of GPT-3.5 and GPT-4 and their potential for use in cultural analytics. The authors identify works memorised by the models by asking for predictions on un-guessable tokens, such as names. These estimates are then used to quantify the degree to which the openai models leverage this memoerisation in answering other tasks. The themes of the paper are copyright, memorisation, opaque models, and cultural analytics. The major contribution is the finding that there is significant impact from more-memorised works. The minor contribution is the use of the task as a determinant. I think this paper lies in the space between computationally-aided linguistic analysis and NLP engineering experiment.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EkDeCJ7s7Y", "reviewer": "Reviewer_gM9J", "summary": "", "questions": "Q_A: LLaMA is not open source. Also, LLaMA 2 is not open source. Only the model weights are open, but the whole corpus used for the training and the code used for training are unavailable. Clarify this aspect.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback, and we appreciate that you find the analysis in this paper clear and helpful. \n\n> The authors provide an accurate methodology, but there is the risk that they already know what they find as result and design experiments to confirm what they want.\n\nWith regard to the risk of confirmation bias in the design of our experiments, we disagree for two reasons. First, we are not aware of any past research that attempted to quantify memorization in ChatGPT and GPT-4, especially in the context of books, so we could not have known our main finding *a priori*. Second, ChatGPT and GPT-4 have not released the details about their training data or protocols, so we could not have known the extent of copyrighted material that these models have been trained on before investigating ourselves.  Our work is designed to flesh out the contours of this memorization so that we, and others, are able to account for it in applying these models to cultural analytics; our work is descriptive and does not hinge on any specific finding of memorization.\n\n> A task that involves fine-tuning should be helpful.\n\nYes, although we’d like to point out that §7 is first written with practitioners of cultural analytics in mind: What must we remember if we engage GPT-3.5/4 for inference tasks to bring to bear on literature and culture? The disparity in memorization leads to disparity in performance, which is important to note before we can draw any meaningful conclusion from the GPT-assisted analysis. The effects of fine-tuning on those tasks, while we agree could help us study memorization, are not what we look to investigate in this section. We’ll also note that fine-tuning with GPT-3.5/4 models was not possible at the time of writing. Thank you, we’ll clarify in the camera ready.\n\n> The paper fails to measure the LLMs' generalization ability in the case of memorization. \n\nIt’s correct that we haven’t measured the generalization ability of these models accounting potentially for memorization, and we would agree that it warrants study, but we could only do that if we knew the boundaries between seen and unseen data for these models. In this light, as we allude to in the discussion, we see this paper as the starting point for this important future work. We’ll make this point explicit in the final version of the paper.\n\n> LLaMA is not open source. Also, LLaMA 2 is not open source. Only the model weights are open, but the whole corpus used for the training and the code used for training are unavailable. \n\nThank you for calling our attention to our description of LLaMA; we will be sure to clarify in the camera-ready. \n\n> Please, provide more details about the human evaluation reported in line 207\n\nYes, we will be including more details on the human evaluation of our name cloze task in the camera-ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Interesting and helpful analysis.\n2) Reproducibility: code and data will be available. It is essential to release data generated by ChatGPT/GPT-4 since they can change.\n3) The paper is clear and well-written.", "reasons_to_reject": "1) Risk of speculations.\nThe authors provide an accurate methodology, but there is the risk that they already know what they find as result and design experiments to confirm what they want. \n2) A task that involves fine-tuning should be helpful.\nThe authors design the prediction of publication year as a zero-shot task. However, they can use a portion of the data for fine-tuning the system and then evaluate the fine-tuned model on the other part. This can help to measure if memorization can also affect fine-tuning.\n3) Limitation.\nOne of the limitations proposed by the author is the central issue of the paper. The paper fails to measure the LLMs' generalization ability in the case of memorization. We know that memorization has several drawbacks (copyright infringement, train-test leakage) and the analysis reported in the paper goes in these directions. However, the main question is: Can LLMs generalize when writing narrative texts? Also, humans improve their writing ability if they read books written by other people. The difference is that we do not memorize the whole passages from books.", "questions_to_authors": "Q_A: LLaMA is not open source. Also, LLaMA 2 is not open source. Only the model weights are open, but the whole corpus used for the training and the code used for training are unavailable. Clarify this aspect.", "paper_topic_and_main_contributions": "The paper describes a data archaeology analysis investigating how closed large language models (ChatGPT/GPT-4) memorize books. This analysis is essential to understand which data are used to train these models. In particular, to investigate if data under copyright are used and to avoid using test data already present in the training (train-test leakage).", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) Please, provide more details about the human evaluation reported in line 207. How many annotators? What is the experience/expertise of the annotators? How many annotations?"}, "extra_text": {}}, {"review_id": "M05IpH5obk", "reviewer": "Reviewer_swER", "summary": "", "questions": "A) The name cloze evaluation, while creative, is an indirect measure of memorization. What do you think of direct methods like targeted probing?\nB) You focus exclusively on fiction books in English. To what extent do you think findings could generalize to other text genres and languages?\nC) The correlations shown between web prevalence and memorization are interesting but do not prove causation. Can you speculate on other factors that could also drive the degree of memorization seen?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your recognition of the “significant implications for bias and validity in downstream tasks”. Our answers to your questions below:\n\n> A) The name cloze evaluation, while creative, is an indirect measure of memorization. What do you think of direct methods like targeted probing?\n\nYes, targeted probing would be a more direct measure of memorization, but it would be most effective when we have hidden representations of the model for the inputs. To the best of our knowledge, the current OpenAI API does not give us access to the hidden representations.\n \n> B) You focus exclusively on fiction books in English. To what extent do you think findings could generalize to other text genres and languages? \n\nWe believe that this is an important empirical question for future work. Our expectation is that similar training processes would yield similar facts of memorization (in which the more times a text is seen, the greater the chance it has been memorized); but as the size of data in different languages and genres vary in orders of magnitude with respect to English fiction, an important open question is many observations of a text are necessary to influence memorization (and downstream work in cultural analytics).\n\n> C) The correlations shown between web prevalence and memorization are interesting but do not prove causation. Can you speculate on other factors that could also drive the degree of memorization seen?\n\nThis is a great point. It’s correct that we don’t specify a causal arrow in which memorization is directly explained by web prevalence; if we were to speculate, it’s likely that a hidden confounder explains them both: popularity.  Moby Dick, for example, is widely read; a.) many copies of it appear in university libraries, b.) multiple copies of it may appear in large-scale training datasets such as books3, c.) quotations of it may appear in unindexed academic journals, and d.) as we see, it appears in multiple places on the public web (both in snippets and in the form of full text).  All four of these factors are likely correlated with each other and commonly explained by the overall popularity of the text itself (concretely, by an unobservable quantity such as the number of times it has been read worldwide).  The correlation we see between web prevalence and memorization may be an indirect reflection of this latent causal graph; we will discuss this relationship (and the lack of direct causation between web prevalence and memorization) further in this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-  Understanding memorization in large language models - that has significant implications for bias and validity in downstream tasks.\n- The analysis is rigorous, evaluating hundreds of books across multiple models. \n- The paper is well-written and presents the analysis clearly. ", "reasons_to_reject": "- The downstream tasks evaluating impact on performance are limited. More analysis of different tasks could strengthen the conclusions.\n- The scale of the experiments, while a strength, required significant resources. This raises reproducibility concerns.", "questions_to_authors": "A) The name cloze evaluation, while creative, is an indirect measure of memorization. What do you think of direct methods like targeted probing?\nB) You focus exclusively on fiction books in English. To what extent do you think findings could generalize to other text genres and languages?\nC) The correlations shown between web prevalence and memorization are interesting but do not prove causation. Can you speculate on other factors that could also drive the degree of memorization seen?", "paper_topic_and_main_contributions": "This paper presents an analysis of the books that are memorized in the training data of ChatGPT, GPT-4 and BERT large language models. \nThe main contribution of this work resides in its results, where these models have extensively memorized many books, especially popular sci-fi/fantasy novels but also including copyrighted bestsellers.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agree that the paper draws attention to a timely and important question: which (potentially copyrighted) books appear in GPT-3.5/4's training data, and to what extend has GPT-3.5/4 memorized these works? Understanding the degree of memorization has important implications for researchers in the humanities who wish to use these models. The reviews also see broader applicability of this work beyond the humanities, contributing to a deeper understanding of the models. The reviewers agree that the experiments are well designed and implemented.

The reviewers raise several concerns, which the authors convincingly address in their rebuttal. Specifically, they explain why the name cloze  task is appropriate, why measuring the perplexity of the prompt is infeasible, and why fine tuning is less relevant in this situation.
Decision: Accept-Main