SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Marc Marone Benjamin Van Durme
{mmarone1,vandurme}@jhu.edu
Foundation models are trained on increasingly immense and opaque datasets. Even while these models are now key in AI system building, it can be difficult to answer the straightforward question: has the model already encountered a given example during training? We therefore propose a widespread adoption of Data Portraits: artifacts that record training data and allow for downstream inspection. First we outline the properties of such an artifact and discuss how existing solutions can be used to increase transparency. We then propose and implement a solution based on data sketching, stressing fast and space efficient querying. Using our tools, we document a popular language modeling corpus (The Pile) and a recently released code modeling dataset (The Stack). We show that our solution enables answering questions about test set leakage and model plagiarism. Our tool is lightweight and fast, costing only  $3\%$  of the dataset size in overhead. We release a live interface of our tools at dataportraits.org and call on dataset and model creators to release Data Portraits as a complement to current documentation practices.
Modern AI is driven by large models trained on large datasets, displaying emergent capabilities as they scale [5]. Despite the foundational nature of these models, it is a natural question to ask of some example, "Was this seen during training?" Understanding the contents of large, opaque, web-scraped datasets is important to assessing downstream model behavior. Web datasets can contain leaked test sets, harmful text, or low quality information; these impact downstream models [12, 26, 15, 8]. Documentation artifacts for datasets and models have been proposed and adopted by the community [14, 27, among others]. However, less work has dealt with tooling to support large dataset inspection.
We suggest that existing practices around foundation models can be improved through the widespread adoption of Data Portraits: artifacts that record training data and allow for efficient inspection. The critical property of a data portrait is membership inference: whether an example was part of a data collection. In discussing these tools we consider three perspectives: content creators (owners), scientists, and content consumers. Creators wish to know whether a dataset contains their content (e.g. copywritten code). Scientists may seek to understand model behavior by assessing test set leakage or knowledge memorization. Content consumers (including downstream applications) may want to know if a model is plagiarizing or citing existing resources.
Our contributions are two-fold. First we call for documentation artifacts based on membership inference, discussing the properties and tradeoffs around existing solutions. We then introduce our own implementation of such an artifact and use it to document a widely used large language model (LLM) corpus: the Pile [13]. We also document a recent code language modeling corpus: the Stack [22]. Our artifact uses data sketching (compressed or approximate views of data, [6]) to enable millisecond latency and minimal compute requirements, using only  $\sim 3\%$  of the original dataset size. We release our tools and a live demo at dataportraits.org.
Table 1: Properties of various membership testing tools. Our proposed implementation, the strided Bloom filter, is probabilistic because it uses hash-based matching, but avoids lossless redistribution of data.  $\sim$  indicates that a tool might support a property. See Table 2 and Section 3 for further discussion about performance.
Method	Avoids Redistribution	Local	Index	Fuzzy	Probabilistic
grep	X	✓	✓	X	X
Full Text Index	X	~	✓	~	X
Suffix Array	X	✓	✓	X	X
Strided Bloom Filter	✓	✓	X	X	✓
2.1 Documentation Artifacts
Much recent work has called for additional artifacts documenting datasets and models [2, 27, 14, among others]. Gebru et al. [14] argues that creators should release a Datasheet artifact documenting the "motivation, composition, collection process" for a dataset. Mitchell et al. [27] earlier proposed a related artifact called Model Cards for trained models. These suggestions have been taken up within the AI community: new models are released with these artifacts (e.g. Zhang et al. [37] provide a Datasheet and Model Card for their models intending to be an open source parallel to OpenAI's closed models) and many resources hosted by Huggingface now include documentation artifacts.
We view a Data Portrait as a complementary type of documentation artifact - one that answers the membership question. There already exist examples of what we would describe as a portrait. Dodge et al. [12] studied multiple aspects of the C4 corpus and released a searchable indexed version. Contemporaneously with this work, Piktus et al. [30] released a search tool for the ROOTS corpus [23]. These tools are increasingly important as web-text corpora continue to outscale the resources of many academic groups.
"Blom filters" are probabilistic data structures used to test whether an element is a member of a set. According to Wikipedia, "A Bloom filter is a space-efficient probabilistic data structure, conceived by Burton Howard Bloom in 1970, that is used to test whether an element is a member of a set. False positives are possible, but false negatives are not - in other words, a query returns either "possibly in set" or "definitely not in set". Elements can be added to the set, but not removed. The more elements that are added to the set, the larger the probability of false positives."
Substring Hashes
Hashes for each of these strings appears in the sketch. Click above to select a new span.
ficient probabilistic data stru
Burton Howard Bloom in 1970, th.
whether an element is a member
Figure 1: Output from ChatGPT [29] when asked to explain Bloom filters using text from Wikipedia. The highlighted string is the longest overlapping string with the Pile. Other overlapping spans are grey. See our demo: dataportraits.org
Datasheets Gebru et al. [14] acknowledge that the contents of a complete datasheet will vary depending on research circumstances.
... datasheets will necessarily vary depending on factors such as the domain or existing organizational infrastructure and workflows ... [such as] academic researchers publicly releasing datasets ... [or] product teams creating internal datasets for training proprietary models. - Gebru et al. 14
For all of these scenarios, even internal proprietary uses, it is beneficial to have a computationally verifiable form of documentation.
Model Cards The Model Cards form seeks to document training data with several sections, including the one duplicated below:
<https://twitter.com/mmitchell.ai/status/1548358023382347777
2https://c4-search.apps.allenai.org/
Training Data. May not be possible to provide in practice. When possible, this section should mirror Evaluation Data. If such detail is not possible, minimal allowable information should be provided here, such as details of the distribution over various factors in the training datasets. - Mitchell et al. 27
We strongly prefer that datasets be open and transparent. However in cases where this is impossible, certain implementations of Data Portraits can still allow for (approximate) membership testing on the dataset used to build a specific model. In Section 3.1, we describe how some implementations can be used without leaking or redistributing data.
2.2 Large Models and Web Data
Other related work analyzes properties of models trained on web-scale corpora. Carlini et al. [9] showed that GPT-2 memorizes and can leak sensitive or private information from its training corpus. Further work [10] examined memorization and showed that it increased with scale. The initial GPT-3 [8] construction suffered from a bug that meant certain testsets were not filtered from their very large training corpus (though they later carefully analyzed the effects of this bug). Conventions and legal precedents around large language models that may memorize and output portions of the training data are still being developed. Kocetkov et al. [22] gather a set of permissively licensed open source code repositories. Luccioni and Viviano [26] analyze a typical source of web data, the Commoncrawl, studying "content that can be generally seen as inappropriate for a language model to generate".
It is not always clear what was used for training existing models. Even though open source models such as OPT [37] provide documentation artifacts, their Datasheet simply notes that the training corpus is a filtered "union of the following datasets..." and they do not host a final accessible artifact.<sup>3</sup>
2.3 Data Structures
Data sketching, storing compressed or approximate views of large datasets, has long been used to enable efficient analysis of large data [6]. GPT-3 and other LLMs ([25, 23] among others) use the MinHash sketch of Broder [6] to deduplicate their training sets. However, they do not release this sketch for downstream use as a documentation tool. Other data sketches, such as Bloom filters [4] and related structures, have a long history of use as efficient storage for NLP tasks. Talbot and Osborne [34] use Bloom filters to construct language models while others used similar structures to count features [16, 36]. More recent work analyzes the use of a Bloom filter to prevent verbatim copying of common n-grams from training sets [19].
Recent work on large datasets has investigated deduplication and memorization analysis. Lee et al. [24] develop an optimized suffix array implementation that can scale to very large datasets. Other work has released traditional full text indexes for datasets they study [30, 12]. These structures can be used as a membership testing tool but are primarily meant for additional use cases and thus may have higher computational demands (see Table 2 and Section 3).
3 Data Portraits
We outline desirable properties of a dataset documentation tool built around membership inference, recognizing that some may induce tradeoffs.
1. Fast: An artifact should support low latency programmatic access.
2. Lightweight: An artifact should not be much more expensive than storing the original data.
3. Avoids Redistribution: Some data cannot be redistributed for legal or proprietary reasons. A tool should avoid propagating harmful or illegal content when possible.
4. Local: A local artifact requires no costly external services. A local tool also provides privacy when querying for PII (personally identifiable information), rather than transmitting PII to a third party.
5. Indexing: A tool should indicate the context in which a match is found.
<sup>3</sup>https://github.com/facebookresearch/metaseq/issues/20
6. Match Flexibility: Fuzzy matching can be useful for similarity searches. Exact matches might be more useful for cases involving PII or memorization.
Other Implementations Examining these, it is clear that there may be many solutions to creating membership inference tools on large datasets. Table 1 has an overview of properties related to data modeling (discussed below) while Table 2 compares space usage between other published tools.
The easiest version of a Data Portrait is to simply store the data on disk. grep or similar tools can be used to query the dataset. This is a typical approach for membership inference; Carlini et al. [9] mention they ask the GPT authors for grep results. Other implementations might include a full-text search using commercial software or a conventional database [30, 12]. Recent work on memorization and data duplication has explored efficient data-structures for deduplicating large text datasets [24].
While these techniques enable tools that we would consider viable Data Portraits, each has certain tradeoffs. Simply storing the corpus on disk and grep-ing is slow and only allows exact or regex matches. This also does not address the governance issue - some data cannot be redistributed. For example, there are restrictions on distributing tweets [35]. However, since grep iterates over the corpus, indexing comes naturally.
A full-text search engine typically relies on an efficient indexing and lookup method and can be very fast. However, this takes additional space or might depend on non-local services. It also does not address the redistribution issue. A full-text search (e.g. BM-25 in Lucene) can support fuzzy or exact match searches, though these may add additional overhead and efficiency concerns.
Structures such as a suffix array are suitable for fast, local use on a single machine but have very high space requirements. Lee et al. [24] use a suffix array to deduplicate C4 [33] but note that "this algorithm still requires that the dataset itself fits in memory" and suggest using a machine with  $>600\mathrm{GB}$  of RAM. These structures allow for indexing into the dataset but do not support fuzzy matching. A local suffix array redistributes data; if exposed as a service, enumerating and extending queries could extract full documents.
Given these tradeoffs, we use a probabilistic approach for membership inference: a data sketch. A sketch provides a compressed and approximate view of data [4, 6, 7]. We use hash-based matching, addressing the redistribution issue, but giving up the ability to index or perform fuzzy matching. Our solution is minimal in that it supports membership inference and nothing more - yielding a small and fast structure. Other tools might support additional features or index the original documents. See Table 2 for a size comparison with other tools, such as the ROOTS Search Tool BM25 index of Piktus et al. [30]
Table 2: Ratio of structure to dataset size for related tools that could be used for membership testing. Tool & Data Sizes in TB (10 $^{12}$  bytes).
Structure	Dataset	Tool	Data	Ratio
grep	(any)	-	-	1.0
RST (BM25)	ROOTS	2.78	1.54	1.80
Suffix Array	C4	1.65	0.86	1.92
Ours	Pile	0.03	0.89	0.03
on the ROOTS corpus [23]; and the Suffix Array of Lee et al. [24] on C4 [33]. Most tools are built for indexing, which causes the structure size to be at least as large as the data. Our minimal implementation uses only  $\sim 3\%$  the space of the original corpus with millisecond query latency.
3.1 Bloom Filter Sketching
We base our sketch implementation of a portrait on Bloom filters [4]. These are a well-known solution for space efficient approximate membership testing, using only a fixed number of bits to record elements of a set. For example, our sketch of the Pile uses only  $\sim 14$  bits per data element. Bloom filters are similar to hash tables, but they store only the hash of a data element rather than storing the element itself. This is accomplished by setting bits in an array indexed by hash functions. See Broder and Mitzenmacher [7] for further details and derivations of hash collision rates. Bloom filters can output false positives but never false negatives. The false positive rate can be tuned to optimize the final size of the structure.
1. Corpus:
2. Hash n-grams: (stride=4)
3. Query:
4. Query n-grams: (stride=1)
5. Matches:
Figure 2: The matching process for a Bloom filter sketch with n-gram width 4. The blue tokens abcdefghijklmn are a string of interest. Checking subsequences indicates several matches (green).
The key is: a] bcde fghi jklm
b c d e f g h i j k l m
3.2 Bloom Filter Construction
Since our use case is membership testing over subsequences of longer documents, we can modify the input to our Bloom filter. Instead of storing all n-grams we store only tiled (or strided) n-grams from a source corpus. When building our Bloom filter data sketch, we hash n-grams with stride  $n$  (i.e. non overlapping, see Figure 2 rows 1 and 2). This implicitly defines an offset or alignment where an n-gram starts (row 2). Each resulting strided n-gram is hashed and stored in the Bloom filter (row 2). At query-time, we extract n-grams of size  $n$  but stride 1 from the input query (Figure 2 rows 3 & 4). Each of these is checked against the Bloom filter and matches are recorded (row 4, green).
This reduces space requirements but carries the risk of missing some string of interest if it is split by a tile boundary in the corpus. This boundary problem can be alleviated by querying for sufficiently long strings and setting an appropriate size of  $n$ . Observe that if a string of interest is at least of length  $2n - 1$  it will necessarily contain at least one tile of size  $n$ , producing a match. However, there may be hanging tokens at the start and end of a string. See Supplementary Appendix B for an example.
**Chaining** We can further chain a set of matching query n-grams into longer sequences. If matches occur  $n$  indices apart, they can be joined as a single inferred string (final row, Figure 2). This does not guarantee that the n-grams occurred in that particular order, but this is unlikely with long chains of long n-grams. Permutation attacks could produce bags of features that would harm other retrieval or indexing tools. See Appendix C for further discussion of adversarial permutations. Examining chains of matches at a document level alleviates the risk of false positives: a single n-gram match might be due to a false positive, but the probability of a chain of such matches decreases exponentially with the chain length.
Redistribution Previously we noted that other tools for inspecting data redistribute the content, leading to legal and privacy concerns. Since Bloom filters distribute only hashes, they provide some obfuscation. Bianchi et al. [3] term this "Better Than Nothing" privacy and discuss information hiding bounds on Bloom filters with various parameters. They note that the protection offered by one-way hashing is vulnerable when "the universe set is easily enumerable". In our case, this is either unicode sequences of length  $n$  or sequences  $V^n$  for tokens in a vocabulary set  $|V|$ . Storing strided n-grams provides additional information hiding without changing the size of the universe. Rather, it becomes harder to guess another member element given an existing match. If we stored every n-gram, an attacker could reconstruct documents by simply guessing single token extensions to an existing member. Since we store strided n-grams, an attacker would need to guess a sequence of length  $n$  to find the next element in a document. Note that in principle, proprietary LLM providers (e.g ChatGPT [29]) could release a Bloom filter based Data Portrait without revealing their exact training data - only the hashes of some substrings.
4 Case Studies
We construct several Data Portraits based on Bloom filters and demonstrate case studies. With the chosen parameters, these sketches are most suitable for checking document level queries. A live interactive interface is available at dataportraits.org.
4.1 Constructing Portraits
We use an optimized ingestion pipeline that requires only a single modest machine. Our pipeline supports both token and character-based sketches. We apply simple whitespace normalization that enables exact string matching on code snippets at different indentation levels. We use Redis<sup>4</sup> to store the resulting hashes.
Further examples in this paper use a sketch built on character n-grams of length 50. This n-gram width was chosen for two reasons. First, it is in-line with related work on dataset filtering and de-contamination. GPT-2 uses 8-grams to analyze test set leakage and GPT-3 uses between 8 and 13-grams for similar analysis based on properties of the test set [32, 8]. We find that an 8-gram corresponds to approximately 52 characters of text in the Pile. Secondly, to further validate this choice of parameter, we analyze a large subset of the Pile and find that out of all extracted 50-grams, only  $\sim 1.6\%$  of extracted n-grams are duplicated elsewhere in the subset. This value is  $7.5\%$  for 25-grams and  $79\%$  for 10-grams. In effect, a 50 character span is already a good document fingerprint. We also found that this width tends to match meaningful spans of text rather than spurious matches of common short spans. See Section 5.1 for further examples.
The Pile We construct a strided Bloom filter on The Pile [13] which is built on 825 GiB of documents. Certain subdatasets are re-weighted and the final dataset is distributed as 1254 GiB of text, after decompression. Our text pipeline takes around 12 hours on a machine with 40 cores and 140 GB of RAM. The token level version takes approximately 24 hours on the same machine, including the time to decompress and tokenize the Pile. The structure takes only 27 GB on disk and has a false positive rate of  $1 \times 10^{-3}$  (see Section 5.1 for further analysis of false positives). We note that these compute resources are fairly modest and our pipeline is stream based - the only requirement is sufficient RAM to hold the final structure. See Table 2 for an outline of the space usage of other structures that could be used for membership inference. With hash-based matching, we use  $\sim 14$  bits per data element (50 UTF-8 characters).
The Stack The Stack is a collection of permissively licensed code data obtained by downloading GitHub repositories [22]. We use the subset of The Stack used to train StarCoder, a 15.5B parameter large language model for code [25]. Specifically we process the data after test-set decontamination and preprocessing ( $\sim$ 800 GB of code). The Stack Portrait uses approximately the same amount of compute as the Pile Portrait, producing a 26GB documentation artifact.
4.2 WMT Overlap
Recent work on large language models (LLMs) has found that they learn language translation implicitly through supervision present in web-scale training sets [8, 37]. A reasonable reaction is to be suspicious that models have memorized target text from the training corpus. We scan WMT (Workshop on Machine Translation) test sets available through SacreBLEU [31], reconstructing documents by concatenating lines. Since line concatenation does not restore paragraph boundaries present, this is only a lower bound on overlap. We analyze non-English references since source documents are typically published English news articles.[5]
Given this collection of documents, we search for overlap using our Pile Bloom filter-based Data Portrait. We chain individual matching n-grams as described in Section 3.2. Recording the max length of a found chain gives us a measure of approximate longest overlapping subsequence. Results are shown in Figure 3 for the 10 longest matches with a more detailed table in Appendix D. Most of the test documents with high overlap are older test sets, where test data may have been publicly available for some time (e.g. on GitHub or rehosted by other projects). One exception is the WMT 2020 English Inuktitut [1] test set (bold). A full document from this test set appears in the Pile.<sup>6</sup> The exact training data is not known for many LLMs: we cannot conclude whether translation targets are memorized by a specific model, but our Pile-based sketch shows that some test data has leaked.
4https://redis.io/
Some WMT sets include reverse-created text [17]. Only recent LLM work [18] has analyzed this effect.
Original Article
Figure 3: Approximate longest overlapping subsequences between individual documents in WMT test sets and the Pile. Y-axis is year and language.
4.3 Overlap Detection
Following Dodge et al. [12, Table 2] we quantify the degree of overlap for selected test sets and the Pile. We do not seek to exactly replicate their table as there are several fundamental differences — they use full text exact match against C4 [33] and we use a measure of expected 50-gram overlap against the Pile.