Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models

Authors: sebastian  bordt
harsha  nori
vanessa  rodrigues
besmira  nushi
rich  caruana
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 15:14:42

Abstract: While many have shown how Large Language Models (LLMs) can be applied to a diverse set of tasks, the critical issues of data contamination and memorization are often glossed over. In this work, we address this concern for tabular data. Specifically, we introduce a variety of different techniques to assess whether a language model has seen a tabular dataset during training. This investigation reveals that LLMs have memorized many popular tabular datasets verbatim. We then compare the few-shot learning performance of LLMs on datasets that were seen during training to the performance on datasets released after training. We find that LLMs perform better on datasets seen during training, indicating that memorization leads to overfitting. At the same time, LLMs show non-trivial performance on novel datasets and are surprisingly robust to data transformations. We then investigate the in-context statistical learning abilities of LLMs. While LLMs are significantly better than random at solving statistical classification problems, the sample efficiency of few-shot learning lags behind traditional statistical learning algorithms, especially as the dimension of the problem increases. This suggests that much of the observed few-shot performance on novel real-world datasets is due to the LLM's world knowledge. Overall, our results highlight the importance of testing whether an LLM has seen an evaluation dataset during pre-training. We release the https://github.com/interpretml/LLM-Tabular-Memorization-Checker Python package to test LLMs for memorization of tabular datasets.

Main paper: 
 introduction: 
 Large Language Models (LLMs) exhibit remarkable performance on a diverse set of tasks (Wei et al., 2022; Bubeck et al., 2023; Liang et al., 2023) . While their prowess in natural language is undeniable, performance in other applications remains an ongoing research topic (Dziri et al., 2023a; Nori et al., 2023) . Recently, LLMs have increasingly been employed for diverse data modalities. This includes an increasing amount of creative applications in domains such as structured learning, tabular data, and time-series forecasting (Yin et al., 2023; Chen et al., 2023; Hegselmann et al., 2023; Jin et al., 2024) .A main question in current research on LLMs is the degree to which these models are able to extrapolate to novel tasks that are unlike what they have seen during training (Wu et al., 2023) . As such, an important aspect of LLM evaluation is to know if a task has been part of the model's training set. In this work, we refer to the case where an LLM is evaluated on a dataset seen during (pre-)training as training data contamination (Magar & Schwartz, 2022; Jiang et al., 2024) . Unfortunately, detecting whether an LLM has seen a certain text or dataset during training is rather challenging (Duan et al., 2024) . This is especially true for models to which we might only have query access via an API. Table 1 : Forecasting financial time series with GPT-3.5 and GPT-4 exhibits remarkable performance differences between years prior to and after 2021 (the cutoff of the training data). We perform few-shot learning, providing the model with the value of the time series on the previous 20 days and asking it to predict the value on the next day. The table depicts the robust mean relative error of the prediction across different years. The second column depicts the minimum and maximum error across the ten years from 2010 to 2019. The baseline predicts the value of the past day for the current day. Details in Supplement C.At the same time, a literature on memorization in LLMs has shown that language models can be prompted to repeat chunks of their training data verbatim (Carlini et al., 2019; 2021; Chang et al., 2023; Nasr et al., 2023) . Research has also shown that memorization occurs if an LLM sees a text repeatedly during training (Carlini et al., 2022b; Biderman et al., 2023) . Because of this, memorization can be seen as an extreme case of training data contamination where a dataset is not only seen during training but repeated within the training set so often that the LLM becomes able to consistently generate it.In this paper, we target the issue of training data contamination when evaluating the fewshot learning performance of LLMs on tasks with tabular data -an aspect often neglected in the rapidly growing literature on applications in this domain (Dinh et al., 2022; Borisov et al., 2023; Narayan et al., 2022; Vos et al., 2022; Hegselmann et al., 2023; Wang et al., 2023; McMaster et al., 2023) . Our first contribution is to develop various methods to detect whether an LLM has seen a tabular dataset during training. This includes four different tests for memorization. Our investigation reveals that GPT-3.5 and GPT-4 have memorized many popular tabular datasets verbatim (OpenAI, 2023) . For example, GPT-4 can consistently generate the entire Iris and Wine datasets from the UCI machine learning repository. What is more, the memorization of tabular datasets in GPT-series models is a robust phenomenon that does not depend on the precise model version.To gauge the effect of training data contamination on evaluations of state-of-the-art LLMs, we compare the few-shot learning performance of GPT-3.5 and GPT-4 on datasets that were seen during training to datasets that were released after training (Brown et al., 2020) . 1 In our experimental design, we also vary the format in which the data is presented to the LLM. In particular, we add small amounts of noise to numerical values in the dataset. The idea is that contamination is more likely to have an effect if the LLM can "recognize" a datapoint from pre-training.We find that LLMs perform better on datasets seen during training, indicating that memorization leads to overfitting. In addition, we find that adding small amounts of noise and other re-formatting techniques leads to an average accuracy drop of 6 percentage points on the memorized datasets. In contrast, the same transformations do not affect the few-shot learning performance on unseen data. To see the significant effect that training data contamination can have on few-shot learning, consider also Table 1 . It shows that GPT-3.5 and GPT-4 exhibit remarkable performance differences when predicting time series data for time periods prior to and after the LLM's training data cutoff.While we find significant evidence of training data contamination, we also find that GPT-3.5 and GPT-4 perform reasonably well on novel datasets. To better understand the LLMs' fewshot learning performance on the novel datasets, we conduct ablation studies and investigate their performance as in-context statistical predictors (Garg et al., 2022) . In particular, we remove the feature names and standardize the data to zero mean and constant variance. We find that GPT-3.5 and GPT-4 can still perform in-context statistical classification better than random but struggle as the dimension of the problem increases. This leads us to conclude that the few-shot learning performance on novel tabular datasets is due to the LLM's world knowledge. Interestingly, we also find that the performance of in-context statistical learning with GPT-4 scales in the number of few-shot examples, whereas it remains more flat for GPT-3.5.The paper is organized as follows. We first present our memorization tests. We then study the few-shot learning performance on memorized and novel tabular datasets. We then study the ability of LLMs to act as statistical predictors during few-shot learning. Finally, we show that LLMs can draw random samples from datasets they have seen during training. 
datasets: 
 In this study, we use two different kinds of tabular data. First, we use datasets that were freely available on the Internet before 2021. This includes the Iris , Wine , Adult and Housing datasets from the UCI machine learning repository (Kelley Pace & Barry, 1997) . It also includes the OpenML Diabetes dataset and the Titanic dataset from Kaggle (Smith et al., 1988) . We use red color to indicate these popular datasets.Second, we use datasets that were not freely available on the Internet before 2022. This includes the ACS Income and ACS Travel datasets (Ding et al., 2021) . These two datasets were derived from the 2022 American Community Survey (ACS). Notably, the ACS Income dataset was constructed by Ding et al. (2021) to obtain a dataset similar to the Adult dataset. We also use the Spaceship Titanic dataset, released on Kaggle in 2022, and the FICO dataset that has been privately held (the dataset is available for download after registration). We also introduce the ICU dataset, a novel small machine learning dataset where the task is to predict whether a patient is being treated in intensive or intermediate care. We derived this dataset from data in the Harvard dataverse (Goad, 2018) . We use blue color to indicate the novel datasets. Additional details on the datasets can be found in Supplement A. 
testing language models for memorization of tabular data: 
 There are various systematic ways to test what a language model knows about a tabular dataset. For example, we can test whether the LLM can list a dataset's feature names or tell the possible values of categorical features in the data. It is also possible to heuristically assess whether the LLM has learned different aspects of the data distribution. In this Section, we focus on tests for verbatim memorization (Carlini et al., 2019; 2021) . Additional heuristics and details are described in Supplement D.We introduce four different tests to detect memorization. These tests use the fact that most tabular datasets have a canonical representation as a CSV file (which is nothing but a text document that might end up in the pre-training data). Intuitively, we say that a tabular dataset is memorized if the LLM can consistently generate it. In our context of tabular datasets, this is justified by the fact that these datasets contain random variables: It is impossible to consistently reproduce the realizations of random variables unless the values of the random variables have been seen before (i.e., during pre-training). 2Previous work on memorization in language models has often relied on the prefix-suffix pattern, where the model is provided with the initial part of a string as a prefix and asked to provide a completion (Carlini et al., 2019; 2021; 2022b) . In this work, we use few-shot learning to condition chat models on the task of completing a given prefix, a prompt strategy Table 2 shows the results of the memorization tests on 11 tabular datasets for GPT-3.5 and GPT-4. The The header test indicates memorization of all the tabular datasets that were publicly available on the internet before 2021. This means that GPT-3.5 and GPT-4 have memorized the initial rows of these datasets. On Titanic , Diabetes , Wine , and Iris , the row completion, feature completion, and first token test also provide evidence of memorization. This means that GPT-3.5 and GPT-4 have not only memorized the initial rows of these datasets (which are frequently in Jupyter notebooks) but also random rows. Of course, this means that GPT-3.5 and GPT-4 have essentially memorized the entire datasets. We also observe that GPT-4 exhibits somewhat stronger evidence for memorization than GPT-3.5, which aligns with the literature that shows that larger models exhibit more memorization (Carlini et al., 2022a) . Interestingly, there is no evidence of memorization of random rows on Adult and Housing .On the novel datasets, there is no evidence of memorization for GPT-3.5 and GPT-4. This is to be expected for datasets released after the model was trained. In the case of the FICO dataset, it suggests that this dataset might have been protected from inclusion in the LLM training data by the need to register prior to access. 3Table 3 shows the results of the header test and the row completion test for five different open LLMs (Groeneveld et al., 2024; Riviere et al., 2024; Dubey et al., 2024; Qwen Team, 2024) . Similar to the results observed in Table 2 , there is evidence for the memorization of the most popular tabular datasets in almost all LLMs. However, with the exception of the Iris dataset, there is usually only evidence for the memorization of the initial rows and not of the entire dataset. 
the first token test:: 
 We prompt the model with a number of contiguous rows from a random position of the CSV file and ask it to complete the first token of the next row. If the rows of the CSV file are known to be random, we can perform a statistical test between the LLM completion accuracy and the accuracy of completion with the mode.It is helpful to think about the proposed memorization tests in terms of power and significance, as we would about a statistical hypothesis test. All the tests are highly significant. This means they provide definitive evidence of memorization (though not necessarily of the entire dataset). Conversely, little can be said about the power of the tests. It seems possible that an LLM has memorized a tabular dataset, but we cannot extract it via prompting.In the previous Section, we demonstrated that LLMs have memorized many popular tabular datasets. In this Section, we investigate the few-shot learning performance of GPT-3.5 and GPT-4 on the novel and memorized datasets. In addition, we replicate the entire analysis in this Section with Llama 3.1 70B and Gemma 2 27B. This can be found in Supplement F.Prompts. We build on previous works and prompt chat models with tabular data in the format "Feature Name = Feature Value" (Borisov et al., 2023; Hegselmann et al., 2023) .System: You are a classification assistant who is an expert in tabular data, data science, and cross-sectional population surveys. We make use of an intriguing property of tabular data: It is possible to make changes to the format of the data without (significantly) affecting the underlying classification problem (compare Figure 1 ). We use this fact to explore (1) the consequences of memorization in Section 4.2 and (2) whether LLMs make use of their world knowledge in Section 4.3. In all few-shot learning experiments, we present the data in one of four standardized formats:Original means that we present the data as it is contained in the CSV file of the dataset. In the perturbed version, we slightly change individual digits in the data that are not relevant to the underlying classification problem. We also deface any unique identifiers, such as observation IDs and names. In the system prompt of both original and perturbed, we tell the model about the origin of the data ("You help to make predictions on the Titanic dataset from Kaggle"). In the task version, we change the names of the features without changing their meaning ("BMI" becomes "Body mass index"), and similarly re-code categorical values ("0" becomes "False" and "United-States" becomes "USA"). We also round numerical values to two digits (unless this interferes with the meaning of a variable) and provide a generic system prompt ('You help to predict the type of a wine from its features."). In the statistical version, all numeric features are standardized to zero mean and constant variance. Feature names are replaced with X1, ..., Xn and strings encoded as categorical variables.An important aspect of our datasets transforms is that they are standardized and comparable across datasets. Additional details on how we standardized the transformations are in Supplement E.Table 4 depicts the few-shot learning performance of GPT-4 and GPT-3.5 on 10 different tabular datasets. On the memorized datasets depicted in Panel A of Table 4 , the performance is quite impressive. In particular, GPT-4 outperforms logistic regression on 3 out of 5 datasets when prompted with the original data. However, the predictive performance of GPT-4 and GPT-3.5 drops as we move from the original data to the perturbed data. As depicted in Figures 11a and 11c , the accuracy of GPT-4 and GPT-3.5 on the memorized datasets consistently drops as we present the data in the original, perturbed, task and statistical format. In particular, GPT-4 on the task data outperforms logistic regression only on 1 out of 5 datasets, and even there, the measured difference lies within the standard error.Panel B of Table 4 depicts the few-shot learning performance of GPT-4 and GPT-3.5 on novel datasets. In contrast to the results in Panel A, GPT-4 in the original format does not outperform logistic regression on any dataset. Moreover, we also don't observe significant changes between the original, perturbed, and task formats. This is again depicted in Figures 11b and 11d . While it is always possible that the few-shot learning performance of an LLM varies with subtleties such as the length of a feature name or the number of significant digits of a number, the results in Panel B of Table 4 indicate that GPT-4 and GPT-3.5 are fairly robust to such changes.The striking difference in performance between the different dataset formats on the memorized datasets, which is completely absent on the novel datasets, strongly suggests that memorization leads to invalid performance estimates of few-shot learning. Even though the datasets in Panel A and Panel B of Table 4 vary along other dimensions than being memorized or novel, it seems highly unlikely that these give rise to the observed performance differences with logistic regression or the observed performance drops between the different dataset versions.We note that the GPT-4 results in Table 4 were obtained with gpt-4-0125-preview, which has seen data up to December 2023. On the memorization tests, gpt-4-0125-preview obtains the same results as the GPT-4 models with a training data cutoff in 2021.4 While the results in Panel A of Table 4 indicate that few-shot learning on datasets seen during training can suffer from overfitting, Panel B demonstrates that GPT-3.5 and GPT-4 perform reasonably well on novel data. 4 .Moreover, on all datasets except Spaceship Titanic, the performance is consistent across the original, perturbed, and task formats but significantly drops for the statistical format. This indicates that the LLMs rely on the natural scale of the feature values and the variable names, which are made unrecognizable in the statistical format. We suggest that this effect is not observed on Spaceship Titanic because this dataset is synthetically generated and about a fictional event with fictional variable names.This effect is also depicted in Figure 2 , where there is a significant drop in accuracy from the task to statistical setting in all subfigures.In the previous section, we have seen a significant performance drop between task and statistical, indicating that the few-shot learning performance of LLMs significantly relies on the LLM's world knowledge (Yu et al., 2023) . In this section, we study the ability of language models to act as in-context statistical learners. This relates to works using small, specifically trained language models to show that in-context learning can learn simple function classes, akin to supervised learning (Garg et al., 2022; Von Oswald et al., 2023) . We investigate the question in the context of GPT-3.5 and GPT-4, similarly to Bhattamishra et al. (2024) .Binary Statistical Classification. We study a simple statistical learning problem with numerical features x i ∼ N (0, I) ∈ R d , an unknown coefficient vector z ∼ N (0, I) ∈ R d , and binary labels y i = (x T i z >= 0). The data is presented to the model in the statistical format (see Figure 1 ). 2023) on our binary statistical classification problem. We see that the performance of the fine-tuning approach scales in the number of samples. 
drawing random samples from datasets seen during training: 
 In the previous Section, we explored the implications of memorization on few-shot learning with tabular data. However, predicting the target label is only one of many tasks that one might want to perform with LLMs and tabular data (Hollmann et al., 2024; Bordt et al., 2024; Sui et al., 2024) . In this section, we demonstrate another implication of seeing a dataset during training: We show that LLMs can draw random samples from these datasets -without any fine-tuning (Borisov et al., 2023) .Figure 4 depicts the longitude and latitude of random samples on the California Housing dataset. To draw these samples, we conditioned GPT-3.5 with samples from other datasets, Figure 4 : GPT-3.5 can draw random saFiguremple from the California Housing dataset. We only provide the model with the name of the dataset and the feature names. The diversity of the generated samples depends on the temperature parameter. For small temperatures, the samples are concentrated around the mode of data. As temperature increases, the samples become more diverse and similar to the data distribution. At large temperatures, some samples lie outside the support of the data distribution. The reader might want to compare with Figure 1 in Borisov et al. (2023) .- revealing no information about the feature values on the California Housing dataset. Interestingly, the samples drawn by the LLM follow the overall summary statistics in the data. For example, the feature correlations of the samples match the feature correlation in the original dataset (Supplement Figure 16 ). At the same time, the samples are not copied from the training data (Supplement Table 11 ). In this sense, it can be said that GPT-3.5 generates novel random samples from the dataset. 
related work: 
 Recent works have demonstrated the capabilities of LLMs on tasks with tabular data (Dinh et al., 2022; Narayan et al., 2022; Vos et al., 2022; Wang et al., 2023; McMaster et al., 2023) Research on memorization has shown that LLMs can be prompted to repeat chunks of their training data verbatim (Carlini et al., 2019; 2021; Chang et al., 2023; Nasr et al., 2023) .Memorization is linked to data-duplication in the pre-training data (Carlini et al., 2022b; Biderman et al., 2023) . Memorization does not necessarily lead to overfitting (Magar & Schwartz, 2022) .LLMs tend to perform better on tasks more likely to occur in the training data (Wu et al., 2023; McCoy et al., 2023; Dziri et al., 2023b) . Our work complements this emerging literature by (1) rigorously demonstrating that GPT-3.5 and GPT-4 have seen certain tasks frequently during training (whereas previous works have relied on heuristics) and ( 2) showing that LLMs overfit on uncommon tasks that were seen frequently during training. 
discussion: 
 In this work, we have first shown that LLMs have memorized many of the popular tabular datasets. We have then used this fact to gauge the amount of overfitting that occurs during in-context learning with such datasets. We find strong evidence of overfitting in in-context learning due to memorization. We have also studied other factors that determine the fewshot learning performance of GPT-3.5 and GPT-4 on prediction tasks with tabular data, finding that these models rely on their world knowledge, but also have the ability to act as statistical predictors.Our study has several limitations, such as that we usually don't know if and in what way the LLMs saw the different tabular datasets during training. Also, it would have been more ideal had we been able to insert datasets into the training corpus at random.Finally, we want to highlight that had we performed our evaluations without carefully controlling for memorization, we might have erroneously concluded that few-shot learning with GPT-4 frequently outperforms traditional statistical learning algorithms. 
ethics statement: 
 All the datasets we use in this study are explicitly available for scientific research. Because our study concerns the scientific question of invalid performance evaluations of LLMs, we don't believe it raises significant ethical concerns. While we consider memorization purely from the perspective of training data contamination, it has broader implications, including copyright and privacy. 
reproducibility statment: 
 We conducted initial experiments with different versions of GPT-3.5 and GPT-4 and found that the results are fairly robust towards the precise model version. This holds true for both the results of the memorization tests in Table 2 and for the few-shot learning results in Table 4 . An exception is the model gpt-3.5-turbo-1106 that performs worse on the few-shot learning tasks than other versions of GPT-3.5. This dataset can be obtained only after signing a data usage agreement and is then available via Google Drive. This is an example of a dataset that is freely available but where the CSV file has not been publicly released on the internet, at least not by the creators of the dataset.Spaceship Titanic. The Spaceship Titanic dataset was released on Kaggle in 2022 https: //www.kaggle.com/c/spaceship-titanic. It is a synthetically generated dataset.ACSIncome and ACSTravel. We created these two datasets using the folktables package and Census Data from the 2022 American Community Survey (ACS). The construction of the datasets is described in (Ding et al., 2021) .ICU. We created the UCI dataset as an example of a novel tabular dataset with few observations and features, similar to some of the traditional machine learning datasets. We created the dataset from the data at https://dataverse.harvard.edu/dataset.xhtml? persistentId=doi:10.7910/DVN/PX9K2R. The dataset has 10 features (patient characteristics) and a binary target. It has 102 observations.Here provide the details of the models that were used to generate the results in the paper.Table 1 : gpt-3.5-turbo-0613 and GPT-4-32k-0315 deployed on Azure.Table 2 : gpt-3.5-turbo-16k-0613 and gpt-4-0613 deployed on Azure, with the exception of gpt-3.5-turbo-0125 and gpt-4-0125-preview with the OpenAI API for the ACS Income, ACS Travel and ICU datasets. We also ran the memorization tests with gpt-3.5-turbo-16k-0613 and gpt-4-0125-preview with the OpenAI API and obtained similar results. The main difference is that gpt-4-0125-preview knows the Spaceship Titanic dataset's feature names and feature values (but there is no evidence of memorization from our tests).Table 3 : gpt-3.5-turbo-0125 and gpt-4-0125-preview with the OpenAI API. Logistic Regression. We train logistic regression using scikit-learn (Pedregosa et al., 2011) . We cross-validate the L 2 regularization constant.Gradient Boosted Tree. We train gradient boosted trees using xgboost (Chen & Guestrin, 2016) . We cross-validate the max depth of the trees and the L 1 and L 2 regularization constants, similar to (Hegselmann et al., 2023) .For each year, we compute the robust mean relative error asEQUATIONWe use a simple system prompt.System: "You are an expert financial market analyst and stock trader and your task is to predict the development of the MSCI World stock index in 2022. You are given the price level of the stock index in the previous 20 days and predict the price of the stock index on the current day."This Section provides additional details on the memorization tests and their quantitative results. We also detail a number of measures that assess what the LLM knows about the dataset (for example, the feature names).With regard to the interpretation of the memorization tests, it is important to note that the quantitative test results (e.g., how many rows the LLM managed to complete in the row completion test) are not sufficient to judge memorization. Instead, one has to consider the completion rate of the LLM and the amount of entropy in the dataset. For example, the OpenML Diabetes dataset contains an individual's glucose level, blood pressure, and BMI, as well as other measurements that are at least in part random and highly unique.If an LLM can consistently generate even a few rows of this dataset, it is strong evidence of memorization. To give another example, the Iris dataset contains many rows that are near-duplicates. This means that an LLM might also achieve a non-zero row completion rate by chance or prediction. Because of this complication, we manually judge the results in Table 2 and Table 3 in the main paper, considering the quantitative results of the different tests and the structure of the dataset.As a general observation, one might expect the results of the tests to be somewhat ambiguous. It turns out, however, that GPT-3.5 and GPT-4 (as well as other LLMs) have the remarkable property of regurgitating quite long and complex strings from their training data. For example, GPT-4 is able to generate the entire Iris dataset, consisting of 151 rows, given only the first couple of rows as prompt. Thus, while there are few cases of ambiguity (marked as question marks in the respective tables), the evidence of memorization is usually fairly clear (and conversely, there are cases where the model gets the dataset's content completely wrong).All memorization tests are conducted at temperature 0.Figures 5-10 on the following pages illustrate the different memorization tests. They depict the Leveshtein string distance between the model response and the actual content in the dataset. For the different tests, we depict two cases: one with evidence of memorization and one without. is depicted in Figure 8 , where the correctly completed row on FICO is one of the many duplicate rows in the dataset that consists repeatedly of the number -9. Of course, this does not provide evidence of memorization.For the feature completion test in Table 5 , we count the number of times that a feature value is correctly completed. We chose the feature to be completed as a highly unique feature (the Iris dataset does not contain such a feature, so we don't conduct the test on this dataset).The results obtained for the feature completion test are very similar to those of the row completion test. Again, we see that the number of feature completions is either zero / in very low digits or fairly large, with over 90% correctly completed feature values on some datasets.For the first token test depicted in Table 6 , we count the number of times that the model correctly predicts the first token of the next row in the CSV file. The prompts for this test are the same as for the row completion test; the difference is that we don't assess whether the entire row is correctly completed but only consider the first token. The first token test is especially interesting for the Adult dataset because we know from the construction of the dataset that the rows are ordered at random. In this case, a model that has never seen the dataset cannot be better at completing the first token than the mode. Interestingly, we see that the completion rate of GPT-4 is exactly at the rate of the mode.Here we provide additional details on the different tests.The feature names test asks the LLM to complete the names of the features of a dataset, given the name of the first feature. The test uses our general prompt structure with few-shot examples, given in Figure 14 . In all cases, the result of this test was unambiguous. The model would either list the names of the different features in the same format as in the CSV file of the dataset, or it would offer clearly made-up feature names.The feature values test asks the LLM to provide observations from the dataset that have valid feature values (most importantly, valid string values for categorical features). This test uses the prompt structure used in Section 5 in the main paper to draw samples from the dataset. We then manually check whether the formatting of the sampled feature values is the same as the formatting in the training data.We ask the model to provide samples from the dataset at temperature 0.7. We then compare the model of the sample feature values with the mode of the feature in the data. This test is also motivated by the empirical observation that GPT-3.5 and GPT-4 model the modal feature values of many publicly available datasets surprisingly well, even if they rarely provide a good model of the finer details of the feature distribution.We draw samples at temperature 0.7 and compare the Pearson correlation coefficients of the samples with the Pearson correlation coefficients in the original dataset (compare Figure 16 ).For the header test, we split the dataset at random positions in rows 2, 4, 6, and 8 and ask the model to complete the dataset from the given position. We condition the model on this task with few-shot examples and consider the best completion. The test succeeds if the model completes at least the next row in the dataset. This was unambiguous on all datasets (the model completed many rows or not even a single row). The prompt is depicted in Figure 13 .Here, we describe the different dataset formats in more detail.Original. We present the feature names and values as they occur in the CSV file of the dataset. In the system prompt, we tell the model about the origin of the data ("You help to make predictions on the Titanic dataset from Kaggle").Perturbed. Starting from the original data, we perturb continuous and discrete variables by randomly perturbing individual digits. We perturb the data so that we (1) change at least a single digit in all non-zero values in the data and (2) perturb with a relative error of approximately 1%. This means that the perturbation depends on a variable's magnitude and number of significant digits. The perturbations do not affect the format of the variables in the data, meaning that the perturbed data still "looks like" the original data. However, the numerical values are slightly different.Because the goal of the perturbed version is to make it harder for the LLM to "recognize" the data, unique identifiers are given special treatment: All unique identifiers are changed to values that do not occur in the original dataset. We note that applying meaningful perturbations requires background knowledge (we only perturb variables that can be slightly changed without changing their meaning in the underlying problem).For perturbed, we use the same system prompt as for original.Task. Starting from the perturbed data, we re-format it. Numerical values are formatted to two decimal places by rounding and adding small amounts of noise. Categorical variables are re-coded ("0" becomes "False" and "United-States" becomes "USA") and features renamed ("BMI" becomes "Body Mass Index"). We also remove any hits on the original dataset from the system prompt ("You help to make predictions on the UCI Wine dataset, predicting the type of a wine from its features." becomes "You help to predict the type of a wine from its features.").While we re-format the values under the task, we do not significantly change the values in the data. However, because the task builds on perturbed, the features still take slightly different values than when we had re-formatted the original dataset.Starting from the task data, we encode categorical variables into numeric values. All numeric features are standardized to zero mean and constant variance. The statistical transform was chosen to transform the appearance of the data without changing the underlying statistics.Table 9 and Table 10 confirm that the different dataset transformations do not change the underlying statistical classification problem in any significant way.In this Section, we replicate the experiments from Section 4 in the main paper using Llama 3.1 70B and Gemma 2 27B (Dubey et al., 2024; Riviere et al., 2024) . We do not perform any changes to the setup of the experiment, that is we send the same prompts to the open-weight models that were previously send to GPT-3.5 and GPT-4 (including the ordering of the few-shot examples). We use https://www.together.ai/ as our inference engine, which means that the experiments can be performed with the same code that was used to send the prompts to the OpenAI API. The model endpoints are Meta-Llama-3.1-70B-Instruct-Turbo and google/gemma-2-27b-it. Code is available at https://github.com/interpretml/LLM-Tabular-Memorization-Checker.While the experiment in this Section can be seen as a replication of the experiment in Section 4 in the main paper, there is an important difference between the experiment detailed in 7 and Table 8 , respectively. This figure replicates Figure 2 in the main paper. Note that we have replaced "memorized" and "novel" in the description of the Figure with "Panel A" and "Panel B". This is because the results of the memorization tests for Llama 3.1 70B and Gemma 2 27B indicate that there is less memorization than for GPT-3.5 and GPT-4.Figure 11 ). On the datasets that are novel for GPT-3.5, both models show a similar pattern across dataset transformations, but Llama 3.1 70B performs, on average, better.The results for Gemma 2 across the popular tabular datasets are similar to the results in the main paper, that is we see the characteristic decaying pattern across dataset transformations (this is depicted in Subfigure (c) of Figure 11 ). Gemma 2 depicts a slightly decaying pattern across dataset transformations also for the datasets that are novel for GPT-3.5 and GPT-4, even if this pattern is less pronounced and not statistically significant (this is depicted in Subfigure (d) of Figure 11 ). This result might be due to random fluctuations, or it might be that some of the datasets are included in the pre-training data of the model. More research would be needed to understand this in detail.At a high level, it is interesting to see that the results obtained with Llama 3.1 70 B and Gemma 2 27B are overall similar to the results obtained with GPT-3.5 and GPT-4, even if some of the assumptions around the training data cutoff and memorization don't necessarily hold in the same way. Of course, this might be because the popular tabular datasets are still more likely to be included in the pre-training data than some of the novel datasets, even if the novel datasets were, strictly speaking, released before the cutoff of the training data.We do not know the training data of the LLMs. We use the term "seen during training" when there is evidence of verbatim memorization of at least part of a dataset.This approach has been lightly formalized inCarlini et al. (2019) andCarlini et al. (2021), where the authors use the term 'canary'.Of course, it is possible that the LLM saw the dataset during training, but there was no memorization, or our memorization tests are not powerful enough.Producing the results in Table4with gpt-4-32k-0613 would have been prohibitively expensive. We conducted limited experiments with gpt-4-32k-0613 on the original data format where the fewshot learning performance matched gpt-4-0125-preview. 
 
 Appendix:  


Decision: This submission investigates the critical issue of data contamination and memorization in large language models (LLMs), specifically focusing on tabular data. The paper introduces various techniques to assess whether an LLM has seen a particular tabular dataset during training. The authors find that GPT-3.5 and GPT-4 have memorized many popular tabular datasets verbatim, leading to better performance on datasets seen during training, indicating overfitting due to memorization. However, the models also demonstrate non-trivial performance on novel datasets and robustness to certain data transformations.

The reviewers acknowledge the paper's value in addressing an important concern regarding data contamination and memorization in LLMs. The focus on tabular data is novel and extends the understanding of LLMs to structured data. The introduction of various techniques to determine whether an LLM has seen a specific dataset during training is a valuable contribution that can be used by other researchers to ensure the integrity of their evaluations.

Reviewer MP4s raises a minor concern about the limited set of datasets used, suggesting that the inclusion of less popular datasets could provide a broader spectrum of results. Reviewer ZJeMS suggests evaluating the memorization issues on a different set of LLMs, especially open-source models, and providing a deeper explanation of the evidence for memorization of tabular datasets from the training data. Reviewer sEZf notes that while the findings on the effects of memorization on few-shot learning are novel for tabular datasets, similar observations have been made for datasets in general in prior work, which should be cited. 

Overall, the reviewers agree that the paper addresses a critical issue, provides valuable contributions, and should be accepted for publication. The outcome is a clear accept. Please address all suggestions of the reviewers for the final version.  A personal request, in line with reviewer ZJeMS is the application of this study to a open source LLMs (open datasets included). This would compare the findings in this paper with a transparent fine-tuning setup.

[comment from the PCs] Please follow up on the AC's request regarding open-weight/source models. This is important for research.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: The paper addresses the issue of data contamination and memorization in LLMs, particularly focusing on tabular data. It introduces various techniques to test whether a language model has seen specific datasets during training. The study finds that GPT-3.5 and GPT-4 can memorize popular tabular datasets verbatim and that these models perform better on datasets they have seen before. This memorization can lead to overfitting. However, the models show non-trivial performance on new, unseen datasets and are somewhat robust to data transformations. The paper concludes that careful testing using the developed exposure tests is essential to determine if a dataset has been seen during pre-training.

Main Findings:
1. Data Contamination: The study reveals that LLMs can memorize large portions of tabular datasets they were trained on, impacting their performance evaluation.
2. Few-Shot Learning Performance: The models perform better on datasets they have seen before. Adding small amounts of noise or reformatting reduces this performance, indicating memorization and data contamination.
3. Performance on Unseen Data: Despite memorization, GPT-3.5 and GPT-4 show significant performance on novel datasets, meaning they are not solely relying on previously seen data.
4. Comparison with Baseline: When compared to traditional statistical learning algorithms, GPT-3.5 and GPT-4 still perform competitively even on unseen datasets, particularly in few-shot learning scenarios.
5. Robustness to Transformations: Even though the models seem robust to certain data transformations, careful evaluation is required to ensure integrity in performance assessments.

Without a doubt, I find this paper valuable for the community. My only concern is the limited set of datasets. Most of them are standard in ML,DL learning which makes them obvious candidates to be memorized. There are several less popular datasets before 2021 that could be used to have a broader spectrum of results.
strength_weakness: strength_weakness: It has more than 9 pages, the appendices also make it particularly long.
A limited set of datasets. Most of them are standard in ML,DL learning which makes them obvious candidates to be memorized.
The related work section looks somewhat lacking; it's evident that the authors sacrificed this section to give space to others.
The use of the word 'elephant' in the title is somewhat strange; the analogy seems excessive to me."
paper_summary: paper_summary: The paper introduces different techniques to evaluate if a language model has seen a specific tabular dataset during training. Second, the authors evaluate fe-shot performance of datasets which were seen during training and datasets that were published after the cut-off date of the LLM training data.  They use a varied set of datasets and perturbations on top of them to study the memorization and few-shot learning capabilities of GPT-3.5 and GPT4.
strength_weakness: strength_weakness: First, I consider that the evaluation only on GPT-3.5 and GPT-4 models is limited and it would be much more valuable to study the memorisation issues on a different set of LLMs and especially open-source models (e.g. Olmo). Also, this addition would help understanding if these datasets are present in the training data and their frequency and not having to use heuristics for predicting if a dataset was present during pertaining.

I also think that the paper lacks a deeper explanation on the evidences of memorisation of the tabular datasets from the training data.
paper_summary: paper_summary: Paper summary: This paper investigates how seeing a dataset during pretraining affects the model's performance on the dataset.  They consider datasets that are stored in tabular form, specifically CSV format, as this is a very common format for storing datasets.  They develop novel exposure tests for this type of data, and find that LLMs perform much better on datasets they have seen during pretraining.  They also investigate the in-context learning abilities of LLMs, and find that without fine-tuning, their in-context learning abilities to be limited.

Review summary: The quality, clarity, originality and significance of this work is quite high.  It is a very good fit for COLM.
strength_weakness: strength_weakness: They call this type of contamination "training data contamination".  In prior work, this has been called "task contamination" (see "Task Contamination: Language Models May Not Be Few-Shot Anymore" Li & Flanigan 2023 https://arxiv.org/abs/2312.16337).  While it's ok that they have used a different term for this, they need to cite this previous work.  Additionally, the above paper somewhat reduces the novelty claims in introduction.
paper_summary: paper_summary: Disclaimer: I am an NLP researcher/computational linguist and do not feel qualified to review this paper. It applies LLMs to tasks that are pretty much outside of NLP.

This paper discusses experiments that use LLMs as "statistical predictors" or classifiers on tabular data, e.g. classification on the famous Iris datasets. It shows that GPT3.5 and GPT4 have memorized a couple of such popular tabular datasets. It describes experiments that perturb these datasets in systematic ways and reports the resulting performance in LLMs.
strength_weakness: strength_weakness: The memorization tests for the tabular datasets seem somewhat simple and are described superficially. The "tests" prompt the LLM with certain elements of the table and record whether the LLM regenerates parts of the tabular data. The results of these tests are presented in a table that just shows some red crosses/green checkmarks for +/- memorized and it is unclear how these crosses and checkmarks have been computed. The supplement gives more detailed numbers, but I could not determine why certain tests/datasets have checkmarks and others do not. For instance, why does the "Adult" dataset have a checkmark for the header test and red crosses for the others? (In supplement D, it looks like the GPTs cannot regenerate any of the elements in this dataset)

It is unclear how exactly the tests have been conducted and evaluated (how many runs, how many rows, features etc.). For instance, which portions of regenerated rows are considered as indications of memorization? 

The results of memorization tests do not seem entirely conclusive, for instance in the "Adult" dataset. Is the dataset memorized or not? What exactly are the end results of the test battery for each dataset? As one of the main contributions of this paper is to show memorization of tabular data, I would have expected more experiments, substance, and systematic validation on this.

This is not really a weakness of the paper per se, but I cannot stop myself from writing this here: given everything we know about how LLMs are built and trained, it is pretty clear that they are *not* reliable statistical predictors. I believe that using an LLM for this task is an "unintended" and maybe even an "unreasonable" use. I would expect at least some discussion and reflection on this in the paper. Since LLMs are now widely used by lay users, unintended uses like this one do come with ethical concerns and should be approached with some care.

Post-rebuttal: The discussion with the authors brought up some interesting points and clarified most of my concerns ... I have adjusted my score accordingly.
questions: questions: See weaknesses above -- more details on how the memorization tests have been evaluated would be very welcome.

Comments: title: Response to comment
Response to comment
Response to rebuttal
comment: Thanks, this helped me to understand what is going on in the memorization tests. I hope that some of these comments and visualizations can find their way into the paper! I have adjusted my score, but I will keep my confidence low as the paper really is outside of my area of expertise.
Thanks for your response!

Although the memorization tests lead to quantitative results, such as the number of rows that were correctly completed, we don't define memorization quantitatively.

Our notion of memorization follows the approach outlined in a series of papers on memorization in LLMs, most importantly Carlini et al. 2019 https://arxiv.org/abs/1802.08232 and Carlini et al. 2021 https://arxiv.org/abs/2012.07805.

Following this literature, we say a text is memorized if a model can consistently generate it.

Of course, this makes sense only for texts that contain sufficient entropy so that the probability of generating the text by chance or prediction is close to zero.

In our context of tabular datasets, we rely on the fact that these datasets contain random variables: It is impossible to consistently reproduce the realizations of random variables unless the values of the random variables have been seen before (i.e., during pre-training). 

For example, the OpenML Diabetes dataset contains an individual's glucose level, blood pressure, and BMI, as well as other measurements that are at least in part random. Now, if an LLM can consistently generate entire rows of this dataset, this is evidence of memorization. We note that this approach has been lightly formalized in the two papers referenced above, where the authors use the term "canary."

In judging memorization, one has to consider the completion rate of the LLM and the amount of entropy in the dataset. To give another example, the Iris dataset contains many rows that are near-duplicates. This means that an LLM might also achieve a non-zero row completion rate by chance or prediction. Because of this complication, we manually judge the results in Table 2, given the results in Supplement Table 4, and the structure of the dataset.

Thankfully for our approach, it turns out that  GPT-3.5 and GPT-4 (as well as other LLMs) have the remarkable property of regurgitating quite long and complex strings from their training data. For example, GPT-4 is able to generate *the entire Iris dataset*, consisting of 151 rows, given only the first couple of rows as prompt. Thus, while there are few cases of ambiguity (marked as question marks in Table 2), the evidence of memorization is usually fairly clear (and conversely, there are the cases where the model gets the dataset's content completely wrong).

To better illustrate the nature of our memorization tests, we have created additional Figures depicting the qualitative test results: https://github.com/colm365/code/blob/main/memorization_tests.pdf. The figures illustrate the Leveshtein string distance between the model response and the actual content in the dataset. For the different tests, we depict two cases, one where there is evidence of memorization and one where there is no evidence of memorization.

We would like to highlight that the anonymous code repository contains a Python package that can perform all the memorization tests (and the package can also print the prompts).

We again thank the reviewer for engaging in the discussion and would be happy to answer any further questions.
Dear Reviewer, could you please check if the author's answers are helpful and if there is any follow up question that you have?
Thanks for the interesting response. The reference to Oswald et al. 2023 would have helped me lot, to understand the general background of this work.

Regarding the memorization tests, I am still not entirely happy. Could you give some more details as to how you defined "memorization" quantitatively? Or is Table 2 obtained from "eyeballing"/manually checking the test battery results?

Your explanation on the adult dataset suggests that memorization is not a binary thing, but that LLMs sometimes seem to memorize particular parts of a dataset.  This is an interesting observation, but it is not discussed in the paper if I am not mistaken. What would be the consequences of "partial memorization"? Do you think that it differs from "full memorization"?
Thank you for the responses! I agree that knowing the pre-training data is impossible, but clues can be used to determine what is common and what is not. For example, the number of views/downloads (available on the UCI repository), the publication date, the number of papers citing it, the number of Google search results for a query, etc. You can include it in your future work!