Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models
Authors: Sebastian Bordt
Harsha Nori
Vanessa Cristiny Rodrigues Vasconcelos
Besmira Nushi
Rich Caruana
Venue: COLM (2024)
Keywords: Memorization, Few-Shot Learning, Tabular Data
TL;DR: We compare the few-shot learning abilities of LLMs on tabular datasets that they have seen during training to datasets that were released after the cutoff date of the training data.

Abstract: While many have shown how Large Language Models (LLMs) can be applied to a diverse set of tasks, the critical issues of data contamination and memorization are often glossed over. In this work, we address this concern for tabular data. Specifically, we introduce a variety of different techniques to assess whether a language model has seen a tabular dataset during training. This investigation reveals that LLMs have memorized many popular tabular datasets verbatim. We then compare the few-shot learning performance of LLMs on datasets that were seen during training to the performance on datasets released after training. We find that LLMs perform better on datasets seen during training, indicating that memorization leads to overfitting. At the same time, LLMs show non-trivial performance on novel datasets and are surprisingly robust to data transformations. We then investigate the in-context statistical learning abilities of LLMs. While LLMs are significantly better than random at solving statistical classification problems, the sample efficiency of few-shot learning lags behind traditional statistical learning algorithms, especially as the dimension of the problem increases. This suggests that much of the observed few-shot performance on novel real-world datasets is due to the LLM's world knowledge. Overall, our results highlight the importance of testing whether an LLM has seen an evaluation dataset during pre-training. We release the https://github.com/interpretml/LLM-Tabular-Memorization-Checker Python package to test LLMs for memorization of tabular datasets.

Published as a conference paper at COLM 2024
Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models
Sebastian Bordt
University of Tübingen, Tübingen AI Center sebastian. bordttuni-tuebingenn
Harsha Nori & Vanessa Rodrigues & Besmira Nushi & Rich Caruana Microsoft Research
{hnori, vanessa. rodr rodrigues, besmi ra. nushi,
Abstract
While many have shown how Large Language Models (LLMs) can be applied to diverse set of tasks, the critical issues of data contamination and memorization are often glossed over. In this work, we address this concern for tabular data. Specifically, we introduce a variety of different techniques to assess whether language model has seen tabular dataset during training. This investigation reveals that LLMs have memorized many popular tabular datasets verbatim. We then compare the few-shot learning performance of LLMs on datasets that were seen during training to the performance on datasets released after training. We find that LLMs perform better on datasets seen during training, indicating that memorization leads overfitting. At the same time, LLMs show non-trivial performance on novel datasets and are surprisingly robust to data transformations. We then investigate the in-context statistical learning abilities of LLMs. While LLMs are significantly better than random at solving statistical classification problems, the sample efficiency of few-shot learning lags behind traditional statistical learning algorithms, especially as the dimension of the problem increases. This suggests that much of the observed few-shot performance novel real-world datasets is due to the LLM's world knowledge. Overall, our results highlight the importance of testing whether an LLM has seen evaluation dataset during pre-training. We release the tabmemcheck Python package to test LLMs for memorization of tabular datasets.
1 Introduction
Large Language Models (LLMs) exhibit remarkable performance on a diverse set of tasks (Wei et al., 2022; Bubeck al., 2023; Liang al., 2023). While their prowess in natural language is undeniable, performance in other applications remains an ongoing research topic (Dziri al., 2023a; Nori al., 2023). Recently, LLMs have increasingly been employed for diverse data modalities. This includes an increasing amount of creative applications in domains such as structured learning, tabular data, and time-series forecasting (Yin al., 2023; Chen et al., 2023; Hegselmann et al., 2023; Jin et al., 2024).
main question in current research on L.MSs is the degree to which these models are able to extrapolate to novel tasks that are unlike what they have seen during training (Wu et al., 2023). As such, an important aspect LLM evaluation is to know if task has been part of the model's training set. In this work, we refer to the case where an LLM is evaluated on a dataset seen during (pre-)training as training data contamination (Magar & Schwartz, 2022; Jiang et al., 2024). Unfortunately, detecting whether LLM has dataset during training is rather challenging (Duan et al., 2024). This is especially true for models to which we might only have query access via API.
certain text or
1

---

Published as a conference paper at COLM 2024
GPT--355
GPT-4
Baseline
Time Series
2020
2022
2020
2022
2022
U.S. Dollars to Yuan
0.09% - 0.20%
0.17%
0.30%
0.13%
0.32%
0.24%
U.S. Dollars to Euro
0.12% -..%5-0..00 0.43%
0.18%
0.58%
0.14%
0.59%
0.44%
NASDAQ
0.36% 0.98%
0.65%
1.35%
0.07%
1.30%
1.20%
MSCI World
0.29% 0.61%
0.82%
1.10%
0.30%
1.03%
0.92%
Netflix
0.96% 1.54%
0.68%
1.58%
0.25%
1.60%
1.47%
Table 1: Forecasting financial time series with GPT-3.55 and GPT-4 exhibits remarkable performance differences between years prior to and after 2021 (the cutoff of the training data). We perform few-shot learning, providing the model with the value of the time series on the previous 20 days and asking it to predict the value on the next day. The table depicts the robust mean relative error of the prediction across different years. The second column depicts the minimum and maximum error across the ten years from 2010 to 2019. The baseline predicts the value of the past day for the current day. Details in Supplement C.
At the same time, literature on memorization in LLMs has shown that language models can be prompted to repeat chunks of their training data verbatim (Carlini al., 2019; 2021; Chang et al., 2023; Nasr et al., 2023). Research has also shown that memorization occurs if an LLM sees a text repeatedly during training (Carlini et al., 2022b; Biderman et al., 2023). Because of this, memorization can be seen as an extreme case of training data contamination where a dataset is not only seen during training but repeated within the training set so often that the LLM becomes able to consistently generate it.
In this paper, we target the issue of training data contamination when evaluating the few- shot learning performance of LLMs on tasks with tabular data an aspect often neglected in the rapidly growing literature on applications in this domain (Dinh et al., 2022; Borisov et al., 2023; Narayan al., 2022; Vos al., 2022; Hegselmann et al., 2023; Wang et al., 2023; McMaster et al., 2023). Our first contribution is to develop various methods to detect whether an LLM has seen tabular dataset during training. This includes four different tests for memorization. Our investigation reveals that GPT-3.5 and GPT-4 have memorized many popular tabular datasets verbatim (OpenAT, 2023). For example, GPT-4 can consistently generate the entire Tris and Wine datasets from the UCI machine learning repository. What more, the memorization of tabular datasets in GPT-series models is robust phenomenon that does not depend on the precise model version.
To gauge the effect of training data contamination on evaluations of state-of-the-art LLMs, we compare the few-shot learning performance of GPT-3.5 and GPT-4 on datasets that were seen during training to datasets that were released after training (Brown et al., 2020). 1 In our experimental design, we also vary the format in which the data is presented to the LLM. In particular, we add small amounts of noise to numerical values in the dataset. The idea is that contamination is more likely to have an effect if the LLM can "recognize" a datapoint from pre-training.
We find that LLMs perform better on datasets seen during training, indicating that memo- rization leads to overfitting. In addition, we find that adding small amounts of noise and other re-formatting techniques leads to an average accuracy drop of 6 percentage points on the memorized dafasets. In contrast, the same transformations do not affect the few-shot learning performance on unseen data. To see the significant effect that training data contamination can have on few-shot learning, consider also Table 1. It shows that GPT-3.55 and GPTT4 ex- hibit remarkable performance differences when predicting time series data for time periods prior to and after the LLM's training data cutoff.
While we find significant evidence of training data contamination, also find that GPT-3.5 and GPT-4 perform reasonably well on novel datasets. To better understand the LLMs' few- shot learning performance the novel datasets, we conduct ablation studies and investigate
We do not know the training data of the LLMs. We use the term "seen during training" when there is evidence of verbatim memorization of at least part of dataset.
2

---

Published as a conference paper at COLM 2024
their performance as in-context statistical predictors (Garg et al., 2022). In particular, remove the feature names and standardize the data to zero mean and constant variance. We find that GPT-3.5 and GPT-4 can still perform in-context statistical classification better than random but struggle as the dimension of the problem increases. This leads us to conclude that the few-shot learning performance on novel tabular datasets is due to the LLM's world knowledge. Interestingly, we also find that the performance of in-context statistical learning with GPT-4 scales in the number of few-shot examples, whereas it remains more flat for GPT-3.5.
The paper is organized as follows. We first present our memorization tests. We then study the few-shot learning performance on memorized and novel tabular datasets. We then study the ability of LLMs to act as statistical predictors during few-shot learning. Finally, we show that LLMs can draw random samples from datasets they have seen during training.
2 Datasets
In this study, we use two different kinds of tabular data. First, we use datasets that were freely available on the Internet before 2021. This includes the Iris, Wine Adult and Housing datasets from the UCI machine learning repository (Kelley Pace & Barry, 1997). It also includes the OpenML Diabetes dataset and the Titanic dataset from Kaggle (Smith et al., 1988). We use red color to indicate these popular datasets.
Second, we use datasets that were not freely available on the Internet before 2022. This includes the ACS Income and ACS Travel datasets (Ding et al., 2021). These two datasets were derived from the 2022 American Community Survey (ACS). Notably, the ACS Income dataset was constructed by Ding et al. (2021) to obtain dataset similar to the Adult dataset. We also use the Spaceship Ti tanic dataset, released on Kaggle in 2022, and the FICO dataset that has been privately held (the dataset is available for download after registration). We also introduce the ICU dataset, a novel small machine learning dataset where the task is to predict whether patient is being treated in intensive or intermediate care. We derived this dataset from data in the Harvard dataverse (Goad, 2018). We use blue color to indicate the novel datasets. Additional details on the datasets can be found in Supplement A.
3 Testing Language Models for Memorization of Tabular Data
There are various systematic ways to test what a language model knows about a tabular dataset. For example, we can test whether the LLM can list dataset's feature names or tell the possible values of categorical features in the data. assess whether the LLM has leameed different aspects of the data distribution. In this Section, we focus on tests for verbatim memorization (Carlini et al., 2019; 2021). Additional heuristics and details are described in Supplement D.
also possible to heuristically
3.1 Testing for memorization
We introduce four different tests to detect memorization. These tests use the fact that most tabular datasets have a canonical representation as a CSV file (which is nothing but text document that might end up in the pre-training data). Intuitively, we say that a tabular dataset is memorized if the LLM can consistently generate it. In our context of tabular datasets, this is justified by the fact that these datasets contain random variables: It is impossible to consistently reproduce the realizations of random variables unless the values of the random variables have been seen before (i.e., during pre-training).
Previous work on memorization in language models has often relied on the srufix-suffix pattern, where the model is provided with the initial part of string as a prefix and asked to provide completion (Carlini et al., 2019; 2021; 2022b). In this work, we use few-shot learning to condition chat models on the task of completing a given prefix,
poomptstrateg
2This approach has been lightly formalized in Carlini et al. (2019) and Carlini et al. (2021), where the authors use the term 'canary'.
3

---

Published as a conference paper at COLM 2024
A. Knowledge and Learning
B. Memorization
Distribution
Conditional
Distribution
Feature
Row Compl.
Feature
First Token Test
Feature Names
Feature Values
Header
Compl.
Test
Test
Test
Titanic
V/V
VIV
V/V
V/V
V/V
V
V
V
V
-/-
Adult
4/V
VI
VI
VI
VIV
X/
X
X/ X
X/
X
Diabetes
V/V
VI
V/
V V
?
V
V
X -/-
Wine
41
VI
?
V
? \
/
Iris
4/V
V
?
-/-
?
Housing
4/V V
V/
v/ V
X/ X
x x
-/-
Sp. Titanic
X/ X
X/ X
1/-
X/ X X/ X
X/ X X AXX X/
X X X X X/ X X/ X
-/-
ACS Income
X/ X
X/ XXX
-/-
-/-
ICU
X/ X
X/ X/XI
X/ X
-/-
FICO
VIV
?/?
X/
X/X
X/ -/-
ACS Travel
X/X
X/X
X
-/-
-/- X/
X/
X/
X
Table 2: GPT--355 and GPT-4 have memorized many of the popular tabular datasets. The table depicts the results of different memorization tests with GPT-3.5 and GPT-4 (depicted in the table */*). Test results are depicted in simplified form, that is, v= evidence of memorization, X= no evidence of memorization, ?= ambiguous result, and = test cannot be conducted. The detailed, quantitative test results can be found in Supplement D.
that works well for GPT-335 and GPT-4, as well as for many other LLMs. The different tests can be performed with the tabmemcheck Python package. The tests are
1. The Header Test:
We prompt the model with the initial rows the CSV file and ask it to complete the next rows verbatim. This test measures the memorization of the initial rows of the dataset.
2. The Row Completion Test: We prompt the model with number of contiguous rows from a random position of the CSV file and ask to complete the next row verbatim. This
3. The Feature Completion Test:
We prompt the model with all feature values of a random row in the dataset except for single highly unique feature and ask it to complete the unique feature value verbatim. Examples of unique features are names and features with inherently high entropy, such as measurements with many decimal places.
4. The First Token Test: a random position of the CSV file and ask complete the first token of the next row. If the rows of the CSV file are known to be random, can perform statistical test between the LLM completion accuracy and the accuracy of completion with the mode.
We prompt the model with a number of contiguous rows from
helpful to think about the proposed memorization tests in terms power and signifi- cance, would about statistical hypothesis test. All the tests are highly significant. This means they provide definitive evidence of memorization (though not necessarily of the entire dataset). Conversely, little can be said about the power of the tests. seems possible that an LLM has memorized tabular dataset, but we cannot extract it via prompting.
3.2 LLMs have memorized many of the popular tabular datasets
Table shows the results of the memorization tests on 11 tabular datasets for GPT-3.5 and GPT-4. The Table depicts the results of the tests in simplified form, where Vindicates that the model is able to generate the respective parts of the dataset. Quantitative test results and model generations can be found in Supplement D.
4

---

Published as a conference paper at COLM 2024
Titanic
Adult
Diabetes
Wine
Iris
Sp.
Titanic
FICO
OLMo 7B
?/1 X
X/ X
X/ X
X
X
?/ V
X/ X
x/ x
Gemma2 27B
V/V 1/V
N/ X
J/ X
V
V/ V
X X
X x
Llama3 70B
V/
V/ X
N/ X
V
V V
X/
X/ X
Qwen1.5 72B
V/ X
// X
VÍ X
X/
X
N/ V
X X
X/ X
Ilama3.1 405B
V V X
X/X
V V/V
?/
X/ x
Table 3: Open-source and open-weight LLMs have memorized parts of the most popular tabular datasets. The table depicts the result of the Header Test and the Row Completion Test (depicted as */*) in the simplified form used in Table 2. Detail in in Supplement D.
The header lest indicates memorization of all the tabular datasets that were publicly available on the internet before 2021. This means that GPT-3.5 and GPT-4 have memorized the initial rows of these datasets. On Ti Faniic, Diabetes, Wine and Iris, the row completion, feature completion, and first token test also provide evidence of memorization. This means that GPI-3.5 and GPT-4 have not only memorized the initial rows of these datasets (which are frequently in Jupyter notebooks) but also random rows. Of course, this means that GPT-3.5 and GPT-4 have essentially memorized the entire datasets. We also observe that GPT-4 exhibits somewhat stronger evidence for memorization than GPT-335,, which aligns with the literature that shows that larger models exhibit more memorization (Carlini et al., 2022a). Interestingly, there is evidence of memorization of random rows Adult and Housing.
On the novel datasets, there no evidence memorization for GPT-3.5 and GPT-4. This is to be expected for datasets released after the model was trained. In the case of the FICO dataset, it suggests that this dataset might have been protected from inclusion in the LLM training data by the need to register prior to access.
Table shows the results of the header lest and the completion test for five different open LLMs (Groeneveld et al., 2024; Riviere et al., 2024; Dubey et al., 2024; Qwen Team, 2024). Similar to the results observed in Table 2, there is evidence for the memorization of the most popular tabular datasets in almost all LLMs. However, with the exception of the Iris dataset, there usually only evidence for the memorization of the initial rows and not of the entire dataset.
Few-Shot Learning with LLMs and Tabular Data
In the previous Section, we demonstrated that LLMs have memorized many popular tabular datasets. In this Section, we investigate the few-shot learning performance of GPT-3.5 and GPT-4 on the novel and memorized datasets. In addition, we replicate the entire analysis in this Section with Llama 3.1 70B and Gemma 2 27B. This can be found in Supplement F.
Prompts. We build on previous works and prompt chat models with tabular data in the format "Feature Name Feature Value" (Borisov et al., 2023; Hegselmann et al., 2023).
Few-Shot Learning with Tabular Data
System: You are a classification assistant who is an expert in tabular data, data science, and cross-sectional population surveys. [..]]
User:
IF Age 30, WorkClass Private, fnlwgt 196945, [...] THEN Income Assistant: >50k
(few-shot examples]
User: Age = 28, WorkClass = Self-emp-inc, fnlwgt 79135, [...] THEN Income [Model Response]
Assistant:
30f course, it is possible that the L.I saw the dataset during training, but there was memoriza- tion, memorization tests powerful enough.

---

Published as a conference paper at COLM 2024
Original. "You help to nake predictions on the tanous Iris flowen data set.."
Task. "You help the user classify difforent spactos of the Iris Flowering plant."
sepal Jength sepal potal, Jength
petal weets
Langth Sepal Jon]
Wath Sepai (cx) Langth (on) With of Peral (en]
of Rower
0.2 Hr 06951
45
30
14
0.2
13
230
Sclove
4,07
321
127
021
Satsan
Hiveso
250
Perturbed.
predictions
Statistical. "You help the user with classification problen, prodicting the target vartable the inputs."
the famous Iris flower data set."
spp. lnggh sase vidn
pasl ligh
pesni with
speciss
5-2
34
13
50
23
13
340
130
472
437
2 422
4.73
401
4.40
5.5
4.32
146
4.00
36
13
Figure 1: Transformations of the Iris dataset. All datasets are presented to the LLM in four different formats: Original, perturbed, task, and statistical. See Section 4.1 for a description.
We ask the LLM to predict the value of the target variable, given the values of the other features in the data. We select 20 few-shot examples randomly and stratify the labels of the few-shot examples to match the label occurrence in the dataset. All experiments are conducted at temperature 0. A full example prompt is given in Supplement G.
4.1 Dataset Format: Original, Perturbed, Task and Statistical
We make of an intriguing property of tabular data: It is possible to make changes to the format of the data without (significantly) affecting the underlying classification problem (compare Figure 1). We use this fact to explore (1) the consequences of memorization in Section 4.2 and (2) whether LLMs make use of their world knowledge in Section 4.3. In all few-shot learning experiments,
present the data in of four standardized formats:
Original means that we present the data as is contained in the CSV file of the dataset. In the perturbed version, we slightly change individual digits in the data that are not relevant to the underlying classification problem. We also deface any unique identifiers, such as observation IDs and names. In the system prompt of both original and perturbed, we tell the model about the origin of the data ("You help to make predictions on the Titanic dataset from Kaggle") ) In the task version, we change the names of the features without changing their meaning ("BMI" becomes "Body mass index"), and similarly re-code categorical values ("0" becomes "False" and "United-States" becomes "USA"). We also round numerical values to two digits (unless this interferes with the meaning of variable) and provide generic system prompt ('ouu help to predict the type wine from its features.") "). In the statistical version, all numeric features standardized to zero mean and constant variance. Feature names are replaced with XI, ..., Xn and strings encoded as categorical variables.
An important aspect of our datasets transforms is that they are standardized and comparable across datasets. Additional details on how we standardized the transformations are in Supplement E.
4.2 Memorization leads to inflated performance estimates
Table 4 depicts the few-shot learning performance of GPT-4 and GPT-3.5 on 10 different tabular datasets. On the memorized datasets depicted in Panel A of Table 4, the performance is quite impressive. In particular, GPT-4 outperforms logistic regression on 3 out of 5 datasets when prompted with the original data. However, the predictive performance of GPT-4 and GPT-3.5 drops in Figures 11a and 11c, the accuracy of GPT-4 and GPT-3.5 on the memorized datasets consistently drops present the data in the original, perturbed, task and statistical
move from the original data to the perturbed data. As depicted

---

Published as a conference paper at COLM 2024
Panel A
Titanic
Adult
Diabetes
Wine
Iris
Original
0.81 m
g
0.78
0.81
0.74.cc
0.74,z2
0.88 sz
0l
0,98
0.99
Perturbed
0.78m
0.82,1
0.78m
0.81 m
0.73
0.73
0.88
0.95mc
0.95
0.95.c
Task
0.77 m
gr
0.75.
p.
0.70.tc
0.73
0.87 as
0.87 m
0.95
0,95.c
Statistical
0.61
0.65
0.70.m
0.63
0.68.s2
0.62
0.86
0.90 m
0.87 n3
0.92
LR / GBT
0.79 / 0.80
0.86
0.87
0.78
0.75
0.98 / 0.96
0.97 / 0.95
Panel B
s. Titanic
ACS Income
ICU
FICO
ACS Travel
Original
0.58
0.67
0.78
0.78
0.69.s 05 0.69 m
0.58
0.67 m
0.54
0.62,c
Perturbed
0.57
0.67 a
0.77 m
0.78
0.69.s m 0.71 m
0.58 mz
0.67 m
0.54
0.62
Task
m.
0.
0.77.
0.77
0.67.z. :6
0.7l
0.61
0.68m
0.54
0.65m
Statistical
0.
0.66,a
0.59,
0.57 A2
0.56gs
0.55
0.60
0.59
0.52,z
0,
LR GBT
0.78 /
0.78
0.80
0.80
0.76
0.66
0.70
0.69
0.64 / 0.67
Table 4: Few-shot learning performance of GPT--3.5 and GPT-4 across different tabular datasets. The table depicts the predictive accuracy and standard error of the LLMs on 10 different datasets. The table depicts the results with GPT-3.5 and GPT-4, separated by space in the same column. Each dataset is presented to the LLM in four different formats: original, perturbed, task and statistical (compare Figure 1). Panel A (top) depicts results on datasets that the LLM has memorized. Panel (bottom) depicts results on novel datasets where there is no evidence of memorization. The table also depicts the predictive accuracy of logistic regression (LR) and a gradient-booster tree (GBT).
format. In particular, GPT-4 on the task data outperforms logistic regression only on 1 out of 5 datasets, and even there, the measured difference lies within the standard error.
Panel B of Table 4 depicts the few-shot learning performance of GPT-4 and GPT-3.55 on novel datasets. In contrast to the results in Paael A, GITT-4 in the original format does not outperform logistic regression on any dataset. Moreover, we also don't observe significant changes between the original, perturbed, and task formats. This is again depicted in Figures 11b and 11d. While it is always possible that the few-shot learning performance of an LLM varies with subtleties such as the length of feature name or the number of significant digits of number, the results in Panel B of Table 4 indicate that GPT-4 and GPT-3355 are fairly robust to such changes.
The striking difference in performance between the different dataset formats on the mem- orized datasets, which is completely absent on the novel datasets, strongly suggests that memorization leads to invalid performance estimates of few-shot learning. Even though the datasets in Panel and Panel B of Table 4 vary along other dimensions than being memo- rized novel, it seems highly unlikely that these give rise to the observed performance differences with logistic regression or the observed performance drops between the different dataset versions.
We note that the GPT-4 results in Table 4 were obtained with has seen data up December 2023. On the memorization tests, gpt-4-0122-prreiee obtains the same results as the GPT-4 models with a training data cutoff in 2021.4
which
4.3 Performance depends feature names and variable format
While the results in Panel A of Table 4 indicate that few-shot learning datasets seen during training can suffer from overfitting, Panel B demonstrates that GPT-3.5 and GPT-4 perform reasonably well on novel data.
*Producing the results in Table 4 with -pt-4-32k-0613 would have been prohibitively expensive. We conducted limited experiments with npt-4-32k-0613 3 on the original data format where the few- shot learning performance matched got-4-8125-previeew
7

---

Published as a conference paper at COLM 2024
0.05
0.00
Accuracy
-0.10
-0.15
Original
Perturbed
Task
Statistical
Original
Perturbed
Task
Statistical
Original
Perturbed
Task
Statistkal
Original
Perturted
Task
Statistical
(a) GPT-4 memorized
(b) GPT-4 novel
(c) GPT-3.5 memorized
(d) GPT-3.5 novel
Figure 2: Few-shot learning performance of GPT-4 and GPT-3.55 across memorized and novel datasets. The Figure depicts the absolute performance difference between the LLM and logistic regression, that is, value of 0 means that the LLM has the same accuracy logistic regression. The Figure depicts the average accuracy across the different datasets. Parentheses indicate standard errors. This Figure summarizes the results in Table 4.
Moreover, on all datasets except Spaceship Titanic, the performance is consistent across the original, perturbed, and task formats but significantly drops for the statistical format. This indicates that the LLMs rely on the natural scale the feature values and the variable names, which are made unrecognizable in the statistical format. We suggest that this effect not observed on Spaceship Titanic because this dataset is synthetically generated and about fictional event with fictional variable names.
This effect also depicted in Figure 2, where there is significant drop in accuracy from the task to statistical setting in all subfigures.
4.4 Language models act as statistical predictors, but lag behind traditional algorithms
In the previous section, we have seen significant performance drop between task and statistical, indicating that the few-shot learning performance of LLMs significantly relies on the LLM's world knowledge (Yu et al., 2023). In this section, we study the ability of language models to act as in-context statistical learners. This relates works using small, specifically trained language models to show that in-context learning can learn simple function classes, akin to supervised learning (Garg et al., 2022; Von Oswald et al., 2023). We investigate the question in the context of GPT--3.5 and GPT-4, similarly to Bhattamishra et al. (2024).
Binary Statistical Classification. We study a simple statistical learning problem with numerical features X; ~ (0, E Rd, an unknown coefficient vector z ~ (0, /) E Rd, and binary labels y/ = (xF >= 0). The data is presented to the model in the statistical
format (see Figure 1).
Few-shot learning performance degrades in the dimension of the problem. Figure 3a depicts the few-shot learning performance on our binary statistical classification problem for a fixed number of 20 few-shot examples as the dimension of the problem increases. For 2, the performance of GPT-3.5 and GPT-4 roughly on par with the 1-Nearest Neighbor classifier and logistic regression. As the dimension of the problem increases, the few-shot learning performance of the LLMs deteriorates, and more quickly so than for the traditional statistical learning algorithms.
Few-shot learning performance of GPT-4 scales the number few-shot examples. Figure 3b depicts the few-shot learning performance our binary statistical classification problem for the fixed dimension of and an increasing number of few-shot examples. Whereas GPT-3.5 scales only very weakly in the number of few-shot examples, the perfor- mance of GPT-4 increases monotonically. For 200 fhw-shot examples, GPT-4 is roughly on par with the 1-Nearest Neighbor classifier, but remains less efficient than logistic regression.
Fine-tuning scales in the number of few-shot examples. interesting to compare the scaling in the number of few-shot examples for in-context learning versus fine-tuning.
8

---

Published as a conference paper at COLM 2024
Scaling in the Dimension (20 Sew-Shot Examples]
Scaling the Number of Few-Shot Examples (d=B)
Accuraci
0.6
0.5
0.5
16
20
50
100
200
Dimension
Number of Few-Shot Examples
(a) Scaling in the dimension d
(b) Scaling in the number of few-shot examples
GPT-3.5
GPT-4
Logistic Regression
1-Nearest Neighbor
TabLLM (Fine-Tuning)
Figure 3: Few-shot learning performance of GPT-3.5, GPT-4, TabLLM (fine-tuning a language model with 11B parameters), Logistic Regression and a 1-Nearest Neighbor classifier across binary classification problems with linear decision boundary. Figure (a) depicts the scaling of the few-shot learning performance in the dimension of the problem (that is, the number of features). We use 20 few-shot examples across all dimensions. Figure (b) depicts the scaling of the few-shot learning performance in the number of few-shot examples (respectively, the size of the training set). We use fixed dimension of Mean and 95% confidence intervals.
Figure 3b depicts the performance of the fine-tuning technique TabLLM by Hegselmann et al. (2023) on our binary statistical classification problem. We see that the performance of the fine-tuning approach scales in the number of samples.
5 Drawing Random Samples from Datasets seen during Training
In the previous Section, we explored the implications of memorization on few-shot learning with tabular data. However, predicting the target label is only one of many tasks that one might want to perform with LLMs and tabular data (Hollmann et al., 2024; Bordt et al., 2024; Sui et al., 2024). In this section, we demonstrate another implication of seeing dataset during training: We show that L.MMs can draw random samples from these datasets without any fine-tuning (Borisov et al., 2023).
Figure 4 depicts the longitude and latitude of random samples on the California Housing dataset. To draw these samples, we conditioned GPT-3.5 with samples from other datasets,
Figure GPT-3.5 can draw random saFiguremple from the California Housing dataset. We only provide the model with the name of the dataset and the feature names. The diversity of the generated samples depends on the temperature parameter. For small temperatures, the samples are concentrated around the mode of data. As temperature increases, the samples become more diverse and similar to the data distribution. At large temperatures, some samples lie outside the support of the data distribution. The reader might want to compare with Figure in Borisov et al. (2023).
43
(a) Temperature 0.2
Temperature 0.7
(c) Temperature 1.2
(d) Dataset

---

Published as a conference paper at COLM 2024
revealing no information about the feature values on the California Housing dataset. Inter- estingly, the samples drawn by the LLM follow the overall summary statistics in the data. For example, the feature correlations of the samples match the feature correlation in the original dataset (Supplement Figure 16). At the same time, the samples are not copied from the training data (Supplement Table 11). In this sense, it can be said that GPT-3.5 generates novel random samples from the dataset.
6 Related Work
Recent works have demonstrated the capabilities of LLMs tasks with tabular data (Dinh et al., 2022; Narayan et al., 2022; Vos et al., 2022; Wang et al., 2023; McMaster et al., 2023). In particular, Borisov et al. (2023) and Hegselmann et al. (2023) have shown that LLMs can be fine-tuned to generate and classify tabular data. The phenomenon of in-context learning has attracted much research interest (Brown et al., 2020; Wei et al., 2023; Li et al., 2023). It has been shown that the forward pass of transformer can mimic gradient descent (Von Oswald et al., 2023) and that LLMs can learn simple function classes in-context (Garg et al., 2022; Bhattamishra et al., 2024).
Data or task contamination is studied in many LLM reports and research papers, often adopting an n-gram-based definition (OpenAI, 2023; Nori et al., 2023; Touvron et al., 2023). Jiang et al. (2024) investigate data contamination in GPT-2 by adding benchmark datasets to the pre-training data. Yang et al. (2023) study the effects of rephrasing widely used benchmarks. Li & Flanigan (2024) analyze how the few-shot learning performance of LLMs varies between datasets released before and after the training data was collected.
Membership inference attacks are set of methods to detect if a model has seen a text during training (Mahloujifar et al., 2021; Choquette-Choo et al., 2021; Carlini et al., 2022a; Mireshghallah et al., 2022; Shi et al., 2024; Mattern et al., 2023). Recently, Duan et al. (2024) question the value of membership inference attacks for large language models. Shi et al. (2024) introduce the Min-K% Prob method to detect pre-training data from LLMs.
Research memorization has shown that LLMs can be prompted to repeat chunks of their training data verbatim (Carlini et al., 2019; 2021; Chang et al., 2023; Nasr et al., 2023). Memorization is linked to data-duplication in the pre-training data (Carlini et al., 2022b; Biderman et al., 2023). Memorization does not necessarily lead to overfitting (Magar & Schwartz, 2022).
LLMs tend to perform better on tasks more likely to occur in the training data (Wu et al., 2023; McCoy et al., 2023; Dziri et al., 2023b). Our work complements this emerging literature by (1) rigorously demonstrating that GPT-3.5 and GPT-4 have seen certain tasks frequently during training (whereas previous works have relied on heuristics) and (2) showing that LLMs overfit on uncommon tasks that were seen frequently during training.
7 Discussion
In this work, we have first shown that LLMs have memorized many of the popular tabular datasets. We have then used this fact to gauge the amount of overfitting that occurs during in-context learning with such datasets. We find strong evidence of overfitting in in-context learning due to memorization. We have also studied other factors that determine the few- shot learning performance of GPT-3.5 and GPT-4 prediction tasks with tabular data, finding that these models rely on their world knowledge, but also have the ability to act as statistical predictors.
Our study has several limitations, such as that we usually don't know if and in what way the LLMS saw the different tabular datasets during training. Also, would have been more ideal had we been able to insert datasets into the training corpus at random.
Finally, we want to highlight that had we performed our evaluations without carefully controlling for memorization, we might have erroneously concluded that few-shot learning with GPT-4 frequently outperforms traditional statistical learning algorithms.
10

---

Published as a conference paper at COLM 2024
8
Ethics Statement
All the datasets we use in this study are explicitly available for scientific research. Because our study concerns the scientific question of invalid performance evaluations of LLMs, we don't believe it raises significant ethical concerns. While we consider memorization purely from the perspective of training data contamination, it has broader implications, including copyright and privacy.
Reproducibility Statment
We conducted initial experiments with different versions of GPT-3.5 and GPT-4 and found that the results are fairly robust towards the precise model version. This holds true for both the results of the memorization tests in Table and for the few-shot learning results in Table 1106 that performs worse on the few-shot
An exception is the model 3pt-3.5-turdee learning tasks than other versions of GPT-3.55. The models that used to run the final experiments are detailed in Supplement B. The cost of replicating all the results in this paper with the Open AI API approximately 1000 USD. By far, the most expensive experiments are the few-shot learning experiments with GPT-4 (they require approximately 1000 queries per data point, sometimes with relatively long context). In contrast, the memorization tests require relatively few queries.
Code to replicate the results in this paper is available at https: //gi thub. com/ com/interpretm// LLM- TTauularMeeoriine ietoorizztion--o Checker... This includes, in particular, the different memoriza- tion tests and the dataset transformations.
Acknowledgments
Sebastian is supported by the Tübingen AI Center and the German Research Foundation through the Cluster of Excellence "Machine Learning New Perspectives for Science" (EXC 2064/1 number 390727645). This project was started while Sebastian was an intern at Microsoft Research. The authors would like to thank Varun Chandrasekaran for helpful discussion.
References
Satwik Bhattamishra, Arkil Patel, Phil Blunsom, and Varun Kanade. Understanding in- context learning in transformers and Iims by learning to learn discrete functions. Interna- tional Conference on Learning Representations (ICLR), 2024.
Stella Biderman, USVSN PRASHANTH, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. In Advances in Neural Information Processing Systems (Neurips), 2023.
Sebastian Bordt, Ben Lengerich, Harsha Nori, and Rich Caruana. Data science with 1lms and interpretable models. XA14Sci Workshop AAAI-24, 2024.
Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. In International Conference on Learning Representations (ICLR), 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (Neurips), 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Farly experiments with gpt-4. Xiv preprint ar Xxx::22333.12722 2023.
11

---

Published as a conference paper at COLM 2024
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19), 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), 2021.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022a.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arv preprint arXiv:2202.07646, 2022b.
Kent K Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. Speak, memory: An archaeology of books known to chatgpt/ gpt-4, arviv preprint ar XXix22305.00115 2023.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 2016.
Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, al. Exploring the potential of large language models (Ilm) in learning on graphs. arviv preprint ar Xiv:2307 003303.....,.... 2023.
Christopher A Choquette-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. In International conference on machine learning (ICML), 2021.
Frances Ding, Moritz Hardt, Miller, and Ludwig Schmidtt. Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems (Neurips), 2021.
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: Language-interfacee fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems (Neurips), 2022.
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do mem- bership inference attacks work on large language models? arXiv preprint ar XrX1:22402.07841, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. Xiv preprint arXiv:2407.21788 211783 2024.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D Hwang, et al. Faith and fate: Limits of transformers on compositionality. ar aio preprint ar NKK::2335.18644 2023a.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang (Lorraine) Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena Hwang, Soumya Sanyal, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers compositionality. In Advances in Neural Information Processing Systems (Neurips), 2023b.
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? case study of simple function classes. Advances in Neural Information Processing Systems (Neurips), 2022.
Nathan Goad. Diabetic Ketoacidosis and Hyperchloremia Full Dataset, 2018 URL https: r/doi.or//11 10...99100000 /
12

---

Published as a conference paper at COLM 2024
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, lan Magmusson, Yizhong Wang, et al. Olmo: Acceler- ating the science of language models. arXiv preprint arXio:2402.00838, 2024.
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics. PMLR, 2023.
Noah S Müller, and Frank Hutter. Large language models for automated data science: Introducing caafe for context-aware automated feature engineering. Advances in Neural Information Processing Systems (Neurips), 2024.
Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. Investigating data contamination for pre-training language models. Xiviv preprint ar Xix:2240
2024.
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, al. Time-Ilm: Time series forecasting by reprogramming large language models. ICLM, 2024.
R. Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 1997.
Changmao Li and Jeffrey Flanigan. Task contamination: Language models may not be few-shot anymore. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Trans- formers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Ya- sunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan An- drew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023.
Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. arXiv preprint arXf::2003.08242 2022.
Saeed Mahloujifar, Huseyin A Inan, Melissa Chase, Esha Ghosh, and Marcello Hasegawa. Membership inference on word embedding and beyond. ar preprint ar arXiv:2106.11384
2021.
Justus Mattern, Fatemehsadat Mireshghallah,
Maireshghallah,hhjiii Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor perg-Kirkpatrick Membership inference attacks against language models via neighbourhood comparison. arXiv preprint ar xxXio22055.84622 2023.
R Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve. Xiv preprint ar Xii::2309..16388 2023.
Christopher McMaster, David FI. Liew, and Douglas EV Pires. models for solving tabular prediction problems in the electronic health record. arviv preprint ar Xi::2303..14900 2023.
Adapting pretrained language
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrics and Reza Shokri. Quantifying privacy risks of masked language models using membership inference attacks. arXiv preprint arXit:2203.03929, 2022.
13

---

Published as a conference paper at COLM 2024
Avanika Narayan, Ines Chami, Laurel Orr, and Christopher Ré. Can foundation models wrangle your data? arXio preprint or arXi::225.00911, 2022.
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramer, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprin! arXir:2311.17035, 2023.
Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of gpt-4 on medical challenge problems. arXiv preprint ar arXio:2303.133 133755 2023.
OpenAl. GPT-4 Technical Report. arxiv preprint ar arXfo:2303.08774, 2023.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, al. Scikit-learn: Machine learning in python. Journal of machine Learning research (JMLR), 2011.
Qwen Team. Introducing Qwen1.5, February 2024. URL https://qwenllm gi github.io/blog! qwen1.5/.
Morgane Riviere, Shreya Pathak, Pier Ciuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et Gemma Improving open language models at a practical size. arXiv preprint ar arXiv:2408.000 00110,, 2024.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. International Conference on Learning Representations (ICLR), 2024.
Jack W Smith, James E Everhart, WC Dickson, William C Knowler, and Robert Scott Jo- hannes. Using the adap learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the annual symposium on computer application in medical care. American Medical Informatics Association, 1988.
Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. Table meets Ilm: Can large language models understand structured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ar preprint ar Xiix:2307.09288, 2023.
Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 2014.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International conference on machine learning (ICML), 2023.
David Vos, Till Döhmen, and Sebastian Schelter. Towards efffiene automation of data wrangling tasks with prefix-tuning In NeurIPS 2022 First Table Representation Workshop, 2022.
Zifeng Wang, Chufan Gao, Cao Xiao, and Jimeng Sun. Anypredict: Foundation model for tabular prediction. arXiv preprint Xio:22005.12081, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (Neurips), 2022
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXit preprint ar 2303:2303003846 2023.
14

---

Published as a conference paper at COLM 2024
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabili- ties and limitations of language models through counterfactual tasks. arXiv preprint ar Xxit:2307.02477, 2023.
Shuo Yang, Wei Chiang, Lianmin Zheng, Joseph E Gonzalez, and lon Stoica. Rethinking benchmark and contamination for language models with rephrased samples. arXiv preprint arXie:2311.04850, 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. ar preprint ar Xr1::3330. 13549, 2023.
Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, et al. Kola: Carefully benchmarking world knowledge of large language models. NeurlPS Datasets and Benchmarks Track, 2023.
15

---

Published as a conference paper at COLM 2024
A
Datasets
Iris. The Iris flower dataset is small dataset (150 observations) that is freely available on the Internet, among others in the UCT repository https://archive. wcwp:ei.ssscccie.i.. wwt.:/iiiiuuaaaaaaaee dataset/ 53/iris.
Wine. The Wine dataset is another small dataset from the UCI repository (178 observa- tions). It is available at https://archiwe itcccccciiiiiii....i......ieiiee uci woci..ed/mmm/mm// http:///aaahiieeeeeinnnn. lecinngdattbbann databases/ wine/wine, data.
The Kaggle Titanic dataset is popular and freely available machine learning dataset. It is available at https://ww.. kaggle cww/ccc/////ooc com//ompetiio coompetitioos/titaaii
Kaggle Titanic.
OpenML Diabetes. The OpenML Diabetes dataset (Smith et al., 1988) is popular dataset that is freely available tps
www.pppeww......erreeeeeeeepeeee /ttp///riittttpttttttttttrrreea 378status=active part of OpenML (Vanschoren et al., 2014).
Adult Income. Historically, the Adult Income dataset is one the most popular machine learning datasets http: /www. cs. toronto. edu/~delve weweeeddaeaaaeeeeeeeeeeeeee htt///ta//aa/aa aaaataaaultt tadu teeaalleeta..... html. Re- cently, researchers have argued that this dataset should no longer be used Ding et al. (2021). The csv file of the dataset can still be found at Kaggle setps://www htt.//www.llllec.ee//// www.kaggle..com/caaastttttt com/datasets/ wenruliu/ adult-income
t- income- aemoom--aate dataset and at many other places.
California Housing. The California Housing dataset (Kelley Pace & Barry, 1997) is freely available and very popular machine learning dataset with 20640 observations.
FICO. The FICO HELIOCv1 dataset was part of the FICO Explainable Machine Learning Challenge https: / /ommumi ty. fico.com// com/s/ http///explainabbiie wetelle-mahiieeeeeee leaieeee-ennnnn---eeeee baarning-challeenn challenge. This dataset can be obtained only after signing data usage agreement and is then available via Google Drive. This is an example of dataset that is freely available but where the CSV file has not been publicly released on the internet, at least not by the creators of the dataset.
Spaceship Titanic. The Spaceship Titanic dataset was released on Kaggle in 2022 https: //www kaggle. cwm/c/ httpm///ppacsshpppppae ti tanic. It is a synthetically generated dataset.
ACSIncome and ACSTravel. We created these two datasets using the folktables package and Census Data from the 2022 American Community Survey (ACS). The construction of the datasets is described in (Ding et al., 2021).
ICU, We created the UCI dataset as an example of a novel tabular dataset with few observations and features, similar to some of the traditional machine learning datasets. harvard.edu/daaaas.. edu/dataset. xhtml? We created the dataset from the data The dataset has 10 features (patient characteristics) persistentId=doi: 10. 7910 1991/DDN/PX9K0 and binary target. It has 102 observations.
B Models
Here provide the details of the models that were used to generate the results in the paper.
Table 3... 5-turbb-000 and GPT-4-32K-0315 deployed on Azure.
Table 3ot-3.5-turbo-16kk
and gpt-4-0613 deployed on Azure, with the ex- ception of gpt-3. 5-turbo-0125 and gpt-4-0125-previee with the OpenAl API for the ACS Income, ACS Travel and ICU datasets. We also ran the memorization tests with 16k-0613 and gpt-4-9125-previer with the OpenAl API and obtained sim- ilar results. The main difference is that gpt-4-01 1aa--011---peieeee knows the Spaceship Titanic dataset's feature names and feature values (but there is no evidence of memorization from our tests).
Table 3: gpt-3. St-ttrooo-eee 0125 and 40t-4-0125-previee with the OpenAl API.
Figure 3: gpt-3.5 5- turbo-urbrooo 000o0-0122 25 with the OpenAl API and gpt-4-0125 1w:0012--peiiiiee deployed on Azure.
16

---

Published as a conference paper at COLM 2024
Figure 4:
deployed on Azure.
Logistic Regression. We train logistic regression using scikit-learn (Pedregosa et al., 2011). We cross-validate the L2 regularization constant.
Gradient Boosted Tree. We train gradient boosted trees using xgboost (Chen & Guestrin, 2016). We cross-validate the max nax.depth of the trees and the L1 and L2 regularization constants, similar to (Hegselmann et al., 2023).
C Time Series Experiments
For each year, we compute the robust mean relative error as
T
19: yi y
Σ
min
0..02..
(1)
T 20
1 20
We use a simple system prompt.
System: "You predict the development of the MSCI World stock index in 2022. You are given the price level of the stock index the previous 20 days and predict the price of the stock index on the current day."
expert financial market analyst and stock trader and your task to
D Memorization Tests
This Section provides additional details on the memorization tests and their quantitative results. We also detail a number measures that assess what the LLM knows about the dataset (for example, the feature names).
With regard to the interpretation of the memorization tests, it is important to note that the quantitative test results (e-g., how many rows the LLM managed to complete in the row completion test) are not sufficient to judge memorization. Instead, one has to consider the completion rate of the LLM and the amount of entropy in the dataset. For example, the OpenML Diabetes dataset contains an individual's glucose level, blood pressure, and BMI, as well as other measurements that are at least in part random and highly unique. If an LLM can consistently generate even few rows of this dataset, it is strong evidence of memorization. To give another example, the Iris dataset contains many rows that are near-duplicates. This means that an LLM might also achieve a non-zero row completion rate by chance or prediction. Because of this complication, we manually judge the results in Table 2 and Table 3 in the main paper, considering the quantitative results of the different tests and the structure of the dataset.
As general observation, one might expect the results of the tests to be somewhat ambiguous. turns out, however, that GPT-3.5 and GPT-4 (as well as other LLMs) have the remarkable property of regurgitating quite long and complex strings from their training data. For example, GPT-4 is able to generate the entire Iris dataset, consisting of 151 rows, given only the first couple of rows as prompt. Thus, while there are few cases of ambiguity (marked as question marks the respective tables), the evidence of memorization usually fairly clear (and conversely, there are cases where the model gets the dataset's content completely wrong).
All memorization tests are conducted at temperature 0.
D.1 Qualitative Examples
Figures 5-10 the following pages illustrate the different memorization tests. They depict the Leveshtein string distance between the model response and the actual content the dataset. For the different tests, depict two cases: one with evidence of memorization and one without.
17

---

Published as a conference paper at COLM 2024
heacer_ proppt header completion, response Tabesschackk, shaeeschackkhhader..... Test; "da "daaaases/ wttp://///auttttttaaaaaaaa http///ddtttttrt..coo csv", pr122speee
V In 45.s
Reader
IniiugtEE.cattion Educationiuur Naritalstatus, Occusation, Relationship.. Cesionnshp..arc..... Gender, Capstaldiinn CopttalLoss,
stare-gnv.
self-eap not 82311, Bachelors, Parried spouse, managerial, Husband, white, 2ale,0.8,13, United States, sadek
38, P-ivate, 215646, HS HS-grao,9, 9, Divorced, Handiers- cleaners, lot-in-fontly, White, Hale,0,8, 8,4,6 Snited-States, <-SaK
55, 23472,, 11th,2, Harried- spous,, handlers eerierr--eeeerrs cleaners, hussand, Black, Pale, 0,0,0,000 26, Private, 358469, Bachetors, 13, Horried civ spuss, Prof specialty, Mife, Black, Fosate, 48, cuba, veaac United-States, c-S0K slack, renale, 52, self-esp-rod not 289542, Ha-grad,9, 9d,9, Married spouse, Exec nec-renageriat, Husband, white. Male,0,0.43 United States, 550K
unitee united-states, <eS0K
P-ivate, 254502, Pasters, Perried
Exec menageras service,,,,,, mut-in
ressle,
31, Fiioste,07701, Mesters,1d, 14, never-married, Prof-spest Prof-specielty, Not-in-fonily, Mhite, reasle. 14004, 0,50, Urited-Stetes, 250K
42, Private,150sed Bachetors, 13, Harried-cly civ sissssss Exec nanggaan,,ra,, Husbard, Weite, Hate,5178,0,40 8,,4, Urited-Staces, H50K
Header Test Legend: Pronpt Correct Incorrect Nissing
Figure Header Test on Adult Income. The LLM prompted with the first couple of rows of the dataset (black text) and responds with the next 7 rows of the dataset, exactly as they appear in the CSV file of the dataset (green text). The text color illustrates the Levenshtein string distance between the text in the CSV file and the model response. An entirely green row means that the model responded with the next row exactly as it occurs in the CSV file. Red color indicates that the model made mistake, and violet color indicates that the model missed a digit. In this particular example, the model response is all green because it is equal to the content of the CSV file. Best viewed in digital format.
rows, responses
tabnencheck. rww.ecckk.ooooooooooooooooooo httppicchccccoooooooleettttoooeo test (irrriss.cs'', 'gpt-4-0125-previer nun (un_queiiss=25)
Info: 1.99% the rows this dataset are duplicates. 5,3.5,1.3,0. 3,
2.13:22.........rriveeercc....
Irisversicolor
0,,.8,,,,,033,riis--.. 1.1.3.1.5..22.4111iivviig
6.3.2.8....111.5.1riivvi.....
7.3.2.9....3.....rriivvvg...
6,2.2.5,1.5,rris-virn
1.12.66.5.6114.Irii..iii
1,8.3.4,1.900.2,Iiiisssttt
1.5.222,5...22331riivvvin.....
633.3.3,4.7.2.6,1111.-.ee
5.9.3.4.2.155.rri..veees1.
1,,3,2,,13,..22,I1is55..0
1,2,411,1.,,00111I1iss5s5
6.7.3.5...7.Iris.versioo.
5.7.4.4,115,0.44Iriis::ttt
6,2.5.1...0.6.Iriiststtt.
7.1.3.5.9.2...1rissvvrinn..
Row Completion Test: 22/25 exact matches. Legend:
Prompt Correct Incorrect Missing
Figure 6: Row Completion Test on Iris. The LLM is prompted with the previous of the dataset and responds with the next row. The figure depicts the Levenshtein string distance between the model responses and the actual next rows in the CSV file. An entirely green row means that the model responded with the next row exactly as it occurs in the CSV file. Red color indicates that the model made a mistake, and violet color indicates that the model missed a digit. Best viewed in digital format.
18

---

Published as a conference paper at COLM 2024
Heeder texts age, Clas of verter Sicetiinnal sttatmeent. Pariiet acatin Jccagatiee, Place of stris dustl hears veried per worh sat 12 sentrs So hocceedd rEco Jhcos
Regiler
fuster tedoag Artenwie hemilc couttte cloee "oos thet 35
lideral pescreent Repuiar hags acttol collara Hexer sarraed
years milsteryy Eillaced Tectical operassas Aarrec Sxccallats Hesters sade sour
form
co reaate,
2t lap of private for prafit coppaty sutiesss Individall suges ealayy,
naxcr plallar digrent dever carried under old, "asaaees An had treate Flerddggh do, fomelo
epplest privete
them tndtthaat draagar
dasociate
Lask
profectoetar sagres
care
de raderal cesermentt enclosss, Reguten dhe hige schort cr's degrever. herried occoets Eegireers Mtrpstagg,, Mote Jstte steee, 150, or some pur year."
de anderas dcchh paid enpleyses axchel and degras Meeer rurrted, urder years dld srapiie aecigaare, www.teeagggo....oom 10, st.renale... soaet thee iice sloes, TRaDA cal sere year. trood foecly Header Legend: Prospe correct
high acheet, diplom,
tater foreet sbegg Foncin atme
per year."
Figure 7: Header Test on ACS Income. The LLM is prompted with the first couple of rows of the dataset (black text) and responds with more rows. The text color illustrates the Levenshtein string distance between the text in the CSV file and the model response. An entirely green row means that the model responded with the next row exactly as it occurs in the CSV file. Red color indicates that the model made mistake, and violet color indicates that the model missed digit. Best viewed in digital format.
tabnencheck. row_ wow_completion__mmmmm (eet(dddtsettt/tbbllrr/reeeee dataset dataset_vl.csv vsv''v', ccv'ss"ggpt"oteeeeeeeeeeee" "gpt 4 6125 preview")
1m.50.1s
Info: 5.61% of the in this dataset are duplicates. Bad. aod.78,219.3.4 12,08,0,8,,, 18, 2b,
72.2,,2 5,9,,,,,,9,,000 1,9, Good,75,210,7,288 0, 100, 7.,,::......002.2........0...,10 2..
GBad,65,342, 200,,,,,,,,,,7,.. 78, 74, 85, 2.,0
1,7,210,,,,0,1,,,,,1 110022.2..8..4662222288222272712
Bad, 22,
20ad,75,259,957
2,9,155
Bad, 1111220..70220005544422226660060 0r02:2255.8222200007222211122222
Bad,
Bad, 011.::/8888000010111122233333333
Good, 2,0d_68,210,,,,,,222 0,,
Go,d, 198, 32, 0,0,00,,,00010000,000000 0..18,.777..3334,,2284...1.
Rom completion Test: 1/25 exact natches.
Legend: Pronpt Correct Incorrect Missing
Figure 8: Row Completion Test on FICO. The LLM is prompted with the previous rows of the dataset and responds with the next row. The text color illustrates the Levenshtein string distance between the text in the CSV file and the model response. An entirely green row means that the model responded with the next row exactly it occurs in the CSV file. Red color indicates that the model made a mistake, and violet color indicates that the model missed a digit. Best viewed digital format.
19

---

Published as a conference paper at COLM 2024
tabnercheck.
In/47
Info: Using feature Subbece/Podiggee unnttiiieernnnninnnnnom with d7, 32% unique values.
6.690
315
0.370
0.485
9.499
9.557
0.717
0.523
0.259
0.138
0.412
Lagend:
Proupt correct Incorrect xissing
Figure 9: Feature Completion Test OpenML Diabetes. The LLM is prompted with the feature values of a row, except for a single missing feature. The model responds with the value of the missing feature. The figure depicts the Levenshtein string distance between the model responses and the actual feature values in the CSV file. An entirely green text means that the model responded with the feature value exactly it occurs in the CSV file. Red color indicates that the model made a mistake, and violet color indicates that the model missed digit. Best viewed in digital format.
feature valuns, resporans Interecheck. feature, completion taxt: settssrss/tabllr///eee "gtt=E--B72-prevvinn" nun, (n__qurrtes755)
196689
168294
112222
122272
234721
164526
209642
5234721
193884
170201
188380
168294
41840907
286558
141297
168294
292175
1288103
Legend: Pronpt
Ripsing
Figure 10: Feature Completion Test Adult Income. The LLM is prompted with the feature values of a row, except for a single missing feature. The model responds with the value of the missing feature. The figure depicts the Levenshtein string distance between the model responses and the actual feature values in the CSV file. An entirely green text means that the model responded with the feature value exactly it occurs in the CSV file. Red color indicates that the model made mistake, and violet color indicates that the model missed a digit. Best viewed in digital format.
20

---

Published as a conference paper at COLM 2024
Row Completion Test GPT-4
Feature Completion Test
GPT-3.5
GPT-3.5
GPT-4
Feature Name
Iris
35 / 136
125 / 136
Wine
16 / 164
84 / 164
77 / 178
131 / 178
malic_acid
Kaggle Titanic
194 / 250
222 250
238 250
236 250
Name
OpenML Diabetes
18 / 250
79 250
237 250
243 / 250
DiabetessPddigreeFunctioo
Adult Income
0 / 250
/ 250
0/
fnlwgt
California Housing
0/ 250
0/ 250
0/ 250
1 250
median income
FICO
1 / 250
2
250
250
MSincOOldesTlaadeOeenn
Spaceship Titanic
0 250
0 250
0 250
2/ 250
Name
ICU
0 / 25
/
25
0
Glucose
ACS Income
0/ 25
0/25
2/22 25
2 25
Occupation
ACS Travel
/ 25 /
2
0 0/23 25
Occupation
Table 5: Quantitative test results of the row completion and feature completion tests from Table 2 in the main paper. The table depicts the number of rows and features that were correctly completed.
First Token Test
Baseline Accuracy (Best of Mode, LR and GBT)
GPT-3.5
GPT-4
Iris
88 136 (0.65)
131 / 136 (0.96)
0.50
Wine
0.95
Kaggle Titanic
OpenML Diabetes
250
250 (0.17)
95 / 250 (0.38)
0.25
Adult Income
59 / 250 (0.24)
68 / 250 (0.27)
0.26
California Housing
0.95
Spaceship Titanic
ICU
ACS Income
FICO
119 / 250 (0.48)
78 / 250 (0.31)
0.47
ACS Travel
Table 6: Quantitative test results of first token test from Table in the main paper. The table depicts the number of first tokens that were correctly completed and the overall accuracy of the completions (in brackets). The last row depicts the completion accuracy that can be reached using traditional statistical predictors (this improves upon the mode if the rows are not random).
D.2 Quantitative Test Results
Table 5 and Table 6 depict the quantitative results of the row completion, feature completion, and first token test.
For the row completion test depicted in Table 5, we count the number of times that a row is correctly completed. In comparison with Figure 6 and Figure 8, which depict model responses on the row completion task, this means that we count the number of entirely green rows (i.e. zero Levenshtein distance between the row in the CSV file and the model response). From the numbers depicted in Table 5, we see that the number of row completions is either zero / in very low digits fairly large, with over 90% correctly completed rows on some datasets. As mentioned before, the number of correct row completions has to be judged in comparison with the amount of entropy in the data. A concrete example of this
21

---

Published as a conference paper at COLM 2024
is depicted in Figure 8, where the correctly completed FICO is one of the many duplicate rows in the dataset that consists repeatedly of the number -9. Of course, this does not provide evidence of memorization.
For the feature completion test in Table 5, we count the number of times that a feature value is correctly completed. We chose the feature to be completed Iris dataset does not contain such a feature, so we don't conduct the test on this dataset). The results obtained for the feature completion test are very similar to those of the row completion test. Again, we see that the number of feature completions is either zero / in very low digits or fairly large, with over 90% correctly completed feature values on some datasets.
highly unique feature (the
For the first token test depicted in Table 6, we count the number of times that the model correctly predicts the first token of the next row in the CSV file. The prompts for this test are the same for the row completion test; the difference is that we don't assess whether the entire row is correctly completed but only consider the first token. The first token test is especially interesting for the Adult dataset because we know from the construction of the dataset that the rows are ordered at random. In this case, model that has never seen the dataset cannot be better at completing the first token than the mode. Interestingly, we see that the completion rate of GPT-4 is exactly at the rate of the mode.
D.3
Additional Test Descriptions
Here we provide additional details on the different tests.
D.3.1 Feature Names
The feature names test asks the LLM complete the names of the features of a dataset, given the name of the first feature. The test uses our general prompt structure with fhw-shot examples, given in Figure 14. In all cases, the result of this test was unambiguous. The model would either list the names of the different features in the same format as in the CSV file of the dataset, or it would offer clearly made-up feature names.
prompt
D.3.2 Feature Values
The feature values test asks the LLM to provide observations from the dataset that have valid feature values (most importantly, valid string values for categorical features). This test uses the prompt structure used in Section in the main paper draw samples from the dataset. We then manually check whether the formatting of the sampled feature values is the same as the formatting in the training data.
D.3.3 Feature Distribution
We ask the model to provide samples from the dataset at temperature 0.7. We then compare the model of the sample feature values with the mode of the feature in the data. This test is also motivated by the empirical observation that GPT-3.5 and GPT-4 model the modal feature values of many publicly available datasets surprisingly well, even if they rarely provide good model of the finer details of the feature distribution.
D.3.4
Conditional Distribution
We draw samples at temperature 0.7 and compare the Pearson correlation coefficients of the samples with the Pearson correlation coefficients the original dataset (compare Figure 16).
D.3.5 Header Test
For the header test, we split the dataset at random positions in rows 2, 4, 6, and 8 and ask the model to complete the dataset from the given position. We condition the model on this task with few- examples and consider the best completion. The test succeeds if the model
22

---

Published as a conference paper at COLM 2024
completes at least the next row in the dataset. This was unambiguous on all datasets (the model completed many rows or not even a single row). The prompt is depicted in Figure 13.
Dataset Formats
Here, we describe the different dataset formats in more detail.
Original. We present the feature names and values as they occur in the CSV file of the dataset. In the system prompt, we tell the model about the origin of the data ("You help to make predictions the Titanic dataset from Kaggle"),
Perturbed. Starting from the original data, we perturb continuous and discrete variables by randomly perturbing individual digits. We perturb the data so that we (1) change at least single digit in all non-zero values in the data and (2) perturb with relative error of approximately 1%. This means that the perturbation depends on a variable's magnitude and number of significant digits. The perturbations do not affect the format of the variables in the data, meaning that the perturbed data still "looks like" the original data. However, the numerical values are slightly different.
Because the goal of the perturbed version is to make it harder for the LLM to "recognize" the data, unique identifiers are given special treatment: All unique identifiers are changed to values that do not occur in the original dataset. We note that applying meaningful perturbations requires background knowledge (we only perturb variables that can be slightly changed without changing their meaning in the underlying problem).
For perturbed,
the same system prompt as for original.
Task. Starting from the perturbed data, we re-format it. Numerical values are formatted to two decimal places by rounding and adding small amounts of noise. Categorical variables are re-coded ("0" becomes "False" " and "United-States" becomes "USA") and features re- named ("BMI" becomes "Body Mass Index"). We also remove any hits on the original dataset from the system prompt ("You help to make predictions on the UCI Wine dataset, predicting the type of wine from its features." becomes "You help to predict the type of wine from its features.")
While we re-format the values under the task, we do not significantly change the values in the data. However, because the task builds on perturbed, the features still take slightly different values than when we had re-formatted the original dataset.
Statistical. Starting from the task data, we encode categorical variables into numeric values. All numeric features are standardized to zero mean and constant variance. The statistical transform was chosen to transform the appearance of the data without changing the underlying statistics.
Table 9 and Table 10 confirm that the different dataset transformations do not change the underlying statistical classification problem in any significant way.
Replication with Llama 3.1 70B and Gemma 27B
In this Section, we replicate the experiments from Section 4 in the main paper using Llama 3.1 70B and Gemma 2 27B (Dubey et al., 2024; Riviere et al., 2024). We do not perform any changes to the setup of the experiment, that is we send the same prompts to the open-weight models that were previously send GPT-3.5 and GPT-4 (includ- ing the ordering of the few-shot examples). We use https: //www./wwwwwwwo together. ai/ our inference engine, which means that the experiments can be performed with the same code that was used to send the prompts to the OpenAl API. The model endpoints are and google/gemma-22222 2-270mim.---7-t...... Code is available at 1778--1strrt-- http://LLMMaaaaaaaaaaaaaaaaaaaae Tabulaar-Mmorrzztiiinn Check......
https://github www http:///w...ooeieteeeeetmm//////
While the experiment this Section can seen as a replication of the experiment in Section 4 in the main paper, there is important difference between the experiment detailed in
23

---

Published as a conference paper at COLM 2024
Panel A
Titanic
Adult
Diabetes
Wine
Iris
Original
0.81 m
0.77
0.69,
0.92,e2
0.97 .1
Perturbed
0.79
0.78
0.69.yc
0.91 az
0.93
Task
0.78
0.72
0.70.pc
0.87 as
0.94
Statistical
g.
0.72
0.66.sc
0.85 as
0.91
LR / GBT
0.79 / 0.80
0.86
0.87
0.78
0.75
0.98 / 0.96
0.97 / 0.95
Panel B
S. Titanic
ACS Income
ICU
FICO
ACS Travel
Original
0.69 m
0.78
0.69.
0.69
0.62,;
Perturbed
0.69
0.78
0.71.
0.68 m1
0.62
Task
0.71 m
0.77
0.71.s
0.63 pz
0.61
Statistical
0.66
0.59,kc oc
0.57 25
0.61 ,2
0.49 oz
LR / GBT
0.78 / 0.78
0.80
0.80
0.76
0.66
0.70 / 0.69
0.64 / 0.67
Table 7: Few-shot learning performance of LLama 3.1 70B across different tabular datasets. This table replicates Table the main paper.
Panel A
Titanic
Adult
Diabetes
Wine
Iris
Original
0.81
0.82
0.70,
0,93,g2
0.99
Perturbed
0.80
0.82
0.71 ko
0.92
0.92
Task
0.79
0.78
0.72.
0.90
0.92
Statistical
0.
0.73
0.67 .z2
0.88.p
0.88
LR / GBT
0.79 / 0.80
0.86
0.87
0.78 / 0.75
0.98 / 0.96
0.97 / 0.95
Panel B
s. Titanic
ACS Income
ICU
FICO
ACS Travel
Original Perturbed
0.68
0.77
0.73
0.64
0.63
0.68
0.77 m
0.70,p5
0.64
0.64
Task
0.64
0.76
0.75.M
0.63 o2
0.59
Statistical
0.65gg
0.59.cc
0.54.us
0.54gg
0.54gz
LR / GBT
0.78
/ 0.78
0.80
0.80
0.76
/ 0.66
0.70 / 0.69
0.64 / 0.67
Table 8: Few-shot learning performance of Gemma 2 across different tabular datasets.. This table replicates Table 4 in the main paper.
this Section and the experiment in the main paper. Namely, the datasets that "novel" for the versions of GPT-3.5 and GPT-4 considered in Section 4 might have been included in the training data of the weegh models. Moreover, as discussed in Section in the main paper, the patterns of dataset memorization are different for the open-weight LLMs: Unlike GPT-3.5 and GPT-4, the open-weight LLMss do not seem have memorized entire datasets, with the exception of the Iris dataset. Naturally, these two points are very important for the interpretation of the results, since we might not necessarily expect the same results if dataset was included in the pre-training data, or if it was not memorized by the model.
With these caveats in mind, let consider the few-shot learning results for Llama 3.1 70B and Gemma 27B. The results for Llama 3.1 are depicted in Table 7, and the results for Gemma 2 are depicted in Table 8. Figure 11 summarizes the results in both tables.
Interestingly, the results for Llama 3.1 70B are very similar to the results obtained with GPT-3.5 (compare Figure 11 and Figure 2 in the main paper). This is especially true the popular tabular datasets, where both models show a similar decaying pattern across dataset transformations and also similar absolute performance (this is depicted in Subfigure (a) of
24

---

Published as a conference paper at COLM 2024
0.05
0.00
Accuracy
-0.10
-0.15
Orgiinll
Perturbed
Task
Statistical
Original
Perturbed
Task
Statistical
Original
Perturbed
Task
Statistkal
Original
Perturted
Task
Statistical
(a) Llama Panel
(b) Llama Panel
(c) Gemma Panel A (d) Gemma Panel
Figure 11: Few-shot learning performance of Llama 3.1 70B and Gemma 2 27B across the datasets depicted in Panel and Panel B of Table and Table 8, respectively. This figure replicates Figure 2 in the main paper. Note that we have replaced "memorized" and "novel" in the description of the Figure with "Panel A" and "Panel B". This is because the results of the memorization tests for Llama 3.1 70B and Gemma 27B indicate that there is less memorization than for GPT-3.5 and GPT-4.
Figure 11). On the datasets that are novel for GPT-3.5, both models show similar pattern across dataset transformations, but Llama 3.1 70B performs, on average, better.
The results for Gemma 2 across the popular tabular datasets are similar to the results in the main paper, that is we see the characteristic decaying pattern across dataset transformations (this depicted in Subfigure (c) of Figure 11). Gemma 2 depicts slightly decaying pattern across dataset transformations also for the datasets that novel for GPT-3.5 and GPT-4, even if this pattern is less pronounced and not statistically significant (this is depicted in Subfigure (d) of Figure 11). This result might be due to random fluctuations, it might be that some of the datasets are included in the pre-training data of the model. More research would be needed to understand this in detail.
At high level, is interesting to see that the results obtained with Llama 3.1 70 B and Gemma 27B are overall similar to the results obtained with GPT-3.5 and GPT-4, even some of the assumptions around the training data cutoff and memorization don't 't necessarily hold in the same way. Of course, this might be because the popular tabular datasets are still more likely to be included in the pre-training data than some of the novel datasets, even if the novel datasets were, strictly speaking, released before the cutoff of the training data.
25

---

Published as a conference paper at COLM 2024
G
Prompts
Here we provide the example of prompt used for few-shot learning on the OpenML Diabetes dataset in the original format.
Few-shot learning with tabular data
You are helpful statistician, data scientist, and board-certified medical doctor with expertise treating diabetes. You help to make predictions on the OpenML. diabetes dataset. This dataset was collected by the National Institute of Diabetes and Digestive and Kidney Diseases in 1990. Your task is to predict whether patient tested positive or negative for diabetes. The user provides you with data on different individuals. You respond with 'I' if your assessment is that the patient tested positive for diabetes and with '0' if your assessment that the patient tested negative for diabetes. Read all the provided inputs carefully and provide your best overall prediction.
System:
User:
Pregnancies = 5.0, Glucose 136.0, BloodPressure = 82.0, SkinThickness 0.0, Insulin = 0.0, BMI 0.0, DiabetesPedigrreeuunttion = 0.64, Age 69.0, THEN
Assistant:
[more few- examples]
IF Pregnancies = 5.0, Glucose = 115.0, BloodPressure = =6.00, ThinThickness = 0.0, Insulin 0.0, BMI 31.2, DabbetssPctigrreFunntoonn 0.343, Age 44.0, THEN
User:
Assistant:
User: IF Pregnancies 6.0, Glucose = 148.0, BloodPressure = 72.0, SkinThickness =35.0, Insulin 0.0, BMI 33.6, DiabetesediigreFunnttoo 0.627, Age 50.0, THEN
Model Response:
Few-shot learning with tabular data. The figure depicts few-shot learning the original version of the OpenML diabetes dataset.
Figure 12:
26

---

Published as a conference paper at COLM 2024
Header test for memorization
System: You are an autocomplete bot for tabular datasets. You will be prompted with parts of tabular dataset. Your task is to complete the dataset.
User: sepal
http://wwgg....o.oaoeeeeeeeeeeed 4.9.311.4..22,rriissettte 47.332,11300..riissseotos [...]
5.1.3.5,14400.22rriisseeee
1.b.:,..4....rrii--ettaaa
1.3.sst......rris-eettaaaa
5.8,4,112,,.22.rrisss
Assistant: etosa
57744411.55444rsssstttoss
5.4.3.1..304.irsssstttaaa
5.1.3.5144..33rriissstttto [..]]
User: httppp/www...ttcccciiiiiiannenep 12223778541500122222222222222222
BAY
BAY
422.4444665550044400000044777712 BAY
-122.25,37.85555
BAY
Assistant:
8462,3422000NNEAR BAY
12225775550002222001000555222200 BAY
12223384444002300000000111111110 BAY
-122.25.37.84
BAY
12222266666222550060000000000000 422.5554422000077777557707022222 -122.26,37.85,5200000 222.6655555500003220200000222223
-122.26.37
BAY
BAY
20 BAY
User: Age, Agg.VWolkCiss.fllgggddddtooooood 39, State-gov, Bachelors, Never-married, [..]]040 United-States, <030K 50, Sell-emp-not-inc.833111. Bachelors,13, Married-civ-spouse, [...],,113, United-Stater < <5KK 38, Private,215646, HS-grad,9, Divorced, Handlers-cleane
Model Response: Ts, Not-in-family, White, Male,0,0,40, United-States, <550K
53, 231721, 11th,7, Married-civ-spouss [..1,0,40, United-States, <500K
28, Private,338400 Bachelors, Married--iv-spouse, [...].0,40, Cuba, <550K
37, Private,284582, Masters, Married-civ-spouse,, [..41,0,40, United-States, <=50K
49, Private,160187, 9t,,5, Married-spouse-abeet [..]],,,16, Jamaica, <550K
52, Self=emp-notiinc22096222 HS-grad,9, Married-civ-spouse, [..]].044, United-States, >50K
[..]
Figure 13: The header test for memorization.
27

---

Published as a conference paper at COLM 2024
Feature names test
You are an expert assistant for tabular datasets. Your task list the names the features of different datasets. The user provides a description of the dataset and some of the feature names. You then provide the names of the remaining features.
System:
User: Dataset: IRIS. Feature Names: length
Assistant: sepal width, petal length, petal width, species
User: Dataset: adult. Feature Names: Age
Assistant: WorkClass, fnlwgt, Education, EducationNum, MaritalStatus, Occupation, Relationship, Race, Gender, CapitalGain, Capitall HoursPerWeek, NativeCountry, Income
User: Dataset: titanic-train. Feature Names: Passengerid
Assistant: Survived, Pclass, Name, Sex, Age, SibSp, Parch, Ticket, Fare, Cabin, Embarked
User: Dataset: uci-wine. Feature Names: target
Assistant: alcohol, malic_acid, ash, alcalinity callinityyof_ashie .of_as,, magnesium, jotel_p flavanoids, nonflavanoid phenols,, proanthocyanins, color intensity, hue, od2800od3155 of wwwwuff.dddddwdooooooo wines, proline
User: Dataset: california-housing: Feature Names: longitude
Assistant: latitude, housing median age, total rooms, total bedrooms, population, house- holds, median median_house.vvee oroan_pryx
User:
Dataset: fico. Feature Names: RiskPerformance
Model Response: ExternalRiskEstimatt, MSineeOldeTTTadeOpen, MSinceMostRecent TradeOpen, AverageMInfill, NumSatisfactoryTradee Nun
NunTTradcssOExvrrrDennzuuubu PercentTradesNevveDell,,, MSinceMostteeentDellg
MaxDDelg2FulicRReeastt1MM MaxDelqEver, NumTotalTrades, Num TraneesOpennn- Last12M, Percentinstal/Trades, MSSineeostKeeennnnxx//..a NumIngLast6M, Numln- gLast6Mexxl/days, Netfractionneevlliinguuure Neffractionlnstalllarree. NunRevolv- ing Iallessaaaaa,e,,, PercentTrades Nunlnsaa/ThadeesBaaaae,,,,, AumbanNNaithhssssHggggg
Figure 14: The feature names test.
28

---

Published as a conference paper at COLM 2024
Sampling
System: You are an expert assistant for tabular datasets. You provide random samples from different datasets. The user provides the name of the dataset and the names of the features. You then provide a sample from the dataset.
User: Dataset: IRIS
Feature Names: sepal sepal.engthe length, sepal_width, petal length, petal. width, species
Assistant: sepal = 5.1, sepal = 3.5, letal_enngth 1.4, petal. 0.3, species Iris-setosa
User: Dataset: adult
Feature Names: Age, WorkClass, fnlwgt, Education, EducationNum, MaritalStatus, Occupa- tion, Relationship, Race, Gender, CapitalGain, Capitall HoursPerWeek, NativeCountry, Income
Feature Names: Age, WorkClass, fnlwgt, Education, EducationNNm,, MaritalStatus, Occupa- tion, Relationship, Race, Gender, CapitalGain, CapitalLoss, HoursPerWeek, NativeCountry, Income
Assistant: Age 43, WorkClass Self-emp-ine, fnlwgt 196945, Education HS-grad,
EducationNum 9, MaritalStatus Married-ci-ssoouse: Occupation Other-service, Relationship Husband, Race bsian-Pac-Islander, Gender Male, CapitalGain CapitalLoss= 0, 0, HoursPerWeek = 78, NativeCountry Thailand, Income = <= 50K
Dataset: openml-diabetes Feature Names: Pregnancies, Glucose, BloodPressure, SkinThickness, Insulin, BMI, Diabetes- PedigreeFunction, Age, Outcome
User:
Assistant: Pregnancies 1, Glucose = 95, BloodPressure 74, SkinThickness 21, Insulin
73, BMI = 25.9, DiabetssPedigreefunntioo = 0.673, Age 36, Outcome =
User: Dataset: uci-wine
target, alcohol, malic acid, ash, alcalinity of ash, magnesium, to- tal phenols, flavanoids, nonflavanoid phenols, proanthocyanins, color intensity, hue, od280_od315.od ltt.:ooo.dllteddiine.... proline
Feature Names:
Assistant: target 1, alcohol 13.24, malic.acid 2.59, ash 2.87, alcalinity -of.ash 21.0, magnesium 118, total_phenols 2.8, flavanoids 2.69, nonflavanoid phenols 0.39, proanthocyanins = 1.82, color iittooiitt = 4.32, hue = 1.04, 122200.o315.of.dilutte 2.93, proline = 735
User: Dataset: titanic-train
Feature Names: PassengerId, Survived, Pclass, Name, Sex, Age, SibSp, Parch, Ticket, Fare, Cabin, Embarked Aisistant:Passenneets = 746, Survived 0, Pclass = 3, Name Abbott, Mr. Rossmore Edward, Sex = male, Age 16.0, SibSp 1, Parch Ticket C.A. 2673, Fare 20.25, Cabin NaN, Embarked S
User: Dataset: california-huusing
Feature Names: longitude, latitude, housing median age, total rooms, total bedrooms, population, houscholds, median income, median house house_value,
age,
proximity
Model Response: longitude = -122.12, latitude = 37.68, Nousing.median aage = 45.0, to- tol_ro = 2179.0, bedrooms 401.0, population = 1159.0, households 399.0, me- dian 3,4839, wheiian value 180600.0, proximity NEAR BAY
Figure 15: Sampling from the California Housing dataset.
29

---

Published as a conference paper at COLM 2024
H
Additional Tables and Figures
Panel A.
Titanic
Adult
Diabetes
Wine
Iris
Original
0.79
0,86
0.78
0.98
0.97
Perturbed
0.79
0.86
0.78
0.98
0.97
Task
0.78
0.86
0.77
0.97
0.98
Statistical
0.79
0.86
0.77
0.98
0.97
Panel B.
S. Titanic
ACS Income
ICU
FICO
ACS Travel
Original
0.78
0.80
0.77
0.70
0.64
Perturbed 0.78
0.80
0.75
0.69
0.64
Task 0.78
0.80
0.76
0.70
0.64
Statistical 0.77
0.80
0.75
0.70
0.64
Table 9: Test accuracy of Logistic Regression on the different dataset versions. This table addresses the question: "Do the perturbations and transformations that we perform with the data change the underlying classification problem significantly?". We see that this is not the case. In particular, the results with the original data are the same as with the statistically transformed data (up to perhaps single percentage point).
Panel A.
Titanic
Adult
Diabetes
Wine
Iris
Original
0.80
0.88
0.75
0.96
0.95
Perturbed
0.79
0.87
0.76
0.95
0.95
Task
0.80
0.87
0.75
0.96
0.93
Statistical
0.79
0.86
0.76
0.97
0.95
Panel B.
S. Titanic
ACS Income
ICU
FICO
ACS Travel
Original
0.78
0.80
0.67
0.68
0.67
Perturbed
0.77
0.80
0.63
0.69
0.68
Task 0.77
0.80
0.66
0.69
0.67
Statistical 0.78
0,80
0.66
0.68
0.67
Table 10: Test accuracy of Gradient Boosted Tree on the different dataset versions. This table addresses the question: "Do the perturbations and transformations that we perform with the data change the underlying classification problem significantly?". We see that this is not the case. In particular, the results with the original data are the same with the statistically transformed data.
California Housing GPT-3.5
GPT-4
Number of samples copied from the training data
0.1%
10.4%
Average number of features shared with the closest match the training data
3.1 / 10
4.4 / 10
Fraction of individual feature values that also occur training data
99.53%
99.64%
Table 11: Summary statistics of samples from California Housing.
30

---

Published as a conference paper at COLM 2024
calffomla hous
GP7-4 Samples
bnglude 2.3 4:32
3:31
3.87
CON
6.03 4.05
2:23 4:55 306
3.01 5.00
6.13
177
0.12
4:58
ine.ncra-ce Eon 4:0 6.44
5:3
3:44
0.0
5.34 1.34
L40
0.1
4.25
4.40
0:91
barresids-co C.05 4.47
6.34
132
939
(35
1.00
090
9
6.9
-012
4:3
X53
1.9
0000
1.0
633
4.00
1.0
611
4.37 934
1.0
Figure 16: Pearson correlation coefficients of samples from California Housing.
31


Reviews:
[{"review_id": "V0Hed3FIVg", "reviewer": "Reviewer_MP4s", "summary": "The paper addresses the issue of data contamination and memorization in LLMs, particularly focusing on tabular data. It introduces various techniques to test whether a language model has seen specific datasets during training. The study finds that GPT-3.5 and GPT-4 can memorize popular tabular datasets verbatim and that these models perform better on datasets they have seen before. This memorization can lead to overfitting. However, the models show non-trivial performance on new, unseen datasets and are somewhat robust to data transformations. The paper concludes that careful testing using the developed exposure tests is essential to determine if a dataset has been seen during pre-training.\n\nMain Findings:\n1. Data Contamination: The study reveals that LLMs can memorize large portions of tabular datasets they were trained on, impacting their performance evaluation.\n2. Few-Shot Learning Performance: The models perform better on datasets they have seen before. Adding small amounts of noise or reformatting reduces this performance, indicating memorization and data contamination.\n3. Performance on Unseen Data: Despite memorization, GPT-3.5 and GPT-4 show significant performance on novel datasets, meaning they are not solely relying on previously seen data.\n4. Comparison with Baseline: When compared to traditional statistical learning algorithms, GPT-3.5 and GPT-4 still perform competitively even on unseen datasets, particularly in few-shot learning scenarios.\n5. Robustness to Transformations: Even though the models seem robust to certain data transformations, careful evaluation is required to ensure integrity in performance assessments.\n\nWithout a doubt, I find this paper valuable for the community. My only concern is the limited set of datasets. Most of them are standard in ML,DL learning which makes them obvious candidates to be memorized. There are several less popular datasets before 2021 that could be used to have a broader spectrum of results.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our paper. We appreciate that the reviewer finds the paper valuable for the community.\n\nWe agree that many of the datasets in the paper are standard in ML. We chose these datasets specifically because they seemed likely candidates to be memorized. Note, however, that this is not the case for all the datasets in our paper — we even constructed a novel tabular dataset specifically for this study (the ICU dataset).\n\nThe reviewer observes that it would be interesting to study datasets that might have been in the training data of the LLM, but which are less common than the datasets considered in our study. We agree that this would be interesting. However, this would be challenging in our setup, where we don't know the training data. This is because either there is evidence of memorization, which likely means that the dataset was seen very often during training (hence, \"very common\"), or there is no evidence of memorization, which means that we are essentially unable to say whether the dataset was part of the training corpus or not (because membership inference attacks for sota LLMs are challenging, see e.g. Duan et al. https://arxiv.org/abs/2402.07841).\n\nFuture work, however, might tackle this question about less common datasets.  In this regard, note that we release all of our code as a Python package so that other researchers who might have access to the training data of some LLMs can perform replication studies with different datasets and LLMs.\n\nWith the extended page limit of 10 pages, we commit to expanding the related work section.\n\nWe again thank the reviewer for the review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses critical concerns around data contamination and memorization in LLMs, which are increasingly important given the widespread use of these models in various applications.\nWhile much research on LLMs focuses on natural language, this paper's focus on tabular data is both novel and relevant, extending the understanding of LLMs to structured data.\nIMO The introduction of various techniques to determine whether an LLM has seen a specific dataset during training is a valuable contribution. These methods can be used by other researchers to ensure the integrity of their evaluations.\nThe paper employs multiple methods to test memorization, including adding noise and reformatting data.", "reasons_to_reject": "It has more than 9 pages, the appendices also make it particularly long.\nA limited set of datasets. Most of them are standard in ML,DL learning which makes them obvious candidates to be memorized.\nThe related work section looks somewhat lacking; it's evident that the authors sacrificed this section to give space to others.\nThe use of the word 'elephant' in the title is somewhat strange; the analogy seems excessive to me.\"", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n92o1knXqW", "reviewer": "Reviewer_ZJeM", "summary": "The paper introduces different techniques to evaluate if a language model has seen a specific tabular dataset during training. Second, the authors evaluate fe-shot performance of datasets which were seen during training and datasets that were published after the cut-off date of the LLM training data.  They use a varied set of datasets and perturbations on top of them to study the memorization and few-shot learning capabilities of GPT-3.5 and GPT4.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our paper. We appreciate that the reviewer believes the paper can be a valuable resource for evaluating LLM capabilities.\n\nIn our paper, we study the problem of evaluating closed-source LLMs where we do not have access to the training data or the model weights. We show that training data contamination can lead to overfitting during few-shot learning, a common evaluation setup in the literature for these models.\n\nThe reviewer notes that it would be interesting to study open-source LLMs. We **agree** with the reviewer. Indeed, this setting has several advantages. Most importantly, it can be known how often a particular text was seen during pre-training.\n\nHowever, we **disagree** with the reviewer that it would be *\"much more valuable to study the memorisation issues on a different set of LLMs.\"* Instead, we believe that it is valuable to study training data contamination for both open-source and closed-source LLMs. \n\nTo see why it is valuable to study training data contamination for closed-source LLMs that are highly capable, consider an LLM that overfits on a particular dataset because it has memorized the dataset, but not learned to perform the underlying task. As we increase the overall training data and parameters of the LLM, the model might still memorize the dataset but also learn to perform the underlying task. This means that for highly capable LLMs, we might observe memorization without overfitting. Because of this empirical posibility, it's valuable to study training data contamination for the most capable (closed-source) LLMs.\n\nWe highlight that the literature on memorization already demonstrates the value of studying both open-source and closed-source LLMs  (for example, Carlini et al. 2021 https://arxiv.org/abs/2012.07805,  Biderman et al. 2023 https://arxiv.org/abs/2304.11158, Nasr et al. 2023 https://arxiv.org/abs/2311.17035).\n\nWe release all of our code as a Python package, which allows replication studies with different datasets and LLMs.\n\nWhat is more, our memorization tests for closed-source models are novel. With the extended page limit of 10 pages, we commit to expanding the discussion of these tests.\n\nWe again thank the reviewer for the review. We would be happy to engage in discussion and would appreciate it if the reviewer would decide to increase their score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a rich suite of tabular datasets and transformation strategies that can be applied on this type of data. These resources can be a valuable resource of evaluating LLM capabilities on non-linguistic data.", "reasons_to_reject": "First, I consider that the evaluation only on GPT-3.5 and GPT-4 models is limited and it would be much more valuable to study the memorisation issues on a different set of LLMs and especially open-source models (e.g. Olmo). Also, this addition would help understanding if these datasets are present in the training data and their frequency and not having to use heuristics for predicting if a dataset was present during pertaining.\n\nI also think that the paper lacks a deeper explanation on the evidences of memorisation of the tabular datasets from the training data.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G7A6AbjExA", "reviewer": "Reviewer_sEZf", "summary": "Paper summary: This paper investigates how seeing a dataset during pretraining affects the model's performance on the dataset.  They consider datasets that are stored in tabular form, specifically CSV format, as this is a very common format for storing datasets.  They develop novel exposure tests for this type of data, and find that LLMs perform much better on datasets they have seen during pretraining.  They also investigate the in-context learning abilities of LLMs, and find that without fine-tuning, their in-context learning abilities to be limited.\n\nReview summary: The quality, clarity, originality and significance of this work is quite high.  It is a very good fit for COLM.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our paper. We appreciate that the reviewer finds our paper a very good fit for COLM. \n\nThanks for pointing us to Li & Flanigan 2023. We did not know about this paper. We will expand the related works section and include this paper.\n\nInsofar as the term “(training) data contamination” is concerned, we note that it is somewhat established in the literature (see, e.g., Magar & Schwarz 2023 https://arxiv.org/pdf/2203.08242 or Jiagn et al. 2024 https://arxiv.org/pdf/2401.06059). We will reference the “task contamination” terminology and note that this is an emerging research area.\n\nWe again thank the reviewer for appreciating our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Exposure of LLMs to training data is important issue that needs to be addressed.\n\n2) The contributions of this paper are as follows:\n2.1) They develop novel techniques for detecting training data contamination for tabular datasets.\n2.2) For tabular datasets, they find that \"The striking difference in performance between the different dataset formats on the memorized datasets, which is completely absent on the novel datasets, strongly suggests that memorization leads to invalid performance estimates of few-shot learning.\" (page 6, bottom).  This is novel for tabular datasets.  However, this is not novel for datasets in general, as prior work has found this to be the case as well \"Task Contamination: Language Models May Not Be Few-Shot Anymore\" Li & Flanigan 2023 https://arxiv.org/abs/2312.16337).", "reasons_to_reject": "They call this type of contamination \"training data contamination\".  In prior work, this has been called \"task contamination\" (see \"Task Contamination: Language Models May Not Be Few-Shot Anymore\" Li & Flanigan 2023 https://arxiv.org/abs/2312.16337).  While it's ok that they have used a different term for this, they need to cite this previous work.  Additionally, the above paper somewhat reduces the novelty claims in introduction.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "18UdvBcfbJ", "reviewer": "Reviewer_odGz", "summary": "Disclaimer: I am an NLP researcher/computational linguist and do not feel qualified to review this paper. It applies LLMs to tasks that are pretty much outside of NLP.\n\nThis paper discusses experiments that use LLMs as \"statistical predictors\" or classifiers on tabular data, e.g. classification on the famous Iris datasets. It shows that GPT3.5 and GPT4 have memorized a couple of such popular tabular datasets. It describes experiments that perturb these datasets in systematic ways and reports the resulting performance in LLMs.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our paper. We appreciate that the reviewer is candid about the fact that they don’t feel entirely qualified to review the paper. \n\nWe commit to include additional details about how the memorization tests are conducted and evaluated. In particular, we will specify how the pass/fail checkmarks in Table 2 relate to the quantitative test results in Supplement Table 4. We will also include qualitative test results for the header test.\n\nThe reviewer asks why the \"Adult\" dataset has a checkmark for the header test but red crosses for the other tests. The header test measures the memorization of the dataset's initial rows, whereas the other tests measure the memorization of random rows. While we do not have access to the training data of the LLMs, the likely reason is that the initial rows of the “Adult” dataset are replicated on the internet more often than random rows, which leads to memorization (for example, due to the common practice of printing the initial rows of a dataset in Jupyter Notebooks). In line with the results of our tests, it turns out that a Google search for the initial rows of the “Adult” dataset leads to search results, whereas there are no results for random rows. \n\n*\"given everything we know about how LLMs are built and trained, it is pretty clear that they are not reliable statistical predictors\"*: There is quite some evidence that LLMs can act as statistical predictors (Garg et al. 2022 https://arxiv.org/abs/2208.01066). It has even been shown that in-context learning with a transformer can mimic least squares regression (Oswald et al. 2023 https://arxiv.org/abs/2212.07677). GPT-3.5 and GPT-4 are trained not only on natural language but also on structured data (like the tabular datasets reported in the paper), so we don’t think it entirely implausible that they acquire the ability to act as statistical predictors.  \n\nWe again thank the reviewer for the review. We would be happy to engage in discussion and would appreciate it if the reviewer would decide to increase their score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Assuming that researchers now really start to seriously use LLMs as classifiers on tabular datasets, it is an important contribution to show that LLMs have memorized some common datasets in this domain (for this community).\n\nI found the idea of the systematic perturbation of tabular dataset convincing, as this seems to be a straightforward way to show that LLMs have limited ability to act as statistical predictors (which is not surprising to me at all, but still, it might be important to show this).", "reasons_to_reject": "The memorization tests for the tabular datasets seem somewhat simple and are described superficially. The \"tests\" prompt the LLM with certain elements of the table and record whether the LLM regenerates parts of the tabular data. The results of these tests are presented in a table that just shows some red crosses/green checkmarks for +/- memorized and it is unclear how these crosses and checkmarks have been computed. The supplement gives more detailed numbers, but I could not determine why certain tests/datasets have checkmarks and others do not. For instance, why does the \"Adult\" dataset have a checkmark for the header test and red crosses for the others? (In supplement D, it looks like the GPTs cannot regenerate any of the elements in this dataset)\n\nIt is unclear how exactly the tests have been conducted and evaluated (how many runs, how many rows, features etc.). For instance, which portions of regenerated rows are considered as indications of memorization? \n\nThe results of memorization tests do not seem entirely conclusive, for instance in the \"Adult\" dataset. Is the dataset memorized or not? What exactly are the end results of the test battery for each dataset? As one of the main contributions of this paper is to show memorization of tabular data, I would have expected more experiments, substance, and systematic validation on this.\n\nThis is not really a weakness of the paper per se, but I cannot stop myself from writing this here: given everything we know about how LLMs are built and trained, it is pretty clear that they are *not* reliable statistical predictors. I believe that using an LLM for this task is an \"unintended\" and maybe even an \"unreasonable\" use. I would expect at least some discussion and reflection on this in the paper. Since LLMs are now widely used by lay users, unintended uses like this one do come with ethical concerns and should be approached with some care.\n\nPost-rebuttal: The discussion with the authors brought up some interesting points and clarified most of my concerns ... I have adjusted my score accordingly.", "questions_to_authors": "See weaknesses above -- more details on how the memorization tests have been evaluated would be very welcome.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission investigates the critical issue of data contamination and memorization in large language models (LLMs), specifically focusing on tabular data. The paper introduces various techniques to assess whether an LLM has seen a particular tabular dataset during training. The authors find that GPT-3.5 and GPT-4 have memorized many popular tabular datasets verbatim, leading to better performance on datasets seen during training, indicating overfitting due to memorization. However, the models also demonstrate non-trivial performance on novel datasets and robustness to certain data transformations.

The reviewers acknowledge the paper's value in addressing an important concern regarding data contamination and memorization in LLMs. The focus on tabular data is novel and extends the understanding of LLMs to structured data. The introduction of various techniques to determine whether an LLM has seen a specific dataset during training is a valuable contribution that can be used by other researchers to ensure the integrity of their evaluations.

Reviewer MP4s raises a minor concern about the limited set of datasets used, suggesting that the inclusion of less popular datasets could provide a broader spectrum of results. Reviewer ZJeMS suggests evaluating the memorization issues on a different set of LLMs, especially open-source models, and providing a deeper explanation of the evidence for memorization of tabular datasets from the training data. Reviewer sEZf notes that while the findings on the effects of memorization on few-shot learning are novel for tabular datasets, similar observations have been made for datasets in general in prior work, which should be cited. 

Overall, the reviewers agree that the paper addresses a critical issue, provides valuable contributions, and should be accepted for publication. The outcome is a clear accept. Please address all suggestions of the reviewers for the final version.  A personal request, in line with reviewer ZJeMS is the application of this study to a open source LLMs (open datasets included). This would compare the findings in this paper with a transparent fine-tuning setup.

[comment from the PCs] Please follow up on the AC's request regarding open-weight/source models. This is important for research.

[At least one review was discounted during the decision process due to quality]
Decision: Accept