Note This folder contains the model evaluations (or "stats") for each model size included in the study. This is the "raw" version where we have stats at the token level. We gathered these statistics "just in case" since the inference process was expensive. However, we provide the sequence-level statistics in the `pietrolesci/pythia-deduped-stats` dataset.

Note This folder contains the model evaluations (or "stats") for each model size included in the study and already aggregated at the sequence level. Based on the "raw" version where we have stats at the token level (`pietrolesci/pythia-deduped-stats-raw`).

Note This is the dataset we used in our study and corresponds to the training set for the models reported below.

Note Sample from the Pile (`EleutherAI/pile-deduped-pythia-preshuffled`) used in the experiments. The unique sequence identified (`seq_idx`) is simply the order of the sequence in the Pile. The dataset is already tokenized.

Note The validation data used in our study. The Pythia suite does not have an official validation. However, we confirmed with the authors that the Pile validation split (this one) was not seen during training. It is still a bit confusing whether the Pile data can be released freely. Thus, we will remove this dataset if required.

EleutherAI/pythia-2.8b-deduped

Text Generation • Updated Jul 9, 2023 • 16.9k • 14

Note This model size was ultimately not included in the analysis as we found what seems to be a potential mismatch between checkpoints and batch indices. Specifically, no instantaneous memorisation was detected, which is puzzling since even the 70M parameters model experiences it.