On Leakage of Code Generation Evaluation Datasets
Authors: name: Alexandre Matton; name: Tom Sherborne; name: Dennis Aumiller; name: Elena Tommasone; name: Milad Alizadeh; name: Jingyi He; name: Raymond Ma; name: Maxime Voisin; name: Ellen Gilsenan-McMahon; name: Matthias Gallé
Published: 2024-07-10 11:50:20

Abstract: In this paper we consider contamination by code generation test sets, in
particular in their use in modern large language models. We discuss three
possible sources of such contamination and show findings supporting each of
them: (i) direct data leakage, (ii) indirect data leakage through the use of
synthetic data and (iii) overfitting to evaluation sets during model selection.
  Key to our findings is a new dataset of 161 prompts with their associated
python solutions, dataset which is released at
https://huggingface.co/datasets/CohereForAI/lbpp .