The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating
  Large Language Models
Authors: name: Jonathan Katzy; name: Razvan Mihai Popescu; name: Arie van Deursen; name: Maliheh Izadi
Published: 2025-01-16 16:48:41

Abstract: The recent rise in the popularity of large language models has spurred the
development of extensive code datasets needed to train them. This has left
limited code available for collection and use in the downstream investigation
of specific behaviors, or evaluation of large language models without suffering
from data contamination. To address this problem, we release The Heap, a large
multilingual dataset covering 57 programming languages that has been
deduplicated with respect to other open datasets of code, enabling researchers
to conduct fair evaluations of large language models without significant data
cleaning overhead.