aalkaswan/the-heap-portrait

Data Portrait: The Heap

A Data Portrait of AISE-TUDelft/the-heap, a contamination-free multilingual code dataset (57 languages, copyleft GitHub repositories) for evaluating code LLMs. Every row is indexed, including rows flagged as exact or near duplicates of a training set.

Text is normalised as in data-portraits and split into 50-character n-grams with stride 50, stored in 1 Bloom filter(s) sized to fit in RAM. A query ORs membership across filters: combined false-positive rate <= 0.01 (each filter 0.010000).

Each partition is a separate revision of this repo (tag p<i>of<k>, branch partition-<n>):

tag	files	sources	size
p0of1	695	ANTLR, Ada, Agda, Apex, Assembly, C, C#, C++, COBOL, Clojure, Common Lisp, Coq, Crystal, Cuda, D, Dart, EJS, Elixir, Emacs Lisp, Erlang, F#, Forth, Go, Groovy, Hack, Haskell, Java, JavaScript, Julia, Kotlin, Less, Lua, Mathematica, Matlab, NetLogo, NewLisp, Nix, OCaml, Objective-C, PHP, Pascal, Perl, Processing, Prolog, Python, R, Raku, Ruby, Rust, SQL, Scala, Scheme, Scilab, Starlark, Swift, Vue, WebAssembly	6 GiB

partitions.json lists the exact files (pinned dataset revision) in each partition. Query with portraits.query:

from portraits.query import SequentialPortrait
portrait = SequentialPortrait("aalkaswan/the-heap-portrait", ["p0of1"])
report = portrait.contains_from_text([text], sort_chains_by_length=True)[0]