aalkaswan/codeparrot-portrait

Data Portrait: CodeParrot training data

A Data Portrait of codeparrot/codeparrot-clean-train (~5.3M deduplicated and filtered Python files from GitHub), the training data of codeparrot/codeparrot (1.5B) and codeparrot/codeparrot-small (110M). The validation split, codeparrot-clean-valid, is not included.

Text is normalised as in data-portraits and split into 50-character n-grams with stride 50, stored in 1 Bloom filter(s) sized to fit in RAM. A query ORs membership across filters: combined false-positive rate <= 0.01 (each filter 0.010000).

Each partition is a separate revision of this repo (tag p<i>of<k>, branch partition-<n>):

tag	files	sources	size
p0of1	53	codeparrot-clean-train	1.0 GiB

partitions.json lists the exact files (pinned dataset revision) in each partition. Query with codeparrot_portrait.query:

from codeparrot_portrait.query import SequentialPortrait
portrait = SequentialPortrait("aalkaswan/codeparrot-portrait", ["p0of1"])
report = portrait.contains_from_text([text], sort_chains_by_length=True)[0]