aalkaswan/codet5p-portrait

Data Portrait: CodeT5+ training data

A Data Portrait of the pretraining data of CodeT5+ (220M, 770M, their -py versions, the 110M embedding and the 220M bimodal model), rebuilt from the description in the CodeT5+ paper (Section 4.1):

• stage 1, github-code: codeparrot/github-code in Python, Java, Ruby, JavaScript, Go, PHP, C, C++ and C#, permissive licenses only, files of 50 to 2000 whitespace-separated tokens, CodeSearchNet repositories removed
• stage 2, codesearchnet: code-search-net/code_search_net train split, functions and docstrings

A github-code file's language comes from its path, as in the dataset's loading script. The paper's "50 to 2000 tokens" are read as whitespace-separated tokens: that reproduces its 51.5B CodeT5 tokens (51.6B, extrapolated from 8 of the 1126 files), while counting CodeT5 tokens would give 24.5B. CodeSearchNet repositories (any language or split) are removed from stage 1 by name, as in the paper.

Which checkpoint saw which stage (model cards): 220M and 770M saw stage 1 only, and their -py versions one more epoch of its Python part; the 110M embedding and 220M bimodal models saw both stages. The 2B, 6B and 16B models were trained on stage 1 and its Python part too, but they start from CodeGen-mono checkpoints whose training data was never released.

Superset: the paper also removed repositories that appear in its other evaluation tasks. It does not list them, so their files are kept here.

Text is normalised as in data-portraits and split into 50-character n-grams with stride 50. Each stage has its own Bloom filter(s) (2 in total), so a match can be traced to a stage. A query ORs membership across filters: combined false-positive rate <= 0.01 over all of them (each filter 0.005013).

Each partition is a separate revision of this repo (tag <stage>-p<i>of<k>, branch partition-<n>):

stage	tag	files	size
github-code	github-code-p0of1	1126	4.0 GiB
codesearchnet	codesearchnet-p0of1	6	42 MiB

partitions.json lists the exact files (pinned dataset revisions) in each partition. Query with codet5p_portrait.query:

from codet5p_portrait.query import SequentialPortrait
portrait = SequentialPortrait("aalkaswan/codet5p-portrait", ["github-code-p0of1", "codesearchnet-p0of1"])
report = portrait.contains_from_text([text], sort_chains_by_length=True)[0]