Data Portrait: RedPajama-Data-1T GitHub
A Data Portrait of the GitHub subset of togethercomputer/RedPajama-Data-1T: 98 jsonl files of source code (~59B tokens, served from data.together.xyz), the code part of the training data of RedPajama-INCITE (Base 3B v1, 7B) and OpenLLaMA v1 (3B, 7B, 13B). Those models were trained on 0.8T to 1T of RedPajama's 1.2T tokens, so they need not have seen all of it. SlimPajama's GitHub part is a filtered, deduplicated (and NFC-normalised) subset of this data.
Text is normalised as in data-portraits and split into 50-character n-grams with stride 50, stored in 1 Bloom filter(s) sized to fit in RAM. A query ORs membership across filters: combined false-positive rate <= 0.01 (each filter 0.010000).
Each partition is a separate revision of this repo (tag p<i>of<k>, branch partition-<n>):
| tag | files | sources | size |
|---|---|---|---|
p0of1 |
98 | github | 4.0 GiB |
partitions.json lists the exact files (URL, size and ETag; URL list at a pinned Hub revision) in each partition. Query with
redpajama_github_portrait.query:
from redpajama_github_portrait.query import SequentialPortrait
portrait = SequentialPortrait("aalkaswan/redpajama-github-portrait", ["p0of1"])
report = portrait.contains_from_text([text], sort_chains_by_length=True)[0]