Data Portrait: Comma v0.1 training data
A Data Portrait of common-pile/comma_v0.1_training_dataset (all 31 Common Pile sources), the training data of common-pile/comma-v0.1-1t and common-pile/comma-v0.1-2t. Both models were trained on this dataset; they differ only in how often each source is repeated, and the cooldown stage uses a subset of the main-stage sources, so this portrait covers both.
Text is normalised as in data-portraits and split into 50-character n-grams with stride 50, stored in 1 Bloom filter(s) sized to fit in RAM. A query ORs membership across filters: combined false-positive rate <= 0.01 (each filter 0.010000).
Each partition is a separate revision of this repo (tag p<i>of<k>, branch partition-<n>):
| tag | files | sources | size |
|---|---|---|---|
p0of1 |
2048 | arxiv_abstracts, arxiv_papers, biodiversity_heritage_library, caselaw_access_project, cccc, data_provenance_initiative, doab, foodista, github_archive, library_of_congress, libretexts, news, oercommons, peS2o, pre_1929_books, pressbooks, project_gutenberg, public_domain_review, pubmed, python_enhancement_proposals, regulations, stackexchange, stackv2_edu, stackv2_html, ubuntu_irc, uk_hansard, usgpo, uspto, wikimedia, wikiteam, youtube | 46 GiB |
partitions.json lists the exact files (pinned dataset revision) in each partition. Query with
comma_portrait.query:
from comma_portrait.query import SequentialPortrait
portrait = SequentialPortrait("aalkaswan/comma-v0.1-portrait", ["p0of1"])
report = portrait.contains_from_text([text], sort_chains_by_length=True)[0]