419 kB

Ctrl+K

2 contributors

History: 33 commits

ViuAI

Fix CUDA OOM on T4: cast model to amp_dtype directly, batch_size=2, safe RMSNorm

9d0f9ca verified about 8 hours ago

add_curated_hinglish.py

3.23 kB
docs/feat: sync translation & hinglish expansion for data/scripts/add_curated_hinglish.py 13 days ago
add_dictionary_stories_uncensored.py

3.7 kB
docs & scripts: add dictionary, expanded stories, and massive uncensored datasets milestone (117GB/83B tokens) 13 days ago
add_full_scale_5_domains.py

10.8 kB
docs & scripts: add full-scale 5-domain milestone (145.93GB/105B tokens) 10 days ago
add_massive_uncensored_alpaca.py

1.38 kB
docs & scripts: add dictionary, expanded stories, and massive uncensored datasets milestone (117GB/83B tokens) 13 days ago
add_open_source_toxicity.py

3.32 kB
feat/docs: sync open-source toxicity datasets & docs for data/scripts/add_open_source_toxicity.py 13 days ago
add_samanantar.py

2.28 kB
docs/feat: sync translation & hinglish expansion for data/scripts/add_samanantar.py 13 days ago
add_science_stem_datasets.py

4.7 kB
docs & scripts: add dedicated science stream milestone (128.75GB/92B tokens) 12 days ago
add_science_stream.py

6 kB
docs & scripts: add dedicated science stream milestone (128.75GB/92B tokens) 12 days ago
add_stories_and_uncensored.py

20 kB
feat/docs: sync stories & uncensored scripts & docs for data/scripts/add_stories_and_uncensored.py 13 days ago
add_wikipedia.py

2.92 kB
docs/feat: sync deep domains, wikipedia ingestion & docs for data/scripts/add_wikipedia.py 13 days ago
check_classic_novels.py

456 Bytes
feat: wipe-proof HF persistence (full ckpt + live log + newest-2 cleanup) 8 days ago
clean_dedup.py

3.26 kB
docs: add official comprehensive data audit report (145.93GB/105.1B tokens - 100% PASS) 10 days ago
collect_automobile_and_tyohar_boost.py

12.3 kB
sync local: enforce_mix sampler, tokenizer v2 (mask fix + 35/30/35), mix optional docs, repair scripts 8 days ago
collect_distilled_corpus.py

11.4 kB
sync local: enforce_mix sampler, tokenizer v2 (mask fix + 35/30/35), mix optional docs, repair scripts 8 days ago
collect_domains.py

30 kB
Add Pillar 26 Grammar & Language Mechanics (Hindi Vyakaran, English Grammar, Hinglish Syntax) 14 days ago
collect_p1_p2_heavy_boost.py

11.6 kB
sync local: enforce_mix sampler, tokenizer v2 (mask fix + 35/30/35), mix optional docs, repair scripts 8 days ago
collect_p1_p2_real_pillars.py

15.3 kB
sync local: enforce_mix sampler, tokenizer v2 (mask fix + 35/30/35), mix optional docs, repair scripts 8 days ago
collect_romance_and_chat_corpus.py

16.8 kB
feat: wipe-proof HF persistence (full ckpt + live log + newest-2 cleanup) 8 days ago
collect_spiritual_and_boost_corpus.py

22.5 kB
fix: exclude non-5col schema paths from HF stream (wikipedia/grammar/gsm8k CastError on shuffle) 8 days ago
filter_by_safety.py

1.73 kB
docs: add official comprehensive data audit report (145.93GB/105.1B tokens - 100% PASS) 10 days ago
generate_basic_arithmetic.py

9.74 kB
docs & scripts: add full-scale 5-domain milestone (145.93GB/105B tokens) 10 days ago
generate_deep_domains.py

68.8 kB
docs/feat: sync deep domains, wikipedia ingestion & docs for data/scripts/generate_deep_domains.py 13 days ago
generate_toxicity_lexicon.py

18.1 kB
docs: add official comprehensive data audit report (145.93GB/105.1B tokens - 100% PASS) 10 days ago
generate_vyakaran_master.py

32 kB
docs: add official comprehensive data audit report (145.93GB/105.1B tokens - 100% PASS) 10 days ago
ingest_erotica_adult.py

14.3 kB
Fix CUDA OOM on T4: cast model to amp_dtype directly, batch_size=2, safe RMSNorm about 8 hours ago
ingest_large_cinema_mythology.py

22.4 kB
Sync data/scripts/ingest_large_cinema_mythology.py with massive Mythology and Bollywood Cinema corpora expansion 3 days ago
inspect_romance_datasets.py

1.66 kB
feat: wipe-proof HF persistence (full ckpt + live log + newest-2 cleanup) 8 days ago
kaggle_collector.py

21.2 kB
feat: 100% overnight-resilient batch repair with 999 retries and persistent rate-limit handling 10 days ago
kaggle_repair_all.py

19.3 kB
sync local: enforce_mix sampler, tokenizer v2 (mask fix + 35/30/35), mix optional docs, repair scripts 8 days ago
push_dataset_to_hf.py

3.63 kB
init: 28L MoE 241M audit-fixed, smoke pass 16 days ago
read_live_samples.py

1.56 kB
docs & scripts: add dictionary, expanded stories, and massive uncensored datasets milestone (117GB/83B tokens) 13 days ago
repair_backfill_hindi.py

4.97 kB
docs: add official comprehensive data audit report (145.93GB/105.1B tokens - 100% PASS) 10 days ago
repair_reshard_large.py

4.91 kB
docs: add official comprehensive data audit report (145.93GB/105.1B tokens - 100% PASS) 10 days ago
standardize_to_unified.py

10.6 kB
docs: add official comprehensive data audit report (145.93GB/105.1B tokens - 100% PASS) 10 days ago
test_xlit.py

2.67 kB
docs & scripts: add dictionary, expanded stories, and massive uncensored datasets milestone (117GB/83B tokens) 13 days ago