Hugging Face's logo

bigscience
/
tr1-13B-data

Model card Files Community
stas commited on
Commit
36b1f8a
•
1 Parent(s): 1638500

indices

Browse files
Files changed (10) hide show
  1. README.md +4 -0
  2. indices/meg-gpt2_text_document_test_indexmap_102400ns_2048sl_42s_doc_idx.npy +3 -0
  3. indices/meg-gpt2_text_document_test_indexmap_102400ns_2048sl_42s_sample_idx.npy +3 -0
  4. indices/meg-gpt2_text_document_test_indexmap_102400ns_2048sl_42s_shuffle_idx.npy +3 -0
  5. indices/meg-gpt2_text_document_train_indexmap_300000000ns_2048sl_42s_doc_idx.npy +3 -0
  6. indices/meg-gpt2_text_document_train_indexmap_300000000ns_2048sl_42s_sample_idx.npy +3 -0
  7. indices/meg-gpt2_text_document_train_indexmap_300000000ns_2048sl_42s_shuffle_idx.npy +3 -0
  8. indices/meg-gpt2_text_document_valid_indexmap_31948800ns_2048sl_42s_doc_idx.npy +3 -0
  9. indices/meg-gpt2_text_document_valid_indexmap_31948800ns_2048sl_42s_sample_idx.npy +3 -0
  10. indices/meg-gpt2_text_document_valid_indexmap_31948800ns_2048sl_42s_shuffle_idx.npy +3 -0
README.md ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+
2
+ - indices - these are shuffled indices that the training was using using. they were generated the first time the training started. So the order is the same if one replays them via the dataloader
3
+ the corresponding dataset is oscar-en that's on JZ at `$six_ALL_CCFRWORK/datasets-custom/oscar-en`
4
+
indices/meg-gpt2_text_document_test_indexmap_102400ns_2048sl_42s_doc_idx.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:946ef63e7414f3e420c0ad4bda8bcf1aff652b0df0d93516a52f85b1dba232a3
3
+ size 1217048
indices/meg-gpt2_text_document_test_indexmap_102400ns_2048sl_42s_sample_idx.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:02025bb405f0b422b18df2437b8b5c1bb02d4677d4ec7be1c04375a1d9751c8e
3
+ size 1099200
indices/meg-gpt2_text_document_test_indexmap_102400ns_2048sl_42s_shuffle_idx.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cddc0ca311a66865cfb91a5c519cc15d046127bb8ef6c8e3b7c32834d486edc7
3
+ size 549660
indices/meg-gpt2_text_document_train_indexmap_300000000ns_2048sl_42s_doc_idx.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b0e844389d995d0cf6c9ab1d890c531367d37be2db4cda480d807c6b6cbe1555
3
+ size 3464576192
indices/meg-gpt2_text_document_train_indexmap_300000000ns_2048sl_42s_sample_idx.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1d4463e4b36f7b51067b1448ec6ddbd1a0eef6e0b5081e62e180a5d4e6069144
3
+ size 3156893488
indices/meg-gpt2_text_document_train_indexmap_300000000ns_2048sl_42s_shuffle_idx.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6c706f60569014b80995da137002dd8095ae990241859185d5ed7ef7b98c7d8a
3
+ size 1578446804
indices/meg-gpt2_text_document_valid_indexmap_31948800ns_2048sl_42s_doc_idx.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:748c9d730e7ddd08b418d3e40a0faae5c7f714c807949371977e7aa701713cbe
3
+ size 304230548
indices/meg-gpt2_text_document_valid_indexmap_31948800ns_2048sl_42s_sample_idx.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d8ee7538902544baf4e641fa6667bedca057599214fd58782de1141a14289bd0
3
+ size 277086568
indices/meg-gpt2_text_document_valid_indexmap_31948800ns_2048sl_42s_shuffle_idx.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e2aad97b7b0dadb92bc6883e5cb09d57563c07878ec516f0af4cd9913ae18d97
3
+ size 138543344