Note Original dataset used to train tokenisers and models.

pietrolesci/tokenisers

Updated 10 days ago

Note Tokenisers trained on the MiniPile. The `_raw_tokenisers` folder contains the original tokenisers trained with a vocabulary size of 320k. Then, each folder is a `transformers`-compatible tokeniser of a smaller size.

Note Tokenised MiniPile dataset(s). Each split correponds to a tokeniser in `pietrolesci/tokenisers`.

pietrolesci/smol_llama-81M-tied_bpe8064minipile

Updated 10 days ago

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/smol_llama-81M-tied_bpe32000minipile

Updated 10 days ago

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/smol_llama-81M-tied_bpe128000minipile

Updated 10 days ago

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/smol_llama-81M-tied_wordpiece32000minipile

Updated 10 days ago

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/smol_llama-81M-tied_bpe2wp32000minipile

Updated 10 days ago

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved every 2k steps. The bpe2wp nomenclature means that we choose the merges using the BPE objective, and we tokenised the MiniPile using the resulting vocabulary and the WordPiece tokenisation function (i.e., longest prefix match).

pietrolesci/smol_llama-370M-tied_bpe32000minipile

Updated 4 days ago • 56

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/smol_llama-1B_bpe32000minipile

Updated 4 days ago • 54

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

Note Log-probabilities computed on the validation set of the MiniPile dataset using the models above.