Bamboo 1.1B
This is a WIP model trained only on public domain (CC0) datasets, primarily in the English language.
Further training is planned, but currently no multi-language datasets are in use or planned; though this may change in the future and the current datasets can contain languages other than English.
Architecture
This is using the Mistral architecture as a base, with the following settings:
- Vocab Size: 32000
- n_embd: 4096
- n_layer: 32
- n_head: 32
- max_position_embeddings: 8192 (context length)
- sliding_window: 4096
License
Though the training data of this model is CC0, the model itself is not. The model is released under the OpenRAIL license, as tagged.
Planned updates
As mentioned, a few updates are planned:
- Further training on more CC0 data, this model's weights will be updated as we pretrain on more of the listed datasets.
- Experiment with exteding the context length using YaRN to 32k tokens.
- Fine-tuning the resulting model for instruct, code and storywriting. These will then be combined using MergeKit to create a MoE model.
- Release a GGUF version and an extended context version of the base model
Test Results
TBD
Tokenizer
Our tokenizer was trained from scratch on 500,000 samples from the Openwebtext dataset. Like Mistral, we use the LlamaTokenizerFast as our tokenizer class; in legacy mode.
Tokenization Analysis
The histogram below illustrates the distribution of token counts per sample in our Mistral tokenizer, trained on 500,000 samples. This analysis provides insights into the tokenizer's behavior and its ability to handle various sentence lengths. The histogram was created using 10,000 samples of tokenization.
Key Observations:
- Broad Range: The tokenizer effectively handles a wide range of sentence lengths, with token counts spanning from 1 to over 50.
- Smooth Distribution: The distribution is relatively smooth, indicating consistent tokenization across different sentence lengths.
- Short Sequence Preference: Most sequences fall below 20 tokens, aligning with the efficiency goals of Mistral models.
- Peak at 7 Tokens: A peak around 7 tokens suggests the tokenizer might have a preference for splitting sentences at this length. This could be investigated further if preserving longer contexts is a priority.
Overall: The tokenizer demonstrates a balanced approach, predominantly producing shorter sequences while still accommodating longer and more complex sentences. This balance promotes both computational efficiency and the ability to capture meaningful context.
Further Analysis: While the tokenizer performs well overall, further analysis of the peak at 7 tokens and the lower frequency of very short sequences (1-3 tokens) could provide additional insights into its behavior and potential areas for refinement.
Theory On 7-token peak: Since English sentences have an average of 15-20 words, our tokenizer may be splitting them into 7-9 tokens, which would possibly explain this peak.