|
--- |
|
model_type: llama |
|
pipeline_tag: text-generation |
|
datasets: |
|
- togethercomputer/RedPajama-Data-1T-Sample |
|
tags: |
|
- llama |
|
--- |
|
|
|
This is [Llama 2 13b](https://huggingface.co/meta-llama/Llama-2-13b-hf) with some additional attention heads from original-flavor Llama 33b frankensteined on. |
|
|
|
Fine-tuned on ~10M tokens from RedPajama to settle in the transplants a little. |
|
|
|
Not intended for use as-is - this model is meant to serve as a base for further tuning, hopefully with a greater capacity for learning than 13b. |
