|
--- |
|
library_name: transformers |
|
tags: [] |
|
--- |
|
|
|
# Model Card for Model ID |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
|
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated. |
|
|
|
- **Developed by:** arcee.ai |
|
- **Finetuned from model:** mistralai/Mistral-7B-Instruct-v0.2 |
|
|
|
|
|
## Uses |
|
|
|
<!-- This has continually trained with the SEC data --> |
|
|
|
### Direct Use |
|
|
|
<!-- Since we used the instruct version of the Mistral we can use this directly. --> |
|
|
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
<!-- We used https://huggingface.co/datasets/arcee-ai/sec-data-full which consists of 2.5B tokens --> |
|
|
|
### Training Procedure |
|
|
|
<!-- We used the CPT method in the hugging face alignment book --> |
|
|
|
#### Preprocessing |
|
|
|
Tokenized and packed into 4096 chunks with the SFT trainer |
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
- **Training regime:** Used SFT trainer <!-- bf16 mixed precision --> |
|
|
|
|
