---
library_name: transformers
tags: []
---

# Model Card for Model ID

The Mistral-7B-v0.1 Large Language Model (LLM) is a pretrained generative text model with 7 billion parameters.
This model uses peft finetuning with NEFTune for robustness.

## Model Details

### Model Description

This model is a finetuned model of the [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1).

## Training Details

### Training Data

This model is finetuned with [kaist-ai/CoT-Collection](https://huggingface.co/datasets/kaist-ai/CoT-Collection).

### Training Procedure 

This model trained with SFT trainer and [NEFTune](https://arxiv.org/abs/2310.05914) method.
(According to the paper, NEFTune adds noise to the embedding vectors during training)

#### Training Hyperparameters

- lora alpha: 16
- lora r: 64
- lora dropout: 0.05
- max sequence length: 4096
- learning rate: 2e-4
- max_grad_norm: 0.3
- weight_decay: 0.001
- gradient checkpoint: True
- optim: paged_adamw_32bit
- use_bf16: True
- use_4bit: True
- use_nested_quant: False
- bnb_4bit_compute_dtype: float16
- bnb_4bit_quant_type: nf4