---
datasets:
- databricks/databricks-dolly-15k
language:
- en
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
---

# Model Card for Model ID

MistralAI 7B model fine-tuned for 1 epoch on Dataricks instruction tuning dataset.

## Model Details

### Model Description

- **Developed by:** Andrew Chahnwoo Park
- **Model type:** [Mistral](https://arxiv.org/pdf/2310.06825.pdf)
- **Language(s) (NLP):** English
- **License:** apache-2.0
- **Finetuned from model:** [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)

### Mistral Repository

- **Repository:** [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)

## Training Details

### Training Data

- [databricks/databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k)

### Training Procedure

- Quantized Low-Rank Adaptation (QLoRA)
- Transformers Trainer
- DataCollatorForSeq2Seq
- Distributed Data Parallel (DDP) across two GPUs

#### Preprocessing

Manually created tokenized 'labels' for the dataset.
Prompt template utilized basic template for instruction-tuning 

### Hardware

Performed fine-tuning with 2 * A100 GPUs
- Provided by Gnewsoft during work period
Model and dataset are too large for free run sessions on Google Colab