---
base_model: unsloth/llama-3.2-1b-instruct-bnb-4bit
language:
- en
license: apache-2.0
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
datasets:
- SkunkworksAI/reasoning-0.01
pipeline_tag: text-generation
---

# Llama 3.2 1B Mango 🥭

- **Developed by:** colesmcintosh
- **License:** apache-2.0
- **Finetuned from model :** unsloth/llama-3.2-1b-instruct-bnb-4bit

This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.

[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)

---

## Model Details
- **Base Model**: unsloth/llama-3.2-1b-instruct-bnb-4bit
- **Training Dataset**: [SkunkworksAI/reasoning-0.01](https://huggingface.co/datasets/SkunkworksAI/reasoning-0.01) – Chain-of-thought reasoning dataset with 29.9k examples to improve the model's ability to solve reasoning problems step-by-step.
- **Techniques**:
  - **LoRA (Low-Rank Adaptation)**: Fine-tuning to enhance conversational abilities without overfitting.
  - **QLoRA (4-bit Quantization)**: Used for reducing model size and speeding up inference times without sacrificing too much accuracy.
  - **RoPE Scaling**: To handle long-sequence token inputs effectively (up to 64k tokens).

---

## Training Details
The fine-tuning process was conducted using the `SFTTrainer` class from the `trl` library, which is optimized for training transformer models using reinforcement learning techniques. The training process was structured as follows:

### Training Configuration
```python
from trl import SFTTrainer
from transformers import TrainingArguments, DataCollatorForSeq2Seq
from unsloth import is_bfloat16_supported

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    dataset_text_field = "text",
    max_seq_length = max_seq_length,
    data_collator = DataCollatorForSeq2Seq(tokenizer = tokenizer),
    dataset_num_proc = 2,
    packing = False, # Can make training 5x faster for short sequences.
    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        max_steps = 60,
        learning_rate = 2e-4,
        fp16 = not is_bfloat16_supported(),
        bf16 = is_bfloat16_supported(),
        logging_steps = 1,
        optim = "adamw_8bit",
        weight_decay = 0.01,
        lr_scheduler_type = "linear",
        seed = 3407,
        output_dir = "outputs",
    ),
)
```

### Key Training Parameters
- **Batch Size**: `2` per device
- **Gradient Accumulation Steps**: `4` to accumulate gradients over multiple forward passes, allowing for effective training with smaller batch sizes.
- **Learning Rate**: `2e-4` with a linear decay schedule.
- **Warmup Steps**: `5` steps used to gradually increase the learning rate at the start of training.
- **Max Training Steps**: `60` total training steps.
- **FP16/BF16 Precision**: The model uses FP16 unless BF16 is supported, in which case it switches to BF16 precision for faster training on GPUs that support it.
- **Optimizer**: `adamw_8bit` – Adam optimizer with 8-bit memory-efficient operations, which reduces GPU memory usage during training.
- **Weight Decay**: `0.01` for regularization, preventing the model from overfitting.

### Dataset
- **Dataset Used for Training**: [SkunkworksAI/reasoning-0.01](https://huggingface.co/datasets/SkunkworksAI/reasoning-0.01) – The dataset contains **29.9k examples** of chain-of-thought reasoning instruction/output pairs.
  
### Collation Strategy
- **Data Collator**: `DataCollatorForSeq2Seq` is used to handle padding and tokenization efficiently, ensuring sequences are of the correct length during training.

---

## Inference Example

To run inference using the fine-tuned model, follow this code snippet:

```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="colesmcintosh/Llama-3.1-8B-Instruct-Mango",
    max_seq_length=64000,
    dtype=None,  # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
    load_in_4bit=True
)

# Enable native 2x faster inference
FastLanguageModel.for_inference(model)

# Prepare the input message
messages = [
    {"role": "user", "content": "Describe a tall tower in the capital of France."},
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,  # Must add for generation
    return_tensors="pt",
).to("cuda")

from transformers import TextStreamer
text_streamer = TextStreamer(tokenizer, skip_prompt=True)

# Generate response from the model
_ = model.generate(input_ids=inputs, streamer=text_streamer, max_new_tokens=128, use_cache=True, temperature=1.5, min_p=0.1)
```

This snippet will generate a response based on the input message and run inference using the `FastLanguageModel` class with the optimizations included in the `Unsloth` framework.

You can also use Hugging Face's AutoModelForPeftCausalLM. Only use this if you do not have unsloth installed. It can be hopelessly slow, since 4bit model downloading is not supported, and Unsloth's inference is 2x faster.
```python
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer
model = AutoPeftModelForCausalLM.from_pretrained(
    "colesmcintosh/Llama-3.1-8B-Instruct-Mango", # YOUR MODEL YOU USED FOR TRAINING
    load_in_4bit = load_in_4bit,
)
tokenizer = AutoTokenizer.from_pretrained("colesmcintosh/Llama-3.1-8B-Instruct-Mango")
```
I highly do NOT suggest - use Unsloth if possible!

## Additional Information
If you have any questions, feedback, or would like to collaborate on projects using this model, feel free to reach out to me on [LinkedIn](https://www.linkedin.com/in/cole-mcintosh) or visit my website at [colemcintosh.io](https://colemcintosh.io/). I’m always open to discussing AI, model development, and innovative solutions!