---
license: mit
base_model: BramVanroy/fietje-2b
tags:
- trl
- fietje
- alignment-handbook
- sft
datasets:
- BramVanroy/ultrachat_200k_dutch
- BramVanroy/no_robots_dutch
- BramVanroy/belebele_dutch
model-index:
- name: fietje-2b-instruct
  results: []
pipeline_tag: text-generation
inference: false
language:
- nl
---

<p align="center" style="margin:0;padding:0">
  <img src="https://huggingface.co/BramVanroy/fietje-2b-instruct/resolve/main/img/fietje-2b-banner-rounded.png" alt="Fietje banner" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
</p>

<div style="margin:auto; text-align:center">
  <h1 style="margin-bottom: 0">Fietje 2B Instruct</h1>
  <em>An open and efficient LLM for Dutch</em>
</div>

<blockquote class="tip">
  <p align="center">
    <a rel="nofollow" href="https://huggingface.co/BramVanroy/fietje-2b">👱‍♀️ Base version</a> -
    <a rel="nofollow" href="https://huggingface.co/BramVanroy/fietje-2b-instruct">🤖 Instruct version</a> (this one) -
    <a rel="nofollow" href="https://huggingface.co/BramVanroy/fietje-2b-chat">💬 Chat version</a> -
    <a rel="nofollow" href="https://huggingface.co/BramVanroy/fietje-2b-chat-GGUF">🚀 GGUF of instruct model</a>
  </p>
</blockquote>

# fietje-2b-sft

This model is a fine-tuned version of [BramVanroy/fietje-2b](https://huggingface.co/BramVanroy/fietje-2b) on the generator dataset.
It achieves the following results on the evaluation set:
- Loss: 0.8818

## Model description

More information needed

## Intended uses & limitations

The same limitations as [phi-2](https://huggingface.co/microsoft/phi-2#limitations-of-phi-2), and LLMs in general, apply here. LLMs hallucinate, make mistakes, and should not be trusted. Use at your own risk!

## Training and evaluation data

Fietje 2B instruct was finetuned from [the base model](https://huggingface.co/BramVanroy/fietje-2b) on the following datasets. Number of training samples per dataset given in brackets, totalling 201,579.

- [BramVanroy/ultrachat_200k_dutch](https://huggingface.co/datasets/BramVanroy/ultrachat_200k_dutch): gpt-4-1106-preview; multi-turn; fully generated (192,598)
- [BramVanroy/no_robots_dutch](https://huggingface.co/datasets/BramVanroy/no_robots_dutch): gpt-4-1106-preview; prompt translate, answer generated; some items have system messages (8181)
- [BramVanroy/belebele_dutch](https://huggingface.co/datasets/BramVanroy/belebele_dutch): Dutch portion of [belebele](https://huggingface.co/datasets/facebook/belebele), formatted into SFT format (800)

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 6e-05
- train_batch_size: 42
- eval_batch_size: 42
- seed: 42
- distributed_type: multi-GPU
- num_devices: 16
- total_train_batch_size: 672
- total_eval_batch_size: 672
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-07
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 3.0

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 0.9325        | 1.0   | 178  | 0.9060          |
| 0.8687        | 2.0   | 356  | 0.8850          |
| 0.8385        | 3.0   | 534  | 0.8818          |


### Framework versions

- Transformers 4.39.1
- Pytorch 2.1.2+cu121
- Datasets 2.18.0
- Tokenizers 0.15.2