Edit model card

Mistral-7B-v0.1-chat-nl

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the Rijgersberg/no_robots_nl and Rijgersberg/ultrachat_10k_nl datasets. It achieves the following results on the evaluation set:

  • Loss: 1.0263

Model description

In order to investigate the effect of pretraining Rijgersberg/GEITje-7B on the finetuning of Rijgersberg/GEITje-7B-chat, I also subjected the base model Mistral 7B v0.1 to the exact same training. This model is called Mistral-7B-v0.1-chat-nl.

More info

Read more about GEITje and GEITje-chat in the 📄 README on GitHub.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 1e-05
  • train_batch_size: 2
  • eval_batch_size: 8
  • seed: 42
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 3

Training results

Training Loss Epoch Step Validation Loss
1.2404 0.2 236 1.1166
1.2103 0.4 472 1.1101
1.0357 0.6 708 1.0739
1.27 0.8 944 1.0540
1.3557 1.0 1180 1.0330
0.7919 1.2 1416 1.0368
0.8701 1.4 1652 1.0193
0.8851 1.6 1888 1.0009
0.7562 1.8 2124 0.9791
0.6838 2.0 2360 0.9823
0.5011 2.2 2596 1.0271
0.4495 2.39 2832 1.0267
0.5625 2.59 3068 1.0250
0.4486 2.79 3304 1.0262
0.5706 2.99 3540 1.0263

Framework versions

  • Transformers 4.36.0.dev0
  • Pytorch 2.1.1+cu121
  • Datasets 2.15.0
  • Tokenizers 0.15.0
Downloads last month
2
Safetensors
Model size
7.24B params
Tensor type
BF16
·

Finetuned from

Datasets used to train Rijgersberg/Mistral-7B-v0.1-chat-nl

Collection including Rijgersberg/Mistral-7B-v0.1-chat-nl