README.md

---
license: apache-2.0
base_model: mistralai/Mistral-7B-Instruct-v0.3
tags:
- axolotl
- generated_from_trainer
model-index:
- name: mistral-7b-english-welsh-translate
  results: []
---

# mistral-7b-english-welsh-translate



This model is a fine-tuned version of [mistralai/Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) on the [Welsh Government Alpaca Welsh-English Instructions](https://huggingface.co/datasets/AndreasThinks/welsh-translation-instruction/blob/main/README.md) dataset.

This model is trained for English-Welsh translation (in any direction), with a focus on government documents, using Markdown formatting.

To ensure the highest quality translations, use the Alpaca instruction prompt format with the below structure.

```
    ### Instruction: {instruction}

    ### Input: {input}

    ### Response:

```

Your instruction should be "Translate the text from English to Welsh." (or vice versa).

The model is also available [quantized as GGUF](https://huggingface.co/AndreasThinks/mistral-7b-english-welsh-translate-GGUF). This version be be [tested in this interactive space](https://huggingface.co/spaces/AndreasThinks/welsh-english-translator).

## Running the model

The model is intended to be run locally, ideally using [Text generation web UI](https://github.com/oobabooga/text-generation-webui) to ensure correct prompt structure.

Start the UI as instructed for your system.

- In the "Model" tab, download either this model or [the quantized version](https://huggingface.co/AndreasThinks/mistral-7b-english-welsh-translate-GGUF). Once the download is complete, load the model.
- In the "Parameters" tab, under "Generation", set "auto_max_new_tokens" to maximum, and "Ban the eos_token" to True. In "Custom stopping strings", add "### Input"
- In the "Notebook" tab, make sure you are using the "Alpaca-with-input" prompt.  Set the instruction as "Translate the text from Welsh to English." (or vice versa).#
- Add the text you would like to translate (replacing "Input"), and hit "generate"

Performance may start to degrade past a certain context window (especially if using the quantized models).  Convert in chunks of under 1000 words to avoid these issues.

## LLM Evals

Thanks to [YALL - Yet Another LLM Leaderboard](https://huggingface.co/spaces/mlabonne/Yet_Another_LLM_Leaderboard).  [Full results available here.](https://gist.github.com/AndreasThinks/d998bc0a607ff5c7df09cd7333ed5c0c)

|                                                    Model                                                    |AGIEval|TruthfulQA|Bigbench|
|-------------------------------------------------------------------------------------------------------------|------:|---------:|-------:|
|[mistral-7b-english-welsh-translate](https://huggingface.co/AndreasThinks/mistral-7b-english-welsh-translate)|  35.31|      54.5|    38.4|


## Training procedure

[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
<details><summary>See axolotl config</summary>

axolotl version: `0.4.1`
```yaml
base_model: mistralai/Mistral-7B-Instruct-v0.3
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer

load_in_8bit: false
load_in_4bit: 
strict: false

# huggingface repo
datasets:
  - path: AndreasThinks/welsh-translation-instruction
    type: alpaca
val_set_size: 0.04
output_dir: ./outputs/mistral-welsh

hub_model_id: AndreasThinks/mistral-7b-english-welsh-translate


sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
eval_sample_packing: false

wandb_project: mistral-nemo-welsh
wandb_entity:
wandb_watch:
wandb_name: mistral-nemo-welsh-v1
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.000005

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:

```

</details><br>

[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/andreasthinks/mistral-nemo-welsh/runs/syq2m3vr)

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 4

### Training results

| Training Loss | Epoch  | Step | Validation Loss |
|:-------------:|:------:|:----:|:---------------:|
| 1.5781        | 0.0013 | 1    | 1.5514          |
| 0.4427        | 0.2506 | 194  | 0.4841          |
| 0.4142        | 0.5011 | 388  | 0.4271          |
| 0.4001        | 0.7517 | 582  | 0.3996          |
| 0.4155        | 1.0023 | 776  | 0.3828          |
| 0.3178        | 1.2296 | 970  | 0.3792          |
| 0.3156        | 1.4801 | 1164 | 0.3732          |
| 0.3115        | 1.7307 | 1358 | 0.3678          |
| 0.2722        | 1.9813 | 1552 | 0.3633          |
| 0.2492        | 2.2089 | 1746 | 0.3809          |
| 0.2159        | 2.4595 | 1940 | 0.3828          |
| 0.2277        | 2.7100 | 2134 | 0.3810          |
| 0.2435        | 2.9606 | 2328 | 0.3809          |
| 0.158         | 3.1899 | 2522 | 0.3961          |
| 0.1802        | 3.4404 | 2716 | 0.3966          |
| 0.1679        | 3.6910 | 2910 | 0.3966          |


### Framework versions

- Transformers 4.42.4
- Pytorch 2.3.1+cu121
- Datasets 2.19.1
- Tokenizers 0.19.1