license: other
license_name: llama-3
license_link: https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/raw/main/LICENSE
base_model: meta-llama/Meta-Llama-3-8B-Instruct
tags:
- generated_from_trainer
model-index:
- name: lightblue/suzume-llama-3-8B-multilingual
results: []
Suzume
This Suzume 8B, a multilingual finetune of Llama 3 (meta-llama/Meta-Llama-3-8B-Instruct).
Llama 3 has exhibited excellent performance on many English language benchmarks. However, it also seemingly been finetuned on mostly English data, meaning that it will respond in English, even if prompted in other languages.
We have fine-tuned Llama 3 on almost 90,000 multilingual conversations meaning that this model has the smarts of Llama 3 but has the added ability to chat in more languages.
Please feel free to comment on this model and give us feedback in the Community tab!
We will release a paper in the future describing how we made the training data, the model, and the evaluations we have conducted of it.
How to use
The easiest way to use this model on your own computer is to use the GGUF version of this model (lightblue/suzume-llama-3-8B-multilingual-gguf) using a program such as jan.ai or LM Studio.
If you want to use this model directly in Python, we recommend using vLLM for the fastest inference speeds.
from vllm import LLM, SamplingParams
sampling_params = SamplingParams(temperature=0.0, max_tokens=100)
llm = LLM(model="lightblue/suzume-llama-3-8B-multilingual")
messages = []
messages.append({"role": "user", "content": "Bonjour!"})
prompt = llm.llm_engine.tokenizer.tokenizer.apply_chat_template(conversation=messages, add_generation_prompt=True, tokenize=False)
prompts = [prompt]
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
Evaluation scores
We achieve the following MT-Bench scores across 6 languages:
| meta-llama/Meta-Llama-3-8B-Instruct | lightblue/suzume-llama-3-8B-multilingual | Nexusflow/Starling-LM-7B-beta | gpt-3.5-turbo | |
|---|---|---|---|---|
| German 🇩🇪 | NaN | 7.26 | 6.99 | 7.68 |
| French 🇫🇷 | NaN | 7.66 | 7.29 | 7.74 |
| Japanese 🇯🇵 | NaN | 6.56 | 6.22 | 7.84 |
| Russian 🇷🇺 * | NaN | 8.19 | 8.28 | 7.94 |
| Chinese 🇨🇳 | NaN | 7.11 | 6.97 | 7.55 |
| English 🇺🇸 | 7.98 | 7.73 | 7.92 | 8.26 |
* (Note the Russian scores exclude code, reasoning and math problems due to not having any translated reference answers for these questions.)
We observe minimal degredation of Llama 3's English ability while achieving best-in-class multilingual abilities compared to the top rated 7B model (Nexusflow/Starling-LM-7B-beta) on the Chatbot Arena Leaderboard.
Here is our evaluation script.
Training data
We train on three sources of data to create this model:
- lightblue/tagengo-gpt4 - 76,338 conversations
- A diverse dataset of initial inputs sampled from lmsys/lmsys-chat-1m and then used to prompt
gpt-4-0125-preview
- A diverse dataset of initial inputs sampled from lmsys/lmsys-chat-1m and then used to prompt
- megagonlabs/instruction_ja - 669 conversations
- A hand-edited dataset of nearly 700 Japanese conversations taken originally from translations of the kunishou/hh-rlhf-49k-ja dataset.
- openchat/openchat_sharegpt4_dataset - 6,206 conversations
- Multilingual conversations of humans talking to GPT-4.
We prepare our data like so:
import pandas as pd
from datasets import Dataset, load_dataset, concatenate_datasets
### Tagengo
gpt4_dataset = load_dataset("lightblue/tagengo-gpt4", split="train")
gpt4_dataset = gpt4_dataset.filter(lambda x: x["response"][1] == "stop")
####
### Megagon
megagon_df = pd.read_json(
"https://raw.githubusercontent.com/megagonlabs/instruction_ja/main/data/data.jsonl",
lines=True,
orient="records"
)
role_map = {"user": "human", "agent": "gpt"}
megagon_df["conversations"] = megagon_df.utterances.apply(lambda x: [{"from": role_map[y["name"]], "value": y["text"]} for y in x])
megagon_df["language"] = "Japanese"
megagon_df = megagon_df[["conversations", "language"]]
megagon_dataset = Dataset.from_pandas(df)
###
### Openchat
openchat_df = pd.read_json("https://huggingface.co/datasets/openchat/openchat_sharegpt4_dataset/resolve/main/sharegpt_gpt4.json?download=true")
openchat_df["conversations"] = openchat_df["items"]
openchat_dataset = Dataset.from_pandas(openchat_df)
###
dataset = concatenate_datasets([gpt4_dataset, megagon_dataset, openchat_dataset])
dataset = dataset.filter(lambda x: not any([y["value"] is None for y in x["conversations"]]))
dataset.select_columns(["conversations"]).to_json("/workspace/llm_training/axolotl/llama3-multilingual/tagengo_openchat_megagon.json")
workspace/llm_training/axolotl/llama3-multilingual/output_tagengo_openchat_megagon_8B_llama3
This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the above described dataset. It achieves the following results on the evaluation set:
- Loss: 0.6595
Training procedure
See axolotl config
axolotl version: 0.4.0
base_model: meta-llama/Meta-Llama-3-8B-Instruct
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer # PreTrainedTokenizerFast
load_in_8bit: false
load_in_4bit: false
strict: false
datasets:
- path: /workspace/llm_training/axolotl/llama3-multilingual/tagengo_openchat_megagon.json
ds_type: json # see other options below
type: sharegpt
conversation: llama-3
dataset_prepared_path: /workspace/llm_training/axolotl/llama3-multilingual/prepared_tagengo_openchat_megagon
val_set_size: 0.01
output_dir: /workspace/llm_training/axolotl/llama3-multilingual/output_tagengo_openchat_megagon_8B_llama3
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
use_wandb: true
wandb_project: wandb_project
wandb_entity: wandb_entity
wandb_name: wandb_name
gradient_accumulation_steps: 2
micro_batch_size: 2
num_epochs: 1
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 1e-5
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
gradient_checkpointing_kwargs:
use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 10
evals_per_epoch: 5
eval_table_size:
saves_per_epoch: 1
debug:
deepspeed: /workspace/axolotl/deepspeed_configs/zero2.json
weight_decay: 0.0
special_tokens:
pad_token: <|end_of_text|>
Note - we added this Llama 3 template to fastchat directly as the Llama 3 chat template was not supported when we trained this model.
from fastchat.conversation import Conversation
from fastchat.conversation import register_conv_template
from fastchat.conversation import SeparatorStyle
register_conv_template(
Conversation(
name="llama-3",
system_template="<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{system_message}",
roles=("<|start_header_id|>user<|end_header_id|>\n", "<|start_header_id|>assistant<|end_header_id|>\n"),
sep_style=SeparatorStyle.ADD_NEW_LINE_SINGLE,
sep="<|eot_id|>",
stop_token_ids=[128009],
stop_str="<|eot_id|>",
)
)
Training hyperparameters
This model was trained using 4 x A100 (80GB) for roughly 2.5 hours.
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 2
- total_train_batch_size: 16
- total_eval_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 1
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.1894 | 0.0 | 1 | 1.0110 |
| 0.8493 | 0.2 | 73 | 0.7057 |
| 0.8047 | 0.4 | 146 | 0.6835 |
| 0.7644 | 0.6 | 219 | 0.6687 |
| 0.7528 | 0.8 | 292 | 0.6615 |
| 0.7794 | 1.0 | 365 | 0.6595 |
Framework versions
- Transformers 4.38.2
- Pytorch 2.2.1+cu121
- Datasets 2.18.0
- Tokenizers 0.15.0
Developer
Peter Devine - (ptrdvn)