Hugging Face's logo

SeaLLMs
/
SeaLLMs-v3-7B-Chat

Text Generation
Transformers
Safetensors
qwen2
sea
multilingual
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
3
SeaLLMs-v3-7B-Chat / README.md
IAMJB's picture
IAMJB HF staff
Update README.md
bb120c4 verified
preview code
|
raw
history blame
16.1 kB
metadata
license: other
license_name: seallms
license_link: https://huggingface.co/SeaLLMs/SeaLLM-13B-Chat/blob/main/LICENSE
language:
  - en
  - zh
  - id
  - vi
  - th
  - ms
tags:
  - sea
  - multilingual

SeaLLMs-v3 - Large Language Models for Southeast Asia

Website    🤗 Tech Memo    🤗 DEMO    Github    Technical Report

We introduce SeaLLMs-v3, the latest series of the SeaLLMs (Large Language Models for Southeast Asian languages) family. It achieves state-of-the-art performance among models with similar sizes, excelling across a diverse array of tasks such as world knowledge, mathematical reasoning, translation, and instruction following. In the meantime, it was specifically enhanced to be more trustworthy, exhibiting reduced hallucination and providing safe responses, particularly in queries closed related to Southeast Asian culture.

🔥 Highlights

  • State-of-the-art performance compared to open-source models of similar sizes, evaluated across various dimensions such as human exam questions, instruction-following, mathematics, and translation.
  • Significantly enhanced instruction-following capability, especially in multi-turn settings.
  • Ensures safety in usage with significantly reduced instances of hallucination and sensitivity to local contexts.

Uses

SeaLLMs is tailored for handling a wide range of languages spoken in the SEA region, including English, Chinese, Indonesian, Vietnamese, Thai, Tagalog, Malay, Burmese, Khmer, Lao, Tamil, and Javanese.

This page introduces the SeaLLMs-v3-7B-Chat model, specifically fine-tuned to follow human instructions effectively for task completion, making it directly applicable to your applications.

You may also refer to the SeaLLMs-v3-1.5B-Chat model which requires much lower computational resources and can be easily loaded locally.

Get started with Transformers

To quickly try the model, we show how to conduct inference with transformers below. Make sure you have installed the latest transformers version (>4.40).

from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained(
  "SeaLLMs/SeaLLMs-v3-7B-Chat", # can change to "SeaLLMs/SeaLLMs-v3-1.5B-Chat" if your resource is limited
  torch_dtype=torch.bfloat16, 
  device_map=device
)
tokenizer = AutoTokenizer.from_pretrained("SeaLLMs/SeaLLMs-v3-7B-Chat")

# prepare messages to model
prompt = "Hiii How are you?"
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
model_inputs = tokenizer([text], return_tensors="pt").to(device)
print(f"Formatted text:\n {text}")
print(f"Model input:\n {model_inputs}")

generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=512, do_sample=True, eos_token_id=tokenizer.eos_token_id)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)

print(f"Response:\n {response[0]}")

You can also utilize the following code snippet, which uses the streamer TextStreamer to enable the model to continue conversing with you:

from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import TextStreamer

device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained(
  "SeaLLMs/SeaLLMs-v3-7B-Chat",  # can change to "SeaLLMs/SeaLLMs-v3-1.5B-Chat" if your resource is limited
  torch_dtype=torch.bfloat16, 
  device_map=device
)
tokenizer = AutoTokenizer.from_pretrained("SeaLLMs/SeaLLMs-v3-7B-Chat")

# prepare messages to model
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
]

while True:
    prompt = input("User:")
    messages.append({"role": "user", "content": prompt})
    text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    model_inputs = tokenizer([text], return_tensors="pt").to(device)
    
    streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
    generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=512, streamer=streamer)
    generated_ids = [
        output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
    ]
    response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
    messages.append({"role": "assistant", "content": response})

Inference with vllm

You can also conduct inference with vllm, which is a fast and easy-to-use library for LLM inference and serving. To use vllm, first install the latest version via pip install vllm.

from vllm import LLM, SamplingParams

prompts = [
    "Who is the president of US?",
    "Can you speak Indonesian?"
]

llm = LLM(ckpt_path, dtype="bfloat16")
sparams = SamplingParams(temperature=0.1, max_tokens=512)
outputs = llm.generate(prompts, sparams)

# print out the model response
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt}\nResponse: {generated_text}\n\n")

Bias, Risks, and Limitations

Terms of Use and License: By using our released weights, codes, and demos, you agree to and comply with the terms and conditions specified in our SeaLLMs Terms Of Use.

Disclaimer: We must note that even though the weights, codes, and demos are released in an open manner, similar to other pre-trained language models, and despite our best efforts in red teaming and safety fine-tuning and enforcement, our models come with potential risks, including but not limited to inaccurate, misleading or potentially harmful generation. Developers and stakeholders should perform their own red teaming and provide related security measures before deployment, and they must abide by and comply with local governance and regulations. In no event shall the authors be held liable for any claim, damages, or other liability arising from the use of the released weights, codes, or demos.

Evaluation

We conduct our evaluation along two dimensions:

  1. Model Capability: We assess the model's performance on human exam questions, its ability to follow instructions, its proficiency in mathematics, and its translation accuracy.
  2. Model Trustworthiness: We evaluate the model's safety and tendency to hallucinate, particularly in the context of Southeast Asia.

Model Capability

Multilingual World Knowledge - M3Exam

M3Exam consists of local exam questions collected from each country. It reflects the model's world knowledge (e.g., with language or social science subjects) and reasoning abilities (e.g., with mathematics or natural science subjects).

Model en zh id th vi avg avg_sea
Sailor-7B-Chat 0.66 0.652 0.475 0.462 0.513 0.552 0.483
gemma-7b 0.732 0.519 0.475 0.46 0.594 0.556 0.510
SeaLLM-7B-v2.5 0.758 0.581 0.499 0.502 0.622 0.592 0.541
Qwen2-7B 0.815 0.874 0.53 0.479 0.628 0.665 0.546
Qwen2-7B-Instruct 0.809 0.88 0.558 0.555 0.624 0.685 0.579
Sailor-14B 0.748 0.84 0.536 0.528 0.621 0.655 0.562
Sailor-14B-Chat 0.749 0.843 0.553 0.566 0.637 0.67 0.585
SeaLLMs-v3-7B 0.814 0.866 0.549 0.52 0.628 0.675 0.566
SeaLLMs-v3-7B-Chat 0.809 0.874 0.558 0.569 0.649 0.692 0.592

Multilingual Instruction-following Capability - SeaBench

SeaBench consists of multi-turn human instructions spanning various task types. It evaluates chat-based models on their ability to follow human instructions in both single and multi-turn settings and assesses their performance across different task types. The dataset and corresponding evaluation code will be released soon!

model id
turn1		 id
turn2		 id
avg		 th
turn1		 th
turn2		 th
avg		 vi
turn1		 vi
turn2		 vi
avg		 avg
Qwen2-7B-Instruct 5.93 5.84 5.89 5.47 5.20 5.34 6.17 5.60 5.89 5.70
SeaLLM-7B-v2.5 6.27 4.96 5.62 5.79 3.82 4.81 6.02 4.02 5.02 5.15
Sailor-14B-Chat 5.26 5.53 5.40 4.62 4.36 4.49 5.31 4.74 5.03 4.97
Sailor-7B-Chat 4.60 4.04 4.32 3.94 3.17 3.56 4.82 3.62 4.22 4.03
SeaLLMs-v3-7B-Chat 6.73 6.59 6.66 6.48 5.90 6.19 6.34 5.79 6.07 6.31

Multilingual Math

We evaluate the multilingual math capability using the MGSM dataset. MGSM originally contains Chinese and Thai testing sets only, we use Google Translate to translate the same English questions into other SEA languages. Note that we adopt the tradition of each country to represent the number, e.g., in Indonesian and Vietnamese, dots are used as thousands separators and commas as decimal separators, the opposite of the English system.

MGSM en id ms th vi zh avg
Sailor-7B-Chat 33.6 22.4 22.4 21.6 25.2 29.2 25.7
Meta-Llama-3-8B-Instruct 77.6 48 57.6 56 46.8 58.8 57.5
glm-4-9b-chat 72.8 53.6 53.6 34.8 52.4 70.8 56.3
Qwen1.5-7B-Chat 64 34.4 38.4 25.2 36 53.6 41.9
Qwen2-7B-instruct 82 66.4 62.4 58.4 64.4 76.8 68.4
aya-23-8B 28.8 16.4 14.4 2 16 12.8 15.1
gemma-1.1-7b-it 58.8 32.4 34.8 31.2 39.6 35.2 38.7
SeaLLM-7B-v2.5 79.6 69.2 70.8 61.2 66.8 62.4 68.3
SeaLLMs-v3-7B-Chat 74.8 71.2 70.8 71.2 71.2 79.6 73.1

Translation

We use the test sets from Flores-200 for evaluation and report the zero-shot chrF scores for translations between every pair of languages. Each row in the table below presents the average results of translating from various source languages into the target languages. The last column displays the overall average results of translating from any language to any other language for each model.

model en id jv km lo ms my ta th tl vi zh avg
Meta-Llama-3-8B-Instruct 51.54 49.03 22.46 15.34 5.42 46.72 21.24 32.09 35.75 40.8 39.31 14.87 31.22
Qwen2-7B-Instruct 50.36 47.55 29.36 19.26 11.06 42.43 19.33 20.04 36.07 37.91 39.63 22.87 31.32
Sailor-7B-Chat 49.4 49.78 28.33 2.68 6.85 47.75 5.35 18.23 38.92 29 41.76 20.87 28.24
SeaLLM-7B-v2.5 55.09 53.71 18.13 18.09 15.53 51.33 19.71 26.1 40.55 45.58 44.56 24.18 34.38
SeaLLMs-v3-7B-Chat 54.68 52.52 29.86 27.3 26.34 45.04 21.54 31.93 41.52 38.51 43.78 26.1 36.52

Model Trustworthiness

Hallucination

Performance of whether a model can refuse questions about the non-existing entity. The following is the F1 score. We use refuse as the positive label. Our test set consists of ~1k test samples per language. Each unanswerable question is generated by GPT4o. The ratio of answerable and unanswerable questions are 1:1. We define keywords to automatically detect whether a model-generated response is a refusal response.

Refusal-F1 Scores en zh vi th id avg
Qwen1.5-7B-Instruct 53.85 51.70 52.85 35.5 58.4 50.46
Qwen2-7B-Instruct 58.79 33.08 56.21 44.6 55.98 49.732
SeaLLM-7B-v2.5 12.90 0.77 2.45 19.42 0.78 7.26
Sailor-7B-Chat 33.49 18.82 5.19 9.68 16.42 16.72
glm-4-9b-chat 44.48 37.89 18.66 4.27 1.97 21.45
aya-23-8B 6.38 0.79 2.83 1.98 14.80 5.36
Llama-3-8B-Instruct 72.08 0.00 1.23 0.80 3.91 15.60
gemma-1.1-7b-it 52.39 27.74 23.96 22.97 31.72 31.76
SeaLLMs-v3-7B-Chat 71.36 78.39 77.93 61.31 68.95 71.588

Safety

Multijaildataset consists of harmful prompts in multiple languages. We take those relevant prompts in SEA languages here and report their safe rate (the higher the better).

Model en jv th vi zh avg
Qwen2-7B-Instruct 0.8857 0.4381 0.6381 0.7302 0.873 0.713
Sailor-7B-Chat 0.7873 0.5492 0.6222 0.6762 0.7619 0.6794
Meta-Llama-3-8B-Instruct 0.8825 0.2635 0.7111 0.6984 0.7714 0.6654
Sailor-14B-Chat 0.8698 0.3048 0.5365 0.6095 0.727 0.6095
glm-4-9b-chat 0.7714 0.2127 0.3016 0.6063 0.7492 0.52824
SeaLLMs-v3-7B-Chat 0.8889 0.6000 0.7333 0.8381 0.927 0.7975

Acknowledgement to Our Linguists

We would like to express our special thanks to our professional and native linguists, Tantong Champaiboon, Nguyen Ngoc Yen Nhi and Tara Devina Putri, who helped build, evaluate, and fact-check our sampled pretraining and SFT dataset as well as evaluating our models across different aspects, especially safety.

Citation

If you find our project useful, we hope you would kindly star our repo and cite our work as follows: 

@misc{zhang2024seallms3openfoundation,
      title={SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languages}, 
      author={Wenxuan Zhang and Hou Pong Chan and Yiran Zhao and Mahani Aljunied and Jianyu Wang and Chaoqun Liu and Yue Deng and Zhiqiang Hu and Weiwen Xu and Yew Ken Chia and Xin Li and Lidong Bing},
      year={2024},
      eprint={2407.19672},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.19672}, 
}

Corresponding Author: l.bing@alibaba-inc.com