Create README.md

README.md

@@ -0,0 +1,149 @@
+---
+license: other
+license_name: seallm
+license_link: https://huggingface.co/SeaLLMs/SeaLLM-13B-Chat/blob/main/LICENSE
+language:
+- en
+- zh
+- hi
+- es
+- fr
+- ar
+- bn
+- ru
+- pt
+- id
+- ur
+- de
+- ja
+- sw
+- ta
+- tr
+- ko
+- vi
+- jv
+- it
+- ha
+- th
+- fa
+- tl
+- my
+tags:
+- multilingual
+- babel
+---
+
+
+# *Babel*: Open Multilingual Large Language Models Serving Over 90% of Global Speakers
+
+<p align="center">
+<a href="https://babel-llm.github.io/babel-llm/" target="_blank" rel="noopener">Website</a>
+&nbsp;&nbsp;
+<a href="https://huggingface.co/Tower-Babel/Babel-9B/" target="_blank" rel="noopener">Model</a>
+&nbsp;&nbsp;
+<a href="https://github.com/babel-llm/babel-llm" target="_blank" rel="noopener">Github</a>
+&nbsp;&nbsp;
+<a href="https://github.com/babel-llm/babel-llm/blob/main/paper/babel.pdf" target="_blank" rel="noopener">Technical Report</a>
+</p>
+
+## Introduction
+
+We introduce **Babel**, a multilingual LLM that covers the top 25 languages by number of speakers, including English, Chinese, Hindi, Spanish, Arabic, French, Bengali, Portuguese, Russian, Urdu, Indonesian, German, Japanese, Swahili, Filipino, Tamil, Vietnamese, Turkish, Italian, Javanese, Korean, Hausa, Persian, Thai, and Burmese. These 25 languages support over 90% of the global population, and include many languages neglected by other open multilingual LLMs. Unlike traditional continued pretraining approaches, Babel expands its parameter count through a layer extension technique that elevates Babel's performance ceiling.
+
+We introduce two variants:  
+- **Babel-9B**, designed for efficient single-GPU inference and fine-tuning  
+- **Babel-83B**, which sets a new standard for open multilingual LLMs  
+
+Extensive evaluations on multilingual tasks demonstrate its superior performance compared to open LLMs of comparable size. In addition, using existing supervised fine-tuning datasets, Babel achieves remarkable performance, with **Babel-9B-Chat** leading among 10B-sized LLMs and **Babel-83B-Chat** setting a new standard for open LLMs, performing comparably to GPT-4o on certain tasks.
+
+This page introduces the **Babel-9B-Base** model
+
+## Supervised Fine-Tuning (SFT) Data
+
+We primarily leverage open-source multilingual SFT training corpora and translated SFT training data. Specifically, we utilize WildChat, a dataset comprising 1 million user-ChatGPT conversations with over 2.5 million interaction turns. Additionally, we employ Everything Instruct Multilingual, an extensive Alpaca-instruct-formatted dataset covering a diverse range of topics. 
+
+
+## Evaluation
+
+We employ multilingual tasks across several categories:
+
+1. **World Knowledge:**  
+   MMMLU ([OpenAI 2024](https://huggingface.co/datasets/openai/MMMLU)), a human-translated version of MMLU ([Hendrycks et al. 2021](https://arxiv.org/abs/2009.03300)) available in 14 languages. For languages not covered, we use Google Translate ([Google Translate API](https://translate.google.com/)) to generate translations. Additionally, we include M3Exam ([M3Exam](https://arxiv.org/abs/2306.05179)), which consists of authentic human exam questions collected from various countries, covering multiple subjects and educational levels.
+
+2. **Reasoning:**  
+   MGSM ([Shi et al. 2022](https://arxiv.org/abs/2210.03057)) and XCOPA ([Ponti et al. 2020](https://aclanthology.org/2020.emnlp-main.185/)).
+
+3. **Understanding:**  
+   XNLI ([Conneau et al. 2018](https://arxiv.org/abs/1809.05053)).
+
+4. **Translation:**  
+   Flores-200 ([NLLB Team 2022](https://arxiv.org/abs/2207.04672)).
+
+
+
+### Get started with `Transformers`
+
+To quickly try the model, we show how to conduct inference with `transformers` below. Make sure you have installed the latest transformers version (>4.40).
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+device = "cuda" # the device to load the model onto
+
+model = AutoModelForCausalLM.from_pretrained(
+  "Tower-Babel/Babel-9B-Chat",
+  torch_dtype=torch.bfloat16, 
+  device_map=device
+)
+tokenizer = AutoTokenizer.from_pretrained("Tower-Babel/Babel-9B-Chat")
+
+# prepare messages to model
+prompt = "Hiii How are you?"
+messages = [
+    {"role": "system", "content": "You are a helpful assistant."},
+    {"role": "user", "content": prompt}
+]
+
+text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+model_inputs = tokenizer([text], return_tensors="pt").to(device)
+print(f"Formatted text:\n {text}")
+print(f"Model input:\n {model_inputs}")
+
+generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=512, do_sample=True, eos_token_id=tokenizer.eos_token_id)
+generated_ids = [
+    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
+]
+response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
+
+print(f"Response:\n {response[0]}")
+```
+
+### Performance of 10B-Size Instruct Models vs. Babel-9B-Chat
+
+| **Dataset**      | **GLM4-9B** | **Gemma2-9B** | **Mistral-12B** | **Llama3.1-8B** | **Qwen2.5-7B** | **Babel-9B** |
+|-------------------|-------------|---------------|-----------------|-----------------|----------------|--------------|
+| MMMLU            | 53.9        | 59.6          | 52.0            | 50.6            | 56.0           | **59.8**     |
+| M3Exam           | 55.0        | **63.2**      | 54.1            | 54.2            | 58.0           | 62.9         |
+| XCOPA            | 86.2        | 87.4          | 83.5            | 82.1            | 80.4           | **88.9**     |
+| MGSM             | 52.2        | 62.4          | 41.4            | 37.2            | 59.1           | **64.3**     |
+| XNLI             | 66.2        | 66.7          | 56.1            | 55.8            | 68.3           | **72.4**     |
+| Flores-200       | 50.8        | 54.8          | 48.9            | 47.3            | 45.8           | **56.7**     |
+| *Average*        | 60.7        | 65.7          | 56.0            | 54.5            | 61.3           | **67.5**     |
+
+**Note that results are achieved purely by leveraging publicly available datasets, showcasing the robust foundational performance of Babel base models. We believe that incorporating more SFT data across diverse types, domains, and formats, along with additional alignment data and preference tuning, will further enhance the chat version beyond its current capabilities.**
+
+## Acknowledgement
+We would like to thank Guanzheng Chen for assisting with the implementation of the training codebase. Our special thanks go to our professional and native linguists—Tantong Champaiboon, Nguyen Ngoc Yen Nhi, and Tara Devina Putri—who contributed to building, evaluating, and fact-checking our sampled pretraining dataset. We also appreciate Fan Wang, Jiasheng Tang, Xin Li, and Hao Zhang for their efforts in coordinating computing resources.
+
+## Citation
+
+If you find our project useful, we hope you would kindly star our repo and cite our work as follows: 
+```
+@article{babel2025,
+      author = {Yiran Zhao and Chaoqun Liu and Yue Deng and Jiahao Ying and Mahani Aljunied and Zhaodonghui Li and Lidong Bing and Hou Pong Chan and Yu Rong and Deli Zhao and Wenxuan Zhang},
+      title = {Babel: Open Multilingual Large Language Models Serving Over 90\% of Global Speakers},
+      year = {2025},
+      note = {Available online: \url{https://github.com/babel-llm/babel-llm/blob/main/paper/babel.pdf}}
+    }
+```
+Corresponding Author: wxzhang@sutd.edu.sg