SEA-LION

SEA-LION is a collection of Large Language Models (LLMs) which has been pretrained and instruct-tuned for the Southeast Asia (SEA) region. The size of the models range from 3 billion to 7 billion parameters. This is the card for the SEA-LION 3B base model.

SEA-LION stands for Southeast Asian Languages In One Network.

Model Details

Model Description

The SEA-LION model is a significant leap forward in the field of Natural Language Processing, specifically trained to understand the SEA regional context.

SEA-LION is built on the robust MPT architecture and has a vocabulary size of 256K.

For tokenization, the model employs our custom SEABPETokenizer, which is specially tailored for SEA languages, ensuring optimal model performance.

The training data for SEA-LION encompasses 980B tokens.

Developed by: Products Pillar, AI Singapore
Funded by: Singapore NRF
Model type: Decoder
Languages: English, Chinese, Indonesian, Malay, Thai, Vietnamese, Filipino, Tamil, Burmese, Khmer, Lao
License: MIT License

Performance Benchmarks

SEA-LION has an average performance on general tasks in English (as measured by Hugging Face's LLM Leaderboard):

Model	ARC	HellaSwag	MMLU	TruthfulQA	Average
SEA-LION 3B	36.26	64.59	24.07	36.46	40.35

Training Details

Data

SEA-LION was trained on 980B tokens of the following data:

Data Source	Unique Tokens	Multiplier	Total Tokens	Percentage
RefinedWeb - English	571.3B	1	571.3B	58.20%
mC4 - Chinese	91.2B	1	91.2B	9.29%
mC4 - Indonesian	3.68B	4	14.7B	1.50%
mC4 - Malay	0.72B	4	2.9B	0.29%
mC4 - Filipino	1.32B	4	5.3B	0.54%
mC4 - Burmese	1.2B	4	4.9B	0.49%
mC4 - Vietnamese	63.4B	1	63.4B	6.46%
mC4 - Thai	5.8B	2	11.6B	1.18%
WangChanBERTa - Thai	5B	2	10B	1.02%
mC4 - Lao	0.27B	4	1.1B	0.12%
mC4 - Khmer	0.97B	4	3.9B	0.40%
mC4 - Tamil	2.55B	4	10.2B	1.04%
the Stack - Python	20.9B	2	41.8B	4.26%
the Stack - Javascript	55.6B	1	55.6B	5.66%
the Stack - Shell	1.2B5	2	2.5B	0.26%
the Stack - SQL	6.4B	2	12.8B	1.31%
the Stack - Markdown	26.6B	1	26.6B	2.71%
RedPajama - StackExchange	21.2B	1	21.2B	2.16%
RedPajama - ArXiv	30.6B	1	30.6B	3.12%

Infrastructure

SEA-LION was trained using MosaicML Composer on the following hardware:

Training Details	SEA-LION 3B
AWS EC2 p4d.24xlarge	30 instances
Nvidia A100 40GB GPU	240
Training Duration	14 days

Configuration

HyperParameter	SEA-LION 3B
Precision	bfloat16
Optimizer	decoupled_adamw
Scheduler	cosine_with_warmup
Learning Rate	1.6e-4
Global Batch Size	1200
Micro Batch Size	5

Technical Specifications

Model Architecture and Objective

SEA-LION is a decoder model using the MPT architecture.

Parameter	SEA-LION 3B
Layers	32
d_model	2560
head_dim	20
Vocabulary	256000
Sequence Length	2048

Tokenizer Details

We sample 20M lines from the training data to train the tokenizer.
The framework for training is SentencePiece.
The tokenizer type is Byte-Pair Encoding (BPE).

The Team

Lam Wen Zhi Clarence
Leong Wei Qi
Li Yier
Liu Bing Jie Darius
Lovenia Holy
Montalan Jann Railey
Ng Boon Cheong Raymond
Ngui Jian Gang
Nguyen Thanh Ngan
Ong Tat-Wee David
Rengarajan Hamsawardhini
Susanto Yosephine
Tai Ngee Chia
Tan Choon Meng
Teo Jin Howe
Teo Eng Sipp Leslie
Teo Wei Yi
Tjhi William
Yeo Yeow Tong
Yong Xianbin

Acknowledgements

AI Singapore is a national programme supported by the National Research Foundation, Singapore and hosted by the National University of Singapore. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.

Contact

For more info, please contact us using this SEA-LION Inquiry Form

Link to SEA-LION's GitHub repository

Disclaimer

This the repository for the base model. The model has not been aligned for safety. Developers and users should perform their own safety fine-tuning and related security measures. In no event shall the authors be held liable for any claim, damages, or other liability arising from the use of the released weights and codes.

References

Thai Pre-Training Data Reference

@misc{lowphansirikul2021wangchanberta,
    title={WangchanBERTa: Pretraining transformer-based Thai Language Models},
    author={Lalita Lowphansirikul and Charin Polpanumas and Nawat Jantrakulchai and Sarana Nutanong},
    year={2021},
    eprint={2101.09635},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

aisingapore
/

sea-lion-3b