---
license: mit
---
# SEA-LION

SEA-LION is a collection of Large Language Models (LLMs) which has been pretrained and instruct-tuned for the Southeast Asia (SEA) region.
The size of the models range from 3 billion to 7 billion parameters.
This is the card for the SEA-LION 3B model.

SEA-LION stands for <i>Southeast Asian Languages In One Network</i>.


## Model Details

### Model Description

The SEA-LION model is a significant leap forward in the field of Natural Language Processing,
specifically trained to understand the SEA regional context.

SEA-LION is built on the robust MPT architecture and has a vocabulary size of 256K.

For tokenization, the model employs our custom SEABPETokenizer, which is specially tailored for SEA languages, ensuring optimal model performance.

The training data for SEA-LION encompasses 980B tokens.

- **Developed by:** Products Pillar, AI Singapore
- **Funded by:** Singapore NRF
- **Model type:** Decoder
- **Languages:** English, Chinese, Indonesian, Malay, Thai, Vietnamese, Filipino, Tamil, Burmese, Khmer, Lao
- **License:** MIT License


## Training Details

### Data

SEA-LION was trained on 980B tokens of the following data:

| Data Source               | Tokens | Percentage |
|---------------------------|-------:|:----------:|
| RefinedWeb - English      | 571.3B |     58.20% |
| mC4 - Chinese             |  91.2B |      9.29% |
| mC4 - Indonesian          |  14.7B |      1.50% |
| mC4 - Malay               |   2.9B |      0.29% |
| mC4 - Filipino            |   5.3B |      0.54% |
| mC4 - Burmese             |   4.9B |      0.49% |
| mC4 - Vietnamese          |  63.4B |      6.46% |
| mC4 - Thai                |  21.6B |      2.20% |
| mC4 - Lao                 |   1.1B |      0.12% |
| mC4 - Khmer               |   3.9B |      0.40% |
| mC4 - Tamil               |  10.2B |      1.04% |
| the Stack - Python        |  41.8B |      4.26% |
| the Stack - Javascript    |  55.6B |      5.66% |
| the Stack - Shell         |   2.5B |      0.26% |
| the Stack - SQL           |  12.8B |      1.31% |
| the Stack - Markdown      |  26.6B |      2.71% |
| RedPajama - StackExchange |  21.2B |      2.16% |
| RedPajama - ArXiv         |  30.6B |      3.12% |

### Infrastructure

SEA-LION was trained using [MosaicML Composer](https://github.com/mosaicml/composer)
on the following hardware:

| Training Details     | SEA-LION 3B  |
|----------------------|:------------:|
| AWS EC2 p4d.24xlarge | 30 instances |
| Nvidia A100 40GB GPU | 240          |
| Training Duration    | 14 days      |


### Configuration

| HyperParameter    | SEA-LION 3B        |
|-------------------|:------------------:|
| Precision         | bfloat16           |
| Optimizer         | decoupled_adamw    |
| Scheduler         | cosine_with_warmup |
| Learning Rate     | 1.6e-4             |
| Global Batch Size | 1200               |
| Micro Batch Size  | 5                  |


## Technical Specifications

### Model Architecture and Objective

SEA-LION is a decoder model using the MPT architecture.

| Parameter       | SEA-LION 3B |
|-----------------|:-----------:|
| Layers          | 32          |
| d_model         | 2560        |
| head_dim        | 20          |
| Vocabulary      | 256000      |
| Sequence Length | 2048        |


### Tokenizer Details

We sample 20M lines from the training data to train the tokenizer.<br>
The framework for training is [SentencePiece](https://github.com/google/sentencepiece).<br>
The tokenizer type is Byte-Pair Encoding (BPE).



## The Team

Lam Zhiwen Clarence<br>
Leong Wei Qi<br>
Li Yier<br>
Liu Darius<br>
Lovenia Holy<br>
Montalan Jann Railey<br>
Ng Raymond<br>
Ngui Jian Gang<br>
Nguyen Thanh Ngan<br>
Ong Tat-Wee David<br>
Rengarajan Hamsawardhini<br>
Susanto Yosephine<br>
Tai Ngee Chia<br>
Tan Choon Meng<br>
Teo Jin Howe<br>
Teo Leslie<br>
Teo Wei Yi<br>
Tjhi William<br>
Yeo Yeow Tong<br>
Yong Xianbin<br>

## Acknowledgements

AI Singapore is a national programme supported by the National Research Foundation, Singapore and hosted by the National University of Singapore.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.

## Contact

For more info, please contact us at sealion@aisingapore.org