Hugging Face's logo

aisingapore
/
sealion-bert-base

Fill-Mask
Transformers
Safetensors
11 languages
bert
custom_code
Model card Files Community
sealion-bert-base
File size: 3,932 Bytes 
426f887
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3455559
426f887
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2eb8615
426f887
 
 
 
 
 
 
 
 
 
1c74a03
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
 
---
license: mit
language:
- en
- zh
- id
- ms
- th
- vi
- tl
- ta
- my
- km
- lo
inference: false
---
# SEA-LION-BERT

SEA-LION stands for <i>Southeast Asian Languages In One Network</i>.

This is the card for the SEA-LION-BERT base model.

## How To Use

```python
from transformers import AutoModelForMaskedLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('aisingapore/sealion-bert-base', trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained('aisingapore/sealion-bert-base', trust_remote_code=True)

# prepare input
text = "Give me a <|mask|>!!!"
encoded_input = tokenizer(text, return_tensors='pt')

```

## Model Details

### Model Description

The SEA-LION-BERT model is built on the MosaicBERT architecture and has a vocabulary size of 256K.

For tokenization, the model employs our custom SEABPETokenizer, which is specially tailored for SEA languages, ensuring optimal model performance.

The training data for SEA-LION-BERT encompasses 790B tokens.

- **Developed by:** Products Pillar, AI Singapore
- **Funded by:** Singapore NRF
- **Model type:** Encoder
- **Languages:** English, Chinese, Indonesian, Malay, Thai, Vietnamese, Filipino, Tamil, Burmese, Khmer, Lao
- **License:** MIT License

## Training Details

### Data

SEA-LION was trained on 790B tokens of the following data:

| Data Source               | Tokens | Percentage |
|---------------------------|-------:|:----------:|
| RefinedWeb - English      | 571.3B |     72.26% |
| mC4 - Chinese             |  91.2B |     11.54% |
| mC4 - Indonesian          |  14.7B |      1.86% |
| mC4 - Malay               |   2.9B |      0.36% |
| mC4 - Filipino            |   5.3B |      0.67% |
| mC4 - Burmese             |   4.9B |      0.61% |
| mC4 - Vietnamese          |  63.4B |      8.02% |
| mC4 - Thai                |  21.6B |      2.74% |
| mC4 - Lao                 |   1.1B |      0.14% |
| mC4 - Khmer               |   3.9B |      0.50% |
| mC4 - Tamil               |  10.2B |      1.29% |

### Infrastructure

SEA-LION was trained using [MosaicML Composer](https://github.com/mosaicml/composer)
on the following hardware:

| Training Details     | SEA-LION-BERT |
|----------------------|:-------------:|
| Nvidia A100 40GB GPU | 4             |
| Training Duration    | 14 days       |


### Configuration

| HyperParameter    | SEA-LION-BERT           |
|-------------------|:-----------------------:|
| Precision         | bfloat16                |
| Optimizer         | decoupled_adamw         |
| Scheduler         | linear_decay_with_warmup|
| Learning Rate     | 5e-4                    |
| Global Batch Size | 448                     |
| Micro Batch Size  | 56                      |


## Technical Specifications

### Model Architecture and Objective

SEA-LION-BERT is an encoder model using the MosaicBERT architecture.

| Parameter       | SEA-LION-BERT |
|-----------------|:-------------:|
| Layers          | 12            |
| d_model         | 768           |
| head_dim        | 12            |
| Vocabulary      | 256000        |
| Sequence Length | 128           |


### Tokenizer Details

We sample 20M lines from the training data to train the tokenizer.<br>
The framework for training is [SentencePiece](https://github.com/google/sentencepiece).<br>
The tokenizer type is Byte-Pair Encoding (BPE).


## The Team

Montalan Jann Railey<br>
Nguyen Thanh Ngan<br>
Rengarajan Hamsawardhini<br>
Teo Eng Sipp Leslie<br>
Tjhi William<br>


## Acknowledgements

AI Singapore is a national programme supported by the National Research Foundation, Singapore and hosted by the National University of Singapore.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.

## Contact

For more info, please contact us using this [SEA-LION Inquiry Form](https://forms.gle/sLCUVb95wmGf43hi6)