File size: 3,730 Bytes
8e85059 2c1575f e93a827 2c1575f 8e85059 2c1575f 50b5a3d 8e85059 2c1575f 0ee90b1 8e85059 2c1575f 03056cd 2c1575f 0ee90b1 2c1575f 3d0b57f 8e85059 2c1575f 2c959b8 2c1575f 2c959b8 2c1575f 8e85059 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 |
---
license: apache-2.0
datasets:
- klue/klue
language:
- ko
metrics:
- f1
- accuracy
- pearsonr
---
# RoBERTa-base Korean
## ๋ชจ๋ธ ์ค๋ช
์ด RoBERTa ๋ชจ๋ธ์ ๋ค์ํ ํ๊ตญ์ด ํ
์คํธ ๋ฐ์ดํฐ์
์์ **์์ ** ๋จ์๋ก ์ฌ์ ํ์ต๋์์ต๋๋ค.
์์ฒด ๊ตฌ์ถํ ํ๊ตญ์ด ์์ ๋จ์ vocab์ ์ฌ์ฉํ์์ต๋๋ค.
## ์ํคํ
์ฒ
- **๋ชจ๋ธ ์ ํ**: RoBERTa
- **์ํคํ
์ฒ**: RobertaForMaskedLM
- **๋ชจ๋ธ ํฌ๊ธฐ**: 256 hidden size, 8 hidden layers, 8 attention heads
- **max_position_embeddings**: 514
- **intermediate_size**: 2,048
- **vocab_size**: 1,428
## ํ์ต ๋ฐ์ดํฐ
์ฌ์ฉ๋ ๋ฐ์ดํฐ์
์ ๋ค์๊ณผ ๊ฐ์ต๋๋ค:
- **๋ชจ๋์๋ง๋ญ์น**: ์ฑํ
, ๊ฒ์ํ, ์ผ์๋ํ, ๋ด์ค, ๋ฐฉ์ก๋๋ณธ, ์ฑ
๋ฑ
- **AIHUB**: SNS, ์ ํ๋ธ ๋๊ธ, ๋์ ๋ฌธ์ฅ
- **๊ธฐํ**: ๋๋ฌด์ํค, ํ๊ตญ์ด ์ํคํผ๋์
์ด ํฉ์ฐ๋ ๋ฐ์ดํฐ๋ **์ฝ 11GB** ์
๋๋ค. **(4B tokens)**
## ํ์ต ์์ธ
- **BATCH_SIZE**: 112 (GPU๋น)
- **ACCUMULATE**: 36
- **Total_BATCH_SIZE**: 8,064
- **MAX_STEPS**: 12,500
- **TRAIN_STEPS * BATCH_SIZE**: **100M**
- **WARMUP_STEPS**: 2,400
- **์ต์ ํ**: AdamW, LR 1e-3, BETA (0.9, 0.98), eps 1e-6
- **ํ์ต๋ฅ ๊ฐ์ **: linear
- **์ฌ์ฉ๋ ํ๋์จ์ด**: 2x RTX 8000 GPU
![Evaluation Loss Graph](https://cdn-uploads.huggingface.co/production/uploads/64a0fd6fd3149e05bc5260dd/-64jKdcJAavwgUREwaywe.png)
![Evaluation Accuracy Graph](https://cdn-uploads.huggingface.co/production/uploads/64a0fd6fd3149e05bc5260dd/LPq5M6S8LTwkFSCepD33S.png)
## ์ฌ์ฉ ๋ฐฉ๋ฒ
### tokenizer์ ๊ฒฝ์ฐ wordpiece๊ฐ ์๋ syllable ๋จ์์ด๊ธฐ์ AutoTokenizer๊ฐ ์๋๋ผ SyllableTokenizer๋ฅผ ์ฌ์ฉํด์ผ ํฉ๋๋ค.
### (๋ ํฌ์์ ์ ๊ณตํ๊ณ ์๋ syllabletokenizer.py๋ฅผ ๊ฐ์ ธ์์ ์ฌ์ฉํด์ผ ํฉ๋๋ค.)
## ์ฑ๋ฅ ํ๊ฐ
- **KLUE benchmark test๋ฅผ ํตํด์ ์ฑ๋ฅ์ ํ๊ฐํ์ต๋๋ค.**
- klue-roberta-base์ ๋นํด์ ๋งค์ฐ ์์ ํฌ๊ธฐ๋ผ ์ฑ๋ฅ์ด ๋ฎ๊ธฐ๋ ํ์ง๋ง hidden size 512์ธ ๋ชจ๋ธ์ ํฌ๊ธฐ ๋๋น ์ข์ ์ฑ๋ฅ์ ๋ณด์์ต๋๋ค.
![image/png](https://cdn-uploads.huggingface.co/production/uploads/64a0fd6fd3149e05bc5260dd/I8e60cf9w-IQCHDgKiooq.png)
![image/png](https://cdn-uploads.huggingface.co/production/uploads/64a0fd6fd3149e05bc5260dd/hkc5ko9Vo-pkKmtouN7xc.png)
## ์ฌ์ฉ ๋ฐฉ๋ฒ
### tokenizer์ ๊ฒฝ์ฐ wordpiece๊ฐ ์๋ syllable ๋จ์์ด๊ธฐ์ AutoTokenizer๊ฐ ์๋๋ผ SyllableTokenizer๋ฅผ ์ฌ์ฉํด์ผ ํฉ๋๋ค.
### (๋ ํฌ์์ ์ ๊ณตํ๊ณ ์๋ syllabletokenizer.py๋ฅผ ๊ฐ์ ธ์์ ์ฌ์ฉํด์ผ ํฉ๋๋ค.)
```python
from transformers import AutoModel, AutoTokenizer
from syllabletokenizer import SyllableTokenizer
# ๋ชจ๋ธ๊ณผ ํ ํฌ๋์ด์ ๋ถ๋ฌ์ค๊ธฐ
model = AutoModelForMaskedLM.from_pretrained("Trofish/korean_syllable_roberta")
tokenizer = SyllableTokenizer(vocab_file='vocab.json',**tokenizer_kwargs)
# ํ
์คํธ๋ฅผ ํ ํฐ์ผ๋ก ๋ณํํ๊ณ ์์ธก ์ํ
inputs = tokenizer("์ฌ๊ธฐ์ ํ๊ตญ์ด ํ
์คํธ ์
๋ ฅ", return_tensors="pt")
outputs = model(**inputs)
```
## Citation
**klue**
```
@misc{park2021klue,
title={KLUE: Korean Language Understanding Evaluation},
author={Sungjoon Park and Jihyung Moon and Sungdong Kim and Won Ik Cho and Jiyoon Han and Jangwon Park and Chisung Song and Junseong Kim and Yongsook Song and Taehwan Oh and Joohong Lee and Juhyun Oh and Sungwon Lyu and Younghoon Jeong and Inkwon Lee and Sangwoo Seo and Dongjun Lee and Hyunwoo Kim and Myeonghwa Lee and Seongbo Jang and Seungwon Do and Sunkyoung Kim and Kyungtae Lim and Jongwon Lee and Kyumin Park and Jamin Shin and Seonghyun Kim and Lucy Park and Alice Oh and Jungwoo Ha and Kyunghyun Cho},
year={2021},
eprint={2105.09680},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```
|