|
|
|
--- |
|
language: |
|
- ko |
|
- en |
|
license: mit |
|
tags: |
|
- electra |
|
- korean |
|
--- |
|
|
|
# Model Card for KcELECTRA: Korean comments ELECTRA |
|
|
|
|
|
# Model Details |
|
|
|
## Model Description |
|
|
|
** Updates on 2022.10.08 ** |
|
|
|
- KcELECTRA-base-v2022 (๊ตฌ v2022-dev) ๋ชจ๋ธ ์ด๋ฆ์ด ๋ณ๊ฒฝ๋์์ต๋๋ค. |
|
- ์ ๋ชจ๋ธ์ ์ธ๋ถ ์ค์ฝ์ด๋ฅผ ์ถ๊ฐํ์์ต๋๋ค. |
|
- ๊ธฐ์กด KcELECTRA-base(v2021) ๋๋น ๋๋ถ๋ถ์ downstream task์์ ~1%p ์์ค์ ์ฑ๋ฅ ํฅ์์ด ์์ต๋๋ค. |
|
|
|
--- |
|
|
|
๊ณต๊ฐ๋ ํ๊ตญ์ด Transformer ๊ณ์ด ๋ชจ๋ธ๋ค์ ๋๋ถ๋ถ ํ๊ตญ์ด ์ํค, ๋ด์ค ๊ธฐ์ฌ, ์ฑ
๋ฑ ์ ์ ์ ๋ ๋ฐ์ดํฐ๋ฅผ ๊ธฐ๋ฐ์ผ๋ก ํ์ตํ ๋ชจ๋ธ์
๋๋ค. ํํธ, ์ค์ ๋ก NSMC์ ๊ฐ์ User-Generated Noisy text domain ๋ฐ์ดํฐ์
์ ์ ์ ๋์ง ์์๊ณ ๊ตฌ์ด์ฒด ํน์ง์ ์ ์กฐ์ด๊ฐ ๋ง์ผ๋ฉฐ, ์คํ์ ๋ฑ ๊ณต์์ ์ธ ๊ธ์ฐ๊ธฐ์์ ๋ํ๋์ง ์๋ ํํ๋ค์ด ๋น๋ฒํ๊ฒ ๋ฑ์ฅํฉ๋๋ค. |
|
|
|
KcELECTRA๋ ์์ ๊ฐ์ ํน์ฑ์ ๋ฐ์ดํฐ์
์ ์ ์ฉํ๊ธฐ ์ํด, ๋ค์ด๋ฒ ๋ด์ค์์ ๋๊ธ๊ณผ ๋๋๊ธ์ ์์งํด, ํ ํฌ๋์ด์ ์ ELECTRA๋ชจ๋ธ์ ์ฒ์๋ถํฐ ํ์ตํ Pretrained ELECTRA ๋ชจ๋ธ์
๋๋ค. |
|
|
|
๊ธฐ์กด KcBERT ๋๋น ๋ฐ์ดํฐ์
์ฆ๊ฐ ๋ฐ vocab ํ์ฅ์ ํตํด ์๋นํ ์์ค์ผ๋ก ์ฑ๋ฅ์ด ํฅ์๋์์ต๋๋ค. |
|
|
|
KcELECTRA๋ Huggingface์ Transformers ๋ผ์ด๋ธ๋ฌ๋ฆฌ๋ฅผ ํตํด ๊ฐํธํ ๋ถ๋ฌ์ ์ฌ์ฉํ ์ ์์ต๋๋ค. (๋ณ๋์ ํ์ผ ๋ค์ด๋ก๋๊ฐ ํ์ํ์ง ์์ต๋๋ค.) |
|
|
|
|
|
|
|
|
|
|
|
- **Developed by:** Junbum Lee |
|
- **Shared by [Optional]:** Hugging Face |
|
- **Model type:** electra |
|
- **Language(s) (NLP):** en |
|
- **License:** MIT |
|
- **Related Models:** |
|
- **Parent Model:** Electra |
|
- **Resources for more information:** |
|
- [GitHub Repo](https://github.com/Beomi/KcBERT-finetune ) |
|
- [Model Space](https://huggingface.co/spaces/BeMerciless/korean_malicious_comment) |
|
- [Blog Post](ttps://monologg.kr/categories/NLP/ELECTRA/) |
|
|
|
# Uses |
|
|
|
|
|
## Direct Use |
|
|
|
This model can be used for the task of |
|
|
|
## Downstream Use [Optional] |
|
|
|
More information needed |
|
|
|
## Out-of-Scope Use |
|
|
|
The model should not be used to intentionally create hostile or alienating environments for people. |
|
|
|
# Bias, Risks, and Limitations |
|
|
|
Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. |
|
|
|
|
|
## Recommendations |
|
|
|
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations. |
|
|
|
|
|
# Training Details |
|
|
|
## Training Data |
|
|
|
ํ์ต ๋ฐ์ดํฐ๋ 2019.01.01 ~ 2021.03.09 ์ฌ์ด์ ์์ฑ๋ **๋๊ธ ๋ง์ ๋ด์ค/ํน์ ์ ์ฒด ๋ด์ค** ๊ธฐ์ฌ๋ค์ **๋๊ธ๊ณผ ๋๋๊ธ**์ ๋ชจ๋ ์์งํ ๋ฐ์ดํฐ์
๋๋ค. |
|
|
|
๋ฐ์ดํฐ ์ฌ์ด์ฆ๋ ํ
์คํธ๋ง ์ถ์ถ์ **์ฝ 17.3GB์ด๋ฉฐ, 1์ต8์ฒ๋ง๊ฐ ์ด์์ ๋ฌธ์ฅ**์ผ๋ก ์ด๋ค์ ธ ์์ต๋๋ค. |
|
|
|
> KcBERT๋ 2019.01-2020.06์ ํ
์คํธ๋ก, ์ ์ ํ ์ฝ 9์ฒ๋ง๊ฐ ๋ฌธ์ฅ์ผ๋ก ํ์ต์ ์งํํ์ต๋๋ค. |
|
|
|
|
|
#### Finetune Samples |
|
|
|
- NSMC with PyTorch-Lightning 1.3.0, GPU, Colab <a href="https://colab.research.google.com/drive/1Hh63kIBAiBw3Hho--BvfdUWLu-ysMFF0?usp=sharing"> |
|
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> |
|
</a> |
|
|
|
|
|
|
|
## Training Procedure |
|
|
|
|
|
### Preprocessing |
|
|
|
PLM ํ์ต์ ์ํด์ ์ ์ฒ๋ฆฌ๋ฅผ ์งํํ ๊ณผ์ ์ ๋ค์๊ณผ ๊ฐ์ต๋๋ค. |
|
|
|
1. ํ๊ธ ๋ฐ ์์ด, ํน์๋ฌธ์, ๊ทธ๋ฆฌ๊ณ ์ด๋ชจ์ง(๐ฅณ)๊น์ง! |
|
|
|
์ ๊ทํํ์์ ํตํด ํ๊ธ, ์์ด, ํน์๋ฌธ์๋ฅผ ํฌํจํด Emoji๊น์ง ํ์ต ๋์์ ํฌํจํ์ต๋๋ค. |
|
|
|
ํํธ, ํ๊ธ ๋ฒ์๋ฅผ `ใฑ-ใ
๊ฐ-ํฃ` ์ผ๋ก ์ง์ ํด `ใฑ-ํฃ` ๋ด์ ํ์๋ฅผ ์ ์ธํ์ต๋๋ค. |
|
|
|
2. ๋๊ธ ๋ด ์ค๋ณต ๋ฌธ์์ด ์ถ์ฝ |
|
|
|
`ใ
ใ
ใ
ใ
ใ
`์ ๊ฐ์ด ์ค๋ณต๋ ๊ธ์๋ฅผ `ใ
ใ
`์ ๊ฐ์ ๊ฒ์ผ๋ก ํฉ์ณค์ต๋๋ค. |
|
|
|
3. Cased Model |
|
|
|
KcBERT๋ ์๋ฌธ์ ๋ํด์๋ ๋์๋ฌธ์๋ฅผ ์ ์งํ๋ Cased model์
๋๋ค. |
|
|
|
4. ๊ธ์ ๋จ์ 10๊ธ์ ์ดํ ์ ๊ฑฐ |
|
|
|
10๊ธ์ ๋ฏธ๋ง์ ํ
์คํธ๋ ๋จ์ผ ๋จ์ด๋ก ์ด๋ค์ง ๊ฒฝ์ฐ๊ฐ ๋ง์ ํด๋น ๋ถ๋ถ์ ์ ์ธํ์ต๋๋ค. |
|
|
|
5. ์ค๋ณต ์ ๊ฑฐ |
|
|
|
์ค๋ณต์ ์ผ๋ก ์ฐ์ธ ๋๊ธ์ ์ ๊ฑฐํ๊ธฐ ์ํด ์์ ํ ์ผ์นํ๋ ์ค๋ณต ๋๊ธ์ ํ๋๋ก ํฉ์ณค์ต๋๋ค. |
|
|
|
6. `OOO` ์ ๊ฑฐ |
|
|
|
๋ค์ด๋ฒ ๋๊ธ์ ๊ฒฝ์ฐ, ๋น์์ด๋ ์์ฒด ํํฐ๋ง์ ํตํด `OOO` ๋ก ํ์ํฉ๋๋ค. ์ด ๋ถ๋ถ์ ๊ณต๋ฐฑ์ผ๋ก ์ ๊ฑฐํ์์ต๋๋ค. |
|
|
|
|
|
|
|
|
|
|
|
### Speeds, Sizes, Times |
|
|
|
More information needed |
|
|
|
# Evaluation |
|
|
|
|
|
## Testing Data, Factors & Metrics |
|
|
|
### Testing Data |
|
|
|
#### Cleaned Data |
|
|
|
- KcBERT ์ธ ์ถ๊ฐ ๋ฐ์ดํฐ๋ ์ ๋ฆฌ ํ ๊ณต๊ฐ ์์ ์
๋๋ค. |
|
|
|
|
|
### Factors |
|
|
|
|
|
### Metrics |
|
|
|
More information needed |
|
## Results |
|
|
|
|
|
(100k step๋ณ Checkpoint๋ฅผ ํตํด ์ฑ๋ฅ ํ๊ฐ๋ฅผ ์งํํ์์ต๋๋ค. ํด๋น ๋ถ๋ถ์ `KcBERT-finetune` repo๋ฅผ ์ฐธ๊ณ ํด์ฃผ์ธ์.) |
|
|
|
๋ชจ๋ธ ํ์ต Loss๋ Step์ ๋ฐ๋ผ ์ด๊ธฐ 100-200k ์ฌ์ด์ ๊ธ๊ฒฉํ Loss๊ฐ ์ค์ด๋ค๋ค ํ์ต ์ข
๋ฃ๊น์ง๋ ์ง์์ ์ผ๋ก loss๊ฐ ๊ฐ์ํ๋ ๊ฒ์ ๋ณผ ์ ์์ต๋๋ค. |
|
|
|
![KcELECTRA-base Pretrain Loss](https://cdn.jsdelivr.net/gh/beomi/blog-img@master/2021/04/07/image-20210407201231133.png) |
|
|
|
### KcELECTRA Pretrain Step๋ณ Downstream task ์ฑ๋ฅ ๋น๊ต |
|
|
|
> ๐ก ์๋ ํ๋ ์ ์ฒด ckpt๊ฐ ์๋ ์ผ๋ถ์ ๋ํด์๋ง ํ
์คํธ๋ฅผ ์งํํ ๊ฒฐ๊ณผ์
๋๋ค. |
|
|
|
![KcELECTRA Pretrain Step๋ณ Downstream task ์ฑ๋ฅ ๋น๊ต](https://cdn.jsdelivr.net/gh/beomi/blog-img@master/2021/04/07/image-20210407215557039.png) |
|
|
|
- ์์ ๊ฐ์ด KcBERT-base, KcBERT-large ๋๋น **๋ชจ๋ ๋ฐ์ดํฐ์
์ ๋ํด** KcELECTRA-base๊ฐ ๋ ๋์ ์ฑ๋ฅ์ ๋ณด์
๋๋ค. |
|
- KcELECTRA pretrain์์๋ Train step์ด ๋์ด๊ฐ์ ๋ฐ๋ผ ์ ์ง์ ์ผ๋ก ์ฑ๋ฅ์ด ํฅ์๋๋ ๊ฒ์ ๋ณผ ์ ์์ต๋๋ค. |
|
|
|
|
|
|
|
\***config์ ์ธํ
์ ๊ทธ๋๋ก ํ์ฌ ๋๋ฆฐ ๊ฒฐ๊ณผ์ด๋ฉฐ, hyperparameter tuning์ ์ถ๊ฐ์ ์ผ๋ก ํ ์ ๋ ์ข์ ์ฑ๋ฅ์ด ๋์ฌ ์ ์์ต๋๋ค.** |
|
|
|
|
|
| | Size<br/>(์ฉ๋) | **NSMC**<br/>(acc) | **Naver NER**<br/>(F1) | **PAWS**<br/>(acc) | **KorNLI**<br/>(acc) | **KorSTS**<br/>(spearman) | **Question Pair**<br/>(acc) | **KorQuaD (Dev)**<br/>(EM/F1) | |
|
| :----------------- | :-------------: | :----------------: | :--------------------: | :----------------: | :------------------: | :-----------------------: | :-------------------------: | :---------------------------: | |
|
| **KcELECTRA-base-v2022** | 475M | **91.97** | 87.35 | 76.50 | 82.12 | 83.67 | 95.12 | 69.00 / 90.40 | |
|
| **KcELECTRA-base** | 475M | 91.71 | 86.90 | 74.80 | 81.65 | 82.65 | **95.78** | 70.60 / 90.11 | |
|
| KcBERT-Base | 417M | 89.62 | 84.34 | 66.95 | 74.85 | 75.57 | 93.93 | 60.25 / 84.39 | |
|
| KcBERT-Large | 1.2G | 90.68 | 85.53 | 70.15 | 76.99 | 77.49 | 94.06 | 62.16 / 86.64 | |
|
| KoBERT | 351M | 89.63 | 86.11 | 80.65 | 79.00 | 79.64 | 93.93 | 52.81 / 80.27 | |
|
| XLM-Roberta-Base | 1.03G | 89.49 | 86.26 | 82.95 | 79.92 | 79.09 | 93.53 | 64.70 / 88.94 | |
|
| HanBERT | 614M | 90.16 | 87.31 | 82.40 | 80.89 | 83.33 | 94.19 | 78.74 / 92.02 | |
|
| KoELECTRA-Base | 423M | 90.21 | 86.87 | 81.90 | 80.85 | 83.21 | 94.20 | 61.10 / 89.59 | |
|
| KoELECTRA-Base-v2 | 423M | 89.70 | 87.02 | 83.90 | 80.61 | 84.30 | 94.72 | 84.34 / 92.58 | |
|
| KoELECTRA-Base-v3 | 423M | 90.63 | **88.11** | **84.45** | **82.24** | **85.53** | 95.25 | **84.83 / 93.45** | |
|
| DistilKoBERT | 108M | 88.41 | 84.13 | 62.55 | 70.55 | 73.21 | 92.48 | 54.12 / 77.80 | |
|
|
|
|
|
|
|
# Model Examination |
|
|
|
More information needed |
|
|
|
# Environmental Impact |
|
|
|
|
|
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). |
|
|
|
- **Hardware Type:** TPU `v3-8` |
|
- **Hours used:** 240 (10 days) |
|
- **Cloud Provider:** More information needed |
|
- **Compute Region:** More information needed |
|
- **Carbon Emitted:** More information needed |
|
|
|
# Technical Specifications [optional] |
|
|
|
## Model Architecture and Objective |
|
|
|
More information needed |
|
|
|
## Compute Infrastructure |
|
|
|
More information needed |
|
|
|
### Hardware |
|
|
|
TPU `v3-8` ์ ์ด์ฉํด ์ฝ 10์ผ ํ์ต์ ์งํํ๊ณ , ํ์ฌ Huggingface์ ๊ณต๊ฐ๋ ๋ชจ๋ธ์ 848k step์ ํ์ตํ ๋ชจ๋ธ weight๊ฐ ์
๋ก๋ ๋์ด์์ต๋๋ค. |
|
|
|
### Software |
|
- `pytorch ~= 1.8.0` |
|
- `transformers ~= 4.11.3` |
|
- `emoji ~= 0.6.0` |
|
- `soynlp ~= 0.0.493` |
|
|
|
|
|
# Citation |
|
|
|
|
|
**BibTeX:** |
|
``` |
|
|
|
@misc{lee2021kcelectra, |
|
author = {Junbum Lee}, |
|
title = {KcELECTRA: Korean comments ELECTRA}, |
|
year = {2021}, |
|
publisher = {GitHub}, |
|
journal = {GitHub repository}, |
|
howpublished = {\url{https://github.com/Beomi/KcELECTRA}} |
|
} |
|
|
|
``` |
|
๋
ผ๋ฌธ์ ํตํ ์ฌ์ฉ ์ธ์๋ MIT ๋ผ์ด์ผ์ค๋ฅผ ํ๊ธฐํด์ฃผ์ธ์. โบ๏ธ |
|
|
|
# Glossary [optional] |
|
More information needed |
|
|
|
# More Information [optional] |
|
|
|
``` |
|
๐ก NOTE ๐ก |
|
General Corpus๋ก ํ์ตํ KoELECTRA๊ฐ ๋ณดํธ์ ์ธ task์์๋ ์ฑ๋ฅ์ด ๋ ์ ๋์ฌ ๊ฐ๋ฅ์ฑ์ด ๋์ต๋๋ค. |
|
KcBERT/KcELECTRA๋ User genrated, Noisy text์ ๋ํด์ ๋ณด๋ค ์ ๋์ํ๋ PLM์
๋๋ค. |
|
``` |
|
|
|
## Acknowledgement |
|
|
|
KcELECTRA Model์ ํ์ตํ๋ GCP/TPU ํ๊ฒฝ์ [TFRC](https://www.tensorflow.org/tfrc?hl=ko) ํ๋ก๊ทธ๋จ์ ์ง์์ ๋ฐ์์ต๋๋ค. |
|
|
|
๋ชจ๋ธ ํ์ต ๊ณผ์ ์์ ๋ง์ ์กฐ์ธ์ ์ฃผ์ [Monologg](https://github.com/monologg/) ๋ ๊ฐ์ฌํฉ๋๋ค :) |
|
|
|
### Github Repos |
|
|
|
- [KcBERT by Beomi](https://github.com/Beomi/KcBERT) |
|
- [BERT by Google](https://github.com/google-research/bert) |
|
- [KoBERT by SKT](https://github.com/SKTBrain/KoBERT) |
|
- [KoELECTRA by Monologg](https://github.com/monologg/KoELECTRA/) |
|
- [Transformers by Huggingface](https://github.com/huggingface/transformers) |
|
- [Tokenizers by Hugginface](https://github.com/huggingface/tokenizers) |
|
- [ELECTRA train code by KLUE](https://github.com/KLUE-benchmark/KLUE-ELECTRA) |
|
|
|
|
|
# Model Card Authors [optional] |
|
|
|
|
|
Junbum Lee in collaboration with Ezi Ozoani and the Hugging Face team |
|
|
|
# Model Card Contact |
|
|
|
More information needed |
|
|
|
# How to Get Started with the Model |
|
|
|
Use the code below to get started with the model. |
|
|
|
<details> |
|
<summary> Click to expand </summary> |
|
|
|
```bash |
|
pip install soynlp emoji |
|
``` |
|
|
|
์๋ `clean` ํจ์๋ฅผ Text data์ ์ฌ์ฉํด์ฃผ์ธ์. |
|
|
|
```python |
|
import re |
|
import emoji |
|
from soynlp.normalizer import repeat_normalize |
|
|
|
emojis = ''.join(emoji.UNICODE_EMOJI.keys()) |
|
pattern = re.compile(f'[^ .,?!/@$%~๏ผ
ยทโผ()\x00-\x7Fใฑ-ใ
ฃ๊ฐ-ํฃ{emojis}]+') |
|
url_pattern = re.compile( |
|
r'https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&//=]*)') |
|
|
|
import re |
|
import emoji |
|
from soynlp.normalizer import repeat_normalize |
|
|
|
pattern = re.compile(f'[^ .,?!/@$%~๏ผ
ยทโผ()\x00-\x7Fใฑ-ใ
ฃ๊ฐ-ํฃ]+') |
|
url_pattern = re.compile( |
|
r'https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&//=]*)') |
|
|
|
def clean(x): |
|
x = pattern.sub(' ', x) |
|
x = emoji.replace_emoji(x, replace='') #emoji ์ญ์ |
|
x = url_pattern.sub('', x) |
|
x = x.strip() |
|
x = repeat_normalize(x, num_repeats=2) |
|
return x |
|
``` |
|
|
|
|
|
</details> |
|
|
|
|