language:
- vi
metrics:
- f1
pipeline_tag: token-classification
tags:
- transformer
- vietnamese
- nlp
- bert
- deberta
- deberta-v3
ViDeBERTa: A powerful pre-trained language model for Vietnamese
ViDeBERTa, a new pre-trained monolingual language model for Vietnamese, with three versions - ViDeBERTa_xsmall, ViDeBERTa_base, and ViDeBERTa_large, which are pre-trained on 138GB of Vietnamese text of high-quality and diverse Vietnamese text using DeBERTaV3 architecture.
Please check the official repository for more implementation details and updates
The DeBERTa V3 xsmall model comes with 12 layers and a hidden size of 384. It has only 22M backbone parameters with a vocabulary containing 128K tokens which introduces 48M parameters in the Embedding layer. This model was trained using CC100 dataset, which consists of 138 GB of Vietnamese text.
Fine-tuning on NLU tasks
We present the dev results on VLSP POS, PhoNER, ViQuAD dataset.
| Model | #Params(M) | POS | NER | MRC |
|---|---|---|---|---|
| XLM-R-base | 125M | 96.2 | - | 82.0 |
| XLM-R-large | 355M | 96.3 | 93.8 | 87.0 |
| PhoBERT-base | 135M | 96.7 | 80.1 | |
| PhoBERT-large | 370M | 96.8 | 83.5 | |
| ViT5-base | 310M | - | 94.5 | - |
| ViT5-large | 866M | - | 93.8 | - |
| ViDeBERTa-xsmall | 22M | 96.4 | 93.6 | 81.3 |
| ViDeBERTa-base | 86M | 96.8 | 94.5 | 85.7 |
| ViDeBERTa-large | 304M | 97.2 | 95.3 | 89.9 |
Citation
If you find ViDeBERTa useful for your work, please cite the following papers:
@article{dao2023videberta,
title={ViDeBERTa: A powerful pre-trained language model for Vietnamese},
author={Dao Tran, Cong and Pham, Nhut Huy and Nguyen, Anh and Son Hy, Truong and Vu, Tu},
journal={arXiv e-prints},
pages={arXiv--2301},
year={2023}
}