abhik1505040 commited on
Commit
5cde4ed
1 Parent(s): 31a5652

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +39 -24
README.md CHANGED
@@ -40,38 +40,53 @@ print("\n" + "-" * 50)
40
 
41
  ## Benchmarks
42
 
43
- | | SC | EC | DC | NER | NLI |
44
- |-------------|--------|-------|-------|----------|----------|
45
- |`Metrics` | `Accuracy` | `F1*` | `Accuracy` | `F1 (Entity)*` | `Accuracy` |
46
- |[mBERT](https://huggingface.co/bert-base-multilingual-cased) | 83.39 | 56.02 | 98.64 | 67.40 | 75.40 |
47
- |[XLM-R](https://huggingface.co/xlm-roberta-base) | 89.49 | 66.70 | 98.71 | 70.63 | 76.87 |
48
- |[sagorsarker/bangla-bert-base](https://huggingface.co/sagorsarker/bangla-bert-base) | 87.30 | 61.51 | 98.79 | 70.97 | 70.48 |
49
- [monsoon-nlp/bangla-electra](https://huggingface.co/monsoon-nlp/bangla-electra) | 73.54 | 34.55 | 97.64 | 52.57 | 63.48 |
50
- |***BanglaBERT*** | **92.18** | **74.27** | **99.07** | **72.18** | **82.94**|
 
 
 
 
 
 
 
 
 
51
 
52
- `*` - Weighted Average
53
 
54
- The benchmarking datasets are as follows:
55
- * **SC:** **[Sentiment Classification](https://ieeexplore.ieee.org/document/8554396/)**
56
- * **EC:** **[Emotion Classification](https://aclanthology.org/2021.naacl-srw.19/)**
57
- * **DC:** **[Document Classification](https://arxiv.org/abs/2005.00085)**
58
- * **NER:** **[Named Entity Recognition](https://content.iospress.com/articles/journal-of-intelligent-and-fuzzy-systems/ifs179349)**
59
- * **NLI:** **[Natural Language Inference](#datasets)**
60
 
61
 
 
 
 
 
 
 
62
  ## Citation
63
 
64
  If you use this model, please cite the following paper:
65
  ```
66
- @article{bhattacharjee2021banglabert,
67
- author = {Abhik Bhattacharjee and Tahmid Hasan and Kazi Samin and Md Saiful Islam and M. Sohel Rahman and Anindya Iqbal and Rifat Shahriyar},
68
- title = {BanglaBERT: Combating Embedding Barrier in Multilingual Models for Low-Resource Language Understanding},
69
- journal = {CoRR},
70
- volume = {abs/2101.00204},
71
- year = {2021},
72
- url = {https://arxiv.org/abs/2101.00204},
73
- eprinttype = {arXiv},
74
- eprint = {2101.00204}
 
 
 
 
 
 
 
75
  }
76
  ```
77
 
 
40
 
41
  ## Benchmarks
42
 
43
+ * Zero-shot cross-lingual transfer-learning
44
+
45
+ | Model | Params | SC (macro-F1) | NLI (accuracy) | NER (micro-F1) | QA (EM/F1) | BangLUE score |
46
+ |----------------|-----------|-----------|-----------|-----------|-----------|-----------|
47
+ |[mBERT](https://huggingface.co/bert-base-multilingual-cased) | 180M | 27.05 | 62.22 | 39.27 | 59.01/64.18 | 50.35 |
48
+ |[XLM-R (base)](https://huggingface.co/xlm-roberta-base) | 270M | 42.03 | 72.18 | 45.37 | 55.03/61.83 | 55.29 |
49
+ |[XLM-R (large)](https://huggingface.co/xlm-roberta-large) | 550M | 68.96 | 78.16 | 57.74 | 71.13/77.70 | 70.74 |
50
+
51
+ * Supervised fine-tuning
52
+
53
+ | Model | Params | SC (macro-F1) | NLI (accuracy) | NER (micro-F1) | QA (EM/F1) | BangLUE score |
54
+ |----------------|-----------|-----------|-----------|-----------|-----------|-----------|
55
+ |[mBERT](https://huggingface.co/bert-base-multilingual-cased) | 180M | 67.59 | 75.13 | 68.97 | 67.12/72.64 | 70.29 |
56
+ |[XLM-R (base)](https://huggingface.co/xlm-roberta-base) | 270M | 69.54 | 78.46 | 73.32 | 68.09/74.27 | 72.82 |
57
+ |[XLM-R (large)](https://huggingface.co/xlm-roberta-large) | 550M | 70.97 | 82.40 | 78.39 | 73.15/79.06 | 76.79 |
58
+ |[sahajBERT](https://huggingface.co/neuropark/sahajBERT) | 18M | 71.12 | 76.92 | 70.94 | 65.48/70.69 | 71.03 |
59
+ |[BanglaBERT](https://huggingface.co/csebuetnlp/banglabert) | 110M | 72.89 | 82.80 | 77.78 | 72.63/79.34 | **77.09** |
60
 
 
61
 
 
 
 
 
 
 
62
 
63
 
64
+ The benchmarking datasets are as follows:
65
+ * **SC:** **[Sentiment Classification](https://aclanthology.org/2021.findings-emnlp.278)**
66
+ * **NER:** **[Named Entity Recognition](https://multiconer.github.io/competition)**
67
+ * **NLI:** **[Natural Language Inference](https://github.com/csebuetnlp/banglabert/#datasets)**
68
+ * **QA:** **[Question Answering](https://github.com/csebuetnlp/banglabert/#datasets)**
69
+
70
  ## Citation
71
 
72
  If you use this model, please cite the following paper:
73
  ```
74
+ @inproceedings{bhattacharjee-etal-2022-banglabert,
75
+ title = {BanglaBERT: Lagnuage Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla},
76
+ author = "Bhattacharjee, Abhik and
77
+ Hasan, Tahmid and
78
+ Mubasshir, Kazi and
79
+ Islam, Md. Saiful and
80
+ Uddin, Wasi Ahmad and
81
+ Iqbal, Anindya and
82
+ Rahman, M. Sohel and
83
+ Shahriyar, Rifat",
84
+ booktitle = "Findings of the North American Chapter of the Association for Computational Linguistics: NAACL 2022",
85
+ month = july,
86
+ year = {2022},
87
+ url = {https://arxiv.org/abs/2101.00204},
88
+ eprinttype = {arXiv},
89
+ eprint = {2101.00204}
90
  }
91
  ```
92