theyorubayesian commited on
Commit
ccd9ab8
1 Parent(s): a6e4ae9

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +21 -13
README.md CHANGED
@@ -44,18 +44,26 @@ In this study, we highlight the importance of enhancing the quality of pretraini
44
  ## Citation Information
45
 
46
  ```bibtex
47
- @article{OladipoBQPD2023EMNLP,
48
- title = "Better Quality Pre-training Data and T5 Models for African Languages",
49
- author = "Oladipo, Akintunde and
50
- Adeyemi, Mofetoluwa and
51
- Ahia, Orevaoghene and
52
- Owodunni, Abraham and
53
- Ogundepo, Odunayo and
54
- Adelani, David and
55
- Lin, Jimmy
56
- ",
57
- booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
58
- publisher = "Association for Computational Linguistics",
59
- year = "2023",
 
 
 
 
 
 
 
60
  }
 
61
  ```
 
44
  ## Citation Information
45
 
46
  ```bibtex
47
+ @inproceedings{oladipo-etal-2023-better,
48
+ title = "Better Quality Pre-training Data and T5 Models for {A}frican Languages",
49
+ author = "Oladipo, Akintunde and
50
+ Adeyemi, Mofetoluwa and
51
+ Ahia, Orevaoghene and
52
+ Owodunni, Abraham and
53
+ Ogundepo, Odunayo and
54
+ Adelani, David and
55
+ Lin, Jimmy",
56
+ editor = "Bouamor, Houda and
57
+ Pino, Juan and
58
+ Bali, Kalika",
59
+ booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
60
+ month = dec,
61
+ year = "2023",
62
+ address = "Singapore",
63
+ publisher = "Association for Computational Linguistics",
64
+ url = "https://aclanthology.org/2023.emnlp-main.11",
65
+ pages = "158--168",
66
+ abstract = "In this study, we highlight the importance of enhancing the quality of pretraining data in multilingual language models. Existing web crawls have demonstrated quality issues, particularly in the context of low-resource languages. Consequently, we introduce a new multilingual pretraining corpus for 16 African languages, designed by carefully auditing existing pretraining corpora to understand and rectify prevalent quality issues. To compile this dataset, we undertake a rigorous examination of current data sources for thirteen languages within one of the most extensive multilingual web crawls, mC4, and extract cleaner data through meticulous auditing and improved web crawling strategies. Subsequently, we pretrain a new T5-based model on this dataset and evaluate its performance on multiple downstream tasks. Our model demonstrates better downstream effectiveness over existing pretrained models across four NLP tasks, underscoring the critical role data quality plays in pretraining language models in low-resource scenarios. Specifically, on cross-lingual QA evaluation, our new model is more than twice as effective as multilingual T5. All code, data and models are publicly available at https://github.com/castorini/AfriTeVa-keji.",
67
  }
68
+
69
  ```