metadata
license: apache-2.0
language:
- yo
oyo-teams-base-discriminator
OYO-BERT (or Oyo-dialect of Yoruba BERT) was created by pre-training a TEAMS model based on ELECTRA architecture on Yoruba language texts for about 100K steps. It was trained using ELECTRA-base architecture with Tensorflow Model Garden
Pre-training corpus
A mix of WURA, Wikipedia and MT560 Yoruba data
Acknowledgment
We thank @stefan-it for providing the pre-processing and pre-training scripts. Finally, we would like to thank Google Cloud for providing us access to TPU v3-8 through the free cloud credits. Model trained using flax, before converted to pytorch.