Before revision:  Although multilingual MLMs have been introduced to serve many languages, these have limitations as to their capacity and the size and diversity of non-English data they are pre-trained on. In this work, we remedy these issues for Arabic by introducing two powerful deep bidirectional transformer-based models, ARBERT and MARBERT , that have superior performance to all existing models .
After revision:  Although multilingual MLMs have been introduced to serve many languages, these have limitations as to their capacity and the size and diversity of non-English data involved in their pre-training. We remedy these issues for Arabic by introducing two powerful deep bidirectional transformer-based models, ARBERT and MARBERT , that have superior performance to all existing models .