SlowGuess/ACL_2020_OCR huggingface dataset:
- Get unique sentences (about 20K sentences).
- The model is likely to classify as Arabizi sentences which contain any letter repeated a lot of times in a row (e.g. "iiiiii...", "uuuuuuu...", "ffffff..."). These sentences are deleted from the dataset.
- Due to the n-gram embeddings, "lah" is considered as a marker of Arabizi, so a lot of sentences containing "blah" are classified as Arabizi. If this phrase appears more than 5 times, the sentence is deleted.
- Figures and numbers are widespread in Arabisi (particularly "3" and "9"), so the model classifies too many sentences which contains only numbers. Therefore, sentences which have more "number of figures" characters than  $80\%$  of the number of characters (excluded figures) are deleted.