Before revision:  Overall, we identify six elements that are potentially necessary for BERT to be multilingual. Architectural factors that contribute to multilinguality are underparameterization, shared special tokens (e.g., " CLS "), shared position embeddings and replacing masked tokens with random tokens.
After revision:  Overall, we identify CLS "), shared position embeddings and replacing masked tokens with random tokens.