Before revision:  Architectural factors that contribute to multilinguality are underparameterization, shared special tokens (e.g., " CLS "), shared position embeddings and replacing masked tokens with random tokens. Factors related to training data that are beneficial for multilinguality are similar word order and comparability of corpora .
After revision:  Architectural factors that contribute to multilinguality are underparameterization, shared special tokens (e.g., " CLS four architectural and two linguistic elements that influence multilinguality. Based on our insights, we experiment with a multilingual pretraining setup that modifies the masking strategy using VecMap, i.e., unsupervised embedding alignment. Experiments on XNLI with three languages indicate that our findings transfer from our small setup to larger scale settings .