Before revision: In this work, we formulate cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts.
After revision: In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts.