SlowGuess/NAACL_2022_OCR huggingface dataset:
It can be observed that: 1) without MCL task, the model performs poorer on the cross-lingual tasks. 2) without parameter sharing, the model performs poorer on all datasets. 3) MCL task can significantly improve the cross-lingual performance on EN-AR, EN-ES, EN-FR, EN-NL. It can be concluded that both MCL task and parameter recurrent play a key role in our method.
4.5 Effect of contrastive learning
To investigate the effects of contrastive learning in stage 4, we select XLM-R  $(b = \text{True}, b s = 128, |R U| = 3)$ , modify the original objective in (5) into three different settings, namely, Bool, CE and w/o CL.
In the Bool setting, the soft label in (5) is re
Settings	EN-AR	EN-DE	EN-TR	EN-ES	EN-FR	EN-IT	EN-NL	Avg.
Ours	78.0	79.8	73.9	80.5	82.1	80.3	81.2	79.4
Bool	77.0↓	80.5↑	73.5↓	79.8↓	80.3↓	80.7↑	81.2	79.0↓
CE	76.6↓	79.9↑	74.3↑	80.0↓	80.8↓	80.6↑	80.7↓	78.9↓
w/o CL	75.9↓	79.7↓	73.2↓	79.9↓	80.4↓	80.4↑	80.5↓	78.5↓
Table 5: Evaluation results of XLM-R ( $b = \text{True}$ ,  $bs = 128$ ,  $|RU| = 3$ ) on the STS2017-extend cross-lingual task with different contrastive learning settings.
Settings	Avg. (Monolingual)	Avg. (Cross-lingual)
Single-stage
Random Initialize	78.1	71.1
+ Pre-Distillation	79.0	73.8
Multi-stage
stage 1 + 2	48.4	20.8
stage 1 + 2 + 3	75.2	70.6
stage 1 + 2 + 3 + 4	82.6	79.4
Table 6: Comparison of using different stage settings on monolingual and multilingual STS task. XLM-R is the basic model. The first three layers from XLM-R are taken as a Recurrent Unit, bottleneck hidden size is 128.
placed with hard label (0 or 1), as (8),
$$
\ell_ {B o o l} = \frac {1}{N ^ {2}} \sum_ {i} ^ {N} \sum_ {j} ^ {N} \left(\delta \left(h _ {T} ^ {s i}, h _ {T} ^ {s j}\right) - \phi \left(h _ {S} ^ {s i}, h _ {S} ^ {t j}\right)\right) ^ {2}, \tag {8}
$$
where  $\delta (x,y) = 1$  , if  $x = y$  , otherwise 0.
In the  $CE$  setting, the objective in (5) is replaced with temperature-scaled cross-entropy, as (9),
$$
\ell_ {C E} = - \sum_ {i} ^ {N} \sum_ {j} ^ {N} \phi_ {T} \log \frac {e ^ {\phi_ {S} / \tau}}{\sum_ {k = 1} ^ {N} e ^ {\phi_ {S} / \tau}}, \tag {9}
$$
where  $\phi_T = \cos(h_T^{si}, h_T^{sj})$ ,  $\phi_S = \cos(h_S^{si}, h_S^{tj})$ ,  $\tau = 0.05$  is a hyperparameter called temperature.
In the  $w / o$  CL setting, the contrastive learning is removed in Stage 4.
Table 5 presents the model performance of crosslingual semantic similarity task with different settings. It can be observed that all the above training objectives can improve the model performance on the cross-lingual task, compared with the w/o CL settings. Model trained with (8) and (9) underperform that trained with (5), especially on EN-AR, EN-ES, EN-FR, EN-NL task.
We plot the convergence process of different settings in Figure 3. On EN-AR, EN-ES, EN-FR tasks, our setting outperforms other settings. It is worth mentioning that on the EN-TR task, our setting underperforms the  $CE$  setting according to Table 5. However, our setting reaches the same level as  $CE$  setting during the 30 to 40 epoch.
Model	AR-AR	ES-ES	EN-EN	Avg.	Embedding size	Encoder size
Teacher model=SBERT-paraphrases, Student model=XLM-R, |RU| = 3
b = True, bs = 128	76.7	84.5	86.6	82.6	32.49M	21.26M
b = True, bs = 256	76.2	84.9	87.4	82.8	64.59M	21.26M
b = False	79.9	86.8	88.4	85.0	192.40M	21.26M
Teacher model=SBERT-paraphrases, Student model=XLM-R, b = True, bs = 128
|RU| = 3	76.7	84.5	86.6	82.6	32.49M	21.26M
|RU| = 6	78.1	84.8	87.4	83.4	32.49M	42.52M
|RU| = 12	79.0	85.5	88.4	84.3	32.49M	85.05M
Teacher model=SBERT-paraphrases, Student model=MiniLM, |RU| = 3
b = True, bs = 128	72.8	79.3	84.4	78.8	32.05M	5.32M
b = True, bs = 256	72.2	81.2	85.2	79.5	64.10M	5.32M
b = False	79.9	85.3	85.6	83.6	96.21M	5.32M
Teacher model=SBERT-paraphrases, Student model=MiniLM, b = True, bs = 128
|RU| = 3	72.8	79.3	84.4	78.8	32.05M	5.32M
|RU| = 6	75.6	83.8	85.1	81.5	32.05M	10.64M
|RU| = 12	79.0	84.4	85.2	82.9	32.05M	21.29M
Table 7: The performance of STS2017 monolingual task based on XLM-R(b=True,  $bs = 128$ ,  $|RU| = 3$ ) and MiniLM(b=True,  $bs = 128$ ,  $|RU| = 3$ ), We evaluated the effect of increasing  $bs$  or  $|RU|$ .
Model	EN-AR	EN-DE	EN-TR	EN-ES	EN-FR	EN-IT	EN-NL	Avg.	Embedding size	Encoder size
Teacher model=SBERT-paraphrases, Student model=XLM-R, |RU| = 3
b = True, bs = 128	78.0	79.8	73.9	80.5	82.1	80.3	81.2	79.4	32.49M	21.26M
b = True, bs = 256	79.2	81.8	73.8	82.3	82.7	81.6	82.6	80.6	64.59M	21.26M
b = False	81.1	84.3	79.8	82.6	84.5	84.8	85.4	83.2	192.40M	21.26M
Teacher model=SBERT-paraphrases, Student model=XLM-R, b = True, bs = 128
|RU| = 3	78.0	79.8	73.9	80.5	82.1	80.3	81.2	79.4	32.49M	21.26M
|RU| = 6	78.8	80.0	74.7	82.9	83.5	83.4	84.6	81.1	32.49M	42.52M
|RU| = 12	79.4	83.6	78.7	83.3	84.2	85.6	84.8	82.8	32.49M	85.05M
Teacher model=SBERT-paraphrases, Student model=MiniLM, |RU| = 3
b = True, bs = 128	73.0	76.0	63.7	71.4	71.8	72.1	74.7	71.8	32.05M	5.32M
b = True, bs = 256	69.7	77.1	66.2	73.5	73.5	74.3	75.6	72.8	64.10M	5.32M
b = False	82.3	82.8	76.9	82.1	80.5	82.3	82.4	81.3	96.21M	5.32M
Teacher model=SBERT-paraphrases, Student model=MiniLM, b = True, bs = 128
|RU| = 3	73.0	76.0	63.7	71.4	71.8	72.1	74.7	71.8	32.05M	5.32M
|RU| = 6	77.1	78.7	68.2	78.1	75.9	77.0	77.6	76.1	32.05M	10.64M
|RU| = 12	79.7	81.0	74.1	81.9	80.1	80.8	80.7	79.8	32.05M	21.29M
Table 8: The performance of STS2017-extend cross-lingual task based on XLM-R(b=True,  $bs = 128$ ,  $|RU| = 3$ ) and MiniLM(b=True,  $bs = 128$ ,  $|RU| = 3$ ), We evaluated the effect of increasing  $bs$  or  $|RU|$ .
4.6 Effect of multi-stages
To verify the effectiveness of multi-stages, we show the performance comparison of using different stage settings in Table 6. In the Single-stage setting, we first initialize the shrunk student model in two ways: (1) Random Initialize: Adding the untrained embedding bottleneck layers to the student model. (2) Pre-Distillation: The student model with bottleneck layer is initialized by distillation using XLM-R and the same corpus as section 4.1. Then we follow Reimers and Gurevych (2020) to align vector space between different languages. In the Multi-stage setting, the performance of the student model is reported after each stage.
As shown in Table 6, the Multi-stage setting outperforms the single-stage one, indicating that our multi-stage framework with an assistant model is effective. Adding stage3 and stage4 further improves the student model performance, suggesting
that multi-stage training are necessary.
4.7 Effect of bottleneck and recurrent unit
In this section, we study the impact of embedding bottleneck and recurrent unit strategies on multilingual semantic learning. We consider three settings for each strategy, as shown in Table 7 and Table 8.
First, we found that both XLM-R and MiniLM perform better as the bottleneck hidden size  $bs$  increases. The performance is best when the entire embedding layer is retained, The MiniLM  $(b = \text{False})$  can outperform its original model in Table 1 and Table 2. But the benefit of increasing  $bs$  is not obvious unless the entire embedding layer is retained.
Second, by increasing the number of recurrent unit layers  $|RU|$ , XLM-R and MiniLM have been steadily improved on these two tasks. The increase in model size caused by the  $|RU|$  is less than the  $bs$ . For example, the performance of MiniLM on cross-lingual tasks increased by  $8\%$ , while its size
only increased by 15.9M.
Finally, it can be observed that when using the bottleneck layer  $(b = \text{True})$ , the model performance will increase steadily as  $|RU|$  increases. The smaller the encoder hidden size, the more significant effect caused by  $|RU|$  increasing ( $\Delta \text{MiniLM} > \Delta \text{XLM-R}$ ). However, the increase of  $bs$  can not improve performance significantly but make the embedding size larger. Therefore, an effective way to compress the multilingual model is reducing  $bs$  while increasing  $|RU|$ . In this way, we shrink XLM-R by  $58\%$ , MiniLM by  $55\%$ , with less than  $1.1\%$  performance degradation.
In this work, we realize that the cross-lingual similarity matching task requires a large model size. To obtain a small-size model with cross-lingual matching ability, we propose a multi-stage distillation framework. Knowledge distillation and contrastive learning are combined in order to compress model with less semantic performance loss.
Our experiments demonstrate promising STS results with three monolingual and six cross-lingual pairs, covering eight languages. The empirical results show that our framework can shrink XLM-R or MiniLM by more than  $50\%$ . In contrast, the performance is only reduced by less than  $0.6\%$  on monolingual and  $1.1\%$  on cross-lingual tasks. If we slack the tolerated loss performance in  $4\%$ , the size of XLM-R can be reduced by  $80\%$ .
Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics (TACL), 7:597-610.
Akari Asai, Jungo Kasai, Jonathan Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021. XOR QA: Cross-lingual open-retrieval question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 547-564.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14.
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao,
Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 3576-3588.
Muthu Chidambaram, Yinfei Yang, Daniel Cer, Steve Yuan, Yunhsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Learning cross-lingual sentence representations via a multi-task dual-encoder model. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 250-259.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 8440-8451.
Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems (NeurIPS).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 4171-4186.
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic bert sentence embedding. arXiv preprint arXiv:2007.01852.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6894-6910.
Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. 2020. EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5427-5444.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop.
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pretraining with multiple cross-lingual tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2485-2494.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163-4174.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X, pages 79-86, Phuket, Thailand.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In Proceedings of International Conference on Learning Representations (ICLR).
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119-9130.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proceedings of the 2019 International Conference on Learning Representations (ICLR).
Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang, and Luo Si. 2021. VECO: Variable and flexible cross-lingual pre-training for language understanding and generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP), pages 3980-3994.
Zhuoyuan Mao, Prakhar Gupta, Chenhui Chu, Martin Jaggi, and Sadao Kurohashi. 2021. Lightweight cross-lingual sentence representation learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP), pages 2902-2913.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan
Ghasemzadeh. 2020. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5191-5198.
Xuan Ouyang, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. ERNIE-M: Enhanced multilingual representation by aligning cross-lingual semantics with monolingual corpora. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 27-38.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. In Journal of Machine Learning Research, pages 1-67.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992.
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512-4525.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2021. WikiMatrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (ECACL), pages 1351-1361.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4323-4332.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. MobileBERT: a compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2158-2170.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC), pages 2214-2218.
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2021. MiniLMv2: Multi-head self-attention relation distillation for compressing pretrained transformers. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 2140–2151.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45.
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2020. Multilingual universal sentence encoder for semantic retrieval. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 87-94.
Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Improving multilingual sentence embedding using bidirectional dual encoder with additive margin softmax. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI), pages 5370-5378.
Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju, and Xiaoyong Du. 2019. UER: An open-source toolkit for pretraining models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 241-246.