Breaking Language Barriers:
Enhancing Multilingual Representation forSentence Alignment and Translation
Zhuoyuan Mao
Jun 20, 2024

Zhuoyuan Mao, 毛 卓遠
Education•••

2013/8 – 2017/7: BSc, Mathematics, East China University of Science and Technology
2019/4 – 2021/3: MInf, Intelligence Science and Technology, Kyoto University
2021/4 – 2024/3: PhD, Intelligence Science and Technology, Kyoto University

Research Career•••••

2022/2 – 2022/3: Research Internship, SenseTime Japan
2022/4 – 2024/3: JSPS Research Fellowship for Young Scientists, DC22022/8 – 2022/10: Student Researcher, Google Japan
2023/6 – 2023/9: Research Internship, Apple Japan
2024/4 – Present: Full-time Researcher, Sony Group Corporation2

Outline•

Introduction to Multilingual Representation Learning•Key Challenges and Our Proposals– High Computation Demands: EMS and LEALLA.
– Data Scarcity: JASS+ENSS, WCL, and AlignInstruct.
– Limitations in Transformer Architecture: INTL and LayerNorm.

•Contributions, Discussions, Conclusion and Future Prospects

3

Introduction to Multilingual
Representation Learning

The Diversity of Global Languages••

Our world is home to over 7,100 languages and 147
language families, each representing a unique culture andcommunity.
However, this diversity often leads to communication
barriers among people who speak different languages.

Figure from Wikipedia.

5Breaking Language Barriers•To bridge these linguistic divides, sentence alignment and machinetranslation techniques emerge as essential tools, enabling clearer andmore effective cross-lingual communication.
Align sentences conveying similar
meanings across languages

Translate sentences
across languages

Sentence Alignment

Machine Translation

Multilingual Representation Learning•

Multilingual representation learning is the core NLP technique that
supports the sentence alignment and translation tasks.
6

What is Multilingual Representation Learning?
• Train one universal neural model for multiple languages(Johnson et al., 2017, Artetxe and Schwenk, 2019)– No need to train separate models for each language– Benefit from knowledge transfer across languages

…

en

fr

jazh

Train

MultilingualModel7

Multilingual Sentence Embedding Learning
Sentence embedding:
I am a student . -> [0.013, -0.065, …, 0.043]

Multilingual sentence embedding:
I am a student. -> [0.013, -0.065, …, 0.043]私は学生です. -> [0.014, -0.063, …, 0.041]
Zero-shot cross-lingual transfer on downstream tasks(document classification, etc.):
Training:

US Professional Sports Teams
Promote Vaccinations

Inference:

新奇超薄膜「二次元材料」の成長と評価

Model fordownstream taske.g. classification

Aligning sentences in ashared embedding space
Initialize

NewsScience8Multilingual Neural Machine Translation•••

Translation for multiple language pairs with a single neural model. (Johnson et al., 2016)
The target language can be specified with a language tag (e.g., [to ja], [to en]).
Multilingual NMT can benefit low-resource languages through cross-lingual transfer.

•Zero-shot translation– Translation for unseen language pairs without training data.
– Promising for its low latency compared with pivot-based translation.

9

Key Challenges and Our Proposals in
Multilingual Representation Learning

An Overview
Core NLP Domain of the Thesis
Multilingual Representation Learning

Fundamental Task 1
Multilingual Sentence Embedding Learning

Key Challenge 1: High
Computational Demands when
Expanding Language Coverage

Fundamental Task 2
Multilingual Neural Machine Translation

Key Challenge 2:
Data Scarcity in Low-resource Languages

Chapter 2: Efficientand Effective
Training Method
for Multilingual
Sentence Learning

Chapter 3: Lightweight
Multilingual Sentence
Embedding Learning
for Efficient Inference

Chapter 4:
Linguistically-driven
Multi-task Pre-training
for Low-resource
Translation

Chapter 5: Word-level
Contrastive Learning
for Low-resource
Translation

Chapter 6: Tuning
LLMs with Alignment
Instructions for Lowresource Translation

ACL 2021,
IEEE/ACM TASLPEACL 2023

LREC 2020,
ACM TALLIPNAACL 2022
Findings

Submitted toLoResMT 2024
Workshop

Key Challenge 3: Limitations in Transformer
architecture for Zero-shot Translation

Chapter 7: Variablelength NeuralInterlingua
Representations forZero-shot Translation

Chapter 8: The Settingof Transformer Layer
Normalization Impacts
Zero-shot Translation

Multi3Generation
2023 Workshop

ACL 2023

11

Key Challenges in Multilingual Representation Learning① High computational demands
A Multilingual Model…

••

Computational demands escalate as we extend multilingual models
to accommodate additional languages.
How to address this challenge has not been explored so far in thecontext of multilingual sentence embedding learning.
12

EMS: Efficient and Effective Massively Multilingual Sentence
Representation Learning (ACL 2021 & IEEE/ACM TASLP)
Goal• Develop an efficient andeffective method for training
multilingual sentence embedding.

�Softmax

Proposals• Proposed effective generativeand contrastive objectives withdecreased amount of data andcomputation overhead.

Softmax

Embedding Layer• Obtained SOTA performance on
six sentence alignment tasks.

Contrastive Objective

Share Weights

Embedding Layer

Share Weights

MLP~MLP

Share Weights

MLPResults• Achieved significant training
acceleration compared withprevious work.

�Generative Objective�

MLP�� ��� ��

Mean-pool

Mean-poolfor inference

Share Weights

Share Weights

Lang. Tag
Embedding
Layer

Lang. Tag
Embedding
Layer

Transformer Encoder

Transformer Encoder

<2fr>

<2en>

I am a student .

Je suis un étudiant .

Take the first sentence pair as an example

A batch sampleof the training dataSS��

I am a student .

Je suis un étudiant .

����

京都 に 住んで いる 。

I live in Kyoto .

��…

…

��I love dancing .

我 喜欢 跳舞 。

��13

LEALLA: Learning Lightweight Language-agnostic Sentence Embeddings
with Knowledge Distillation (EACL 2023)
Goal• Develop lightweight language-agnostic
sentence embedding models based on LaBSE(Feng et al., 2022), for faster inference.
Explorations and Proposals• Explored the optimal dimension of languageagnostic sentence embeddings.
• Explored the optimal model architecture.
• Proposed two distillation methods toenhance the lightweight model.
Results• Obtained SOTA performance for cross-lingual
sentence retrieval for 109 languages.
• Released LEALLA models on TFHub:

https://tfhub.dev/google/collections/LEALLA/114

Key Challenges in Multilingual Representation Learning② Data scarcity••

The scarcity of training data in low-resource languages often leads toinferior performance of multilingual models for these languages.
Prior research has focused on pre-training, cross-lingual transfer, and data
augmentation for low-resource languages. Our approach advances this byutilizing more fine-grained linguistic features or alignments to enhance
performance for low-resource languages.

15

ENSS & JASS: Linguistically-driven Multi-task Pre-training for Low-resource
Neural Machine Translation (LREC 2020 & ACM TALLIP)
ENSS: PMASS & HFSSGoal• Improve sequence-to-sequence pre-training
for NMT in low-resource languages.
Proposals• PMASS and HFSS: Linguistically-driven pretraining methods for NMT involving English.

• BMASS and BRSS: Linguistically-driven pretraining methods for NMT involving Japanese.
Results

JASS: BMASS & BRSS

• Achieved up to 7.0 BLEU improvements on
4 datasets in low-resource scenarios.

16WCL: When do Contrastive Word Alignments Improve Many-to-many
Neural Machine Translation? (NAACL 2022 Findings)
Goal• Improve encoder-side representations for lowresource languages in multilingual NMT.
Proposals

Results•
Improved the translation quality on several low-resource
language pairs, which comes from better aligned
encoder representations.

• Proposed word-level contrastive learning toaugment and improve the cross-lingual signals for
multilingual NMT.

17

AlignInstruct: Tuning LLMs with Contrastive Alignment Instructions for MachineTranslation in Unseen, Low-resource Languages (Submitted to LoResMT 2024)
Goal• Adapt multilingual LLMs to unseen, low-resource languages
in MT tasks with enhanced cross-lingual signals.
Explorations and Proposals

Results•AlignInstruct led to consistent improvements intranslation quality across 48 translation directions
involving English and 30 zero-shot directions.

• Proposed AlignInstruct for enhancing cross-lingual signals
for better aligning low-resource languages in MT instruction
fine-tuning.
Instruction Fine-tuning

MTInstruct--- Input --Translate from English to Japanese.
English: What’s the future of generative AI?
Japanese:--- Output --生成 AI の未来はど う なる で し ょ う か?
A Sentence Pair Example
between English and Japanese

AlignInstruct--- Input --Given the following parallel sentence between English and Japanese,judge whether the assertion is True or False.
English: What’s the future of generative AI?
Japanese: 生成 AI の未来はど う なる で し ょ う か?
Assertion: “generative” can be aligned with “生成” statistically.
--- Output --True (or False)
A Sentence Pair Example
between English and JapaneseEnglish-Japanese
Parallel Corpora
ABC --- いろ は
Multilingual Parallel Corpora

Multilingual Training

Alignment Extractor:
IBM Model 2 (FastAlign)

future ↔ 未来
generative ↔ 生成of ↔ のInference English-JapaneseWord Alignments

18Key Challenges in Multilingual Representation Learning③ Limitations in Transformer architectureTransformer is initially designed for the English—French bilingual translation task.

••

The Transformer architecture, initially designed for bilingualtranslation, may not be ideally suited for multilingual contexts.
Addressing this architecture's limitations in zero-shot translation
was underexplored, with only two studies by Zhu et al. (2020) and
Liu et al. (2021) delving into it.

19Variable-length Neural Interlingua Representations for Zero-shot
Neural Machine Translation (Multi3Generation 2023 Workshop)
An overview of the proposed variable-length interlingua representations.

Goal• Figure out a better Transformer
architecture for zero-shot NMT.
Proposals• Proposed variable-length interlingua
representations to constructlanguage-agnostic representation
sequence with variable length.
Results• Improved and stabilized translation
quality in 48 zero-shot translation
directions on three datasets.

The proposed variable-length interlingua module.

20

PostNorm: Exploring the Impact of Layer Normalization for
Zero-shot Neural Machine Translation (ACL 2023)
Goal• Figure out the Transformer component that
impacts zero-shot NMT.

BLEU scores (and off-target rates) of PreNorm vs. PostNorm acrossdifferent language tag and residual connection settings.

Explorations and Proposals• Explored PostNorm in different language tagand residual connection settings.
• Proposed LLR score to analyze encoder anddecoder representations for zero-shot NMT.
Results• PostNorm was consistently better thanPreNorm across various settings.
• LLR revealed that PostNorm leads to betteraligned encoder representations and morelanguage-specific decoder representations,which benefited zero-shot translation.

LLR scores for recognizing source or target languages using PreNorm or PostNorm.

21

Conclusion and Future Prospects

Conclusion•

We explored methods to tackle the three challenges in multilingual
representation learning, focusing on the sentence alignment andtranslation tasks.

•The methods proposed offer strategies for training more efficientand effective multilingual models and can potentially benefit theresearch pertaining to multilingual LLMs, thus broadening thereach of NLP techniques to a wider audience.

23

Future Prospects•

Further expanding language coverage– Low-resource languages, regional dialects, etc.

•Confirming the effectiveness of the proposed methods for LLMs•

Integration with multimodal data
– Images and videos as universal pivots for further improving
multilingual models.

•Cross-culture understanding– Detecting and addressing cultural nuances.

24

Thanks for your listening!

Publication List
[1] Zhuoyuan Mao, Fabien Cromieres, Raj Dabre, Haiyue Song and Sadao Kurohashi. JASS: Japanesespecific Sequence to Sequence Pre-training for Neural Machine Translation. In Proceedings of the 12thLanguage Resources and Evaluation Conference, pages 3683–3691, 2020.
[2] Zhuoyuan Mao, Prakhar Gupta, Chenhui Chu, Martin Jaggi and Sadao Kurohashi. Lightweight CrossLingual Sentence Representation Learning. In Proceedings of the Joint Conference of the 59th Annual
Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing, pages 2902–2913, 2021.
[3] Zhuoyuan Mao, Chenhui Chu, Raj Dabre, Haiyue Song, Zhen Wan and Sadao Kurohashi. When doContrastive Word Alignments Improve Many-to-many Neural Machine Translation? In Findings of theAssociation for Computational Linguistics: NAACL 2022, pages 1766–1775, 2022.
[4] Zhuoyuan Mao, Chenhui Chu and Sadao Kurohashi. Linguistically-driven Multi-task Pre-training for
Low-resource Neural Machine Translation. In ACM Trans. Asian Low-Resour. Lang. Inf. Process., 21, 4,Article 68 (Jul. 2022), 29 pages.
[5] Zhuoyuan Mao and Tetsuji Nakagawa. LEALLA: Learning Lightweight Language-agnostic Sentence
Embedding with Knowledge Distillation. In Proceedings of the 17th Conference of the European Chapterof the Association for Computational Linguistics, pages 1886–1894, 2023.

26

Publication List
[6] Zhuoyuan Mao, Haiyue Song, Raj Dabre, Chenhui Chu and Sadao Kurohashi. Variable-length NeuralInterlingua Representations for Zero-shot Neural Machine Translation. In the 1st Workshop on
Multilingual, Multimodal and Multitask Language Generation, pages 16–25, 2023.
[7] Zhuoyuan Mao, Raj Dabre, Qianying Liu, Haiyue Song, Chenhui Chu and Sadao Kurohashi. Exploring
the Impact of Layer Normalization for Zero-shot Neural Machine Translation. In Proceedings of the 61stAnnual Meeting of the Association for Computational Linguistics, pages 1300–1316, 2023.
[8] Zhuoyuan Mao and Yen Yu. Tuning LLMs with Contrastive Alignment Instructions for MachineTranslation in Unseen, Low-resource Languages. Submitted to LoResMT 2024 Workshop, 2023.
[9] Zhuoyuan Mao, Chenhui Chu and Sadao Kurohashi. EMS: efficient and effective massively multilingual
sentence representation learning. In IEEE ACM Trans. Audio Speech Lang. Process. 32: 2841-2856, 2024.

27