CoNLL 2019

Multilingual model using
cross-task embedding projection
Jin Sakuma

Naoki Yoshinaga

T h e U n i ve rsi ty o f To kyo Insti tute of Indu stri a l S c i en c e,
the U ni versi ty of To ky o

Needs for resources in deep learning
Improvements in various tasks by deep learningl Task-specific representation learning needs more data
Translation performance (DE->EN) [Sennrich+ 2019]
BLEU

Large gain
with rich resources

Small gain
with poor resources

Size of training data

Need massive data for every pair of task and lang2Gaps in available resources across langs
Among 7097 languages in the world [Simons+ 2018],
massive resources are obtainable only in few
l Universal dependencies project covers only 76[Nivre+ 201X]

Large gap in model performances among languages

Can we exploit resources of resource-rich languages
for training in resource-poor languages?

3

Problem settings of this study
Available resources:l Labeled data for training in the source languagel Raw corpora in both languages
SourceAnnotated data
Raw corpus(e.g. Wikipedia)

TargetNo annotated data
Raw corpus(e.g. Wikipedia)

Applicable to various target languages and tasks
4

Problem settings of this study
Available resources:l Labeled data for training in the source language
No cross-linguall Raw corpora in both languages
resources
SourceAnnotated data
Raw corpus(e.g. Wikipedia)

TargetNo annotated data
Raw corpus(e.g. Wikipedia)

Applicable to various target languages
5Preliminary:

Cross-lingual word embeddings (CLWE)
Language-independent representation of words[Mikolov+ 13]

l Words from two lang. are represented in a shared spacel Similar words from different languages are closecatchien (dog)chat (cat) dog

eat

nourriture (food)manger (eat)badterrible
merveilleux(wonderful)

foodmauvaise (bad)goodbien (good)wonderful6

Existing multilingual models
Fix the emb. layer to general CLWE during training[Duong+ 17, Chen+ 18]

Train

chien

Input words

dog catgoodbien
bad mauvaise

chat
Sentiment model

General CLWE

l Enables cross-lingual transfer
l The embedding layer is not optimized for the task7

Related work:

Task-specific CLWE with specialized dict.
Utilize task-specific bilingual dict. to obtain CLWE
[Gouws+ 15]

Task-specific dict.
Input words

cat
chat

bien
good

Train

chien

bad mauvaisedog

Sentiment modelTask-specific CLWE

l The embedding layer is optimized for the taskl Additional cross-lingual resources are required8

Related work:

Task-specific CLWE with specialized dict.
Utilize task-specific bilingual dict. to obtain CLWE
[Gouws+ 15]

Task-specific dict.

Train

bien
cat
good
In thisstudy:chatInputwords
Obtain task-specific CLWE chienwithout relying ondog
bad mauvaiseresources
any cross-lingual
Sentiment modelTask-specific CLWE

l The embedding layer is optimized for the taskl Requires additional cross-lingual resources9Proposal:

Multilingual model with task-spec emb.
Project general CLWE to the emb. layer optimized
for the task by cross-task embedding projection
Sentiment
model

chen

Train data

dog

cat
chat

mauvaise(bad)badbien (good)
good

General CLWE

Cross-task emb.
projection(locally linear
mapping, LLM)

catbaddoggoodrunTask-specific word emb.
Input words
10

Locally linear mapping:

Idea: local topology of embeddings
Assumption:l Words adequately close in the general CLWE are also
close in task-specific space
chen

cat
chat

dog

mauvaisebien

wonderfulbad

goodwonderful

bien

good
food
cat
chat dogfoodnourriture
General CLWE

nourriturechenmauvaisebad(Ideal) task-specific CLWE
11

Locally linear mapping:

Step 1: selecting nearest neighbors
For each target word (bien), select 𝒌-nearest neighbors
in the general CLWE
chen

cat
chat

dog

wonderfulmauvaise
bien

bad

good

good
wonderful

food
cat

food

dog

nourriture
badGeneral CLWE

Task-specific word emb.
12

Locally linear mapping:

Step 2: local topology in general space
In general CLWE, learn linear combination of nearest
neighbors that reconstructs the the target wordchencatdog

chatfoodbien
nourriture↵ˆ w⇤ = arg min Ywgen↵w⇤<latexit sha1_base64="PsS4ZPlKqWUdb+bKGeNdRllb4fk=">AAAFBnicjVPNaxNBFH+pUWv8aKoXwctiqGjQMEkrFUEo6sGL2g/TRrrtsruZbIbsF7OTpHW7d/EsePCk4EFEb+JV8OI/4KF/gnisIIgH30yWmFhNOsPuvPnN+/3em3kzVuiySBCym5k4lD185OjksdzxEydPTeWnT69GQZvbtGoHbsBrlhlRl/m0KphwaS3k1PQsl65ZrVtyfa1DecQC/4HYDumGZzo+azDbFAgZ+ad60xS66YZN04i7xeSGbnLHY74RD4CJ7tKG2Il1FS+um7zFaT3RHhrdTV3QLcG92KF+klzRo7ZnxExnvnbP6CZ9DZYMki23TROtZrBhts6Z0xQ7mxUjXyAlopq23yinRgHSthhMZ96CDnUIwIY2eEDBB4G2CyZE2NehDARCxDYgRoyjxdQ6hQRyyG2jF0UPE9EW/h2craeoj3OpGSm2jVFc/DgyNZghX8hrskc+kzfkK/k1QqsOHYwZ4tqw3taQ4v/4B/eT2nLP2zhavRxpaEw9ObvyYyzLw1FA8w9rBMPC7o3MZPx6gL2FEU20o5G+HuKuqhpHTndM1QQ04Jo6XYanHSpE1tHu7azz6NneyvXlmfgCeUm+YQVfkF3yCWvod77br5bo8nOlfhs5vTvE0bqbns191KOIyJmsRRGj6MrHwSxlvCS9Yw7il/tYTt1QipnJOR2oZoy49G/hKbiYK1UKkm8pi0AJP4JamrLnYBbm4Wp/Lu2D6ct91P+hPqg2GEnaCb7G8t9vb7+xWimVZ0uVpbnCws30XU7COTgPF/HtzcMC3IFFqGJWPzNa5lKmmH2cfZd9n/3Qc53IpJwzMNSyH38D0CYqEQ==</latexit>wonderfulmauvaise
bien

good

badgood
wonderful

food
cat

good,wonderful
Xi2NwGeneral CLWEdog2↵wi Xigen

badTask-specific word emb.
13

Locally linear mapping:

Step 3: task-specific word embeddings
Compute task-specific word emb. of the target word
as the linear combination with the induced weights
chen

cat

dog

chat

food

bien
nourriture↵ˆ w⇤ = arg min Ywgen↵w⇤<latexit sha1_base64="PsS4ZPlKqWUdb+bKGeNdRllb4fk=">AAAFBnicjVPNaxNBFH+pUWv8aKoXwctiqGjQMEkrFUEo6sGL2g/TRrrtsruZbIbsF7OTpHW7d/EsePCk4EFEb+JV8OI/4KF/gnisIIgH30yWmFhNOsPuvPnN+/3em3kzVuiySBCym5k4lD185OjksdzxEydPTeWnT69GQZvbtGoHbsBrlhlRl/m0KphwaS3k1PQsl65ZrVtyfa1DecQC/4HYDumGZzo+azDbFAgZ+ad60xS66YZN04i7xeSGbnLHY74RD4CJ7tKG2Il1FS+um7zFaT3RHhrdTV3QLcG92KF+klzRo7ZnxExnvnbP6CZ9DZYMki23TROtZrBhts6Z0xQ7mxUjXyAlopq23yinRgHSthhMZ96CDnUIwIY2eEDBB4G2CyZE2NehDARCxDYgRoyjxdQ6hQRyyG2jF0UPE9EW/h2craeoj3OpGSm2jVFc/DgyNZghX8hrskc+kzfkK/k1QqsOHYwZ4tqw3taQ4v/4B/eT2nLP2zhavRxpaEw9ObvyYyzLw1FA8w9rBMPC7o3MZPx6gL2FEU20o5G+HuKuqhpHTndM1QQ04Jo6XYanHSpE1tHu7azz6NneyvXlmfgCeUm+YQVfkF3yCWvod77br5bo8nOlfhs5vTvE0bqbns191KOIyJmsRRGj6MrHwSxlvCS9Yw7il/tYTt1QipnJOR2oZoy49G/hKbiYK1UKkm8pi0AJP4JamrLnYBbm4Wp/Lu2D6ct91P+hPqg2GEnaCb7G8t9vb7+xWimVZ0uVpbnCws30XU7COTgPF/HtzcMC3IFFqGJWPzNa5lKmmH2cfZd9n/3Qc53IpJwzMNSyH38D0CYqEQ==</latexit>

wonderfulmauvaisebienbadgoodbien

goodwonderful

foodcatgood,wonderfulXbien2↵wi Xigen

Ywspec =i2Nw

General CLWE

<latexit sha1_base64="kzvuJCPxb4qsqUKFM2rIjWT9y/E=">AAAE2HicjVO9bxMxFH8pB5Tw0RQWJJYTURFCKHLaoiIkpAoYWIB+kDbQK6e7i5NY8X3I5yQtp5PYEIwsDEwgMSDEX8HCwMrQPwGxUSQGOvDsnEpCIKmtOz///H6/9+xnuxFnsSRkJzdxyDh85OjksfzxEydPTRWmT6/FYVt4tOKFPBRV14kpZwGtSCY5rUaCOr7L6brbuqnW1ztUxCwM7svtiG76TiNgdeY5EiG7UE8sLZLUHNEStJaaD+zuI0vSLSn8JI6ol6bXrbjt2wmzWGDetbup1XSk5fCo6dhJl6X9Ci5v09SsDknYhSIpEd3MYaOcGUXI2lI4nfsAFtQgBA/a4AOFACTaHByIsW9AGQhEiG1CgphAi+l1CinkkdtGL4oeDqIt/DdwtpGhAc6VZqzZHkbh+AlkmjBDvpB3ZJd8Iu/JV7I3QqsGHYwZ4dqg3taA4v/4B/dT2mrP2zi6vRxpZE89P7v6cyzLx1FC8w9rBMPF7o/MZPx6iL2FER2045G+PuJcV00gpzumahLqcFWfLsPTjjSi6uj1dtZ5/HJ39drKTHKBvCHfsIKvyQ75iDUMOj+8t8t05ZVWv4Wc3h0SaN3JzuYe6lFE1EzV4hJGsbRPA7NU8dLsjjUQv7yP5fUNpZiZmtO+aiaIK/8WngLHXKlWUHxXWwRK+BHUMrU9D3OwAFf258o+mL7aR+0f6v1q/ZGUrV5j+e+3N2yszZbKc6XZ5fni4o3sXU7COTgPF/HtLcAi3IYlqGBWn+E7/II946HxxHhqPOu5TuQyzhkYaMaL33oNGRc=</latexit>

doggood,Xwonderfuli2Nw↵ˆ wi Xwspec

badTask-specific word emb.
14

Proposal:

Hyperparameter search
Dev. set in the target language is required to tune
the hyperparameter 𝑘 (size of nearest neighbors)
Tuning to the task (no additional resources)
Assume the best 𝑘 is independent of language
Apply LLM to the embeddings of the source language
and evaluate on the dev. set of the source language(Tuning to the task/language)
Utilize small development set (100 examples)of the target language
15Experimental setup (1/2)
Goal:l Does out task-specific word embeddings improvethe multilingual model?

Task:l Topic classification task (and sent. analysis)

Languages:l Source language: English (en)l Target languages:l Danish (da), Italian (it), French (fr), Swedish (sv)

Datasets:l RCV1/2 dataset (four topics)
16

Experimental setup (2/2):

Models to compareCompare the following two models to evaluate
the effect of task-specific CLWE
l Experiments on more models on the paper
CLWE fixed
CLWE opt (proposal)
SoftMax
SoftMax
1-Layer FFNN
catchatdog chenbien good courir
bad mauvaise runGeneral CLWE

1-Layer FFNN
LLM

catchenchatdog
badmauvaise

bien
good
courirrun

Task-specific CLWE
17

Results:

Topic classification task
Classification accuracies in four languages
Method𝒌-tuning en-da
CLWE fixed
0.621
CLWE opt (Proposed)
task
0.672
CLWE opt (Proposed) task/lang 0.687

en-it

en-fr en-sv

0.535 0.772 0.816
0.623 0.885 0.831
0.615 0.879 0.830

l CLWE opt outperforms the baselinel Tuning 𝑘 for task and language is not necessary

18Conclusion and future work
Conclusionl Proposed a method to build a multilingual modelwith task-specific word embeddingsl Evaluated our method on real tasks
and confirmed its effectiveness

Future workl Evaluate this method on wider range of tasks, languages,
and models
l Further improve the quality of locally linear mapping

19