Artificial Intelligence,
Machine Translation andFuture Linguists Like You
Graham Neubig@New York Circle of Translators Meeting (9/21/2020)

翻訳とは、⾔語において、Aという形で表現されているものを、その意味に対応するBという形に置き換える⾏為をさす。
具体的には、⾃然⾔語において、起点⾔語 による⽂章を、別の⽬標⾔語 による⽂章に変換する⾏為をさす。

an expression that takes
a different form
Translation refers to the act of replacing what is expressed in
the form of A in a language with the form of B that corresponds
to that meaning. Specifically, in natural language, it refers tothe act of converting a sentence in the source language into a
sentence in another target language.
the samea⽥中花⼦は友⼈が少ないが、数少ない友⼈をとても⼤事にする。
Hanako Tanaka has few friends, but she cherishes the few friends verymuch.
佐藤太郎は友⼈が少ないが、数少ない友⼈をとても⼤事にする。
Taro Sato has few friends, but he cherishes a few friends very much.
⽥中花⼦は友⼈が少なく佐藤太郎は友⼈が多い。だが、⽥中は数少ない友⼈をとても⼤事にする。
Hanako Tanaka has few friends and Taro Sato has many friends.
However, Tanaka takes great care of his few friends.

コード
ko-do(code, cord, chord)

電気のコードが切れた。
The electric cord has broken.
プログラムのコードを書いた。
I wrote the code for the program.
楽譜にコードを書いた。
I wrote the chord on the score.

プログラマーなのでコードを読むのが得意。
As a programmer, I am good at reading code.
ミュージシャンなのでコードを読むのが得意。
As a musician, I am good at reading codes.
Javaで、ギターのコードを表示するコードを書いた。
In Java, I wrote a chord that displays the chord of the guitar.

Why Can Machines
Translate?

Translation Methods•

•

•

•

Rule-based: Do linguistic analysis, syntactic transformation, and word/phrase
replacement•+ Can be complicated/sophisticated in design•- Hard to handle every contingency, not enough linguists to build for everylanguage

Translation memory: Look up most similar sentence in database•+ Easy to understand - look up most recent sentence•- Cannot generalize to new sentences

Phrase-based translation: Look up translations for small chunks then combine thesmall chunks together•+ Can generalize better as long as small chunks are covered•- Hard to generate fluent sentences, make mistakes on syntax

Neural machine translation: Feed input into probabilistic model that predicts nextword•+ Better at syntactic correctness, fluency, and leveraging context•- Can make big mistakes (dropping/hallucinating content)

data-driven methods

Available Data•

Parallel corpora: Translated sentences in source/targetlanguage•

Built by translators or hobbyists, for government offices,
newspapers, companies, end-users

OPUS Open SourceCorpus
http://opus.nlpl.eu/
Google's MT Data [Arivazhagan+ 19]

Parallel Training Data
Source Language(X)

Target Language(Y)
Training Process

Source Language Input(X)

Probabilistic
Translation ModelP(Y|X)

Target Language Output(Y)Translation Process

Image Credit: pngtree

NMT Basic Idea: Predict Next Word's
Probability and Choose Max
X = watashi wa kouen wo shiteimasu
P(y1=I|X) = 0.96
P(y1=talk|X) = 0.03

P(y1=it|X) = 0.01
...

y1=I

P(y2=am|X,y1) = 0.90
P(y2=was|X,y1) = 0.06

P(y2=will|X,y1) = 0.02
...

y2=am

P(y3=giving|X,y1,2) = 0.41
P(y3=talking|X,y1,2) = 0.20

P(y3=presenting|X,y1,2) = 0.15...

y3=giving

P(y4=a|X,y1-3) = 0.85
P(y4=the|X,y1-3) = 0.10

P(y4=my|X,y1-3) = 0.04
...

y4=am

P(y5=talk|X,y1-4) = 0.51
P(y5=lecture|X,y1-4) = 0.30

P(y5=presentation|X,y1-4) = 0.12...

y5=talkP(y6=<done>|X,y1-5) = 0.51 P(y6=with|X,y1-5) = 0.12
P(y6=and|X,y1-5) = 0.30...

y6=<done>

How to Make Difficult Predictions?
Neural Networks

An Aside: What is "ArtificialIntelligence?"

From "Wired"
From Willyam BradberryAn Aside: What is "ArtificialIntelligence?"•

Artificial Intelligence: Technology that does
something that seems intelligent•Machine Learning: Technology that learns fromdata to do something that cannot be done easily
otherwise••(translation is a good example)

In it's current actualization: mostly just computationgraphs with parameters trained on lots of data

“Neural” Networks
Original Motivation: Neurons in the Brain

Current Conception: Computation Graphs
f (x1 , x2 , x3 ) =X

xii

f (M, v) = Mv
f (U, V) = UV
f (u) = u>

Axf (u, v) = u · v

bc

Image credit: Wikipedia

expression:>y = x Ax + b · x + cgraph:xexpression:>y = x Ax + b · x + cgraph:f (u) = u>xexpression:>y = x Ax + b · x + cgraph:f (U, V) = UV
f (u) = u>Axexpression:>y = x Ax + b · x + cgraph:f (M, v) = Mv
f (U, V) = UV
f (u) = u>Axexpression:>y = x Ax + b · x + cgraph:f (x1 , x2 , x3 ) =Xxiif (M, v) = Mv
f (U, V) = UV
f (u) = u>Axf (u, v) = u · v

bcexpression:>y = x Ax + b · x + cgraph:f (x1 , x2 , x3 ) =yX

xiif (M, v) = Mv
f (U, V) = UV
f (u) = u>

Axf (u, v) = u · v

bc

"Learning" Neural Networks•

Build a graph to calculate something you want tomaximize/minimize (e.g. probability of translation
given an input)•

Calculate the value itself•

Calculate the derivatives of model parameters
given the value (back propagation)•Update the model parameters to increase/decrease the value

Forward Propagationgraph:f (x1 , x2 , x3 ) =X

xiif (M, v) = Mv
f (U, V) = UV
f (u) = u>Axf (u, v) = u · v

bcForward Propagationgraph:f (x1 , x2 , x3 ) =X

xii

f (M, v) = Mv
f (U, V) = UV
f (u) = u>

Axf (u, v) = u · v

bcForward Propagationgraph:f (x1 , x2 , x3 ) =Xxiif (M, v) = Mv
f (U, V) = UV
f (u) = u>

Axf (u, v) = u · v

bcForward Propagationgraph:f (x1 , x2 , x3 ) =Xxii

f (M, v) = Mv
f (U, V) = UV
f (u) = u>Ax>xf (u, v) = u · v

bcForward Propagationgraph:f (x1 , x2 , x3 ) =

X

xi

i

f (M, v) = Mv
f (U, V) = UV
x> A
f (u) = u>Ax>xf (u, v) = u · v

bcForward Propagationgraph:f (x1 , x2 , x3 ) =Xxiif (M, v) = Mv
f (U, V) = UV
x> A
f (u) = u>

Ax>xf (u, v) = u · v
b·xbc

Forward Propagationgraph:f (x1 , x2 , x3 ) =Xxiif (M, v) = Mv
x> Ax
f (U, V) = UV
x> Af (u) = u>

Ax>xf (u, v) = u · v
b·xbc

Forward Propagationgraph:f (x1 , x2 , x3 ) =X

xii>

x Ax + b · x + cf (M, v) = Mv
x> Ax
f (U, V) = UV
x> A
f (u) = u>Ax>xf (u, v) = u · v
b·xbc

Back Propagationgraph:f (x1 , x2 , x3 ) =X

xii

f (M, v) = Mv
f (U, V) = UV
f (u) = u>Axf (u, v) = u · vbc

(A Gross Oversimplification of)

Neural Machine Translation

Recurrent Neural Networks
IRNN

hateRNN

thisRNNmovieRNN•

Read in one word at a time and extract "features"•

Word embeddings (red): features of each individualword•

Hidden words (purple): features of all words seen so farEncoder- Decoders
Encoder
kono

LSTM

eiga

LSTM

I

LSTM

ga

LSTM

hate

LSTM

kirai

LSTM

LSTM

thisLSTM

movie

LSTM

argmaxargmaxargmaxargmaxargmaxI

hate

thismovie


Decoder
Sentence Representations
this is an example•"read this document, remember it, and translate it"this is an example•"read this document, reference it while translating"Calculating Attention (1)•Use “query” vector (decoder state) and “key” vectors (all encoder states)•

For each query-key pair, calculate weight•

Normalize to add to one using softmax

konoeigagakirai

a1=2.1 a2=-0.1

a3=0.3a4=-1.0

KeyVectors
I hateQuery Vector

softmax

α1=0.76 α2=0.08 α3=0.13 α4=0.03

Calculating Attention (2)•Combine together value vectors (usually encoderstates, like key vectors) by taking the weighted sumkonoeigagakiraiValueVectors****
α1=0.76 α2=0.08 α3=0.13 α4=0.03•Use this in any part of the model you like

A Graphical Example

Image from Bahdanau et al. (2015)

"Deep" Learning
this is an example
Attention

this is an example
Attention

Attention

Attention
Attention

x6?
x12?
x24?

Again, Why Can/Can't
Machines Translate?

What Can/Can't MachineTranslation Currently Do?
•Theoretically: Neural networks can model arbitrarily
complicated functions -- i.e. they can do anything•

In Practice:•

MT is very good at modeling direct associations•

MT is pretty good at modeling associations that
require capturing word similarity•

MT is not great at capturing phenomena that
require more than one step of "reasoning"•

MT cannot learn things that are not included in itsdata

プログラム の コード を 書いた 。
.

programof

ko-do

OBJ

wrote.

I wrote the code for the program.

電気 の コード が 切れた 。
electricof

ko-do

SUBJ

broke. .

The electric cord has broken.

ミュージシャンなのでコードを読むのが得意。

"reading" more strongly
associated with "codes" than"ミュージシャン" with "chords"?

As a musician, I am good at reading codes.
⽥中花⼦は友⼈が少なく佐藤太郎は友⼈が多い。
だが、⽥中は数少ない友⼈をとても⼤事にする。

Hanako Tanaka has few friends and Taro
Sato has many friends. However, Tanaka
takes great care of his few friends.

Too many steps of between
stereotypically female name
"Hanako" and the final predictionof a gendered pronoun.

Data Efficiency

MT Performance by Data [Arivazhagan+ 19]

What Does this Mean forFuture Linguists like You?

Overall Implications•Machine translation is pretty good, and it will continue to getbetter•

However, it's not perfect••

Especially in domains or languages where little data exists•

Especially in non-literal translations•

Especially in any situation where you really have to think

Barring a major technological breakthrough, the above weakpoints will improve gradually, not be solved any time soon

Shared Strengths••

Humans and machines are better at different things•

Machines: memory, speed•

Humans: reasoning, estimating uncertainty,
communication with clients

For high-quality translation, the answer will likely
not be human translation or machine translation,
but how we can best combine both

Case Study: Chinese-English News Translation

More mistaken words by MT
More missing words byhuman translator B

Many more grammaticalerrors by MT

Samuel Laubli, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen, Antonio Toral. dA Set of Recommendations forAssessing Human–Machine Parity in Language Translation. Journal of Artificial Intelligence Research (JAIR). February 2020.

An Example of Symbiosis:
Terminology Assistance for Interpreters
Simultaneous interpretation is an strenuous task, and term recall is difficult• One solution: leave interpreting to humans, but have machines assist in term recall•• Nikolai Vogler, Craig Stewart, Graham Neubig.

Lost in Interpretation: Predicting Untranslated Terminology in Simultaneous Interpretation.
Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL). June 2019.

Conclusion•

Modern MT can seem like magic at times, but it's notf (x1 , x2 , x3 ) =X

xi

if (M, v) = Mv
f (U, V) = UV
f (u) = u>

Axf (u, v) = u · vbc•It does lots of things well, and lots of things poorly•

Understanding it and working with it is likely the pathforwardThank You! Questions?