Machine Translation. This story includes pre-neural machine…
By: Rachel Chen
Tags: NLP, Machine Learning, Data Science, Seq2seq, Machine Translation

2. Neural machine translation

Neural Machine Translation (NMT) is a way to do Machine Translation with a single neural network. The neural network architecture is called sequence-to-sequence (aka seq2seq) and it involves two RNNs

seq2seq NMT decoding example.Source: the course slide

Sequence-to-sequence(seq2seq) is useful for more than just MT. Many NLP tasks can be phrased as sequence-to-sequence: summarization( from long text to short text), dialogue (from previous utterances to next utterance), parsing (from input text into output parse as a sequence), and Code generation (from natural language into python code). The sequence-to-sequence model is an example of a conditional Language Model. It is a Language Model because the decoder is predicting the next word of the target sentence y. It is Conditional because its predictions are also conditioned on the source sentence x. NMT directly calculate P(y|x).

NMT directly calculates P(y|x)

seq2seq NMT training process.Source: the course slide

Getting a big parallel corpus, we could train an NMT system. Seq2seq is optimized as a single system, backpropagation operates “end-to-end”.

We saw how to generate (or “decode”) the target sentence by taking argmax on each step of the decoder. This is greedy decoding, which is taking the most probable word on each step). The problem with greedy decoding is it has no way to undo decisions. Ideally, we want to find a (length T) translation y that maximizes p(y|x). This is the exhaustive search decoding algorithm.

Ideally, we want to find a (length T) translation y that maximizes p(y|x).Source: the course slide

We could try computing all possible sequences y. This means that on each step t of the decoder, we are tracking Vt possible partial translations, where V is vocab size. This O(V^T) complexity is far too expensive!

We could use beam search decoding. On each step of decoder, it keeps track of the k most probable partial translations ( which we call hypotheses), where K is the beam size ( in practice around 5 to 10).

score in beam search.Source: the course slide

A hypothesis y1,….yt has a score which is its log probability. Scores are all negative, and the higher score is better. We search for high-scoring hypotheses, tracking top k on each step. Beam search is not guaranteed to find optimal solutions, but it is much more efficient than the exhaustive search!

beam search decoding example.Source: the course slide

Let’s talk about stopping criterion in decoding. In greedy decoding, we usually decode until model produces a <END> token, for example, `<START> he hit me with a pie <END>`. In beam search decoding, different hypotheses may produce <END> tokens on different timesteps. When a hypothesis produces <END>, that hypothesis is complete. Beam searching places it aside and continues exploring other hypotheses. We usually continue beam search until we reach timestep T ( where T is some pre-defined cutoff), or we have at least n completed hypotheses (where n is pre-defined cutoff).

Each hypothesis y1,…yt on our list has a score

normalized beam search score.Source: the course slide

Now we have our list of completed hypotheses, but how to select the top one with the highest score? Each hypothesis y1,…yt on our list has a score. The problem with this is that longer hypotheses have lower scores. We could fix this by normalizing the score by a length t.

Compared to SMT, NMT has many advantages: 1) it has better performance: more fluent, better use of context and better use of phrase similarities; 2) A single neural network to be optimized end-to-end: no subcomponents to be individually optimized; 3) it requires much less human engineering effort: no feature engineering, and same method for all language pairs. Compared to SMT, there are also some disadvantages, such as NMT is less interpretable and difficult to control (s.t cant easily specify rules or guidelines for translation).

We use BLEU to evaluate ML. BLEU stands for Bilingual Evaluation Understudy. It compares the machine-written translation to one or serval human-written translation(s) and computes a similarity score based on: n-gram precision (usually for 1,2,3 and 4-grams), plus a penalty for too-short system translations. BLEU is useful but imperfect. There are many valid ways to translate a sentence. So a good translation can get a poor BLEU score because it has low n-gram overlap with human translation.

NMT makes good progress, but machine translation is not solved. NMT is the flagship task for NLP Deep Learning. NMT research has pioneered many of the recent innovations of NLP Deep learning. In 2019, NMT research continues to thrive. Researchers have found many, many improvements to the “vanilla” seq2seq NMT system we’ve presented today. But one improvement is so integral that it is the new vanilla, attention.