Building a Translation System In Minutes | by Ceshine Lee | Towards Data Science

Sequence-to-sequence(seq2seq)[1] is a versatile structure and capable of many things (language translation, text summarization[2], video captioning[3], etc.). For a short introduction to seq2seq, here are some good posts: [4][5].
Sean Robertson’s tutorial notebook[6] and Jeremy Howard’s lectures [6][7] are great starting points to get a firm grasp on the technical details of seq2seq. However, I’d try to avoid implementing all these details myself when dealing with real-world problems. It’s usually not a good idea to reinvent the wheel, especially when you’re very new to this field. I’ve found that OpenNMT project is very active, has good documentation, and can be used out-of-the-box:
opennmt.net
There also are some more general frameworks (for example, [8]), but may need some customization to make it work on your specific problem.
There are two official versions of OpenNMT:
OpenNMT-Lua (a.k.a. OpenNMT): the main project developed with LuaTorch.Optimized and stable code for production and large scale experiments.
OpenNMT-py: light version of OpenNMT using PyTorch.Initially created by the Facebook AI research team as a sample project for PyTorch, this version is easier to extend and is suited for research purpose but does not include all features.
We’re going to use the PyTorch version in the following sections. We will walk you through the steps needed to create a very basic translation system with a medium-sized dataset.
Clone the OpenNMT-py git repository on Github into a local folder:
github.com
You might want to fork the repository on Github if you’re planning to customize or extend it later. Also it is suggested in the README:
Codebase is nearing a stable 0.1 version. We currently recommend forking if you want stable code.
Here we’re going to use the dataset from AI Challenger — English-Chinese Machine Translation competition. It is a dataset with 10 million English-Chinese sentence pairs. The English copora are conversational English extracted from English learning websites and movie subtitles. From my understanding, most of the translation are submitted by enthusiasts, not necessarily professionals. The translated Chinese sentences are checked by human annotators.
challenger.ai
Downloading the dataset requires account sign-up and possibly ID verification (can’t remember whether the latter is mandatory). If that’s a problem for you, you can try datasets from WMT17.
There are some problems to the AI Challenger dataset: 1. The quality of the translation is not consistent. 2. Because many of sentences are from movie subtitles, the translation are often context-dependent (related to the previous or the next sentence). However, there are no context information available in the dataset.
Let’s see how the out-of-the-box model perform on this dataset. Because of memory restriction, I down-sampled the dataset to 1 million sentences.
(We’ll assume that you put the dataset into folder challenger under the OpenNMT root directory.)
The validation and test dataset comes in XML format. We need to convert it to plain text files where a line consists of a single sentence. A simple way to do that is using BeautifulSoup. Here’s a sample chunk of code:
with open(input_file, "r") as f:    soup = BeautifulSoup(f.read(), "lxml")    lines = [      (int(x["id"]), x.text.strip()) for x in soup.findAll("seg")]    # Ensure the same order    lines = sorted(lines, key=lambda x: x[0])
The input sentence must be tokenized with tokens space-separated.
For English, there are a few tokenizers to choose from. One example is nltk.tokenize.word_tokenize :
with open(output_file, "w") as f:    f.write(        "\n".join([             " ".join(word_tokenize(l[1]))             for l in lines         ])    )
It turns “It’s a neat one — two. Walker to Burton.” into “It ‘s a neat one — two . Walker to Burton .”.
For Chinese, we use the simplest character-level tokenization, that is, treat each character as a token:
with open(output_file, "w") as f:    f.write(        "\n".join([            " ".join([c if c != " " else "
" for c in l[1]])            for l in lines        ])    )
It turns “我就一天24小时都得在她眼皮子底下。” into “我 就 一 天 2 4 小 时 都 得 在 她 眼 皮 子 底 下 。”. (Note because the token are space-separated, we need a special token “
” to represent the space characters.)
(I didn’t provide full code for step 3 and step 4 because it’s really beginner-level Python programming. You should be able to complete these tasks by yourself.)
Simply run the following command in the root directory:
python preprocess.py -train_src challenger/train.en.sample \      -train_tg challenger/train.zh.sample \      -valid_src challenger/valid.en \      -valid_tgt challenger/valid.zh  \      -save_data challenger/opennmt -report_every 10000
The preprocessing script will go through the dataset, keep track of token frequencies, and construct a vocabulary list. I ran into memory problem here and had to down-sample the training dataset to 1 million rows, but I think the raw dataset should fit into 16GB memory with some optimization.
python train.py -data challenger/opennmt \    -save_model models/baseline -gpuid 0 \    -learning_rate 0.001 -opt adam -epochs 20
It’ll use your first GPU to train a model. The default model structure is:
NMTModel (  (encoder): RNNEncoder (    (embeddings): Embeddings (      (make_embedding): Sequential (        (emb_luts): Elementwise (          (0): Embedding(50002, 500, padding_idx=1)        )      )    )    (rnn): LSTM(500, 500, num_layers=2, dropout=0.3)  )  (decoder): InputFeedRNNDecoder (    (embeddings): Embeddings (      (make_embedding): Sequential (        (emb_luts): Elementwise (          (0): Embedding(6370, 500, padding_idx=1)        )      )    )    (dropout): Dropout (p = 0.3)    (rnn): StackedLSTM (      (dropout): Dropout (p = 0.3)      (layers): ModuleList (        (0): LSTMCell(1000, 500)        (1): LSTMCell(500, 500)      )    )    (attn): GlobalAttention (      (linear_in): Linear (500 -> 500)      (linear_out): Linear (1000 -> 500)      (sm): Softmax ()      (tanh): Tanh ()    )  )  (generator): Sequential (    (0): Linear (500 -> 6370)    (1): LogSoftmax ()  ))
The vocabulary size of source and target corpora is 50,002 and 6,370, respectively. The source vocabulary is obviously truncated to 50,000. The target vocabulary is relatively small because there are not that many common Chinese characters.
python translate.py \    -model models/baseline_acc_58.79_ppl_7.51_e14  \    -src challenger/valid.en -tgt challenger/valid.zh \    -output challenger/valid_pred.58.79 -gpu 0 -replace_unk
Replace models/baseline_acc_58.79_ppl_7.51_e14 with your own model. The model naming should be obvious: this is a model after 14 epochs of training, with 58.79 accuracy and 7.51 perplexity on validation set.
You can also calculate BLEU score with the following:
wget https://raw.githubusercontent.com/moses-smt/mosesdecoder/master/scripts/generic/multi-bleu.perlperl multi-bleu.perl challenger/valid.zh \     < challenger/valid_pred.58.79
Now you have a working translation system!
If you want to submit the translation to AI Challenger, you need to reverse Step 4 and then Step 3. Again, they should be quite simple to implement.
English: You knew it in your heart you haven’t washed your hairChinese(pred): 你心里清楚你没洗头发Chinese(gold): 你心里知道你压根就没洗过头
English: I never dreamed that one of my own would be going off to a University, but here I stand,Chinese(pred): 我从来没梦到过我的一个人会去大学,但是我站在这里,Chinese(gold): 我从没想过我的孩子会上大学,但我站在这,
English: We just don’t have time to waste on the wrong man.Chinese(pred): 我们只是没时间浪费人。Chinese(gold): 如果找错了人我们可玩不起。
The above three examples are, from top to bottom, semantically correct, partially correct, and entirely incomprehensible. After examining a few examples, I found most of the machine translated sentences were partially correct, and there were surprising amount of semantically correct ones. Not a bad result, considering how little effort we’ve had put in so far.
If you submit the result you should get around .22 BLEU. The current top BLEU score is .33, so there’s a lot of rooms for improvement. You can check out opts.py in the root folder for more built-in model parameters. Or dive deep into the codebase to figure out how things work and where might be improved.
The other paths include applying word segmentation on Chinese sentences, adding named entity recognition, using pronunciation dictionary[10] to guess translation to unseen English names, etc.
(Update on 2017/10/14: If you use jieba and jieba.cut with default settings to tokenize the Chinese sentence, you’d get around .20 BLEU on public leaderboad. One of the possible reasons to the score drop is its much larger Chinese vocabulary size. You can tell from the number of <unk> in the output.)
Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks.Nallapati, R., Zhou, B., dos Santos, C., & Xiang, B. (2016). Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond.Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., & Saenko, K. (2015). Sequence to SequenceSequence to sequence model: Introduction and conceptsseq2seq: the clown car of deep learningPractical PyTorch: Translation with a Sequence to Sequence Network and AttentionCutting Edge Deep Learning For Coders, Part 2, Lecture 12 — Attention ModelsCutting Edge Deep Learning For Coders, Part 2, Lecture 13 — Neural TranslationGoogle/seq2seq: A general-purpose encoder-decoder framework for TensorflowThe CMU Pronouncing Dictionary
Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks.
Nallapati, R., Zhou, B., dos Santos, C., & Xiang, B. (2016). Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond.
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., & Saenko, K. (2015). Sequence to Sequence
Sequence to sequence model: Introduction and concepts
seq2seq: the clown car of deep learning
Practical PyTorch: Translation with a Sequence to Sequence Network and Attention
Cutting Edge Deep Learning For Coders, Part 2, Lecture 12 — Attention Models
Cutting Edge Deep Learning For Coders, Part 2, Lecture 13 — Neural Translation
Google/seq2seq: A general-purpose encoder-decoder framework for Tensorflow
The CMU Pronouncing Dictionary