Repository: tensorflow/tensor2tensor

Issue:
problem TranslateEnzhWmt8k
I run the Chinese-to-English translation task as the default set, the "t2t-decoder" scripts can not translate Chinese segment into English sentence, but only outputs some short Chinese sentence:

INFO:tensorflow:Inference results INPUT: 雇主们 希望 继续 积极 探讨 其他 重要 话题 —— 如 部署 、 培训 的 灵活性 、 针对 那些 在 职业 生涯 中断 后重 上岗 之 人 的 其他 培训 、 培训 成本 、 对 课程 提纲 的 共同 认知 、 进修 假 和 医疗 领域 的 性别 工资 差异 ， 这些 说明 了 雇主 、 英格兰 健康 教育 组织 ( Health Education England ) 和 卫生部 对 11月 、 2月 及 5月 与 BMA 达成 的 协议 很 重视 。
INFO:tensorflow:Inference results OUTPUT: 这 是 一 个 错误 。
INFO:tensorflow:Inference results INPUT: 在 该 法 院本 周 的 听证会 上 ， 加莱 政府 辩称 法国 政府 应 有 权 拆除 该 难民营 内 总计 72 间 临时 搭建 的 商店 。 民众 在 难民营 中的 这些 商店 里 销售 从 香烟 到 能量 饮料 在内 的 商品 、 提供 理发 服务 ， 还 在 临时 搭建 的 餐馆 中 以 50 美分 （ 40 便士 ） 的 价格 卖茶 或 以 几 欧 元 的 价格 售卖 基本 餐点 。
INFO:tensorflow:Inference results OUTPUT: 但 这 一 比例 并非 如此 。

 although the outputs of "t2t-datagen" and "t2t-trainer" scripts seems fine:

INFO:tensorflow:Saving dict for global step 250000: global_step = 250000, loss = 1.29398, metrics-translate_enzh_wmt8k/accuracy = 0.622824, metrics-translate_enzh_wmt8k/accuracy_per_sequence = 0.0, metrics-translate_enzh_wmt8k/accuracy_top5 = 0.892174, metrics-translate_enzh_wmt8k/approx_bleu_score = 0.452572, metrics-translate_enzh_wmt8k/neg_log_perplexity = -1.56626, metrics-translate_enzh_wmt8k/rouge_2_fscore = 0.756268, metrics-translate_enzh_wmt8k/rouge_L_fscore = 0.538652

I have two question:
(1) Why the Chinese-to-English translation task do not work, and the problem name "TranslateEnzhWmt8k" seems to mean translating English to Chinese. 
(2) Why not split the Chinese sentence into Chinese words before training and decoding~

Thanks in advance!


Discussion:
Hi, have you tried feed in English sentences to it and see if it works properly?
tokenization is done in the script with a vocab size of 8k. 

I agree it is misleading (and I would suggest to fix it) that "TranslateEnzhWmt8k" internally uses _ZHEN_TRAIN_DATASETS, `source_vocab` is zh and `target_vocab` is en. However, there is [a trick](https://github.com/tensorflow/tensor2tensor/blob/37465a1759e278e8f073cd04cd9b4fe377d3c740/tensor2tensor/data_generators/wmt.py#L496-L500) where instead of "lang1, lang2", it uses "lang2, lang1". So in the end I guess `translate_enzh_wmt8k` is (as the name suggests) translation **from English to Chinese**. If you need **Chinese to English**, try `translate_enzh_wmt8k_rev`. 
(I am not the author. I have not even tried zh translation. This is just my guess based on the source code.)
@colmantse @martinpopel, thanks for your kind help.
I also try translate English to Chinese using the "t2t-decoder" script with the same trained model, but the outputs are also not right:
 
2017-09-21 20:48:13.634873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1055] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:02:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /home/mosesli/T4/transformer/z2e_train/translate_enzh_wmt8k/transformer-transformer_base_single_gpu/model.ckpt-250000
INFO:tensorflow:Inference results INPUT: employers were hopeful that the continued positive engagement on other important topics - such as deployment , flexibility in training , additional training for those returning from career breaks , costs of training , mutual recognition of syllabus , study leave and the gender pay gap in medicine - were a sign of how serious employers , health education england and the department of health were about honouring the agreements reached with the bma in november , february and may .
INFO:tensorflow:Inference results OUTPUT: 这 是 一 个 错误 。
INFO:tensorflow:Inference results INPUT: " it 's either that they had no diverse representation of people of color on their staff to the point where they 're able to make decisions like this , " ms. zhu , who is chinese-american , said , " or they do have some people of color who are working there , but they 're not on positions where they feel safe or comfortable to speak up . "
INFO:tensorflow:Inference results OUTPUT: 这 是 一 个 错误 。

I notice that the corresponding codes in "wmt.py" is change from lines start with "-" to lines start with "+" on 17 Aug by rsepassi, but I donot understand it.
 
 -@registry.register_problem("wmt_zhen_tokens_8k") # line 443
 -class WMTZhEnTokens8k(WMTProblem): #line 444
 +@registry.register_problem
 +class TranslateEnzhWmt8k(TranslateProblem):
 +"""Problem spec for WMT Zh-En translation."""
  
 -return bi_vocabs_token_generator(data_path + ".lang1", data_path + ".lang2",
 -#We generate English->X data by convention, to train reverse translation
 +# just add the "_rev" suffix to the problem name, e.g., like this.
 +#--problems=translate_enzh_wmt8k_rev
 +return bi_vocabs_token_generator(data_path + ".lang2", data_path + ".lang1", source_vocab, target_vocab, EOS)
- means remove, + means add. It looks like you have trained a chi --> en model, what is your loss and bleu?
Since the translation outputs are obviously wrong, I do not measure the BLEU score on the test set, but the results when training are as follows:

INFO:tensorflow:loss = 0.434695, step = 249601 (17.925 sec)
INFO:tensorflow:global_step/sec: 5.56531
INFO:tensorflow:loss = 0.395525, step = 249701 (17.968 sec)
INFO:tensorflow:global_step/sec: 5.55805
INFO:tensorflow:loss = 0.370602, step = 249801 (17.992 sec)
INFO:tensorflow:global_step/sec: 5.54219
INFO:tensorflow:loss = 0.401325, step = 249901 (18.044 sec)
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236], 'window_size': 720, 'max_length': 1000000000, 'batch_sizes': [240, 180, 180, 180, 144, 144, 144, 120, 120, 120, 90, 90, 90, 90, 80, 72, 72, 60, 60, 48, 48, 48, 40, 40, 36, 30, 30, 24, 24, 20, 20, 18, 18, 16, 15, 12, 12, 10, 10, 9, 8, 8], 'shuffle_queue_size': 270}
INFO:tensorflow:datashard_devices: ['gpu:0']
INFO:tensorflow:caching_devices: None
INFO:tensorflow:Unsetting shared_embedding_and_softmax_weights.
INFO:tensorflow:Doing model_fn_body took 1.501 sec.
INFO:tensorflow:This model_fn took 1.705 sec.
INFO:tensorflow:Starting evaluation at 2017-09-13-16:05:08
2017-09-14 00:05:09.127576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1055] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:02:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /home/mosesli/T4/transformer/z2e_train/translate_enzh_wmt8k/transformer-transformer_base_single_gpu/model.ckpt-248416
INFO:tensorflow:Evaluation [1/10]
INFO:tensorflow:Evaluation [2/10]
INFO:tensorflow:Evaluation [3/10]
INFO:tensorflow:Evaluation [4/10]
INFO:tensorflow:Evaluation [5/10]
INFO:tensorflow:Evaluation [6/10]
INFO:tensorflow:Evaluation [7/10]
INFO:tensorflow:Evaluation [8/10]
INFO:tensorflow:Evaluation [9/10]
INFO:tensorflow:Evaluation [10/10]
INFO:tensorflow:Finished evaluation at 2017-09-13-16:05:22
INFO:tensorflow:Saving dict for global step 248416: global_step = 248416, loss = 1.32257, metrics-translate_enzh_wmt8k/accuracy = 0.627322, metrics-translate_enzh_wmt8k/accuracy_per_sequence = 0.0, metrics-translate_enzh_wmt8k/accuracy_top5 = 0.888772, metrics-translate_enzh_wmt8k/approx_bleu_score = 0.451552, metrics-translate_enzh_wmt8k/neg_log_perplexity = -1.60012, metrics-translate_enzh_wmt8k/rouge_2_fscore = 0.758087, metrics-translate_enzh_wmt8k/rouge_L_fscore = 0.539738
INFO:tensorflow:Validation (step 250000): loss = 1.32257, metrics-translate_enzh_wmt8k/neg_log_perplexity = -1.60012, metrics-translate_enzh_wmt8k/accuracy = 0.627322, metrics-translate_enzh_wmt8k/accuracy_per_sequence = 0.0, metrics-translate_enzh_wmt8k/rouge_2_fscore = 0.758087, metrics-translate_enzh_wmt8k/approx_bleu_score = 0.451552, metrics-translate_enzh_wmt8k/accuracy_top5 = 0.888772, metrics-translate_enzh_wmt8k/rouge_L_fscore = 0.539738, global_step = 248416
INFO:tensorflow:Saving checkpoints for 250000 into /home/mosesli/T4/transformer/z2e_train/translate_enzh_wmt8k/transformer-transformer_base_single_gpu/model.ckpt.
...
INFO:tensorflow:Finished evaluation at 2017-09-13-16:05:42
INFO:tensorflow:Saving dict for global step 250000: global_step = 250000, loss = 1.29398, metrics-translate_enzh_wmt8k/accuracy = 0.622824, metrics-translate_enzh_wmt8k/accuracy_per_sequence = 0.0, metrics-translate_enzh_wmt8k/accuracy_top5 = 0.892174, metrics-translate_enzh_wmt8k/approx_bleu_score = 0.452572, metrics-translate_enzh_wmt8k/neg_log_perplexity = -1.56626, metrics-translate_enzh_wmt8k/rouge_2_fscore = 0.756268, metrics-translate_enzh_wmt8k/rouge_L_fscore = 0.538652

      The final loss is 1.29398, and the BLEU score is 0.452572.
hi, i am not sure what has gone wrong, but it might be due to small data size that it did not work so well? I suggest you use some dev set sentence and check how they perform, at least then you would be able to figure if it output chi or en.
@colmantse @martinpopel OK, thank you very much!
Finally, I change the line 502 in data_generators/wmt.py from

-#return bi_vocabs_token_generator(data_path + ".lang2", data_path + ".lang1", source_vocab, target_vocab, EOS)

to 

+return bi_vocabs_token_generator(data_path + ".lang1", data_path + ".lang2", source_vocab, target_vocab, EOS)

then it can translate Chinese to English, and the BLEU score on the test set of newstest2017-zhen is about 11.07 with the default datasets and parameters. 
I think I can answer the question why the words split is not done. From our experiments, the  character based Chinese encoding leads to better results than the one using Chinese word split. More corpus you feed, better results you will get. The transformer model can learn how to form words from characters and it's a better way to do this in domain specific corpus from my point of view.
I suppose we should change the name from translate_enzh_wmt8k to translate_enzh_wmt8k right? Is there anything else we should do to correct this in the next release? Thanks for your suggestions!
> change the name from translate_enzh_wmt8k to translate_enzh_wmt8k

I guess, you mean from translate_enzh_wmt8k to translate_zhen_wmt8k.
I have another suggestion: follow the style of other datasets, that is:
* change _ZHEN_TRAIN_DATASETS to _ENZH_TRAIN_DATASETS
* get rid of [the hack](https://github.com/tensorflow/tensor2tensor/blob/37465a1759e278e8f073cd04cd9b4fe377d3c740/tensor2tensor/data_generators/wmt.py#L496-L500) where instead of "lang1, lang2", it uses "lang2, lang1"
* This way the default will stay en->zh as it is now. Whoever wants zh->en, must use `translate_enzh_wmt8k_rev`.

Resolving patch:
diff --git a/tensor2tensor/data_generators/translate_enzh.py b/tensor2tensor/data_generators/translate_enzh.py
--- a/tensor2tensor/data_generators/translate_enzh.py
+++ b/tensor2tensor/data_generators/translate_enzh.py
@@ -35,21 +35,23 @@
 
 # End-of-sentence marker.
 EOS = text_encoder.EOS_ID
-
-_ZHEN_TRAIN_DATASETS = [[("http://data.statmt.org/wmt17/translation-task/"
+# This is far from being the real WMT17 task - only toyset here
+# you need to register to get UN data and CWT data
+# also by convention this is EN to ZH - use translate_enzh_wmt8k_rev for ZH to EN task
+_ENZH_TRAIN_DATASETS = [[("http://data.statmt.org/wmt17/translation-task/"
                           "training-parallel-nc-v12.tgz"),
-                         ("training/news-commentary-v12.zh-en.zh",
-                          "training/news-commentary-v12.zh-en.en")]]
+                         ("training/news-commentary-v12.zh-en.en",
+                          "training/news-commentary-v12.zh-en.zh")]]
 
-_ZHEN_TEST_DATASETS = [[
+_ENZH_TEST_DATASETS = [[
     "http://data.statmt.org/wmt17/translation-task/dev.tgz",
-    ("dev/newsdev2017-zhen-src.zh.sgm", "dev/newsdev2017-zhen-ref.en.sgm")
+    ("dev/newsdev2017-zhen-src.en.sgm", "dev/newsdev2017-zhen-ref.zh.sgm")
 ]]
 
 
 @registry.register_problem
 class TranslateEnzhWmt8k(translate.TranslateProblem):
-  """Problem spec for WMT Zh-En translation."""
+  """Problem spec for WMT En-Zh translation."""
 
   @property
   def targeted_vocab_size(self):
@@ -61,16 +63,16 @@ def num_shards(self):
 
   @property
   def source_vocab_name(self):
-    return "vocab.zhen-zh.%d" % self.targeted_vocab_size
+    return "vocab.enzh-en.%d" % self.targeted_vocab_size
 
   @property
   def target_vocab_name(self):
-    return "vocab.zhen-en.%d" % self.targeted_vocab_size
+    return "vocab.enzh-zh.%d" % self.targeted_vocab_size
 
   def generator(self, data_dir, tmp_dir, train):
-    datasets = _ZHEN_TRAIN_DATASETS if train else _ZHEN_TEST_DATASETS
-    source_datasets = [[item[0], [item[1][0]]] for item in _ZHEN_TRAIN_DATASETS]
-    target_datasets = [[item[0], [item[1][1]]] for item in _ZHEN_TRAIN_DATASETS]
+    datasets = _ENZH_TRAIN_DATASETS if train else _ENZH_TEST_DATASETS
+    source_datasets = [[item[0], [item[1][0]]] for item in _ENZH_TRAIN_DATASETS]
+    target_datasets = [[item[0], [item[1][1]]] for item in _ENZH_TRAIN_DATASETS]
     source_vocab = generator_utils.get_or_generate_vocab(
         data_dir, tmp_dir, self.source_vocab_name, self.targeted_vocab_size,
         source_datasets)
@@ -79,21 +81,18 @@ def generator(self, data_dir, tmp_dir, train):
         target_datasets)
     tag = "train" if train else "dev"
     data_path = translate.compile_data(tmp_dir, datasets,
-                                       "wmt_zhen_tok_%s" % tag)
-    # We generate English->X data by convention, to train reverse translation
-    # just add the "_rev" suffix to the problem name, e.g., like this.
-    #   --problems=translate_enzh_wmt8k_rev
-    return translate.bi_vocabs_token_generator(data_path + ".lang2",
-                                               data_path + ".lang1",
+                                       "wmt_enzh_tok_%s" % tag)
+    return translate.bi_vocabs_token_generator(data_path + ".lang1",
+                                               data_path + ".lang2",
                                                source_vocab, target_vocab, EOS)
 
   @property
   def input_space_id(self):
-    return problem.SpaceID.ZH_TOK
+    return problem.SpaceID.EN_TOK
 
   @property
   def target_space_id(self):
-    return problem.SpaceID.EN_TOK
+    return problem.SpaceID.ZH_TOK
 
   def feature_encoders(self, data_dir):
     source_vocab_filename = os.path.join(data_dir, self.source_vocab_name)


Test patch: