Repository: tensorflow/tensor2tensor

Issue:
translate_enzh_wmt8k does not work
It seems the t2t-trainer was expecting a single vocab but clearly the datagen generated 2 vocab (respective language) for this problem. t2t-trainer subsequently could not train this problem. 

`NewRandomAccessFile failed to Create/Open: L:/python34_colman/projects/trans/t2t_data\vocab.zh.8192 : The system cannot find the file specified.`

vocab.zhen-en.8192
vocab.zhen-zh.8192  however are found at the directory instead.


Discussion:
Stack trace? 

And why does the final part of the path there have a `\` instead of a `/`?
i have not really an idea, the directories are all with / i.e. folder/folder/lastfolder 
seems \ is more of t2t-trainer's doing.

trace here:
```
Traceback (most recent call last):
  File "L:/python34_colman/tensor2tensor/tensor2tensor/bin/t2t-trainer", line 87, in <module>
    tf.app.run()
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "L:/python34_colman/tensor2tensor/tensor2tensor/bin/t2t-trainer", line 83, in main
    schedule=FLAGS.schedule)
  File "l:\python34_colman\tensor2tensor\tensor2tensor\utils\trainer_utils.py", line 290, in run
    exp = exp_fn(output_dir)
  File "l:\python34_colman\tensor2tensor\tensor2tensor\utils\trainer_utils.py", line 138, in experiment_fn
    eval_steps=eval_steps)
  File "l:\python34_colman\tensor2tensor\tensor2tensor\utils\trainer_utils.py", line 146, in create_experiment
    hparams = create_hparams(FLAGS.hparams_set, data_dir)
  File "l:\python34_colman\tensor2tensor\tensor2tensor\utils\trainer_utils.py", line 262, in create_hparams
    return add_problem_hparams(hparams, FLAGS.problems)
  File "l:\python34_colman\tensor2tensor\tensor2tensor\utils\trainer_utils.py", line 234, in add_problem_hparams
    p_hparams = problem.internal_hparams(hparams)
  File "l:\python34_colman\tensor2tensor\tensor2tensor\data_generators\problem.py", line 239, in internal_hparams
    self.internal_build_encoders(model_hparams.data_dir)
  File "l:\python34_colman\tensor2tensor\tensor2tensor\data_generators\problem.py", line 234, in internal_build_encoders
    self._encoders = self.feature_encoders(data_dir)
  File "l:\python34_colman\tensor2tensor\tensor2tensor\data_generators\wmt.py", line 489, in feature_encoders
    source_token = text_encoder.SubwordTextEncoder(source_vocab_filename)
  File "l:\python34_colman\tensor2tensor\tensor2tensor\data_generators\text_encoder.py", line 309, in __init__
    self._load_from_file(filename)
  File "l:\python34_colman\tensor2tensor\tensor2tensor\data_generators\text_encoder.py", line 581, in _load_from_file
    for line in f:
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 213, in __next__
    return self.next()
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 207, in next
    retval = self.readline()
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 176, in readline
    self._preread_check()
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 78, in _preread_check
    compat.as_bytes(self.__name), 1024 * 512, status)
  File "C:\ProgramData\Anaconda3\lib\contextlib.py", line 88, in __exit__
    next(self.gen)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: L:/python34_colman/projects/trans/t2t_data\vocab.zh.8192 : The system cannot find the file specified.
```
the `\` might be a windows thing, but I am also running into same issue on ubuntu.

The file is being called by `def feature_encoder()`, and if i'm not mistaken, differs from `de-en` tasks in that there no subword/bpe encodings. Is it intended to use the vocab files directly? 

`vocab.zh.8192` => `vocab.zhen.zh.8192`
`vocab.en.8192` => `vocab.zhen.en.8192`



````
$ t2t-trainer   --generate_data   --data_dir=~/t2t_data   --problems=translate_enzh_wmt8k   --model=transformer   
--hparams_set=transformer_base_single_gpu   --output_dir=~/t2t_train/base
INFO:tensorflow:Generating data for translate_enzh_wmt8k
INFO:tensorflow:Found vocab file: /home/paperspace/t2t_data/vocab.zhen-zh.8192
INFO:tensorflow:Found vocab file: /home/paperspace/t2t_data/vocab.zhen-en.8192
INFO:tensorflow:Not downloading, file already found: /tmp/t2t_datagen/training-parallel-nc-v12.tgz
INFO:tensorflow:Found vocab file: /home/paperspace/t2t_data/vocab.zhen-zh.8192
INFO:tensorflow:Found vocab file: /home/paperspace/t2t_data/vocab.zhen-en.8192
INFO:tensorflow:Not downloading, file already found: /tmp/t2t_datagen/dev.tgz
INFO:tensorflow:Generating case 100000.
INFO:tensorflow:Generating case 200000.
INFO:tensorflow:Shuffling data...
Traceback (most recent call last):
  File "/usr/local/bin/t2t-trainer", line 87, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/usr/local/bin/t2t-trainer", line 83, in main
    schedule=FLAGS.schedule)
  File "/usr/local/lib/python3.4/dist-packages/tensor2tensor/utils/trainer_utils.py", line 298, in run
    exp = exp_fn(output_dir)
  File "/usr/local/lib/python3.4/dist-packages/tensor2tensor/utils/trainer_utils.py", line 140, in experiment_fn
    eval_steps=eval_steps)
  File "/usr/local/lib/python3.4/dist-packages/tensor2tensor/utils/trainer_utils.py", line 149, in create_experiment
    FLAGS.hparams_set, FLAGS.problems, data_dir, passed_hparams=FLAGS.hparams)
  File "/usr/local/lib/python3.4/dist-packages/tensor2tensor/utils/trainer_utils.py", line 270, in create_hparams
    return add_problem_hparams(hparams, problems)
  File "/usr/local/lib/python3.4/dist-packages/tensor2tensor/utils/trainer_utils.py", line 240, in add_problem_hparams
    p_hparams = problem.internal_hparams(hparams)
  File "/usr/local/lib/python3.4/dist-packages/tensor2tensor/data_generators/problem.py", line 242, in internal_hparams
    self.internal_build_encoders(model_hparams.data_dir)
  File "/usr/local/lib/python3.4/dist-packages/tensor2tensor/data_generators/problem.py", line 237, in internal_build_encoders
    self._encoders = self.feature_encoders(data_dir)
  File "/usr/local/lib/python3.4/dist-packages/tensor2tensor/data_generators/wmt.py", line 516, in feature_encoders
    source_token = text_encoder.SubwordTextEncoder(source_vocab_filename)
  File "/usr/local/lib/python3.4/dist-packages/tensor2tensor/data_generators/text_encoder.py", line 364, in __init__
    self._load_from_file(filename)
  File "/usr/local/lib/python3.4/dist-packages/tensor2tensor/data_generators/text_encoder.py", line 674, in _load_from_file
    self._load_from_file_object(f)
  File "/usr/local/lib/python3.4/dist-packages/tensor2tensor/data_generators/text_encoder.py", line 657, in _load_from_file_object
    for line in f:
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/lib/io/file_io.py", line 213, in __next__
    return self.next()
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/lib/io/file_io.py", line 207, in next
    retval = self.readline()
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/lib/io/file_io.py", line 176, in readline
    self._preread_check()
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/lib/io/file_io.py", line 78, in _preread_check
    compat.as_bytes(self.__name), 1024 * 512, status)
  File "/usr/lib/python3.4/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_stat
us
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: /home/paperspace/t2t_data/vocab.zh.8192
````
> differs from de-en tasks in that there no subword/bpe encodings

No. It uses the same text_encoder.SubwordTextEncoder. But unlike en-de tasks, it uses the subword encoder separately for the source (zh) and target (en) language.
I think the problem will be solved by fixing lines [513 and 515 of wmt.py](https://github.com/tensorflow/tensor2tensor/blob/56cb37f64f0415e6a99005dcac9d04d7ecd0da94/tensor2tensor/data_generators/wmt.py#L513-L515),
"vocab." => "vocab.zhen-", so they are compatible with lines [489 and 492](https://github.com/tensorflow/tensor2tensor/blob/56cb37f64f0415e6a99005dcac9d04d7ecd0da94/tensor2tensor/data_generators/wmt.py#L489-L492).
thanks, going to try it out asap. 

Resolving patch:
diff --git a/tensor2tensor/data_generators/wmt.py b/tensor2tensor/data_generators/wmt.py
--- a/tensor2tensor/data_generators/wmt.py
+++ b/tensor2tensor/data_generators/wmt.py
@@ -510,9 +510,9 @@ def target_space_id(self):
   def feature_encoders(self, data_dir):
     vocab_size = self.targeted_vocab_size
     source_vocab_filename = os.path.join(data_dir,
-                                         "vocab.zh.%d" % vocab_size)
+                                         "vocab.zhen-zh.%d" % vocab_size)
     target_vocab_filename = os.path.join(data_dir,
-                                         "vocab.en.%d" % vocab_size)
+                                         "vocab.zhen-en.%d" % vocab_size)
     source_token = text_encoder.SubwordTextEncoder(source_vocab_filename)
     target_token = text_encoder.SubwordTextEncoder(target_vocab_filename)
     return {


Test patch: