Repository: OpenNMT/OpenNMT-tf

Issue:
Error when using pre-trained word embedding features
System configuration:
Tensorflow : 1.8.0 
OpenNMT-tf: 1.5.0

I use pre-trained word embedding for training Transformer, and it can only last for one epoch and will be failed when it is constructing the model for the next epoch. Here is my code, the "embedding_file_key" and "embedding_file_with_header" caused the error:
```

class Transformer(onmt.models.Transformer):
  """Defines a Transformer model as decribed in https://arxiv.org/abs/1706.03762."""
  def __init__(self):
    super(Transformer, self).__init__(
        source_inputter=onmt.inputters.WordEmbedder(
            vocabulary_file_key="source_words_vocabulary",
            embedding_file_key="source_words_embeddings",
            embedding_file_with_header=False,
            embedding_size=512),
        target_inputter=onmt.inputters.WordEmbedder(
            vocabulary_file_key="target_words_vocabulary",
            embedding_file_key="target_words_embeddings",
            embedding_file_with_header=False,
            embedding_size=512),
        num_layers=6,
        num_units=512,
        num_heads=8,
        ffn_inner_dim=2048,
        dropout=0.1,
        attention_dropout=0.1,
        relu_dropout=0.1)

model = Transformer()

runner = Runner(
      model,
      config,
      seed=None,
      num_devices=1,
      gpu_allow_growth=True)

runner.train_and_evaluate()
```
My config file is almost the same with [OpenNMT-tf/scripts/wmt/config/wmt_ende.yml](https://github.com/OpenNMT/OpenNMT-tf/blob/master/scripts/wmt/config/wmt_ende.yml) except adding "source_words_embeddings" and "target_words_embeddings" :
```
# The directory where models and summaries will be saved. It is created if it does not exist.
#model_dir: /home/lgy/deepModels/tf_models/wmt_ende_transformer_4gpu_lr2_ws8000_dur2_0.998
model_dir: /home/lgy/deepModels/tf_models

data:
  train_features_file: /home/lgy/data/wmt/wmt14-de-en/amax/train.en
  train_labels_file: /home/lgy/data/wmt/wmt14-de-en/amax/train.de
  eval_features_file: /home/lgy/data/wmt/wmt14-de-en/amax/valid.en
  eval_labels_file: /home/lgy/data/wmt/wmt14-de-en/amax/valid.de
  source_words_vocabulary: /home/lgy/data/wmt/wmt14-de-en/amax/wmtende.vocab
  target_words_vocabulary: /home/lgy/data/wmt/wmt14-de-en/amax/wmtende.vocab
  source_words_embeddings: /home/lgy/deepModels/tf_models/wmt_ende_transformer_4gpu_lr2_ws8000_dur2_0.998/compositional_encode_M32K16.txt
  target_words_embeddings: /home/lgy/deepModels/tf_models/wmt_ende_transformer_4gpu_lr2_ws8000_dur2_0.998/compositional_decode_M32K16.txt
```
I can run the first epoch successfully, but failed when the model is constructing for the next epoch. **It should be noted that if I delete the the "embedding_file_key" and "embedding_file_with_header" in the code, it ran successfully without error.** Here is the error message:
```
INFO:tensorflow:Saving checkpoints for 16221 into /home/lgy/deepModels/tf_models/wmt_ende_transformer_1gpu_lr2_ws8000_dur2_0.998_compositionalencoding_M64K16/model.ckpt.
INFO:tensorflow:Loss for final step: 3.9958026.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2018-06-14-15:33:54
INFO:tensorflow:Graph was finalized.
2018-06-14 23:33:54.898526: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-14 23:33:54.898579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-14 23:33:54.898588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-14 23:33:54.898594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-14 23:33:54.898894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15127 MB memory) -> physical GPU (device: 0, name: Tesla P100-SXM2-16GB, pci bus id: 0000:89:00.0, compute capability: 6.0)
INFO:tensorflow:Restoring parameters from /home/lgy/deepModels/tf_models/wmt_ende_transformer_1gpu_lr2_ws8000_dur2_0.998_compositionalencoding_M64K16/model.ckpt-16221
INFO:tensorflow:Running local_init_op.
2018-06-14 23:33:55.571416: I tensorflow/core/kernels/lookup_util.cc:373] Table trying to initialize from file /home/lgy/data/wmt/wmt14-de-en/amax/wmtende.vocab is already initialized.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Evaluation predictions saved to /home/lgy/deepModels/tf_models/wmt_ende_transformer_1gpu_lr2_ws8000_dur2_0.998_compositionalencoding_M64K16/eval/predictions.txt.16221
INFO:tensorflow:BLEU evaluation score: 15.020000
INFO:tensorflow:Finished evaluation at 2018-06-14-15:39:44
INFO:tensorflow:Saving dict for global step 16221: global_step = 16221, loss = 3.0185592
INFO:tensorflow:Calling model_fn.
Traceback (most recent call last):
  File "tmp_train_transformer.py", line 74, in <module>
    runner.train_and_evaluate()
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/OpenNMT_tf-1.5.0-py2.7.egg/opennmt/runner.py", line 148, in train_and_evaluate
    tf.estimator.train_and_evaluate(self._estimator, train_spec, eval_spec)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 439, in train_and_evaluate
    executor.run()
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 518, in run
    self.run_local()
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 657, in run_local
    eval_result = evaluator.evaluate_and_export()
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 858, in evaluate_and_export
    self._export_eval_result(eval_result, is_the_final_export)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 889, in _export_eval_result
    is_the_final_export=is_the_final_export)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/estimator/exporter.py", line 232, in export
    is_the_final_export)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/estimator/exporter.py", line 123, in export
    strip_default_attrs=self._strip_default_attrs)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 613, in export_savedmodel
    config=self.config)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 831, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/OpenNMT_tf-1.5.0-py2.7.egg/opennmt/models/model.py", line 128, in _model_fn
    _, predictions = self._build(features, labels, params, mode, config=config)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/OpenNMT_tf-1.5.0-py2.7.egg/opennmt/models/sequence_to_sequence.py", line 185, in _build
    return_alignment_history=True))
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/OpenNMT_tf-1.5.0-py2.7.egg/opennmt/decoders/self_attention_decoder.py", line 315, in dynamic_decode_and_search
    eos_id=end_token)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/OpenNMT_tf-1.5.0-py2.7.egg/opennmt/utils/beam_search.py", line 557, in beam_search
    back_prop=False)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3224, in while_loop
    result = loop_context.BuildLoop(cond, body, loop_vars, shape_invariants)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2956, in BuildLoop
    pred, body, original_loop_vars, loop_vars, shape_invariants)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2893, in _BuildLoop
    body_result = body(*packed_vars_for_body)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/OpenNMT_tf-1.5.0-py2.7.egg/opennmt/utils/beam_search.py", line 485, in inner_loop
    i, alive_seq, alive_log_probs, states)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/OpenNMT_tf-1.5.0-py2.7.egg/opennmt/utils/beam_search.py", line 379, in grow_topk
    flat_logits, flat_states = symbols_to_logits_fn(flat_ids, i, flat_states)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/OpenNMT_tf-1.5.0-py2.7.egg/opennmt/decoders/self_attention_decoder.py", line 101, in _impl
    inputs = embedding_fn(ids[:, -1:])
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/OpenNMT_tf-1.5.0-py2.7.egg/opennmt/models/sequence_to_sequence.py", line 103, in _target_embedding_fn
    return self.target_inputter.transform(ids, mode=mode)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/OpenNMT_tf-1.5.0-py2.7.egg/opennmt/inputters/text_inputter.py", line 390, in transform
    trainable=self.trainable)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1317, in get_variable
    constraint=constraint)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1079, in get_variable
    constraint=constraint)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 425, in get_variable
    constraint=constraint)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 394, in _true_getter
    use_resource=use_resource, constraint=constraint)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 751, in _get_single_variable
    "reuse=tf.AUTO_REUSE in VarScope?" % name)
ValueError: Variable transformer/decoder/w_embs does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=tf.AUTO_REUSE in VarScope?
``` 

This message starts from the end of the first epoch including evaluating BLEU on the validation set and ends at the ValueError. According to the error message,  I edited line 103 in “/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/OpenNMT_tf-1.5.0-py2.7.egg/opennmt/models/sequence_to_sequence.py” from 
```
def _scoped_target_embedding_fn(self, mode, scope):
    def _target_embedding_fn(ids):
      try:
        with tf.variable_scope(scope):
          return self.target_inputter.transform(ids, mode=mode)
      except ValueError:
        with tf.variable_scope(scope, reuse=True):
          return self.target_inputter.transform(ids, mode=mode) # line 103
    return _target_embedding_fn
```
to
```
def _scoped_target_embedding_fn(self, mode, scope):
    def _target_embedding_fn(ids):
      try:
        with tf.variable_scope(scope):
          return self.target_inputter.transform(ids, mode=mode)
      except ValueError:
        with tf.variable_scope(scope, reuse=tf.AUTO_REUSE): # I edit here
          return self.target_inputter.transform(ids, mode=mode)
    return _target_embedding_fn
```
Then the running failed again at the same place but with a different error message (The first epoch is also successful and it fails at the beginning of the second epoch). The message is
```
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/OpenNMT_tf-1.1.0-py2.7.egg/opennmt/models/sequence_to_sequence.py", line 105, in _target_embedding_fn
    return self.target_inputter.transform(ids, mode=mode)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/OpenNMT_tf-1.1.0-py2.7.egg/opennmt/inputters/text_inputter.py", line 390, in transform
    trainable=self.trainable)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1297, in get_variable
    constraint=constraint)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1093, in get_variable
    constraint=constraint)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 439, in get_variable
    constraint=constraint)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 408, in _true_getter
    use_resource=use_resource, constraint=constraint)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 800, in _get_single_variable
    use_resource=use_resource)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 2157, in variable
    use_resource=use_resource)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 2147, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 2130, in default_variable_creator
    constraint=constraint)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 235, in __init__
    constraint=constraint)
  File "/home/lgy/test/nmt_transformer_opennmt_tf-workspace/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 356, in _init_from_args
    "initializer." % name)
ValueError: Initializer for variable transformer/decoder/w_embs_1/ is from inside a control-flow construct, such as a loop or conditional. When creating a variable inside a loop or conditional, use a lambda as the initializer.
```  

I think this is a bug, but I am not sure whether it is the bug of OpenNMT-tf or tensorflow, and I found a issue https://github.com/tensorflow/tensorflow/issues/14729 talks about a similar error in github of tensorflow.


Discussion:

Resolving patch:
diff --git a/opennmt/inputters/text_inputter.py b/opennmt/inputters/text_inputter.py
--- a/opennmt/inputters/text_inputter.py
+++ b/opennmt/inputters/text_inputter.py
@@ -376,12 +376,12 @@ def transform(self, inputs, mode):
             case_insensitive_embeddings=self.case_insensitive_embeddings)
         self.embedding_size = pretrained.shape[-1]
 
-        shape = None
-        initializer = tf.constant(pretrained.astype(self.dtype.as_numpy_dtype()))
+        initializer = tf.constant_initializer(
+            pretrained.astype(self.dtype.as_numpy_dtype()), dtype=self.dtype)
       else:
-        shape = [self.vocabulary_size, self.embedding_size]
         initializer = None
 
+      shape = [self.vocabulary_size, self.embedding_size]
       embeddings = tf.get_variable(
           "w_embs",
           shape=shape,


Test patch: