Repository: OpenNMT/OpenNMT-tf

Issue:
GPT-2 training has problem: ValueError: Duplicate node name in graph: 'lm/w_embs'
env: opennmt-tf=master, cuda=9.0,gpu=4
model:gpt_2.py
error info:
```python
INFO:tensorflow:Using parameters:
data:
  eval_features_file: dev_mini.txt
  train_features_file: train_full.txt
  vocabulary: tokens_v3.gpt2
eval:
  batch_size: 32
  eval_delay: 1200
  exporter: last
  exporters: last
  external_evaluators: null
  num_threads: 1
  prefetch_buffer_size: 1
  save_eval_predictions: false
infer:
  batch_size: 32
  bucket_width: 5
  n_best: 1
  num_threads: 1
  prefetch_buffer_size: 1
  with_scores: false
model_dir: run
params:
  average_loss_in_time: true
  beam_width: 4
  decay_params:
    max_step: 1000000
    model_dim: 512
    warmup_steps: 8000
  decay_type: noam_decay_v2
  gradients_accum: 1
  label_smoothing: 0.1
  learning_rate: 2.0
  length_penalty: 0.6
  maximum_iterations: 50
  minimum_decoding_length: 0
  optimizer: LazyAdamOptimizer
  optimizer_params:
    beta1: 0.9
    beta2: 0.998
  weight_decay: 0.01
score:
  batch_size: 64
train:
  average_last_checkpoints: 8
  batch_size: 4096
  batch_type: tokens
  bucket_width: 8
  effective_batch_size: null
  keep_checkpoint_max: 8
  maximum_features_length: 256
  maximum_labels_length: 256
  num_threads: 8
  prefetch_buffer_size: 4
  sample_buffer_size: 100000
  save_checkpoints_secs: null
  save_checkpoints_steps: 5000
  save_summary_steps: 100
  train_steps: 1000000

2019-03-06 23:09:26.843446: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-03-06 23:09:27.382716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: Tesla K40c major: 3 minor: 5 memoryClockRate(GHz): 0.745
pciBusID: 0000:02:00.0
totalMemory: 11.17GiB freeMemory: 11.09GiB
2019-03-06 23:09:27.645287: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties: 
name: Tesla K40c major: 3 minor: 5 memoryClockRate(GHz): 0.745
pciBusID: 0000:04:00.0
totalMemory: 11.17GiB freeMemory: 11.09GiB
2019-03-06 23:09:27.910482: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 2 with properties: 
name: Tesla K40c major: 3 minor: 5 memoryClockRate(GHz): 0.745
pciBusID: 0000:83:00.0
totalMemory: 11.17GiB freeMemory: 11.09GiB
2019-03-06 23:09:28.184100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 3 with properties: 
name: Tesla K40c major: 3 minor: 5 memoryClockRate(GHz): 0.745
pciBusID: 0000:84:00.0
totalMemory: 11.17GiB freeMemory: 11.09GiB
2019-03-06 23:09:28.184971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1, 2, 3
2019-03-06 23:09:35.080283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-06 23:09:35.080335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1 2 3 
2019-03-06 23:09:35.080348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N Y N N 
2019-03-06 23:09:35.080356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   Y N N N 
2019-03-06 23:09:35.080364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 2:   N N N Y 
2019-03-06 23:09:35.080372: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 3:   N N Y N 
2019-03-06 23:09:35.081500: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 10747 MB memory) -> physical GPU (device: 0, name: Tesla K40c, pci bus id: 0000:02:00.0, compute capability: 3.5)
2019-03-06 23:09:35.082024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:1 with 10747 MB memory) -> physical GPU (device: 1, name: Tesla K40c, pci bus id: 0000:04:00.0, compute capability: 3.5)
2019-03-06 23:09:35.082351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:2 with 10747 MB memory) -> physical GPU (device: 2, name: Tesla K40c, pci bus id: 0000:83:00.0, compute capability: 3.5)
2019-03-06 23:09:35.082622: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:3 with 10747 MB memory) -> physical GPU (device: 3, name: Tesla K40c, pci bus id: 0000:84:00.0, compute capability: 3.5)
INFO:tensorflow:Using config: {'_model_dir': '/mnt/yardcephfs/mmyard/g_wxg_td_prc/chriscxyan/OpenNMT-TF/opennmt-tf/code/data/gpt2/run', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 5000, '_save_checkpoints_secs': None, '_session_config': gpu_options {
}
allow_soft_placement: true
graph_options {
  rewrite_options {
    layout_optimizer: OFF
  }
}
, '_keep_checkpoint_max': 8, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f079dd36cc0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 5000 or save_checkpoints_secs None.
INFO:tensorflow:Calling model_fn.
Traceback (most recent call last):
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1628, in _create_c_op
    c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Duplicate node name in graph: 'lm/w_embs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/yardcephfs/mmyard/g_wxg_td_prc/chriscxyan/OpenNMT-TF/opennmt-tf/code/../OpenNMT-tf-master/opennmt/bin/main.py", line 201, in <module>
    main()
  File "/mnt/yardcephfs/mmyard/g_wxg_td_prc/chriscxyan/OpenNMT-TF/opennmt-tf/code/../OpenNMT-tf-master/opennmt/bin/main.py", line 172, in main
    runner.train_and_evaluate(checkpoint_path=args.checkpoint_path)
  File "/mnt/yardcephfs/mmyard/g_wxg_td_prc/chriscxyan/OpenNMT-TF/opennmt-tf/OpenNMT-tf-master/opennmt/runner.py", line 295, in train_and_evaluate
    result = tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 471, in train_and_evaluate
    return executor.run()
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 610, in run
    return self.run_local()
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 711, in run_local
    saving_listeners=saving_listeners)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 354, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1207, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1237, in _train_model_default
    features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1195, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/mnt/yardcephfs/mmyard/g_wxg_td_prc/chriscxyan/OpenNMT-TF/opennmt-tf/OpenNMT-tf-master/opennmt/estimator.py", line 162, in _fn
    _loss_op, local_model, features_shards, labels_shards, params, mode)
  File "/mnt/yardcephfs/mmyard/g_wxg_td_prc/chriscxyan/OpenNMT-TF/opennmt-tf/OpenNMT-tf-master/opennmt/utils/parallel.py", line 151, in __call__
    outputs.append(funs[i](*args[i], **kwargs[i]))
  File "/mnt/yardcephfs/mmyard/g_wxg_td_prc/chriscxyan/OpenNMT-TF/opennmt-tf/OpenNMT-tf-master/opennmt/estimator.py", line 231, in _loss_op
    logits, _ = model(features, labels, params, mode)
  File "/mnt/yardcephfs/mmyard/g_wxg_td_prc/chriscxyan/OpenNMT-TF/opennmt-tf/OpenNMT-tf-master/opennmt/models/model.py", line 85, in __call__
    return self._call(features, labels, params, mode)
  File "/mnt/yardcephfs/mmyard/g_wxg_td_prc/chriscxyan/OpenNMT-TF/opennmt-tf/OpenNMT-tf-master/opennmt/models/language_model.py", line 56, in _call
    self.examples_inputter.build()
  File "/mnt/yardcephfs/mmyard/g_wxg_td_prc/chriscxyan/OpenNMT-TF/opennmt-tf/OpenNMT-tf-master/opennmt/inputters/text_inputter.py", line 414, in build
    name=compat.name_from_variable_scope("w_embs"))
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 183, in __call__
    return cls._variable_v1_call(*args, **kwargs)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 146, in _variable_v1_call
    aggregation=aggregation)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 125, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2444, in default_variable_creator
    expected_shape=expected_shape, import_scope=import_scope)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 187, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 1329, in __init__
    constraint=constraint)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 1443, in _init_from_args
    name=name)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 77, in variable_op_v2
    shared_name=shared_name)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 1357, in variable_v2
    shared_name=shared_name, name=name)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1792, in __init__
    control_input_ops)
  File "/data1/qspace/chriscxyan/anaconda2/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1631, in _create_c_op
    raise ValueError(str(e))
ValueError: Duplicate node name in graph: 'lm/w_embs'
```


Discussion:
Thanks for reporting.

Does it work on a single GPU?
Yep, multi-GPU is currently broken. Will push a fix shortly.

Resolving patch:
diff --git a/opennmt/models/language_model.py b/opennmt/models/language_model.py
--- a/opennmt/models/language_model.py
+++ b/opennmt/models/language_model.py
@@ -48,11 +48,7 @@ def auto_config(self, num_devices=1):
         }
     })
 
-  def _call(self, features, labels, params, mode):
-    training = mode == tf.estimator.ModeKeys.TRAIN
-    outputs, predictions = None, None
-
-    # Initialize input and output layers.
+  def _build(self):
     self.examples_inputter.build()
     vocab_size = self.examples_inputter.vocabulary_size
     output_layer = None
@@ -64,6 +60,10 @@ def _call(self, features, labels, params, mode):
           dtype=self.examples_inputter.dtype)
     self.decoder.initialize(vocab_size=vocab_size, output_layer=output_layer)
 
+  def _call(self, features, labels, params, mode):
+    training = mode == tf.estimator.ModeKeys.TRAIN
+    outputs, predictions = None, None
+
     ids, length = features["ids"], features["length"]
     if mode != tf.estimator.ModeKeys.PREDICT:
       # For training and evaluation, forward the full sequence.
diff --git a/opennmt/models/model.py b/opennmt/models/model.py
--- a/opennmt/models/model.py
+++ b/opennmt/models/model.py
@@ -9,6 +9,7 @@
 
 from opennmt import estimator
 from opennmt import inputters
+from opennmt.utils import compat
 from opennmt.utils.optim import optimize_loss
 
 
@@ -82,6 +83,8 @@ def __call__(self, features, labels, params, mode, config=None):  # pylint: disa
       the arguments of this function.
     """
     with tf.variable_scope(self.name, initializer=self._initializer(params)):
+      if not compat.reuse():
+        self._build()  # Always rebuild unless the scope is marked for reuse.
       return self._call(features, labels, params, mode)
 
   def _initializer(self, params):
@@ -99,6 +102,10 @@ def _initializer(self, params):
           minval=-param_init, maxval=param_init, dtype=self.dtype)
     return None
 
+  def _build(self):
+    """Builds stateful layers."""
+    return
+
   @abc.abstractmethod
   def _call(self, features, labels, params, mode):
     """Creates the graph.
diff --git a/opennmt/models/sequence_to_sequence.py b/opennmt/models/sequence_to_sequence.py
--- a/opennmt/models/sequence_to_sequence.py
+++ b/opennmt/models/sequence_to_sequence.py
@@ -135,6 +135,7 @@ def __init__(self,
     self.encoder = encoder
     self.decoder = decoder
     self.share_embeddings = share_embeddings
+    self.output_layer = None
 
   def auto_config(self, num_devices=1):
     config = super(SequenceToSequence, self).auto_config(num_devices=num_devices)
@@ -153,9 +154,19 @@ def auto_config(self, num_devices=1):
         }
     })
 
+  def _build(self):
+    self.examples_inputter.build()
+    if EmbeddingsSharingLevel.share_target_embeddings(self.share_embeddings):
+      self.output_layer = layers.Dense(
+          self.labels_inputter.vocabulary_size,
+          weight=self.labels_inputter.embedding,
+          transpose=True,
+          dtype=self.labels_inputter.vocabulary_size.dtype)
+      with tf.name_scope(tf.get_variable_scope().name + "/"):
+        self.output_layer.build([None, self.decoder.output_size])
+
   def _call(self, features, labels, params, mode):
     training = mode == tf.estimator.ModeKeys.TRAIN
-    self.examples_inputter.build()
 
     features_length = self.features_inputter.get_length(features)
     source_inputs = self.features_inputter.make_inputs(features, training=training)
@@ -167,16 +178,6 @@ def _call(self, features, labels, params, mode):
 
     target_vocab_size = self.labels_inputter.vocabulary_size
     target_dtype = self.labels_inputter.dtype
-    output_layer = None
-    if EmbeddingsSharingLevel.share_target_embeddings(self.share_embeddings):
-      output_layer = layers.Dense(
-          target_vocab_size,
-          weight=self.labels_inputter.embedding,
-          transpose=True,
-          dtype=target_dtype)
-      with tf.name_scope(tf.get_variable_scope().name + "/"):
-        output_layer.build([None, self.decoder.output_size])
-
     if labels is not None:
       target_inputs = self.labels_inputter.make_inputs(labels, training=training)
       with tf.variable_scope("decoder"):
@@ -195,7 +196,7 @@ def _call(self, features, labels, params, mode):
             initial_state=encoder_state,
             sampling_probability=sampling_probability,
             embedding=self.labels_inputter.embedding,
-            output_layer=output_layer,
+            output_layer=self.output_layer,
             mode=mode,
             memory=encoder_outputs,
             memory_sequence_length=encoder_sequence_length,
@@ -228,7 +229,7 @@ def _call(self, features, labels, params, mode):
               end_token,
               vocab_size=target_vocab_size,
               initial_state=encoder_state,
-              output_layer=output_layer,
+              output_layer=self.output_layer,
               maximum_iterations=maximum_iterations,
               minimum_length=minimum_length,
               mode=mode,
@@ -247,7 +248,7 @@ def _call(self, features, labels, params, mode):
                   end_token,
                   vocab_size=target_vocab_size,
                   initial_state=encoder_state,
-                  output_layer=output_layer,
+                  output_layer=self.output_layer,
                   beam_width=beam_width,
                   length_penalty=length_penalty,
                   maximum_iterations=maximum_iterations,


Test patch: