Repository: OpenNMT/OpenNMT-tf

Issue:
Failed to restore checkpoint during evaluation when using DenseBridge
As it seems there still is an issue, after writing 1st checkpoint OpenNMT fails:
```
INFO:tensorflow:Starting evaluation at 2019-03-04T18:24:11Z
INFO:tensorflow:Graph was finalized.
2019-03-04 20:24:11.831808: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-03-04 20:24:11.831888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-04 20:24:11.831898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0
2019-03-04 20:24:11.831906: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N
2019-03-04 20:24:11.832092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15190 MB memory) -> physical GPU (device: 0, name: Tesla P100-SXM2-16GB, pci bus id: 0002:01:00.0, compute capability: 6.0)
WARNING:tensorflow:From /home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from /projects/aigine/35/tf/m-1088-3-5-5-s/model.ckpt-4000
2019-03-04 20:24:12.044384: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key seq2seq/decoder/dense/bias_2 not found in checkpoint

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.

Traceback (most recent call last):
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Key seq2seq/decoder/dense/bias_2 not found in checkpoint
         [[{{node save/RestoreV2}}]]
         [[{{node save/RestoreV2}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1276, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key seq2seq/decoder/dense/bias_2 not found in checkpoint
         [[node save/RestoreV2 (defined at /home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py:1537) ]]
         [[node save/RestoreV2 (defined at /home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py:1537) ]]

Caused by op 'save/RestoreV2', defined at:
  File "opennmt/bin/main.py", line 201, in <module>
    main()
  File "opennmt/bin/main.py", line 172, in main
    runner.train_and_evaluate(checkpoint_path=args.checkpoint_path)
  File "/home/ari/tf/onmt-120/OpenNMT-tf/opennmt/runner.py", line 293, in train_and_evaluate
    result = tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 471, in train_and_evaluate
    return executor.run()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 611, in run
    return self.run_local()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 712, in run_local
    saving_listeners=saving_listeners)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 358, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1124, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model_default
    saving_listeners)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1407, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 676, in run
    run_metadata=run_metadata)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1171, in run
    run_metadata=run_metadata)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1255, in run
    return self._sess.run(*args, **kwargs)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1335, in run
    run_metadata=run_metadata))
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/basic_session_run_hooks.py", line 582, in after_run
    if self._save(run_context.session, global_step):
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/basic_session_run_hooks.py", line 607, in _save
    if l.after_save(session, step):
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 517, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 537, in _evaluate
    self._evaluator.evaluate_and_export())
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 913, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 469, in evaluate
    name=name)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 511, in _actual_eval
    return _evaluate()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 500, in _evaluate
    output_dir=self.eval_dir(name))
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1537, in _evaluate_run
    config=self._session_config)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/evaluation.py", line 271, in _evaluate_once
    session_creator=session_creator, hooks=hooks) as session:
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 934, in __init__
    stop_grace_period_secs=stop_grace_period_secs)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 648, in __init__
    self._sess = _RecoverableSession(self._coordinated_creator)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1122, in __init__
    _WrappedSession.__init__(self, self._create_session())
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1127, in _create_session
    return self._sess_creator.create_session()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 805, in create_session
    self.tf_sess = self._session_creator.create_session()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 562, in create_session
    self._scaffold.finalize()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 217, in finalize
    self._saver = training_saver._get_saver_or_default()  # pylint: disable=protected-access
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 604, in _get_saver_or_default
    saver = Saver(sharded=True, allow_empty=True)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 832, in __init__
    self.build()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 844, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 881, in _build
    build_save=build_save, build_restore=build_restore)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 507, in _build_internal
    restore_sequentially, reshape)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 385, in _AddShardedRestoreOps
    name="restore_shard"))
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps
    restore_sequentially)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2
    name=name)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Key seq2seq/decoder/dense/bias_2 not found in checkpoint
         [[node save/RestoreV2 (defined at /home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py:1537) ]]
         [[node save/RestoreV2 (defined at /home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py:1537) ]]


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1286, in restore
    names_to_keys = object_graph_key_mapping(save_path)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1591, in object_graph_key_mapping
    checkpointable.OBJECT_GRAPH_PROTO_KEY)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 370, in get_tensor
    status)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "opennmt/bin/main.py", line 201, in <module>
    main()
  File "opennmt/bin/main.py", line 172, in main
    runner.train_and_evaluate(checkpoint_path=args.checkpoint_path)
  File "/home/ari/tf/onmt-120/OpenNMT-tf/opennmt/runner.py", line 293, in train_and_evaluate
    result = tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 471, in train_and_evaluate
    return executor.run()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 611, in run
    return self.run_local()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 712, in run_local
    saving_listeners=saving_listeners)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 358, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1124, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model_default
    saving_listeners)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1407, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 676, in run
    run_metadata=run_metadata)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1171, in run
    run_metadata=run_metadata)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1270, in run
    raise six.reraise(*original_exc_info)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1255, in run
    return self._sess.run(*args, **kwargs)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1335, in run
    run_metadata=run_metadata))
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/basic_session_run_hooks.py", line 582, in after_run
    if self._save(run_context.session, global_step):
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/basic_session_run_hooks.py", line 607, in _save
    if l.after_save(session, step):
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 517, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 537, in _evaluate
    self._evaluator.evaluate_and_export())
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 913, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 469, in evaluate
    name=name)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 511, in _actual_eval
    return _evaluate()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 500, in _evaluate
    output_dir=self.eval_dir(name))
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1537, in _evaluate_run
    config=self._session_config)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/evaluation.py", line 271, in _evaluate_once
    session_creator=session_creator, hooks=hooks) as session:
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 934, in __init__
    stop_grace_period_secs=stop_grace_period_secs)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 648, in __init__
    self._sess = _RecoverableSession(self._coordinated_creator)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1122, in __init__
    _WrappedSession.__init__(self, self._create_session())
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1127, in _create_session
    return self._sess_creator.create_session()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 805, in create_session
    self.tf_sess = self._session_creator.create_session()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 571, in create_session
    init_fn=self._scaffold.init_fn)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/session_manager.py", line 281, in prepare_session
    config=config)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/session_manager.py", line 195, in _restore_checkpoint
    saver.restore(sess, checkpoint_filename_with_path)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1292, in restore
    err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key seq2seq/decoder/dense/bias_2 not found in checkpoint
         [[node save/RestoreV2 (defined at /home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py:1537) ]]
         [[node save/RestoreV2 (defined at /home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py:1537) ]]

Caused by op 'save/RestoreV2', defined at:
  File "opennmt/bin/main.py", line 201, in <module>
    main()
  File "opennmt/bin/main.py", line 172, in main
    runner.train_and_evaluate(checkpoint_path=args.checkpoint_path)
  File "/home/ari/tf/onmt-120/OpenNMT-tf/opennmt/runner.py", line 293, in train_and_evaluate
    result = tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 471, in train_and_evaluate
    return executor.run()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 611, in run
    return self.run_local()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 712, in run_local
    saving_listeners=saving_listeners)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 358, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1124, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model_default
    saving_listeners)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1407, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 676, in run
    run_metadata=run_metadata)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1171, in run
    run_metadata=run_metadata)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1255, in run
    return self._sess.run(*args, **kwargs)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1335, in run
    run_metadata=run_metadata))
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/basic_session_run_hooks.py", line 582, in after_run
    if self._save(run_context.session, global_step):
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/basic_session_run_hooks.py", line 607, in _save
    if l.after_save(session, step):
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 517, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 537, in _evaluate
    self._evaluator.evaluate_and_export())
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 913, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 469, in evaluate
    name=name)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 511, in _actual_eval
    return _evaluate()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 500, in _evaluate
    output_dir=self.eval_dir(name))
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1537, in _evaluate_run
    config=self._session_config)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/evaluation.py", line 271, in _evaluate_once
    session_creator=session_creator, hooks=hooks) as session:
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 934, in __init__
    stop_grace_period_secs=stop_grace_period_secs)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 648, in __init__
    self._sess = _RecoverableSession(self._coordinated_creator)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1122, in __init__
    _WrappedSession.__init__(self, self._create_session())
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1127, in _create_session
    return self._sess_creator.create_session()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 805, in create_session
    self.tf_sess = self._session_creator.create_session()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 562, in create_session
    self._scaffold.finalize()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 217, in finalize
    self._saver = training_saver._get_saver_or_default()  # pylint: disable=protected-access
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 604, in _get_saver_or_default
    saver = Saver(sharded=True, allow_empty=True)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 832, in __init__
    self.build()
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 844, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 881, in _build
    build_save=build_save, build_restore=build_restore)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 507, in _build_internal
    restore_sequentially, reshape)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 385, in _AddShardedRestoreOps
    name="restore_shard"))
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps
    restore_sequentially)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2
    name=name)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key seq2seq/decoder/dense/bias_2 not found in checkpoint
         [[node save/RestoreV2 (defined at /home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py:1537) ]]
         [[node save/RestoreV2 (defined at /home/ari/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py:1537) ]]


```

_Originally posted by @Dagamies in https://github.com/OpenNMT/OpenNMT-tf/issues/359#issuecomment-469378670_


Discussion:

Resolving patch:
diff --git a/opennmt/layers/bridge.py b/opennmt/layers/bridge.py
--- a/opennmt/layers/bridge.py
+++ b/opennmt/layers/bridge.py
@@ -49,8 +49,11 @@ def __call__(self, encoder_state, decoder_zero_state):  # pylint: disable=argume
       The decoder initial state.
     """
     inputs = [encoder_state, decoder_zero_state]
-    # Always build for backward compatibility.
-    self.build(compat.nest.map_structure(lambda x: x.shape, inputs))
+    if compat.is_tf2():
+      return super(Bridge, self).__call__(inputs)
+    # Build by default for backward compatibility.
+    if not compat.reuse():
+      self.build(compat.nest.map_structure(lambda x: x.shape, inputs))
     return self.call(inputs)
 
   @abc.abstractmethod
diff --git a/opennmt/layers/position.py b/opennmt/layers/position.py
--- a/opennmt/layers/position.py
+++ b/opennmt/layers/position.py
@@ -63,9 +63,13 @@ def __call__(self, inputs, sequence_length=None, position=None):  # pylint: disa
       A ``tf.Tensor`` of shape :math:`[B, T, D]` where :math:`D` depends on the
       :attr:`reducer`.
     """
-    # Always build for backward compatibility.
+    if compat.is_tf2():
+      return super(PositionEncoder, self).__call__(
+          inputs, sequence_length=sequence_length, position=position)
     self._dtype = inputs.dtype
-    self.build(inputs.shape)
+    # Build by default for backward compatibility.
+    if not compat.reuse():
+      self.build(inputs.shape)
     return self.call(
         inputs, sequence_length=sequence_length, position=position)
 
diff --git a/opennmt/utils/compat.py b/opennmt/utils/compat.py
--- a/opennmt/utils/compat.py
+++ b/opennmt/utils/compat.py
@@ -53,6 +53,10 @@ def name_from_variable_scope(name=""):
     compat_name = "%s/%s" % (var_scope, compat_name)
   return compat_name
 
+def reuse():
+  """Returns ``True`` if the current variable scope is marked for reuse."""
+  return tf_compat(v1="get_variable_scope")().reuse
+
 def _string_to_tf_symbol(symbol):
   modules = symbol.split(".")
   namespace = tf


Test patch: