Repository: OpenNMT/OpenNMT-tf

Issue:
DataLossError when infer with the model from checkpoints averaging
Hi,

tensorflow-gpu 1.8.0
opennmt-tf 1.5.0

Model is trained with the same parameter of [wmt_ende.yml](https://github.com/OpenNMT/OpenNMT-tf/blob/master/scripts/wmt/config/wmt_ende.yml) with 4 x 1080Ti

After the checkpoints averaging (max_count=5), 
```bash
onmt-average-checkpoints --max_count=5 --model_dir=model/ --output_dir=model/avg/
```
do the inference in averaged model, but it return an error,
```bash
onmt-main infer --model_type Transformer --config train_enfr.yml --features_file newstest2014-fren-src.en.sp --predictions_file newstest2014-fren-src.en.sp.pred --checkpoint_path /model/avg/
```

```bash
DataLossError (see above for traceback): Invalid size in bundle entry: key transformer/decoder/LayerNorm/beta; stored size 4096; expected size 2048
         [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_INT64, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]
         [[Node: save/RestoreV2/_301 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_306_save/RestoreV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
``` 
But there is **no error** if I do inference with each single checkpoint, 
```bash
onmt-main infer --model_type Transformer --config train_enfr.yml --features_file newstest2014-fren-src.en.sp --predictions_file newstest2014-fren-src.en.sp.pred --checkpoint_path /model/model.ckpt-345921
```

I wonder if there something wrong when I average the checkpoints ?

Thanks !



Discussion:
Hello,

I did not succeed in reproducing the issue. Are you running these commands on the same server?
Thanks for your reply,

I run all the commands on the same server, I trained the model with the same parameter but on en-fr data of wmt14

The strange thing is that all single checkpoints work but the average one doesn't

Here is the screen shot of my model directory,

<img width="505" alt="screen shot 2018-07-02 at 19 49 21" src="https://user-images.githubusercontent.com/10563679/42178544-1cf16fde-7e31-11e8-9ade-cac9d75eeb80.png">

Did you try re-running the `average-checkpoints` script? Seems like it saved the wrong `dtype` for a variable.

If you can, I suggest to try the code on the master branch. There was some changes on checkpoint management that could help in your case.
Yes, I re-running and get the same error.

I have installed the lastest code by ***python setup.py install***

I re-running the code with the same data in [this script](https://github.com/OpenNMT/OpenNMT-tf/blob/master/scripts/wmt/README.md), I download the data in lazy run part and train then eval the model with the provided bash script. But finally get the same error when inference with the average checkpoint,
```bash
WARNING:tensorflow:You provided a model configuration but a checkpoint already exists. The model configuration must define the same model as the one used for the initial training. However, you can change non structural values like dropout.
2018-07-03 14:37:57.691948: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-07-03 14:38:00.441040: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:05:00.0
totalMemory: 10.92GiB freeMemory: 10.76GiB
2018-07-03 14:38:00.601709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 1 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:06:00.0
totalMemory: 10.92GiB freeMemory: 10.76GiB
2018-07-03 14:38:00.778173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 2 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:09:00.0
totalMemory: 10.92GiB freeMemory: 10.76GiB
2018-07-03 14:38:00.953637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 3 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:0a:00.0
totalMemory: 10.92GiB freeMemory: 10.76GiB
2018-07-03 14:38:00.958714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0, 1, 2, 3
2018-07-03 14:38:01.741761: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-03 14:38:01.741793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0 1 2 3
2018-07-03 14:38:01.741800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N Y Y Y
2018-07-03 14:38:01.741804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 1:   Y N Y Y
2018-07-03 14:38:01.741809: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 2:   Y Y N Y
2018-07-03 14:38:01.741813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 3:   Y Y Y N
2018-07-03 14:38:01.742552: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10412 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
2018-07-03 14:38:01.843134: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10413 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:06:00.0, compute capability: 6.1)
2018-07-03 14:38:01.942467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10413 MB memory) -> physical GPU (device: 2, name: GeForce GTX 1080 Ti, pci bus id: 0000:09:00.0, compute capability: 6.1)
2018-07-03 14:38:02.041326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10413 MB memory) -> physical GPU (device: 3, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
2018-07-03 14:38:02.142136: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0, 1, 2, 3
2018-07-03 14:38:02.142277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-03 14:38:02.142287: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0 1 2 3
2018-07-03 14:38:02.142294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N Y Y Y
2018-07-03 14:38:02.142300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 1:   Y N Y Y
2018-07-03 14:38:02.142304: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 2:   Y Y N Y
2018-07-03 14:38:02.142310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 3:   Y Y Y N
2018-07-03 14:38:02.142658: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/device:GPU:0 with 10412 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
2018-07-03 14:38:02.142747: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/device:GPU:1 with 10413 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:06:00.0, compute capability: 6.1)
2018-07-03 14:38:02.142848: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/device:GPU:2 with 10413 MB memory) -> physical GPU (device: 2, name: GeForce GTX 1080 Ti, pci bus id: 0000:09:00.0, compute capability: 6.1)
2018-07-03 14:38:02.142959: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/device:GPU:3 with 10413 MB memory) -> physical GPU (device: 3, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Using config: {'_model_dir': 'wmt_ende_transformer_4gpu_lr2_ws8000_dur2_0.998', '_tf_random_seed': None, '_save_summary_steps': 50, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': gpu_options {
}
allow_soft_placement: true
, '_keep_checkpoint_max': 10, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 50, '_train_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f1c8befe0b8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Graph was finalized.
2018-07-03 14:38:06.116385: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0, 1, 2, 3
2018-07-03 14:38:06.116505: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-03 14:38:06.116515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0 1 2 3
2018-07-03 14:38:06.116524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N Y Y Y
2018-07-03 14:38:06.116531: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 1:   Y N Y Y
2018-07-03 14:38:06.116539: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 2:   Y Y N Y
2018-07-03 14:38:06.116543: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 3:   Y Y Y N
2018-07-03 14:38:06.116864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10412 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
2018-07-03 14:38:06.116954: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10413 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:06:00.0, compute capability: 6.1)
2018-07-03 14:38:06.117021: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10413 MB memory) -> physical GPU (device: 2, name: GeForce GTX 1080 Ti, pci bus id: 0000:09:00.0, compute capability: 6.1)
2018-07-03 14:38:06.117084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10413 MB memory) -> physical GPU (device: 3, name: GeForce GTX 1080 Ti, pci bus id: 0000:0a:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from wmt_ende_transformer_4gpu_lr2_ws8000_dur2_0.998/avg/model.ckpt-8934
2018-07-03 14:38:06.270786: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Data loss: Invalid size in bundle entry: key transformer/decoder/LayerNorm/beta; stored size 4096; expected size 2048

Traceback (most recent call last):
  File "/home/XXXX/anaconda3/envs/dlnlp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/home/XXXX/anaconda3/envs/dlnlp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/XXXX/anaconda3/envs/dlnlp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.DataLossError: Invalid size in bundle entry: key transformer/decoder/LayerNorm/beta; stored size 4096; expected size 2048
         [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_INT64, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]
         [[Node: save/RestoreV2/_301 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_306_save/RestoreV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
```


Thank you for the details. I'm running out of ideas at the moment but can you also share the Numpy version that is installed and the output of:

```bash
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
```

just to make sure.
Thanks for your help,

My numpy version is **1.14.0**

output of the command is
```bash
/home/XXXX/anaconda3/envs/dlnlp/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
v1.8.0-0-g93bc2e2072 1.8.0
```
@silverguo The same problem, have you solved it?

Resolving patch:
diff --git a/opennmt/utils/checkpoint.py b/opennmt/utils/checkpoint.py
--- a/opennmt/utils/checkpoint.py
+++ b/opennmt/utils/checkpoint.py
@@ -191,7 +191,7 @@ def average_checkpoints(model_dir, output_dir, max_count=8, session_config=None)
   avg_values = {}
   for name, shape in var_list:
     if not name.startswith("global_step"):
-      avg_values[name] = np.zeros(shape)
+      avg_values[name] = np.zeros(shape, dtype=np.float32)
 
   for checkpoint_path in checkpoints_path:
     tf.logging.info("Loading checkpoint %s" % checkpoint_path)


Test patch: