Repository: OpenNMT/OpenNMT-tf

Issue:
Error when running inference on the multi-source Transformer
```
2018-11-28 10:58:38.020515: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:344] Starting optimization for grappler item: tf_graph
INFO:tensorflow:Running local_init_op.
2018-11-28 10:58:38.257302: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:344] Starting optimization for grappler item: tf_graph
INFO:tensorflow:Done running local_init_op.
2018-11-28 10:58:38.349261: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:344] Starting optimization for grappler item: tf_graph
2018-11-28 10:58:38.580481: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:344] Starting optimization for grappler item: tf_graph
Traceback (most recent call last):
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1292, in _do_call
    return fn(*args)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1277, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1365, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 1818624 values, but the requested shape has 3637248
	 [[{{node transformer/decoder/while/Reshape_70}} = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](transformer/decoder/while/layer_5/multi_head_1/cond/Merge_1, transformer/decoder/while/Reshape_70/shape)]]
	 [[{{node transformer/Cast/_1585}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1967_transformer/Cast", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "opennmt/bin/main.py", line 192, in <module>
    main()
  File "opennmt/bin/main.py", line 175, in main
    log_time=args.log_prediction_time)
  File "/home/ari/tf/Onmt-14/OpenNMT-tf/opennmt/runner.py", line 329, in infer
    hooks=infer_hooks):
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 568, in predict
    preds_evaluated = mon_sess.run(predictions)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1148, in run
    run_metadata=run_metadata)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1239, in run
    raise six.reraise(*original_exc_info)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1224, in run
    return self._sess.run(*args, **kwargs)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1296, in run
    run_metadata=run_metadata)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1076, in run
    return self._sess.run(*args, **kwargs)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 887, in run
    run_metadata_ptr)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1110, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1286, in _do_run
    run_metadata)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1306, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 1818624 values, but the requested shape has 3637248
	 [[node transformer/decoder/while/Reshape_70 (defined at /home/ari/tf/Onmt-14/OpenNMT-tf/opennmt/utils/beam_search.py:88)  = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](transformer/decoder/while/layer_5/multi_head_1/cond/Merge_1, transformer/decoder/while/Reshape_70/shape)]]
	 [[{{node transformer/Cast/_1585}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1967_transformer/Cast", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op 'transformer/decoder/while/Reshape_70', defined at:
  File "opennmt/bin/main.py", line 192, in <module>
    main()
  File "opennmt/bin/main.py", line 175, in main
    log_time=args.log_prediction_time)
  File "/home/ari/tf/Onmt-14/OpenNMT-tf/opennmt/runner.py", line 329, in infer
    hooks=infer_hooks):
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 550, in predict
    features, None, model_fn_lib.ModeKeys.PREDICT, self.config)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1168, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/home/ari/tf/Onmt-14/OpenNMT-tf/opennmt/models/model.py", line 152, in _model_fn
    _, predictions = self._build(features, labels, params, mode, config=config)
  File "/home/ari/tf/Onmt-14/OpenNMT-tf/opennmt/models/sequence_to_sequence.py", line 277, in _build
    return_alignment_history=True))
  File "/home/ari/tf/Onmt-14/OpenNMT-tf/opennmt/decoders/decoder.py", line 378, in dynamic_decode_and_search
    min_decode_length=minimum_length)
  File "/home/ari/tf/Onmt-14/OpenNMT-tf/opennmt/utils/beam_search.py", line 613, in beam_search
    back_prop=False)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3274, in while_loop
    return_same_structure)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2994, in BuildLoop
    pred, body, original_loop_vars, loop_vars, shape_invariants)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2929, in _BuildLoop
    body_result = body(*packed_vars_for_body)
  File "/home/ari/tf/Onmt-14/OpenNMT-tf/opennmt/utils/beam_search.py", line 539, in inner_loop
    i, alive_seq, alive_log_probs, states)
  File "/home/ari/tf/Onmt-14/OpenNMT-tf/opennmt/utils/beam_search.py", line 428, in grow_topk
    lambda t: _unmerge_beam_dim(t, batch_size, beam_size), flat_states)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/nest.py", line 347, in map_structure
    structure[0], [func(*x) for x in entries])
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/nest.py", line 347, in <listcomp>
    structure[0], [func(*x) for x in entries])
  File "/home/ari/tf/Onmt-14/OpenNMT-tf/opennmt/utils/beam_search.py", line 428, in <lambda>
    lambda t: _unmerge_beam_dim(t, batch_size, beam_size), flat_states)
  File "/home/ari/tf/Onmt-14/OpenNMT-tf/opennmt/utils/beam_search.py", line 88, in _unmerge_beam_dim
    return tf.reshape(tensor, new_shape)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 6296, in reshape
    "Reshape", tensor=tensor, shape=shape, name=name)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/home/ari/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 1818624 values, but the requested shape has 3637248
	 [[node transformer/decoder/while/Reshape_70 (defined at /home/ari/tf/Onmt-14/OpenNMT-tf/opennmt/utils/beam_search.py:88)  = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](transformer/decoder/while/layer_5/multi_head_1/cond/Merge_1, transformer/decoder/while/Reshape_70/shape)]]
	 [[{{node transformer/Cast/_1585}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1967_transformer/Cast", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
```

The model trains just fine (and it is a copy from example)

_Originally posted by @Dagamies in https://github.com/OpenNMT/OpenNMT-tf/issues/275#issuecomment-442370984_


Discussion:
@Dagamies, can you share your run configuration?
Model is from example, other parameters are:
```
model_dir: /data/20/tf/multi_tf

data:
  train_features_file: 
    - /data/20/tf/a_train.words.txt
    - /data/20/tf/a_train.feats.txt
  train_labels_file: /data/19/tf/a_train_r.words.txt
  eval_features_file: 
    - /data/20/tf/a_validate.words.txt
    - /data/20/tf/a_validate.feats.txt
  eval_labels_file: /data/20/tf/a_validate_r.words.txt
  source_vocabulary_1: /data/20/tf/a_train.words-vocab.txt
  source_vocabulary_2: /data/20/tf/a_train.feats-vocab.txt
  target_vocabulary: /data/20/tf/a_train_r.words-vocab.txt


params:
  optimizer: AdamOptimizer
  learning_rate: 4.0  
  decay_type: noam_decay
  decay_rate: 512  
  decay_steps: 1000  


  decay_step_duration: 8  

  average_loss_in_time: true
  label_smoothing: 0.01  # was 0.1

  beam_width: 16
  # length_penalty: 0.6
  clip_gradients: 8.0

  optimizer_params: 
    beta1: 0.9
    beta2: 0.998


train:
  batch_size: 32
  batch_type: tokens
  bucket_width: 1

  maximum_features_length: 351
  maximum_labels_length: 767

  save_checkpoints_steps: 20000
  keep_checkpoint_max: 8
  save_summary_steps: 100
  train_steps: 10000000
  clip_gradients: 8.0
  # Consider setting this to -1 to match the number of training examples.
  sample_buffer_size: -1

eval:
  batch_size: 32
  eval_delay: 3600  

infer:
  batch_size: 1
  bucket_width: 0
```

Resolving patch:
diff --git a/opennmt/models/sequence_to_sequence.py b/opennmt/models/sequence_to_sequence.py
--- a/opennmt/models/sequence_to_sequence.py
+++ b/opennmt/models/sequence_to_sequence.py
@@ -236,7 +236,7 @@ def _build(self, features, labels, params, mode, config=None):
 
     if mode != tf.estimator.ModeKeys.TRAIN:
       with tf.variable_scope("decoder", reuse=labels is not None):
-        batch_size = tf.shape(encoder_sequence_length)[0]
+        batch_size = tf.shape(tf.contrib.framework.nest.flatten(encoder_outputs)[0])[0]
         beam_width = params.get("beam_width", 1)
         maximum_iterations = params.get("maximum_iterations", 250)
         minimum_length = params.get("minimum_decoding_length", 0)


Test patch: