Repository: OpenNMT/OpenNMT-tf

Issue:
InvalidArgumentError when using scheduled sampling
Hi,

I am trying the ListenAttendSpell model working on character level outputs.

My train_labels_file is of this kind (one space separated sentence per line):
```
<char> <char> <char> <space> <char> <char>
<char> <char> <char> <char> <char> <space>
.
.
.
I <space> a m <space> t a l l 
```
etc.

and I built the vocab file (length ~160) out of it using the onmt-build-vocab script.

When running the model I get an InvalidArgumentError and this is the generated stack:

```
Traceback (most recent call last):
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1323, in _do_call
    return fn(*args)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1302, in _run_fn
    status, run_metadata)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[0] = 319 is not in [0, 167)
         [[Node: seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/decoder_1/embedding_lookup = Gather[Tindices=DT_INT3
2, Tparams=DT_FLOAT, _class=["loc:@seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/decoder_1/convert_gradient_to_tens
or_cc661786"], validate_indices=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrain
ingHelperNextInputs/cond/decoder_1/convert_gradient_to_tensor_cc661786, seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/co
nd/GatherNd)]]
         [[Node: seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/Greater/Switch/_525 = _HostRecv[client_terminated=fa
lse, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_7992_seq2seq/pa
rallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/Greater/Switch", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0
/device:GPU:0"](^_cloopseq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/TrainingHelperNextInputs/GreaterEqual/_132)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/hltsrv0/rdessi/sw/anaconda3/bin/onmt-main", line 11, in <module>
    load_entry_point('OpenNMT-tf==1.5.0', 'console_scripts', 'onmt-main')()
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/OpenNMT_tf-1.5.0-py3.6.egg/opennmt/bin/main.py", line 133, in main
    runner.train_and_evaluate()
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/OpenNMT_tf-1.5.0-py3.6.egg/opennmt/runner.py", line 148, in train_and_evaluate
    tf.estimator.train_and_evaluate(self._estimator, train_spec, eval_spec)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 432, in train_and_evaluate
    executor.run_local()
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 611, in run_local
    hooks=train_hooks)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 302, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 783, in _train_model
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 521, in run
    run_metadata=run_metadata)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 892, in run
    run_metadata=run_metadata)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 967, in run
    raise six.reraise(*original_exc_info)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 952, in run
    return self._sess.run(*args, **kwargs)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1024, in run
    run_metadata=run_metadata)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 827, in run
    return self._sess.run(*args, **kwargs)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 889, in run
    run_metadata_ptr)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
    options, run_metadata)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[0] = 319 is not in [0, 167)
         [[Node: seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/decoder_1/embedding_lookup = Gather[Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/decoder_1/convert_gradient_to_tensor_cc661786"], validate_indices=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/decoder_1/convert_gradient_to_tensor_cc661786, seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/GatherNd)]]
         [[Node: seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/Greater/Switch/_525 = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_7992_seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/Greater/Switch", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](^_cloopseq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/TrainingHelperNextInputs/GreaterEqual/_132)]]

Caused by op 'seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/decoder_1/embedding_lookup', defined at:
  File "/hltsrv0/rdessi/sw/anaconda3/bin/onmt-main", line 11, in <module>
    load_entry_point('OpenNMT-tf==1.5.0', 'console_scripts', 'onmt-main')()
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/OpenNMT_tf-1.5.0-py3.6.egg/opennmt/bin/main.py", line 133, in main
    runner.train_and_evaluate()
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/OpenNMT_tf-1.5.0-py3.6.egg/opennmt/runner.py", line 148, in train_and_evaluate
    tf.estimator.train_and_evaluate(self._estimator, train_spec, eval_spec)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 432, in train_and_evaluate
    executor.run_local()
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 611, in run_local
    hooks=train_hooks)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 302, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 711, in _train_model
  features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 694, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/OpenNMT_tf-1.5.0-py3.6.egg/opennmt/models/model.py", line 103, in _model_fn
    _loss_op, features_shards, labels_shards, params, mode, config)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/OpenNMT_tf-1.5.0-py3.6.egg/opennmt/utils/parallel.py", line 148, in __call__
    outputs.append(funs[i](*args[i], **kwargs[i]))
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/OpenNMT_tf-1.5.0-py3.6.egg/opennmt/models/model.py", line 66, in _loss_op
    logits, _ = self._build(features, labels, params, mode, config=config)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/OpenNMT_tf-1.5.0-py3.6.egg/opennmt/models/sequence_to_sequence.py", line 144, in _build
    memory_sequence_length=encoder_sequence_length)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/OpenNMT_tf-1.5.0-py3.6.egg/opennmt/decoders/rnn_decoder.py", line 130, in decode
    outputs, state, length = tf.contrib.seq2seq.dynamic_decode(decoder)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/seq2seq/python/ops/decoder.py", line 286, in dynamic_decode
    swap_memory=swap_memory)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2816, in while_loop
    result = loop_context.BuildLoop(cond, body, loop_vars, shape_invariants)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2640, in BuildLoop
    pred, body, original_loop_vars, loop_vars, shape_invariants)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2590, in _BuildLoop
    body_result = body(*packed_vars_for_body)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/seq2seq/python/ops/decoder.py", line 234, in body
    decoder_finished) = decoder.step(time, inputs, state)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/seq2seq/python/ops/basic_decoder.py", line 147, in step
    sample_ids=sample_ids)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/seq2seq/python/ops/helper.py", line 342, in next_inputs
    all_finished, lambda: base_next_inputs, maybe_sample)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 316, in new_func
    return func(*args, **kwargs)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1864, in cond
    orig_res_f, res_f = context_f.BuildCondBranch(false_fn)
File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1725, in BuildCondBranch
    original_result = fn()
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/seq2seq/python/ops/helper.py", line 331, in maybe_sample
    sampled_next_inputs = self._embedding_fn(sample_ids_sampling)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/OpenNMT_tf-1.5.0-py3.6.egg/opennmt/models/sequence_to_sequence.py", line 103, in _target_embedding_fn
    return self.target_inputter.transform(ids, mode=mode)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/OpenNMT_tf-1.5.0-py3.6.egg/opennmt/inputters/text_inputter.py", line 392, in transform
    outputs = embedding_lookup(embeddings, inputs)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/OpenNMT_tf-1.5.0-py3.6.egg/opennmt/layers/common.py", line 30, in embedding_lookup
    return tf.nn.embedding_lookup(params, ids)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/embedding_ops.py", line 328, in embedding_lookup
    transform_fn=None)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/embedding_ops.py", line 150, in _embedding_lookup_and_transform
    result = _clip(_gather(params[0], ids, name=name), ids, max_norm)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/embedding_ops.py", line 54, in _gather
    return array_ops.gather(params, ids, name=name)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 2486, in gather
    params, indices, validate_indices=validate_indices, name=name)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1834, in gather
    validate_indices=validate_indices, name=name)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
    op_def=op_def)
  File "/hltsrv0/rdessi/sw/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1470, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): indices[0] = 319 is not in [0, 167)
         [[Node: seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/decoder_1/embedding_lookup = Gather[Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/decoder_1/convert_gradient_to_tensor_cc661786"], validate_indices=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/decoder_1/convert_gradient_to_tensor_cc661786, seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/GatherNd)]]
         [[Node: seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/Greater/Switch/_525 = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_7992_seq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/cond/Greater/Switch", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](^_cloopseq2seq/parallel_0/seq2seq/decoder/decoder_1/while/BasicDecoderStep/ScheduledEmbeddingTrainingHelperNextInputs/TrainingHelperNextInputs/GreaterEqual/_132)]]
```

From what I understood it seems like the WordEmbedder is querying the lookup table for a value that is outside the vocab length range: 

`InvalidArgumentError (see above for traceback): indices[0] = 319 is not in [0, 167)
` 

Is that correct?
Could it be fixed with <unk> token? I am not sure how the model deals with OOV internally. 

Thanks in advance!


Discussion:
Hello,

Tokens not present in the vocabulary are mapped to a single index, the last entry in the lookup table.

Is the error happening during the initial training, or after continuing from a checkpoint? Could you double check the number of lines in the vocabulary file you set in the training configuration?
It is happening during initial training. I just ran the command again with a different vocabulary of length 82 and the error I get (with the same traceback) is:

`InvalidArgumentError (see above for traceback): indices[0] = 445 is not in [0, 83)
`
Could you share the complete training configuration and the command line you used? Thanks.
I ran:
`onmt-main train_and_eval --model_type ListenAttendSpell --config ../exp_code/config.yml`

```
model_dir: LAS_default

data:

  train_features_file: /hltsrv3/rdessi/train.tfrecords 
  train_labels_file: /hltsrv3/rdessi/train.de 

  eval_features_file: /hltsrv3/rdessi/dev.tfrecords 
  eval_labels_file: /hltsrv3/rdessi/dev.de 


  target_words_vocabulary: /hltsrv3/rdessi/vocab.de                                                                                                                                      


params:

  optimizer: GradientDescentOptimizer  

  learning_rate: 0.2 


  param_init: 0.1 

  clip_gradients: 1.0 


  average_loss_in_time: true

  decay_type: exponential_decay

  decay_rate: 0.98

  decay_steps: 2674

  decay_step_duration: 1

  start_decay_steps: 40106

  minimum_learning_rate: 0.00001

  scheduled_sampling_type: constant

  scheduled_sampling_read_probability: 0.9

  scheduled_sampling_k: 0

  label_smoothing: 0.1

  beam_width: 5

  length_penalty: 0.2

  maximum_iterations: 200

  replace_unknown_target: false


train:
  batch_size: 64


  batch_type: examples

  save_checkpoints_steps: 2674

  keep_checkpoint_max: 8

  save_summary_steps: 100

  train_steps: 1000000                    

  single_pass: false

  bucket_width: 1

  num_threads: 4

  sample_buffer_size: 500000

  prefetch_buffer_size: null
  average_last_checkpoints: 8
```
Thank you. I'm reproducing the issue that is related to the scheduled sampling. If you want to continue your experiments, you can disable scheduled sampling by setting:

```yaml
params:
  scheduled_sampling_read_probability: 1
```
It seems to be working fine now. Thanks!
In the paper a 10% probability of sampling form previous outputs rather than from the ground truth is used, how could I replicate that?
Your settings were correct but scheduled sampling appears to be broken. I'm pushing a fix.

Resolving patch:
diff --git a/opennmt/decoders/rnn_decoder.py b/opennmt/decoders/rnn_decoder.py
--- a/opennmt/decoders/rnn_decoder.py
+++ b/opennmt/decoders/rnn_decoder.py
@@ -104,8 +104,10 @@ def decode(self,
           sequence_length,
           embedding,
           sampling_probability)
+      fused_projection = False
     else:
       helper = tf.contrib.seq2seq.TrainingHelper(inputs, sequence_length)
+      fused_projection = True  # With TrainingHelper, project all timesteps at once.
 
     cell, initial_state = self._build_cell(
         mode,
@@ -118,9 +120,6 @@ def decode(self,
     if output_layer is None:
       output_layer = build_output_layer(self.num_units, vocab_size, dtype=inputs.dtype)
 
-    # With TrainingHelper, project all timesteps at once.
-    fused_projection = isinstance(helper, tf.contrib.seq2seq.TrainingHelper)
-
     decoder = tf.contrib.seq2seq.BasicDecoder(
         cell,
         helper,


Test patch: