Repository: magenta/magenta

Issue:
issue training / generating with polyphony_rnn
Hello,

After (apparently successfully) learning an polyphony_rnn model, I cannot generate any sequence.
If I try to generate from the checkpoint, it returns the following:

> INFO:tensorflow:Checkpoint used: ./2017_05_04//run1/train/model.ckpt-3122
INFO:tensorflow:Restoring parameters from ./2017_05_04//run1/train/model.ckpt-3122
2017-05-04 18:18:18.663347: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key rnn/multi_rnn_cell/cell_2/basic_lstm_cell/weights not found in checkpoint
2017-05-04 18:18:18.666763: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key rnn/multi_rnn_cell/cell_2/basic_lstm_cell/biases not found in checkpoint

It is followed by a long traceback call message (that I'll provide if it helps).

Unsurprisingly I also cannot generate a bundle file from my checkpoint, for the same reason.

Any suggestion?



Discussion:
Can you include the commands you were using to train and generate? Also might as well include the traceback just to be sure.
Sure thing!
I've been using some shell scripts to execute those commands, hence the ${n}, I guess the actual values are not relevant.

Training command:
`python ${MAGENTA_PATH}/models/polyphony_rnn/polyphony_rnn_train.py --run_dir=${1}/run${3} --sequence_example_file=${1}/tfSequences/training_poly_tracks.tfrecord --hparams="{'batch_size':64,'rnn_layer_sizes':[64,64],'initial_learning_rate':0.0001}" --num_training_steps=${4}`

Generation command:
```
python ${MAGENTA_PATH}/models/polyphony_rnn/polyphony_rnn_generate.py  --run_dir=${1}/run${2} --output_dir=${1}/generated --num_outputs=${3} --num_steps=128 \
--primer_pitches=${4} \
--condition_on_primer=true \
--inject_primer_during_generation=false
```

Generation traceback:

> INFO:tensorflow:hparams = {'rnn_layer_sizes': [256, 256, 256], 'decay_rate': 0.95, 'dropout_keep_prob': 1.0, 'batch_size': 1, 'decay_steps': 1000, 'clip_norm': 5, 'initial_learning_rate': 0.001, 'skip_first_n_losses': 10}
> WARNING:tensorflow:<tensorflow.contrib.rnn.python.ops.core_rnn_cell_impl.BasicLSTMCell object at 0x7fddfb80e690>: Using a concatenated state is slower and will soon be deprecated.  Use state_is_tuple=True.
> WARNING:tensorflow:<tensorflow.contrib.rnn.python.ops.core_rnn_cell_impl.BasicLSTMCell object at 0x7fddfb80e5d0>: Using a concatenated state is slower and will soon be deprecated.  Use state_is_tuple=True.
> WARNING:tensorflow:<tensorflow.contrib.rnn.python.ops.core_rnn_cell_impl.BasicLSTMCell object at 0x7fde05c6dd50>: Using a concatenated state is slower and will soon be deprecated.  Use state_is_tuple=True.
> INFO:tensorflow:Checkpoint used: ./2017_05_04//run1/train/model.ckpt-3122
> INFO:tensorflow:Restoring parameters from ./2017_05_04//run1/train/model.ckpt-3122
> 2017-05-04 20:47:29.315896: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key rnn/multi_rnn_cell/cell_2/basic_lstm_cell/weights not found in checkpoint
> 2017-05-04 20:47:29.320142: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key rnn/multi_rnn_cell/cell_2/basic_lstm_cell/biases not found in checkpoint
> Traceback (most recent call last):
>   File "/home/pierre/HumtapGit/magenta/magenta//models/polyphony_rnn/polyphony_rnn_generate.py", line 265, in <module>
>     console_entry_point()
>   File "/home/pierre/HumtapGit/magenta/magenta//models/polyphony_rnn/polyphony_rnn_generate.py", line 261, in console_entry_point
>     tf.app.run(main)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
>     _sys.exit(main(_sys.argv[:1] + flags_passthrough))
>   File "/home/pierre/HumtapGit/magenta/magenta//models/polyphony_rnn/polyphony_rnn_generate.py", line 257, in main
>     run_with_flags(generator)
>   File "/home/pierre/HumtapGit/magenta/magenta//models/polyphony_rnn/polyphony_rnn_generate.py", line 227, in run_with_flags
>     generated_sequence = generator.generate(primer_sequence, generator_options)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/magenta/music/sequence_generator.py", line 203, in generate
>     self.initialize()
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/magenta/music/sequence_generator.py", line 149, in initialize
>     self._model.initialize_with_checkpoint(checkpoint_file)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/magenta/music/model.py", line 63, in initialize_with_checkpoint
>     saver.restore(self._session, checkpoint_file)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1545, in restore
>     {self.saver_def.filename_tensor_name: save_path})
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 786, in run
>     run_metadata_ptr)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 994, in _run
>     feed_dict_string, options, run_metadata)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1044, in _do_run
>     target_list, options, run_metadata)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1064, in _do_call
>     raise type(e)(node_def, op, message)
> tensorflow.python.framework.errors_impl.NotFoundError: Key rnn/multi_rnn_cell/cell_2/basic_lstm_cell/weights not found in checkpoint
> 	 [[Node: save/RestoreV2_7 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_7/tensor_names, save/RestoreV2_7/shape_and_slices)]]
> 
> Caused by op u'save/RestoreV2_7', defined at:
>   File "/home/pierre/HumtapGit/magenta/magenta//models/polyphony_rnn/polyphony_rnn_generate.py", line 265, in <module>
>     console_entry_point()
>   File "/home/pierre/HumtapGit/magenta/magenta//models/polyphony_rnn/polyphony_rnn_generate.py", line 261, in console_entry_point
>     tf.app.run(main)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
>     _sys.exit(main(_sys.argv[:1] + flags_passthrough))
>   File "/home/pierre/HumtapGit/magenta/magenta//models/polyphony_rnn/polyphony_rnn_generate.py", line 257, in main
>     run_with_flags(generator)
>   File "/home/pierre/HumtapGit/magenta/magenta//models/polyphony_rnn/polyphony_rnn_generate.py", line 227, in run_with_flags
>     generated_sequence = generator.generate(primer_sequence, generator_options)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/magenta/music/sequence_generator.py", line 203, in generate
>     self.initialize()
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/magenta/music/sequence_generator.py", line 149, in initialize
>     self._model.initialize_with_checkpoint(checkpoint_file)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/magenta/music/model.py", line 60, in initialize_with_checkpoint
>     saver = tf.train.Saver()
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1139, in __init__
>     self.build()
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1170, in build
>     restore_sequentially=self._restore_sequentially)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 691, in build
>     restore_sequentially, reshape)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
>     tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
>     [spec.tensor.dtype])[0])
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 669, in restore_v2
>     dtypes=dtypes, name=name)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
>     op_def=op_def)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
>     original_op=self._default_original_op, op_def=op_def)
>   File "/home/pierre/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1228, in __init__
>     self._traceback = _extract_stack()
> 
> NotFoundError (see above for traceback): Key rnn/multi_rnn_cell/cell_2/basic_lstm_cell/weights not found in checkpoint
> 	 [[Node: save/RestoreV2_7 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_7/tensor_names, save/RestoreV2_7/shape_and_slices)]]

I notice that at the beginning of the traceback of the generation part, it displays 'rnn_layer_sizes': [256, 256, 256] (while the training was actually done with size [64, 64]). Is it relevant?
Yes, that's the issue. You need to specify the same layer sizes via the hparams flag when you generate as when you trained.
I actually had tried that, using this generation command instead:

```
python ${MAGENTA_PATH}/models/polyphony_rnn/polyphony_rnn_generate.py  --run_dir=${1}/run${2} --output_dir=${1}/generated --num_outputs=${3} --num_steps=128 \
--hparams="{'batch_size':64,'rnn_layer_sizes':[64,64]}"\
--primer_pitches=${4} \
--condition_on_primer=true \
--inject_primer_during_generation=false
```
It yields exactly the same result. (Including the display of 'rnn_layer_sizes': [256, 256, 256])
The hparams parsing issue fixing documented in #592 is only applied to the training and also needs applied on the generation part.
I did it and I can now generate.
Ah, good catch. I'll take a look at that.

Resolving patch:
diff --git a/magenta/models/polyphony_rnn/polyphony_rnn_generate.py b/magenta/models/polyphony_rnn/polyphony_rnn_generate.py
--- a/magenta/models/polyphony_rnn/polyphony_rnn_generate.py
+++ b/magenta/models/polyphony_rnn/polyphony_rnn_generate.py
@@ -108,6 +108,11 @@
     'log', 'INFO',
     'The threshold for what messages will be logged DEBUG, INFO, WARN, ERROR, '
     'or FATAL.')
+tf.app.flags.DEFINE_string(
+    'hparams', '{}',
+    'String representation of a Python dictionary containing hyperparameter '
+    'to value mapping. This mapping is merged with the default '
+    'hyperparameters.')
 
 
 def get_checkpoint():
@@ -239,6 +244,7 @@ def main(unused_argv):
   tf.logging.set_verbosity(FLAGS.log)
 
   config = polyphony_model.default_configs[FLAGS.config]
+  config.hparams.parse(FLAGS.hparams)
 
   generator = polyphony_sequence_generator.PolyphonyRnnSequenceGenerator(
       model=polyphony_model.PolyphonyRnnModel(config),


Test patch: