Repository: magenta/magenta

Issue:
DRUMS_RNN: when trying to run --eval on drums_rnn I get "ValueError: invalid every_n_iter=0."
Some very strange behavior here.

I am comparing lstm behavior here to that of [LSTMetallica](https://github.com/keunwoochoi/LSTMetallica). So using the same midi library of metallica drums, I went through the normal drums_rnn pipeline:

1. I created a tfrecord (notesequence) of the entire metallica midi library
2. I turned that notesequence into a training and validation split of 10 percent
3. I started a training on the training sequence on the training data
4. I started a validation (--eval) on the eval data and IMMEDIATELY get a valueerror:
```
Traceback (most recent call last):
  File ".\drums_rnn_train.py", line 113, in <module>
    console_entry_point()
  File ".\drums_rnn_train.py", line 109, in console_entry_point
    tf.app.run(main)
  File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\platform\app.py", line 124, in run
    _sys.exit(main(argv))
  File ".\drums_rnn_train.py", line 99, in main
    events_rnn_train.run_eval(build_graph_fn, train_dir, eval_dir, num_batches)
  File "C:\Program Files\Python35\lib\site-packages\magenta\models\shared\events_rnn_train.py", line 114, in run_eval
    EvalLoggingTensorHook(logging_dict, every_n_iter=num_batches),
  File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 209, in __init__
    raise ValueError("invalid every_n_iter=%s." % every_n_iter)
ValueError: invalid every_n_iter=0.

```

I've played around with the training and the evaluation percentages, and if I do 40%!!! VALIDATION data then it doesn't give that error any more, but that is way too much data for validation. (the tfrecord size for training data is 103 megs and the the eval data is 75 megs). 

The rest of the data sets i've tried this on has similar properties, which led me to believe that perhaps these other data sets were too small. But a weird quirk is that if I used a much smaller data set (training set size of about 20megs, and eval data of 10megs), it would still give the same error on using the validation data to run the eval, but it doesn't fail if i used the training data for evaluation. So somehow a 20 meg file of training data for a smaller data set works validation but it fails for this metallica data set if I don't give it at least 70 megs of evaluation data. Is there just a weird problem with "drums_rnn_create_dataset"

Any ideas?

Thanks in advance


Discussion:
I'm running into a very similar error in an experiment I'm doing with custom data on polyphony_rnn:
```
polyphony_rnn_train --run_dir=./polyphony_rnn/logdir/run1 --sequence_example_file=./polyphony_rnn/sequence_examples/eval_poly_tracks.tfrecord --hparams="batch_size=64,rnn_layer_sizes=[64,64]" --num_training_steps=20000 --eval
INFO:tensorflow:hparams = {'rnn_layer_sizes': [64, 64], 'learning_rate': 0.001, 'clip_norm': 5, 'batch_size': 64, 'dropout_keep_prob': 0.5}
INFO:tensorflow:Train dir: ./polyphony_rnn/logdir/run1/train
INFO:tensorflow:Eval dir: ./polyphony_rnn/logdir/run1/eval
INFO:tensorflow:Counting records in ./polyphony_rnn/sequence_examples/eval_poly_tracks.tfrecord.
INFO:tensorflow:Total records: 54
INFO:tensorflow:[<tf.Tensor 'ParseSingleSequenceExample/ParseSingleSequenceExample:0' shape=(?, 259) dtype=float32>, <tf.Tensor 'ParseSingleSequenceExample/ParseSingleSequenceExample:1' shape=(?,) dtype=int64>, <tf.Tensor 'strided_slice:0' shape=() dtype=int32>]
Traceback (most recent call last):
  File "/Users/ianconway/anaconda/envs/magenta/bin/polyphony_rnn_train", line 11, in <module>
    sys.exit(console_entry_point())
  File "/Users/ianconway/anaconda/envs/magenta/lib/python2.7/site-packages/magenta/models/polyphony_rnn/polyphony_rnn_train.py", line 113, in console_entry_point
    tf.app.run(main)
  File "/Users/ianconway/anaconda/envs/magenta/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/Users/ianconway/anaconda/envs/magenta/lib/python2.7/site-packages/magenta/models/polyphony_rnn/polyphony_rnn_train.py", line 103, in main
    events_rnn_train.run_eval(build_graph_fn, train_dir, eval_dir, num_batches)
  File "/Users/ianconway/anaconda/envs/magenta/lib/python2.7/site-packages/magenta/models/shared/events_rnn_train.py", line 114, in run_eval
    EvalLoggingTensorHook(logging_dict, every_n_iter=num_batches),
  File "/Users/ianconway/anaconda/envs/magenta/lib/python2.7/site-packages/tensorflow/python/training/basic_session_run_hooks.py", line 209, in __init__
    raise ValueError("invalid every_n_iter=%s." % every_n_iter)
ValueError: invalid every_n_iter=0.
```

At first I had issues because the script which prepares the tf_records expects the files be above/below a given minimum/maximum length. After I wrote a script to split my midi files into appropriately sized chunks, I was left with only about 65 files. Before on an experiment where I was using more examples, I did not get this error. Just a hunch, but I think it may have to do with the number of midi files being too small, even though you have enough data. Maybe consider splitting the files and see if that helps. If I figure out a solution on my end I will post it.

It's really strange. I encountered this error as well. However, when I change the eval radio to 40% there's no error and then I change it back to 10% and it works too. I am so confused what happened. 

I've been running into this same error when using melody_rnn. After some digging I found some relevant code in magenta/models/shared/events_rnn_train.py.

It seems that a flag 'num_eval_examples' is set to a default value of 0. Later on, num_batches is assigned to num_eval_examples (if set) otherwise it attempts to count the number of examples. 

`num_batches = (
        (FLAGS.num_eval_examples if FLAGS.num_eval_examples else
         magenta.common.count_records(sequence_example_file_paths)) //
        config.hparams.batch_size)`

I haven't been setting the num_eval_examples flag so for some reason I suppose the method of counting the eval examples doesn't work properly.

I was able to successfully run evaluation after I added this flag to the melody_rnn_train script:
--num_eval_examples=100

Resolving patch:
diff --git a/magenta/models/shared/events_rnn_train.py b/magenta/models/shared/events_rnn_train.py
--- a/magenta/models/shared/events_rnn_train.py
+++ b/magenta/models/shared/events_rnn_train.py
@@ -98,7 +98,13 @@ def run_eval(build_graph_fn, train_dir, eval_dir, num_batches,
     num_batches: The number of full batches to use for each evaluation step.
     timeout_secs: The number of seconds after which to stop waiting for a new
         checkpoint.
+  Raises:
+    ValueError: If `num_batches` is less than or equal to 0.
   """
+  if num_batches <= 0:
+    raise ValueError(
+        '`num_batches` must be greater than 0. Check that the batch size is '
+        'no larger than the number of records in the eval set.')
   with tf.Graph().as_default():
     build_graph_fn()
 


Test patch: