Repository: magenta/magenta

Issue:
Odd behaviour with 'skip_first_n_losses' not equal to zero.
When I run the following, things work:

bazel-bin/magenta/models/melody_rnn/melody_rnn_train \
--config=basic_rnn \
--run_dir=/homes/sturm/magenta_session \
--sequence_example_file=/homes/sturm/datasets/basicrnnseq/training_melodies.tfrecord
\
--hparams="{'batch_size':100,'rnn_layer_sizes':[512,512,512],'decay_rate':0.97,
'initial_learning_rate':0.03,'decay_steps':1,'dropout_keep_prob':0.5,'skip_first_n_losses':0}"
\
--num_training_steps=40000

As soon as I add a non-zero value to 'skip_first_n_losses', e.g.,

bazel-bin/magenta/models/melody_rnn/melody_rnn_train \
--config=basic_rnn \
--run_dir=/homes/sturm/magenta_session \
--sequence_example_file=/homes/sturm/datasets/basicrnnseq/training_melodies.tfrecord
\
--hparams="{'batch_size':100,'rnn_layer_sizes':[512,512,512],'decay_rate':0.97,
'initial_learning_rate':0.03,'decay_steps':1,'dropout_keep_prob':0.5,'skip_first_n_losses':8000}"
\
--num_training_steps=40000

It breaks for some reason.

INFO:tensorflow:hparams = {'rnn_layer_sizes': [512, 512, 512],
'decay_rate': 0.97, 'dropout_keep_prob': 0.5, 'batch_size': 100,
'decay_steps': 2, 'clip_norm': 5, 'initial_learning_rate': 0.03,
'skip_first_n_losses': 8000}
INFO:tensorflow:Train dir: /homes/sturm/magenta_session/train
WARNING:tensorflow:From
/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py:344
in __init__.: __init__ (from tensorflow.python.training.summary_io) is
deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.FileWriter. The interface and behavior is
the same; this is just a rename.
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0
with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:84:00.0
Total memory: 11.92GiB
Free memory: 11.86GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context
when one is currently active; existing: 0x514f840
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 1
with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:85:00.0
Total memory: 11.92GiB
Free memory: 11.86GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 1
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 1:   Y Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating
TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id:
0000:84:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating
TensorFlow device (/gpu:1) -> (device: 1, name: Tesla K80, pci bus id:
0000:85:00.0)
INFO:tensorflow:Starting training loop...
INFO:tensorflow:global_step/sec: 0
W tensorflow/core/framework/op_kernel.cc:975] Invalid argument:
Incompatible shapes: [0] vs. [51100]
     [[Node: mul_4 = Mul[T=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/gpu:0"](ToFloat_2,
Reshape_4)]]
W tensorflow/core/framework/op_kernel.cc:975] Invalid argument:
Incompatible shapes: [0] vs. [51100]
     [[Node: mul_4 = Mul[T=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/gpu:0"](ToFloat_2,
Reshape_4)]]
W tensorflow/core/framework/op_kernel.cc:975] Invalid argument:
Incompatible shapes: [0] vs. [51100]
     [[Node: mul_4 = Mul[T=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/gpu:0"](ToFloat_2,
Reshape_4)]]
W tensorflow/core/framework/op_kernel.cc:975] Invalid argument:
Incompatible shapes: [0] vs. [51100]
     [[Node: mul_4 = Mul[T=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/gpu:0"](ToFloat_2,
Reshape_4)]]
INFO:tensorflow:Error reported to Coordinator: <class
'tensorflow.python.framework.errors_impl.InvalidArgumentError'>,
Incompatible shapes: [0] vs. [51100]
     [[Node: mul_4 = Mul[T=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/gpu:0"](ToFloat_2,
Reshape_4)]]
     [[Node: mul_7/_103 = _Recv[client_terminated=false,
recv_device="/job:localhost/replica:0/task:0/cpu:0",
send_device="/job:localhost/replica:0/task:0/gpu:0",
send_device_incarnation=1, tensor_name="edge_351_mul_7",
tensor_type=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op u'mul_4', defined at:
  File "/homes/sturm/magenta/bazel-bin/magenta/models/melody_rnn/melody_rnn_train.runfiles/__main__/magenta/models/melody_rnn/melody_rnn_train.py",
line 103, in <module>
    console_entry_point()
  File "/homes/sturm/magenta/bazel-bin/magenta/models/melody_rnn/melody_rnn_train.runfiles/__main__/magenta/models/melody_rnn/melody_rnn_train.py",
line 99, in console_entry_point
    tf.app.run(main)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py",
line 43, in run
    sys.exit(main(sys.argv[:1] + flags_passthrough))
  File "/homes/sturm/magenta/bazel-bin/magenta/models/melody_rnn/melody_rnn_train.runfiles/__main__/magenta/models/melody_rnn/melody_rnn_train.py",
line 78, in main
    mode, config, sequence_example_file_paths)
  File "/homes/sturm/magenta/bazel-bin/magenta/models/melody_rnn/melody_rnn_train.runfiles/__main__/magenta/models/shared/events_rnn_graph.py",
line 143, in build_graph
    tf.to_float(tf.not_equal(labels_flat, no_event_label)) * mask_flat)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py",
line 814, in binary_op_wrapper
    return func(x, y, name=name)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py",
line 987, in _mul_dispatch
    return gen_math_ops.mul(x, y, name=name)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py",
line 1613, in mul
    result = _op_def_lib.apply_op("Mul", x=x, y=y, name=name)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py",
line 759, in apply_op
    op_def=op_def)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py",
line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py",
line 1128, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Incompatible shapes:
[0] vs. [51100]
     [[Node: mul_4 = Mul[T=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/gpu:0"](ToFloat_2,
Reshape_4)]]
     [[Node: mul_7/_103 = _Recv[client_terminated=false,
recv_device="/job:localhost/replica:0/task:0/cpu:0",
send_device="/job:localhost/replica:0/task:0/gpu:0",
send_device_incarnation=1, tensor_name="edge_351_mul_7",
tensor_type=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Traceback (most recent call last):
  File "/homes/sturm/magenta/bazel-bin/magenta/models/melody_rnn/melody_rnn_train.runfiles/__main__/magenta/models/melody_rnn/melody_rnn_train.py",
line 103, in <module>
    console_entry_point()
  File "/homes/sturm/magenta/bazel-bin/magenta/models/melody_rnn/melody_rnn_train.runfiles/__main__/magenta/models/melody_rnn/melody_rnn_train.py",
line 99, in console_entry_point
    tf.app.run(main)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py",
line 43, in run
    sys.exit(main(sys.argv[:1] + flags_passthrough))
  File "/homes/sturm/magenta/bazel-bin/magenta/models/melody_rnn/melody_rnn_train.runfiles/__main__/magenta/models/melody_rnn/melody_rnn_train.py",
line 95, in main
    FLAGS.summary_frequency)
  File "/homes/sturm/magenta/bazel-bin/magenta/models/melody_rnn/melody_rnn_train.runfiles/__main__/magenta/models/shared/events_rnn_train.py",
line 75, in run_training
    tf.logging.info('Training complete.')
  File "/homes/sturm/anaconda2/lib/python2.7/contextlib.py", line 35,
in __exit__
    self.gen.throw(type, value, traceback)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py",
line 974, in managed_session
    self.stop(close_summary_writer=close_summary_writer)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py",
line 802, in stop
    stop_grace_period_secs=self._stop_grace_secs)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py",
line 386, in join
    six.reraise(*self._exc_info_to_raise)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py",
line 296, in stop_on_exception
    yield
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py",
line 487, in run
    self.run_loop()
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py",
line 1004, in run_loop
    self._sv.global_step])
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py",
line 766, in run
    run_metadata_ptr)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py",
line 964, in _run
    feed_dict_string, options, run_metadata)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py",
line 1014, in _do_run
    target_list, options, run_metadata)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py",
line 1034, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError:
Incompatible shapes: [0] vs. [51100]
     [[Node: mul_4 = Mul[T=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/gpu:0"](ToFloat_2,
Reshape_4)]]
     [[Node: mul_7/_103 = _Recv[client_terminated=false,
recv_device="/job:localhost/replica:0/task:0/cpu:0",
send_device="/job:localhost/replica:0/task:0/gpu:0",
send_device_incarnation=1, tensor_name="edge_351_mul_7",
tensor_type=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op u'mul_4', defined at:
  File "/homes/sturm/magenta/bazel-bin/magenta/models/melody_rnn/melody_rnn_train.runfiles/__main__/magenta/models/melody_rnn/melody_rnn_train.py",
line 103, in <module>
    console_entry_point()
  File "/homes/sturm/magenta/bazel-bin/magenta/models/melody_rnn/melody_rnn_train.runfiles/__main__/magenta/models/melody_rnn/melody_rnn_train.py",
line 99, in console_entry_point
    tf.app.run(main)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py",
line 43, in run
    sys.exit(main(sys.argv[:1] + flags_passthrough))
  File "/homes/sturm/magenta/bazel-bin/magenta/models/melody_rnn/melody_rnn_train.runfiles/__main__/magenta/models/melody_rnn/melody_rnn_train.py",
line 78, in main
    mode, config, sequence_example_file_paths)
  File "/homes/sturm/magenta/bazel-bin/magenta/models/melody_rnn/melody_rnn_train.runfiles/__main__/magenta/models/shared/events_rnn_graph.py",
line 143, in build_graph
    tf.to_float(tf.not_equal(labels_flat, no_event_label)) * mask_flat)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py",
line 814, in binary_op_wrapper
    return func(x, y, name=name)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py",
line 987, in _mul_dispatch
    return gen_math_ops.mul(x, y, name=name)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py",
line 1613, in mul
    result = _op_def_lib.apply_op("Mul", x=x, y=y, name=name)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py",
line 759, in apply_op
    op_def=op_def)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py",
line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/homes/sturm/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py",
line 1128, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Incompatible shapes:
[0] vs. [51100]
     [[Node: mul_4 = Mul[T=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/gpu:0"](ToFloat_2,
Reshape_4)]]
     [[Node: mul_7/_103 = _Recv[client_terminated=false,
recv_device="/job:localhost/replica:0/task:0/cpu:0",
send_device="/job:localhost/replica:0/task:0/gpu:0",
send_device_incarnation=1, tensor_name="edge_351_mul_7",
tensor_type=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/cpu:0"]()]]


Discussion:

Resolving patch:
diff --git a/magenta/models/shared/events_rnn_graph.py b/magenta/models/shared/events_rnn_graph.py
--- a/magenta/models/shared/events_rnn_graph.py
+++ b/magenta/models/shared/events_rnn_graph.py
@@ -117,14 +117,15 @@ def build_graph(mode, config, sequence_example_file_paths=None):
     logits_flat = tf.contrib.layers.linear(outputs_flat, num_classes)
 
     if mode == 'train' or mode == 'eval':
+      labels_flat = tf.reshape(labels, [-1])
+      mask = tf.sequence_mask(lengths)
       if hparams.skip_first_n_losses:
-        logits = tf.reshape(logits_flat, [hparams.batch_size, -1, num_classes])
-        logits = logits[:, hparams.skip_first_n_losses:, :]
-        logits_flat = tf.reshape(logits, [-1, num_classes])
-        labels = labels[:, hparams.skip_first_n_losses:]
+        skip = tf.minimum(lengths, hparams.skip_first_n_losses)
+        skip_mask = tf.sequence_mask(skip, maxlen=tf.reduce_max(lengths))
+        mask = tf.logical_and(mask, tf.logical_not(skip_mask))
+      mask = tf.cast(mask, tf.float32)
+      mask_flat = tf.reshape(mask, [-1])
 
-      labels_flat = tf.reshape(labels, [-1])
-      mask_flat = tf.reshape(tf.sequence_mask(lengths, dtype=tf.float32), [-1])
       num_logits = tf.to_float(tf.reduce_sum(lengths))
 
       with tf.control_dependencies(


Test patch: