Issue 1:
While trying to optimize training of a network by moving most work to the GPU, I encountered the following error below. Although this references a variable defined in the batch_normalization helper class I've used, removing that class (replacing all calls to it with an identify function) results in a similar error elsewhere.

Traceback (most recent call last):
File "train_resnet.py", line 106, in <module>
tf.app.run()
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "train_resnet.py", line 103, in main
run_training()
File "train_resnet.py", line 97, in run_training
sess.run(tf.initialize_all_variables())
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 333, in run
run_metadata_ptr)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 573, in _run
feed_dict_string, options, run_metadata)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 648, in _do_run
target_list, options, run_metadata)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 668, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'gradients/resnet_module_1/bn/moments/moments/mean_ss_grad/Maximum/y': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available
[[Node: gradients/resnet_module_1/bn/moments/moments/mean_ss_grad/Maximum/y = Const[_class=["loc:@resnet_module_1/bn/moments/moments/mean_ss"], dtype=DT_INT32, value=Tensor<type: int32 shape: [] values: 1>, _device="/device:GPU:0"]()]]
Caused by op 'gradients/resnet_module_1/bn/moments/moments/mean_ss_grad/Maximum/y', defined at:
File "train_resnet.py", line 106, in <module>
tf.app.run()
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "train_resnet.py", line 103, in main
run_training()
File "train_resnet.py", line 89, in run_training
train_op = opt.minimize(loss, colocate_gradients_with_ops=True, aggregation_method=2)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py", line 193, in minimize
grad_loss=grad_loss)
File "/data/Ray/policy_network/gpu/clipopt.py", line 9, in compute_gradients
*args, **kwargs)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py", line 250, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/gradients.py", line 481, in gradients
in_grads = _AsList(grad_fn(op, *out_grads))
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/math_grad.py", line 41, in _SumGrad
tile_scaling = _safe_shape_div(input_shape, output_shape_kept_dims)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/math_grad.py", line 33, in _safe_shape_div
return x // math_ops.maximum(y, 1)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 1173, in maximum
result = _op_def_lib.apply_op("Maximum", x=x, y=y, name=name)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/op_def_library.py", line 455, in apply_op
as_ref=input_arg.is_ref)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 620, in convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/constant_op.py", line 179, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/constant_op.py", line 166, in constant
attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1224, in __init__
self._traceback = _extract_stack()

...which was originally created as op 'resnet_module_1/bn/moments/moments/mean_ss', defined at:
File "train_resnet.py", line 106, in <module>
tf.app.run()
[elided 1 identical lines from previous traceback]
File "train_resnet.py", line 103, in main
run_training()
File "train_resnet.py", line 77, in run_training
depth=FLAGS.num_features)
File "train_resnet.py", line 45, in model
model = res_3x3_pair(model, depth, "resnet_module_{}".format(idx + 1), training_switch, keep_prob_var)
File "train_resnet.py", line 29, in res_3x3_pair
normed = tf.nn.relu(batch_norm(data, depth, training_switch))
File "/data/Ray/policy_network/gpu/batchnorm.py", line 22, in batch_norm
batch_mean, batch_var = tf.nn.moments(x, [0, 1, 2], name='moments')
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/nn.py", line 712, in moments
name=name)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/nn.py", line 649, in sufficient_statistics
m_ss = math_ops.reduce_sum(m_ss, axes, keep_dims=keep_dims, name="mean_ss")
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 909, in reduce_sum
keep_dims, name=name)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 2087, in _sum
keep_dims=keep_dims, name=name)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/op_def_library.py", line 704, in apply_op
op_def=op_def)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1224, in __init__
self._traceback = _extract_stack()

Environment info

Operating System: Ubuntu 16.06

Installed version of CUDA and cuDNN:

-rw-r--r-- 1 root root 189170 May 24 11:22 /usr/local/cuda/lib/libcudadevrt.a
lrwxrwxrwx 1 root root 16 May 24 11:22 /usr/local/cuda/lib/libcudart.so -> libcudart.so.7.5
lrwxrwxrwx 1 root root 19 May 24 11:22 /usr/local/cuda/lib/libcudart.so.7.5 -> libcudart.so.7.5.18
-rwxr-xr-x 1 root root 311596 May 24 11:22 /usr/local/cuda/lib/libcudart.so.7.5.18
-rw-r--r-- 1 root root 558020 May 24 11:22 /usr/local/cuda/lib/libcudart_static.a

If installed from binary pip package, provide:

pip installed from May 25 nightly build of Linux GPU version.

python -c "import tensorflow; print(tensorflow.__version__)"
0.8.0

Steps to reproduce

grab python files from https://gist.github.com/thouis/9bbd330a153b1f553fd58743a3ae4c9a

python -u -i train_resnet.py --learning_rate 0.005 --num_modules 16 --num_features 192 --summary_dir Workspace --checkpoint_dir Workspace --batch_size 64

What have you tried?

I tried to cast the values in _safe_shape_div() to int64 in case that was the underlying cause, but that seemed to cause different problems.

Logs or other output that would be helpful

All files (python, run log) at:

https://gist.github.com/thouis/9bbd330a153b1f553fd58743a3ae4c9a

Issue 2:
I apologize if this is not actually a issue. When trying to put tf.reduce_mean() on the gpu, I get the following error:

tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'gradients/Mean_grad/mod': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available
[[Node: gradients/Mean_grad/mod = Mod[T=DT_INT32, _device="/device:GPU:0"](gradients/Mean_grad/add, gradients/Mean_grad/Size)]]

I am on commit 43a8c49 from Tue 5/24/2016.

You can reproduce this by modifying the mnist/convolutional.py file by putting the tf.reduce_mean() and optimizer.minimize() inside "with tf.device('/gpu:0'):" statements.

I am trying to do gradient calculations on their respective gpu towers, but compute_gradients() gets hung up on tf.reduce_mean().

Thank you,

Mark