Issue 1:
Please go to Stack Overflow for help and support:

https://stackoverflow.com/questions/tagged/tensorflow

If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).

The form below must be filled out.

It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary):

TensorFlow version (use command below):

Python version:

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version:

GPU model and memory:

Exact command to reproduce:

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

Issue 2:
I want to train with multiple gpu with tensorflow-1.10. I use the estimator and MirroredStrategy. The model is resnet50 which is official model of tensorflow . The following is my code and error log. Am I use MirroredStrategy in the wrong way?

import tensorflow as tf
import numpy as np
from tensorflow.contrib.slim.nets import resnet_v2
import tensorflow.contrib.slim as slim

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data(path='mnist.npz')
x_train = np.expand_dims(x_train, 3).astype(np.float32)
y_train = y_train.astype(np.int32)
x_test = np.expand_dims(x_test, 3).astype(np.float32)
y_test = y_test.astype(np.int32)

tf.logging.set_verbosity(tf.logging.INFO)

cls_num = 10

def model_fn(features, labels, mode):
is_training = (mode == tf.estimator.ModeKeys.TRAIN)
with slim.arg_scope(resnet_v2.resnet_arg_scope()):
logits, endpoints = resnet_v2.resnet_v2_50(features,
num_classes=cls_num,
is_training=is_training)

logits = tf.squeeze(logits, [1, 2])
preds = tf.argmax(logits, 1)
loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)
optimizer = tf.train.AdagradOptimizer(learning_rate=0.01)
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
train_op = optimizer.minimize(loss, global_step=tf.train.get_global_step())

accuracy = tf.metrics.accuracy(labels=labels, predictions=preds)
metrics = {'accuracy': accuracy}

if mode == tf.estimator.ModeKeys.EVAL:
return tf.estimator.EstimatorSpec(mode, loss=loss, eval_metric_ops=metrics)

return tf.estimator.EstimatorSpec(mode, loss=loss, train_op=train_op)

def train_input_fn():
dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
dataset = dataset.repeat(1).batch(128)
return dataset

def eval_input_fn():
dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))
dataset = dataset.repeat(1).batch(128)
return dataset

distribution = tf.contrib.distribute.MirroredStrategy()
config = tf.estimator.RunConfig(train_distribute=distribution)
estimator = tf.estimator.Estimator(model_fn=model_fn, model_dir='logs', config=config)
train_spec = tf.estimator.TrainSpec(input_fn=train_input_fn)
eval_specs = tf.estimator.EvalSpec(input_fn=eval_input_fn)
for _ in xrange(100):
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs)

This is the error log:

2018-08-30 16:58:47.446417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0 1
2018-08-30 16:58:47.446428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0: N Y
2018-08-30 16:58:47.446437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1: Y N
2018-08-30 16:58:47.446877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/device:GPU:0 with 21544 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:83:00.0, compute capability: 6.1)
2018-08-30 16:58:47.447098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/device:GPU:1 with 21544 MB memory) -> physical GPU (device: 1, name: Tesla P40, pci bus id: 0000:84:00.0, compute capability: 6.1)
INFO:tensorflow:Device is available but not used by distribute strategy: /device:CPU:0
INFO:tensorflow:Configured nccl all-reduce.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Error reported to Coordinator:
Traceback (most recent call last):
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 297, in stop_on_exception
yield
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/distribute/python/mirrored_strategy.py", line 519, in run
self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 1133, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "test_mnist.py", line 22, in model_fn
is_training=is_training)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py", line 287, in resnet_v2_50
scope=scope)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py", line 216, in resnet_v2
net = resnet_utils.stack_blocks_dense(net, blocks, output_stride)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
return func(*args, **current_args)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_utils.py", line 215, in stack_blocks_dense
net = block.unit_fn(net, rate=1, **unit)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
return func(*args, **current_args)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py", line 101, in bottleneck
inputs, activation_fn=nn_ops.relu, scope='preact')
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
return func(*args, **current_args)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 650, in batch_norm
outputs = layer.apply(inputs, training=is_training)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 805, in apply
return self.__call__(inputs, *args, **kwargs)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/layers/base.py", line 362, in __call__
outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 736, in __call__
outputs = self.call(inputs, *args, **kwargs)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/layers/normalization.py", line 158, in call
return super(BatchNormalization, self).call(inputs, training=training)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py", line 514, in call
outputs = self._fused_batch_norm(inputs, training=training)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py", line 420, in _fused_batch_norm
momentum)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py", line 369, in _assign_moving_average
with ops.colocate_with(variable):
File "/home/soft/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3939, in _colocate_with_for_gradient
with self.colocate_with(op, ignore_existing):
File "/home/soft/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3992, in colocate_with
op = internal_convert_to_tensor_or_indexed_slices(op, as_ref=True).op
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1255, in internal_convert_to_tensor_or_indexed_slices
value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1094, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/distribute/python/values.py", line 414, in _tensor_conversion_mirrored
assert not as_ref
AssertionError
Traceback (most recent call last):
File "test_mnist.py", line 58, in <module>
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 451, in train_and_evaluate
return executor.run()
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 590, in run
return self.run_local()
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 691, in run_local
saving_listeners=saving_listeners)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 376, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 1143, in _train_model
return self._train_model_distributed(input_fn, hooks, saving_listeners)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 1255, in _train_model_distributed
self.config)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/training/distribute.py", line 777, in call_for_each_tower
return self._call_for_each_tower(fn, *args, **kwargs)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/distribute/python/mirrored_strategy.py", line 308, in _call_for_each_tower
coord.join(threads)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 297, in stop_on_exception
yield
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/distribute/python/mirrored_strategy.py", line 519, in run
self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 1133, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "test_mnist.py", line 22, in model_fn
is_training=is_training)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py", line 287, in resnet_v2_50
scope=scope)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py", line 216, in resnet_v2
net = resnet_utils.stack_blocks_dense(net, blocks, output_stride)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
return func(*args, **current_args)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_utils.py", line 215, in stack_blocks_dense
net = block.unit_fn(net, rate=1, **unit)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
return func(*args, **current_args)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py", line 101, in bottleneck
inputs, activation_fn=nn_ops.relu, scope='preact')
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
return func(*args, **current_args)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 650, in batch_norm
outputs = layer.apply(inputs, training=is_training)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 805, in apply
return self.__call__(inputs, *args, **kwargs)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/layers/base.py", line 362, in __call__
outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 736, in __call__
outputs = self.call(inputs, *args, **kwargs)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/layers/normalization.py", line 158, in call
return super(BatchNormalization, self).call(inputs, training=training)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py", line 514, in call
outputs = self._fused_batch_norm(inputs, training=training)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py", line 420, in _fused_batch_norm
momentum)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/keras/layers/normalization.py", line 369, in _assign_moving_average
with ops.colocate_with(variable):
File "/home/soft/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3939, in _colocate_with_for_gradient
with self.colocate_with(op, ignore_existing):
File "/home/soft/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3992, in colocate_with
op = internal_convert_to_tensor_or_indexed_slices(op, as_ref=True).op
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1255, in internal_convert_to_tensor_or_indexed_slices
value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/soft/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1094, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/soft/lib/python2.7/site-packages/tensorflow/contrib/distribute/python/values.py", line 414, in _tensor_conversion_mirrored
assert not as_ref
AssertionError