Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

Yes, duplicated the minimal code from documentation snippet here

But the included keras_mnist.py example is broken as well.

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Docker container tensorflow/tensorflow:1.11.0-rc2-gpu-py3

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary):

TensorFlow version (use command below):

Python version:

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version:

GPU model and memory:

2 x 1080ti

Exact command to reproduce:

Describe the problem

keras_mnist.py example seems to break. Investigating further...

Optimizers other than tf.train.GradientDescentOptimizer are not working with MirroredStrategy for Keras models as shown in this official example

Source code / logs

Setup, copied from the example:

inputs = tf.keras.layers.Input(shape=(1,))
predictions = tf.keras.layers.Dense(1)(inputs)
model = tf.keras.models.Model(inputs=inputs, outputs=predictions)

features = tf.data.Dataset.from_tensors([1.]).repeat(10000).batch(10)
labels = tf.data.Dataset.from_tensors([1.]).repeat(10000).batch(10)
train_dataset = tf.data.Dataset.zip((features, labels))

distribution = tf.contrib.distribute.MirroredStrategy()

Then, this works:

model.compile(loss='categorical_crossentropy',
optimizer=tf.train.GradientDescentOptimizer(learning_rate=0.2),
distribute=distribution)
model.fit(train_dataset, epochs=5, steps_per_epoch=10)

But, Keras optimizers don't seem to work:

model.compile(loss='categorical_crossentropy',
optimizer=tf.keras.optimizers.SGD(lr=0.2, momentum=0.9),
distribute=distribution)
model.fit(train_dataset, epochs=5, steps_per_epoch=10)

and these Tensorflow Optimizers are throwing errors as well:

model.compile(loss='categorical_crossentropy',
optimizer=tf.train.AdamOptimizer(learning_rate=0.2),
distribute=distribution)
model.fit(train_dataset, epochs=5, steps_per_epoch=10)

model.compile(loss='categorical_crossentropy',
optimizer=tf.train.MomentumOptimizer(learning_rate=0.2, momentum=0.9),
distribute=distribution)
model.fit(train_dataset, epochs=5, steps_per_epoch=10)

model.compile(loss='categorical_crossentropy',
optimizer=tf.train.RMSPropOptimzier(learning_rate=0.2),
distribute=distribution)
model.fit(train_dataset, epochs=5, steps_per_epoch=10)

Error looks like:

File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/engine/training.py", line 1590, in fit
validation_steps=validation_steps)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/engine/training_distributed.py", line 125, in fit_loop
orig_model_weights = model.get_weights()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/engine/network.py", line 483, in get_weights
return backend.batch_get_value(weights)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/backend.py", line 2717, in batch_get_value
return get_session().run(tensors)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/backend.py", line 465, in get_session
_initialize_variables(session)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/backend.py", line 710, in _initialize_variables
variables = _get_variables(ops.get_default_graph())
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/backend.py", line 704, in _get_variables
variables.update(opt.optimizer.variables())
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/optimizer.py", line 785, in variables
optimizer_variables = [v for v in self._non_slot_variables()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/optimizer.py", line 786, in <listcomp>
if _from_current_graph(v)]
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/optimizer.py", line 779, in _from_current_graph
return variable.op.graph is current_graph
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/distribute/python/values.py", line 305, in op
return self.get().op
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/distribute/python/values.py", line 73, in get
(device, self._index.keys(), device_util.current())), e)
File "<string>", line 3, in raise_from
ValueError: Device /replica:0/task:0/device:CPU:0 not found in dict_keys(['/replica:0/task:0/device:GPU:1', '/replica:0/task:0/device:GPU:0']) (current device )

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Linux Ubuntu 16.04

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary):binary (conda install tensorflow-gpu==1.14.0)

TensorFlow version (use command below): 1.14.0

Python version: Python 3.7.3

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version: 10

GPU model and memory: nvidia 1080, 8GB

You can collect some of this information using our environment capture

script

You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior

Attempting to use progbar as a callback to model.fit() fails.

Describe the expected behavior

If runing model.fit() with progbar callback and verbosity=0 should be identical to verbosity=1 and no progbar callback

Code to reproduce the issue

Provide a reproducible test case that is the bare minimum necessary to generate the problem.

current output:

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'

import tensorflow as tf

print('tf_version:', tf.__version__, 'gpu available:', tf.test.is_gpu_available())
model = tf.keras.applications.ResNet50()

print('compiling model')
model.compile(optimizer='SGD', loss=tf.keras.losses.categorical_crossentropy)

print('running fit function')
x = tf.data.Dataset.from_tensors(tf.zeros([16]+model.input.shape.as_list()[1:]))
y = tf.data.Dataset.from_tensors(tf.zeros([16]+model.output.shape.as_list()[1:]))
print('x:', x, '\ny', y)
model.fit(tf.data.Dataset.zip((x,y)).repeat().shuffle(buffer_size=1),
steps_per_epoch=10,
verbose=0,
callbacks=[tf.keras.callbacks.ProgbarLogger('steps')])
print('done')

Other info / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

running the code above recreates the problem.

tf_version: 1.14.0 gpu available: True
WARNING: Logging before flag parsing goes to stderr.
W1003 10:20:49.769316 140195608012544 deprecation.py:506] From /home/hackerman/anaconda3/envs/py36/lib/python3.7/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
compiling model
running fit function
x: <DatasetV1Adapter shapes: (16, 224, 224, 3), types: tf.float32>
y <DatasetV1Adapter shapes: (16, 1000), types: tf.float32>
done

expected output (reproduced by commenting out callback, and setting verbose to 1):

tf_version: 1.14.0 gpu available: True
WARNING: Logging before flag parsing goes to stderr.
W1003 10:21:47.987383 140242665334528 deprecation.py:506] From /home/hackerman/anaconda3/envs/py36/lib/python3.7/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
compiling model
running fit function
x: <DatasetV1Adapter shapes: (16, 224, 224, 3), types: tf.float32>
y <DatasetV1Adapter shapes: (16, 1000), types: tf.float32>
10/10 [==============================] - 6s 592ms/step - loss: 0.0000e+00
done

bonus bug (if keeping callback and setting verbosity to 1):

tf_version: 1.14.0 gpu available: True
WARNING: Logging before flag parsing goes to stderr.
W1003 10:23:10.488721 140225721059072 deprecation.py:506] From /home/hackerman/anaconda3/envs/py36/lib/python3.7/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
compiling model
running fit function
x: <DatasetV1Adapter shapes: (16, 224, 224, 3), types: tf.float32>
y <DatasetV1Adapter shapes: (16, 1000), types: tf.float32>
10/10 [==============================] - 6s 597ms/step - loss: 0.0000e+00
10/10 [==============================] - 6s 597ms/step - loss: 0.0000e+00
done