Issue 1:
I'm using a LeNet-5 mnist example from Udacity's course. Link to the source code is below.

Training works ok on a CPU (config = tf.ConfigProto(device_count = {'GPU': 0})),

but fails in a GPU mode with the following 'CUDA_ERROR_ILLEGAL_ADDRESS' error:

I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:885] Found device 0 with properties:

name: GeForce GTX 1060 6GB

major: 6 minor: 1 memoryClockRate (GHz) 1.7845

pciBusID 0000:01:00.0

Total memory: 6.00GiB

Free memory: 5.01GiB

I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:906] DMA: 0

I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:916] 0: Y

I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0)

Training...

E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS

F c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_event_mgr.cc:198] Unexpected Event status: 1

I have reproduced same error on two setups.

Environment 1 (Home PC):

windows 10;

latest anaconda 4.2.0, python 3.5;

cuda 8

cudnn 5.1 (for win10)

tensorflow 0.12.0 gpu (https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-0.12.0-cp35-cp35m-win_amd64.whl)

GeForce GTX 1060 6Gb

Environment 2 (Work PC):

windows 7;

latest anaconda 4.2.0, python 3.5;

cuda 8

cudnn 5.1 (for win7)

tensorflow 0.12.0 gpu (https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-0.12.0-cp35-cp35m-win_amd64.whl)

GeForce GTX 660 3Gb

I'm sharing two scripts with minor changes that allow a workaround:

https://drive.google.com/open?id=0B6jkkqMOGy5cNHh3TVpxU283Ykk

Main difference:

Script from example that crashes (LabLenetBad.py) uses raw mnist label data with the tf.one_hot() call.

The workaround (LabLenetGood.py) reads mnist data with (one_hot=True) flag and does not use tf.one_hot() call on the Y placeholder.

I think that tf.one_hot does not work properly on the gpu.

Issue 2:
Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary): binary

TensorFlow version (use command below): 1.13.1

Python version: 3.5.2

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version: CUDA 10.0

GPU model and memory:

Describe the current behavior

Using tf.summary.scalar in keras loss/metrics function, then use model_to_estimator to convert keras model to estimator, tensorboard get duplicate scalars with totally same value. (e.g., metrics/accuracy/train and metrics/accuracy/train_1)

Describe the expected behavior

It should only summary one scalar.

Code to reproduce the issue

A example here:

...

model = tf.keras.Model(inputs=input_layer, outputs=[output_layer1, output_layer2])

def accuracy(y_true, y_pred):
y_pred, y_true = reshape_keras_tensor(y_pred, y_true)
acc = tf.reduce_mean(
tf.cast(tf.equal(tf.cast(y_true, tf.int64), tf.argmax(input=y_pred, axis=1)), tf.float32))
# Where I summary, without this line, it will not summary anything during training
tf.summary.scalar('train', acc)
return acc

def total_loss(y_true, y_pred):
y_pred, y_true = reshape_keras_tensor(y_pred, y_true)
loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
labels=y_true,
logits=y_pred)
loss = tf.reduce_mean(cross_entropy)
# Where I summary, without this line, it will not summary anything during training
tf.summary.scalar('train', loss)
return loss

...

model.compile(optimizer=optimizer_builder.build(config.optimizer),
loss={output_layer1: total_loss},
metrics={output_layer1: accuracy})

estimator = tf.keras.estimator.model_to_estimator(
keras_model=model,
config=run_config,
)
train_spec = tf.estimator.TrainSpec(
input_fn=lambda: train_input_fn(config))
eval_spec = tf.estimator.EvalSpec(
input_fn=lambda: eval_input_fn(config))
tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)

Other info / logs

TensorBoard with duplicate scalars: