Issue 1:
Use tf.one_hot() on Windows 10, GPU, Nvidia 970. Error CUDA_ERROR_LAUNCH_FAILED happens.

What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?

I have post an issue in http://stackoverflow.com/questions/41115476/tensorflow-gpu-cuda-error-launch-failed-on-tf-one-hot . No solution found. More people report the same issue.

Environment info

TensorFlow 0.12.0-rc1

Python 3.5

CUDA 8.0

cuDNN 5.1

OS: Windows 10

GPU: GeForce GTX 970

If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)

import tensorflow as tf
idx_0 = tf.placeholder(tf.int64, [None])
mask = tf.one_hot(idx_0, 3, axis=-1)
sess = tf.Session()
sess.run(tf.global_variables_initializer())
a = sess.run([mask],feed_dict={idx_0:[0,1,2]})
print(a)

What other attempted solutions have you tried?

Run the code in same machine, Linux, GPU. No error.

Run the code in same machine, Windows, CPU. No error.

Logs or other output that would be helpful

(If logs are large, please upload as attachment or provide link).

E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_driver.cc:1177] could not synchronize on CUDA context: CUDA_ERROR_LAUNCH_FAILED :: No stack trace available
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_LAUNCH_FAILED
F c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_util.cc:370] GPU sync failed

Issue 2:
I'm using a LeNet-5 mnist example from Udacity's course. Link to the source code is below.

Training works ok on a CPU (config = tf.ConfigProto(device_count = {'GPU': 0})),

but fails in a GPU mode with the following 'CUDA_ERROR_ILLEGAL_ADDRESS' error:

I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:885] Found device 0 with properties:

name: GeForce GTX 1060 6GB

major: 6 minor: 1 memoryClockRate (GHz) 1.7845

pciBusID 0000:01:00.0

Total memory: 6.00GiB

Free memory: 5.01GiB

I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:906] DMA: 0

I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:916] 0: Y

I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0)

Training...

E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS

F c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_event_mgr.cc:198] Unexpected Event status: 1

I have reproduced same error on two setups.

Environment 1 (Home PC):

windows 10;

latest anaconda 4.2.0, python 3.5;

cuda 8

cudnn 5.1 (for win10)

tensorflow 0.12.0 gpu (https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-0.12.0-cp35-cp35m-win_amd64.whl)

GeForce GTX 1060 6Gb

Environment 2 (Work PC):

windows 7;

latest anaconda 4.2.0, python 3.5;

cuda 8

cudnn 5.1 (for win7)

tensorflow 0.12.0 gpu (https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-0.12.0-cp35-cp35m-win_amd64.whl)

GeForce GTX 660 3Gb

I'm sharing two scripts with minor changes that allow a workaround:

https://drive.google.com/open?id=0B6jkkqMOGy5cNHh3TVpxU283Ykk

Main difference:

Script from example that crashes (LabLenetBad.py) uses raw mnist label data with the tf.one_hot() call.

The workaround (LabLenetGood.py) reads mnist data with (one_hot=True) flag and does not use tf.one_hot() call on the Y placeholder.

I think that tf.one_hot does not work properly on the gpu.