Issue 1:
Operating System:

Ubuntu 16.04

Installed version of CUDA and cuDNN:

server@server:~$ ls -l /usr/local/cuda/lib64/libcud*

-rw-r--r-- 1 root root 560184 9月 7 09:56 /usr/local/cuda/lib64/libcudadevrt.a

lrwxrwxrwx 1 root root 16 9月 7 09:56 /usr/local/cuda/lib64/libcudart.so -> libcudart.so.8.0

lrwxrwxrwx 1 root root 19 9月 7 09:56 /usr/local/cuda/lib64/libcudart.so.8.0 -> libcudart.so.8.0.27

-rwxr-xr-x 1 root root 394472 9月 7 09:56 /usr/local/cuda/lib64/libcudart.so.8.0.27

-rw-r--r-- 1 root root 737516 9月 7 09:56 /usr/local/cuda/lib64/libcudart_static.a

The commit hash (git rev-parse HEAD)

server@server:~/tensorflow$ git rev-parse HEAD

2ab7e63

The output of bazel version

server@server:~$ bazel version

Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar

Build time: Thu Jan 01 00:00:00 1970 (0)

Build timestamp: Thu Jan 01 00:00:00 1970 (0)

Build timestamp as int: 0

Here is my bazel build result:

INFO: From Compiling tensorflow/core/kernels/tile_ops_gpu.cu.cc:

nvcc warning : option '--relaxed-constexpr' has been deprecated and replaced by option '--expt-relaxed-constexpr'.

nvcc warning : option '--relaxed-constexpr' has been deprecated and replaced by option '--expt-relaxed-constexpr'.

INFO: From Compiling tensorflow/core/kernels/tile_ops_gpu.cu.cc:

nvcc warning : option '--relaxed-constexpr' has been deprecated and replaced by option '--expt-relaxed-constexpr'.

nvcc warning : option '--relaxed-constexpr' has been deprecated and replaced by option '--expt-relaxed-constexpr'.

Target //tensorflow/cc:tutorials_example_trainer up-to-date:

bazel-bin/tensorflow/cc/tutorials_example_trainer

INFO: Elapsed time: 985.075s, Critical Path: 952.03s

Here is my bazel-bin result:

000008/000006 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]

000007/000009 lambda = 39.000000 x = [3.605551 -0.000000] y = [10.816654 -3.605551]

000006/000006 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]

000001/000002 lambda = nan x = [ nan nan] y = [ nan nan]

000002/000006 lambda = nan x = [ nan nan] y = [ nan nan]

000006/000008 lambda = nan x = [ nan nan] y = [ nan nan]

000004/000004 lambda = 1.200000 x = [-0.894427 0.447214] y = [-0.894427 0.894427]

000006/000007 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]

000005/000005 lambda = 249281724416.000000 x = [0.894427 -0.447214] y = [278705438720.000000 -130.000000]

000008/000003 lambda = nan x = [ nan nan] y = [ nan nan]

000007/000009 lambda = 2.400000 x = [0.948683 -0.316228] y = [2.213594 -0.948683]

000008/000004 lambda = nan x = [ nan nan] y = [ nan nan]

000003/000009 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]

000000/000004 lambda = 2.255962 x = [0.930408 -0.366524] y = [2.058176 -0.930408]

000001/000002 lambda = nan x = [ nan nan] y = [ nan nan]

000004/000004 lambda = nan x = [ nan 0.707107] y = [ nan nan]

000008/000000 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]

000003/000008 lambda = nan x = [ nan nan] y = [ nan nan]

000006/000007 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]

000007/000009 lambda = 2.172414 x = [0.919145 -0.393919] y = [1.969597 -0.919145]

000008/000006 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]

000008/000003 lambda = nan x = [ nan nan] y = [ nan nan]

000002/000006 lambda = nan x = [ nan nan] y = [ nan nan]

000000/000004 lambda = 2.115613 x = [0.911220 -0.411921] y = [1.909816 -0.911220]

000003/000008 lambda = nan x = [ nan nan] y = [ nan nan]

000008/000004 lambda = nan x = [ nan nan] y = [ nan nan]

000003/000009 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]

000006/000006 lambda = 1786627917493567488.000000 x = [715827904.000000 348406848.000000] y = [2844297216.000000 -715827904.000000]

000008/000000 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]

000007/000009 lambda = 2.080292 x = [0.906183 -0.422885] y = [1.872779 -0.906183]

000008/000003 lambda = nan x = [ nan nan] y = [ nan nan]

000003/000008 lambda = nan x = [ nan nan] y = [ nan nan]

000008/000004 lambda = nan x = [ nan nan] y = [ nan nan]

000006/000007 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]

000008/000006 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]

000004/000004 lambda = nan x = [ nan nan] y = [ nan nan]

000005/000005 lambda = 3.000000 x = [1.000000 -0.000000] y = [3.000000 -1.000000]

000003/000009 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]

000008/000000 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]

000008/000006 lambda = 2.000000 x = [0.894427 -0.447214] y = [1.788854 -0.894427]

000006/000006 lambda = 2.584623 x = [0.969760 -0.244061] y = [2.421158 -0.969760]

000004/000004 lambda = nan x = [ nan nan] y = [ nan nan]

000007/000009 lambda = 2.038786 x = [0.900159 -0.435561] y = [1.829356 -0.900159]

I've searched the nan problem, according to #2037, i've tried "--num_concurrent_sessions=1" and "--num_concurrent_steps=1", and it seems to be fine

Here is the output:

server@server:~/tensorflow$ bazel-bin/tensorflow/cc/tutorials_example_trainer --use_gpu --num_concurrent_sessions=1 --num_concurrent_steps=1

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so.8.0 locally

I tensorflow/stream_executor/dso_loader.cc:105] Couldn't open CUDA library libcudnn.so.5. LD_LIBRARY_PATH: /usr/local/cuda-8.0/lib64:

I tensorflow/stream_executor/cuda/cuda_dnn.cc:3304] Unable to load cuDNN DSO

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so.8.0 locally

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so.8.0 locally

I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:

name: GeForce GTX TITAN X

major: 5 minor: 2 memoryClockRate (GHz) 1.076

pciBusID 0000:02:00.0

Total memory: 11.92GiB

Free memory: 11.40GiB

I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0

I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0: Y

I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:02:00.0)

000000/000000 lambda = 1.841570 x = [0.669396 0.742906] y = [3.493999 -0.669396]

000000/000000 lambda = 2.708984 x = [0.982138 -0.188162] y = [2.570089 -0.982138]

000000/000000 lambda = 2.284280 x = [0.934118 -0.356965] y = [2.088423 -0.934118]

000000/000000 lambda = 2.127152 x = [0.912847 -0.408302] y = [1.921937 -0.912847]

000000/000000 lambda = 2.060266 x = [0.903291 -0.429029] y = [1.851815 -0.903291]

000000/000000 lambda = 2.029356 x = [0.898775 -0.438410] y = [1.819504 -0.898775]

000000/000000 lambda = 2.014491 x = [0.896580 -0.442881] y = [1.803979 -0.896580]

000000/000000 lambda = 2.007199 x = [0.895499 -0.445064] y = [1.796367 -0.895499]

000000/000000 lambda = 2.003588 x = [0.894962 -0.446143] y = [1.792599 -0.894962]

000000/000000 lambda = 2.001791 x = [0.894694 -0.446679] y = [1.790724 -0.894694]

000000/000000 lambda = 2.000895 x = [0.894561 -0.446947] y = [1.789788 -0.894561]

000000/000000 lambda = 2.000447 x = [0.894494 -0.447080] y = [1.789321 -0.894494]

000000/000000 lambda = 2.000224 x = [0.894460 -0.447147] y = [1.789088 -0.894460]

000000/000000 lambda = 2.000112 x = [0.894444 -0.447180] y = [1.788971 -0.894444]

000000/000000 lambda = 2.000056 x = [0.894436 -0.447197] y = [1.788913 -0.894436]

000000/000000 lambda = 2.000028 x = [0.894431 -0.447205] y = [1.788883 -0.894431]

000000/000000 lambda = 2.000014 x = [0.894429 -0.447209] y = [1.788869 -0.894429]

When I build the pip-packages. i got another error.

Here is the output:

server@server:~/tensorflow$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

ERROR: no such package '@local_config_cuda//crosstool': BUILD file not found on package path.

ERROR: no such package '@local_config_cuda//crosstool': BUILD file not found on package path.

INFO: Elapsed time: 0.037s

Where am I wrong?

Issue 2:
Environment info

Operating System:

OS 10.10.5

Installed version of CUDA and cuDNN:

$ ls -l /usr/local/cuda/lib/libcud*
-rwxr-xr-x 1 root wheel 8280 Apr 13 01:02 /usr/local/cuda/lib/libcuda.dylib
lrwxr-xr-x 1 root wheel 45 Apr 13 01:03 /usr/local/cuda/lib/libcudadevrt.a -> /Developer/NVIDIA/CUDA-7.5/lib/libcudadevrt.a
lrwxr-xr-x 1 root wheel 50 Apr 13 01:03 /usr/local/cuda/lib/libcudart.7.5.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudart.7.5.dylib
lrwxr-xr-x 1 root wheel 46 Apr 13 01:03 /usr/local/cuda/lib/libcudart.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudart.dylib
lrwxr-xr-x 1 root wheel 49 Apr 13 01:03 /usr/local/cuda/lib/libcudart_static.a -> /Developer/NVIDIA/CUDA-7.5/lib/libcudart_static.a
-rwxr-xr-x@ 1 production204 staff 60108616 Feb 8 2016 /usr/local/cuda/lib/libcudnn.4.dylib
lrwxr-xr-x 1 root admin 47 Aug 29 18:08 /usr/local/cuda/lib/libcudnn.5.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudnn.5.dylib
lrwxr-xr-x 1 root admin 45 Aug 29 18:08 /usr/local/cuda/lib/libcudnn.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudnn.dylib
-rw-r--r--@ 1 production204 staff 59311504 Feb 8 2016 /usr/local/cuda/lib/libcudnn_static.a

The output from python -c "import tensorflow; print(tensorflow.__version__)".

(can't get that far, but i'm using 0.10)

>>> import tensorflow
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.dylib locally
Segmentation fault: 11

If installed from source, provide

The commit hash (git rev-parse HEAD)

4c49dbebef05442c7e72d6129a30574fcd13f0e1

The output of bazel version

$ bazel version
Build label: 0.3.1-homebrew
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Aug 4 09:59:58 2016 (1470304798)
Build timestamp: 1470304798
Build timestamp as int: 1470304798

If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)

$ bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer
ERROR: no such package '@local_config_cuda//crosstool': BUILD file not found on package path.
ERROR: no such package '@local_config_cuda//crosstool': BUILD file not found on package path.
INFO: Elapsed time: 0.076s

What other attempted solutions have you tried?

Downgrading to cuDNN4, switching between 4 and 5

Re-installing bazel

Modifying CROSSTOOL file according to various threads

Manually linking CUDA libraries during ./configure to not use symlinked libraries

Various other hacks over the last week 😭