Issue 1:
ERROR message

ERROR: /csproject/dygroup2/czeng/downloads/tensorflow/tensorflow/core/kernels/BUILD:1575:1: undeclared inclusion(s) in rule '//tensorflow/core/kernels:training_ops_gpu':
this rule is missing dependency declarations for the following files included by 'tensorflow/core/kernels/training_ops_gpu.cu.cc':

Environment info

Operating System:

CentOS 6.5

Installed version of CUDA and cuDNN:

cuda-7.5

cuDNN v5

Procedure to reproduce:

I used a custom GCC version (which is in /usr/local/GNU/gcc-4.9.2/) following the guide in #2109 , and executed

bazel build --verbose_failures -c opt --config=cuda --genrule_strategy=standalone -j 10 //tensorflow/cc:tutorials_example_trainer

Output of $ echo | gcc4 -E -xc++ - -v

Using built-in specs.
COLLECT_GCC=gcc4
Target: x86_64-unknown-linux-gnu
Configured with: ../gcc-4.9.2/configure --prefix=/usr/local/GNU/gcc-4.9.2 --enable-clocale=generic
Thread model: posix
gcc version 4.9.2 (GCC)
COLLECT_GCC_OPTIONS='-E' '-v' '-mtune=generic' '-march=x86-64'
/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../libexec/gcc/x86_64-unknown-linux-gnu/4.9.2/cc1plus -E -quiet -v -iprefix /export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/ -D_GNU_SOURCE - -mtune=generic -march=x86-64
ignoring nonexistent directory "/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../x86_64-unknown-linux-gnu/include"
ignoring duplicate directory "/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2"
ignoring duplicate directory "/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/x86_64-unknown-linux-gnu"
ignoring duplicate directory "/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/backward"
ignoring duplicate directory "/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include"
ignoring duplicate directory "/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include-fixed"
ignoring nonexistent directory "/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../x86_64-unknown-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
/csproject/dygroup2/czeng/venv/cudnnv5/include
/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2
/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/x86_64-unknown-linux-gnu
/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/backward
/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include
/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include-fixed
/usr/local/include
/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/../../include
/usr/include
End of search list.
#1 "<stdin>"
#1 "<built-in>"
#1 "<command-line>"
#1 "<stdin>"
COMPILER_PATH=/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../libexec/gcc/x86_64-unknown-linux-gnu/4.9.2/:/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../libexec/gcc/
LIBRARY_PATH=/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/:/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/:/csproject/dygroup2/czeng/venv/cudnnv5/lib64/../lib64/:/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/csproject/dygroup2/czeng/venv/cudnnv5/lib64/:/export/centos6_usr_local/GNU/gcc-4.9.2/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-E' '-v' '-mtune=generic' '-march=x86-64'

Issue 2:
Environment info

Operating System: Ubuntu 16.04, GeForce GTX TITAN X

Installed version of CUDA and cuDNN:

(please attach the output of ls -l /path/to/cuda/lib/libcud*):

ls /usr/local/cuda-8.0/lib64/libcud*

/usr/local/cuda-8.0/lib64/libcudadevrt.a

/usr/local/cuda-8.0/lib64/libcudart.so

/usr/local/cuda-8.0/lib64/libcudart.so.8.0

/usr/local/cuda-8.0/lib64/libcudart.so.8.0.44

/usr/local/cuda-8.0/lib64/libcudart_static.a

/usr/local/cuda-8.0/lib64/libcudnn.so

/usr/local/cuda-8.0/lib64/libcudnn.so.5

/usr/local/cuda-8.0/lib64/libcudnn.so.5.1.10

/usr/local/cuda-8.0/lib64/libcudnn.so.5.1.5

/usr/local/cuda-8.0/lib64/libcudnn_static.a

If installed from binary pip package, provide:

Installed with pip tensorflow-gpu

~ python -c "import tensorflow; print(tensorflow.version)

1.0.0

If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)

Using the provided tf.contrib.learn model in get_started/get_started documentation

import tensorflow as tf
import numpy as np

features = [tf.contrib.layers.real_valued_column("x", dimension=1)]
estimator = tf.contrib.learn.LinearRegressor(feature_columns=features)
x = np.array([1., 2., 3., 4.])
y = np.array([0., -1., -2., -3.])
input_fn = tf.contrib.learn.io.numpy_input_fn({"x":x}, y, batch_size=4, num_epochs=1000)
estimator.fit(input_fn=input_fn, steps=10)

The Core model works fine.

Logs or other output that would be helpful

(If logs are large, please upload as attachment or provide link).

python test-oom.py
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpew3U6z
WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.
WARNING:tensorflow:From /home/gajop/projekti/ubuntu-ranking-dataset-creator/env/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/head.py:1362: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:910] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:01:00.0
Total memory: 11.92GiB
Free memory: 11.52GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 11.92G (12799180800 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY