Issue 1:
Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Debian

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary):binary

TensorFlow version (use command below):1.13.1,1.14.0，2.0.0a0，1.9.0

Python version:3.7,3.6

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version:Cuda 10/7.5.0,7.4.2,7.4.1,7.4.0,Cuda 9 can't support 2060

GPU model and memory:rtx 2060 6GB

You can collect some of this information using our environment capture

script

You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior

Describe the expected behavior

Code to reproduce the issue

conda activate base

2.0.0-alpha0

2019-05-29 19:58:31.543654: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA

2019-05-29 19:58:31.557759: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1

2019-05-29 19:58:31.670769: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1009] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2019-05-29 19:58:31.672348: I tensorflow/compiler/xla/service/service.cc:162] XLA service 0x55c9b57a0a60 executing computations on platform CUDA. Devices:

2019-05-29 19:58:31.672364: I tensorflow/compiler/xla/service/service.cc:169] StreamExecutor device (0): Graphics Device, Compute Capability 7.5

2019-05-29 19:58:31.693201: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2904000000 Hz

2019-05-29 19:58:31.693619: I tensorflow/compiler/xla/service/service.cc:162] XLA service 0x55c9b580db30 executing computations on platform Host. Devices:

2019-05-29 19:58:31.693637: I tensorflow/compiler/xla/service/service.cc:169] StreamExecutor device (0): ,

2019-05-29 19:58:31.693791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1467] Found device 0 with properties:

name: Graphics Device major: 7 minor: 5 memoryClockRate(GHz): 1.71

pciBusID: 0000:01:00.0

totalMemory: 5.76GiB freeMemory: 5.17GiB

2019-05-29 19:58:31.693804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1546] Adding visible gpu devices: 0

2019-05-29 19:58:31.693844: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0

2019-05-29 19:58:31.694402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1015] Device interconnect StreamExecutor with strength 1 edge matrix:

2019-05-29 19:58:31.694413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1021] 0

2019-05-29 19:58:31.694419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1034] 0: N

2019-05-29 19:58:31.694516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1149] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4990 MB memory) -> physical GPU (device: 0, name: Graphics Device, pci bus id: 0000:01:00.0, compute capability: 7.5)

Number of training examples: 60000

Number of test examples: 10000

Epoch 1/5

2019-05-29 19:58:32.786674: W ./tensorflow/core/framework/model.h:202] Encountered a stop event that was not preceded by a start event.

2019-05-29 19:58:36.114172: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0

2019-05-29 19:58:36.288348: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7

2019-05-29 19:58:36.934479: E tensorflow/stream_executor/cuda/cuda_dnn.cc:338] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

2019-05-29 19:58:36.945020: E tensorflow/stream_executor/cuda/cuda_dnn.cc:338] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

2019-05-29 19:58:36.945127: W tensorflow/core/common_runtime/base_collective_executor.cc:214] BaseCollectiveExecutor::StartAbort Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

[[{{node conv2d/Conv2D}}]]

Other info / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

Issue 2:
Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes and No (described below)

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Manjaro

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary): tf-nightly-gpu (Dec 19, r1.13)

TensorFlow version (use command below): 1.13.0-dev20181219

Python version: 3.7.1

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version: CUDA 10 with cuDNN 7.4.1

GPU model and memory: RTX 2070 8GB

Describe the current behavior

I'm running the CNN model on MNIST. When I'm running with the GPU, I am encountering

2018-12-20 20:09:13.644176: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

I did some digging and realized that it is a memory issue (which shouldn't be the case as I have 32GB of RAM and 64GB of swap. I ran htop when running the model and I have 20+GB free, which is more than enough to fit the 8GB vRAM mappings.

Using the gpu_options.allow_growth = True gets the model to work properly, and setting os.environ['CUDA_VISIBLE_DEVICES'] = '-1' also works. This means that I AM facing a memory issue, but I don't see how.

Also, using gpu_options.allow_growth = True does not fix the same issue when trying to run tensorflow/models/official/mnist/ model, which should have a similar behavior with my code.

Code to reproduce the issue

import os
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
import math
import time
# Killing optional CPU driver warnings
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
# os.environ['CUDA_VISIBLE_DEVICES'] = '-1'
tf.logging.set_verbosity(tf.logging.ERROR)

class Model:

def __init__(self, image, label):
"""
A Model class contains a computational graph that classifies images
to predictions. Each of its methods builds part of the graph
on Model initialization. Do not modify the constructor, as doing so
would break the autograder. You may, however, add class variables
to use in your graph-building. e.g. learning rate,

image: the input image to the computational graph as a tensor
label: the correct label of an image as a tensor
prediction: the output prediction of the computational graph,
produced by self.forward_pass()
optimize: the model's optimizing tensor produced by self.optimizer()
loss: the model's loss produced by computing self.loss_function()
accuracy: the model's prediction accuracy
"""
self.image = image
self.label = label

# TO-DO: Add any class variables you want to use.

self.prediction = self.forward_pass()
self.loss = self.loss_function()
self.optimize = self.optimizer()
self.accuracy = self.accuracy_function()

def forward_pass(self):
"""
Predicts a label given an image using convolution layers

:return: the prediction as a tensor
"""
filter_1 = tf.Variable(tf.truncated_normal([3, 3, 1, 8], stddev=0.1))
conv_1 = tf.nn.conv2d(self.image, filter_1, [1, 1, 1, 1], "SAME")

reshaped = tf.reshape(conv_1, shape=[50, -1])

L1 = reshaped.shape[1].value
L2 = 500
W1 = tf.Variable(tf.random_normal([L1, L2], mean=0, stddev=0.01))
b1 = tf.Variable(tf.random_normal([L2], mean=0, stddev=0.01))
relu_1 = tf.nn.relu(tf.matmul(reshaped, W1) + b1)

W2 = tf.Variable(tf.random_normal([L2, 10], mean=0, stddev=0.01))
b2 = tf.Variable(tf.random_normal([10], mean=0, stddev=0.01))
logits = tf.nn.relu(tf.matmul(relu_1, W2) + b2)
return logits

def loss_function(self):
"""
Calculates the model cross-entropy loss

:return: the loss of the model as a tensor
"""
loss = tf.losses.softmax_cross_entropy(onehot_labels=self.label, logits=self.prediction)
return loss

def optimizer(self):
"""
Optimizes the model loss using an Adam Optimizer

:return: the optimizer as a tensor
"""
learning_rate = 0.1
sgd = tf.train.GradientDescentOptimizer(learning_rate)
train = sgd.minimize(self.loss)
return train

def accuracy_function(self):
"""
Calculates the model's prediction accuracy by comparing
predictions to correct labels – no need to modify this

:return: the accuracy of the model as a tensor
"""
correct_prediction = tf.equal(tf.argmax(self.prediction, 1),
tf.argmax(self.label, 1))
return tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

def main():
t_start = time.time()

mnist = input_data.read_data_sets("data/mnist/", one_hot=True)
batch_sz = 50
batch = 2000

inputs = tf.placeholder(shape=[batch_sz, 28, 28, 1], dtype=tf.float32)
labels = tf.placeholder(shape=[batch_sz, 10], dtype=tf.float32)

model = Model(inputs, labels)

session_config = tf.ConfigProto(gpu_options=tf.GPUOptions(allow_growth=True))
sess = tf.Session(config=session_config)

# sess = tf.Session()

sess.run(tf.global_variables_initializer())
for i in range(batch):
next_image, next_label = mnist.train.next_batch(batch_sz)
next_image = next_image.reshape((batch_sz, 28, 28, 1))
sess.run(model.optimize, feed_dict={inputs: next_image, labels: next_label})

acc, test_images, test_labels = 0, mnist.test.images, mnist.test.labels
test_batch = math.ceil(len(test_images) / batch_sz)
for i in range(test_batch):
batch_images = test_images[i * batch_sz: (i + 1) * batch_sz]
batch_images = batch_images.reshape((batch_sz, 28, 28, 1))
batch_labes = test_labels[i * batch_sz: (i + 1) * batch_sz]
acc += sess.run(model.accuracy, feed_dict={inputs: batch_images, labels: batch_labes})
acc /= test_batch
print(acc)

print(time.time() - t_start, 'seconds')

return

if __name__ == '__main__':
main()