Issue 1:
System information

OS - High Sierra 10.13

Tensorflow - 1.4

Keras - 2.0.9

CUDA - 9

cuDNN - 7

Describe the problem

CUDA_ERROR_OUT_OF_MEMORY running tensorflow on GPU

Simple program:

import tensorflow as tf
with tf.device('/gpu:0'):
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)

with tf.Session() as sess:
print (sess.run(c))

Output:

(tensorflow) Smit-Shilu:Desktop smitshilu$ python gputest.py
2017-11-07 08:55:50.690390: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:856] OS X does not support NUMA - returning NUMA node zero
2017-11-07 08:55:50.690536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1031] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:c3:00.0
totalMemory: 11.00GiB freeMemory: 10.81GiB
2017-11-07 08:55:50.690560: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:c3:00.0, compute capability: 6.1)
2017-11-07 08:55:50.690914: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 10.27G (11026294784 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2017-11-07 08:55:50.691022: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 9.24G (9923664896 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
[[ 22. 28.]
[ 49. 64.]]

CUDA_ERROR_OUT_OF_MEMORY when import Keras after upgrading to 10.13 and 1.4

>>> from keras.callbacks import *
Using TensorFlow backend.
2017-11-06 16:30:54.704584: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:856] OS X does not support NUMA - returning NUMA node zero
2017-11-06 16:30:54.704700: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1031] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:c3:00.0
totalMemory: 11.00GiB freeMemory: 10.81GiB
2017-11-06 16:30:54.704725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:c3:00.0, compute capability: 6.1)
2017-11-06 16:30:54.705019: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 10.27G (11026489344 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2017-11-06 16:30:54.705125: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 9.24G (9923840000 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): https://github.com/tkuanlun350/Tensorflow-SegNet

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 7 x64

TensorFlow installed from (source or binary): https://pypi.python.org/pypi/tensorflow-gpu/1.4.0rc1

TensorFlow version (use command below): 1.4.0

Python version: 3.5

CUDA/cuDNN version: Cuda release 8.0, V8.0.60. cuDNN 6.

GPU model and memory: NVIDIA P4

Exact command to reproduce: c:\python35\python3 main.py --log_dir=./logs --image_dir={image dir} --val_dir= {validation dir} --batch_size=15 --training=True

Describe the problem

Under 1.3.0 I was able to use a batch size of {15, put your max batch size here} for training. Under 1.4.0 I get Resource Exhausted errors for that batch size. So use of GPU resources is going up. Not the right direction.

For me here are the performance effects:

TensorFlow GPU 1.3.0: 9.8 images/sec for batch size: 15

TensorFlow GPU 1.4.0: Can't do batch size: 15. 7.8 images/sec for batch size: 12

Source code / logs

tf_bug2.txt