Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

no (slightly modified)

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Ubuntu 16.04

TensorFlow installed from (source or binary):

binary

TensorFlow version (use command below):

tensorflow-gpu 1.8.0

Python version:

Python version: 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19)

[GCC 7.2.0]

Bazel version (if compiling from source):

N/A

GCC/Compiler version (if compiling from source):

N/A

CUDA/cuDNN version:

9.0.176 / 7.1.2

GPU model and memory:

GPU[0] GeForce GTX 1080 Ti

Exact command to reproduce:

import tensorflow as tf
import numpy
import matplotlib.pyplot as plt
import sklearn.datasets.samples_generator as sample_gen
rng = numpy.random

# Parameters
learning_rate = 0.01
training_epochs = 500
display_step = 50

# Training set
n_features=1
n_samples=100

# Training Data
# train_X = numpy.asarray([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,
# 7.042,10.791,5.313,7.997,5.654,9.27,3.1])
# train_Y = numpy.asarray([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,
# 2.827,3.465,1.65,2.904,2.42,2.94,1.3])
train_X, train_Y = sample_gen.make_regression(n_samples=n_samples, n_features=n_features, random_state=0)
# n_samples = train_X.shape[0]

# tf Graph Input
X = tf.placeholder("float")
Y = tf.placeholder("float")

# Set model weights
W = tf.Variable(rng.randn(), name="weight")
b = tf.Variable(rng.randn(), name="bias")

# Construct a linear model
pred = tf.add(tf.multiply(X, W), b)

# Mean squared error
cost = tf.reduce_sum(tf.pow(pred-Y, 2))/(2*n_samples)
# Gradient descent
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

# Initialize the variables (i.e. assign their default value)
init = tf.global_variables_initializer()

# Start training
with tf.Session() as sess:
sess.run(init)

# Fit all training data
for epoch in range(training_epochs):
for (x, y) in zip(train_X, train_Y):
sess.run(optimizer, feed_dict={X: x, Y: y})

#Display logs per epoch step
if (epoch+1) % display_step == 0:
c = sess.run(cost, feed_dict={X: train_X, Y:train_Y})
print("Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(c),
"W=", sess.run(W), "b=", sess.run(b))

print("Optimization Finished!")
training_cost = sess.run(cost, feed_dict={X: train_X, Y: train_Y})
print("Training cost=", training_cost, "W=", sess.run(W), "b=", sess.run(b), '\n')

#Graphic display
plt.plot(train_X, train_Y, 'ro', label='Original data')
plt.plot(train_X, sess.run(W) * train_X + sess.run(b), label='Fitted line')
plt.legend()
plt.show()
plt.show()

Describe the problem

Same issue as #15880 here, with a fully reproducible example using latest TF 1.8 with CUDA 9.0 and cuDNN 7.1 on Ubuntu 16.04. So same old story, but this time I'm giving you a model solution to GPU memory management - the Catboost library by Yandex.

I confirm that Tensorflow does not release GPU memory after preallocating most of the available VRAM

(leaving only a few percent free). This memory should be freed by TF immediately after use for other modeling frameworks to use, so it is a bug that needs to be repaired. To see how it is done, refer to how Catboost manages GPU resources.

If you are using Jupyter Notebook, restarting python kernel is relatively easy and it "solves" the problem, but of course at the cost of losing all your data loaded to CPU memory.

Issue 2:
When using the TF C++ library inside an application that also uses GPUs for other tasks (not implemented in TF), it would be useful to be able to deallocate all the GPU memory TF has allocated once the session is closed, and no further TF calls are expected for the time being. gpu_options.allow_growth keeps TF's allocated pool small, but it still can grow to several GB. Even after the session is deleted, the pool doesn't shrink. To free it up, the whole application must be restarted, if I'm not mistaken.

Being able to destroy the ProcessState singleton seems to solve it without breaking anything. However, its destructor is protected. Alternatively, getting the Allocator for each GPU from ProcessState and manually destroying them does the trick, but renders TF unusable for all future operations because ProcessState still thinks the Allocators exist and doesn't recreate them when they are required again.

I think making the ProcessState destructor public (or adding a public method to invoke similar code) would be the best solution, but maybe I'm missing an obvious solution that already exists?