Issue 1:
Environment info

Operating System: Ubuntu 14.04,

GPU server with 8 Tesla K80 GPUs,

Python 2.7 (Anaconda)

Installed version of CUDA and cuDNN:

Cuda 7.5, cuDNN v4

If installed from binary pip package, provide:

Pip package: GPU-enabled, Python 2.7

https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.10.0rc0-cp27-none-linux_x86_64.whl

The output from python -c "import tensorflow; print(tensorflow.__version__)":

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so locally

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally

0.9.0

Steps to reproduce

import tensorflow as tf
sess = tf.Session()

yields the error:

W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active;

see the detailed logs below please:

What have you tried?

I checked the output of the nvidia-smi. There are other processes run by other users currently. Below is the output.

I re-installed tensorflow again using pip, it didn't solve the problem.

+------------------------------------------------------+
| NVIDIA-SMI 352.39 Driver Version: 352.39 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K80 Off | 0000:05:00.0 Off | 0 |
| N/A 33C P0 59W / 149W | 190MiB / 11519MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K80 Off | 0000:06:00.0 Off | 0 |
| N/A 30C P8 30W / 149W | 22MiB / 11519MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Tesla K80 Off | 0000:09:00.0 Off | 0 |
| N/A 25C P8 27W / 149W | 22MiB / 11519MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Tesla K80 Off | 0000:0A:00.0 Off | 0 |
| N/A 28C P8 31W / 149W | 22MiB / 11519MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 4 Tesla K80 Off | 0000:84:00.0 Off | 0 |
| N/A 24C P8 28W / 149W | 22MiB / 11519MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 5 Tesla K80 Off | 0000:85:00.0 Off | 0 |
| N/A 54C P0 106W / 149W | 11461MiB / 11519MiB | 58% Default |
+-------------------------------+----------------------+----------------------+
| 6 Tesla K80 Off | 0000:88:00.0 Off | 0 |
| N/A 49C P0 134W / 149W | 11461MiB / 11519MiB | 97% Default |
+-------------------------------+----------------------+----------------------+
| 7 Tesla K80 Off | 0000:89:00.0 Off | 0 |
| N/A 69C P0 131W / 149W | 11462MiB / 11519MiB | 57% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 155948 C ../../../build/tools/caffe 82MiB |
| 0 196261 C .build_release/test/test_all.testbin 81MiB |
| 5 19542 C python 11436MiB |
| 6 18802 C python 11436MiB |
| 7 85515 C python 11437MiB

Logs

I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:05:00.0
Total memory: 11.25GiB
Free memory: 11.00GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x2f85bd0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 1 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:06:00.0
Total memory: 11.25GiB
Free memory: 11.16GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x339e320
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 2 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:09:00.0
Total memory: 11.25GiB
Free memory: 11.16GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x37b5f60
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 3 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:0a:00.0
Total memory: 11.25GiB
Free memory: 11.16GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x3bd1860
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 4 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:84:00.0
Total memory: 11.25GiB
Free memory: 11.16GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x3ff0cd0
Segmentation fault (core dumped)

Issue 2:
When I set the nvidia driver in exclusive mode and one of the GPU is already used by another process, I get a segmentation fault:

$ python -c "import tensorflow as tf;tf.InteractiveSession()"
I tensorflow/core/common_runtime/local_device.cc:25] Local device intra op parallelism threads: 12
Segmentation fault (core dumped)

If I limit the visible GPUs to only GPUs that have nothing running on them, it don't segfault:

$CUDA_VISIBLE_DEVICES=1 python -c "import tensorflow as tf;tf.InteractiveSession()"
I tensorflow/core/common_runtime/local_device.cc:25] Local device intra op parallelism threads: 12
I tensorflow/core/common_runtime/gpu/gpu_init.cc:88] Found device 0 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:09:00.0
Total memory: 12.00GiB
Free memory: 11.87GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:112] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:122] 0: Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:643] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:09:00.0)
I tensorflow/core/common_runtime/gpu/gpu_region_allocator.cc:47] Setting region size to 12105628263
I tensorflow/core/common_runtime/local_session.cc:45] Local session inter op parallelism threads: 12

Here is my output of nvidia-smi:

$ nvidia-smi
Wed Nov 11 16:48:27 2015
+------------------------------------------------------+
| NVIDIA-SMI 352.39 Driver Version: 352.39 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 750 Off | 0000:05:00.0 On | N/A |
| N/A 48C P8 0W / 38W | 25MiB / 2047MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX TIT... Off | 0000:06:00.0 Off | N/A |
| 42% 82C P2 127W / 250W | 262MiB / 12287MiB | 39% E. Process |
+-------------------------------+----------------------+----------------------+
| 2 GeForce GTX TIT... Off | 0000:09:00.0 Off | N/A |
| 22% 46C P8 17W / 250W | 23MiB / 12287MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 3 GeForce GTX TIT... Off | 0000:0A:00.0 Off | N/A |
| 22% 37C P8 15W / 250W | 361MiB / 12287MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1171 G /usr/bin/X 17MiB |
| 1 32740 C python 209MiB |
| 3 9429 C python 336MiB |
+-----------------------------------------------------------------------------+

I suppose that in the code that check the available GPUs, it don't handle correctly a case when one of the GPUs can't be used.