Issue 1:
Environment info

Operating System: Ubuntu 14.04, AWS g2.2xlarge

Installed version of CUDA and cuDNN: CUDA 7.5, cuDNN 4

Tried 3 different version using pip package:

0.8.0:

sudo pip install --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.8.0-cp27-none-linux_x86_64.whl

0.9.0:

sudo pip install --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.9.0-cp27-none-linux_x86_64.whl

0.10.0 rc

sudo pip install --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.10.0rc0-cp27-none-linux_x86_64.whl

If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)

On the same machine, install different version of tf and run the cifar10_train.py

Logs or other output that would be helpful

For version 0.8.0 and 0.9.0, the GPU usage is about 30%

+------------------------------------------------------+
| NVIDIA-SMI 352.99 Driver Version: 352.99 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GRID K520 Off | 0000:00:03.0 Off | N/A |
| N/A 37C P0 51W / 125W | 3854MiB / 4095MiB | 35% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 16558 C python 3841MiB |
+-----------------------------------------------------------------------------+

But the speed is about ~0.24 sec/batch for version 0.8.0 and ~0.22 sec/batch for version 0.9.0

2016-08-26 21:20:41.309814: step 230, loss = 4.28 (522.0 examples/sec; 0.245 sec/batch)
2016-08-26 21:20:43.625017: step 240, loss = 4.26 (582.3 examples/sec; 0.220 sec/batch)
2016-08-26 21:20:45.953772: step 250, loss = 4.24 (544.4 examples/sec; 0.235 sec/batch)
2016-08-26 21:20:48.302202: step 260, loss = 4.23 (540.1 examples/sec; 0.237 sec/batch)
2016-08-26 21:20:50.643760: step 270, loss = 4.21 (554.6 examples/sec; 0.231 sec/batch)
2016-08-26 21:20:52.955326: step 280, loss = 4.20 (545.6 examples/sec; 0.235 sec/batch)
2016-08-26 21:20:55.399758: step 290, loss = 4.18 (476.4 examples/sec; 0.269 sec/batch)
2016-08-26 21:20:57.825254: step 300, loss = 4.17 (548.0 examples/sec; 0.234 sec/batch)
2016-08-26 21:21:00.453533: step 310, loss = 4.15 (543.6 examples/sec; 0.235 sec/batch)
2016-08-26 21:21:02.876055: step 320, loss = 4.14 (513.9 examples/sec; 0.249 sec/batch)
2016-08-26 21:21:05.229421: step 330, loss = 4.13 (580.3 examples/sec; 0.221 sec/batch)
2016-08-26 21:21:07.614095: step 340, loss = 4.11 (528.6 examples/sec; 0.242 sec/batch)

For 0.10.0.rc0, the GPU usage is ~90％：

+------------------------------------------------------+
| NVIDIA-SMI 352.99 Driver Version: 352.99 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GRID K520 Off | 0000:00:03.0 Off | N/A |
| N/A 35C P0 46W / 125W | 3818MiB / 4095MiB | 90% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 16702 C python 3805MiB |
+-----------------------------------------------------------------------------+

But the speed is only ~0.34 sec/batch:

016-08-26 21:24:11.512601: step 30, loss = 4.38 (371.3 examples/sec; 0.345 sec/batch)
2016-08-26 21:24:14.875387: step 40, loss = 4.42 (379.0 examples/sec; 0.338 sec/batch)
2016-08-26 21:24:18.248093: step 50, loss = 4.27 (368.0 examples/sec; 0.348 sec/batch)
2016-08-26 21:24:21.609797: step 60, loss = 4.24 (379.8 examples/sec; 0.337 sec/batch)
2016-08-26 21:24:24.987058: step 70, loss = 4.25 (376.4 examples/sec; 0.340 sec/batch)
2016-08-26 21:24:28.387080: step 80, loss = 4.38 (381.0 examples/sec; 0.336 sec/batch)
2016-08-26 21:24:31.775519: step 90, loss = 4.18 (377.8 examples/sec; 0.339 sec/batch)

Issue 2:
Environment info

Operating System:

Linux 4.4.11-23.53

Observations

I am testing on a Tesla K80 (details below) using the following lines:

nvidia-docker run --rm -it -v /tmp/cifar10_data:/tmp/cifar10_data tensorflow/tensorflow:0.9.0-devel-gpu bash -c 'ln -s /usr/local/nvidia/lib64/libcuda.so.1 /usr/lib/x86_64-linux-gnu/libcuda.so && python /usr/local/lib/python2.7/dist-packages/tensorflow/models/image/cifar10/cifar10_multi_gpu_train.py'

nvidia-docker run --rm -it -v /tmp/cifar10_data:/tmp/cifar10_data tensorflow/tensorflow:0.10.0rc0-devel-gpu python /usr/local/lib/python2.7/dist-packages/tensorflow/models/image/cifar10/cifar10_multi_gpu_train.py

On r0.0.9, I get: ~575 examples/sec.

On r0.10.0rc0 I get ~425 examples/sec.

Here are the device stats:

major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:04.0
Total memory: 11.17GiB
Free memory: 11.11GiB

I have the: Nvidia Driver Version: 367.35