Issue 1:
I'm running my ResNet-32 benchmark model. My code looks quite similar with the code in TensorFlow GitHub:

https://github.com/tensorflow/tensorflow/tree/master/tensorflow/models/image/cifar10

but the model is changed into my own ResNet-32 implementation.

I'm measuring how long it takes to train a minibatch with size 128, and here are brief results. I used one GPU, 64bit Ubuntu 14.04 and Python 2.7 for all experiments.

TensorFlow Version
CUDA/cuDNN Version
GPU
Elapsed Time (ms)

0.9.0 (pip installed)
7.5 / 4.0.7
GTX Titan X
75

0.9.0 (745f16f)
8.0 RC / 5.0.5
GTX 1080
330

0.8.0 (ea9e00a)
8.0 RC / 5.0.5
GTX 1080
60

The 0.8.0 version commit contains the first support for CUDA 8.0 RC as far as I know and I found this to use my GTX 1080 without performance drop. Now I can use it with reasonable performance, but I want to figure out why the 0.9.0 compiled version is 4 times slower than before.

Issue 2:
Nightly built, python2 gpu, cuda 7.5, cudnn 4.0.7, archlinux. TitanX.

I run batch_norm_benchmark.py and got the following output:

Forward convolution (lower layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.043865 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.053864 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.088060 secs
=== op vs py: 22.8% ===
=== py vs slow: 63.5% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.287913 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.284179 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.287409 secs
=== op vs py: -1.3% ===
=== py vs slow: 1.1% ===
Forward/backward convolution (lower layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:True - 0.220112 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.201284 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.295103 secs
=== op vs py: -8.6% ===
=== py vs slow: 46.6% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:True - 2.108785 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.578407 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.863753 secs
=== op vs py: -59.0% ===
=== py vs slow: -96.6% ===
Forward convolution (higher layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.025443 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.033344 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.048162 secs
=== op vs py: 31.1% ===
=== py vs slow: 44.4% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.164241 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.161473 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.163212 secs
=== op vs py: -1.7% ===
=== py vs slow: 1.1% ===
Forward/backward convolution (higher layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:True - 0.111931 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.118556 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.163289 secs
=== op vs py: 5.9% ===
=== py vs slow: 37.7% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:True - 1.194288 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.329403 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.491005 secs
=== op vs py: -72.4% ===
=== py vs slow: 49.1% ===
Forward fully-connected.
cpu shape:2/1 #layers:10 mode:py scale:True train:False - 0.001866 secs
cpu shape:2/1 #layers:10 mode:slow scale:True train:False - 0.002859 secs
=== py vs slow: 53.2% ===
gpu shape:2/1 #layers:10 mode:py scale:True train:False - 0.002420 secs
gpu shape:2/1 #layers:10 mode:slow scale:True train:False - 0.002523 secs
=== py vs slow: 4.3% ===
Forward/backward fully-connected.
cpu shape:2/1 #layers:10 mode:py scale:True train:True - 0.004973 secs
cpu shape:2/1 #layers:10 mode:slow scale:True train:True - 0.006781 secs
=== py vs slow: 36.4% ===
gpu shape:2/1 #layers:10 mode:py scale:True train:True - 0.006370 secs
gpu shape:2/1 #layers:10 mode:slow scale:True train:True - 0.008182 secs
=== py vs slow: 28.4% ===

Although sess.run in a loop might not be an accurate way to benchmark, I think the performance regression exists because running the same script with an earlier binary built (not sure which commit it is) is 10x-20x faster.