Issue 1:
Nightly built, python2 gpu, cuda 7.5, cudnn 4.0.7, archlinux. TitanX.

I run batch_norm_benchmark.py and got the following output:

Forward convolution (lower layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.043865 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.053864 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.088060 secs
=== op vs py: 22.8% ===
=== py vs slow: 63.5% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.287913 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.284179 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.287409 secs
=== op vs py: -1.3% ===
=== py vs slow: 1.1% ===
Forward/backward convolution (lower layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:True - 0.220112 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.201284 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.295103 secs
=== op vs py: -8.6% ===
=== py vs slow: 46.6% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:True - 2.108785 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.578407 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.863753 secs
=== op vs py: -59.0% ===
=== py vs slow: -96.6% ===
Forward convolution (higher layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.025443 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.033344 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.048162 secs
=== op vs py: 31.1% ===
=== py vs slow: 44.4% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.164241 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.161473 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.163212 secs
=== op vs py: -1.7% ===
=== py vs slow: 1.1% ===
Forward/backward convolution (higher layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:True - 0.111931 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.118556 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.163289 secs
=== op vs py: 5.9% ===
=== py vs slow: 37.7% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:True - 1.194288 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.329403 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.491005 secs
=== op vs py: -72.4% ===
=== py vs slow: 49.1% ===
Forward fully-connected.
cpu shape:2/1 #layers:10 mode:py scale:True train:False - 0.001866 secs
cpu shape:2/1 #layers:10 mode:slow scale:True train:False - 0.002859 secs
=== py vs slow: 53.2% ===
gpu shape:2/1 #layers:10 mode:py scale:True train:False - 0.002420 secs
gpu shape:2/1 #layers:10 mode:slow scale:True train:False - 0.002523 secs
=== py vs slow: 4.3% ===
Forward/backward fully-connected.
cpu shape:2/1 #layers:10 mode:py scale:True train:True - 0.004973 secs
cpu shape:2/1 #layers:10 mode:slow scale:True train:True - 0.006781 secs
=== py vs slow: 36.4% ===
gpu shape:2/1 #layers:10 mode:py scale:True train:True - 0.006370 secs
gpu shape:2/1 #layers:10 mode:slow scale:True train:True - 0.008182 secs
=== py vs slow: 28.4% ===

Although sess.run in a loop might not be an accurate way to benchmark, I think the performance regression exists because running the same script with an earlier binary built (not sure which commit it is) is 10x-20x faster.

Issue 2:
This issue is a follow-up to this posting on stack overflow. As noted, the 0.10 version is ~3x slower than version 0.9 for a particular training script I am using. The same slow-down is observed when using the pip version with CuDNN4 or with a version compiled from source and CuDNN 5.1.

I'm not sure what information about the training script would help. It is a fairly complex script that is training a network for object detection using methods similar to YOLO and SSD. It won't be possible to post the entire set of code, but if there is a way to isolate the source of the delay I may be able to modify it and post that portion.

While trying to get insight into the source of the slow-down I also noted that I wasn't able to compile from source using CuDNN4 as noted in the SO posting. This is a secondary issue, but may also warrant investigation.