Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Archlinux

TensorFlow installed from (source or binary): binary

TensorFlow version (use command below): v1.12.1-3374-g9eb67b17bf 2.0.0-dev20190605

Python version: 3.6

CUDA/cuDNN version: 10

GPU model and memory: 1080 Ti

Describe the current behavior

I expect to do a forward pass with a model with a BachNormalization layer in training mode, when using the tf.distribuite.MirroredStrategy but I can't, because it reises the following exception:

RuntimeError: add_update was called in a cross-replica context. This is not expected. If you require this feature, please file an issue.

Why it is not expected?

Describe the expected behavior

It should work.

The commit that introduced this behavior is: 316cd57#diff-8eb7e20502209f082d0cb15119a50413

Code to reproduce the issue

import tensorflow as tf

model = tf.keras.Sequential(
[
tf.keras.layers.Dense(10),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Dense(1),
]
)

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
out = model(tf.zeros((1, 10)), training=True)

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

$ uname -mrs
Linux 4.14.0-49.el7a.ppc64le ppc64le

TensorFlow installed from (source or binary): Binary

TensorFlow version (use command below): 1.6.0

Python version: 2.7.5

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version: CUDA 9.1, CuDNN 7.0.5

GPU model and memory: Tesla V100:

== nvidia-smi ===================================================
Thu Apr 26 18:35:34 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.19 Driver Version: 396.19 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... Off | 00000004:04:00.0 Off | 0 |
| N/A 29C P0 52W / 300W | 0MiB / 15360MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+

Exact command to reproduce: N/A

We would like to be able to have access to the CuDNN autotuner in tf.keras module to access optimal algorithms for a given hardware (or, perhaps passing a custom convolutional algorithm from config). In TensorFlow, I can specify to use the CuDNN autotuner by setting:

os.environ['TF_CUDNN_USE_AUTOTUNE'] = "1"

(currently enabled by default), which improves performance significantly on Volta GPUs and especially with FP16.

However, I am unable to access this performance improvement when running pure tf.keras, where setting this environmental variable does not have any effect.

Source code / logs

Following simple example could be used to reproduce the issue: https://gist.github.com/ASvyatkovskiy/8d1dd622e447d9d8de1ec4e238e0dbaa