Issue 1:
Recently introduced batch renormalization could be very useful, but without support for a fused batch norm, it has limited use (XLA JIT still has too many problems to be used instead of a fused version)

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

MacOSX 10.13.6

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

No

TensorFlow installed from (source or binary):

Binary

TensorFlow version (use command below):

tf.version.VERSION=2.0.0-dev20190404

tf.version.GIT_VERSION=v1.12.0-11729-g98c3cfbf74

Python version:

3.6.8

Bazel version (if compiling from source):

N/A

GCC/Compiler version (if compiling from source):

N/A

CUDA/cuDNN version:

N/A

GPU model and memory:

N/A

Describe the current behavior

When using a + b then keras.layers.Add in the same Keras model, I get an exception saying the name "add" was used twice. The problem goes away if I use keras.layers.Add first, or if I use twice the same operation (either + twice, or keras.layers.Add twice).

Describe the expected behavior

I don't expect any name conflicts.

Code to reproduce the issue

The following code raises a ValueError exception (full stacktrace below).

from tensorflow import keras

inputs = keras.layers.Input(shape=[2])
add1 = inputs + inputs
add2 = keras.layers.Add()([inputs, inputs])
model = keras.models.Model(inputs=[inputs], outputs=[add1, add2])

Other info / logs

Here is the full stacktrace:

ValueError Traceback (most recent call last)
<ipython-input-1-07b5faf0201d> in <module>
4 add1 = inputs + inputs
5 add2 = keras.layers.Add()([inputs, inputs])
----> 6 model = keras.models.Model(inputs=[inputs], outputs=[add1, add2])

.../tensorflow/python/keras/engine/training.py in __init__(self, *args, **kwargs)
121
122 def __init__(self, *args, **kwargs):
--> 123 super(Model, self).__init__(*args, **kwargs)
124 # initializing _distribution_strategy here since it is possible to call
125 # predict on a model without compiling it.

.../tensorflow/python/keras/engine/network.py in __init__(self, *args, **kwargs)
137 'inputs' in kwargs and 'outputs' in kwargs):
138 # Graph network
--> 139 self._init_graph_network(*args, **kwargs)
140 else:
141 # Subclassed network

.../tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
456 self._setattr_tracking = False # pylint: disable=protected-access
457 try:
--> 458 result = method(self, *args, **kwargs)
459 finally:
460 self._setattr_tracking = previous_value # pylint: disable=protected-access

.../tensorflow/python/keras/engine/network.py in _init_graph_network(self, inputs, outputs, name)
284 # Keep track of the network's nodes and layers.
285 nodes, nodes_by_depth, layers, layers_by_depth = _map_graph_network(
--> 286 self.inputs, self.outputs)
287 self._network_nodes = nodes
288 self._nodes_by_depth = nodes_by_depth

.../tensorflow/python/keras/engine/network.py in _map_graph_network(inputs, outputs)
1916 if all_names.count(name) != 1:
1917 raise ValueError('The name "' + name + '" is used ' +
-> 1918 str(all_names.count(name)) + ' times in the model. '
1919 'All layer names should be unique.')
1920 return network_nodes, nodes_by_depth, layers, layers_by_depth

ValueError: The name "add" is used 2 times in the model. All layer names should be unique.