Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): 20.04

I have implemented a classification model , which has Batchnormalization layers into it. While training the model , i used tf.keras.layers.BatchNormalization layer for training. Then manually extracted the bn weights (gamma, beta, mean , var) from checkpoint file and called the bn using tf.nn.batch_normalization layer using above statistics for inference. I observed a drop in testing accuracy (~75%) as compared to the scenario where i load the weights in the model using keras itself model.load_weights(chkpnt_file) (~92%).

Though if i replace the tf.nn.batch_normalization call to keras bn layer call by calling set_weights function , both the cases gived 0.0 diff, which apprently led me to believe that there is diff in both the calls while the input and bn_weights are exactly same. epsilon is also same.

Standalone code to reproduce the issue

Please refer to this colab notebook(https://colab.research.google.com/drive/1nysYyCR5Ay_jkPlh1WD9A1Mf01eVvhMu? usp=sharing) for a small example. Here the error is coming to be a margin of 10^-3 which theoretically shouldn't be there.

Issue 2:
Hi Everyone,

It seems TimeDistributed wrapper forces a single shape on data unlike Recurrent layers. To reuse the Dense layer you need to create 2 TimeDistributed layers that reference the same Dense layer, otherwise you get a shape error. I am not entirely sure this behavior is desired since it duplicates the weights in the network. Here is a minimal example that should illustrate this behavior:

#Breaks
import keras.layers as L

shared=L.TimeDistributed(L.Dense(1))

inp1=L.Input((2,1))
inp2=L.Input((3,1))

hidden1=shared(inp1)
hidden2=shared(inp2)

Here is the error:

ValueError: Input 0 is incompatible with layer time_distributed_19: expected shape=(None, 2, 1), found shape=(None, 3, 1)

As you can see Keras 'remembered the shape of the inp1 and threw an error when found mismatch

This on the other hand works fine:

#Works but duplicates the share layer weights and unnecessarily consumes memory:
shared=(L.Dense(1))

inp1=L.Input((2,1))
inp2=L.Input((3,1))

hidden1=L.TimeDistributed(shared)(inp1)
hidden2=L.TimeDistributed(shared)(inp2)

Now the same logic but for the Recurrent layer works without any issue even with the wrapper :

#Works without layer duplication
shared=L.Bidirectional(L.GRU(1))

inp1=L.Input((2,1))
inp2=L.Input((3,1))

hidden1=shared(inp1)
hidden2=shared(inp2)

Please let me know your thoughts or if I should provide more detail.

Thanks,

Tim

[ x] Check that you are up-to-date with the master branch of Keras. You can update with:

pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

[ x] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

[x ] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:

pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

[x ] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).