Issue 1:
When batchnorm is used inside a model that is invoked more than once (e.g inside an auto regression step) it produces very different results during prediction (non train time). I have isolated a toy problem and the suspected reason is that the updated values are accumulated across all invocations and then updated using K.moving_average_update. In the example bellow we use momentum 0.5 on the means 2, 4, 8 then we should get 6.5 = ((0.5 * 0 + 0.5 * 2) * 0.5 + 0.5 * 4) + 0.5 * 8 or 2.33 = 0.5 * 0 +0.5* (2 + 4 + 8) / 3 but we have (with tensorflow)

7 = 0.5 * 0 + 0.5 * (2 + 4 + 8) also if we use momentum=0.1 for example the output is 12.666 = 0.1 * + 0.9 * (2 + 4 + 8)

With theano, only the first invocation is considered

import numpy as np
from keras.models import Model
from keras.layers import Layer, Input, Dense
from keras import initializers
import keras.backend as K

class CollectAvg(Layer):
def build(self, input_shape):
self.moving_mean = self.add_weight(shape=(input_shape[1], ), name='moving_mean',
initializer=initializers.get('zeros'),
trainable=False)
self.built = True

def call(self, inputs):
mean = K.mean(inputs)
self.add_update([K.moving_average_update(self.moving_mean, mean, 0.5)], inputs)
return inputs

def get_shared_model():
x = shared_input = Input(shape=(1, ))

w = np.array([2]).reshape((1, 1, 1))
x = Dense(1, use_bias=False, weights = w)(x)

x = CollectAvg()(x)

shared_model = Model(inputs=[shared_input], outputs=[x])
return shared_model

def get_auto_regressive(submodel, n):
x = model_input = Input(shape=(1, ))
outputs = []
for i in range(n):
x = submodel(x)
outputs.append(x)

model = Model(inputs=[model_input], outputs = outputs)
model.compile(optimizer='sgd', loss='mse')
return model

submodel = get_shared_model()
submodel.summary()
model = get_auto_regressive(submodel, 3)
model.summary()
sample_input = np.asarray([[1]])
for i in range(2):
sample_output = model.predict(sample_input)
model.train_on_batch(sample_input, sample_output)
print ("collected avg:", submodel.layers[-1].get_weights())
print ("outputs:", np.array(sample_output))

with output being

Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) (None, 1) 0
_________________________________________________________________
dense_1 (Dense) (None, 1) 1
_________________________________________________________________
collect_avg_1 (CollectAvg) (None, 1) 1
=================================================================
Total params: 2
Trainable params: 1
Non-trainable params: 1
_________________________________________________________________
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_2 (InputLayer) (None, 1) 0
_________________________________________________________________
model_1 (Model) (None, 1) 2
=================================================================
Total params: 2
Trainable params: 1
Non-trainable params: 1
_________________________________________________________________

for theano:

collected avg: [array([ 1.], dtype=float32)]
collected avg: [array([ 1.5], dtype=float32)]
outputs: [[[ 2.]]

[[ 4.]]

[[ 8.]]]

for tensorflow

collected avg: [array([ 7.], dtype=float32)]
collected avg: [array([ 3.5], dtype=float32)]
outputs: [[[ 2.]]

[[ 4.]]

[[ 8.]]]

Issue 2:
Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.

Thank you!

Check that you are up-to-date with the master branch of Keras. You can update with:

pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:

pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

This is a duplicate of: #7365.

Bidirectional.call(self, inputs, training=None, mask=None): does not accept an initial_state keyword argument.

Furthermore, invoking Bidirectional.call with a list of inputs doesn't allow you to specify the initial state either, since Recurrent.call only has this code to handle list inputs:

if isinstance(inputs, list):
inputs = inputs[0]

I would fix this myself but I'm not sure what the right solution is. My guess is to switch Bidirectional to use self.forward_layer.__call__ instead of self.forward_layer.call.