Repository: keras-team/keras

Issue #11927: Batchnorm in shared layers goes to nan
Created: 2018-12-24T11:52:50Z
Hi,
There seems to be a bug with Batch Normalization layer when using it for shared layers. I traced the problem to the running mean growing uncontrollably and then going to nan. It surfaced in my kind of a triplet loss model. See code attached to reproduce. Several other issues have this problem:
#11897
#9646
import numpy as np

import keras

import keras.backend as K

from keras.optimizers import Adam
from keras.applications import ResNet50
from keras.applications.resnet50 import preprocess_input as preprocess_resnet50
from keras.models import Model, Input

from keras.layers import Input,Lambda,subtract,GlobalMaxPooling2D,Dense,GlobalAveragePooling2D,concatenate,Activation
from keras.layers import concatenate, Lambda, Input, Dense, Dropout, Flatten, Conv2D, MaxPooling2D, \
        BatchNormalization, Activation, GlobalAveragePooling2D,GlobalMaxPooling2D, SeparableConv2D,DepthwiseConv2D

def get_keras_model(model, input_shape):   
    
    i = Input(shape=input_shape)
    x=model.output
    x = GlobalMaxPooling2D()(x)
    x=BatchNormalization()(x)
    x=Dense(512,kernel_initializer='he_normal')(x)
    x=Activation('relu')(x)
    x=BatchNormalization()(x)
    o=Dense(5005,kernel_initializer='he_normal', activation='softmax')(x)
    return Model(inputs=model.input, outputs=o)

def get_resnet50(input_shape=(192,256,3)):
    K.clear_session()
    resnet=ResNet50(include_top=False, weights='imagenet', input_tensor=None, input_shape=input_shape, pooling=None)

    for l in range(0,len(resnet.layers)):
        if type(resnet.layers[l])==keras.layers.normalization.BatchNormalization:
            resnet.layers[l].trainable=True
        else:
            resnet.layers[l].trainable=False
    return resnet


resnet=ResNet50(include_top=False, weights='imagenet', input_tensor=None, input_shape=(192,256,3), pooling=None)
model=get_keras_model(get_resnet50(input_shape=(192,256,3)),input_shape=(192,256,3))
#model.load_weights(path+'models/Resnet50/loss_{:.4f}_{:.4f}_{:02d}.hdf5'.format(4.9773,0.6173,1))
trimmed_resnet50=Model(inputs=model.input,outputs=model.layers[173].output)
x=trimmed_resnet50.output
x = GlobalAveragePooling2D()(x)
x=BatchNormalization()(x)
x=Dense(1024,kernel_initializer='he_normal')(x)
x=Activation('relu')(x)
x=BatchNormalization()(x)
o=Dense(128,kernel_initializer='he_normal')(x)
base_net=Model(inputs=model.input,outputs=o)
#base_net.summary()


def create_model(base_model, input_shape=(192,256,3)):
    input_tensor1 = Input(shape=input_shape)
    input_tensor2 = Input(shape=input_shape)
    input_tensor3 = Input(shape=input_shape)

    reduce_sum = Dense(1,activation='linear',kernel_initializer='ones',bias_initializer='zeros',name='reduce_sum')

    x1 = base_model(input_tensor1)
    x2 = base_model(input_tensor2)
    x3 = base_model(input_tensor3)

    d12 = subtract([x1,x2])
    d13 = subtract([x1,x3])

    d12 = Lambda(lambda val: (val)**2)(d12)
    d13 = Lambda(lambda val: (val)**2)(d13)

    d12 = reduce_sum(d12)
    d13 = reduce_sum(d13)
    
    d12 = Lambda(lambda val: keras.backend.sqrt(val+K.epsilon()))(d12)
    d13 = Lambda(lambda val: keras.backend.sqrt(val+K.epsilon()))(d13)
    
    d = concatenate([d12,d13])
    d = Activation('softmax')(d)

    model = Model(inputs=[input_tensor1,input_tensor2,input_tensor3], outputs=d)
    metric = Model(inputs=input_tensor1, outputs=x1)


    for l in model.layers:
        if l.name == 'reduce_sum':
            print('reduce sum')
            l.trainable=False
    return model,metric
    
triplet_model,metric = create_model(base_net)

train_1=np.random.uniform(-128,128,size=(1000,192,256,3))
train_2=np.random.uniform(-128,128,size=(1000,192,256,3))
train_3=np.random.uniform(-128,128,size=(1000,192,256,3))
train_y=np.array([1,0]*1000).reshape(1000,2)

triplet_model.compile(loss='categorical_crossentropy',optimizer=Adam(1e-5, clipnorm=1.0),metrics=['accuracy'])
triplet_model.fit([train_1,train_2,train_3], train_y,batch_size=16,epochs=5)

intermediate=Model(inputs=metric.layers[1].get_input_at(0), outputs=metric.layers[1].layers[3].output)
output=intermediate.predict(train_1[0:1])
print(metric.layers[1].layers[3].name)
print(output.shape, output)

print(intermediate.layers[3].weights[2])
print(K.eval(intermediate.layers[3].weights[2]))


I'm trying to train a kind-of triplet loss CNN which has three shared Resnet50 models.
Please make sure that the boxes below are checked before you submit your issue.
If your issue is an implementation question, please ask your question on StackOverflow or on the Keras Slack channel instead of opening a GitHub issue.
Thank you!


[ X] Check that you are up-to-date with the master branch of Keras. You can update with:
pip install git+git://github.com/keras-team/keras.git --upgrade --no-deps


[ X] Check that your version of TensorFlow is up-to-date. The installation instructions can be found here.


[ X] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

Duplicate issue #15210: Tutorial model consistently contains NaN
Created: 2021-08-20T05:28:41Z
System information.

Have I written custom code (as opposed to using a stock example script provided in Keras): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Colab
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below):  2.6.0
Python version: 3.7.11
GPU model and memory:  Tesla T4 15GB
Exact command to reproduce: See underneeth

Describe the problem.
Model works, but contains a NaN weight after training.
Describe the current behavior.
After training, model contains NaN.
Describe the expected behavior.
It shouldn't contain NaNs.
Standalone code to reproduce the issue.
Relevant issue: tensorflow/tfjs#5374
(originally thought to be a tfjs bug, now more likely a keras bug)

I went to https://www.tensorflow.org/tutorials/generative/pix2pix
Ran it in colab, as is
Inspect the weights:

generator.weights[35]
batch_normalization_8/moving_variance:0 <tf.Variable 'batch_normalization_8/moving_variance:0' shape=(512,) dtype=float32, numpy=
array([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan], dtype=float32)>

Closing comment (2021-08-24T22:41:06Z): possibly related to #11927