Issue 1:
I use the following script to remove the 2 inputs from a siamese network and the output (a cosine distance layer).

#!/usr/bin/env python

from dir.model import load_siamese_network
import sys

def load_siamese_network(filepath, include_distance=True):
custom_objects = {
'contrastive_loss': contrastive_loss
}
model = load_model(filepath, custom_objects=custom_objects)
if not include_distance:
# layer 0 and 1 are inputs
tuned_model = model.layers[2]
return tuned_model
return model

src = sys.argv[1]
dst = sys.argv[2]
model = load_siamese_network(src, include_distance=False)

model.save(dst)

Here's the code I use to generate the siamese network from a base network: https://gist.github.com/aeec5b2a6efb1090375d4211aabbed6f

It appears to be working as expected but I was surprised to see that the saved file is twice smaller than the original one (90MB instead of 180MB). How could removing 2 inputs and a distance layer with no weights so dramatically reduce the file size?

Since this is a siamese network, both inputs share the same network + weights so removing an input shouldn't really affect the file size that much. That being said, it's possible that Keras isn't that smart about saving the model to disk and is duplicating the network weights for each input although they really are shared.

Is this what is happening? Or did I setup the siamese network wrong?

Issue 2:
System information

Have I written custom code (as opposed to using example directory):

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS Linux release 7.2.1511 (Core)

TensorFlow backend (yes / no): YES

TensorFlow version: v1.12.3-0-g41e0a4f56c 1.12.3

Keras version: 2.1.6-tf

Python version: python 3.6.5

CUDA/cuDNN version: 9.0.176/7.6.3.30

GPU model and memory: Tesla P100-PCIE 16GB x4

My model is shown below, with many BN layers.

def DnCNN(depth, filters=64, image_channels=1, use_bnorm=True):
layer_count = 0
inpt = Input(shape=(None, None, image_channels), name='input' + str(layer_count))
# 1st layer, Conv+relu
layer_count += 1
x = Conv2D(filters=filters, kernel_size=(3, 3), strides=(1, 1), kernel_initializer='Orthogonal', padding='same',
name='conv' + str(layer_count))(inpt)
layer_count += 1
x = Activation('relu', name='relu' + str(layer_count))(x)
# depth-2 layers, Conv+BN+relu
for i in range(depth - 2):
layer_count += 1
x = Conv2D(filters=filters, kernel_size=(3, 3), strides=(1, 1), kernel_initializer='Orthogonal', padding='same',
use_bias=False, name='conv' + str(layer_count))(x)
if use_bnorm:
layer_count += 1
# x = BatchNormalization(axis=3, momentum=0.1,epsilon=0.0001, name = 'bn'+str(layer_count))(x)
x = BatchNormalization(axis=3, momentum=0.0, epsilon=0.0001, name='bn' + str(layer_count))(x)
layer_count += 1
x = Activation('relu', name='relu' + str(layer_count))(x)
# last layer, Conv
layer_count += 1
x = Conv2D(filters=image_channels, kernel_size=(3, 3), strides=(1, 1), kernel_initializer='Orthogonal',
padding='same', use_bias=False, name='conv' + str(layer_count))(x)
layer_count += 1
x = Subtract(name='subtract' + str(layer_count))([inpt, x]) # input - noise
model = Model(inputs=inpt, outputs=x)

return model

My test set is (128 * n, 40, 40, 1), batch_size = 128

It's normal for me to run my code on a single GPU

However, when I use multiple GPUs（multi_gpu_num）, the weight of the BN layer becomes NAN.

Strangely, this seems to be related only to GPU_NUM :

When GPU_NUM = 2, the weights become Nan at the 129th step of training,

When GPU_NUM = 4, the weights become Nan at the 65th step of training,

Whether I change the batch_size to 64, 128, or 512, the above behaviors still exist

code:

if __name__ == '__main__':
# model selection

# with tf.device('/cpu:0'):
# model = DnCNN(depth=17, filters=64, image_channels=1, use_bnorm=True)
model = DnCNN(depth=17, filters=64, image_channels=1, use_bnorm=True)
model.summary()

# for reproducibility
np.random.seed(seed=10)

# load the last model in matconvnet style
initial_epoch = findLastCheckpoint(save_dir=save_dir)
if initial_epoch > 0:
print('resuming by loading epoch %03d' % initial_epoch)
model = load_model(os.path.join(save_dir, 'model_%03d.hdf5' % initial_epoch), compile=False)

# compile the model
gpu_num = 2
print("[INFO] training with {} GPUs...".format(gpu_num))
# model = DnCNN(depth=17, filters=64, image_channels=1, use_bnorm=True)
parallel_model = multi_gpu_model(model, gpus=gpu_num)
# model.compile(optimizer=Adam(0.001), loss=sum_squared_error)
parallel_model.compile(optimizer=Adam(0.001), loss=sum_squared_error)

# use call back functions
args.train_data = r'data/Train400' #train set
args.batch_size = 512
args.epoch = 2
checkpointer = ModelCheckpoint(os.path.join(save_dir, 'model_{epoch:03d}.hdf5'),
verbose=1, save_weights_only=False, period=args.save_every)
csv_logger = CSVLogger(os.path.join(save_dir, 'log.csv'), append=True, separator=',')
lr_scheduler = LearningRateScheduler(lr_schedule)
data = train_datagen(batch_size=args.batch_size, data_dir=args.train_data)
# history = model.fit_generator(data,
# steps_per_epoch=2000, epochs=args.epoch, verbose=1, initial_epoch=initial_epoch,
# callbacks=[checkpointer, csv_logger, lr_scheduler])
history = parallel_model.fit_generator(data,
steps_per_epoch=2000,epochs=args.epoch, verbose=1, initial_epoch=initial_epoch,
callbacks=[csv_logger, lr_scheduler])
model.save('test.h5')
parallel_model.save('testpa.h5')

How can I solve this problem？