Repository: keras-team/keras

Issue #11104: Cannot save optimizer weights due to h5 error "object header message is too large"
Created: 2018-09-07T08:36:58Z
When trying to save my model I get the runtime error below. There was a similar issue when model layers names were too long and it can be solved by giving layers shorter names. This time the error pops up when saving optimizer weights. getattr(model.optimizer,'weights') shows
[<tf.Variable 'Adam/iterations:0' shape=() dtype=int64_ref>,
 <tf.Variable 'training/Adam/Variable:0' shape=(3, 3, 1, 64) dtype=float32_ref>,
 <tf.Variable 'training/Adam/Variable_1:0' shape=(64,) dtype=float32_ref>,
 <tf.Variable 'training/Adam/Variable_2:0' shape=(64,) dtype=float32_ref>,
...]


and if I convert it to numpy array its length is above the 64k limits which gives h5 runtime. I can save the model if I use save_model(....,include_optimizer=False) but I need the optimizer state. Is there any way I can reduce the length of "training/Adam/Variable:0"... names so as to fit them into 64k hdf5 table limit. Thanks.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-130-d231b4a5a40c> in <module>()
----> 1 model.save('model')

C:\Anaconda3\lib\site-packages\keras\engine\network.py in save(self, filepath, overwrite, include_optimizer)
   1083             raise NotImplementedError
   1084         from ..models import save_model
-> 1085         save_model(self, filepath, overwrite, include_optimizer)
   1086 
   1087     def save_weights(self, filepath, overwrite=True):

C:\Anaconda3\lib\site-packages\keras\engine\saving.py in save_model(model, filepath, overwrite, include_optimizer)
    173                     #print('Weight names',weight_names,len(weight_names),np.asarray(weight_names).nbytes)
    174                     optimizer_weights_group.attrs[
--> 175                         'weight_names'] = weight_names
    176                     for name, val in zip(weight_names, weight_values):
    177                         param_dset = optimizer_weights_group.create_dataset(

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

C:\Anaconda3\lib\site-packages\h5py\_hl\attrs.py in __setitem__(self, name, value)
     93         use the methods create() and modify().
     94         """
---> 95         self.create(name, data=value, dtype=base.guess_dtype(value))
     96 
     97     @with_phil

C:\Anaconda3\lib\site-packages\h5py\_hl\attrs.py in create(self, name, data, shape, dtype)
    186 
    187             try:
--> 188                 attr = h5a.create(self._id, self._e(tempname), htype, space)
    189             except:
    190                 raise

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

h5py\h5a.pyx in h5py.h5a.create()

RuntimeError: Unable to create attribute (object header message is too large)

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.
Thank you!


 Check that you are up-to-date with the master branch of Keras. You can update with:
pip install git+git://github.com/keras-team/keras.git --upgrade --no-deps


 If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.


 If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps


 Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

Duplicate issue #11200: Crash in ModelCheckpoint callback
Created: 2018-09-22T16:41:11Z
python3 train_classes_v2.py --train 1/train/ --val 1/val/ --all 1/all/
Using TensorFlow backend.
2018-09-22 10:35:27.903598: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled
to use: AVX2 FMA
2018-09-22 10:35:28.554300: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but the
re must be at least one NUMA node, so returning NUMA node zero
2018-09-22 10:35:28.554868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:01:00.0
totalMemory: 7.92GiB freeMemory: 7.80GiB
2018-09-22 10:35:28.554883: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-09-22 10:35:28.731514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-22 10:35:28.731544: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-09-22 10:35:28.731550: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-09-22 10:35:28.731704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0
with 7537 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
Found 6400 images belonging to 32 classes.
Found 1306 images belonging to 32 classes.
Found 6522 images belonging to 32 classes.
{'SKU49862': 15, 'SKU49871': 21, 'SKU49844': 3, 'SKU49845': 4, 'SKU49882': 22, 'SKU49981': 25, 'SKU49861': 14, 'SKU49985': 29, 'SKU49982': 26, 'SKU49883': 23
, 'SKU49865': 17, 'SKU49851': 10, 'SKU49984': 28, 'SKU49850': 9, 'SKU49843': 2, 'SKU49987': 31, 'SKU49849': 8, 'SKU49847': 6, 'SKU49855': 11, 'SKU49846': 5,
'SKU49860': 13, 'SKU49983': 27, 'SKU49864': 16, 'SKU49842': 1, 'SKU49866': 18, 'SKU49841': 0, 'SKU49986': 30, 'SKU49884': 24, 'SKU49867': 19, 'SKU49848': 7,
'SKU49868': 20, 'SKU49859': 12}
steps_per_epoch: 6400.0
validation_steps: 1306.0
all_steps: 6522.0
Downloading data from https://github.com/titu1994/Keras-NASNet/releases/download/v1.2/NASNet-mobile-no-top.h5
19996672/19993432 [==============================] - 5s 0us/step
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
NASNet (Model)               (None, 21, 7, 1056)       4269716
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 21, 7, 32)         33824
_________________________________________________________________
global_average_pooling2d_1 ( (None, 32)                0
_________________________________________________________________
softmax_1 (Softmax)          (None, 32)                0
=================================================================
Total params: 4,303,540
Trainable params: 33,824
Non-trainable params: 4,269,716
_________________________________________________________________
None
Epoch 1/1
6400/6400 [==============================] - 181s 28ms/step - loss: 3.4429 - acc: 0.0819 - val_loss: 3.4432 - val_acc: 0.0704
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
NASNet (Model)               (None, 21, 7, 1056)       4269716
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 21, 7, 32)         33824
_________________________________________________________________
global_average_pooling2d_1 ( (None, 32)                0
_________________________________________________________________
softmax_1 (Softmax)          (None, 32)                0
=================================================================
Total params: 4,303,540
Trainable params: 4,266,802
Non-trainable params: 36,738
_________________________________________________________________
None
Epoch 2/151
6400/6400 [==============================] - 1308s 204ms/step - loss: 2.1879 - acc: 0.6333 - val_loss: 3.3195 - val_acc: 0.4556
Epoch 3/151
6400/6400 [==============================] - 1265s 198ms/step - loss: 0.4105 - acc: 0.8980 - val_loss: 2.2126 - val_acc: 0.5383
Epoch 4/151
6400/6400 [==============================] - 1268s 198ms/step - loss: 0.1253 - acc: 0.9513 - val_loss: 1.4821 - val_acc: 0.6715
Epoch 5/151
6400/6400 [==============================] - 1270s 198ms/step - loss: 0.0915 - acc: 0.9677 - val_loss: 1.3620 - val_acc: 0.6715
Epoch 6/151
6400/6400 [==============================] - 1271s 199ms/step - loss: 0.0861 - acc: 0.9783 - val_loss: 1.4910 - val_acc: 0.7021
Epoch 7/151
6400/6400 [==============================] - 1272s 199ms/step - loss: 0.0590 - acc: 0.9908 - val_loss: 1.3748 - val_acc: 0.7175
Epoch 8/151
6400/6400 [==============================] - 1272s 199ms/step - loss: 0.0343 - acc: 0.9961 - val_loss: 1.4008 - val_acc: 0.7060
Epoch 9/151
6400/6400 [==============================] - 1270s 198ms/step - loss: 0.0249 - acc: 0.9972 - val_loss: 1.4543 - val_acc: 0.7136
Epoch 10/151
6400/6400 [==============================] - 1271s 199ms/step - loss: 0.0108 - acc: 0.9987 - val_loss: 1.4282 - val_acc: 0.6983

Epoch 00010: saving model to epoch-010_loss-0.01085_acc-0.99875_val_loss-1.42822_val_acc-0.69832.h5
Traceback (most recent call last):
  File "train_classes_v2.py", line 338, in <module>
    _main(parser.parse_args())
  File "train_classes_v2.py", line 292, in _main
    use_multiprocessing=True,
  File "/home/michael/.local/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/michael/.local/lib/python3.5/site-packages/keras/engine/training.py", line 1415, in fit_generator
    initial_epoch=initial_epoch)
  File "/home/michael/.local/lib/python3.5/site-packages/keras/engine/training_generator.py", line 247, in fit_generator
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "/home/michael/.local/lib/python3.5/site-packages/keras/callbacks.py", line 77, in on_epoch_end
    callback.on_epoch_end(epoch, logs)
  File "/home/michael/.local/lib/python3.5/site-packages/keras/callbacks.py", line 455, in on_epoch_end
    self.model.save(filepath, overwrite=True)
  File "/home/michael/.local/lib/python3.5/site-packages/keras/engine/network.py", line 1085, in save
    save_model(self, filepath, overwrite, include_optimizer)
  File "/home/michael/.local/lib/python3.5/site-packages/keras/engine/saving.py", line 174, in save_model
    'weight_names'] = weight_names
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/michael/.local/lib/python3.5/site-packages/h5py/_hl/attrs.py", line 95, in __setitem__
    self.create(name, data=value, dtype=base.guess_dtype(value))
  File "/home/michael/.local/lib/python3.5/site-packages/h5py/_hl/attrs.py", line 188, in create
    attr = h5a.create(self._id, self._e(tempname), htype, space)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5a.pyx", line 47, in h5py.h5a.create
RuntimeError: Unable to create attribute (object header message is too large)


Code:
import os.path

import keras
from keras.applications import NASNetMobile
from keras.models import Sequential, model_from_json
from keras.layers import Conv2D, GlobalAveragePooling2D, Softmax, Dropout
from keras.callbacks import ModelCheckpoint, ReduceLROnPlateau
from keras.preprocessing.image import ImageDataGenerator

import tensorflow as tf

import json
import argparse
import glob
import cv2
import numpy as np
from sklearn.utils import class_weight

BATCH_SIZE = 1
IMG_HEIGHT = 224 * 3  # ResNetInceptionv2 & Xception: 299, ResNet50 & VGG: 224
IMG_WIDTH = 224
DIMS = (IMG_HEIGHT, IMG_WIDTH, 3)

HEAD_EPOCHES = 1 
FINETUNING_EPOCHES = 150
ALL_EPOCHES = 50


def save_model(model, name, classes):
    # модель
    json_string = model.to_json()
    open(name + '.model.json', 'w').write(json_string)

    # список классов
    json_string = json.dumps(classes, indent="\t")
    open(name + '.classes.json', 'w').write(json_string)

    # веса модели
    model.save_weights(name + ".weights.h5")


# set tf backend to allow memory to grow, instead of claiming everything
def get_session(cpu_only=False):

    num_cores = 4

    config = tf.ConfigProto()
    if cpu_only:
        config = tf.ConfigProto(
            intra_op_parallelism_threads=num_cores,
            inter_op_parallelism_threads=num_cores,
            allow_soft_placement=True,
            device_count={
                'CPU': 1,
                'GPU': 0
            })
    else:
        config = tf.ConfigProto(
            intra_op_parallelism_threads=num_cores,
            inter_op_parallelism_threads=num_cores,
            allow_soft_placement=True,
            device_count={
                'CPU': 1,
                'GPU': 1
            })
        config.gpu_options.allow_growth = True

    return tf.Session(config=config)


def _main(args):

    trainPath = args.train
    valPath = args.val
    allPath = args.all

    keras.backend.tensorflow_backend.set_session(get_session())

    train_datagen = ImageDataGenerator(
        rescale=1. / 255,
        rotation_range=8,
        width_shift_range=0.1,
        height_shift_range=0.1,
        shear_range=8,
        zoom_range=0.1,
        samplewise_center=True,
        samplewise_std_normalization=True,
        fill_mode="constant",
        cval=0,
    )

    val_datagen = ImageDataGenerator(
        rescale=1. / 255,
        samplewise_center=True,
        samplewise_std_normalization=True,
    )

    train_generator = train_datagen.flow_from_directory(
        trainPath,
        target_size=(IMG_HEIGHT, IMG_WIDTH),
        batch_size=BATCH_SIZE,
        class_mode='categorical',
        interpolation="bicubic",
        #save_to_dir='aug',
    )

    val_generator = val_datagen.flow_from_directory(
        valPath,
        target_size=(IMG_HEIGHT, IMG_WIDTH),
        batch_size=BATCH_SIZE,
        class_mode='categorical',
        interpolation="bicubic",
        #save_to_dir='aug2',
    )

    all_generator = train_datagen.flow_from_directory(
        allPath,
        target_size=(IMG_HEIGHT, IMG_WIDTH),
        batch_size=BATCH_SIZE,
        class_mode='categorical',
        interpolation="bicubic",
        #save_to_dir='aug3',
    )

    class_indices = train_generator.class_indices
    print(class_indices)

    num_of_classes = train_generator.num_classes 
    num_of_samples = train_generator.samples

    steps_per_epoch = num_of_samples / BATCH_SIZE
    validation_steps = val_generator.samples / BATCH_SIZE
    all_steps = all_generator.samples / BATCH_SIZE

    print('steps_per_epoch: ' + str(steps_per_epoch))
    print('validation_steps: ' + str(validation_steps))
    print('all_steps: ' + str(all_steps))

    class_weights = class_weight.compute_class_weight(
        'balanced', np.unique(train_generator.classes),
        train_generator.classes)

    class_weights_all = class_weight.compute_class_weight(
        'balanced', np.unique(all_generator.classes), all_generator.classes)

    callbacks = [
        ModelCheckpoint(
            filepath='epoch-{epoch:03d}_loss-{loss:.5f}_acc-{acc:.5f}_val_loss-{val_loss:.5f}_val_acc-{val_acc:.5f}.h5',
            verbose=1,
            period=10),
    ]

    base_model = NASNetMobile(include_top=False, input_shape=DIMS)
    base_model.trainable = False

    model = Sequential()
    model.add(base_model)
    model.add(Conv2D(num_of_classes, (1, 1), activation='relu'))
    model.add(GlobalAveragePooling2D())
    model.add(Softmax())

    for layer in base_model.layers:
        if hasattr(layer, 'kernel_regularizer'):
            layer.kernel_regularizer= keras.regularizers.l2(0.00001)

    model.compile(
        loss='categorical_crossentropy',
        optimizer=keras.optimizers.adam(lr=1e-5, clipnorm=0.001),
        metrics=['acc'])

    print(model.summary())

    history = model.fit_generator(
        train_generator,
        class_weight=class_weights,
        steps_per_epoch=steps_per_epoch,
        epochs=HEAD_EPOCHES,
        validation_data=val_generator,
        validation_steps=validation_steps,
        verbose=1,
        callbacks=callbacks,
        workers=8,
        use_multiprocessing=True,
    )

    base_model.trainable = True

    model.compile(
        loss='categorical_crossentropy',
        optimizer=keras.optimizers.adam(lr=1e-5, clipnorm=0.001),
        metrics=['acc'])

    print(model.summary())

    history = model.fit_generator(
        train_generator,
        class_weight=class_weights,
        steps_per_epoch=steps_per_epoch,
        initial_epoch=HEAD_EPOCHES,
        epochs=HEAD_EPOCHES + FINETUNING_EPOCHES,
        validation_data=val_generator,
        validation_steps=validation_steps,
        verbose=1,
        callbacks=callbacks,
        workers=8,
        use_multiprocessing=True,
    )

    callbacks = [
        ModelCheckpoint(
            filepath='epoch-{epoch:03d}_loss-{loss:.5f}_acc-{acc:.5f}.h5',
            verbose=1,
            period=10),
        ReduceLROnPlateau(
            monitor='acc',
            factor=0.94,
            patience=5,
            verbose=1,
            min_lr=0),
    ]

    history = model.fit_generator(
        all_generator,
        class_weight=class_weights_all,
        steps_per_epoch=all_steps,
        initial_epoch=HEAD_EPOCHES + FINETUNING_EPOCHES,
        epochs=HEAD_EPOCHES + FINETUNING_EPOCHES + ALL_EPOCHES,
        verbose=1,
        callbacks=callbacks,
        workers=8,
        use_multiprocessing=True,
    )

    save_model(model, 'image-classification', class_indices)


if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        epilog="Example: python3 train_classes.py --train tmp/train/ --val tmp/val/ --all tmp/all/")
    parser.add_argument(
        '--train',
        help='Path to tarin dataset. Example: --train tmp/train/')
    parser.add_argument(
        '--val',
        help='Path to validation dataset. Example: --val tmp/val/')
    parser.add_argument(
        '--all',
        help='Path to all dataset with tain and validation images. Example: --all tmp/all/')

    _main(parser.parse_args())

Closing comment (2018-09-22T18:10:29Z): Closing as duplicate of #11104