Issue 1:
python3 train_classes_v2.py --train 1/train/ --val 1/val/ --all 1/all/
Using TensorFlow backend.
2018-09-22 10:35:27.903598: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled
to use: AVX2 FMA
2018-09-22 10:35:28.554300: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but the
re must be at least one NUMA node, so returning NUMA node zero
2018-09-22 10:35:28.554868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:01:00.0
totalMemory: 7.92GiB freeMemory: 7.80GiB
2018-09-22 10:35:28.554883: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-09-22 10:35:28.731514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-22 10:35:28.731544: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0
2018-09-22 10:35:28.731550: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N
2018-09-22 10:35:28.731704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0
with 7537 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
Found 6400 images belonging to 32 classes.
Found 1306 images belonging to 32 classes.
Found 6522 images belonging to 32 classes.
{'SKU49862': 15, 'SKU49871': 21, 'SKU49844': 3, 'SKU49845': 4, 'SKU49882': 22, 'SKU49981': 25, 'SKU49861': 14, 'SKU49985': 29, 'SKU49982': 26, 'SKU49883': 23
, 'SKU49865': 17, 'SKU49851': 10, 'SKU49984': 28, 'SKU49850': 9, 'SKU49843': 2, 'SKU49987': 31, 'SKU49849': 8, 'SKU49847': 6, 'SKU49855': 11, 'SKU49846': 5,
'SKU49860': 13, 'SKU49983': 27, 'SKU49864': 16, 'SKU49842': 1, 'SKU49866': 18, 'SKU49841': 0, 'SKU49986': 30, 'SKU49884': 24, 'SKU49867': 19, 'SKU49848': 7,
'SKU49868': 20, 'SKU49859': 12}
steps_per_epoch: 6400.0
validation_steps: 1306.0
all_steps: 6522.0
Downloading data from https://github.com/titu1994/Keras-NASNet/releases/download/v1.2/NASNet-mobile-no-top.h5
19996672/19993432 [==============================] - 5s 0us/step
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
NASNet (Model) (None, 21, 7, 1056) 4269716
_________________________________________________________________
conv2d_1 (Conv2D) (None, 21, 7, 32) 33824
_________________________________________________________________
global_average_pooling2d_1 ( (None, 32) 0
_________________________________________________________________
softmax_1 (Softmax) (None, 32) 0
=================================================================
Total params: 4,303,540
Trainable params: 33,824
Non-trainable params: 4,269,716
_________________________________________________________________
None
Epoch 1/1
6400/6400 [==============================] - 181s 28ms/step - loss: 3.4429 - acc: 0.0819 - val_loss: 3.4432 - val_acc: 0.0704
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
NASNet (Model) (None, 21, 7, 1056) 4269716
_________________________________________________________________
conv2d_1 (Conv2D) (None, 21, 7, 32) 33824
_________________________________________________________________
global_average_pooling2d_1 ( (None, 32) 0
_________________________________________________________________
softmax_1 (Softmax) (None, 32) 0
=================================================================
Total params: 4,303,540
Trainable params: 4,266,802
Non-trainable params: 36,738
_________________________________________________________________
None
Epoch 2/151
6400/6400 [==============================] - 1308s 204ms/step - loss: 2.1879 - acc: 0.6333 - val_loss: 3.3195 - val_acc: 0.4556
Epoch 3/151
6400/6400 [==============================] - 1265s 198ms/step - loss: 0.4105 - acc: 0.8980 - val_loss: 2.2126 - val_acc: 0.5383
Epoch 4/151
6400/6400 [==============================] - 1268s 198ms/step - loss: 0.1253 - acc: 0.9513 - val_loss: 1.4821 - val_acc: 0.6715
Epoch 5/151
6400/6400 [==============================] - 1270s 198ms/step - loss: 0.0915 - acc: 0.9677 - val_loss: 1.3620 - val_acc: 0.6715
Epoch 6/151
6400/6400 [==============================] - 1271s 199ms/step - loss: 0.0861 - acc: 0.9783 - val_loss: 1.4910 - val_acc: 0.7021
Epoch 7/151
6400/6400 [==============================] - 1272s 199ms/step - loss: 0.0590 - acc: 0.9908 - val_loss: 1.3748 - val_acc: 0.7175
Epoch 8/151
6400/6400 [==============================] - 1272s 199ms/step - loss: 0.0343 - acc: 0.9961 - val_loss: 1.4008 - val_acc: 0.7060
Epoch 9/151
6400/6400 [==============================] - 1270s 198ms/step - loss: 0.0249 - acc: 0.9972 - val_loss: 1.4543 - val_acc: 0.7136
Epoch 10/151
6400/6400 [==============================] - 1271s 199ms/step - loss: 0.0108 - acc: 0.9987 - val_loss: 1.4282 - val_acc: 0.6983

Epoch 00010: saving model to epoch-010_loss-0.01085_acc-0.99875_val_loss-1.42822_val_acc-0.69832.h5
Traceback (most recent call last):
File "train_classes_v2.py", line 338, in <module>
_main(parser.parse_args())
File "train_classes_v2.py", line 292, in _main
use_multiprocessing=True,
File "/home/michael/.local/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/michael/.local/lib/python3.5/site-packages/keras/engine/training.py", line 1415, in fit_generator
initial_epoch=initial_epoch)
File "/home/michael/.local/lib/python3.5/site-packages/keras/engine/training_generator.py", line 247, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)
File "/home/michael/.local/lib/python3.5/site-packages/keras/callbacks.py", line 77, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "/home/michael/.local/lib/python3.5/site-packages/keras/callbacks.py", line 455, in on_epoch_end
self.model.save(filepath, overwrite=True)
File "/home/michael/.local/lib/python3.5/site-packages/keras/engine/network.py", line 1085, in save
save_model(self, filepath, overwrite, include_optimizer)
File "/home/michael/.local/lib/python3.5/site-packages/keras/engine/saving.py", line 174, in save_model
'weight_names'] = weight_names
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/home/michael/.local/lib/python3.5/site-packages/h5py/_hl/attrs.py", line 95, in __setitem__
self.create(name, data=value, dtype=base.guess_dtype(value))
File "/home/michael/.local/lib/python3.5/site-packages/h5py/_hl/attrs.py", line 188, in create
attr = h5a.create(self._id, self._e(tempname), htype, space)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5a.pyx", line 47, in h5py.h5a.create
RuntimeError: Unable to create attribute (object header message is too large)

Code:

import os.path

import keras
from keras.applications import NASNetMobile
from keras.models import Sequential, model_from_json
from keras.layers import Conv2D, GlobalAveragePooling2D, Softmax, Dropout
from keras.callbacks import ModelCheckpoint, ReduceLROnPlateau
from keras.preprocessing.image import ImageDataGenerator

import tensorflow as tf

import json
import argparse
import glob
import cv2
import numpy as np
from sklearn.utils import class_weight

BATCH_SIZE = 1
IMG_HEIGHT = 224 * 3 # ResNetInceptionv2 & Xception: 299, ResNet50 & VGG: 224
IMG_WIDTH = 224
DIMS = (IMG_HEIGHT, IMG_WIDTH, 3)

HEAD_EPOCHES = 1
FINETUNING_EPOCHES = 150
ALL_EPOCHES = 50

def save_model(model, name, classes):
# модель
json_string = model.to_json()
open(name + '.model.json', 'w').write(json_string)

# список классов
json_string = json.dumps(classes, indent="\t")
open(name + '.classes.json', 'w').write(json_string)

# веса модели
model.save_weights(name + ".weights.h5")

# set tf backend to allow memory to grow, instead of claiming everything
def get_session(cpu_only=False):

num_cores = 4

config = tf.ConfigProto()
if cpu_only:
config = tf.ConfigProto(
intra_op_parallelism_threads=num_cores,
inter_op_parallelism_threads=num_cores,
allow_soft_placement=True,
device_count={
'CPU': 1,
'GPU': 0
})
else:
config = tf.ConfigProto(
intra_op_parallelism_threads=num_cores,
inter_op_parallelism_threads=num_cores,
allow_soft_placement=True,
device_count={
'CPU': 1,
'GPU': 1
})
config.gpu_options.allow_growth = True

return tf.Session(config=config)

def _main(args):

trainPath = args.train
valPath = args.val
allPath = args.all

keras.backend.tensorflow_backend.set_session(get_session())

train_datagen = ImageDataGenerator(
rescale=1. / 255,
rotation_range=8,
width_shift_range=0.1,
height_shift_range=0.1,
shear_range=8,
zoom_range=0.1,
samplewise_center=True,
samplewise_std_normalization=True,
fill_mode="constant",
cval=0,
)

val_datagen = ImageDataGenerator(
rescale=1. / 255,
samplewise_center=True,
samplewise_std_normalization=True,
)

train_generator = train_datagen.flow_from_directory(
trainPath,
target_size=(IMG_HEIGHT, IMG_WIDTH),
batch_size=BATCH_SIZE,
class_mode='categorical',
interpolation="bicubic",
#save_to_dir='aug',
)

val_generator = val_datagen.flow_from_directory(
valPath,
target_size=(IMG_HEIGHT, IMG_WIDTH),
batch_size=BATCH_SIZE,
class_mode='categorical',
interpolation="bicubic",
#save_to_dir='aug2',
)

all_generator = train_datagen.flow_from_directory(
allPath,
target_size=(IMG_HEIGHT, IMG_WIDTH),
batch_size=BATCH_SIZE,
class_mode='categorical',
interpolation="bicubic",
#save_to_dir='aug3',
)

class_indices = train_generator.class_indices
print(class_indices)

num_of_classes = train_generator.num_classes
num_of_samples = train_generator.samples

steps_per_epoch = num_of_samples / BATCH_SIZE
validation_steps = val_generator.samples / BATCH_SIZE
all_steps = all_generator.samples / BATCH_SIZE

print('steps_per_epoch: ' + str(steps_per_epoch))
print('validation_steps: ' + str(validation_steps))
print('all_steps: ' + str(all_steps))

class_weights = class_weight.compute_class_weight(
'balanced', np.unique(train_generator.classes),
train_generator.classes)

class_weights_all = class_weight.compute_class_weight(
'balanced', np.unique(all_generator.classes), all_generator.classes)

callbacks = [
ModelCheckpoint(
filepath='epoch-{epoch:03d}_loss-{loss:.5f}_acc-{acc:.5f}_val_loss-{val_loss:.5f}_val_acc-{val_acc:.5f}.h5',
verbose=1,
period=10),
]

base_model = NASNetMobile(include_top=False, input_shape=DIMS)
base_model.trainable = False

model = Sequential()
model.add(base_model)
model.add(Conv2D(num_of_classes, (1, 1), activation='relu'))
model.add(GlobalAveragePooling2D())
model.add(Softmax())

for layer in base_model.layers:
if hasattr(layer, 'kernel_regularizer'):
layer.kernel_regularizer= keras.regularizers.l2(0.00001)

model.compile(
loss='categorical_crossentropy',
optimizer=keras.optimizers.adam(lr=1e-5, clipnorm=0.001),
metrics=['acc'])

print(model.summary())

history = model.fit_generator(
train_generator,
class_weight=class_weights,
steps_per_epoch=steps_per_epoch,
epochs=HEAD_EPOCHES,
validation_data=val_generator,
validation_steps=validation_steps,
verbose=1,
callbacks=callbacks,
workers=8,
use_multiprocessing=True,
)

base_model.trainable = True

model.compile(
loss='categorical_crossentropy',
optimizer=keras.optimizers.adam(lr=1e-5, clipnorm=0.001),
metrics=['acc'])

print(model.summary())

history = model.fit_generator(
train_generator,
class_weight=class_weights,
steps_per_epoch=steps_per_epoch,
initial_epoch=HEAD_EPOCHES,
epochs=HEAD_EPOCHES + FINETUNING_EPOCHES,
validation_data=val_generator,
validation_steps=validation_steps,
verbose=1,
callbacks=callbacks,
workers=8,
use_multiprocessing=True,
)

callbacks = [
ModelCheckpoint(
filepath='epoch-{epoch:03d}_loss-{loss:.5f}_acc-{acc:.5f}.h5',
verbose=1,
period=10),
ReduceLROnPlateau(
monitor='acc',
factor=0.94,
patience=5,
verbose=1,
min_lr=0),
]

history = model.fit_generator(
all_generator,
class_weight=class_weights_all,
steps_per_epoch=all_steps,
initial_epoch=HEAD_EPOCHES + FINETUNING_EPOCHES,
epochs=HEAD_EPOCHES + FINETUNING_EPOCHES + ALL_EPOCHES,
verbose=1,
callbacks=callbacks,
workers=8,
use_multiprocessing=True,
)

save_model(model, 'image-classification', class_indices)

if __name__ == '__main__':
parser = argparse.ArgumentParser(
epilog="Example: python3 train_classes.py --train tmp/train/ --val tmp/val/ --all tmp/all/")
parser.add_argument(
'--train',
help='Path to tarin dataset. Example: --train tmp/train/')
parser.add_argument(
'--val',
help='Path to validation dataset. Example: --val tmp/val/')
parser.add_argument(
'--all',
help='Path to all dataset with tain and validation images. Example: --all tmp/all/')

_main(parser.parse_args())

Issue 2:
When trying to save my model I get the runtime error below. There was a similar issue when model layers names were too long and it can be solved by giving layers shorter names. This time the error pops up when saving optimizer weights. getattr(model.optimizer,'weights') shows

[<tf.Variable 'Adam/iterations:0' shape=() dtype=int64_ref>,
<tf.Variable 'training/Adam/Variable:0' shape=(3, 3, 1, 64) dtype=float32_ref>,
<tf.Variable 'training/Adam/Variable_1:0' shape=(64,) dtype=float32_ref>,
<tf.Variable 'training/Adam/Variable_2:0' shape=(64,) dtype=float32_ref>,
...]

and if I convert it to numpy array its length is above the 64k limits which gives h5 runtime. I can save the model if I use save_model(....,include_optimizer=False) but I need the optimizer state. Is there any way I can reduce the length of "training/Adam/Variable:0"... names so as to fit them into 64k hdf5 table limit. Thanks.

---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-130-d231b4a5a40c> in <module>()
----> 1 model.save('model')

C:\Anaconda3\lib\site-packages\keras\engine\network.py in save(self, filepath, overwrite, include_optimizer)
1083 raise NotImplementedError
1084 from ..models import save_model
-> 1085 save_model(self, filepath, overwrite, include_optimizer)
1086
1087 def save_weights(self, filepath, overwrite=True):

C:\Anaconda3\lib\site-packages\keras\engine\saving.py in save_model(model, filepath, overwrite, include_optimizer)
173 #print('Weight names',weight_names,len(weight_names),np.asarray(weight_names).nbytes)
174 optimizer_weights_group.attrs[
--> 175 'weight_names'] = weight_names
176 for name, val in zip(weight_names, weight_values):
177 param_dset = optimizer_weights_group.create_dataset(

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

C:\Anaconda3\lib\site-packages\h5py\_hl\attrs.py in __setitem__(self, name, value)
93 use the methods create() and modify().
94 """
---> 95 self.create(name, data=value, dtype=base.guess_dtype(value))
96
97 @with_phil

C:\Anaconda3\lib\site-packages\h5py\_hl\attrs.py in create(self, name, data, shape, dtype)
186
187 try:
--> 188 attr = h5a.create(self._id, self._e(tempname), htype, space)
189 except:
190 raise

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

h5py\h5a.pyx in h5py.h5a.create()

RuntimeError: Unable to create attribute (object header message is too large)

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.

Thank you!

Check that you are up-to-date with the master branch of Keras. You can update with:

pip install git+git://github.com/keras-team/keras.git --upgrade --no-deps

If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:

pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).