Repository: keras-team/keras

Issue:
Model serialization
This discussion started in #51, but as things can get complicated I decided to start another issue.

It seems to be a good idea to store weights for a model separately in an HDF5 file (or even a Numpy npy file, but HDF5 would be more portable). I wanted to compare how large is a serialized model with and without the weights, so I did the following test:

```
    model = Sequential()
    model.add(Dense(n_input, 2048, init='uniform', activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(2048, 2048, init='uniform', activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(2048, 2048, init='uniform', activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(2048, 2048, init='uniform', activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(2048, n_output, init='uniform', activation='linear'))
```

(the model is intentionally large!)

I then compiled the model, serialized it after compilation, and removed the weights and post-compilation Theano objects/functions as follows:

```
    for l in model.layers:
        weights.append(l.get_weights())
        l.params = []
        try:
            l.W = None
            l.b = None
        except ValueError:
            pass
    model.X = None
    model.y = None
    model.y_test = None
    model.y_train = None
    model._train  = None
    model._train_with_acc  = None
    model._test = None
    model._test_with_acc = None
    model._predict = None
```

The full compiled model ends up with 243 MB, and the cleaned-up model with 120 MB (which is exacly the same we would get from pickling the non-compiled models with the weight matrices deleted). Is there anything else we could remove to make the serialized model smaller?



Discussion:
In theory, besides the weights, we only need the parameters to the .compile() method, the names of the layer classes (or rather their import path) and the parameters that were passed to each layer. It should take a few kilobytes in total. 

There are hack-ish ways to recover attributes from an arbitrary layer, but that's kind of ugly. Here's an example:

``` python
 for l in layers:
    attributes = []
    for a in dir(l):
        if a[:2] != '__' and a not in ['params', 'previous_layer', 'input']:
            if type(getattr(l, a)) != types.MethodType:
                attributes.append((a, getattr(l, a)))
```

Besides being ugly, it might be somewhat brittle. 

Here's a better alternative: have all layers expose a .config or .get_config attribute or method that returns the constructor arguments to the layer, as a dict. Then it's easy to reconstruct the layer:

```
config = layer.get_config()
weights = layer.get_weights()

new_layer = Layer(**config)
new_layer.set_weights(weights)
```

That's a good approach, but we'll also need the correct layer type and a way to call the right constructor, as we cannot call the base class constructor with arbitrary params. Some alternatives could be:
1. Long function matching each layer class (which would be a pain to maintain). 
2. Keeping all the layer subclasses in a list and matching over that list (with `__name__`). Also not great for maintaining, but better than 1. 
3. Dynamically calling class names from strings, which is possible but hacky and unsafe.

I think the simplest thing to do would be to dynamically instantiate the layers from their import path, which we would have saved as part of the serialization. This give users the freedom to use the .save() function with custom layers not part of the Keras source. 

Of course, it's sort of hacky. But at this point all our options are looking somewhat hacky. 

Alternatively, to do something cleaner we would have to restrict ourselves to known layers from the source. 

Maybe we should search for inspiration somewhere else. Do you know how Torch7 implements their saving function?

I've been mostly working with Theano-based code and Caffe (which does not have this problem as everything is in the config file). By reading Torch7 [code](https://github.com/torch/torch7/blob/a99177e7be941e50792e4cc15d247462399cdcce/File.lua) used for serialization, it seems they serialize the whole objects without any "smart" parameter saving besides avoiding to store the same object twice on disk.

[Mocha.jl](https://github.com/pluskid/Mocha.jl) (a Julia-based library inspired by Caffe) only stores parameters and requires the user to have code to rebuild the model. They only have a simple function to load the stored parameters to a network. Maybe having something like this would be enough, given the minimalist approach in keras?

I don't think we should be restricted to saving layers from the source, as adding new kinds of layers is something I suspect people would want to do often. I'll work tomorrow on an approach based on either just loading params from HDF5 files or with dynamic instantiation (as it seems the most flexible way to do it).

Ok, great! Looking forward to seeing what you come up with.


Resolving patch:
diff --git a/keras/models.py b/keras/models.py
--- a/keras/models.py
+++ b/keras/models.py
@@ -211,7 +211,31 @@ def evaluate(self, X, y, batch_size=128, show_accuracy=False, verbose=1):
             return tot_score/len(batches)
 
 
-                
-
+    def save_weights(self, filepath):
+        # Save weights from all layers to HDF5
+        import h5py
+        # FIXME: fail if file exists, or add option to overwrite!
+        f = h5py.File(filepath, 'w')
+        f.attrs['nb_layers'] = len(self.layers)
+        for k, l in enumerate(self.layers):
+            g = f.create_group('layer_{}'.format(k))
+            weights = l.get_weights()
+            g.attrs['nb_params'] = len(weights)
+            for n, param in enumerate(weights):
+                param_name = 'param_{}'.format(n)
+                param_dset = g.create_dataset(param_name, param.shape, dtype='float64')
+                param_dset[:] = param
+        f.flush()
+        f.close()
+
+    def load_weights(self, filepath):
+        # Loads weights from HDF5 file
+        import h5py
+        f = h5py.File(filepath)
+        for k in range(f.attrs['nb_layers']):
+            g = f['layer_{}'.format(k)]
+            weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]
+            self.layers[k].set_weights(weights)
+        f.close()
 
 


Test patch:
diff --git a/test/test_save_weights.py b/test/test_save_weights.py
new file mode 100644
--- /dev/null
+++ b/test/test_save_weights.py
@@ -0,0 +1,40 @@
+from keras.models import Sequential
+from keras.layers.core import Dense, Dropout, Activation
+from keras.optimizers import SGD
+
+import sys
+sys.setrecursionlimit(10000) # to be able to pickle Theano compiled functions
+
+import pickle, numpy
+
+def create_model():
+    model = Sequential()
+    model.add(Dense(256, 2048, init='uniform', activation='relu'))
+    model.add(Dropout(0.5))
+    model.add(Dense(2048, 2048, init='uniform', activation='relu'))
+    model.add(Dropout(0.5))
+    model.add(Dense(2048, 2048, init='uniform', activation='relu'))
+    model.add(Dropout(0.5))
+    model.add(Dense(2048, 2048, init='uniform', activation='relu'))
+    model.add(Dropout(0.5))
+    model.add(Dense(2048, 256, init='uniform', activation='linear'))
+    return model
+
+model = create_model()
+sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
+model.compile(loss='mse', optimizer=sgd)
+
+pickle.dump(model, open('/tmp/model.pkl', 'wb'))
+model.save_weights('/tmp/model_weights.hdf5')
+
+model_loaded = create_model()
+model_loaded.load_weights('/tmp/model_weights.hdf5')
+
+for k in range(len(model.layers)):
+    weights_orig = model.layers[k].get_weights()
+    weights_loaded = model_loaded.layers[k].get_weights()
+    for x, y in zip(weights_orig, weights_loaded):
+        if numpy.any(x != y):
+            raise ValueError('Loaded weights are different from pickled weights!')
+
+