Repository: keras-team/keras

Issue:
Keras Examples without Magic Numbers
Hello,

Looking at the CIFAR10 example:
https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py
(3-layered 32x32 images)

I tried to run the model on the LFW (Faces in the Wild) dataset
http://vis-www.cs.umass.edu/lfw/
(3-layered 250x250 images)

I have reshaped the data to the same shape as the CIFAR10 data, but I get other errors:

``` python
ValueError: matrices are not aligned
Apply node that caused the error: dot(Reshape{2}.0, HostFromGpu.0)
Inputs types: [TensorType(float32, matrix), TensorType(float32, matrix)]
Inputs shapes: [(3, 246016), (4096, 512)]
Inputs strides: [(984064, 4), (2048, 4)]
Inputs values: ['not shown', 'not shown']
```

If I understand correctly, this happens because I use the CIFAR10 model as-is, but its parameters - magic numbers such as 32 (which is also the image dimensions of the CIFAR10 dataset are all around the model definition:

``` python
model = Sequential()

model.add(Convolution2D(32, 3, 3, 3, border_mode='full'))
model.add(Activation('relu'))
model.add(Convolution2D(32, 32, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(poolsize=(2, 2)))
model.add(Dropout(0.25))

model.add(Convolution2D(64, 32, 3, 3, border_mode='full'))
model.add(Activation('relu'))
model.add(Convolution2D(64, 64, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(poolsize=(2, 2)))
model.add(Dropout(0.25))

model.add(Flatten())
model.add(Dense(64*8*8, 512))
model.add(Activation('relu'))
model.add(Dropout(0.5))

model.add(Dense(512, nb_classes))
model.add(Activation('softmax'))
```

It would be great for newbies such as myself is instead of magic numbers, the way to calculate them would be stored in an variable and then those variable would be used in building the model. Even for those variables which are selected arbitrarily and on which no other variables are dependent would be better marked. Take for instance batch sizes - of course they affect the training, but on the other hand we can toy around with them and the code would still run (which is my main goal right now).

I know that this is partially because I'm not well-versed in ANNs and in ConvNets, but believe me that understanding things first via code, then by reading is the best method for me. Let's be Pythonic about it and make keras friendly to newbies.

Thanks!



Discussion:
Here is what I meant: I finally broke down the CIFAR10 demo and replaced the magic numbers with variables. Now I can separate between what variables are my choice (kernel size, number of kernels) and what isn't (things that derive from each other).

Of course my degree of freedom is not unlimited - I can't choose too big (or too small) a kernel size (etc.), but a code like this gave me much more understanding regarding how convolution layers work.

``` python
# Initial arguments
nr_kernels1 = 32 # arbitrary choice
kernel_size = 3  # arbitrary choice
nr_channels = X_train.shape[1]  # RGB inputs => 3 channels

#1
# model.add(Convolution2D(32, 3, 3, 3, border_mode='full'))

model.add(Convolution2D(nr_kernels1,
                        nr_channels,
                        kernel_size,
                        kernel_size,
                        border_mode='full'))

model.add(Activation('relu'))
# (b x 3 x 32 x 32) -> (b x 32 x 32 x 32)


nr_kernels2 = 32 #arbitrary choice
model.add(Convolution2D(
        nr_kernels2,  # I chose it
        nr_kernels1,  # derived from output of previous layer
        kernel_size,  # I chose it
        kernel_size)) # I chose it

model.add(Activation('relu'))


#3
model.add(MaxPooling2D(poolsize=(2, 2))) # arbitrary choice
model.add(Dropout(0.25))
# (b x 32 x 32 x 32) -> (b x 32 x 16 x 16)


#4
nr_kernels3=64
model.add(Convolution2D(nr_kernels3, # I chose it
                        nr_kernels2,
                        kernel_size,
                        kernel_size,
                        border_mode='full'))
model.add(Activation('relu'))
# (b x 32 x 16 x 16) -> (b x 64 x 16 x 16)


#5
nr_kernels4=64
model.add(Convolution2D(nr_kernels4,
                        nr_kernels3,
                        kernel_size,
                        kernel_size))
model.add(Activation('relu'))
# (b x 64 x 16 x 16) -> (b x 64 x 16 x 16)


#6
model.add(MaxPooling2D(poolsize=(2, 2))) # arbitrary
model.add(Dropout(0.25))
# (b x 64 x 16 x 16) -> (b x 64 x 8 x 8)


#7
model.add(Flatten())


#8
# The dimensions of each image is (N = (X_train.shape[2]) ** 2),
# We had two downsampling layers of 2x2 maxpooling, so we divide each dimension twice by 2 (/2 /2).
# The input to this layer is the 64 "channels" that the previous layer outputs. Thus we have a layer of
# nr_kernels * (N / 2 / 2) * (N / 2 / 2)

flat_layer_size = nr_kernels4 * (X_train.shape[2] / 2 / 2) ** 2
final_layer_size=512 # I chose it
model.add(Dense(flat_layer_size, final_layer_size))
model.add(Activation('relu'))
model.add(Dropout(0.5))


#9
model.add(Dense(final_layer_size, nb_classes))
model.add(Activation('softmax'))
```

If you're willing to give it more formalism (comments and variable names), you could submit a pull request with this. 

Sure thing. Thanks for keras!

@guy4261 I agree with you. Such comments and style makes it more readable for newbies. I would like to see you make comments in other examples in keras(./keras/examples.py) especially for those RNN example codes.

It's a little confusing but if you read some of the other issue posts and read up on the network type, you should be able to fill in the holes. But if you can't just holla ;) 

@guy4261 - a little late to the party but: I have an [an example](https://github.com/dribnet/lfw_fuel/blob/857dfb43bbcdeb5d820c95b541569af181617409/example/run-lfw.py) of using LFW with Keras that has fewer magic numbers. Since 250x250x3x2 is huge, I [crop and scale the image pairs](https://github.com/dribnet/lfw_fuel/blob/857dfb43bbcdeb5d820c95b541569af181617409/example/run-lfw.py#L29-L35) in a paramaterized way, and [adjust the network](https://github.com/dribnet/lfw_fuel/blob/857dfb43bbcdeb5d820c95b541569af181617409/example/run-lfw.py#L61) accordingly.

Also - if you use that that [lfw_fuel repo](https://github.com/dribnet/lfw_fuel/) itself as a dependency, you'll have LFW data in any of of three formats (regular, funneled, deepfunneled) with the official test/train splits ready to use with Keras. My example runs but doesn't perform well, so suggestions getting the accuracy up (and the magic numbers down) also appreciated!


Resolving patch:
diff --git a/examples/cifar10_cnn.py b/examples/cifar10_cnn.py
--- a/examples/cifar10_cnn.py
+++ b/examples/cifar10_cnn.py
@@ -28,6 +28,17 @@
 nb_epoch = 200
 data_augmentation = True
 
+# shape of the image (SHAPE x SHAPE)
+shapex, shapey = 32, 32
+# number of convolutional filters to use at each layer
+nb_filters = [32, 64]
+# level of pooling to perform at each layer (POOL x POOL)
+nb_pool = [2, 2]
+# level of convolution to perform at each layer (CONV x CONV)
+nb_conv = [3, 3]
+# the CIFAR10 images are RGB
+image_dimensions = 3
+
 # the data, shuffled and split between tran and test sets
 (X_train, y_train), (X_test, y_test) = cifar10.load_data()
 print('X_train shape:', X_train.shape)
@@ -40,22 +51,24 @@
 
 model = Sequential()
 
-model.add(Convolution2D(32, 3, 3, 3, border_mode='full'))
+model.add(Convolution2D(nb_filters[0], image_dimensions, nb_conv[0], nb_conv[0], border_mode='full'))
 model.add(Activation('relu'))
-model.add(Convolution2D(32, 32, 3, 3))
+model.add(Convolution2D(nb_filters[0], nb_filters[0], nb_conv[0], nb_conv[0]))
 model.add(Activation('relu'))
-model.add(MaxPooling2D(poolsize=(2, 2)))
+model.add(MaxPooling2D(poolsize=(nb_pool[0], nb_pool[0])))
 model.add(Dropout(0.25))
 
-model.add(Convolution2D(64, 32, 3, 3, border_mode='full'))
+model.add(Convolution2D(nb_filters[1], nb_filters[0], nb_conv[0], nb_conv[0], border_mode='full'))
 model.add(Activation('relu'))
-model.add(Convolution2D(64, 64, 3, 3))
+model.add(Convolution2D(nb_filters[1], nb_filters[1], nb_conv[1], nb_conv[1]))
 model.add(Activation('relu'))
-model.add(MaxPooling2D(poolsize=(2, 2)))
+model.add(MaxPooling2D(poolsize=(nb_pool[1], nb_pool[1])))
 model.add(Dropout(0.25))
 
 model.add(Flatten())
-model.add(Dense(64*8*8, 512))
+# the image dimensions are the original dimensions divided by any pooling
+# each pixel has a number of filters, determined by the last Convolution2D layer
+model.add(Dense(nb_filters[-1] * (shapex / nb_pool[0] / nb_pool[1]) * (shapey / nb_pool[0] / nb_pool[1]), 512))
 model.add(Activation('relu'))
 model.add(Dropout(0.5))
 
diff --git a/examples/mnist_cnn.py b/examples/mnist_cnn.py
--- a/examples/mnist_cnn.py
+++ b/examples/mnist_cnn.py
@@ -22,11 +22,20 @@
 nb_classes = 10
 nb_epoch = 12
 
+# shape of the image (SHAPE x SHAPE)
+shapex, shapey = 28, 28
+# number of convolutional filters to use
+nb_filters = 32
+# level of pooling to perform (POOL x POOL)
+nb_pool = 2
+# level of convolution to perform (CONV x CONV)
+nb_conv = 3
+
 # the data, shuffled and split between tran and test sets
 (X_train, y_train), (X_test, y_test) = mnist.load_data()
 
-X_train = X_train.reshape(X_train.shape[0], 1, 28, 28)
-X_test = X_test.reshape(X_test.shape[0], 1, 28, 28)
+X_train = X_train.reshape(X_train.shape[0], 1, shapex, shapey)
+X_test = X_test.reshape(X_test.shape[0], 1, shapex, shapey)
 X_train = X_train.astype("float32")
 X_test = X_test.astype("float32")
 X_train /= 255
@@ -41,15 +50,18 @@
 
 model = Sequential()
 
-model.add(Convolution2D(32, 1, 3, 3, border_mode='full'))
+model.add(Convolution2D(nb_filters, 1, nb_conv, nb_conv, border_mode='full'))
 model.add(Activation('relu'))
-model.add(Convolution2D(32, 32, 3, 3))
+model.add(Convolution2D(nb_filters, nb_filters, nb_conv, nb_conv))
 model.add(Activation('relu'))
-model.add(MaxPooling2D(poolsize=(2, 2)))
+model.add(MaxPooling2D(poolsize=(nb_pool, nb_pool)))
 model.add(Dropout(0.25))
 
 model.add(Flatten())
-model.add(Dense(32*196, 128))
+# the resulting image after conv and pooling is the original shape
+# divided by the pooling with a number of filters for each "pixel"
+# (the number of filters is determined by the last Conv2D)
+model.add(Dense(nb_filters * (shapex / nb_pool) * (shapey / nb_pool), 128))
 model.add(Activation('relu'))
 model.add(Dropout(0.5))
 


Test patch: