Network In Network
Authors: Min Lin, Qiang Chen, Shuicheng Yan
ICLR 2014

Abstract: We propose a novel network structure called 'Network In Network' (NIN) to enhance the model discriminability for local receptive fields. The conventional convolutional layer uses linear filters followed by a nonlinear activation function to scan the input. Instead, we build micro neural networks with more complex structures to handle the variance of the local receptive fields. We instantiate the micro neural network with a nonlinear multiple layer structure which is a potent function approximator. The feature maps are obtained by sliding the micro networks over the input in a similar manner of CNN and then fed into the next layer. The deep NIN is thus implemented as stacking of multiple sliding micro neural networks. With the enhanced local modeling via micro network, we are able to utilize global average pooling over feature maps in the classification layer, which is more interpretable and less prone to overfitting than traditional fully connected layers. We demonstrated state-of-the-art classification performances with NIN on CIFAR-10, CIFAR-100 and SVHN datasets.


Peer review:
[Review]:
Interesting Paper. I think there are a lot of possibilities hidden in
the Ideas brought up here - as well as in the general Idea brought up
by the maxout work. I have a short comment to make regarding the
performance that you achieve with the 'Network in Network' model:

Although I do not think that this should influence the decision on
this paper (nor do I think it takes anything away from the Network 
in Network idea) I want to make you aware that I believe a large part
of your performance increase over maxout stems from your choice of
hyperparameters.

I am currently running a hyperparameter search for maxout for a paper
submitted to ICLR (the 'Improving Deep Neural Networks with
Probabilistic Maxout Units' paper). The preliminary best result that I
obtained for a maxout network on CIFAR-10 without data augmentation 
(using the same amount of units per layer as in the original maxout
paper) is 10.92 % error. If I understand it correctly then this is
approximately the same as the NiN model with a fully connected layer. 
The hyperparameter settings for this model are very similar to the
settings I assume were used in your paper (based on the parameter file 
you posted here
https://github.com/mavenlin/cuda-convnet/blob/master/NIN/cifar-10_def). 

The most crucial ingredient seems to be the pooling and filter/kernel
size. I will post more details on the hyperparameter settings in the
discussion on the 'Probabilistic Maxout' paper.