Network In Network
Authors: Min Lin, Qiang Chen, Shuicheng Yan
ICLR 2014

Abstract: We propose a novel network structure called 'Network In Network' (NIN) to enhance the model discriminability for local receptive fields. The conventional convolutional layer uses linear filters followed by a nonlinear activation function to scan the input. Instead, we build micro neural networks with more complex structures to handle the variance of the local receptive fields. We instantiate the micro neural network with a nonlinear multiple layer structure which is a potent function approximator. The feature maps are obtained by sliding the micro networks over the input in a similar manner of CNN and then fed into the next layer. The deep NIN is thus implemented as stacking of multiple sliding micro neural networks. With the enhanced local modeling via micro network, we are able to utilize global average pooling over feature maps in the classification layer, which is more interpretable and less prone to overfitting than traditional fully connected layers. We demonstrated state-of-the-art classification performances with NIN on CIFAR-10, CIFAR-100 and SVHN datasets.


Peer review:
[Review]:
That is an interesting paper. I have a few suggestions and comments about it:

(i) We used the same architecture in a paper publised at ICLR 2013 [1] for a specific problem and we called it as SMLP. Two differences in their approach from our paper are, for NIN authors stack several layers of locally connected MLPs(mlpconv) with tied weights, whereas we used only one layer of mlpconv and we didn't use Global Average Pooling. However sliding neural network over an image to do detection/classification is quiet old [2]. I think authors should cite those papers.

(ii) Moreover I think authors should provide more details about their experiments and hyperparameters that they have used (such as size of the local receptive fields, size of the strides).

(iii) A speed comparison between regular convolutional neural networks and NIN would be also interesting. 

[1] Gülçehre, Çağlar, and Yoshua Bengio. 'Knowledge matters: Importance of prior information for optimization.' arXiv preprint arXiv:1301.4083 (2013). 
[2] Rowley, Henry A., Shumeet Baluja, and Takeo Kanade. Human face detection in visual scenes. Pittsburgh, PA: School of Computer Science, Carnegie Mellon University, 1995.