Network In Network
Authors: Min Lin, Qiang Chen, Shuicheng Yan
ICLR 2014

Abstract: We propose a novel network structure called 'Network In Network' (NIN) to enhance the model discriminability for local receptive fields. The conventional convolutional layer uses linear filters followed by a nonlinear activation function to scan the input. Instead, we build micro neural networks with more complex structures to handle the variance of the local receptive fields. We instantiate the micro neural network with a nonlinear multiple layer structure which is a potent function approximator. The feature maps are obtained by sliding the micro networks over the input in a similar manner of CNN and then fed into the next layer. The deep NIN is thus implemented as stacking of multiple sliding micro neural networks. With the enhanced local modeling via micro network, we are able to utilize global average pooling over feature maps in the classification layer, which is more interpretable and less prone to overfitting than traditional fully connected layers. We demonstrated state-of-the-art classification performances with NIN on CIFAR-10, CIFAR-100 and SVHN datasets.


Peer review:
[Review]:
> - A brief summary of the paper's contributions, in the context of prior work.

Convolutional neural networks have been an essential part of the recent breakthroughs deep learning has made on pattern recognition problems such as object detection and speech recognition. Typically, such networks consist of convolutional layers (where copies of the same neuron look at different patches of the same image), pooling layers, normal fully-connected layers, and finally a softmax layer.

This paper modifies the architecture in two ways. Firstly, the authors explore an extremely natural generalization of convolutional layers by changing the unit of convolution: instead of running a neuron in lots of locations, they run a 'micro network.' Secondly, instead of having fully-connected layers, they have features generated by the final convolutional layer correspond to categories, and perform global average pooling before feeding the features into a softmax layer. Dropout is used between mlpconv layers.

The paper reports new state-of-the-art results with this modified architecture on a variety of benchmark datasets: CIFAR-10, CIFAR-100, and SVHN. They also achieve near state-of-the-art performance on MNIST.

> - An assessment of novelty and quality.

The reviewer is not an expert but believes this to be the first use of the 'Network In Network' architecture in the literature. The most similar thing the reviewer is aware of is work designing more flexible neurons and using them in convolutional layers (eg. maxout by Goodfellow et al, cited in this paper). The difference between a very flexible neuron and a small network with only one output may become a matter of interpretation at some point.

The paper very clearly outlines the new architecture, the experiments performed, and the results.

> - A list of pros and cons (reasons to accept/reject).

Pros:
* The work is performed in an important and active area.
* The paper explores a very natural generalization of convolutional layers. It's really nice to have this so thoroughly explored.
* The authors perform experiments to understand how global average pooling affect networks independently of mlpconv layers.
* The paper reports new state-of-the-art results on several standard datasets.
* The paper is clearly written.

Cons:
* All the datasets the model is tested on are classification of rather small images (32x32 and 28x28). One could imagine a few stories where the mlpconv layers would have a comparative advantage on small images (eg. the small size makes having lots of convolutional layers tricky, so it's really helpful to have each layer be more powerful individually). If this was the case, mlpconv would still be useful and worth publishing, but it would be a bit less exciting. That said, it clearly wouldn't be reasonable to demand the architecture be tested on ImageNet -- the reviewer is just very curious.
* It would also be nice to know what happens if you apply dropout to the entire model instead of just between mlpconv layers. (Again, the reviewer is just curious.)