Network In Network
Authors: Min Lin, Qiang Chen, Shuicheng Yan
ICLR 2014

Abstract: We propose a novel network structure called 'Network In Network' (NIN) to enhance the model discriminability for local receptive fields. The conventional convolutional layer uses linear filters followed by a nonlinear activation function to scan the input. Instead, we build micro neural networks with more complex structures to handle the variance of the local receptive fields. We instantiate the micro neural network with a nonlinear multiple layer structure which is a potent function approximator. The feature maps are obtained by sliding the micro networks over the input in a similar manner of CNN and then fed into the next layer. The deep NIN is thus implemented as stacking of multiple sliding micro neural networks. With the enhanced local modeling via micro network, we are able to utilize global average pooling over feature maps in the classification layer, which is more interpretable and less prone to overfitting than traditional fully connected layers. We demonstrated state-of-the-art classification performances with NIN on CIFAR-10, CIFAR-100 and SVHN datasets.


Peer review:
[Review]:
Authors propose the following modification to standard architecture:

Replace:
convolution -> relu
with
convolution -> relu -> convolution (1x1 filter) -> relu

Additionally, instead of using fully connected layers, the depth of the last conv layer is the same as number of classes, which they average over x,y position. This generates a vector of per-class scores. 

There's a number of internal inconsistencies and omitted details which make reproducing the results impossible. Those issues must be fixed to be considered for acceptance. They do bring up one intriguing idea which gives a novel approach to localization.

- Authors would be better off using standard terminology, like I did above, that makes reading the paper easier.

- Some space is unnecessarily taken up by issues that are irrelevant/speculative, like discussing that this architecture allows for feature filters that are 'universal function approximators.' Do we have any evidence this is actually needed for performance?

- In section 3.2 they say that last averaged layer is fed into softmax, but this contradicts Figure 4 where it seems that last layer features actually correspond to classes, and no softmax is needed. I assumed the later was the intention.

- Following sentence is unclear, consider expanding or removing:

'A vectorized view of the global average pooling is that the output of the last mlpconv layer is forced into orthogonal subspaces for different categories of inputs'

- Most serious shortcoming of this paper is lack of detailed explanation of architecture. All I have to go on is the picture in Figure 2, which looks like 3 spatial pooling layers and 6 convolutional layers. Authors need to provide following information for each layer -- filter size/pooling size/stride/number of features. Ideally it would be in a succinct format like Table 2 of 'OverFeat' paper (1312.6229). We have implemented NIN idea on our own network we used for SVHN and got worse results. Since detailed architecture spec is missing, I can't tell if it's the problem of the idea or the particulars of the network we used.

One intriguing/promising idea they bring up is the idea of using averaging instead of fully connected layers. I expected this to allow one to be able to localize the object by looking at the outputs of the last conv layer before averaging, which indeed seems like the case from Figure 4.