Network In Network
Authors: Min Lin, Qiang Chen, Shuicheng Yan
ICLR 2014

Abstract: We propose a novel network structure called 'Network In Network' (NIN) to enhance the model discriminability for local receptive fields. The conventional convolutional layer uses linear filters followed by a nonlinear activation function to scan the input. Instead, we build micro neural networks with more complex structures to handle the variance of the local receptive fields. We instantiate the micro neural network with a nonlinear multiple layer structure which is a potent function approximator. The feature maps are obtained by sliding the micro networks over the input in a similar manner of CNN and then fed into the next layer. The deep NIN is thus implemented as stacking of multiple sliding micro neural networks. With the enhanced local modeling via micro network, we are able to utilize global average pooling over feature maps in the classification layer, which is more interpretable and less prone to overfitting than traditional fully connected layers. We demonstrated state-of-the-art classification performances with NIN on CIFAR-10, CIFAR-100 and SVHN datasets.


Peer review:
[Review]:
Summary of contributions:
Proposes a new activation function for backprop nets. Advocates using global mean pooling instead of densely connected layers at the output of convolutional nets.

Novelty: moderate
Quality: moderate

Pros:
	-Very impressive results on CIFAR-10 and CIFAR-100
	-Acceptable results on SVHN and MNIST
	-Experiments distinguish between performance improvements due to NIN structure and performance improvements due to global average pooling
Cons:
	-Explanation of why NIN works well doesn’t make a lot of sense

I suspect that NIN’s performance has more to do with the way you apply dropout to the model, rather than the explanations you give in the paper. I elaborate more below in the detailed comments.

Did you ever try NIN without dropout?

Maxout without dropout generally does not work all that well, except in cases where each maxout unit has few filters, or the dataset size is very large. I suspect your NIN units don’t work well without dropout either, unless the micro-net is very small or you have a lot of data. I find it very weird that you don’t explore how well NIN works without dropout, and that your explanation of NIN’s performance doesn’t involve dropout at all.

This paper has strong results but I think a lot of the presentation is misleading. It should be published after being edited to take out some of the less certain stuff from the explanations. It could be a really great paper if you had a better story for why NIN works well, including experiments to back up this story. I suspect the story you have now is wrong though, and I suspect the correct story involves the interaction between NIN and dropout.

I’ve hear Geoff Hinton proposed using some kind of unit similar to this during a talk at the CIFAR summer school this year. I’ll ask one of the summer school students to comment on this paper. I don’t think this subtracts from your originality but it might be worth acknowledging his talk, depending on what the summer school student says.

Detailed comments:

Abstract:
	I don’t understand what it means “to enhance the model discriminability for local receptive fields.”
	

Introduction
	Paragraph 1: I don’t think we can confidently say that convolutional net features are generally related to binary discrimination of whether a specific feature is present, or that they are related to probabilities. For example, some of them might be related to measurements (“how red is this patch?” rather than “what is the probability that this patch is red?”) In general, our knowledge of what features are doing is fairly primitive, informal, and ad hoc. Note that the other ICLR submission “Intriguing Properties of Neural Networks” has some strong arguments against the idea of looking at the meaning of individual features in isolutian, or interpreting them as probabilistic detectors. Basically I think you could describe conv nets in the intro without committing to these less well-established ideas about how conv nets work.
	
	Paragraph 2: I understand that many interesting features can’t be detected by a GLM. But why does the first layer of the global architecture need to be a nonlinear feature detector? Your NIN architecture still is built out of GLM primitives. It seems like it’s a bit arbitrary which things you say can be linear versus non-linear. i.e., why does it matter that you group all of the functionality of the micro-networks and say that together those are non-linear? Couldn’t we just group the first two layers of a standard deep network and say they form a non-linear layer? Can’t we derive a NIN layer just by restricting the connective of multiple layers of a regular network in the right way?

	Paragraph 3: Why call it an mlpconv layer? Why not call it a NIN layer for consistency with the title of the paper?

	Last paragraph: why average pooling? Doesn’t it get hard for this to have a high confidence output if the spatial extent of the layer gets large?

Section 2: Convolutional Neural Networks

	eqn 1: use 	ext{max} so that the word “max” doesn’t appear in italics. italics are for variable names.

	Rest of the section: I don’t really buy your argument that people use overcompleteness to avoid the limitations of linear feature detectors. I’d say instead they use multiple layers of features. When you use two layers of any kind of MLP, the second layer can include / exclude any kind of set, regardless of whether the MLP is using sigmoid or maxout units, so I’m not sure why it matters that the first layer can only include / exclude linear half-spaces for sigmoid units and can only exclude convex sets for maxout units.

	Regarding maxout: I think the argument here could use a little bit more detail / precision. I think what you’re saying is that if you divide input space into an included set and an excluded set by comparing the value of a single unit against some threshold t, then traditional GLM feature detectors can only divide the input into two half-spaces with a linear boundary, while maxout can divide the input space into a convex set and its complement. Your presentation is a little weird though because it makes it sound like maxout units are active (have value > threshold) within a convex region, when in fact the opposite is true. Maxout units are active *outside* a convex region. It also doesn’t make a lot of sense to refer to “separating hyperplanes” anymore when you’re talking about this kind of convex region discrimination. 

Section 3.1
	Par 1: I’d argue that an RBF network is just an MLP with a specific kind of unit.

	Equation 2: again, “max” should not be in italics

Section 4.1
	Let me be sure I understand how you’re applying dropout. You drop the output of each micro-MLP, but you don’t drop the hidden units within the micro-MLP, right? I bet this is what leads to your performance improvement: you’ve made the unit of dropping have higher capacity. The way you group things to be dropped for the droput algorithm actually has a functional consequence. The way you group things when looking for linear versus non-linear feature detectors is relatively arbitrary. So I don’t really buy your story in sections 1-3 about why NIN performs better, but I bet the way you use dropout could explain why it works so well.

Section 4.2
	These results are very impressive!
	While reading this section I wanted to know how much of the improvements were due to global averaging pooling versus NIN. I see you’ve done those experiemnts later in section 4.6. I’d suggest bringing Table 5 into this section so all the CIFAR-10 experiments are together and readers won’t think of this objection without knowing you’ve addressed it.
	
Section 4.3
	Convolutional maxout is actually not the previous state of the art for this dataset. The previous state of the art is 36.85% error, in this paper:
http://www.cs.toronto.edu/~nitish/treebasedpriors.pdf

	Speaking of which, you probably want to ask to have your results added to this page, to make sure you get cited:
http://rodrigob.github.io/are_we_there_yet/build/

Section 4.4
	http://arxiv.org/pdf/1312.6082.pdf gets an error rate of only 2.16% with convolutional maxout + convolutional rectifiers + dropout. Also, when averaging the output of many nets, the DropConnect paper gets down to 1.94% (even when not using dropout / DropConnect). Your results are still impressive but I think it’s worth including these results in the table for most accurate context.

Section 4.5
	I think the table entries should be sorted by accuracy, even if that means your method won’t be at the bottom.

Section 4.6
	It’s good that you’ve shown that the majority of the performance improvement comes from NIN rather than global average pooling.
	It’s also interesting that you’ve shown that moving from a densely connected layer to GAP regularizes the net more than adding dropout to the densely connected layer does.

Section 4.7
	What is the difference between the left panel and the right panel? Are these just examples of different images, or is there a difference in the experimental setup?