Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper experiments with channel to channel sparse neural networks.
The paper is well written and the analysis is useful. The sparse connection is not new but has not been experimented on large-scale problems like ImageNet. One of the reasons for that is the unavailability of fast implementations of randomly connected convolutional layers.
The results displayed in figures 2, 3, 4, and, 5 show that sparse connections need the same number of parameters as the dense networks to reach to the best performance on the given tasks, but can provide better performance when there is a limited budget for the #parameters and #multiplyAdds. 
This paper is definitely informative but it does not reach to the conference acceptance level, simply because the idea is not new, the sparse connection implementation is poor, and the results are not very surprising.
review_title: Useful analysis 

final_score: rating: 7: Good paper, accept
review_content: The paper is about channel sparsity in Convolution layer.
The paper is well written and it elaborately discussed and investigated different approaches for applying sparsity.  The paper contains detailed literature review.
In result section, it showed the approach gives good results using 60% sparsity with reducing number of parameters, which can be useful in some embedded application with limited resource i.e. mobile devices.
The main point is that the paper needs more detailed investigation on different dropout schedule.
As mentioned implementation details section, they deactivate the connections by applying masks to parameter tensors, which is not helpful in speeding up the training and computation in convolution layer. They can optimize implementation to reduce computation time.

review_title: Less parameters using sparse random connection structure and more discriminative power by gradual densifying of the network

final_score: rating: null
review_content: In the conclusion section, the paper says: "For example, this simple method achieves AlexNet-level accuracy with fewer than 400K parameters and VGG level with roughly 1M parameters." I couldn't see these results in any of the graphs. By looking at the graphs, my understanding is there was no experiment conducted with AlexNet and the number of parameters needed to achieve VGG level accuracy is the same as the dense connected convolutional networks. Can you please clarify this statement from the conclusion? 
review_title: clarification on the results

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper aims to improve efficiency of convolutional networks by using a sparse connection structure in the convolution filters at each layer.  Experiments are performed using MNIST, CIFAR-10 and ImageNet, comparing the sparse connection kernels against dense convolution kernels with about the same number of connections, showing the sparse structure (with more feature maps) generally performs better for similar numbers of parameters.

Unfortunately, any theoretical efficiencies are not realized, since the implementation enforces the sparse structure using a zeroing mask on the weights.  In addition, although the paper mentions that this method can be implemented efficiently and take advantage of contiguous memory reads/writes of current architectures, I still find it unclear whether this would be the case:  The number of activation units is no smaller than when using a dense convolution of same dimension, and these activations (inputs and outputs) must be loaded/stored.  The fact that the convolution is sparse saves only on the multiply/addition operation cost, not memory access for the activations, which can often be the larger amount of time spent.

The section on incremental training is interesting, but feels short and preliminary, and any gains here also have yet to be realized.  The precision is no better than for the original network, and as mentioned above, the implementation of the sparse structure is no faster than the original.

Overall, the method and evaluations show that the basic approach has promise.  However, it is unclear how real gains (in either speed or accuracy) might actually be found with it.  Without this last step, it still seems incomplete to me for a conference paper.

review_title: seems promising, but incomplete without real gains

final_score: rating: null
review_content: Computational cost is reported in terms of multiply-adds.  On what real architectures (current or possible future) would this result in wall-time speedups?  On many arch implementations, elements stored in contiguous memory regions are faster to load (amortized per element) compared to spread-out random reads.  Is it possible to measure wall-time in addition to multiply-adds?

review_title: Wall-time measurements?

Meta-review: The reviewers agreed that the main contribution is the first empirical analysis on large-scale convolutional networks concerning layer-to-layer sparsity. The main concerns were that of novelty (connection-wise sparsity being explored previously but not in large-scale domains) and the importance given the current state of fast implementations of sparsely connected CNNs. The authors argued that the point of their paper was to drive software/hardware co-evolution and guide the next generation of, e.g. CUDA tools development. The confident scores were borderline while the less confident reviewer was pleased with the paper so I engaged the reviewers in a discussion post author-feedback. The consensus was that the paper presented promising but not fully developed nor convincing research.
Decision: Reject