Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes a new way to create compact neural net, named Atomic Compression Networks (ACN). An immediate related work is LayerNet, where a deep neural net is created by replicating the same layer. Here, this paper extends replication down to the neuron level. 

I am leaning towards rejecting this paper because the experimental setup is not well justified and a few important details are missing before conclusions can be drawn. I would like to ask a few clarification questions. Depending on the authors’ answers, I might be willing to adjust my rating. 

(1) Is there missing a delta in the first half of line 6 in Algorithm 1? 

(2) Throughout the experiments, for the same hyperparameter (e.g. Table 4 in A.2) do you run Algorithm 1 more than once and select the best sample architecture? If the answer is yes, summarizing all masks as one parameter will not be reasonable. Given a yes answer, I would also like to ask if the same number of samples have been considered for FC (for the same hyperparameter). 

(3) Is there any intuition behind why FC does a much worse job of fitting curves than ACN with much less parameters? This refers to Fig. 2, if we compare FC with 41 parameters to ACN with 18 parameters. I am confused because MSE on sampled points often goes down when we increase the number of parameters for the application of curve fitting.

(4) Convolution can be thought of as a special case of ACN. ConvNet is the default architecture for working on image datasets. Since MNIST and CIFAR are considered, why not also compare to ConvNet?

(5) The claims that “ACNs achieve compression rates of up to three orders of magnitudes compared to fine-tuned fully-connected neural networks with only a fractional deterioration of classification accuracy” is quite misleading. Given fully-connected neural networks achieve up to 528 times with also a fractional deterioration (Sec. 4.3), by presumably having a shallower architecture. 
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The paper describes a new method called Atomic Compression Network for constructing neural networks. The idea is straightforward. Basically, firstly create some neurons in random fashion, then reuse a subset of those neurons in each layer. The experiments shows ACN produces better accuracy than baseline models including a FC network, a Baysesina compression method, etc. for MINIST, etc. The paper also show ACN uses much less numbers of parameters and achieves similar accuracy when comparing with a large optima FC network on a set of datasets. 

Overall, I don’t support accepting this paper. First, I don’t think the proposed idea is very innovative. Please elaborate why this method seems to work well when comparing baseline models. Is it just randomly constructed network also perform well?  Secondly, I’m not convinced we will use this method to build network in real world applications. The model size is small, but in what cases this small model size matters? Is this a reliable way to create useful models? 

On page 7, in Figure 3, why logistic regression only has a single point in some of the plots?


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper explores the use of replicating neurons across and within layers to compress fully connected neural networks. The idea is simple, and is evaluated on a number of datasets and compared with fully connected, single layer, and several compression schemes. 

Strengths: a lot of nice experiments with clearly advantageous results are given.

Weaknesses: One obvious baseline missing is sparse compression, which can be achieved using either l1 regularization, or hard thresholding + fine tuning, both of which are easy to implement and appear in several works, e.g.

Scalable Neural Network Compression and Pruning Using Hard Clustering and L1 Regularization (Yang, Ruozzi, Gogate)
Training skinny deep neural networks with iterative hard thresholding methods (Yin, Yuan, Feng, Yan)

... many others just via googling ... 

Also, I think this work should be compared with compression schemes that work via kronecker product, which seem very similar to this scheme (but where the kronecker matrix is binary to produce replication)

Compression of Fully-Connected Layer in Neural Network by Kronecker Product (Zhou, Wu)
(more via google)

One obvious advantage of replication over kronecker product is lower complexity, but nonetheless, the methods belong in a similar family.

Otherwise, I think the work makes sense, the idea is nice, and the results show promise!

After rebuttal: I have read the rebuttal and the authors have basically addressed all my concerns. It is a bit disappointing that simple L1 regularization can give competitive results, but the fact that the authors are willing to do the experiment and incorporate the results convinces me that there's nothing being hidden here, and the reader can make a fair and informed conclusion, so I have no more complaints.
review_title: Official Blind Review #2

Meta-review: This paper proposed a very general idea called Atomic Compression Networks (ACNs) to construct neural networks. The idea looks simple and effective.  However, the reason why it works is not well explained.  The experiments are not sufficient enough to convince the reviewers.
Decision: Reject