Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper studies a fast algorithm for choosing networks with interleaved block types – BlockSwap. It passes a single minibatch of training data through randomly initialized networks and gauging their Fisher potential. 
The teacher-student network is used here to learn compressed network on a budget. They conduct various experiments, including cifar-10, imagenet, and coco-detection. All the experiments show the advance of proposed model, which is quite remarkable.
Overall, this paper is well organized, and very well written. The insightful experiments thoroughly discuss and compare the proposed method. 

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: 1) in the introduction, it's not clear what attention transfer means 
  2) why did they specifically choose these four blocks? 
  3) (3.1) substitute blocks $S$: ``each using $N$ lots of $N \times k \times k$ filters" should it be ``$N$ number of $k \times k$ filters"? 
  4) (3.2) ``consider a choice of layers $i = 1, 2, \dots, N_L$" what is $L$ and why does the number of layers depend on $L$? 
  5)  (3.2) Why is that specific $f$ chosen? 
  6)  (3.3) They could elaborate more on why they chose Fisher potential instead of other metrics for architecture selection. Currently, they only provided the intuition of the metric. Since the paper suggests the Fisher potential is a crucial part of their method, they could provide more theoretical justification about this choice. 
  7) (4.0) How did they decide on the hyperparameters? 
  8) In the introduction, they suggest that the major novelty of their method ``assigns non-uniform importance to each block by cheapening them to different extents", but in their method they only randomly assembled the mixed-blocktype models. 
  9) Section 4.1 suggests good mixed blocktype yields low error but they didn't address how good mixed blocktype can be found 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper introduces an approach to compressing a deep network so as to satisfy a given budget. In contrast to methods that replace all convolutional blocks with the same cheaper alternative, the authors argue that mixing different types of such cheap alternatives should be effective. To achieve this in a fast manner, they propose to randomly sample architectures with mixed blocktypes that satisfy the given budget and rank these architectures using Fisher information.

Originality:
- The approach is simple, but seems effective. Although the different components that make up the final method are known, they are put together so as to address a different task, in a way that I find interesting.

Methodology:
- From the paper, it is not entirely clear how sampling under budget constraints is achieved. I imagine that one could do it in a naive way, by choosing the first block uniformly randomly, and then removing the candidates that do not fit the budget for the subsequent blocks. This, however, would seem to give a different importance to the early blocks than to the later ones, since the former would essentially not have any budget constraints. I would appreciate it is the authors could explain their strategy in more detail.
- The assumption that the number of blocks in the student is the same as in the teacher seems constraining. A better architecture might be obtained by having fewer, wider blocks, or more, thinner blocks. Do the authors see a potential way to address this limitation?
- In principle, it seems that the proposed Fisher Information-based strategy could be used for general NAS, not just for compact architectures. Have the authors investigated this direction?

Related work:
- It seems to me that the literature review on compression/pruning is a bit shallow. I acknowledge, however, that most works do not tackle the scenario where a budget is given. However, Chen et al., "Constraint-aware Deep Neural Network Compression", ECCV 2018, do, and it would be interesting to discuss and provide comparisons with this work.

Experiments:
- I appreciate the ablation study, which answered several of my questions.
- In Table 1, it seems counterintuitive that the (negative) correlation becomes smaller as the number of mini-batches increases. Do the authors have an explanation for this?
- In Section 5, are the baseline compact networks all trained using the same Attention Transfer algorithm as for the BlockSwap ones?
- In Table 2, the budget values (P. (K)) seem fairly arbitrary? How were they obtained? They seem to match those of SNIP. Is this because the authors ran SNIP, and then set their budget accordingly?
- Below Fig. 3, the authors mention that they are generous with sparsity-based methods because they count the number of non-zero parameters. Note that several structured-sparsity compression methods have been proposed (Alvarez & Salzmann, NIPS 2016, 2017; Wen et al., NIPS 2016), which, by contrast with regular sparsity ones, would cancel out entire filters.

Summary:
Overall, I like the simple idea proposed in this paper. I would however appreciate it if the authors could clarify the way they sample the architectures and address my questions about their experiments.
review_title: Official Blind Review #3

Meta-review: Two reviewers recommend acceptance. One reviewer is negative, however, does not provide reasons for rejection. The AC read the paper and agrees with the positive reviewers. in that the paper provides value for the community on an important topic of network compression.
Decision: Accept (Poster)