Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper addresses the problem of limited resources that may generally be available at inference time (as compared to the lack of constraints during training time). The paper addresses architecture search as well as model compression by simultaneously optimizing all layers/weights/submodules of the network. The approach is motivated well in terms of comparisons to other work: Although I am not an expert in this particular area of model compression, the intuitive comparisons and mentions of prior work felt satisfying and informative. This problem is definitely timely, and the paper showed results of improved inference speed and memory footprint on both a smaller 10-way classification task as well as a larger 1000-way one. 

My current decision is a weak accept, for a well-written paper and an interesting problem, but in the presence of some concerns and suggestions as listed below. 

(1) My main concern is that without weight regularization, this optimization problem seems ill-posed. Looking at algorithm 1 specifically, consider the case where j=10 and the learned alphas are indeed sparse, like (0.1, 0, 0,…, 0, 0.9). The cost of J=0 here might be very high (i.e., large, slow, expensive model) and cost of J=9 might be very low (i.e., small fully-connected network that basically maps to an identity function). Here, the cost during optimization could be very low, and the results could be very accurate, but the weights of J=0 may be very high such that the entire work of getting the right answer is from this model. More generally, a low alpha can be attained for any model by simply increasing the network weights themselves. This doesn’t seem to be taken care of anywhere in this paper; if not addressed in any way, it would be a large reason for me to suggest rejecting this work.

(2) For the experiments, both of the chosen tasks were classification. Although one was smaller/easier and one was larger/harder, it would serve to be much more convincing if a task requiring regression and/or higher-dimensional output were tested. For these classification tasks, I can imagine that for certain model changes, the decision boundaries may remain the same even if the network itself is changing in detrimental ways. Thus, a task such as an image-to-image depth perception task (or other tasks in this category) would solidify the results and make it more convincing that DARC does indeed help.

Minor: 

(a) The motivation of going from L0 to L1 for the cost constraint seems relevant also to the choice of h being the convex combination of different models. Can the motivation and problem setup address these motivations together, instead of separately?  

(b) Although this paper defined R(h) and it also defined h=(\sum (alpha_j * h_j), it would still be helpful for the reader if the paper could explicitly define R(\sum (alpha_j * h_j)) somewhere. Although I know what was intended was to compose these h_j functions as sequential stages to pass data through, the sum notation seems a bit misleading like you may be averaging the final predictions/outputs from different models and computing the loss on these averaged results. 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new architecture search method called "DARC" that utilizes a differentiable objective function. Since a naive formulation of architecture search is reduced to a combinatorial optimization which is not differentiable, the optimization requires much computational cost. To overcome this difficulty, this paper proposes a L1-norm relaxation and apply such relation in a layer-wise manner. The method shares a similar spirit with NAS, but the proposed model is more like "model selection" from a fixed candidates, and thus there is a Rademacher complexity guarantee. The effectiveness of DARC is justified by thorough numerical experiments.

Although the idea is rather straight-forward, the effectiveness of the method is well supported by the thorough experiments. In particular, it works as a model compression method and shows a favorable performances compared with SOTA methods.

The pros and cons are summarized as follows.
Pros:
- The proposed method is simple and rather easy to implement.
- The numerical experiments show the proposed method gives favorable performances compared with the existing methods.

Cons:
- The idea itself is rather straight-forward.
- The theoretical analysis is instructive but its derivation does not require new techniques.
- Compared with NAS, the proposed method should prepare a set of candidates which restrict the search space. This ensures generalization but limits its flexibility.

More comments:
- I could not see how efficient the method is in terms of memory. It prepares several models in each layer, thus it requires large memories. Can it be performed on more large networks?
- The setting of C_j affects the result. How did you set C_j in the experiments?


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The authors proposed a new gradient-based architecture search method that tries to find more efficient alternatives starting from the pre-trained model. The approach is similar to DARTS (Liu et al., 2019) with a budget constraint, such as size and throughput. One major difference is to modify the update of architectural parameters, i.e., mixing weights of all candidate operations, to induce the sparsity rather than to keep the weighted sum of all possible operations. Another difference is that it starts from a well-defined architecture with pre-trained weights. It is simple to apply, but hard to tell. The recognized strengths and concerns are as follows.

< Strengths >
1. The proposed update method to induce the sparsity of the architecture during the search seems applicable to all gradient-based search methods, and more important for the budget-constrained search minimizing the discrepancy between the ensemble architectures during the search and the final architecture derived after the search.
2. The setup to initialize with the well-trained model seems practically useful rather than starting the search with a random initial model.
3. The proposed methods seem easy to apply.

< Concerns & Questions >
1. The algorithm does not seem efficient because it continues training iteratively while increasing the strength (\lambda) of the L1 regularizer by the estimated cost until the budget constraint is met.
2. No details to calculate the cost C_l0(\alpha) for each architecture candidate (e.g., the throughput cost of the architecture with a 3x3 DS operation in layer 1).
3. No number of operations (e.g., # FLOPs, # MACs) is reported. Since the throughput is strongly dependent on the underlying hardware, the number of operations also needs to be shown as a more general estimation of the model inference latency in various hardware devices.
4. The models under comparison are out-dated. It needs to be compared with the latest models designed with computational efficiency in mind, e.g., EfficientNet (Tan et al. 2019a), MixNet (Tan et al. 2019b), MobileNet V3 (Howard et al., 2019).

- Liu et al., DARTS: differentiable architecture search, ICLR 2019
- Tan et al., EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, ICML 2019a
- Tan et al., MixConv: Mixed Depthwise Convolutional Kernels, arxiv:1907.09595, 2019b
- Howard et al., Searching for MobileNetV3, arxiv:1905.02244, 2019
review_title: Official Blind Review #1

Meta-review: The paper is on the borderline. A rejection is proposed due to the percentage limitation of ICLR.  
Decision: Reject