Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a way of exploring compressible NNs across different architectures, sparsity levels, and quantization levels during training. The proposed framework is called Unified DNAS for Comppresssible (UDC) NNs and it can yield compressed NNs that show better accuracy-model size trade-offs than prior work. 
strengths_and_weaknesses: The paper tackles an important problem and the proposed solution seems reasonable. The authors claim that UDC outperforms prior work in terms of accuracy-model size and the experiments seem to support this. A brief description of the prior methods that the UDC framework is compared to such as HAQ, Gong et al. [27], McDonnel, [51], FBNet, MBNetV2, Choi et al., [18] would be really helpful for readers. 

In my opinion, the paper is a bit hard to follow. Specifically, Introduction could be a lot clearer with a more structured presentation and perhaps with a few subsections.
questions: Please see the previous section. 
limitations: The authors discussed the limitations of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a differentiable NAS approach targetting compressible NN for NPUs with low memory footprint.
The approach conducts a joint search over NN architecture, weight bitwidths and layer-wise weight sparsity levels to find models with the smallest model size.
strengths_and_weaknesses: Strengths:
- A theoretical lower-bound of the storage size that is cheap to be computed.
- Clear three-stage training process to deal with quantization and sparsity.
- Improved pareto-front of model size vs accuracy over previous work.
- The search algorithm has better sample efficiency vs random method.

Weaknesses:
- Make it clear that the target device is mobile-based NPU with limited model storage.
- Models targetting small NPU and MCU have strigent latency and memory utilization constraints, which are not discussed in this paper.
- Limited comparison with other NAS algorithms.
- There are other NAS papers for MCU, please also cite and compare to them. For example: https://arxiv.org/abs/2010.14246
questions: - What is the decompression overhead when the compressed model is executed on the target device?
limitations: The limitations are adequately discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors present am updated Differential Neural Architecture Search algorithm, UDC, which incorporates model compression features such as weight sparsity and quantization as part of the search space for the ideal (high accuracy, small model size) neural network (NN) model. In addition, keeping in mind the constraints typically faced by TinyML models, the authors also enforce a strict limit on the model size to ensure it can be implemented in resource constrained IoT applications. Further, the presented algorithm enables effective exploration of accuracy vs model size tradeoff. Lastly, the paper demonstrates the pareto dominance of NNs designed by UDC compared to prior art for a range of ML benchmarks and varying hardware constraints on Ethos U55 NPU.
strengths_and_weaknesses: The paper presents a thorough comparison of UDC against past techniques that attempt similar co-optimization of ML models while accounting for hardware constraints. UDC tackles a wider range of constraints compared to all listed in prior art. 

Section 3 and 4 describe the details of the design model and the proposed DNAS algorithm. These sections were somewhat hard to follow, however, it is understandable considering the mathematical nature of problem. It is not obvious what these variables denote, ξ & ν, kindly add a description. 

Section 6 shows a comparative study of generating NNs using UDC and other prior art for state-of-the-art (SOTA) ML benchmarks under different memory size constraints. Figure 4 illustrates the superiority of UDC in terms of accuracy and model size, could the authors also shed light on the runtime it takes to converge to the final NN design UDC picks compared to other approaches (e.g., MCUNet). Considering UDC accounts for a larger set of constraints compared to other approaches it would be interesting if it can demonstrate comparable runtimes too. 

Figure 5b, compares UDC results to random search, and the authors do share some insight on UDC runtime compared to training a baseline network (~2x), if they could share this runtime information it would be handy. Also note that authors should be careful that while comparing to random search the resource utilization for UDC should be similar e.g., running equal number of parallel GPU instances for a fair comparison. It would be interesting to know how long it takes to converge the random search to the optimal design picked up by UDC (withing reasonable time constraints of course).
 
Additionally, could authors share how they selected the memory size constraint values for the experiments shared in the paper, is there some understanding if the superiority of UDC holds for larger/smaller mem constraints and if yes how a user could understand if they fall in that range. 

To conclude, this work promises a very useful optimization tool for designing small but high accuracy NNs that can fit on typical IoT devices servicing TinyML applications.

questions: Refer to questions addressed in Strengths/Weakness section.
limitations: Authors do not present any data showing the timing performance of UDC compared to other approaches, some perspective on that would be helpful. Also any explanation on how the memory size constraints were selected for presented experiments would be helpful.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper is on model compression (quantize and prune) with hardware constraints. The proposed search method is an extension of differentiable NAS to learn weight sparsity ratios and bit-widths per layer. Conceptually, this work combines previous ideas, such as DNAS (of layer width, depth, or operator) and learning of layer-wise bitwidth and sparsity, and yield pareto-optimal results on model size vs. accuracy. 

strengths_and_weaknesses: - The methods used in the paper are not new. But this work is a sort of combination of well-known techniques. Thus, making it work, i.e., DNAS for model architecture with pruning and quantization, is a new contribution. The related work part discusses the differences from prior work, and the related work is adequately cited to my knowledge.
- The submission is technically sound with mostly empirical results. But it does not evaluate the weaknesses. For example, the searching costs of the proposed algorithm. 
- The writing is okay. The design decisions and the proposed algorithm are described with details. Since the major contribution, to my understanding, is the practical issues of making DNAS with compression techniques working, I think more details are necessary to help readers and practitioners to use the findings in this work. For example,  experiment setups and code examples are helpful, especially if no source code available. 
- The results are useful but not significant. In some scenarios with strict limitations on model size, the proposed method could be used. The reported benefits (model size vs. accuracy) are not unique.
questions: - What is the role of the Vela compiler in the proposed framework? Are the results on model size, e.g., in Fig. 4 & Fig. 5, from the use of the Vela compiler afterwards? 
- What are the performance results, in terms of latency/inference time, of the searched, compressed, and deployed models? Setting the optimization goal to be smallest model size is reasonable if the model size matters the most. But what is the impact on inference performance? The work mentioned using unstructured pruning, would that hurt the performance? Please comment on that.
- Can you compare the searching costs with other work? For example, the GPU days and training FLOPs could be helpful in understanding this work when compared with others.
limitations: The authors have addressed the potential societal impact. 
The limitations are not addressed enough.
review_title: null

Meta-review: metareview: In this paper, the authors present a new way to obtain compressible neural networks to fit on resource-constrained NPU-based hardware.

Initial reviews were mixed, but the authors successfully managed to respond to reviewers' concerns during the rebuttal period. Several reviewers pointed out clarity issues, but (1) some of these issues came from reviewers not reading the paper carefully enough, and (2) others were properly addressed by the authors. I also want to acknowledge that the most negative review is a short one, falling below NeurIPS quality standards.

After discussion, all reviewers are leaning towards acceptance, agreeing that the paper successfully demonstrates the superiority of the proposed method vs. existing relevant baselines. As a result, I also recommend acceptance.
confidence: Less certain
Decision: Accept