Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes a method which is able to learn a hardware performance predictor for a wide range of devices with only a few training samples(such as 10), through first one-time meta-learning a hardware embedding with the samples from a large device-architecture-performance pool and then adapt to the seen/unseen devices through inner-gradient manner with only a few training samples; throughout the process, this work treats the hardware devices as black boxes outputing the performance metric, thus not holding the statistical assumption about hardware devices' performance.


main_review: Overall this work 
1. Proposes a good method to predict the hardware performance of many devices while only requiring a few training samples without the statistical assumption between the hardware devices and their performance. 
2.This work will certainly expedite and facilitate the process of hardware-aware NAS. 
3.The writing is clear and to the points, pleasant to read

However, I have a couple of concerns which would be great if the authors can address them.

1. Algorithm-task level generality: It would be better to show some end-to-end results on more datasets/tasks, or model types(if possible), e.g., GNN, transformers...

2. More device variety and metric types: Different devices may have different calculation process for some performance metric, like throughput. For latency metric, most of the devices have a accumulative relation between layer(component)-wise latency and the system-latency. However, for throughput on architectures like cell-phones, Eyeriss or the FPGA in HW-NAS-Bench, the final throughput equals the inverse of system-latency, as they execute the workload layers sequentially. However, some architectures, such as the one in DNNBuilder,  execute the workload layers in a pipelined fashion. Thus the throughput will instead be the inverse of the bottleneck-layer latency. It would great to show some results/insights on how meta-learning on one type of devices generalize to the other type.

DNNBuilder: an Automated Tool for Building High-Performance DNN Hardware Accelerators for FPGAs
limitations_and_societal_impact: Yes, it's addressed
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper aims to improve the estimation of latency of neural models on unseen hardware devices. Its contributions are:
- Utilize a collected pool of devices and architectures to meta-train the predictor.
- Adapt the predictor to unseen hardware devices/platforms by considering hardware device embedding -- the latencies of the unseen devices on a fixed set of reference neural architectures.
- Introduce a "hardware-adaptive modulator" to improve the prediction.
- Reduce the latency estimator building cost of N devices from O(N) to O(1).
- Apply the latency estimator to latency-constrained NAS.
main_review: **Originality**
The contributions of this paper are quite outstanding:
- Improve SOTA latency predictor by formulating it as a few-shot regression problem.
- Use latencies on reference architectures as hardware embeddings.
- Combine with exisiting NAS framework to accelerate latency-constrained NAS.
- Good combination of existing techniques: GCN and MLP to encode architectures and devices, meta learning to adapt the predictor to unseen devices.
- Improve number of samples required to adapt to an unseen device.

**Quality**
- The proposed methods are sound and convincing.
- The empirical results support the arguments.
- The meta-learning approach looks simple and effective.
- The hardware-adaptive modulator needs more explanation.

**Clarity**
Some explanations in the paper are not clear. I hope the author can address the problems and make it a solid paper.
- Table 2: what are HW-condition, few-shot adapt and z modulator? It will be clearer to relate to the sections explicitly.
- Hardware-adaptive modulator seems to be an important contribution. However the explanation is not very clear -- How is the hardware-adaptive modulator be designed, implementated and trained?
- Why is the latency model builing time so long for ProxylessNAS and OFA-layerwise? They use layer-wise predictor which are the summation of profiled per-layer latency. Should be fast.
- Check mathematical notations. For example: f vs f^{\tau}, V_h^{(0)} -- what is (0)? 2) 
- Figure 3 and 5: What is the unseen device? How to select the device in the pool for each point in the x-axis? 

**Significance**
- The paper has reduced the latency estimator building cost by meta-learning a few-shot regression model that can generalize across hardware devices.
- The source code is available to reproduce the results. It would be more helpful if the dataset (latency measurement from the device pool) is also available.

**Post-rebuttal updates**
Thank you to the authors for answering my questions and clarifying the technical details. I am convinced that meta learning + hardware performance predictor is a solid contribution. I will increase my score to accept this paper.
limitations_and_societal_impact: The authors have addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a hardware latency modeling and prediction methodology that adapts itself to multiple devices through few shot meta-learning. The overall goal of the paper is to create a hardware predictor for use within hardware-aware neural architecture search (NAS). 

A graph convolutional network (GCN) is first trained using latency measurements from a number of devices (18) then few-shot learning (with as little as 10 measurements) is done to adapt the predictor to predict the latency of unseed devices (3 devices are similar to the devices used for training, and 3 devices are different). The problem addressed with this work is important as it is often impractical to get numerous measurements from real hardware devices. Few-shot metalearning is one way to adapt a latency predictor for a new device and this is the first time I have seen metalearning applied to the specific problem of hardware latency for different devices. 

main_review: While the results in the paper are strongly in favour of the proposed methodology compared to prior work, there are many questionable assumptions in those comparisons and missing ablations that need to be addressed to prove the efficacy of the proposed meta-learning approach. I am strongly in favour of seeing this work published but I find it hard to give a high score without addressing the major issues listed below.

### Strengths:
The topic is important and interesting and the use of meta-learning is spot-on. This is a great way to formulate the hardware latency prediction problem and provide a low-cost way to adapt to different devices.

### Major issues:
- Without code release, it is difficult to trust the results.
- Choice of "unseen device" and "unseen platform" seems to be hand-picked for good results. An ablation would really help in understanding the pros/cons of using HELP. For example, what happens with fewer train devices? What happens when you swap train and test devices? What happens when your "completely unseen platforms" more complex? Latency prediction for ASIC, FPGA and MCU devices is *very* simple and could usually be modeled analytically by counting the number of compute cycles. Additionally, the "unseed devices" CPU/GPU/Pixel are **very** similar to devices in the meta-train pool for which many training samples are provided.
- Some results are overstated. When comparing the number of samples used to train each predictor, all training points should be reported in some way. For HELP in Table 3 for example, it only has '10' measurements needed but the actual number is 900*18+10=16210. I do understand that most of this is a fixed cost that is not incurred with each new device but it still needs to be clearly spelled out -- I found the current presentation of results slightly misleading in some places.
- Again in Table 3, it is difficult to discern **why** HELP is better than BRP-NAS? Both predictors are based on GCNs. The only reason I can think of has to be related to the fact that HELP is trained on more data points (18 devices * 900 points) and so that data allows it to achieve a better prediction. I would've expected HELP to be an *approximation* of BRP-NAS so to speak. A discussion of the reason behind this improvement is warranted.

### Minor issues:
- Equation 3 is odd. It combines set notation and function notation. It is unclear to me what it is saying.
- Why are some results missing? For example in Table 3 the layerwise predictor is not present for 4 devices.
- In Table 3, why is the FLOPs correlation coefficient so high for the GPU when Fig. 4(a) suggests that it should be very low?
- I don't view this as the main contribution of the work, but in the comparison to baseline metalearning approaches like meta-SGD in Fig. 5, the baseline seem to be completely failing - is it just untuned? or is this a trivial baseline? Or is the fact that you use a modulator making all the difference? A discussion on this would help.
- Table 1 is misleading in my opinion.

## Post-rebuttal
All of my concerns were addressed - please see the conversation below with the authors. I will increase my score to reflect that I am satisfied with their responses. As mentioned below: With the code release and additional ablations (especially ones that show that this method generalizes with different devices in test vs. train pool), I would like to vote for accepting this paper. I think it is a solid contribution that straddles the boundary between two important areas: metalearning and hardware device modeling. I would suggest adding many of the ablations in this rebuttal to the submitted paper (or its appendix).
limitations_and_societal_impact: The main limitations of the paper are stated in the main review. I do not have comments on the societal impact as this work is quite technical and abstracted away from the important societal issues in the machine learning community.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Modern neural architecture search methods often require us to train a *latency prediction model* which can predict the latency of any candidate network architecture within a human-defined search space. These are commonly obtained by (i) sampling a set of random network architectures from the search space, (ii) benchmarking each one on the target inference hardware device, and (iii) training a regression model on these <architecture, latency> pairs in order to generalize to new, unseen architectures. The submission explores the idea of (i) training a single model which can generate predictions for many different hardware devices, and (ii) using meta-learning to allow the model to generalize to a new hardware device with little additional data (e.g., 10 samples).
main_review: **Note**: I'll focus on the Neural Architecture Search / latency aspects of the submission (rather than the meta-learning aspects), since that's what I'm most familiar with.

# Overview
The paper's central claim is that meta-learning can be used to train a latency prediction model which can generalize to new (unseen) hardware devices with minimal training data (e.g., benchmarks of 10 random architectures from a large search space). The main steps are to (i) select a "seed" set of hardware devices, (ii) collect benchmark latencies for ~900 - 4000 random architectures for each device, and (iii) use the resulting <architecture, device, latency> triples to train a model which can predict latencies for architectures and hardware devices not seen in the training dataset (Section 4).

One of the inputs to the model is a representation of the target hardware device. This is generated in an interesting/clever way. The authors propose sampling/fixing 10 random architectures from the search space, and using the vector `<latency of architecture 1, latency of architecture 2, ...>` as their representation of the hardware device (Section 3.2). The submission proposes a "z-modulator" (weight adaptation mechanism) to adjust the network weights for a target hardware device.

The submission provides two sets of evaluations to test the proposed method. First (Figure 3, Figure 4, Figure 5, Table 2): evaluating how well the method performs on new hardware devices that were not seen during training, and comparing against a set of standard baselines (e.g. using FLOPS instead of on-device performance, comparing against a previously proposed method called BRP-NAS which trains a separate latency predictor for each device). In the authors' experiments, the newly proposed method significantly outperforms these baselines. And second (Table 3, Table 4, Table 5, Figure 6): plugging the proposed latency estimator into previously proposed NAS algorithms (e.g., Once-for-All, MetaD2A) and measuring the end-to-end quality impact on NASBench-201 or the Once-for-All search space. Both sets of experiments provide evidence that the latency predictor, once trained, can generalize reasonably well to new hardware devices with only ~10 benchmark samples.

**Clarity**: The submission seems clear and well-written.

**Significance**: The problem of trying to reduce the cost of training latency models for new hardware-devices seem well-motivated, and this is the first paper I've seen which tries to directly address this problem. It also seems like a nice application of ideas from few-shot learning. The results might be useful for practitioners who wish to simultaneously target a large number of different hardware devices.

**Incorrect Use of Big-O Notation**: The submission makes the following claim on Line 209 and Table 1 ("cost of NAS and latency estimation"):  "Since conventional approaches require to collect a large number of samples and train a device-specific latency estimator for each target device, while HELP only needs to collect 10 samples per device, HELP reduces the time complexity of obtaining latency estimations on N devices from O(N) to O(1)."

This seems like an incorrect use of big-O notation. The cost of training models for N hardware devices is reduced from `900*N` to `10*N+C`, where C is the cost of training the meta-model. Both of these costs are technically `O(N)`, since the cost grows linearly with the number of target devices as N goes to infinity. **Because the claims about big-O complexity are technically incorrect, I would not feel comfortable accepting the paper until the issue is addressed, or until the authors clarify their meaning (if I misunderstood their claims).**

**Caveats (different devices in same family may be strongly correlated)**: While the ability to adapt to a new hardware device with only 10 samples is impressive, it's possible that this is possible because many of the devices in the submission's pool are quite similar to each other. For example, it might be reasonable to expect Intel Xeon Silver and Intel Xeon Gold CPUs to have strongly correlated performance. This is significant because (i) it's not obvious to me that we actually need separate neural network architectures for the two CPUs, and (ii) it's possible that we could get an excellent predictor for Intel Xeon Gold by taking latency predictions for Xeon Silver and rescaling them a bit, without any kind of fancy meta-learning tricks.

Overall, the target devices the authors evaluate can be broken down into the following categories (Appendix A.2):
1. Desktop GPU (5 devices, all made by NVIDIA).
2. Edge GPU (1 device, made by NVIDIA).
3. Server CPU (3 devices, all made by Intel).
4. Mobile Phone CPU (5 devices, made by Samsung and Qualcomm) 
5. Raspberry Pi (1 device).
6. ASIC (1 device, made by Eyeriss).
7. FPGA (1 device).

I have mixed feelings about this dataset. On the positive side, it covers a diverse set of hardware devices of different types/sizes. On the negative side, if we split the list of devices randomly, we might end up with an overly optimistic estimate of the model's performance. Given a random split, with high probability, at least one mobile desktop GPU will end up in the training set and another very similar device will end up in the test set.

For this reason, I think that evaluating the latency predictor's ability to generalize across device families is importance. The submission takes a first step towards addressing this in Table 5, where the meta-model is trained only on CPUs but is evaluated on GPUs + Intel CPUs. This suggests that in terms of end-to-end performance (plugging in the latency predictor to a previously proposed NAS algorithm), the current method gets reasonable generalization across hardware families. But it would be helpful to see a more careful analysis of the predictor's raw performance as well.

# Additional Detailed Comments
Figure 3: Each line shows a shaded error region, but I didn't notice an explanation for how these shaded regions were derived (e.g., worst run vs. 1 standard deviation below the mean).
limitations_and_societal_impact: I believe the authors' discussion of limitations and societal impact in Appendix D adequately addresses the NeurIPS review requirements.

I would be interested in seeing a more detailed study of the meta-model's generalization performance within vs. across hardware families, but I think the submission already has enough interesting results for me to recommend its acceptance.
review_title: null

Meta-review: This is a timely paper. It addresses a relevant problem using novel meta-learning methods and obtains strong results.
All reviewers were in favour of acceptance.

There was a long internal discussion with dozens of posts about the comparison with BRP-NAS. The authors first misleadingly claimed that the results shown in the paper are consistent with the ones in the BRP-NAS paper, but they are clearly not (with a huge difference in Spearman correlation of roughly 0.8 vs. 0.99!). During the review process, the authors' code for running BRP-NAS was checked and found to be fine, so that is not the reason for the performance difference. 
The most likely explanations for the different performance of BRP-NAS are (1) The data may be measured differently and/or (2) BRP-NAS may need different hyperparameters for new data. Concerning the latency measurement methodology, the BRP-NAS paper described their detailed methodology for measurements in detail: they pruned NAS-Bench-201 graphs and optimized them before running them on the device, then discarded the first few measurements and averaged multiple runs. The current paper doesn't give full details about the measurement pipeline to the paper, and I strongly encourage the authors to add more details. In the further discussion, the authors indeed explained that they use a less controlled (and arguably, more realistic) noise than in the BRP-NAS paper, which reduces the predictability. 

The authors also added a convincing comparison to BRP-NAS on BRP-NAS' own benchmark LatBench.

An orthogonal concern that was brought up in the reviewer discussion was that Table 5 is overclaiming: OFA + HELP obviously does *not* take a total time of 26 seconds, as OFA takes very substantial time to construct the one-shot model. I strongly encourage the authors to fix this by reporting the total time spent, in order to present their contribution realistically.
Another concern that came up was that authors were given an additional page this year and encouraged to use this page to discuss the broader impact and societal impact; the authors instead deferred that discussion to the appendix, which could be seen as cheating the page limit. I strongly encourage them to include a discussion in the main paper for the final version and move some less important details to the appendix instead.

Overall, the work addresses an important and relevant problem; the use of meta-learning for NAS is much underexplored and thus very timely. I am therefore clearly in favour of acceptance.
Decision: Accept (Spotlight)