Question: Does the reported D(best) include points in the validation set?
Evidence: Our evaluations aim to answer the following questions: Q(1) Can PRIME design accelerators tailored for a given application that are better than the best observed configuration in the training dataset, and comparable to or better than state-of-the-art simulation-driven methods under a given simulator-query budget? Q(2) Does PRIME reduce the total simulation time compared to other methods? Q(3) Can PRIME produce hardware accelerators for a family of different applications? Q(4) Can PRIME trained for a family of applications extrapolate to designing a high-performing accelerator for a new, unseen application, thereby enabling data reuse? Additionally, we ablate various properties of PRIME (Appendix A.6) and evaluate its efficacy in designing accelerators with distinct dataflow architectures, with a larger search space (up to 2.5×10 114 possible candidates).
Baselines and comparisons. We compare PRIME against three online optimization methods that actively query the simulator: (1) evolutionary search with the firefly optimizer (Yazdanbakhsh et al., 2021a) ("Evolutionary"), which is the shown to outperform other online methods for accelerator design; (2) Bayesian Optimization ("Bayes Opt"), (3) MBO. In all the experiments, we grant all the methods the same number of feasible points. Note that our method do not get to select these points, and use the same exact offline points across all the runs, while the online methods can actively select which points to query, and therefore require new queries for every run. "D(Best in Training)" denotes the best latency value in the training dataset used in PRIME. We also present ablation results with different components Architecting application-specific accelerators. We first evaluate PRIME in designing specialized accelerators for each of the applications in Table. We train a conservative surrogate using the method in Section 4 on the logged dataset for each application separately. The area constraint α (Equation) is set to α = 29 mm 2 , a realistic budget for accelerators. Table summarizes the results. On average, the best accelerators designed by PRIME outperforms the best accelerator configuration in the training dataset (last row Table), by 2.46×. PRIME also outperforms the accelerators in the best online method by 1.54× (up to 5.80× and 6.62× in t-RNN Dec and t-RNN Enc, respectively). Moreover, perhaps surprisingly, PRIME generates accelerators that are better than all the online optimization methods in 7/9 domains, and performs on par in several other scenarios (on average only 6.8% slowdown compared to the best accelerator with online methods in M5 and M6). These results indicates that offline optimization of accelerators using PRIME can be more data-efficient compared to online methods with active simulation queries. To answer Q(2), we compare the total simulation time of PRIME and the best evolutionary approach from Table on the MobileNetEdge domain. On average, not only that PRIME outperforms the best online method that we evaluate, but also considerably reduces the total simulation time by 93%, as shown in Figure. Even the total simulation time to the first occurrence of the final design that is eventually returned by the online methods is about 11× what PRIME requires to fine a better design. This indicates that data-driven PRIME is much more preferred in terms of the performance-time trade-off. The fact that our offline approach PRIME outperforms the online evolutionary method (and also other state-of-the-art online MBO methods; see Table) is surprising, and we suspect this is because online methods get stuck early-on during optimization, while utilizing offline data allows us PRIME to find better solutions via generalization (see Appendix B.1.1).
Table: Optimized objective values (i.e., latency in milliseconds) obtained by various methods for the task of learning accelerators specialized to a given application. Lower latency is better. From left to right: our method, online Bayesian optimization ("Bayes Opt"), online evolutionary algorithm ("Evolutionary"), and the best design in the training dataset + validation dataset. On average (last row), PRIME improves over the best in the dataset by 2.46× (up to 6.69× in t-RNN Dec) and outperforms best online optimization methods by 1.54× (up to 6.62× in t-RNN Enc). The best accelerator configurations identified is highlighted in bold.  Architecting accelerators for multiple applications. To answer Q(3), we evaluate the efficacy of the contextual version of PRIME in designing an accelerator that attains the lowest latency averaged over a set of application domains. As discussed previously, the training data used does not label a given accelerator with latency values corresponding to each application, and thus, PRIME must extrapolate accurately to estimate the latency of an accelerator for a context it is not paired with in the training dataset. This also means that PRIME cannot simply return the accelerator with the best average latency and must run non-trivial optimization. We evaluate our method in seven different scenarios (Table), comprising various combinations of models from Table and under different area constraints, where the smallest set consists of the three MobileNet variants and the largest set consists of nine models from image classification, object detection, image segmentation, and speech recognition. This scenario is also especially challenging for online methods since the number of jointly feasible designs is expected to drop significantly as more applications are added. For instance, for the case of the MobileNet variants, the training dataset only consists of a few (20-30) accelerator configurations that are jointly feasible and high-performing (Appendix B.2-Figure).
Table shows that, on average, PRIME finds accelerators that outperform the best online method by 1.2× (up to 41%). While PRIME performs similar to online methods in the smallest three-model scenario (first row), it outperforms online methods as the number of applications increases and the set of applications become more diverse. In addition, comparing with the best jointly feasible design point across the target applications, PRIME finds significantly better accelerators (3.95×). Finally, as the number of model increases the total simulation time difference between online methods and PRIME further widens (Figure). These results indicate that PRIME is effective in designing accelerators jointly optimized across multiple applications while reusing the same dataset as for the single-task, and scales more favorably than its simulation-driven counterparts. Appendix A.4 expounds the details of the designed accelerators for nine applications, comparing our method and the best online method.  Accelerating previously unseen applications ("zero-shot" optimization). Finally, we answer Q() by demonstrating that our data-driven offline method, PRIME enables effective data reuse by using logged accelerator data from a set of applications to design an accelerator for an unseen new application, without requiring any training on data from the new unseen application(s). We train a contextual version of PRIME using a set of "training applications" and then optimize an accelerator using the learned surrogate with different contexts corresponding to "test applications," without any additional query to the test application dataset. Table shows, on average, PRIME outperforms the best online method by 1.26× (up to 66%) and only 2% slowdown in 1/4 cases. Note that the difference in performance increases as the number of training applications increases. These results show the effectiveness of PRIME in the zero-shot setting (more results in Appendix A.5).
Applying PRIME on other accelerator architectures and dataflows. Finally, to assess the the generalizability of PRIME to other accelerator architectures, we evaluate PRIME to optimize latency of two style of dataflow accelerators-NVDLA-style and ShiDianNao-style-across three applications (Appendix C details the methodology). As shown in Table, PRIME outperforms the online evolutionary method by 6% and improves over the best point in the training dataset by 3.75×. This demonstrates the efficacy of PRIME with different dataflows and large design spaces.
Our evaluations aim to answer the following questions: Q(1) Can PRIME design accelerators tailored for a given application that are better than the best observed configuration in the training dataset, and comparable to or better than state-of-the-art simulation-driven methods under a given simulator-query budget? Q(2) Does PRIME reduce the total simulation time compared to other methods? Q(3) Can PRIME produce hardware accelerators for a family of different applications? Q(4) Can PRIME trained for a family of applications extrapolate to designing a high-performing accelerator for a new, unseen application, thereby enabling data reuse? Additionally, we ablate various properties of PRIME (Appendix A.6) and evaluate its efficacy in designing accelerators with distinct dataflow architectures, with a larger search space (up to 2.5×10 114 possible candidates).
Baselines and comparisons. We compare PRIME against three online optimization methods that actively query the simulator: (1) evolutionary search with the firefly optimizer (Yazdanbakhsh et al., 2021a) ("Evolutionary"), which is the shown to outperform other online methods for accelerator design; (2) Bayesian Optimization ("Bayes Opt"), (3) MBO. In all the experiments, we grant all the methods the same number of feasible points. Note that our method do not get to select these points, and use the same exact offline points across all the runs, while the online methods can actively select which points to query, and therefore require new queries for every run. "D(Best in Training)" denotes the best latency value in the training dataset used in PRIME. We also present ablation results with different components Architecting application-specific accelerators. We first evaluate PRIME in designing specialized accelerators for each of the applications in Table. We train a conservative surrogate using the method in Section 4 on the logged dataset for each application separately. The area constraint α (Equation) is set to α = 29 mm 2 , a realistic budget for accelerators. Table summarizes the results. On average, the best accelerators designed by PRIME outperforms the best accelerator configuration in the training dataset (last row Table), by 2.46×. PRIME also outperforms the accelerators in the best online method by 1.54× (up to 5.80× and 6.62× in t-RNN Dec and t-RNN Enc, respectively). Moreover, perhaps surprisingly, PRIME generates accelerators that are better than all the online optimization methods in 7/9 domains, and performs on par in several other scenarios (on average only 6.8% slowdown compared to the best accelerator with online methods in M5 and M6). These results indicates that offline optimization of accelerators using PRIME can be more data-efficient compared to online methods with active simulation queries. To answer Q(2), we compare the total simulation time of PRIME and the best evolutionary approach from Table on the MobileNetEdge domain. On average, not only that PRIME outperforms the best online method that we evaluate, but also considerably reduces the total simulation time by 93%, as shown in Figure. Even the total simulation time to the first occurrence of the final design that is eventually returned by the online methods is about 11× what PRIME requires to fine a better design. This indicates that data-driven PRIME is much more preferred in terms of the performance-time trade-off. The fact that our offline approach PRIME outperforms the online evolutionary method (and also other state-of-the-art online MBO methods; see Table) is surprising, and we suspect this is because online methods get stuck early-on during optimization, while utilizing offline data allows us PRIME to find better solutions via generalization (see Appendix B.1.1).
Table: Optimized objective values (i.e., latency in milliseconds) obtained by various methods for the task of learning accelerators specialized to a given application. Lower latency is better. From left to right: our method, online Bayesian optimization ("Bayes Opt"), online evolutionary algorithm ("Evolutionary"), and the best design in the training dataset + validation dataset. On average (last row), PRIME improves over the best in the dataset by 2.46× (up to 6.69× in t-RNN Dec) and outperforms best online optimization methods by 1.54× (up to 6.62× in t-RNN Enc). The best accelerator configurations identified is highlighted in bold.  Architecting accelerators for multiple applications. To answer Q(3), we evaluate the efficacy of the contextual version of PRIME in designing an accelerator that attains the lowest latency averaged over a set of application domains. As discussed previously, the training data used does not label a given accelerator with latency values corresponding to each application, and thus, PRIME must extrapolate accurately to estimate the latency of an accelerator for a context it is not paired with in the training dataset. This also means that PRIME cannot simply return the accelerator with the best average latency and must run non-trivial optimization. We evaluate our method in seven different scenarios (Table), comprising various combinations of models from Table and under different area constraints, where the smallest set consists of the three MobileNet variants and the largest set consists of nine models from image classification, object detection, image segmentation, and speech recognition. This scenario is also especially challenging for online methods since the number of jointly feasible designs is expected to drop significantly as more applications are added. For instance, for the case of the MobileNet variants, the training dataset only consists of a few (20-30) accelerator configurations that are jointly feasible and high-performing (Appendix B.2-Figure).
Table shows that, on average, PRIME finds accelerators that outperform the best online method by 1.2× (up to 41%). While PRIME performs similar to online methods in the smallest three-model scenario (first row), it outperforms online methods as the number of applications increases and the set of applications become more diverse. In addition, comparing with the best jointly feasible design point across the target applications, PRIME finds significantly better accelerators (3.95×). Finally, as the number of model increases the total simulation time difference between online methods and PRIME further widens (Figure). These results indicate that PRIME is effective in designing accelerators jointly optimized across multiple applications while reusing the same dataset as for the single-task, and scales more favorably than its simulation-driven counterparts. Appendix A.4 expounds the details of the designed accelerators for nine applications, comparing our method and the best online method.  Accelerating previously unseen applications ("zero-shot" optimization). Finally, we answer Q() by demonstrating that our data-driven offline method, PRIME enables effective data reuse by using logged accelerator data from a set of applications to design an accelerator for an unseen new application, without requiring any training on data from the new unseen application(s). We train a contextual version of PRIME using a set of "training applications" and then optimize an accelerator using the learned surrogate with different contexts corresponding to "test applications," without any additional query to the test application dataset. Table shows, on average, PRIME outperforms the best online method by 1.26× (up to 66%) and only 2% slowdown in 1/4 cases. Note that the difference in performance increases as the number of training applications increases. These results show the effectiveness of PRIME in the zero-shot setting (more results in Appendix A.5).
Applying PRIME on other accelerator architectures and dataflows. Finally, to assess the the generalizability of PRIME to other accelerator architectures, we evaluate PRIME to optimize latency of two style of dataflow accelerators-NVDLA-style and ShiDianNao-style-across three applications (Appendix C details the methodology). As shown in Table, PRIME outperforms the online evolutionary method by 6% and improves over the best point in the training dataset by 3.75×. This demonstrates the efficacy of PRIME with different dataflows and large design spaces.
From left to right: our method, online Bayesian optimization (“Bayes Opt”), online evolutionary algorithm (“Evolutionary”), and the best design in the training dataset + validation dataset. ****PRIME generalizes to other classes of accelerators with larger search space and outperforms the best online method by 1.06× and the and the best design in the training dataset + validation dataset by 3.75× (last column).
Answer: Yes it does, we have added this information in the caption of Tables 3 and 6.