SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Metrics There are two tasks for our dataset: regression and classification. The goal of the regression task is to predict the five targets: PERF, DSP, BRAM, LUT, and FF. We use rooted mean square error (RMSE) to evaluate each method. And the goal of the classification task is to predict whether a design is valid or not, i.e. whether the downstream RTL and physical synthesis are likely to complete or not. We use the classification accuracy as the evaluation metric.
In addition, recall that we have two versions of the dataset (SDX (v1) and VITIS (v2)). Each (version, task) combination receives a separate evaluation. For each evaluation, there are two evaluating settings: transductive and inductive testing. Specifically, for each (version, task) combination: (1) We select six kernels as the held-out testing kernels. These kernels are never seen during training and are used for the inductive testing; (2) For the rest of the kernels, we merge all the labeled designs, and randomly split them into training, validation, and transductive testing designs with the 70:15:15 ratio; (3) Using the training designs for 1000 epochs for the regression task, and 200 epochs for the classification task, we train each baseline method. We employ the validation set to determine the best epoch to use for testing; (4) We test the trained model on the transductive testing set. It is called transductive ("Trans") since this testing set contains designs from kernels that are seen during training; (5) We test the trained model on the held-out six kernels. Specifically, we (5.1) select the  $30\%$  designs from each held-out kernel as the testing designs, (5.2) then repeat the following procedure 5 times. For each kernel, from the rest of the  $70\%$  remaining designs, 10 designs are sampled and are utilized to adapt the trained model for 10 epochs. Then, the adapted model is tested on the  $30\%$  designs for that held-out kernel. We call such a setting inductive because the model is tested on six kernels that are not visible in the training stage in a zero-shot ("Ind") or few-shot learning setting ("Ind Adapt").
The overall regression and classification results are exhibited in Tables 5 and 6. Tables 7, 8, 9, and 10 reveal the breakdown results over individual held-out kernels.
Observation 1: There is no consistent winner among the baselines. For the regression task, G-CODEBERT and G-CODEBERT-L achieve the lowest error when adapted to the held-out kernels, whereas, for the classification task, GNN-based methods perform better. Such a phenomenon calls for a hybrid model utilizing both the source code and the assembly code graph, e.g., the concatenation models [CODET5,GNN-GSE] and [G-CODEBERT,GNN-GSE]. However, a simple concatenation of the
Table 6: Classification accuracy on SDx (v1) and VITIS (v2).
Method	SDx (v1)	VITIS (v2)
Trans	Ind	Ind Adapt	Trans	Ind	Ind Adapt
CODE2VEC [2]	0.7576	0.5662	0.6617	0.7060	0.5444	0.6337
CODET5-RAND	0.8524	0.5257	0.7015	0.7924	0.4851	0.6291
CODET5-FROZEN	0.7515	0.6098	0.7486	0.7334	0.4161	0.6061
CODET5 [39]	0.9501	0.6394	0.7447	0.9045	0.4781	0.6734
G-CODEBERT [18]	0.9536	0.6478	0.7610	0.9233	0.5415	0.7024
G-CODEBERT-L [18]	0.9204	0.5730	0.7701	0.8970	0.5342	0.7180
GNN-DSE [30]	0.9422	0.6529	0.7623	0.9045	0.4781	0.6734
GNN-DSE-2L	0.8912	0.6085	0.7421	0.9126	0.5303	0.7632
[CODET5,GNN-GSE]	0.9434	0.6141	0.7385	0.9195	0.5053	0.7002
[G-CODEBERT,GNN-GSE]	0.9212	0.6174	0.7446	0.9126	0.5001	0.7160
Table 7: Regression result breakdown on SDx (v1) on individual test kernels.
Method	DOITGEN-R	FDTD-2D	GEMM-N	JACOBI-2D	STENCIL-3D	TRMM-OPT
CODE2VEC [2]	2.5711±0.08	3.9973±0.19	5.6191±0.08	2.5710±0.11	2.9029±0.04	2.8322±0.30
CODET5-RAND	1.2123±0.02	3.3038±0.10	3.8933±0.11	1.8584±0.12	0.9961±0.14	2.1133±0.09
CODET5-FROZEN	1.3197±0.00	3.2819±0.06	4.3073±0.03	1.7756±0.04	0.6807±0.03	2.0248±0.09
CODET5 [39]	0.9990±0.27	2.9818±0.05	3.6910±0.41	1.3470±0.12	0.3695±0.06	1.5336±0.14
G-CODEBERT [18]	0.8301±0.17	2.5938±0.15	3.3672±0.45	1.1581±0.06	0.4830±0.16	0.8990±0.19
G-CODEBERT-L [18]	0.5376±0.08	2.8015±0.18	3.1310±0.52	1.0214±0.12	0.3636±0.07	0.9384±0.10
GNN-DSE [30]	0.5278±0.03	2.6449±0.10	2.8907±0.51	1.3855±0.10	0.6918±0.23	0.9673±0.05
GNN-DSE-2L	0.9113±0.16	2.6677±0.11	2.7881±0.28	1.5230±0.07	0.6984±0.09	1.1499±0.05
[CODET5,GNN-GSE]	0.9637±0.20	2.7998±0.30	2.9908±0.16	1.2991±0.14	0.7373±0.14	1.0056±0.04
[G-CODEBERT,GNN-GSE]	1.0409±0.21	2.7988±0.17	3.2942±0.23	1.0845±0.12	0.3072±0.06	0.7891±0.24
Table 8: Regression result breakdown on VITIS (v2) on individual test kernels.
Method	COVARIANCE	FDTD-2D-L	GEMM-N	GEMM-P-L	SYMM	TRMM-OPT
CODE2VEC [2]	2.4977±0.09	3.1259±0.18	4.5526±0.19	3.7389±0.03	1.4209±0.11	2.6379±0.21
CODET5-RAND	1.6388±0.13	2.4587±0.22	2.8477±0.15	3.0722±0.04	0.4969±0.03	1.2295±0.09
CODET5-FROZEN	1.2137±0.02	2.4494±0.22	2.6576±0.10	3.1420±0.02	0.3854±0.02	1.2199±0.02
CODET5 [39]	1.3594±0.15	2.2958±0.07	2.8436±0.37	2.7803±0.07	0.4092±0.09	0.8427±0.14
G-CODEBERT [18]	1.0652±0.06	1.9245±0.09	2.3772±0.31	2.3913±0.04	0.4019±0.08	0.9452±0.09
G-CODEBERT-L [18]	0.9107±0.08	1.7740±0.11	2.6635±0.33	2.5029±0.09	0.3384±0.03	1.1469±0.11
GNN-DSE [30]	1.1496±0.04	1.8897±0.04	2.3157±0.23	2.7828±0.18	0.4110±0.02	0.7790±0.06
GNN-DSE-2L	1.1457±0.02	1.9521±0.09	2.0248±0.26	2.9691±0.10	0.4067±0.04	0.9146±0.03
[CODET5,GNN-GSE]	1.0988±0.07	1.8532±0.10	2.2430±0.23	2.5489±0.04	0.3766±0.06	1.0440±0.09
[G-CODEBERT,GNN-GSE]	1.1182±0.07	1.9088±0.10	2.5352±0.25	2.7655±0.08	0.3842±0.05	0.8986±0.13
Table 9: Classification result breakdown on SDx (v1) on individual test kernels.
Method	DOITGEN-R	FDTD-2D	GEMM-N	JACOBI-2D	STENCIL-3D	TRMM-OPT
CODE2VEC [2]	0.6449±0.07	0.6192±0.01	0.4598±0.03	0.8122±0.05	0.6085±0.03	0.8256±0.04
CODET5-RAND	0.6663±0.05	0.5283±0.08	0.6366±0.08	0.8384±0.03	0.5887±0.11	0.9509±0.00
CODET5-FROZEN	0.7472±0.00	0.6414±0.00	0.6286±0.17	0.8889±0.00	0.6338±0.00	0.9516±0.00
CODET5 [39]	0.7236±0.04	0.6455±0.02	0.5804±0.04	0.8681±0.01	0.7042±0.05	0.9467±0.01
G-CODEBERT [18]	0.6607±0.06	0.6455±0.03	0.6714±0.03	0.8509±0.02	0.8000±0.04	0.9377±0.02
G-CODEBERT-L [18]	0.7236±0.08	0.6556±0.03	0.5741±0.07	0.8792±0.01	0.8394±0.05	0.9488±0.01
GNN-DSE [30]	0.6506±0.06	0.6889±0.02	0.6402±0.03	0.8652±0.03	0.7803±0.03	0.9488±0.00
GNN-DSE-2L	0.6674±0.05	0.6778±0.03	0.6545±0.02	0.8534±0.03	0.6648±0.04	0.9349±0.03
[CODET5,GNN-GSE]	0.7022±0.07	0.6364±0.02	0.6438±0.03	0.8419±0.03	0.6620±0.04	0.9446±0.01
[G-CODEBERT,GNN-GSE]	0.7124±0.06	0.7091±0.04	0.6393±0.04	0.8616±0.04	0.6028±0.09	0.9426±0.02
Table 10: Classification result breakdown on VITIS (v2) on individual test kernels.
Method	COVARIANCE	FDTD-2D-L	GEMM-N	GEMM-P-L	SYMM	TRMM-OPT
CODE2VEC [2]	0.5830±0.05	0.5750±0.02	0.5099±0.02	0.6678±0.05	0.7191±0.05	0.7476±0.03
CODET5-RAND	0.6453±0.03	0.6444±0.04	0.6185±0.03	0.6678±0.04	0.6936±0.05	0.5048±0.25
CODET5-FROZEN	0.6698±0.00	0.4778±0.11	0.4951±0.04	0.6610±0.00	0.5234±0.11	0.8095±0.00
CODET5 [39]	0.6698±0.04	0.5750±0.06	0.5728±0.03	0.6983±0.05	0.6766±0.04	0.8476±0.03
G-CODEBERT [18]	0.5585±0.07	0.6111±0.07	0.5840±0.03	0.8373±0.04	0.9021±0.05	0.7214±0.07
G-CODEBERT-L [18]	0.5887±0.05	0.6583±0.08	0.6778±0.03	0.8203±0.07	0.7702±0.07	0.7929±0.02
GNN-DSE [30]	0.6509±0.01	0.7000±0.07	0.6840±0.06	0.7559±0.05	0.8170±0.08	0.8381±0.03
GNN-DSE-2L	0.6396±0.07	0.7639±0.10	0.7222±0.02	0.7763±0.06	0.8340±0.06	0.8429±0.02
[CODET5,GNN-GSE]	0.6491±0.06	0.6361±0.02	0.6494±0.03	0.7017±0.04	0.8128±0.04	0.7524±0.09
[G-CODEBERT,GNN-GSE]	0.6396±0.03	0.6083±0.05	0.6691±0.05	0.7695±0.03	0.8213±0.07	0.7881±0.03
design embeddings does not consistently yield better performance. Particularly, [CODET5,GNN-GSE] and [G-CODEBERT,GNN-GSE] achieve the lowest regression error under the transductive setting but fall short when adapted to new kernels. Such results imply that future efforts can be made on studying the generalization abilities of machine learning models on our HLSYN benchmark.
Observation 2: In general, pre-training helps with the performance of source code transformer models. This can be seen by comparing CODET5-RAND and CODET5, where the former starts training from scratch while the latter loads a pre-trained model released by [39]. This should not come as a surprise, because pre-training has been shown to demonstrate success in natural language processing. Our experimental results verify the effectiveness of pre-training on the HLSYN benchmark. One implication is that one can design better pre-training methods on source code related to electronic design automation, or even design pre-training for GNNs operating on assembly-level graphs.
Observation 3: More GNN message passing layers does not necessarily improve the performance of GNN models. In many cases, the 2-layer version, GNN-DSE-2L, performs even better than the 8-layer version, GNN-DSE. This may be attributed to the fact that an attention-based global readout function is used to aggregate node embeddings into a graph-level embedding representing the entire design, and thus each node does not necessarily need to reach far-away nodes in the local message passing stage.
Observation 4: Generalization to new kernels is difficult, and the performance after adaptation differs across kernels. For example, G-CODEBERT-L achieves the overall lowest error ("Ind Adapt") on SDX (v1) on the regression task, but Table 7 demonstrates that G-CODEBERT-L does not always yield the lowest error on each of the six held-out kernels. For example, [G-CODEBERT,GNN-GSE] performs the best on STENCIL-3D and TRMM-OPT, but poorly on DOITGEN-R, and thus the average regression error over the six held-out kernels is higher than G-CODEBERT-L. This calls for a more in-depth study of the discrepancy between kernels and a model that is capable of generalizing to a diverse set of kernels. In general, adaptation is necessary, since across all the experiments, without any adaptation ("Ind"), directly applying the trained model to new unseen kernels leads to poor regression error and classification accuracy.
This work introduces the task of design quality prediction in the forms of regression and classification tasks and presents the HLSYN benchmark to evaluate state-of-the-art program representation learning methods. Although there is no method that consistently outperforms all the other methods, we notice several trends and identify promising directions toward a more accurate prediction model design. As program representation learning is a continuously growing research domain, we plan to maintain the benchmark to test new methods. For example, our recent work [31] uses hierarchical graphs for program representations to predict design performance. Our HLSYN benchmark is a growing project. We expect to include more kernels and labeled designs running newer versions of HLS tools and establish a leaderboard  ${}^{2}$  to encourage participation. In addition, the current benchmark does not consider the design space exploration (DSE) stage, which will be added as the project develops. In fact, the regression task aims to be eventually integrated into the DSE process, which traverses the design space in order to find the optimal pragma setting.
This work was partially supported by NSF 2211557, NSF 1937599, NSF 2119643, NSF 2303037, NASA, SRC JUMP 2.0 Center, Okawa Foundation, Amazon Research, Cisco, Pictsart, Snapchat, and CDSC industrial partners (https://cdsc.ucla.edu/partners/). We would also like to thank AMD/Xilinx for HACC equipment donation and Marci Baun for editing the paper.
[1] Sergi Abadal, Akshay Jain, Robert Guirado, Jorge López-Alonso, and Eduard Alarcón. Computing graph neural networks: A survey from algorithms to accelerators. ACM Computing Surveys (CSUR),
$^{2}$ https://github.com/UCLA-DM/HLSyn#leaderboard
[39] Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. EMNLP, 2021.
[40] Nan Wu, Yuan Xie, and Cong Hao. Ironman-pro: Multi-objective design space exploration in HLS via reinforcement learning and graph neural network based modeling. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2022.
[41] Nan Wu, Hang Yang, Yuan Xie, Pan Li, and Cong Hao. High-level synthesis performance prediction using GNNs: Benchmarking, modeling, and advancing. In Proceedings of the 59th ACM/IEEE Design Automation Conference, pages 49-54, 2022.
[42] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. ICML, 2018.
[43] Cody Hao Yu, Peng Wei, Max Grossman, Peng Zhang, Vivek Sarker, and Jason Cong. S2FA: an accelerator automation framework for heterogeneous computing in datacenters. In DAC, pages 1-6, 2018.
[44] Tomofumi Yuki and Louis-Noël Pouchet. Polybench/c.
[45] Jieru Zhao, Liang Feng, Sharad Sinha, Wei Zhang, Yun Liang, and Bingsheng He. COMBA: A comprehensive model-based analysis framework for high level synthesis of real applications. In ICCAD, pages 430-437, 2017.
[46] Guanwen Zhong, Alok Prakash, Yun Liang, Tulika Mitra, and Smail Niar. Lin-analyzer: a high-level performance analysis tool for FPGA-based accelerators. In DAC, pages 1-6, 2016.
[47] Guanwen Zhong, Vanchinathan Venkataramani, Yun Liang, Tulika Mitra, and Smail Niar. Design space exploration of multiple loops on FPGAs using high level synthesis. In ICCD, pages 456-463, 2014.