Towards a Comprehensive Benchmark for High-Level Synthesis Targeted to FPGAs
Authors: Yunsheng Bai
Atefeh Sohrabizadeh
Zongyue Qin
Ziniu Hu
Yizhou Sun
Jason Cong
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: integrated circuit design
FPGAs
hardware design automation
hardware design optimization
representation learning
source code
assembly code
multi-modality
graph neural network
transformer
pretraining
high level synthesis
benchmark

Abstract: High-level synthesis (HLS) aims to raise the abstraction layer in hardware design, enabling the design of domain-specific accelerators (DSAs) like field-programmable gate arrays (FPGAs) using C/C++ instead of hardware description languages (HDLs). Compiler directives in the form of pragmas play a crucial role in modifying the microarchitecture within the HLS framework. However, the space of possible microarchitectures grows exponentially with the number of pragmas. Moreover, the evaluation of each candidate design using the HLS tool consumes significant time, ranging from minutes to hours, leading to a time-consuming optimization process. To accelerate this process, machine learning models have been used to predict design quality in milliseconds. However, existing open-source datasets for training such models are limited in terms of design complexity and available optimizations. In this paper, we present HLSyn, the first benchmark that addresses these limitations. It contains more complex programs with a wider range of optimization pragmas, making it a comprehensive dataset for training and evaluating design quality prediction models. The HLSyn benchmark consists of 42 unique programs/kernels, resulting in over 42,000 labeled designs. We conduct an extensive comparison of state-of-the-art baselines to assess their effectiveness in predicting design quality. As an ongoing project, we anticipate expanding the HLSyn benchmark in terms of both quantity and variety of programs to further support the development of this field.

Towards a Comprehensive Benchmark for High-Level Synthesis Targeted to FPGAs
Yunsheng Bai, Atefeh Sohrabizadeh, Zongyue Qin, Ziniu Hu, Yizhou Sun, Jason Cong Department of Computer Science
University of California, Los Angeles
{yba, atea www.aeesssginzzngguuuuu bull, yzsun, cong) www..ucla.............. aell.edi edu
Abstract
High-level synthesis (HLS) aims to raise the abstraction layer in hardware design, enabling the design of domain-specific accelerators (DSAs) targeted for field- programmable gate arrays (FPGAs) using C/C++ instead of hardware description languages (HDLs). Compiler directives in the form of pragmas play a crucial role modifying the microarchitecture within the HL. framework. However, the number of possible microarchitectures grows exponentially with the number of pragmas. Moreover, the evaluation of each candidate design using the HLS tool consumes significant time, ranging from minutes to hours, leading optimization process. To accelerate this process, machine learning models have been used to predict design quality in milliseconds. However, existing open-source datasets for training such models are limited in terms of design complexity and present HLSYN, new benchmark
slow
available optimizations. In this paper, that addresses these limitations. It contains more complex programs with a wider range of optimization pragmas, making it comprehensive dataset for training and evaluating design quality prediction models. The HL.SYN benchmark consists of 42 unique programs/kernels, each of which has many different pragma configurations, resulting in over 42,000 labeled designs. We conduct an extensive comparison of state-of-the-ar baselines to assess their effectiveness in predicting design quality, As an ongoing project, we anticipate expanding the HLSYN benchmark in terms of both quantity and variety of programs further support the development of this field.
Introduction
In recent decades, the demand for specialized computing systems tailored to specific applications has significantly increased. This has led to the emergence of being implemented in either opplication-specife integrated circuits (ASICs) or field programmable gate arrays (FPGAs). By leveraging the unique characteristics of specific workloads, designer can design DSAs to enhance performance and energy efficiency. This becomes particularly valuable when general-pon purepss processors like CPUs and GPUs cannot meet the performance or efficiency requirements of certain applications due to the end of Dennard scaling [8, 14]. For instance, Google has developed its custom-designed DSA the form of an ASIC named the Tensor Processing Unit (TPU) [21], which is highly optimized for machine learning workloads, offering remarkably faster performance and improved energy efficiency compared to CPUs and GPUs. In addition, FPGAs offer cost-effective alternative with reconfigurability, making them increasingly appealing for accelerating applications across various domains, including search engines and numerous datacenter applications [25, 9, 19], machine learning inference acceleration [15, 17, 32, 1, 27]. and autonomous vehicles [7]. among others.
accelerators (DSAs)
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Nevertheless, the design of DSAs poses distinct difficulties in contrast to perrooe hardware like CPUs and GPUs [34, 8). DSAs are com- monly developed using hardware description lan- guages (HDLs) at the register-transfer level (RTL), specifically Verilog and VHDL, which are primar- ily known to circuit designers. To address this challenge, high-level synthesis (HLS) [11, 10] was introduced and is now supported by most EDA (Electronic Design Automation) and FPGA compa- nies [4, 6, 20, 24, 29]. HLS raises the level of design abstraction to //C===/OpeCL/SSytteem
void kernel, 2in int ist nk,ing nl,dochle
alpha double beta, double tmp [40] [50] ddoubl A [To], ,oubbe atro] [50] double
C150] (80) double [840]
4;
int Ji
ist k;
//Apragna scop
/a D :- alpherte3>C beca<D -/
bgragza 4D00. P17
1o0e THE PACTOR-
Ppragea AS98, PANMIEN
Pragmas
For (1
I++)
Norazza A200, P1PPo LAk Flatban
Spragga Aooes, 7115 FACTOR 1
Apragen Adoai, PASITE FACTOP-
For [j 0: 50; jrt)
enabling de- signers to describe the high-level behavioral repre- sentation of their designs rather than the low-level data transition in RTL. This abstraction eliminates the need for explicit clock scheduling specifications in the HLS code. Instead, HLS tools analyze the behavior description to schedule operations across different clock cycles, assign operations to available resources, and establish the required control struc- ture. Finally, the HLS tool automatically generates code based on these analyses. It can take several minutes to hours for the HIS tool to generate this RTL code [34]. The RTL can then be passed through logic synthesis and physical design steps, which can consume several hours, to be implemented on the target FPGA. This HLS tool enhances design produc- tivity, shortens design cycles, and allows designers to rapidly explore various design options without the need for manual RTL code writing.
tmp[s] [j] - 0.0;
ACDEL PASALLEL
BACTOR
70;
tap [5] alpha A( (1)
Sprogae ADDEL PIPELTHE off Seragea KRDEL THE PACTOR-
RO0EL PIZALLEL EL FICTUET!
Moragga AoWES
Spsagga RDDEL
Poragga A20EL, PARATEEL FACTOR
For () 0: < 90; f++)
Disl [g] e- beta;
Boragea Roven FISTreL reruccion FACTORESS
(x [f) txpli) (5):
50; +98)
//Aprugxa andscop
>
Source Code with Pragmas
IILS followed by RTL and Physical Synthesis
Despite the increased level of design abstraction of- fered by high-level synthesis (HLS) tools, they still require considerable amount of hardware design expertise to utilize synthesis directives in the form of pragmas. These pragmas play a crucial role in spec- ifying various aspects of the design, such as memory organization, caching strategies, memory buffer par- titioning, parallelization and pipelining of computa- tions, etc. As demonstrated by Chi et al. [8]. although performance-
FPGA Implementation for the Source Code
Figure I:
High-level synthesis (HLS) trans- forms the source code of the kernel 2MM writ- ten in C into lower-level programming lan- guage and eventually implements the design on the target FPGA as shown with the chip die photo.
the performance of DSA with optimizing pragmas can be 108 x slower than a CPU, through proper optimization, it can achieve re- markable performance improvement, surpassing a CPU by 89x. However, the optimization process for architecture-specffic enhancements is typically limited to hardware programmers and falls beyond the capabilities of the average software programmers. Consequently, there has been growing focus on automating this optimization process. While some approaches treat the HLS tool as a black box and develop custom- designd heuristics to search through design candidates, a more recent research paradigm leverages machine learning and deep learning techniques. These approaches either learn the behavior of the HLS tool and construct predictive models or employ data-driven exploration methods to search through the solution space. The goal of automating the optimization process to democratize customized computing and make it more accessible for the average software programmers, allowing them to utilize the tailored hardware acceleration.
To address the need for automating pragma insertion and parameter tuning in high-level synthesis (HLS), machine learning techniques can be used. This approach aims to achieve optimal quality in terms of latency and resource utilization. However, the lack of open-source datasets in this domain and the limitations of existing datasets, which are constrained in terms of design complexity and available optimizations, restrict their practicality. To bridge this gap, this paper introduces HLSYN the first

---

comprehensive benchmark for HLS designs targeted to FPGAs for performance optimization. This benchmark provides more complex programs and a wider range of optimization pragmas, facilitating advanced research and facilitating in-depth exploration of machine learning techniques in the context of HI... In our study, we define a design as a piece of C/C++ source code (referred to as a kernel) with associated pragmas. Our primary focus is on predicting the quality of design using supervised learning by running a model trained on a collection of labeled designs with corresponding quality metrics. However, our dataset can be utilized in various training scenarios, including training agents to efficiently explore the solution space.
2 Background
The task of HLSYN is to predict the quality of the HLS design specified by program (kernel) with a specific optimization pragma design. Our target implementation platform is FPGA, although similar techniques can be ap- plied to ASIC accelcrator designs as well. The quality of a design is defined as a function of its performance, which is measured by its latency in cycle counts, and its area/re- source utilization, such as the usage of digital signal pro- cessing blocks (DSP), blocks' RAMs (BRAM), flip-flps (FF), and lookup-tables (LUTT), which are the fundamental building blocks for implementing digital logic circuits in FPGA designs.
PARA
PARA
185
PARA
125
PARA 133
PARA 14:
Val-DSP DSP 0.1000.
PARA 15%
PIPE uu.
PIPE LI
Val--LIT: 0.8000.
Aeil-91 010000
PIPE 12:.0
PIPE Yontent
TIL LIP
VILE LI. I.
TILE 12
TILE LS 80
Design
In this work, we specifically consider the optimization pragmas of the Merlin Compiler, an open-source source- to-source optimization tool used for efficient AMD/Xilinx HLS designs. The Merlin Compiler provides three types of optimization pragmas, namely PIPELINE, PARALLEL, and TILE to define the desired microarchitecture [34].
pust 4.7000
WIE-DE DEP--PPP 20000
WIL-DRAM
POPE
"11l-1177 IN 11000
PIPE 1.1
VIB-FF 0.1000
PIP Hamon
TILE LI:
As illustrated in Figure 1, these pragmas can be applied at the loop level and offer control over the type of pipelining. the parallelization factor, and the amount of data caching. If setting the pragmas properly to non-default parameters for proper parallelizing and pipelining the computation, the resulting accelerator can be 10x or even 100x faster than a single-core CPU, However, without any pragma insertion, the resulting hardware can be 10x slower than a CPU. Figure 2 demonstrates an example where the pre- diction targets are sensitive to the pragma settings. It is noteworthy that in some other examples, a change in the pragma settings does not lead to any change in the output targets. The machine learning model must learn from ex- isting labeled designs and understand the source code as well as the pragmas in order to accurately forecast the outcome of each design when eventually being exccuted on an FPGA.
TIEE 125
1.3
Design 2
Figure 2: Two example designs selected from the 2MM kernel. In Figure 1, the kernel contains 14 pragmas, and each row has the format "* ""<pragma name>": <pragma setting>". A small change in one of the pragma parameters leads to changes in the FPGA HLS results, i... the five prediction targets for the regres- sion task.
Table 1 summarizes the parameter space of these pragmas. For a given program/kernel. any change in the option of any of the pragmas results in different design with unique microarchitecture, The "fg" option in pipelining refers to the case where all the inner loops are unrolled (parallelized with separate logic) and each parallel unit is pipelined. The "cg" option, on the other hand, results in mparse-grained processing elements (PEs) that are pipelined together. For example, it can create pipelined aad-compute-stonn units. The PARALLEL and TILE pragma take numeric values that determine the degree of parallelization and loop tiling, respectively.
3

---

Table 1: Target pragmas with their options.
Pragma Name
Parameter Name
Parameter Space
Examples of Pragma Settings
PARALLEL
factor mode factor
integer "eg", "fg", off integer
4,
PIPELINE
"flatten' resulting in the "fg" mode
TILE
2,
3 Related Work
In previous research, optimizing HLS designs has been approached in different ways. One category of methods treats the HLS tool as a black box and utilizes problem-independenn heuristics or develops dedicated heuristics to explore the solution space and evaluate the quality of results (QoR) directly using the tool [34, 43, 35]. However, this approach is consuming as each evaluation takes several minutes to hours. To mitigate this issue, another category of methods aims to create surrogate models for the HI.S tool. Some of these methods construct dependency graphs of the program and employ traditional graph analysis techniques to schedule operations and estimate the QoR accordingly [46, 38J, while others develop analytical performance and area models to estimate the QoR [45, 47]. Nevertheless, due to the different heuristics employed by HLS tools in the design process, these models may not accurately predict the QoR [34]. Some methods address this limitation by focusing on designs that can exploit pre-defined microarchitecture templates or follow specific computation patterns [33, 37, 12], but this can limit their generality. Alternatively, data-driven approach utilizing machine learning and deep learning models has been proposed to enhance prediction accuracy [23, 22]. Graph neural networks (GNNs) have gained attention in this context and demonstrated promising results in enhancing prediction accuracy [30, 5, 40, 36).
fundamental aspect of these approaches is the availability of a large-scale database to effectively train the models. Recent works have focused on gathering such databases [16, 41, 30J. Unfortunately, existing datasets have limitations. The dataset in [41] predominantly consists of synthetic programs that do not utilize any pragmas. DB4HLS [16] targets programs from the MachSuite benchmark[26] but overlooks the inclusion a key optimization pragma, pipelining. Additionally, DB4HLS views each function in the program an individual kernel. ENN-DSE [30] targets programs from the Polyhedral benchmark [44], which features more complex kernels for FPGA mapping, in addition to kernel,
the MachSuite benchmark. This dataset considers program with all its fub-fuuctions further increasing design complexity. Despite covering a wide range of optimization pragmas for each kernel, the generated dataset is small, with only target kernels and total of 4,752 data points. To address these limitations, we propose HLSYN, which includes kernels from both the Polyhedral and MachSuite benchmarks. It encompasses diverse range of optimization pragmas that can pipeline and/or parallelize computation, as well as adjust data caching. Our benchmark is comprised of 42 unique kernels from various domains summarized in Table totaling over 42,000 design points, providing comprehensive resource for advancing research and facilitating an in-depth exploration of machine learning techniques for HL..
The HLSYN Benchmark
In this section, we introduce the datasets in HLSYN', The input data source comes from 42 selected kernels in the MACHSUITE benchmark [26] and the POLY FoLYBENCH benchmark suite [44]. Our selected 42 kernels cover a wide range of applications whose descriptions are shown in Table 2.
The benchmark consists of 2 datasets accumulated in the past three years, corresponding to two versions of AMD/Xilinx HLS tools: (1) SDx (v1) [3] and VITIS (v2) [4], with the AMD/Xilinx Alveo U200 as the target FPGA and a working frequency of 250MHz. For each dataset, we select 6 kernels as the held-out testing kernels. They will test the ability of model generalize to kernels that it did not see during the training. For the rest of the kernels, we perform a random split with the training, validation, and testing ratio being 70: 15:15. Summary statistics of datasets given in Table 3.
For each dataset kernel in each dataset, we run the two versions of the tools described above obtain a set of labeled designs. Since the design space size is exponential to the number of pragmas, we rely
'https://githain http://gitthuu..cmocA..LA.. DM/HHSyn

---

Table 2: There are 42 kernels in total across SDX (v1) and VITIS (v2) spanning multiple domains such as linear algebra on vectors and matrices, data mining, stencil operations, encryption, dynamic programming, etc. #p denotes the number of pragmas the kernel. # in vl and # in v2 denote the number of labeled designs in SDX (v1) and VITIS (v2) respectively.
Kernel
Source
Description
praggmas
v1
v2
2MM
POLYBENCH
2 Matrix Muliplications
14
812
861
3MM
POLYBENCH
Matrix Mulúplicatioos
ADI
POLYBENCH
Altemating Direction Implicit solver
13
551
AES
MACHSUITE
Advanced Encryption Standard
45
ATAX
POLYBENCU
Matri Transpose and Voctor Multiplication
884
902
ATAX-MEDIUM
POLYBENCH
Matrix Transpose and Vector Multiplicution
362
544
POLYBENCH
BiCG Sub Kernel of BiCGStab Linear Solver
512
498
BICG-LARGE
POLYBENCH
BICG Sub Keree of BICGStab Lincar Solver
456
BICG-MEDIUM
POLYBENCH
BiCG Kernel BiCCStubb Linear Solver
CORRELATION
POLYBENCE
Comelation Computation
17
1522
699
COVARIANCE
BENCHEN
Covariance Computation
13
356
DOITGEN
POLYBENCH
Multresolution Analysis
179
172
DOITGEN-R
PoLY BENCH
Multiresolation Analysis
5Y5
230
FDID-2D
POLYBENCH
2-D Finite Different Time Domain Kemel
16
FDTD-2D-L
POLYBENCH
2-D Finite Different Time Domsin Kemel
16
240
GEMM-B
MACHSUITE
Blocked Version Matrix Muliiplication
GEMM-N
MACHSUITE
Matrix Transpose and Vector Multiplication Matrix Multiplication
749
540
GEMM-P
POLY BONYE
1160
714
GEMM-P-L
POLYBENCH
Marix Transpose and Voctor Multiplication Vecior Multiplication and Matrix Addition
199
GEMVER
POLYBENCH
13
924
712
GEMVER-M
POLYBENCH
Vector Multiplication and Matrix Addition
13
3365
GESUMMY
POLYBENCH
Scalar, Vector and Matrix Multipiiatioo Scalar, Vector Matrix Multiplication Heat Equation over 3D Data Domain
442
371
GESUMMY-M M
POLYBENCH
HEAT-3D
BENCH
11
1664
JACOBI-ID
PoLY BOLYBENCH
I-D Jacobi Stencil Computation
595
JACOBI-2D
POLYBENCH
2-D Jacobi Stencil Computation
11
1862
MD
MACHSUITE
n hody Molecular Dynamics
12
MVT
POLYBENCH
Matrix Vector Product Transpose
1175
MVT-M
MACHSUITE
Matrix Vector Product and Transpose
416
NW
MACHSUITE
Dynamic Programming for Sequence Aligament
1347
6155
SEIDEL-20
POLYBENCH
2-D Seidel Stencil Computation
1314
SPMV-CRS
MACHSUTTE
Sparse Mat- Vee Mult. w/ Variable-Len. Neighbor
114
114
SPMY-ELLPACK
MACHSUITE
Sparse Mat- Vec Muli. w/ Fixed size Neighbor
114
102
STENCH.
MACHSUITE
A Wvo-Dimensional Stencil Computation
1404
1016
STENCIL-3D
MACHSUITE
A Three- Dimensional Stencil Computation
SYMM
POLYBENCH
Symmetric Matrix Multiplication
153
158
SYMM-OPT
PoLY BENCH
Symmetric Matrix Multiplication
SYMM-OPT-M
BOLYBENCH
Symmetric Matrix Multiplication
315
SYR2K
BOLYBENCH
Symmetric Rank-2k Operations
433
793
SYRK
POLYBENCH
Symmetric Rank Operations
660
234
TRMM
PolY BELYBENCH
Triangular Matrix Multiplication
231
968
TRMM-OFT
POLYBENCH
Triangular Matrix Multiplication
Table 3: Dataset statistics for the input. The meanings of columns are: #K: # kernels, A#K: average # pragmas per kernel', A#T: average # source code tokens per kernel, A#N: average # nodes per kernel's graph, A#E; average # edges per kernel's graph, #nt: # node types, #pt: # pragma node types, nr: numeric attribute range, #it: instruction type, #ft: # flow types, #bt: block types, #ept: # edge position types, #eft: # edge position type.
Dataset
#K
A#K
A#T
A#N
A#E
#nt
Apt
Att
#nt
#bt
#ept
#eft
SDX (VI)
8.0
629.9
366.7
589.6
4
7
[0, 494]
8
56
3
VITIS (V2)
629.9
334.0
534.7
4
7
494]
3
on heuristics provided by AutoDSE [34] to generate the labels for subset of all possible designs. The labels come in the form of 5 target values: PERF, DSP, BRAM, LUT, and FF. In addition, according to whether the PERF is greater than a threshold value, we classify design into two categories: valid and invalid. For the valid designs, we perform the regression task of predicting each one of the target values. Table 4 shows the statistics of the prediction targets, i.e., output of a machine learning model.
Table 4: Dataset statistics for the output. The meanings of columns are #D for r: # designs for the regression task, PERF: range of the PERF target, DSP: range of the DSP target, BRAM: range of the BRAM target, LUT: range of the LUT target, FF: range of the FF target, #D for e: # designs for the
classification task, T:F: True class (Valid) False class (Invalid) design ratio.
Dataset
#D for
PERF
DSP
BRAM
FF
#D for
T:F
SDX (V1)
9439
[6.5, 1.5]
[0.0. 8.4)
[.00. 3.0]
[0.0, 6.5]
[0.0, 3.2]
28017
9439:18578
VITIS (V2)
5027
1-3.6, 6.6)
10.0. 6.61
0..0,0.1
10.0. 5.61
10.0, 1.91
14273
5027:9246
5

---

(a) Graph of 2MM consist-(b) Graph of CORRELA-(C) Graph of MVT consist- (d) Graph of Nw consist- ing of 354 nodes and 566 TION consisting 662 ing of 206 nodes and 328 ing 439 nodes and 704 edges.
nodes and 1071 edges. edges.
edges.
Figure 3: Visualization of the pragma-augmented PROGRAML graphs of four selected kernels. Node colors indicate the block attribute derived from the assembly code. Edge colors indicate the edge flow.
Since our task aims to predict the validity (classification) and quality (5-target regression) of the designs, we provide both the source code and the graph representation derived from PROGRAMI. [13]. Specifically, we follow [13] to compile a kernel's source code into assembly code, and then trans- form the assembly codes into control data flow graph (CDFG) with call relation, and eventually add pragma nodes to the PROGRAML graph following [30]. The resulting pragma-augmented PROGRAML graphs for four selected kernels depicted in Figure 3.
is noteworthy that there are multiple ways to represent the input source programs. As an illustration, we include in this dataset the graph representation used in [30]. Other representations are possible, as such abstract syntax trees. Since we release the source code, is possible to derive and generate such other representations.
5
Experiments on HLSYN
This section provides several baseline experiments with their results to investigate the performance of various methods on the task of design quality prediction.
5.1 Baseline Methods
All the baseline methods share the same encoder-decoder architecture and differ only encoder. Specifically, each method encodes a design into D-dimensional vector where D = 512 by default and uses a ML.--based decoder transform the design embedding into the targets.
CODE2VEC [2] CODE2VEC is path-based attention model. It first decomposes the code to collection of paths in its abstract syntax tree and represent them as bag of distributed vector representations. Then it uses an attention mechanism to compute a learned weighted average of the path vectors as the overall representation of the code.
CODET5-RAND, CODETS-FROZEN, and CODET5 [39] These three methods are based on the CODE2VEC method which performs pre- training small version released by the authors to encode the design. CODET5 fine-tunes both the encoder and the decoder, -ODETS-FROZEN freezes the encoder and only fine-tunes the decoder, while CODET5- RAND fine-tunes both but replaces the encoder parameters/weights with a random initialization to study the effect of pre-training on our tasks.
large amount of source code. We utilize the
In order to handle the long source code as sequence code tokens, we set the maximum sub- sequence length (i.e., the maximum number of tokens allowed in sub-sequence) to be 64, and apply a sliding window of size 64 over the source code to obtain multiple sub-sequences as input to the transformer-bast based encoder. At the beginning of each sub-sequence, special starting token is inserted and its embedding is taken as the sub-sequence level embedding, and all the sub-sequence embeddings are aggregated into the final D-dimensional embedding for the design.
G-CODEBERT [18] and G-CODEBERT-L, Similar to CODET5, G-CODEBERT is another pre-trained source code encoder, yet with larger embedding dimension (768 instead of 512), and is thus

---

Table 5: Regression (RMSE) SDx (v1) and VITIS (v2).
SDX (v1)
VITIS (V2)
Method
Trans
Ind
Ind Adapt
Trans
Ind
Ind Adapt
[ODE2VEC[2]
3.2877
4,2186
3.4156
2.9685
3.9076
2,9577
CODETS-RAND
1.7100
3,2206
2.2295
1,9999
2.9592
1.9573
CODEYS-FROZEN
2.6808
2.9447
2.2317
2.1822
2,9452
1.8447
CODETS [39]
0.5515
2,8301
1.8203
0.5270
3,4191
1.7552
G-CODEBERT [18]
0.5541
2.6639
1.5552
0.6637
3.1158
1,5175
G.CODERERT--.
2.7668
1,6666
3,2129
1.5561
GNN-DSE [30]
0.8641
3,1366
1,5180
0.5816
3.6764
1,5546
GNN-DSE-2L
0,8020
2.6887
1.6231
0.7644
3.2825
1.5688
[CONFTS.ONN-GSF]
0.4648
1,5274
(G-CODEBERT,,NN-GSSE)
0,5502
2.8521
1.5525
0.3940
3,6146
1.6017
presumably more expressive. G-CODEBERT-L uses larger maximum sub-sequence length, 128 instead of 64, which would capture a longer dependency between source code tokens, and may yield better results.
GNN-DSE [30] and GNN-DSE-2L In contrast to the previous methods which only receive the source code as input, these methods receive the assembly-level graph (with examples in Figure 3) input and use TRANSFORMERCONY [28] with a jumping knowledge network [42] to produce the design embeddings. GNN-DSE DSE employs 8 layers whereas GNN-DSE-2L. only utilizes 2.
[CODETS,GNN-GSE] and C--CODEBERT,,NN--E embeddings produced by source code transformer and a graph neural network- encoder, i.e., the input embedding into the decoder 2 x D instead of D.
These two methods concatenate the design
5.2 Evaluation Protocol
Metries There are two tasks for our dataset: regression and classification. The goal of the regression task is to predict the five targets: PERF, DSP, BRAM, LUT, and FF. We use rooted mean square error (RMSE) to evaluate each method. And the goal of the classification task is to predict whether design valid or not, i.e. whether the downstream RTL and physical synthesis are likely to complete or not. We use the classification accuracy as the evaluation metric.
In addition, recall that have versions of the dataset (SDx (V1) and VITIS (v2)). Each (ver- sion, combination receives separate evaluation. For each evaluation, there are two evaluating settings: transductive and inductive testing. Specifically, for each (version, task) combination: (1) We select six kernels as the held-out testing kernels. These kernels are never seen during training and are used for the inductive testing: (2) For the rest of the kernels, we merge all the labeled designs, and randomly split them into training, validation, and transductive testing designs with the 70:15:15 ratio; (3) Using the training designs for 1000 epochs for the regression task, and 200 epochs for the classification task, we train each baseline method. We employ the validation set to determine the best epoch to use for testing; (4) We test the trained model on the transductive testing set. is called transductive ("Trans") since this testing set contains designs from kernels that are seen during training; (5) We test the trained model on the held-out six kernels. Specifically, we (5.1) select the 30% designs from each held-out kernel the testing designs, (5.2) then repeat the following procedure times. For each kernel, from the rest of the 70% remaining designs, 10 designs are sampled and are utilized to adapt the trained model for 10 epochs. Then, the adapted model is tested on the 30% designs for that held-out kernel. We call such a setting inductive because the model is tested six kernels that are not visible in the training stage in zero-shot ("Ind") or few-shot learning setting I'Ind Adapt").
5.3 Results and Analysis
The overall regression and classification results are exhibited in Tables 5 and 6. Tables 7, 8, 9, and 10 reveal the breakdown results over individual held-out kernels.
There is no consistent winner among the baselines. For the regression task, G-CODEBERT and G-CODEBERT-L achieve the lowest error when adapted to the held-out kernels, whereas, for the classification task, GNN-based methods perform better. Such phenomenon calls for a hybrid model utilizing both the source code and the assembly code graph, e.g., the concatenation models [CODET5, SANUT5,GNN-GO GNN-GSE] and (G-CODEBERT,GNN--GS However, a simple concatenation of the
Observation 1:
7

---

Table 6: Classification aceuracy on SDX (v1) and VITIS (V2).
SDX (VI)
VITIS (V2)
Method
Trans
Ind
Ind Adapt
Irans
Ind
Ind Adapt
CODEZVEC
0.7576
0.5662
0.66117
0.7060
0,5444
0.6337
CODETS-RAND
0.8524
0.5257
0,4851
0.6291
CODEYS-FROZEN
0.7515
0.6098
0.7486
0.7334
0.4161
0.6061
CODETS [39]
0.9501
0.6394
0.7447
0.9045
0.4781
0.6734
G-CODEBERT
0.9536
0.6478
0.7610
0,9233
0.5415
0.7024
B-CODEBERT-LL
0.9204
0.5730
0.7701
0.8970
0.5342
0.7180
GNN-DSE [30]
0.9422
0,4781
0.6734
GNN-DSE-2L
0.8912
0.6085
0.7421
0.9126
0,5303
0,7632
[CODETS,GNN-OSP
0.9434
0.6141
0.7385
0.9195
0.5053
0.7002
(G-CODEBERT.ONN-OSEE
0.9212
0.6174
0.7446
0.9126
0.5001
0,7160
Table 7: Regression result breakdown on SDX (v1) on individual test kernels.
Method
DOITGEN-R
FDTD-2D
GEMM-N
JACOBI-2D
STENCIL-3D
TRMM-OPT
CODEZVEC
2.57TTH0.08
3.997320.19
5.619120.08
2.5710=0.11
2.902950.04
2.8322=0.30
CODETS-RAND
1.2123+0.02
3.3038+0. 10
3.8933+0.1 11
1.85840.. 12
0.9961 14
2,1133=0.009
CODET5-FROZEN
1.3197+0.00 00
3,2819+0.06
4.3073=0.03
1.7756+0.04
0.6807=0.03
2.0248=0.09
CODETS [39]
0,9990s0.27
2.981810.05
3.691020.41
1.347050.12
0.369520.06
1.533620,14
G G-CODEBERT [18]
0.8301+0.11
2.5938:0.15
3.3672:0.45
1.1581
0.4830=0.16
0.8990:0.
G-CODERERT-L. [18]
0.5376+0.08 08
2.8015#0.1
3.1310+0.52
1.0214:0.12
0.3636+0.07
0,9384=0,10
GNN-DSE [30]
0.527820003
2644910.10
2.890720.51
1.3855 10
0.691820.25
0.967320.05
GNN-DSF-21.
0.6984+0.09
1.1499
[CODFT5.ONN-OSE]
0.963740.20 20
2.799840.36
2.990840.16
1,299 +0.11
0.7373+0.
1.006600.04
[G-CODUDEXT,NNNGGEE]
100409+0221
2,7968+0.17
3,2942:0.23
1.0845+0.12
0.3072:0.06
0.7891#0.24
Table 8: Regression result breakdown on VITIS (v2) on individual test kernels.
Method
COVARIANCE
FDTD-2D-1.
GEMM-N
GEMM-P-L
TRMM-OFT
[ODE2VEC[2]
2.497720.0W
3.125920.18
4,552620.19 19
3.738920.03
L440950. 1I
2.637950.21
CODETŠ-RAND
1.638820.13
2.47770..5
3.0722a0.04
0.496950.01
CODETŠ-TROZEN
1.2137=0.02
2,449420,22
2.657640.10
3.1420#0.00
0. 0.385420.02
1.2199=0.00
CODETS [39]
1.359420.15
2.2958a0.07
2.843640.37
2.780340.07
0.4092s0.09
0.8427.0.14
G-CODERERT [18]
1.0652=0.06
1.9245:0.09
2.377240.31
2.391340.04
0.4019+0.08
0.9452+0.09
G-CODERFRT-L [18]
0.9107=0.08
L.770=00.1
2.663540.35
2.5022+0.00 09
0. 0.3384+0.03
1.1469+D.
GNN-DSE [30]
1.149620.04
2.3157400.23
2.7828+0.18
0.4110=0.02
0.779020.06
GNN-DSF-21.
1.1457=0.02
1.952100099
2.0248
2.9691 10
0.4067+0.04
0.9146+0.03
[ONDET5,ONNGGSE]
1.098:0007
1855220.10
2.2450H0.23
2.548950.04
0.376650006
1.44000.00
(G-CODEBERT,ONN-GSSE
1.1182:0.07
1.908820.10
2.535240.25
2.765520.08
0.3842:0.05
0.898620.11
Table 9: Classification result breakdown on SDX (vl) on individual test kernels.
Method
DOITGEN-R
FDTD-2D
GEMM-N
JACOBI-2D
STENCIL-3D
TRMM-OPT
0.644910.07
0.6192x0.01
0.459820:03
0.812220.05
0.608520.03
0.825620004
CODETS CODETS-RAND
0.666320.05
0.5283+0.08
0.636620.08
0.838420.03
0.588740.1 11
0.
CODETS-FROCE
0.7472+0.00
0.6414+0.00
0.6286#0. 17
0.8589:0.00
0.6318+0.00
0.9516200.0
CODET5
0.723610.04
0.645520.02
0.580420.04
0..68140001
0,704220.05
0.946720.01
G G-CODERERT [18]
0.660740.06
0.6455:0.03
0.671420.03
0.8509+0.02
0.8000=0.00 04
0.9377#0.02
G-CODEBERT-L [18]
0.7236+0.00 08
0.6556+0003
0.5741+0.07
0.8792+0.01
0.8394:0.05
0,9488=0.01
GNN DSE [30]
0.6506x00006
0.6889:0.02
0.640220.03
0.8652=0.03
0.7803=0.03
0.9488a0.00
GNN-DSF-21.
0.6674+0.05
0.6778+0.03
0.6545+0.02
0.8534+0.03
0.6648+0.04
0.9349+0.03
[CODETS.ONN-GS]]
0.702240.07
0.6:6450.02
0.643860.03
0.841950.03
0.662000.04
0,944620.01
[G-CODBBERT.GNN-GSE)
0.712420.06
0.7091:0.04
0.639320.04
0.861620.04
0.602820.09
0.942620.02
Table 10: Classification result breakdown on VITIS (v2) on individual test kernels.
Method
COVARIANCE
FDTD-2D-1.
GEMM-N
GEMM-P-1
TRMM-OPT
CODEZVEC [2]
0.583020.05
0,575020.02
0.509920.02
0.667820005
0.719120:05
0.747620.03
CODETS. RAND
0.6453=0.03
0.6444:0.04
0.618540.03
0.667840.04
0.6936=00005
0.5048=0.225
CODFTS-FRO7EN
0.6698=0.00
0.4778+0.11
0.4951 +0.04
0.6610+0.00
0.5234+0. II
0.8095+0.00
CODETS [39]
0.669820.04
0.575020.06
0.572840.03
0.698340.05
0.676620004
0.847620.03
n-CODERFRT [18]
0.5585=0.07
0.6111=0.07
0.584040.03
01837344004
0.9021
0.7214+0.07
G-CODEBERT-L [18]
0.5887=0.05
0.6583+0.08
0.6778+0.00
0.8203+0.00
0.7702+0.07
0.7929 0.002
ONN-DSE [30]
0.6509=0.01
0.684D40.06
0.755940.05
0.8170=0.08
0.8381
ONN-DSE-2L
0.6396e0.00 07
0.7639+0.10 10
0.7222+0.02
0.7763+0.00 06
0.8340+0.06
0.8429+0 02
[CODET5.cmm GNN GSE
0.649120.06
0.6361
0.649420.03
0.701740.04
0.812860.04
0.7524+0.09
|C-CODEBERT.ONN-GSEE
0.639620.03
0.608330.05
0.669110.05
0.769520.03
0.821320.07
0.788120.03
8

---

design embeddings does not consistently yield better performance. Particularly, [CODETS,GN] GNNGGSE] and IN-NODEBERT,GNN GNN-GSE] achieve the lowest regression error under the transductive setting but fall short when adapted to new kernels. Such results imply that future efforts can be made on studying the generalization abilities of machine learning models on our HI.SYN benchmark.
In general, pre-training helps with the performance of source code transformer models. This can be seen by comparing CODET5-RAND and CODETS, where the former starts training from scratch while the latter loads a pre-trained model released by [39]. This should not come as a surprise, because pre-training has been shown to demonstrate success in natural language processing. Our experimental results verify the effectiveness of pre-training on the HLSy YN benchmark. One implication that one can design better pre-training methods on source code related to electronic design automation, or even design pre- trainini for GNNs operating on assembly-level graphs.
Observation 2:
Ohservation 3: More GNN message passing layers does not necessarily improve the performance of GNN models. In many cases, the 2-layer version, GNN-DSE-2L, performs even better than the 8-layer version, GNN-DSE. This may be attributed to the fact that an attention-based global readout function is used to aggregate node embeddings into a graph-level embedding representing the entire design, and thus each node does not necessarily need to reach far-away nodes in the local message passing stage.
Observation 4: Generalization to new kernels is difficult, and the performance after adaptation differs across kernels. For example, G-CODERERT-LL achieves the overall lowest ("Ind Adapt") on SDX (V1) on the regression task, but Table demonstrates that G-CODEBERT-L does not always yield the lowest error each of the six held-out kernels. For example, [G-CODEBERT, GNN-GSEE performs the best on STENCIL-3D and TRMM-OPT, but poorly on DOITGEN-R, and thus the average regression error over the six held-out kernels is higher than G-CODEBERT-L. This calls for a more in-depth study of the discrepancy between kernels and a model that is capable of generalizing to a diverse of kernels. In general, adaptation is necessary, since across all the experiments, without any adaptation ("Ind"), directly applying the trained model to new unseen kemels leads to poor regression error and classification accuracy.
6 Conclusion and Future Work
This work introduces the task of design quality prediction in the forms of regression and classification tasks and presents the HLSYN benchmark evaluate state-of-the-ar program representation learning methods. Although there is no method that consistently outperforms all the other methods, we notice several trends and identify promising directions toward a more accurate prediction model design. As program representation learning is continuously growing research domain, we plan to maintain the benchmark to test new methods. For example, our recent work [31] uses hierarchical graphs for program representations to predict design performance. Our HLSYN benchmark is growing project. We expect to include more kernels and labeled designs running newer versions of HLS tools and establish leaderboard to encourage participation. In addition, the current benchmark does not consider the design space exploration (DSE) stage, which will be added as the project develops. In fact, the regression task aims to be eventually integrated into the DSE process, which traverses the design space in order to find the optimal pragma setting.
7 Acknowledgement
This work was partially supported by NSF 2211 NSF 1937599, NSF 2119643, NSF 2303037, NASA, SRC JUMP 2.0 Center, Okawa Foundation, Amazon Research, Cisco, Picsart, Snapchat, and CDSC industrial partners http:://cdsccucch..dd//rrrrr))r We would also like to thank AMD/Xiliix for HACC equipment donation and Marci Baun for editing the paper.
References
[1] Sergi Abadal, Akshay Jain, Robert Guirado, Jorge López-Alons and Eduard Alarcón. Computing graph neural networks: A survey from algorithms to accelerators. ACM Computing Surveys (CSUR).
hnttps://githin httph//w...cc//AAAA/////////////

---

54(9):1-38, 2021.
[2] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: Learning distributed representations of code. Proceedings of the ACM on LanguagenggLnggugsss,..o
Languages, 3PPPL): Lang:ggs.3/PPPP)11113 2019.
[3] AMD/Xilinx Vivado
HLS.
B-Engishh/
ug902- - www.viivvvviiiiie -ech-eeee--ae synthesis.
[4] AMD/Xilinx Vivado HLS.
https://docs. iitesxxxiiiix.. wttpiiiw..o///////////e/e/ 2- English/
[5] Yunsheng Bai, Atefeh Sohrabizadeh, Yizhou Sun, and Jason Cong. Improving GNN-hased accelerator design automation with meta learning In Proceedings of the 59th ACM/IEEE Design Automation Conference, pages 1347-1350. 2022.
[6] Cadence Stratus High-Level Synthesis. digital-design-ane httpeeinnnndddiido//oo/e//// https///ssp.peesssssssaaaaaaaaaa htmm.
https:// cttpiieeeecoommeeuSeSSSSSSS http://ennSSShmmeeeeeeeee
[7] David Castells-Rufas, Vinh Ngo, Juan Borrego-Carazo, Mare Codina, Carles Sanchez, Dehora Gil, and Jordi Carrabina. A survey of FPGA-based vision systems for autonomous cars. IARE Access, 13:5252 132563, 2022.
[8] Yuze Chi, Weikang Qiao, Atefeh Sohrabizadeh, Jie Wang, and Jason Cong. Democratizing domain-specific computing. Communications of the ACM, 66(1):74-85) 85, 2022.
[9] Eric Chung, Jeremy Fowers. Kalin Ovtcharov, Michael Papamichael. Adrian Caulfield, Todd Massengill, Ming Liu, Daniel Lo. Shlomi Alkalay, Michael Haselman, et al. Serving DNNs in real time at datacenter scale with project brainwave. iEEE Micro, 38(2):8-20, 2018.
[10] Jason Cong, Jason Lau, Gai Liu, Stephen Neuendorffer, Peichen Pan, Kees Vissers, and Zhiru Zhang. FPGA HLS today: successes, challenges, and opportunitics.. ACM Transactions on Reconfigurable Technology and Systems (TRETS), 15(4):1-22 2022.
[11] Jason Cong, Bin Liu, Stephen Neuendorffer, Juanjo Noguera, Kees Vissers, and Zhiru Zhang. High-level synthesis for FPGAs: From prototyping to deployment. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 30(4):473-44) 491, 2011.
[12] Jason Cong, Peng Wei, Cody Hao Yu, and Peng Zhang. Automated accelerator generation and optimization with composable, parallel and pipeline architecture. DAC, 2018
[13] Chris Cummins, Zacharias Fisches, Tal Ben-Nun, Torsten Hoefler, Michael FP O' O'Boyle, and Hugh Leather. Programl: A graph-hased program representation for data flow analysis and compiler optimiza- tions. In International Conference on Machine Learning, pages
PMLR, 2021.
[14] William J Dally, Yatish Turakhia, and Song Han. Domain-specific hardware accelerators. Communications of the ACM, 63(7):48-57, 2020.
[15] Javier Duarte, Philip Harris, Scott Hauck, Burt Holzman, Shih Chich Hsu, Sergo Jindariani, Suffian Khan, Benjamin Kreis, Brian Lee, Mia Liu, et al. FPGA-aacelerated machine learning inference as a service for particle physics computing. Computing and Software for Big Science, 3:1-15, 2019.
[16] Lorenzo Ferretti, Jihye Kwon, Giovanni Ansaloni, Giuseppe Di Gugliclmo, Luca Carloni, and Laura Pozzi. DB4HLS: databasc of high-level synthesis design space explorations. IEEE Embedded Systems Letters, 13(4))194-15) 197, 2021.
[17] Jeremy Fowers, Kalin Ovtcharov, Michael Papamichael, Todd Massengill, Ming Liu, Daniel Lo, Shlomi Alkalay, Michael Haselman, Logan Adams, Mahdi Ghandi, et al. A configurable cloud-scale DNN processor for real-time AT. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), pages 1-14. TEFF, 2018.
[18] Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujic Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Grapheodebert: Pre-training code representations with data flow arXiv preprint htt:::00000886666 2020.
[19] Muhuan Huang. Di Wu, Cody Hao Yu. Zhenman Fang, Matteo Interlandi, Tyson Condie, and Jason Cong. Programming and runtime support to blaze FPGA accelerator deployment at datacenter scale. In Proceedings of the Seventh ACM Symposium Cloud Computing, pages 456-469, 2016.
10

---

[20] Intel High Level Synthesis Compiler. http:///wtp/////aiaiie...oo.oo.
programmable/ a pawparruuuspiiieeee/-ecomamppppp
html.
[21] Norman P Jouppi, Cliff Young. Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of tensor processing unit In Proceedings of the 44th annual international symposium on computer architecture, pages 1-12, 2017.
[22] David Koeplinger, Raghu Prabhakar, Yaqi Zhang, Christina Delimitrou, Christos Kozyrakis, and Kunle Olukotun. Automatic generation of efficient accelerators for reconfigurable hardware. ISCA, pages 115-127, 2016.
[23] Hung-Yi Yi and Luca P Carloni. On learning-based methods for design-space exploration with high-level synthesis. In DAC, pages 1-7, 2013.
[24] NEC CyberWorkBench. https://wwwl http::///wwwncccc new. cww/e// global/prodleooooooooooooo http://ww//iitt /it//xinxxxx.///// httml.
[25]
Andrew Putnam, Adrian Caulfield, Eric Chung, Derek Chiou, Kypros Constantinides, John Demme, Hadi Esmaeilzadeh, Jeremy Fowers, Jan Gray, Michael Haselman, Scott Hauck, Stephen Heil, Amir Hormati, Joo- Young Kim, Sitaram Lanka, Eric Peterson, Aaron Smith, Jason Thong. Phillip Yi Xiao, Doug Burger, Jim Larus, Gopi Prashanth Gopal, and Simon Pope. A reconfigurable fabric for accelerating large-scale datacenter services. In Proceeding of the 41st Annual International Symposium Computer Architecuture (ISCA), pages 13-24. TEEE Press, June 2014. Selected
TEEE Micro TopPick.
[26] Brandon Reagen, Robert Adolf, Yakun Sophia Shao, Gu-Yeon Wei, and David Brooks. Machsuite: Benchmarks for accelerator design and customized architectures. In IISCC, 2014.
[27] Ahmad Shawahna, Sadiq M Sait, and Aiman EI-Maleh. FPGA-based accelerators of deep learning networks for learning and classification: A review icce Access, 7:7823-7859, 2018.
[28] Yunsheng Shi, Zhengjie Huang. Shikun Feng. Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. IJCAI, 2021.
[29] Siemens Catapult High-Level Synthesis. https://eda. sw. www.sssess.... cot.en..comeeenne/eeeeeeeeee Bttee/n///////ceeeeeeeeeee hi hetp-leeel-syttheeee wwweeeeesssadddddeeeedeeee aend-verifiaation wtt.rriiaaaiioopppaarroomoooo//
[30] Atefeh Sohrabizadch, Yunsheng Bai, Yizhou Sun, and Jason Cong. Automated accelerator optimization aided by graph neural networks. In Proceedings of the 59th ACM/IEEE Design Automation Conference. pages 55-60, 2022.
[31] Atefeh Sohrabizadeh, Yunsheng Bai. Yizhou Sun. and Jason Cong. Robust GNN- representation learning for HLS. In The International Conference Computer-Aided Design, 2023.
[32] Atefch Sohrabizadeh, Yuze Chi, and Jason Cong. StreamGCN: Accelcrating graph convolutional networks with streaming processing. In 2022 IEEE Custom Integrated Circuits Conference (CICC), pages 1-8.
IEEE, 2022.
[33] Atefeh Sohrabizadeh, Jie Wang, and Jason Cong. End-to-end optimization of deep learning applications. In Proceedings of the 2020 ACM/SIGDA International Symposium on Proel--rorramabblee Gate Arryys pages 133-139, 2020.
[34] Atefeh Sohrabizadeh, Cody Hao Yu, Min Gao, and Jason Cong. AstoDDE: Enabling software programmers to design efficient FPGA accelerators. ACM Transactions on Design Automation of Electronic Systems (TODAES). 27(4) 2022.
[35] Qi Sun. Tinghuan Chen, Siting Liu, Jin Miao, Jianli Chen, Hao Yu, and Bei Correlated oulti-objective multi-fidel culti-fidelity optimization for HLS directives design. In IEEE/ACM Proceedings Design, Automation and Test in Europe (DATE), pages 01-05, 2021.
[36] Ecenur Ustun, Chenhui Deng, Debjit Pal, Zhijing Li, and Zhiru Zhang. Accurate operation delay prediction for FPGA HLS using graph neural networks. In Proceedings of the 39th International Conference on Computer-Aided Design, pages 1-9, 2020.
[37] Jie Wang. Licheng Guo, and Jason Cong. AutoSA: A polyhedral compiler for high-perffrmance systolic arrays on FPGA In Proceedings of the 2021 ACM/SIGDA international symposium on prelaamrorr gate arrays, 2021.
[38] Shuo Wang, Yun Liang, and Wei Zhang. Flexel: An analytical performance model for OpenCL workloads flexible FPGAs In DAC, pages 1-6, 2017.

---

[39] Yue Wang, Weishi Wang. Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. EMNLP. 2021.
[40] Nan Wu, Yuan Xie, and Cong Hao. Ironman-pro: Multi-objective design space exploration in HI.S via reinforcement learning and graph neural network based modeling. IEEE Transactions Computer-Aides Design of Integrated Circuits and Systems, 2022.
[41] Nan Wu, Hang Yang. Yuan Xie, Pan Li, and Cong Hao. High-level synthesis performance prediction using GNNs: Benchmarking, modeling, and advancing Proceedines of 59th ACM/IEEE Design Automation Conference, pages 49 54, 2022.
[42] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. ICML, 2018
[43] Cody Hao Yu. Peng Wei, Max Grossman, Peng Zhang, Vivek Sarker, and Jason Cong. S2FA: an accelerator automation framework for heterogeneous computing in datacenters.. In DAC, pages 1-6, 2018.
[44] Tomofumi Yuki and Louis-Noël Pouchet. Polybench/c.
[45] Jieru Zhao, Liang Feng, Sharad Sinha, Wei Zhang, Yun Liang, and Bingsheng He. COMBA: compre- hensive model-based analysis framework for high level synthesis real applications. ICCAD, pages 430-437, 2017.
[46] Guanwen Zhong, Alok Prakash, Yun Liang, Tulika Mitra, and Smail Niar. Lin-analyzer: a high-level performance analysis tool for FPGA-based accelerators. In DAC, pages 1-6, 2016.
[47] Guanwen Zhong, Vanchinathan Venkataramani, Liang, Tulika Mitra, and Smail Niar. Design space exploration of multiple loops on FPGAs using high level synthesis. In ICCD, pages 456-463, 2014.
12


Reviews:
[{"review_id": "hGaLZSk15q", "reviewer": "Reviewer_qZxJ", "summary": "", "questions": "", "limitations": "Yes, this paper discussed its limitations and future work in the last section. ", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The collected design points are one order of magnitude larger than previous databases. The performance prediction aims to predict both the validity (classification) and quality (5-target regression) of the design. \n2. Various SOTA baseline methods (i.e., 10 methods in total) are used for experiments on HLSyn. In addition, detailed evaluations are provided. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "vePogdtPRU", "reviewer": "Reviewer_mcz1", "summary": "", "questions": "", "limitations": "There are minor improvements which authors can easily take care of:\n\n1) Authors claim more complex programs (line 11 and line 92) however they used kernels from Machsuite and polybench which exist in HLS literature. If authors are not including new programs, I suggest dropping the term of more complex programs compared to prior work which is misleading.\n\n2) Line 147, “Nevertheless, due to the different heuristics employed by HLS tools in the design process, these models may not accurately predict the QoR”. Provide citation or detailed explanation on this statement.\n\n3) Line 225: “How are the subsequence embedding aggregated?, Please explain.”\n\n4) Section 5.2 description is *too* much confusing: especially the “Ind-adapt”. When authors say 10 designs out of 70% of designs, do they mean 70% designs out of training data or 70% designs created out of held-out kernels? Based on the realistic assumption of Ind-adapt, I consider 10 designs created out of held-out kernels for adaptation for 10 epochs. But, what does adaptation indicate? Fine-tuning? Please explain.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "Authors have combined POLYBENCH and MACHSUITE benchmarks; used pragma configurations existing on the SOTA methods and generated a set of labeled data to be used for training a regressor (instead of using HLS tools to get accurate results)", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "sokOGWEwcl", "reviewer": "Reviewer_C7Q3", "summary": "", "questions": "", "limitations": "1. It is unclear how accurate the evaluated ML-based methods are compared to a fast compiler-based analysis to predict the latency and resource utilization (that is present in most of the existing HLS tools, such as AMD Vivado HLS/SDx/VITIS and Intel OpenCL). The paper failed to mention the comparison to the existing non-ML-based analysis in terms of both accuracy and speed. \n\n2. The dataset is composed of programs from existing benchmarks and only targets AMD/Xilinx HLS compiler as the pragmas are critical to performance in these tools. The labels and evaluations are collected on a specific FPGA Alveo U200. It would be more useful to the ML for the compiler community if it targets multiple devices and not only covers various pragmas. ", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "This work provides valuable insights into how different program feature extraction networks perform in predicting HLS program validity and QoR. For instance,  the study (table 5 and 6) demonstrates that G-CODEBERT models achieve high ind-adapt accuracy in predicting QoR of the HLS programs, while GNN-DSE exhibits high accuracy for classification tasks. It also demonstrates that the generalizability of the networks trained on the dataset is poor, which suggests that a better network or a more comprehensive dataset is needed for training. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Zr1nDLzUvM", "reviewer": "Reviewer_JA6Z", "summary": "", "questions": "", "limitations": "The authors have discussed the limitation of their work in the last section of the main paper.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "* Inclusive collection of kernels spanning diverse application domains.\n* Evaluation of multiple SOTA design quality prediction methods utilizing the proposed dataset, yielding valuable observations for enhanced model design.\n* Provision of quality source code and comprehensive readme for training and testing of the methods.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "1suhXtjqHI", "reviewer": "Reviewer_455o", "summary": "", "questions": "", "limitations": "Currently three optimization pragmas (i.e., PIPELINE, PARALLEL, and TILE) are considered. Can we include more pragma types? Also, there could be more options for each pragma. Of course, we understand that the generation of a larger dataset takes time and we expect the expansion of this dataset in the future. \n", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "HLSYN encompasses a diverse range of optimization pragmas that can pipeline and/or parallelize computation, as well as adjust data caching. The benchmark is comprised of 42 unique kernels from various domains, totaling over 42,000 design points, providing a comprehensive resource for advancing research and facilitating an in-depth exploration of machine learning techniques for HLS. \n\nThe paper & document are detailed and well structured. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: All the reviewers are overall positive about the paper. ACs don't see obvious reason to overturn the ratings.
Decision: Accept (Poster)