SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yunsheng Bai, Atefeh Sohrabizadeh, Zongyue Qin, Ziniu Hu, Yizhou Sun, Jason Cong
{yba,atefehsz,qinzongyue,bull,yzsun,cong}@cs.ucla.edu
High-level synthesis (HLS) aims to raise the abstraction layer in hardware design, enabling the design of domain-specific accelerators (DSAs) targeted for field-programmable gate arrays (FPGAs) using C/C++ instead of hardware description languages (HDLs). Compiler directives in the form of pragmas play a crucial role in modifying the microarchitecture within the HLS framework. However, the number of possible microarchitectures grows exponentially with the number of pragmas. Moreover, the evaluation of each candidate design using the HLS tool consumes significant time, ranging from minutes to hours, leading to a slow optimization process. To accelerate this process, machine learning models have been used to predict design quality in milliseconds. However, existing open-source datasets for training such models are limited in terms of design complexity and available optimizations. In this paper, we present HLSYN, a new benchmark that addresses these limitations. It contains more complex programs with a wider range of optimization pragmas, making it a comprehensive dataset for training and evaluating design quality prediction models. The HLSYN benchmark consists of 42 unique programs/kernels, each of which has many different pragma configurations, resulting in over 42,000 labeled designs. We conduct an extensive comparison of state-of-the-art baselines to assess their effectiveness in predicting design quality. As an ongoing project, we anticipate expanding the HLSYN benchmark in terms of both quantity and variety of programs to further support the development of this field.
In recent decades, the demand for specialized computing systems tailored to specific applications has significantly increased. This has led to the emergence of domain-specific accelerators (DSAs) being implemented in either application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). By leveraging the unique characteristics of specific workloads, the designer can design DSAs to enhance performance and energy efficiency. This becomes particularly valuable when general-purpose processors like CPUs and GPUs cannot meet the performance or efficiency requirements of certain applications due to the end of Dennard scaling [8, 14]. For instance, Google has developed its custom-designed DSA in the form of an ASIC named the Tensor Processing Unit (TPU) [21], which is highly optimized for machine learning workloads, offering remarkably faster performance and improved energy efficiency compared to CPUs and GPUs. In addition, FPGAs offer a cost-effective alternative with reconfigurability, making them increasingly appealing for accelerating applications across various domains, including search engines and numerous datacenter applications [25, 9, 19], machine learning inference acceleration [15, 17, 32, 1, 27], and autonomous vehicles [7], among others.
Nevertheless, the design of DSAs poses distinct difficulties in contrast to general-purpose hardware like CPUs and GPUs [34, 8]. DSAs are commonly developed using hardware description languages (HDLs) at the register-transfer level (RTL), specifically Verilog and VHDL, which are primarily known to circuit designers. To address this challenge, high-level synthesis (HLS) [11, 10] was introduced and is now supported by most EDA (Electronic Design Automation) and FPGA companies [4, 6, 20, 24, 29]. HLS raises the level of design abstraction to  $\mathrm{C / C + + } / \mathrm{OpenCL} / \mathrm{SystemC}$ , enabling designers to describe the high-level behavioral representation of their designs rather than the low-level data transition in RTL. This abstraction eliminates the need for explicit clock scheduling specifications in the HLS code. Instead, HLS tools analyze the behavior description to schedule operations across different clock cycles, assign operations to available resources, and establish the required control structure. Finally, the HLS tool automatically generates RTL code based on these analyses. It can take several minutes to hours for the HLS tool to generate this RTL code [34]. The RTL can then be passed through logic synthesis and physical design steps, which can consume several hours, to be implemented on the target FPGA. This HLS tool enhances design productivity, shortens design cycles, and allows designers to rapidly explore various design options without the need for manual RTL code writing.
Despite the increased level of design abstraction offered by high-level synthesis (HLS) tools, they still require a considerable amount of hardware design expertise to utilize synthesis directives in the form of pragmas. These pragmas play a crucial role in specifying various aspects of the design, such as memory organization, caching strategies, memory buffer partitioning, parallelization and pipelining of computations, etc. As demonstrated by Chi et al. [8], although the performance of a DSA with no performance-optimizing pragmas can be  $108 \times$  slower than a CPU, through proper optimization, it can achieve a remarkable performance improvement, surpassing a CPU by  $89 \times$ . However, the optimization process for
architecture-specific enhancements is typically limited to hardware programmers and falls beyond the capabilities of the average software programmers. Consequently, there has been a growing focus on automating this optimization process. While some approaches treat the HLS tool as a black box and develop custom-designed heuristics to search through design candidates, a more recent research paradigm leverages machine learning and deep learning techniques. These approaches either learn the behavior of the HLS tool and construct predictive models or employ data-driven exploration methods to search through the solution space. The goal of automating the optimization process is to democratize customized computing and make it more accessible for the average software programmers, allowing them to utilize the tailored hardware acceleration.
To address the need for automating pragma insertion and parameter tuning in high-level synthesis (HLS), machine learning techniques can be used. This approach aims to achieve optimal quality in terms of latency and resource utilization. However, the lack of open-source datasets in this domain and the limitations of existing datasets, which are constrained in terms of design complexity and available optimizations, restrict their practicality. To bridge this gap, this paper introduces HLSYN, the first
FPGA Implementation for the Source Code
Figure 1: High-level synthesis (HLS) transforms the source code of the kernel 2MM written in C into a lower-level programming language and eventually implements the design on the target FPGA as shown with the chip die photo.
comprehensive benchmark for HLS designs targeted to FPGAs for performance optimization. This benchmark provides more complex programs and a wider range of optimization pragmas, facilitating advanced research and facilitating in-depth exploration of machine learning techniques in the context of HLS. In our study, we define a design as a piece of  $\mathsf{C} / \mathsf{C} + +$  source code (referred to as a kernel) with associated pragmas. Our primary focus is on predicting the quality of a design using supervised learning by running a model trained on a collection of labeled designs with corresponding quality metrics. However, our dataset can be utilized in various training scenarios, including training agents to efficiently explore the solution space.
The task of HLSYN is to predict the quality of the HLS design specified by a program (kernel) with a specific optimization pragma design. Our target implementation platform is FPGA, although similar techniques can be applied to ASIC accelerator designs as well. The quality of a design is defined as a function of its performance, which is measured by its latency in cycle counts, and its area/resource utilization, such as the usage of digital signal processing blocks (DSP), blocks' RAMs (BRAM), flip-flops (FF), and lookup-tables (LUT), which are the fundamental building blocks for implementing digital logic circuits in FPGA designs.
In this work, we specifically consider the optimization pragmas of the Merlin Compiler, an open-source source-to-source optimization tool used for efficient AMD/Xilinx HLS designs. The Merlin Compiler provides three types of optimization pragmas, namely PIPELINE, PARALLEL, and TILE to define the desired microarchitecture [34].
As illustrated in Figure 1, these pragmas can be applied at the loop level and offer control over the type of pipelining, the parallelization factor, and the amount of data caching. If setting the pragmas properly to non-default parameters for proper parallelizing and pipelining the computation, the resulting accelerator can be  $10 \times$  or even  $100 \times$  faster than a single-core CPU. However, without any pragma insertion, the resulting hardware can be  $10 \times$  slower than a CPU. Figure 2 demonstrates an example where the prediction targets are sensitive to the pragma settings. It is noteworthy that in some other examples, a change in the pragma settings does not lead to any change in the output targets. The machine learning model must learn from existing labeled designs and understand the source code as
well as the pragmas in order to accurately forecast the outcome of each design when eventually being executed on an FPGA.
Table 1 summarizes the parameter space of these pragmas. For a given program/kernel, any change in the option of any of the pragmas results in a different design with a unique microarchitecture. The "fg" option in pipelining refers to the case where all the inner loops are unrolled (parallelized with separate logic) and each parallel unit is pipelined. The "cg" option, on the other hand, results in coarse-grained processing elements (PEs) that are pipelined together. For example, it can create pipelined load-compute-store units. The PARALLEL and TILE pragma take numeric values that determine the degree of parallelization and loop tiling, respectively.
Figure 2: Two example designs selected from the 2MM kernel. In Figure 1, the kernel contains 14 pragmas, and each row has the format "<pragma name>": <pragma setting>". A small change in one of the pragma parameters leads to changes in the FPGA HLS results, i.e., the five prediction targets for the regression task.
Table 1: Target pragmas with their options.
Pragma Name	Parameter Name	Parameter Space	Examples of Pragma Settings
PARALLEL	factor	integer	4,8
PIPELINE	mode	“cg”, “fg”, off	‘flatten’ resulting in the "fg” mode
TILE	factor	integer	2,4
In previous research, optimizing HLS designs has been approached in different ways. One category of methods treats the HLS tool as a black box and utilizes problem-independent heuristics or develops dedicated heuristics to explore the solution space and evaluate the quality of results (QoR) directly using the tool [34, 43, 35]. However, this approach is time-consuming as each evaluation takes several minutes to hours. To mitigate this issue, another category of methods aims to create surrogate models for the HLS tool. Some of these methods construct dependency graphs of the program and employ traditional graph analysis techniques to schedule operations and estimate the QoR accordingly [46, 38], while others develop analytical performance and area models to estimate the QoR [45, 47]. Nevertheless, due to the different heuristics employed by HLS tools in the design process, these models may not accurately predict the QoR [34]. Some methods address this limitation by focusing on designs that can exploit pre-defined microarchitecture templates or follow specific computation patterns [33, 37, 12], but this can limit their generality. Alternatively, a data-driven approach utilizing machine learning and deep learning models has been proposed to enhance prediction accuracy [23, 22]. Graph neural networks (GNNs) have gained attention in this context and demonstrated promising results in enhancing prediction accuracy [30, 5, 40, 36].
A fundamental aspect of these approaches is the availability of a large-scale database to effectively train the models. Recent works have focused on gathering such databases [16, 41, 30]. Unfortunately, existing datasets have limitations. The dataset in [41] predominantly consists of synthetic programs that do not utilize any pragmas. DB4HLS [16] targets programs from the MachSuite benchmark[26] but overlooks the inclusion of a key optimization pragma, pipelining. Additionally, DB4HLS views each function in the program as an individual kernel. GNN-DSE [30] targets programs from the Polyhedral benchmark [44], which features more complex kernels for FPGA mapping, in addition to the MachSuite benchmark. This dataset considers a program with all its sub-functions as a kernel, further increasing design complexity. Despite covering a wide range of optimization pragmas for each kernel, the generated dataset is small, with only 9 target kernels and a total of 4,752 data points. To address these limitations, we propose HLSYN, which includes kernels from both the Polyhedral and MachSuite benchmarks. It encompasses a diverse range of optimization pragmas that can pipeline and/or parallelize computation, as well as adjust data caching. Our benchmark is comprised of 42 unique kernels from various domains summarized in Table 2, totaling over 42,000 design points, providing a comprehensive resource for advancing research and facilitating an in-depth exploration of machine learning techniques for HLS.
4 The HLSYN Benchmark
In this section, we introduce the datasets in HLSYN<sup>1</sup>. The input data source comes from 42 selected kernels in the MACHSUITE benchmark [26] and the POLYBENCH benchmark suite [44]. Our selected 42 kernels cover a wide range of applications whose descriptions are shown in Table 2.
The benchmark consists of 2 datasets accumulated in the past three years, corresponding to two versions of AMD/Xilinx HLS tools: (1) SDX (v1) [3] and VITIS (v2) [4], with the AMD/Xilinx Alveo U200 as the target FPGA and a working frequency of  $250\mathrm{MHz}$ . For each dataset, we select 6 kernels as the held-out testing kernels. They will test the ability of a model to generalize to kernels that it did not see during the training. For the rest of the kernels, we perform a random split with the training, validation, and testing ratio being 70:15:15. Summary statistics of datasets are given in Table 3.
For each dataset kernel in each dataset, we run the two versions of the tools described above to obtain a set of labeled designs. Since the design space size is exponential to the number of pragmas, we rely
<sup>1</sup>https://github.com/UCLA-DM/HLSyn
Table 2: There are 42 kernels in total across SDx (v1) and VITIS (v2) spanning multiple domains such as linear algebra on vectors and matrices, data mining, stencil operations, encryption, dynamic programming, etc. #p denotes the number of pragmas in the kernel. #in v1 and #in v2 denote the number of labeled designs in SDx (v1) and VITIS (v2) respectively.
Kernel	Source	Description	# pragmas	# in v1	# in v2
2MM	POLYBENCH	2 Matrix Multiplications	14	812	861
3MM	POLYBENCH	3 Matrix Multiplications	21	848	-
ADI	POLYBENCH	Alternating Direction Implicit solver	13	551	-
AES	MACHSUITE	Advanced Encryption Standard	3	45	43
ATAX	POLYBENCH	Matrix Transpose and Vector Multiplication	5	884	902
ATAX-MEDIUM	POLYBENCH	Matrix Transpose and Vector Multiplication	5	362	544
BICG	POLYBENCH	BiCG Sub Kernel of BiCGStab Linear Solver	5	512	498
BICG-LARGE	POLYBENCH	BiCG Sub Kernel of BiCGStab Linear Solver	4	-	456
BICG-MEDIUM	POLYBENCH	BiCG Sub Kernel of BiCGStab Linear Solver	5	316	-
CORRELATION	POLYBENCH	Correlation Computation	17	1522	699
COVARIANCE	POLYBENCH	Covariance Computation	13	-	356
DOITGEN	POLYBENCH	Multiresolution Analysis	6	179	172
DOITGEN-R	POLYBENCH	Multiresolution Analysis	7	595	230
FDTD-2D	POLYBENCH	2-D Finite Different Time Domain Kernel	16	660	-
FDTD-2D-L	POLYBENCH	2-D Finite Different Time Domain Kernel	16	-	240
GEMM-B	MACHSUITE	Blocked Version of Matrix Multiplication	9	775	440
GEMM-N	MACHSUITE	Matrix transpose and Vector Multiplication	7	749	540
GEMM-P	POLYBENCH	Matrix Multiplication	8	1160	714
GEMM-P-L	POLYBENCH	Matrix Transpose and Vector Multiplication	8	-	199
GEMVER	POLYBENCH	Vector Multiplication and Matrix Addition	13	924	712
GEMVER-M	POLYBENCH	Vector Multiplication and Matrix Addition	13	3365	-
GESUMMV	POLYBENCH	Scalar, Vector and Matrix Multiplication	4	442	371
GESUMMV-M	POLYBENCH	Scalar, Vector and Matrix Multiplication	4	304	-
HEAT-3D	POLYBENCH	Heat Equation over 3D Data Domain	11	1664	-
JACOBI-1D	POLYBENCH	1-D Jacobi Stencil Computation	5	595	-
JACOBI-2D	POLYBENCH	2-D Jacobi Stencil Computation	11	1862	-
MD	MACHSUITE	n-body Molecular Dynamics	3	12	-
MVT	POLYBENCH	Matrix-Vector Product and Transpose	8	1175	1452
MVT-M	MACHSUITE	Matrix-Vector Product and Transpose	8	416	-
NW	MACHSUITE	Dynamic Programming for Sequence Alignment	6	1347	615
SEIDEL-2D	POLYBENCH	2-D SeidelStencil Computation	7	1314	-
SPMV-CRS	MACHSUITE	Sparse Mat-Vec Mult. w/ Variable-Len. Neighbor	3	114	114
SPMV-ELLPACK	MACHSUITE	Sparse Mat-Vec Mult. w/ Fixed-size Neighbor	3	114	102
STENCIL	MACHSUITE	A Two-DimensionalStencil Computation	7	1404	1016
STENCIL-3D	MACHSUITE	A Three-DimensionalStencil Computation	5	239	239
SYMM	POLYBENCH	Symmetric Matrix Multiplication	7	153	158
SYMM-OPT	POLYBENCH	Symmetric Matrix Multiplication	8	-	281
SYMM-OPT-M	POLYBENCH	Symmetric Matrix Multiplication	8	315	-
SYR2K	POLYBENCH	Symmetric Rank-2k Operations	8	433	793
SYRK	POLYBENCH	Symmetric Rank-k Operations	8	660	234
TRMM	POLYBENCH	Triangular Matrix Multiplication	7	231	968
TRMM-OPT	POLYBENCH	Triangular Matrix Multiplication	7	964	281
Table 3: Dataset statistics for the input. The meanings of columns are: #K: # kernels, A#K: average # pragmas per kernel', A#T: average # source code tokens per kernel, A#N: average # nodes per kernel's graph, A#E: average # edges per kernel's graph, #nt: # node types, #pt: # pragma node types, nr: numeric attribute range, #it: # instruction type, #ft: # flow types, #bt: # block types, #ept: # edge position types, #eft: # edge position type.
Dataset	#K	A#K	A#T	A#N	A#E	#nt	#pt	nr	#it	#ft	#bt	#egt	#eft
SDX (v1)	37	8.0	629.9	366.7	589.6	4	7	[0, 494]	82	8	56	3	4
ViTIS (v2)	29	7.6	629.9	334.0	534.7	4	7	[0, 494]	74	8	56	3	4
on heuristics provided by AutoDSE [34] to generate the labels for a subset of all possible designs. The labels come in the form of 5 target values: PERF, DSP, BRAM, LUT, and FF. In addition, according to whether the PERF is greater than a threshold value, we classify a design into two categories: valid and invalid. For the valid designs, we perform the regression task of predicting each one of the 5 target values. Table 4 shows the statistics of the prediction targets, i.e., output of a machine learning model.
Table 4: Dataset statistics for the output. The meanings of columns are #D for r: # designs for the regression task, PERF: range of the PERF target, DSP: range of the DSP target, BRAM: range of the BRAM target, LUT: range of the LUT target, FF: range of the FF target, #D for c: # designs for the classification task, T:F: True class (Valid) vs False class (Invalid) design ratio.
Dataset	#D for r	PERF	DSP	BRAM	LUT	FF	#D for c	T:F
SDx (v1)	9439	[-6.5, 1.5]	[0.0, 8.4]	[0.0, 3.0]	[0.0, 6.5]	[0.0, 3.2]	28017	9439:18578
ViTIS (v2)	5027	[-3.6, 6.6]	[0.0, 6.6]	[0.0, 0.7]	[0.0, 5.6]	[0.0, 1.9]	14273	5027:9246
Figure 3: Visualization of the pragma-augmented PROGRAML graphs of four selected kernels. Node colors indicate the block attribute derived from the assembly code. Edge colors indicate the edge flow.
Since our task aims to predict the validity (classification) and quality (5-target regression) of the designs, we provide both the source code and the graph representation derived from PROGRAML [13]. Specifically, we follow [13] to compile a kernel's source code into assembly code, and then transform the assembly codes into a control data flow graph (CDFG) with call relation, and eventually add pragma nodes to the PROGRAML graph following [30]. The resulting pragma-augmented PROGRAML graphs for four selected kernels are depicted in Figure 3.
It is noteworthy that there are multiple ways to represent the input source programs. As an illustration, we include in this dataset the graph representation used in [30]. Other representations are possible, as such abstract syntax trees. Since we release the source code, it is possible to derive and generate such other representations.
5 Experiments on HLSYN
This section provides several baseline experiments with their results to investigate the performance of various methods on the task of design quality prediction.
All the baseline methods share the same encoder-decoder architecture and differ only in the encoder. Specifically, each method encodes a design into a  $D$ -dimensional vector where  $D = 512$  by default and uses a MLP-based decoder to transform the design embedding into the targets.
CODE2VEC [2] CODE2VEC is a path-based attention model. It first decomposes the code to a collection of paths in its abstract syntax tree and represent them as a bag of distributed vector representations. Then it uses an attention mechanism to compute a learned weighted average of the path vectors as the overall representation of the code.
CODET5-RAND, CODET5-FROZEN, and CODET5 [39] These three methods are based on the CODE2VEC method which performs pre-training on a large amount of source code. We utilize the small version released by the authors to encode the design. CODET5 fine-tunes both the encoder and the decoder, CODET5-FROZEN freezes the encoder and only fine-tunes the decoder, while CODET5-RAND fine-tunes both but replaces the encoder parameters/weights with a random initialization to study the effect of pre-training on our tasks.
In order to handle the long source code as a sequence of code tokens, we set the maximum sub-sequence length (i.e., the maximum number of tokens allowed in a sub-sequence) to be 64, and apply a sliding window of size 64 over the source code to obtain multiple sub-sequences as input to the transformer-based encoder. At the beginning of each sub-sequence, a special starting token is inserted and its embedding is taken as the sub-sequence level embedding, and all the sub-sequence embeddings are aggregated into the final  $D$ -dimensional embedding for the design.
G-CODEBERT [18] and G-CODEBERT-L Similar to CODET5, G-CODEBERT is another pre-trained source code encoder, yet with a larger embedding dimension (768 instead of 512), and is thus
Table 5: Regression error (RMSE) on SDx (v1) and VITIS (v2).
Method	SDx (v1)	VITIS (v2)
Trans	Ind	Ind Adapt	Trans	Ind	Ind Adapt
CODE2VEC [2]	3.2877	4.2186	3.4156	2.9685	3.9076	2.9957
CODET5-RAND	1.7100	3.2206	2.2295	1.4939	2.9592	1.9573
CODET5-FROZEN	2.6808	2.9447	2.2317	2.1822	2.9452	1.8447
CODET5 [39]	0.5515	2.8301	1.8203	0.5270	3.4191	1.7552
G-CODEBERT [18]	0.5541	2.6639	1.5552	0.6637	3.1158	1.5175
G-CODEBERT-L [18]	0.5651	2.7668	1.4656	0.4910	3.2129	1.5561
GNN-DSE [30]	0.8641	3.1366	1.5180	0.5816	3.6764	1.5546
GNN-DSE-2L	0.8020	2.6587	1.6231	0.7644	3.2825	1.5688
[CODET5,GNN-GSE]	0.4648	2.6620	1.6327	0.4785	3.2893	1.5274
[G-CODEBERT,GNN-GSE]	0.5502	2.8521	1.5525	0.3940	3.6146	1.6017
presumably more expressive. G-CODEBERT-L uses a larger maximum sub-sequence length, 128 instead of 64, which would capture a longer dependency between source code tokens, and may yield better results.
GNN-DSE [30] and GNN-DSE-2L In contrast to the previous methods which only receive the source code as input, these methods receive the assembly-level graph (with examples in Figure 3) as input and use TRANSFORMERCONV [28] with a jumping knowledge network [42] to produce the design embeddings. GNN-DSE employs 8 layers whereas GNN-DSE-2L only utilizes 2.
[CODET5,GNN-GSE] and [G-CODEBERT,GNN-GSE] These two methods concatenate the design embeddings produced by a source code transformer and a graph neural network-based encoder, i.e., the input embedding into the decoder is  $2 \times D$  instead of  $\bar{D}$ .
5.2 Evaluation Protocol