# An Evaluation of Directive-Based Parallelization on the GPU Using a Parboil Benchmark
## 7 Discussion  
After an overall evaluation of the OpenACC programming model, we can conclude that it has a lot of merits. Low-level programming models like CUDA and OpenCL, which give access to a lot of hardware-specific mechanisms, do indeed achieve the best performance in most cases. The difference between implementations using these programming models are insignificant except in the cases where applications rely heavily on optimizations. However, these programming models require a lot of knowledge about the underlying hardware and its features. In addition, the implementation itself can be very long and challenging, as in most cases it requires significant code restructuring. On the other hand, OpenACC hides all hardware specifics behind its directives and allows users to parallelize their code with minimal effort. However, there are a few things that need to be considered before using a programming model like OpenACC. Firstly, like CUDA and OpenCL, OpenACC does not contribute much to applications with an irregular control flow (a lot of conditional jumps). As can be seen from the BFS application, the solution in these cases is to use a proficient compiler which better utilizes the underlying CPU features. Furthermore, these programming models also have a small impact on applications with irregular memory access patterns [44] or strided memory accesses [21]. A good example in this case is the SpMV application. Although, unlike the problem with irregular control flow, a large amount of data can hide this caveat. Secondly, some of the applications require a certain degree of code modifications. Compute-bound applications, like CUTCP and MRI-Q, only required a simple loop interchanging. In the case of memory-bound applications, the modifications are more complicated. For applications like LBM, the code needed to be modified to achieve a different memory access pattern in order to improve memory coalescing similar to [27]. Applications like SAD and TPACF required a degree of code inlining. The aim of this code inlining was grouping more loops in order to use the collapse directive and achieve a higher level of parallelism. A similar approach is used for sparse matrix multiplication in [10]. Finally, applications like SpMV with an irregular memory access pattern required loop unrolling. Thirdly, some of the applications cannot be implemented without using low-level CUDA features or they benefit from them. For example, BFS cannot be correctly implemented using the atomic directives provided by the OpenACC programming model. To solve this problem, CUDA-style atomics were used as we executed our code on an NVIDIA  
Figure 11: Speedup of TPACF. GPU. Furthermore, applications like HISTO benefit from the CUDA utility functions. Rather than entering a parallel region for memory initialization, we used the cudaMemset function, which is faster. However, this choice of action can have an impact on portability since the availability of atomic operations is dependent on the platform itself. Low-level programming models share similar constructs under different names; thus, portability issues can be easily fixed in most cases. Morever, nowadays lightweight portability layers exist for major architectures, such as AMD HIP, which allows us to use low-level programming functions across different platforms, e.g., NVIDIA and AMD GPUs. Finally, it should be noted that typical parallelization of an application via OpenACC requires significantly less programming time and code restructuring, as also noted in other studies in the field [12]. Also, it allows us to maintain a similar codebase with the sequential application, which is beneficial in the case of further application development. Moreover, as stated in [10], avoiding model-specific and hardware-specific features enables portability between different platforms. Performance-wise, some of the applications, such as CUTCP, SGEMM, MRI-Q, and Stencil, show slightly better performances in the case of OpenACC implementations in this study.