# High Performance Computing in Satellite SAR Interferometry: A Critical Perspective
## 4 High Performance Computing: Fundamentals Concepts and Models
### Parallel Programming Models for HPC Systems  
This section provides a brief overview of parallel programming models and tools for phrasing structured algorithms in concrete HPC implementations. In particular, to achieve high speed-ups on a large number of processing units, the classical sequential SAR processing algorithms need to be translated into their parallel counterparts by making explicit the parallelism onto the underlying hardware. Different parallel programming models for parallel HPC systems might be used for such a purpose, and the most popular ones are briefly summarized in the following. Open multi-processing (OpenMP) is typically used for writing parallel multithreaded programs. Accordingly, OpenMP is a de facto standard oriented to software development for shared-memory parallel machines, in which multiple computational units share access to a common memory [160]. It offers support for dynamic parallelism, reductions, explicit synchronization, and techniques for performance optimization. OpenMP supports Fortran, C, and C++. As far as GPU computing is concerned, we consider the two most prevalent frameworks that provide developers with fine control over code implementation and performance: they are the Compute Unified Device Architecture (CUDA) [161] and the Open Computing Language (OpenCL). First, CUDA is an NVIDIA proprietary parallel computing platform and programming model for writing multithreaded programs running on  
Figure 7: Hybrid systems: multicore CPUs and GPU (graphical processing unit), with node-to-node connection via the network. NVIDIA GPUs, supporting various compiled languages (e.g., C, C++, and FORTRAN). Compared with other alternatives, CUDA is easier to use, and it provides extensive support to developers; however, CUDA can run only on CUDA-enabled NVIDIA GPUs. Second, OpenCL is an open standard that allows parallel programming targeted for heterogeneous systems. It is used for writing parallel multithreaded programs running on multicore CPUs and/or GPUs; it is implemented as a C/C++ language dialect. Message passing is a very broad parallel programming paradigm for implementing parallel algorithms on computational platforms with a distributed memory topology. In particular, Message Passing Interface (MPI) is currently considered the de facto standard for developing HPC applications on distributed memory architectures, even though other parallel-programming models exist [162]. As a matter of fact, a typical application is composed of many processes running in parallel, and MPI implements communication among processors through explicit messages, including point-to-point or collective type operations, thus allowing data passing and synchronization among concurrent processes [162]. MPI supports Fortran and C and offers portability, standardization, performance, and functionalities. In summary, scientific computing nowadays relies on modern HPC platforms embracing heterogeneous systems with GPU-accelerated and CPU-only nodes (see Figure 7), thus supporting diverse parallel programming models such as MPI, OpenMP, CUDA, OpenCL, for C, C++, and Fortran codes. However, it is worth noting that a selected combination of the mentioned parallel programming models (e.g., MPI + CUDA, MPI + OpenMP, and CUDA + OpenMP) may be employed in a hybrid approach to parallelization (as further discussed for specific cases of interest in Section 5). In particular, OpenCL can perform computations on multicore CPUs and GPUs installed within the same workstation. As a result, the most used parallel programming model relies on different levels of parallelism (multi-level), in which MPI is used for internode parallelism, and OpenMP is used for intranode parallelism, augmented with libraries and tools (e.g., CUDA and OpenCL) for coprocessor use. It is reasonable to assume that this mainstream programming paradigm will still be adopted in the long term for HPC systems [163].