# High Performance Computing in Satellite SAR Interferometry: A Critical Perspective
## 4 High Performance Computing: Fundamentals Concepts and Models
### Parallel Computing Architectures  
Parallel HPC platforms can be primarily categorized according to their memory access; therefore, a first distinction concerns _shared_- and _distributed_-memory systems. Distributed memory topology generally requires that each processor have a separate address space (private memory area). Conversely, in shared-memory architectures, multiple processors can operate independently but share the same memory resources. As a matter of fact, the largest and fastest computers in the world today employ both shared and distributed memory architectures [157]. A multiprocessor system includes two or more processors (CPUs), and a _multicore_ system is essentially a low-end multiprocessor. Nowadays, multicore machines have become nearly ubiquitous, as multiple cores in the CPU permit overcoming the limited performance achievable by boosting the clock speed of a single-core architecture. Multiprocessors are indeed _shared-memory_ systems since they share the same physical RAM. Furthermore, a _threaded_ program will have several instantiations, called _threads_, which work jointly to achieve parallelism. They share the program's data in common, and they run independently, with each thread on a different core (Figure 6). In the computer systems landscape, new opportunities for accelerating scientific algorithms have been offered by the rise of the _graphics processing unit_ (GPU). A GPU uses thousands of stream processors to increase the chip-level parallelism, thus potentially generating huge speed-ups. It should be stressed that GPUs are somewhat different from CPUs in terms of frequencies, latencies, and hardware capabilities. In recent years, general-purpose computation on GPUs (GPGPU) has been proposed to leverage the powerful GPU capability in speeding up general computation, thus rapidly creating new processing challenges and new opportunities for accelerating scientific processing. As a general guideline for using GPU architectures, the user should minimize the use of global memory, thus preferring shared memory access, wherever possible, to avoid redundant transfers from global memory. Nowadays, CPUs and GPUs are widely available on several hardware platforms, ranging from laptops to multiple-GPU servers. Clusters of commodity and purpose-built processors have dominated the previous decade [164; 165]. In a cluster, multiple nodes are usually connected through a fast local network, and each computing node is a multi-processor parallel computer in itself. More recently, these clusters have been augmented with computational accelerators (coprocessors and GPUs). The modern architecture of the HPC clusters, based on innovative software technologies and supported by high-speed, low-latency interconnections (such as Infiniband), ensures the best scalability and overall performance for scientific applica  
Figure 6: Scheme for multiple threads on a shared memory machine. tions (Figure 7). Accordingly, cluster computing relies on a set of individual computers connected through specialized hardware and software, and hybrid parallel systems might be equipped with multi-core CPUs and many-core GPUs connected through network communications (Figure 7) [154; 155; 156]. As a matter of fact, constant growth in the performance of high-performance computing systems has characterized the recent history of parallel computing. In particular, power consumption and performance for the top clusters according to the TOP500 are available in [157]. Efficient parallelization of applications is aimed at minimizing the application execution time. According, potential parallelism of the algorithmic schemes have to be exposed to obtain parallel implementations of the corresponding algorithms. This generally requires equally engaging all the available processing units for computations (_load balancing_) and minimizing communication and synchronization overheads resulting from parallelization. Accordingly, developing scalable parallel code for larger-scale systems is not a trivial task.