# Enhancement of In-Plane Seismic Full Waveform Inversion with CPU and GPU Parallelization
## 3 Parallel Computational Model
### Gpu  
GPUs are programmable processors among the most powerful computing devices available in terms of performance and price. GPUs are high-density processors known to accelerate graphics applications compared to CPUs due to their multi-core design and high memory bandwidth. Because of the GPU's parallel architecture, it has sustained a steady performance boost. A current-generation high-end GPU exceeds a high-end general-purpose CPU in terms of processing power. On a GPU, a shared memory pool available to all threads in the block performs substantially better than a global memory pool (See Figure 6. The programming model is based on the hierarchy of hardware parallelism. The GPU kernel divides all threads into equal-sized static workgroups of several hundred. Workgroups effectively exchange state and synchronise, allowing for coordinated data processing. A workgroup's threads are all scheduled to run on a single core simultaneously. Serial execution of the code takes place on the CPU. On the GPU, parallelism is represented via a kernel function that runs many threads simultaneously.  
\begin{table}
\begin{tabular}{l l l} \hline \hline  & **OpenMP** & **CUDA** \\ \hline Parallelism & -Data Parallelism & -Data Parallelism \\  & -Asynchronous task parallelism & -Asynchronous task parallelism \\  & -Host and device & -Device only \\ Architecture abstraction & -Memory hierarchy & -Memory hierarchy \\  & -Data and computation binding & -Explicit data mapping and movement \\  & -Explicit data mapping and movement & -Explicit data mapping and movement \\ Synchronisation & -Barrier & -Barrier \\  & -Reduction & -Joint \\ Framework Implementation & -Compiler directives for C/C++ and & \\  & Fortran & C/C++ extensions \\ \hline \hline \end{tabular}
\end{table}
Table 1: Major features of OpenMP and CUDA [37].  
Figure 5: A flowchart of parallel computation using OpenMP.  
Figure 6: CPU and GPU Architecture.  
mance, making it a popular parallelism option. CUDA is a C-based language designed to use GPUs' massive parallelism. Nvidia's CUDA framework comprises an expanded C/C++ programming language and optimised libraries (cuBLAS, cuFFT) for common mathematical tasks. CUDA provides a basic abstract programming model that may be used on several GPU generations in the future.  
The steps to parallelise the code using CUDA on the GPU have been elaborated in Figure 7. The prepossessing is conducted on the CPU using Python programming language and is saved as binary files. The files are read in C++ host code, from which the necessary variables are transferred to the GPU device for the parallel computation of forward and FWI simulations. The GPU is used for the rest of the calculation. The forward simulations are run over the initial input parameters. Then the output from the forward simulations is compared with the field measurement records to calculate the data misfit. If the data misfit is less than the minimum, the parameters (Lami's constants: \(\lambda,\mu\) and Density: \(\rho\)) are estimated to the desired accuracy. Suppose the data misfit is more than the minimum; the simulation is forwarded to an adjoint simulation where the gradients of the parameters are calculated. The computed gradients are then smoothed and optimised, and the best fitting step length is calculated using a parabolic line search algorithm. With the gradients and the step length the medium parameters are updated as \(m_{i+1}=l\;\partial m+m_{i}\), where \(i\) is the iteration step, \(m_{i}\) is the medium parameters at \(i^{th}\) iteration step, \(\partial m\) the gradient of the parameter and \(l\) the computed step length. The forward simulation is run in the next iteration with the updated medium parameters. The loop runs until the misfit is obtained within the desired limits. After the desired misfits are obtained, the latest obtained medium parameters are copied to the host CPU as binary files, where the post-processing of the results continues in Python.  
Figure 7: A flow chart of parallel computation of seismic FWI in GPU.