Parallel general-purpose reservoir simulation with coupled reservoir models and multisegment wells [electronic resource]
Keywords: Energy Resources Engineering

The development of a parallel general-purpose simulation framework for coupled reservoir models and multisegment wells is the subject of this dissertation. With this work, the General Purpose Research Simulator (GPRS) based on a flexible Automatic Differentiation (AD) framework, AD-GPRS, is now a powerful and flexible platform for modeling thermal-compositional fluid flow in reservoir models with fully unstructured grids. AD-GPRS has advanced and extensible spatial and temporal discretization schemes, advanced linear solvers, and a generalized multisegment well model. In addition, AD-GPRS supports OpenMP parallelization on multicore platforms and a Nested Factorization (NF) linear solver for systems with multiple GPUs (Graphics Processing Units). AD-GPRS employs generalized MultiPoint Flux Approximations (MPFA) for spatial discretization and a multilevel Adaptive Implicit Method (AIM) for time discretization. A generalized connection list is used to locate the block nonzero entries in the system matrix associated with general MPFA discretization. Our AIM implementation allows for new fluid models and nonlinear formulations. The framework can deal with any combination of TPFA (Two-Point Flux Approximation), MPFA, FIM (Fully Implicit Method), and AIM. For efficient linear solution of coupled reservoir models and advanced wells, AD-GPRS supports linear systems based on the MLBS (MultiLevel Block Sparse) data structure. MLBS was first designed and implemented in the original GPRS and extended further in AD-GPRS. Equipped with the CPR (Constrained Pressure Residual) preconditioner, MLBS is very powerful and highly efficient. MLBS has a hierarchical data structure and can accommodate systems with general MPFA discretization and AIM. For accurate well modeling, a general MultiSegment (MS) well model is implemented in AD-GPRS. In this model, variables and equations are defined for both nodes and connections. The general MS well model allows for the following advanced features: general branching, loops with arbitrary flow directions, multiple exit connections with different constraints, and special nodes (e.g., separators, valves). The linear and nonlinear solvers are extended to address the numerical challenges brought about by the general MS well model. Parallel reservoir simulation has recently drawn a lot of attention, and specializing the algorithms for the target parallel architecture has grown in importance. We describe an architecture-aware approach to parallelization. First, multithreading parallelization of AD-GPRS is described. Parallel Jacobian construction is achieved with a thread-safe extension of the ADETL library. For linear solution, we use a two-stage CPR preconditioning strategy, which combines the parallel multigrid solver XSAMG and the Block Jacobi technique with Block ILU(0) applied locally. We also describe multi-GPU parallelization of Nested Factorization (NF). We build on the Massively Parallel NF (MPNF) framework described by Appleyard et al. (SPE RSS 2011). The most important features of our GPU-based implementation of MPNF include: 1) special ordering of the matrix elements to maximize coalesced access to the GPU global memory, 2) application of `twisted factorization' to increase the number of concurrent threads at no additional cost, and 3) multi-GPU extension of the algorithm by first performing computations in the halo region of each GPU, and then overlapping the peer-to-peer memory transfer between GPUs with the computations of the interior regions.