Megacell Reservoir Simulation
Authors: Dogru Ali H. 
Published: June 2014

Distinguished Author Series articles are general, descriptive representations that summarize the state of the art in an area of technology by describing recent developments for readers who are not specialists in the topics discussed. Written by individuals recognized to be experts in the area, these articles provide key references to more definitive work and present specific details only to illustrate the technology. Purpose: to inform the general readership of recent advances in various areas of petroleum engineering.

Summary
Reservoir simulation has moved into a new era with the rapid advancements in parallel-computer hardware and software technologies. In the past decade, the total number of gridblocks used in a typical reservoir study increased from thousands to millions. Mega (million) -cell simulation capability allows accurate representation of reservoirs because of its ability to include detailed information provided by logs, 3D seismic, horizontal wells, and stratigraphic studies. One of the main advantages of megacell simulation is that it minimizes or eliminates upscaling effects, making more-accurate reservoir-performance predictions possible. Furthermore, the speed of parallel computers significantly reduces the manpower-intensive history-matching process. Megacell simulators typically are developed on parallel computers. With the current low-cost parallel computers, a million-cell simulation run covering 20 years of production history takes less than 1 hour. State-of-the-art computer graphics provide an effective means of visualizing large data sets covering million-cell descriptions of a reservoir and its performance. This paper presents an overview of megacell simulation technologies in the oil industry and academia.

Introduction
The development of parallel computers allowed faster reservoir simulation, placing heavy demand on number crunching and data storage. Parallel computers are classified in two categories: shared- and distributed-memory computers.
Shared-memory computers, such as Cray vector supercomputers, are composed of multiple central processing units (CPU's) sharing a common memory. Each CPU of this type of machine is usually very fast and uses vector processing. In contrast, in distributed-memory machines, each CPU has its own memory and communicates with the other nodes (CPU's) through a high-speed network. The number of CPU's in shared-memory machines cannot be increased beyond a specific number (usually 4, 8, or 16), but distributed-memory machines have no such limitations. Distributed-memory machines [sometimes called massively parallel processors (MPP's)] can be made up of hundreds or thousands of CPU's. Each CPU can be very low cost (the cost of a PC); however, the network communication and programming language become major challenges in this type of architecture. Both major and small hardware companies manufacture massively parallel computers. The collection (cluster) of workstations and, more recently, the collection of PC's with a high-speed switch box provide both significant cost savings and attractive performance. Ref. 1 gives a comprehensive description of parallel computers and the associated programming languages, and Abate et al.* discuss the latest efforts and provide an example of a reservoir simulator running on cost-effective PC clusters.

Early Oil Industry Efforts
As the developments described were taking place in the computer industry, researchers in the oil industry began investigating the benefits of the parallelization in reservoir simulation. The research effort on parallel reservoir simulation during the 1980's is widely discussed in the literature.

Shared Memory.
Refs. 2 through 4 present early work on parallelization of reservoir simulators for shared-memory/vector computers. The results presented in these works demonstrate the potential benefits of parallel processing.

Distributed Memory.
Wheeler5 developed a black-oil simulator on a hypercube, Killough and Bhogeswara6 presented a compositional simulator on an Intel iPSC/860, and Rutledge et al.7 constructed a black-oil simulator for the CM-2 and showed that reservoir models with more than 2 million gridblocks could be run on this type of machine with 65,536 processors.
The mid-1990's brought further distributed-memory publications. Kaarstad et al.8 presented a 2D oil/water simulator for a 16,384-processor MasPar MP-2. He showed that a model problem with 1 million gridblocks could be solved in a few minutes. Rame and Delshad9 parallelized a chemical-flooding code and tested scalability on a variety of systems.
Distributed-memory simulators began to move from research to production in 1997 with the presentation of real field applications with massively parallel simulators. Shiralkar et al.10 presented FALCON, a distributed-memory simulator that uses several programming languages to handle data distribution and interprocessor communication. These languages include high-performance FORTRAN and FORTRAN 90 coupled with message passing [parallel virtual machine or message-passing interface (MPI)].1 The simulator was tested on various computing platforms, including TMC CM-5, IBM SP-2, SGI Power Challenge, Cray T3D and T3E, and Origin 2000.
Chien et al.11 presented a distributed-memory simulator based on an existing FORTRAN 77 simulator. They used domain decomposition and MPI message-passing libraries on an IBM SP-2. Their largest reported field was more than 1 million gridblocks running on a 16- or 32-node SP-2 system. Killough et al.12 presented locally refined grids based on domain decomposition for a distributed-memory simulator. The largest field model featured implicit pressure/explicit saturations (black oil and compositional) for more than 1 million gridblocks and up to 32 SP-2 processors.