**The Applied Partial Differential Equations Center (APDEC): Software
Development**

Principal Investigators: J. Bell, M. Berger, P. Colella, L. Diachin, M.
Minion, E.G. Puckett, C. Rutland, R. Samtaney. Software Development
Team: K. Chand, D. Graves, N. Keen, T. Ligocki, P. McCorquodale, A.
Petersson, T. Sternberg, B. van Straalen.

*Summary*

***The APDEC ISIC has developed a suite of reusable tools for
implementing block-structured adaptive mesh refinement methods and
embedded boundary methods for complex geometries, including solvers,
grid generation tools, and support for interoperability of the APDEC
tools with software developed by other ISICs and by SciDAC applications
projects.***

**1. Introduction**

The software framework being developed in APDEC is based on the use of
block-structured adaptive mesh refinement (AMR), in which the dependent
variables are discretized on a spatial grid consisting of nested
rectangles of varying spatial resolution, organized into blocks (figure
1).

![](media/image1.wmf)

***Figure 1 . An example of a block-structured refined grid in three*
*dimensions. The individual grid cells are shown in white, and the black
lines indicate the organization into blocks.***

This is combined with an embedded boundary

approach to the treatment of complex boundary geometries in which the
irregular boundary is represented by its intersection with the
rectangular grid (figure 2).

One of the principal characteristics of this class of algorithms is that
they are difficult to implement: they are more complicated than
traditional finite difference methods, and the data structures involved
are not easily represented in the traditional procedural programming
environments used in scientific computing. To manage this algorithmic
and software complexity, we have developed a collection of libraries
written in a mixture of C++ and Fortran that implements a set of
abstractions specific to this algorithmic domain.

![](media/image2.jpeg){width="1.832638888888889in"
height="1.5055555555555555in"}

***Figure 2. Example of an embedded boundary representation of a nozzle
geometry. The cells that covered by the solid body are shown in red. The
blue cells constitute locations where the discretized unknowns are
defined.***

The starting point for the software development part of APDEC is the
Chombo software library, developed by the Applied Numerical Algorithms
Group at LBNL. Chombo is a version of the BoxLib family of libraries
developed by the Center for Computational Sciences and Engineering at
LBNL that has been refactored to support a broader range of algorithmic
needs and to enhance its reusability.

**2. Accomplishments**

In the area of core infrastructure support, we have implemented a
variety of new tools in response to the needs of various applications or
the needs of other software developers.

These included parallel data structures for bin-sorted particles on
unions of rectangles, support for periodic boundary conditions in one or
more directions; a visualization and analysis package, called ChomboVis
(jointly funded by SciDAC and the NASA CT program). We have also
developed a number of interoperability tools for allowing Chombo code to
call and to be called by other libraries developed under SciDAC. These
include interfaces to the Hypre solver package developed by the TOPS
ISIC and to the PAPI performance tools developed by the PERC ISIC, as
will as a CCA-compliant interface to the Chombo AMR elliptic solvers.

![](media/image3.jpeg){width="2.8333333333333335in"
height="1.0166666666666666in"}

**Figure 3. Nodal-point AMR solution of Poisson's equation. The
conducting surfaces of a high-current ion beam accelerator are colored
with the electric field strength.**

We developed a variety of solvers for cell-centered discretizations of
classical PDEs, including iterative solvers for variable coefficient
elliptic equations and a general package for an unsplit higher-order
Godunov method for hyperbolic conservation laws, for which the user need
only provide a small amount of physics-dependent code. We also developed
node-centered Poisson solvers in support of AMR for particle-in-cell
methods, including a treatment of irregular boundaries based on the
Shortley-Weller algorithm (figure 3).

![](media/image4.jpeg){width="2.5840277777777776in"
height="1.2979166666666666in"}

**Figure 4. Calculation of an axisymmetric gas jet expanding into a
vacuum. The calculation was performed using an embedded boundary AMR
method. The covered cells are colored black.**

We developed the extensions of the Chombo layered architecture required
to support embedded boundary methods, including distributed data
containers, AMR interlevel transfer operations, and AMR elliptic and
hyperbolic solvers for problems with complex geometries (figure 4). We
have also developed a variety of grid generation tools, including
interfaces to the Cart3D package from NASA for generating embedded
boundary grid descriptions from a collection of surface triangulations
obtained from a CAD package (figure 1), as well as generating such
descriptions from level-set representations of the surface. This enables
the generation of grids from sources such as digital elevation maps and
image data. (figure 5.)

![](media/image5.jpeg){width="1.4916666666666667in"
height="1.5416666666666667in"}

![](media/image6.jpeg){width="1.6631944444444444in"
height="1.1416666666666666in"}

**Figure 5. Embedded boundary grid generation using level set
representations. Left: grid generated from SEM image of a neutrophil.
Right: grid for flow simulation in San Francisco bay generated from a
digital elevation map.**

**3. Future Plans**

There are three major goals for the software development component of
APDEC over the next two years. The first is to add new solver
capabilities particularly for anisotropic problems that arise, for
example, in magnetic fusion. The second is to continue to improve the
interoperability of APDEC software with tools developed in the other
SciDAC ISICs and elsewhere. The third is to continue to improve
performance by attacking the problem at all levels, from the development
of new low-level data representations to the introduction of new
analysis-based low-communcations parallel algorithms for elliptic PDE.

**For further information on this subject contact:**

Phillip Colella

Lawrence Berkeley National Laboratory

Phone: 510-486-5412

E-mail: <Pcolella@lbl.gov>

APDEC software web site: <http://davis.lbl.gov/APDEC/software.html>