# Presentation: 864798

## Data Management Challenges in Petascale Simulations of Turbulent Combustion

**Combustion Research Facility**
**Sandia National Laboratories**
- _[jhchen@sandia.gov](mailto:jhchen@sandia.gov)_
** ****Ramanan Sankaran**
**Oak Ridge National Laboratory**
**Supported by **
**Division of Chemical Sciences, Geosciences, and Biosciences, **
**Office of Basic Energy Sciences**
***Computing: ORNL NLCF, NERSC, PNNL, SNL***

**Data Management Challenges in Petascale Simulations of Turbulent Combustion**

## Combustion and energy security

**Combustion and energy security**

- Combustion accounts for 3/4 of energy used in U.S. manufacturing
- Ground transportation accounts for 2/3 of petroleum usage
- Potential for improvement in thermal efficiency (30%45%)
- Low temperature combustion (LTC) concepts for automobiles
- Savings of 3 million barrels of oil per day (out of 20M)
- Design improvements are difficult
- Low hanging fruits have already been picked
- Advanced concepts require combustion operating at the edge
- Sound scientific understanding is necessary

**Notes:**
Combustion currently provides 85% of our nation’s energy needs.  Furthermore, because of the large infrastructure costs combustion will continue to be the predominant source of energy for the near and middle term. 

## Turbulent combustion is a grand challenge!

**Diesel Engine Autoignition, Soot Incandescence**
**Chuck Mueller, Sandia National Laboratories**

**Stiffness : wide range of length and time scales**
**turbulence**
**flame reaction zone**
**Chemical complexity**
**large number of species and reactions (100’s of species, thousands of reactions)**
**Multi-Physics complexity **
**multiphase (liquid spray, gas phase, soot, surface)**
**thermal radiation **
**acoustics ...**
**All these are tightly coupled**

**Notes:**
This soot incandescence image of auto-igniting fuel sprays in a diesel engine demonstrates many of the challenges involved in turbulent combustion simulation.

First, there is a very wide range of length and time scales.  There is a range of scales of the turbulent flow, and in a reacting flow the problem is usually compounded by even smaller reactive scales.

Then, there is a chemical complexity.  If we make a continuum assumption, this is manifested by large numbers of species and chemical reactions.  Typical hydrocarbon fuel kinetics may involve hundreds of species and thousands of chemical reactions.

Then, there is multi-physics complexity.  We can have multi-phase flows, including sprays and soot, and other factors like surface interactions, thermal radiation and acoustics.

All of these phenomena and scales are tightly coupled, and this means that the simulation of turbulent combustion is a daunting prospect, and will remain so for the foreseeable future.

And, I would like to add that many of these considerations also apply to other reacting flows.

## Direct Numerical Simulation (DNS)

- DNS is a tool for fundamental studies of the micro-physics of turbulent reacting flows
- Full access to time resolved fields
- Physical insight into chemistry turbulence interactions
- A tool for the development and validation of reduced model descriptions used in macro-scale simulations of engineering-level systems

## DNS capability at Sandia

- Solves compressible reacting Navier-Stokes equations.
- High fidelity numerical methods.
- 8th order finite-difference
- 4th order explicit RK integrator
- Hierarchy of molecular transport models
- Detailed chemistry
- Multi-physics (sprays, radiation and soot)
- From SciDAC-TSTC  (Terascale Simulation of Turbulent Combustion)
- Fortran90 and MPI
- Highly scalable and portable

**S3D is a state-of-the-art DNS code developed with **
**13 years of BES sponsorship.**

## S3D parallel performance 

**S3D scales with 90% parallel efficiency on 10000 cores on CrayXT3 (ORNL)**

## Achieving quantitative predictability requires petascale computing

- Petascale computers needed to achieve relevant parameters spaces for turbulent combustion: N ~ Re9/4  turbulence plus flame scales (3-4 decades of scales)
- Relevant parameter regimes of real devices and laboratory-scale flames Re>15,000.
- Terascale computing: Re~O(10,000), fully-developed turbulence
- Turbulence-chemistry interactions requires transporting 20-80 species plus turbulence

## DNS of canonical laboratory configurations

- Lean premixed flames
- Extinction and reignition in nonpremixed flames
- Flame stabilization in autoigniting flames

## DNS of lean premixed methane/air combustion

**DNS of lean premixed methane/air combustion**

- Goals
- Better understanding of lean premixed combustion in natural-gas based stationary gas turbines
- Model validation and development
- Simulation details
- Detailed CH4/Air chemistry (18 D.O.F.)
- Slot burner configuration with mean shear
- Spatially developing and statistically stationary simulation.
- Better suited for model development
- Parametric study: 3 simulations at increasing turbulence intensities
- to understand the effect of turbulence on flame structure and burning speed.

***Sankaran et al. 2006***

**Notes:**
Better images from more recent simulations can go here. When they are ready. May be an animation too.

## Parametric study with increasing Reynolds number

**Parametric study with increasing Reynolds number**

| u’/SL | 3 | 6 | 10 |
| --- | --- | --- | --- |
| Mesh | 52M | 88M | 200M |
| Performed on | X1E 
512 MSPs | XT3
4800 CPUs | XT3
7200 cores |
| CPU-hours | 200k | 500K  | 1.2M |

**Small-scale intense turbulence leads to **
**flame broadening in the preheat zone **
**and increased flame shortening**

**Notes:**
Better images from more recent simulations can go here. When they are ready. May be an animation too.

## Effect on flame structure

- Progress variable defined based on O2 mass fraction
- Instantaneous slices shown on the left for Case 1
**-**	**Progress variable from 0 (blue) to 1 (red) in color **
**-****	****Heat release rate as line contours**
- Considerable influence on preheat zone
- Reaction zone relatively intact

## Turbulent nonpremixed combustion

** ****Fuel and air segregated**
** ****Mixing limited**
** ****Extinction**
** ****Reignition**
** ****Flame stabilization**

## Extinction and reignition in a CO/H2 jet flame

**Extinction and reignition in a CO/H2 jet flame**

**Understanding extinction/reignition in non-premixed combustion is key to flame stability and emission control in aircraft and power producing gas-turbines**
***Discovered dominant reignition mode is due to engulfment of product gases, not flame propagation***

**Scalar dissipation rate **

**Burning**

**Extinguished**

- The ***largest ever simulations of combustion*** have been performed to advance this goal:
-   500 million grid points
-   11 species and 21 reactions
-   16 DOF per grid point
-   512 Cray X1E processors
-   30 TB raw data
-   2.5M hours on IBM SP NERSC (INCITE);
-     400K hours on Cray X1E (ORNL)

***Hawkes, Sankaran, Sutherland, Chen – 2006, DOE INCITE 2005, early user LCF /ORNL 2005***

**Notes:**
Direct numerical simulations of turbulent combustion are used to investigate key turbulence-chemistry interactions.  Recently, through a DOE INCITE award (2.5 million cpu hrs at NERSC) and a Leadership Computing Facility award of 3.6million cpu-hours, we have performed the world’s largest DNS with detailed chemistry to understand extinction and reignition in turbulent jet flames.   These simulations involve 1⁄2 billion grid points and transporting 11 species and 21 reactions (Dryer’s CO/H2 mechanism) and represent jet Reynolds numbers (9500) comparable to laboratory experiments (e.g. Su and Clemens, Muniz and Mungal, Han and Mungal).   Having access to space and time resolved reactive scalars, temperature, and the turbulence fields we were able to determine the primary mechanism for reignition of parcels of fluid that are extinguished due to excessive scalar dissipation rate.  An outstanding debate in the combustion community is whether reignition results from flame propagation in the plane of the stoichiometric surface (i.e. in the plane of the flame front if the flame was still burning), or whether reignition occurs due to engulfment of hot product and intermdiate species from adjacent burning flame surfaces brought together with the quenched regions by compressive strain through the turbulence cascade.  Statistics from a joint pdf of the scalar dissipation rate evaluated at edges of the flame holes (i.e. intersection between the stoichiometric mixture fraction surface and the OH isosurface equal to 1⁄2 of its value at the steady state extinction value) and the propagation speed at the edge of the holes reveal that there exist a high probability for positive flame speeds at moderately high scalar dissipation rates.  These speeds are much higher than the laminar propagation speed, and therefore are attributable to turbulent transport of heat and radicals from neighboring flames.  

## Description of runs- Temporally Evolving Non-premixed Plane Jet Flame

**Description of runs****- Temporally Evolving Non-premixed Plane Jet Flame**

- Jet develops temporally.
- Shear-driven turbulence interacts with the flame.

- Initial condition

- Later time

- Spanwise BC:
- periodic

- Streamwise BC:
- periodic

**Notes:**
Now I would like to quickly describe the runs.  The runs we are doing are in a canonical configuration which we call a temporally evolving nonpremixed plane jet flame.

The reason for choosing this configuration is that it allows a high level of turbulence for a lower computational cost, basically a high Reynolds number.  This flow does retain many characteristics of practical nonpremixed combustion flows.

I hope this cartoon helps to explain what is going on.  On the left is the initial condition and on the right is a later time.  We are showing contours of heat release by the way.

We have a fuel jet in the core which flows to the right, and there are counter-flowing air streams at the top and the bottom.  The boundary conditions are periodic in the streamwise and spanwise directions.  Between the two streams there is a turbulent mixing layer, and that is also where the reaction occurs.  The interaction of the reaction and mixing and turbulence is basically what we are studying. 

## Volume rendering of scalar dissipationYu, Ma, Chen, Chen, Hawkes, StorCloud demo at SC05. 

- Hardware accelerated parallel volume rendering (nearly interactive – read and render in 2 sec).
- 4D time-varying terascale data
- Multi-variate volume rendering
- Intelligent visualization

**Notes:**
I would like to show you an animation to give you a feel for how the run progresses.

One of the major challenges we have faced is that the tools even to simply visualize the data just aren’t adequate.  In order to solve this problem we have teamed up with Computer Scientists at UC Davis and at Sandia.  The animation shows a volume rendering of scalar dissipation rate, which we generated at the Supercomputing conference in Seattle last month.  This was generated on an 8-node cluster at Sandia with hardware accelerated volume rendering software written by Hongfeng Yu and Kwan-Liu Ma at UC Davis.   Of course we can’t render this at all in serial, and existing parallel rendering tools such as Paraview or Ensight would have required hundreds of nodes in order to render these data at interactive rates.

So, we start from a laminar flame solution, and we see that the scalar dissipation field is simply 2 parallel sheets.  Then, the intrinsic shear instabilities are triggered by a fairly small initial forcing.  At early times the structures are fairly organized, and clearly aligned with principal strain directions.  Later, the jet breaks down and a more turbulent, isotropic structure exists.  You can really see that we are getting a good amount of turbulent structure in these calculations.  Qualitatively, these observations are consistent with what people obverse in experimental jet flames.

## Reynolds number effects on 

- Higher Re:
- more fine-scale intermittent structure
- higher fluctuations of 

**Case L**

**Case H**

**Case M**

**Notes:**
So, what is the effect of Reynolds number on chi?

Here we have a visualisation of some slices through the field data.  It is pretty clear that as we increase Re we are getting a lot more fine-scale structure.  Basically, the intermittency is increasing.  This is consistent with what we know from nonreacting flow experiments and simulations.

Quantitatively, it translates to higher fluctuations in chi as we see in this plot against the transverse distance.

## Reynolds number effects on extinction

**20t****j****     Simulation Time      40t****j**

**Re=2500**

**Re=4500**

**Re=9000**

**Notes:**
So, what happens as we increase the Reynolds number?

This first case is what I showed you on the last slide.  But the simulation time is going down.

As you can see, as we increase Re we get a lot more extinction!

Consequently, the mean temperatures go down a lot and the reignition also takes a lot longer.  In fact in the highest Re case we’ve still got signiifcant extinction at 40 jet times.  It takes until 50 jet times to reignite this case.

We attribute this to the increased intermittency in scalar dissipation that we observed a few slides back.

## How is extinction correlated with local mixing rates?

**Scalar Dissipation (mixing rate)**

**Extinguished Regions**

**Notes:**

It’s interesting to see whether this correlates with scalar dissipation.

On the left here I’m showing the scalar dissipation and on the right the extinguished regions.

You can see that at early times, there is definitely a correlation.

At later times, it’s too complex to see in this image, at least for my tiny brain.

## Quantification of Extinction- Extinguished Flame Area

- Reaction rate related to the conditional fine-grained surface-density of the stoichiometric surface
- Isosurface extraction from volume data through triangulation
- Data analysis on iso-surface and local normal vector
- Identify flame holes
- Flame edge analysis
- Edge propagation speed
- Study reignition mechanisms
-   Analysis code has to be parallel, suitable for large data and reside within S3D

**S****e**

## Joint PDF of edge speed and mixing rate

- Color scale: Joint PDF, Black line: conditional mean speed.
- First, mainly negative speeds, strong negative correlation with .
- Then, broader PDF, *with 2 branches*
- negatively correlating branch at very high 
- positively correlating branch at low-intermediate 
*Peak positive edge speed occurs at quite high ****!!!*

**Simulation Time**

**Notes:**
Now, I’d like to show a higher level of information – let’s look at the correlations with the scalar dissipation.

I’m plotting the joint pdf of scalar dissipation and edge speed on the flame edge on the color scale and the black line shows the conditional mean.

We start out here at early times with a strong negative correlation and predominately negative speed.

Then, later on, the PDF becomes more broad, and it has 2 branches – a negatively correlating upper branch at very high chi and a positively correlating lower branch at lower chi.

But what’s really interesting about the later times, and the last one in particular is that the maximum positive edge speed is not occurring at the minimum chi but actually occurs a quite a high value of chi!

## Physical interpretation

- 2 basic ideas for reignition:
- Propagating edges along the stoichiometric contour.

**Burning**

**Extinguished**

**Z=Z****st**

**Notes:**
We have a tentative interpretation of this phenomenon.

First, I should describe two competing theories for reignition.

One idea is that the holes are healed by flame edges which propagate along the stoichiometric isosurface, and that these have a laminar-edge flame type structure.

Another idea is that reignition mainly occurs when burning elements of flame are folded onto non-burning elements by the turbulent flow.

## Tentative Interpretation

- 2 basic ideas for reignition:
- Propagating edges along the stoichiometric contour.

**Burning**

**Extinguished**

**Z=Z****st**

**O(S****L****)**

**Notes:**
We have a tentative interpretation of this phenomenon.

First, I should describe two competing theories for reignition.

One idea is that the holes are healed by flame edges which propagate along the stoichiometric isosurface, and that these have a laminar-edge flame type structure.

Another idea is that reignition mainly occurs when burning elements of flame are folded onto non-burning elements by the turbulent flow.

## Tentative Interpretation

- 2 basic ideas for reignition:
- Turbulence folds burning flames onto non-burning areas.

**Burning**

**Extinguished**

**Z=Z****st**

**Notes:**
We have a tentative interpretation of this phenomenon.

First, I should describe two competing theories for reignition.

One idea is that the holes are healed by flame edges which propagate along the stoichiometric isosurface, and that these have a laminar-edge flame type structure.

Another idea is that reignition mainly occurs when burning elements of flame are folded onto non-burning elements by the turbulent flow.

## Tentative Interpretation

- 2 basic ideas for reignition:
- Turbulence folds burning flames onto non-burning areas.

**Burning**

**Extinguished**

**Z=Z****st**

**Notes:**
We have a tentative interpretation of this phenomenon.

First, I should describe two competing theories for reignition.

One idea is that the holes are healed by flame edges which propagate along the stoichiometric isosurface, and that these have a laminar-edge flame type structure.

Another idea is that reignition mainly occurs when burning elements of flame are folded onto non-burning elements by the turbulent flow.

## Tentative Interpretation

**Tentative Interpretation**

- u’>>sL indicates laminar edge flame propagation unimportant.
- Expect reignition by turbulent flame-folding.
- To bring burning and non-burning surfaces together, compressive strain is required, leading to high dissipation.
- Consistent with our result – but more work needed to confirm.

**Notes:**
In our parametric range, and that which we believe to exist in most high Re applications where extinction occurs, the turbulence velocity fluctuation is much bigger than the laminar edge flame speed, which indicates that the flame-folding must be dominant.

And, to bring those burning and non-burning surfaces together, we are going to need a compressive strain, and that is also going to lead to at least a moderately high dissipation.   But, if the dissipation is too high, both the flames will extinguish.

This is consistent with our result - the peak in the conditional mean edge speed as a function of scalar dissipation.

## Understanding the effect of ignition on lift-off stabilization of n-heptane diesel jet requires petascale computing

- How is combustion stabilized at the lift-off   length in a n-heptane diesel jet?
- Flame propagation
- Autoignition (1st stage cool flame)
- Is lift-off stabilization supported by
- premixed flame propagation into a cool flame mixture, or by second-stage autoignition?
- Is lift-off scaling parameterized by
- ignition or by flame propagation?
- How does turbulent mixing affect the transition from first-stage to second-stage, high temperature ignition?
- How are soot precursors affected by ignition quality?

- High-speed Chemiluminescence
- Imaging in a combustion vessel
- near the lift-off length, Lyle Pickett,SNL

**Notes:**
Broadband chemi. Imaging where quasi steady liftoff height is 30 mm.  See formation of detached ignition kernels from the high temperature combustion as ignition occcurs starting at 1 ms.  However, the details of the reaction zone and dynamics of ignition kernels cannot be determined from the image obtained from aline-of-sight chemilim. Measurement.

## Stabilization of Lifted Autoigniting Flames

- Stabilization mechanisms in lifted, vitiated flames
- Hydrogen lifted flame
- High flow velocity to lift off flame
- Auto-ignition may occur below the base of the lifted flame due to high co-flow temperature
- Approximately 1.2 million CPU hours per a simulation on Jaguar in NCCS
- 9 species with 21 elementary reactions
- 24 x 24 x 5.4mm3 domain size with 1600 x 1600 x 270 grid resolution ~0.7 billion grids
- 6~8 flow-though times w/ Umean = 347 m/s
- Total 3~4 simulations

**Highest heat release rate (green), *****Y*****HO2****(red), **
**and vorticity (aqua) of hydrogen lifted **
**jet flame at *****t***** = 0.3ms**

**Liftoff**
**height**

## Animation of hydroxyl radical in lifted hydrogen/air lifted flame

## n-Heptane Lifted Autoigniting Flame

- CPU hour estimation of n-heptane lifted flame
- Skeletal n-heptane mechanism
- 88 species; 384 reaction steps
- 40 x 30 x 30 mm3 domain size
-  2000 x 1200 x 1200 grid resolution (3 billion grids)
- 12 flow-though time (2.4ms) calculation 200-300 m/s mean flow velocity
- CPU hours
- (2000 x 1200 x 1200 grid points) x (240,000 time steps) x (92 equations) x (3.0 x 10-9 hour/NgridNtimeNeq)  240 million CPU hours (on Jaguar in NCCS)
- Requires petascale computer!

**Propane lifted flames in hot coflow (from Kim *****et. al*****, *****Proc. Combust. Inst.***** 31, 2007 to appear)**

## Community Data Sets

- Precedents for comparison of measured and modeled results
- TNF workshop
-     _http://www.ca.sandia.gov/TNF/abstract.html_
- Premixed flame workshops
-      _http://eetd.lbl.gov/aet/combustion/workshop/workshop.html_
- Addition of high-fidelity numerical benchmarks for model validation and development

## Data Sharing

**What is the best model for sharing data, especially large data?**
- Reduced data on the web?
- RANS, LES means, conditional means, slices, chunks
- What is the basic set of data to make available?
- Visiting research projects?
- Giving out whole data?
- What, if any, tools/procedures are needed to enable this?
- Visualization
- Data movement
- Generic data-processing

## Challenges of Petascale Computation: Mountains of Data

**Data storage:**
- Long term: where do we put 500TB?
- Short term: scratch ~ 1TB, but need ~ 10TB!
**Data movement:**
- Archive to scratch (~ 1 week to move 10TB)
- HPC facility to local analysis cluster (longer)
**Data processing:**
- Everything must be parallel, scalable.
- IO speed, memory are the bottlenecks.
**Visualization:**
- Parallel, Multi-variate
- Multi-scale phenomena

**HPSS storage facility at NERSC**

**Interpretation:**
- Physics are more complex
- Wider range of scales, manual sifting is impossible.
- Multi-scale analysis methods
- Feature detection, growing, and tracking

## S3D I/O Requirements

- I/O sizes of current terascale
- S3D combustion simulations

|  | Grid points | Platform | Size per dump |
| --- | --- | --- | --- |
| Jet, Re=3,000 | 150M | XT3 (NCCS) | 19GB |
| Jet, Re=5,000 | 350M | SP (NERSC) | 45GB |
| Jet, Re=10,000 | 500M | X1E (NCCS) | 64GB |
| Bunsen, u’/Sl=3 | 52M | X1E (NCCS) | 8GB |
| Bunsen, u’/Sl=6 | 88M | XT3 (NCCS) | 13GB |
| Bunsen, u’/Sl=10 | 200M | XT3(NCCS) | 29GB |

- Jet simulation (Re=10,000) on X1E (20TF).
- At the rate of one data dump every hour, I/O rate is 64GB/hour
- On a Petaflop system, required I/O rate is 3.2TB/hour
- To achieve 5% maximum overhead, I/O has to occur at 64TB/hour or 17GB/s
- It will be useful to dump data more often than once an hour if higher I/O rates are available.

## Data Movement and Workflow

- 40TB of data generated last year (excluding prep runs and analysis output)
- Geographically distributed HPC resources/archives
- ORNL, NERSC, PNNL, Sandia (CA)
- Need automatic handling of data
- Kepler workflow for S3D,  Ramanan Sankaran and Scott Klasky (ORNL) and Norbert Podhorszki (UC Davis) (SDM Center)

## Parallel feature detection and tracking  - gleaning insight from large data

- Large and complex multiscale data
- Interesting regions sparse
- Temporal evolution of features

##  Interactive Scalable Feature Detection and Tracking

***DNS of Autoignition of H2/Air in Inhomogeneous ***
***Mixtures, Echekki and Chen, 2001***

**Feature definition: threshold on HO2 concentration**
**Feature detection and tracking: **
**Born**
**Merge**
**Die**
**Grow**
**Split**
**Feature statistics**
**Feature-based analysis**

***W. Koegler 1996 IEEE Vis.***

## Feature Statistics

***Feature Graph***

***time***

***Koegler 1996 IEEE Vis.***

## Future of automated data discovery

- Novel approach for data discovery:
- Hierarchical features allow humans to frame hypotheses at different levels of detail.
- Quantitative feature definitions – beyond static thresholds
- Complex isosurfaces
- Topological methods for scalar and vector definitions
- Online distributed feature detection and tracking; computational steering of I/O and analysis
- FastBit technology for efficient search and query
- Frame hypotheses as parametric relationships between feature properties.
- New rendering tools allow humans to visually and quantitatively review multiple features *in context* and provide training data to machine learning.

**Notes:**
We were criticized for being unclear as to the novelty and risk of our proposal.
There are several novel aspects to this research, and they're based on a few simple observations of the challenges in dealing with simulations that are so large in size. The first observation is that large domains are required in combustion because features occur at many length and time scales. Features are not arbitrarily chosen regions and so they have some built-in statistical bias compared to the entire domain... that's how they are chosen. Why they are chosen is another matter: we hope that examining them will help us prove or disprove some hypothesis. We can't examine every node or cell and so we prune pieces and hope to see patterns in the remainder.
 For large simulations, we contend that there's a need to “move up a level” and define something above features. First, we're proposing hierarchical features. One way to scale an algorithm is to use a tree structure, because those allow us to traverse data more quickly – with the log of the number of children at each tree node, as a matter of fact. The second novel thing we propose is multiple feature definitions because we are scaling up the number of variables compared to other types of simulations and some variables define flow features, others flame features. The third way to “move up a level” is to help scientists direct their exploration of the data quantitatively. Previously, you could subset data and examine (visually or quantitatively) results over those subsets and that was the only tool you had to correlate things like flow and combustion. We contend that scientists need a way to express a hypothesis as a parametric relationship and test it. We can't test it on every node of the grid, but if the parameters are feature statistics, then it's tractable. And because people's hypotheses tend to be generalizations rather than precise statements true at every point in a domain, we want hypotheses expressed as fuzzy relationships between feature properties. So we're planning to employ Bayesian methods that allow random variables in the model and provide confidence of the fit. This requires ways to render different values on different features in the same view (context) so users can define features and train Bayesian models correlating quantities across multiple features of varying type.
This third step is the riskiest, but even if it fails, we're left with a lot of machinery to help humans deal with data on larger scales and we may even be able to perform some non-parametric feature identification to help humans extract knowledge from the data.

## Scalable Interactive Combustion Analysis Toolkit

**Scalable Interactive Combustion Analysis Toolkit**

**Raw DNS Data**

**Feature Definition**

**Demand-driven I/O preprocessor**

**Feature detection**

**Feature tracking**

**Feature-based analysis**

**Flame surface statistics**

**Chemical Analysis**

**Conditional statistics**

**Spectra – Fourier, wavelets**

**Averaging and filtering**

**Reduced order representation- POD, topology**

***Analysis Libraries***

***Feature Detection/Tracking ***
***Demand-driven I/O Libraries ***

## Acknowledgments

- Chunsang Yoo (SNL)
- Ramanan Sankaran (ORNL)
- Scott Klasky (ORNL)
- Evatt Hawkes (SNL)
- Mark Fahey (ORNL)
- David Skinner (LBNL)
- David Lignell (SNL)
- Andrea Gruber (SINTEF, U. Trondheim)
- Tianfeng Lu (Princeton U.)
- Chung Law (Princeton U.)
- Kwan-Liu Ma (U. C. Davis)
- Hiroshi Akiba (U. C. Davis)
- Hongfeng Yu (U. C. Davis)
- Scott Klasky (ORNL)
- Wendy Doyle (SNL)
- John Wu (LBNL)
- Arnaud Trouve (U. Maryland)
- Hong Im (U. Michigan)
- Chris Rutland (U. Wisconsin)

## Contacts

- Jacqueline Chen
- Combustion Research Facility
- Sandia National Laboratories
- Livermore, CA  94550
- _[jhchen@sandia.gov](mailto:jhchen@sandia.gov)_
- Ramanan Sankaran
- National Center for Computational Sciences
- Oak Ridge National Laboratory
- sankaranr_[@ornl.gov](mailto:jhchen@sandia.gov)_

-    Presenter_date

## DNS of turbulent flame-wall interaction

- DNS of a premixed flame interacting with turbulence in a wall-bounded channel flow
- Detailed H2/air chemistry (14 D.O.F.)
- Inflow turbulence obtained from a separate simulation of inert channel flow
- Cost: 100K hours on Cray X1E
- Goals
**Material failure from fluctuating thermal stress in micro-gas turbines**
**Study of spatial and temporal patterns of wall heat flux**
***Discovered correlation of near-wall coherent turbulence structures and exothermic radical recombination reactions with wall heat flux***

*Wall heat fluxes in turbulent flame wall interaction at low Reynolds number*

***A. Gruber, R. Sankaran, E. R. Hawkes, and J. H. Chen - 2006***

## Flame stabilization in lifted autoigniting hydrogen flames

- Goal: determine stabilization mechanisms in lifted, autoigniting hydrogen/air flames
- Hydrogen lifted flame
- Important submechanism for n-heptane
- Comparison with experiment (Chung, Cabra)
- Auto-ignition may occur below the base of the lifted flame due to high co-flow temperature
- Approximately 1.5 million CrayXT3 CPU hours
- 9 species; 21 elementary reaction steps (Li et al. 2006)
- 24 x 20 x 3.6mm3 domain size; 1600 x 1000 x 240 grid resolution=~400M grids
- 4 flow-though times; Umean = 347 m/s, Re=7000
- Parametric study with coflow temp, flow velocity, additives
- Total 3~4 simulations (~3.0 million cpu-hrs
- 	on CrayXT3 ORNL 2006-2007)

- Heat release rate and vorticity in a H2/air lifted slot jet flame in heated coflow

**Notes:**
U_jet = 347 m/s; u_coflow = 4 m/s
U’/U_jet = 0.1, u’=34.7 m/s
D =1.6 mm, ~2 mm
L_t= 1.6 mm; L_11 = 0.54mm
Domain = 24x20x3.6mm^3
Grids: 1600x1000x240 grids
T_jet = 300K; T_coflow=1100K 
65%hydrogen/35% nitrogen by volume
Coflow air
Stoichiometric mixture fraction = 0.2
Re = 7000 (d=1.6 mm)
Timestep = 2e-9 s
Simulate for 4 flow through times = 240 microseconds
240 microseconds/2e-9s = 140,000 timesteps
Ignition delay ~ 90 microseconds at 1100K

## Motivation
Soot is a pollutant and lowers combustion efficiencies in diesel engines
Soot radiation is the major heat transfer mode
Goals
Quantify soot formation and transport mechanisms in turbulent flames  
Approach
DNS of turbulent sooting flames with detailed chemistry, transport, radiation
2–3 moment soot particle model with semi-empirical soot chemistry
3M cpu-hours on CrayXT3 at ORNL to observe slow soot processes

- Motivation
**Soot is a pollutant and lowers combustion efficiencies in diesel engines**
**Soot radiation is the major heat transfer mode**
- Goals
**Quantify soot formation and transport mechanisms in turbulent flames  **
- Approach
**DNS of turbulent sooting flames with detailed chemistry, transport, radiation**
**2****–3**** moment soot particle model with semi-empirical soot chemistry**
**3M cpu-hours on CrayXT3 at ORNL to observe slow soot processes**

***D. O. Lignell, P. J. Smith, and J. H. Chen***

- DNS of non-premixed sooting ethylene flames

**Notes:**
Picture shows DNS as a small region of a larger, more physically relevant flow.  Smoke breakthrough/shielding near the top.
Ethylene flame with extinction.
Colored by temperature with soot contours
Plot is mean soot mass fraction versus mixture fraction and time for the blue Kelvin Helmolz plot.  
Soot breakthrough is important in fires since smoke can shield the environment from radiation.  
How soot gets from fuel rich regions to a queched state outside the fire is a mystery.

## DNS of stabilization in lifted auto-igniting flames (planned)

- Goal: Determine stabilization mechanisms in lifted, auto-igniting flames relevant to efficient, clean burning low-temperature compression ignition engines and flashback in aircraft gas turbine engines
- Hydrogen lifted flame
**300K hydrogen turbulent jet; 1100K air coflow jet**
**Important submechanism for n-heptane**
**Comparison with experiment (Chung, Cabra)**
**Auto-ignition may occur below the base of the lifted flame due to high co-flow temperature **
**Approximately 1.2 million CPU hours per simulation on Jaguar at NCCS**
**-****	****9 species; 21 elementary reaction steps (Li et al. 2006)**
**-****	****24x20x3.6 mm****3**** domain size; 1600x1000 x240 grid resolution= ~400M grids**
**-****	****4–6 flow-though times; U****mean**** = 350 m/s**
**-****	****Total 3~4 simulations (~3.0 million cpu-hours on CrayXT4 at ORNL 2006–2007)**

**Heat release (gold) and vorticity in a lifted autoigniting H2/air jet flame**

## Transport Effects on Mixing Timescale

- Mixing time scale is key element of combustion models (transported PDF method)
- Most models assume mixing time scale same as mechanical time scale for all species
- Detailed transport and chemistry effects can alter the observed mixing timescales

## Acknowledgments

- Chunsang Yoo (SNL)
- Ramanan Sankaran (ORNL)
- Evatt Hawkes (SNL)
- Mark Fahey (ORNL)
- David Lignell (SNL)
- NLCF Computational Combustion End Station (3.6M cpu-hours, 2006)
- NERSC Incite Award 2005 (2.5M cpu-hours)