# Presentation: 847560

## Scientific Computing Group

- Ricky A. Kendall** **
**Scientific Computing**
**National Center for Computational Sciences**

## Scientific Computing

**Scientific Computing facilitates the delivery of leadership science****by partnering with users to effectively utilize computational science, visualization, and work flow technologies on LCF resources** **to:**

**Port, tune, augment, and develop current and future applications at scale**

**Provide visualizations****to present scientific results and augment discovery processes**

**Automate the scientific computational method**

-    Kendall_SciComp_SC07

## National Center for ComputationalSciences/Leadership Computing Facility

**LCF System Architect**
- S. Poole

**National Center for Computational****Sciences/Leadership Computing Facility**
**A. Bland, **Director
**D. Kothe, **Director of Science
**J. Rogers, **Director of Operations
**L. Gregg, **Division Secretary

**Operations Council**
**M. Dobbs,** Facility Mgmt.**J. Joyce,** Recruiting
**D. Vasil,** Cyber Security
**W. McCrosky,** Finance Officer
**M. Palermo,** HR Mgr.
**R. Toedte,** Safety & Health
**N. Wright,** Org. Mgmt. Specialist

*Site Preparation*
- K. Dempsey
*Hardware Acquisition*
- A. Bland*
*Test  and Acceptance*
- Development
- S. Canon
*Commissioning*
- A. Baker
*Project Management*
- D. Hudson5
*Project R&D*
- A. Geist
*Cray Project Director*
- K. Kafka4

- 1Student
- 2Postgraduate
- 3JICS
- 4Cray Inc.
- 5Matrixed
- 6Subcontract
- *Interim

**Deputy Project Director**
- K. Boudwin

**Chief Technology Officer**
- A. Geist

**Advisory Committee**
- J. Dongarra
- T. Dunning
- K. Droegemeier

- J. Hack
- S. Karin
- D. Reed

**User Assistance**
**and Outreach**
- J. White
- L. Rael
- T. Anderson6
- D. Frederick
- C. Fuson
- B. Gajus6,5
- C. Halloy3
- S. Hempfling
- M. Henley
- J. Hines
- D. Levy5
- B. Renaud
- B. Whitten
- L. Williams5
- K. Wong3

**Cray **
**Supercomputing**
**Center for **
**Excellence**
- J. Levesque4
- S. Allen
- L. DeRose4
- D. Kiefer4
- J. Larkin4
- N. Wichmann4

**Technology **
**Integration**
- S. Canon
- S. Allen
- T. Barron
- R. Graham
- K. Matney
- M. Minich
- S. Oral
- G. Shipman
- D. Steinert
- F. Wang
- V. White
- W. Yu5

**High-Performance**
**Computing Operations**
- A. Baker
- S. Allen

**Scientific Computing**
- R. Kendall
- L. Rael

- D. Londo4
- J. Lothian
- D. Maxwell#
- M. McNamara4
- J. Miller6
- G. Phipps, Jr.6
- G. Pike
- S. Shpanskiy
- D. Vasil
- S. White
- C. Willis4
- T. Wilson6

- M. Bast
- J. Becklehimer4
- J. Breazeale6
- J. Brown6
- M. Disney
- A. Enger4
- C. England
- J. Evanko4
- A. Funk4
- M. Griffith
- V. Hazelwood5
- J. Hill
- C. Leach6

- C. Jin
- S. Klasky#
- J. Kuehn5
- V. Lynch5
- B. Messer
- R. Mills5
- G.Ostrouchov5
- D. Pugmire
- N. Podhorszki
- R. Sankaran
- A. Tharrington
- S. Thornton3
- R. Toedte
- T. White

- S. Ahern##
- S. Alam5
- E. Apra5
- D. Banks3
- R. Barreto2
- R. Barrett5
- J. Daniel
- M. Eisenbach
- M. Fahey
- J. Gergel5
- R. Hartman-Baker
- S. Hodson

- #End-to-End Solutions Lead
- ##Viz Task Lead

- #Technical Coordinator

**Notes:**
Left side is User Support
Right side is HW and SW support
Tremendous linkage among groups.
All staff in one building, groups collocated

## Visualization and data analytics

- End-to-End Solutions

- Visualization and data analytics

- Visualization

**Researchers must analyze, organize, and transfer an enormous quantity****of data. The End-to-End task group streamlines the work flow for system users so that their time is not eaten up by slow and repetitive chores.**
**Automate routine activities, e.g., job monitoring at multiple sites**

**Once users have completed their runs, the Visualization task group helps them make sense of the sometimes overwhelming amount of information they generate.**
**Viewing at a 30******** x 8******** PowerWall**
**Cluster with GPUs for remote visualization**

## Scientific computing user support model

**“****Whatever it takes” is the motto. **
**Share expertise in algorithms and application-development strategies.**
**Provide porting, tuning, optimization.**
**Help users in running applications, using application development tools and libraries.**
**Ensure application readiness by partnering with users to develop current next-generation applications.**
**Represent users’ needs in LCF planning and reporting exercises:**
**application requirements,**
**scientific progress and highlights,**
**issues with current resources.**

**The LCF provides experts in user support, including Ph.D.-level liaisons from fields such as chemistry, climate, physics, astrophysics, mathematics, numerical analysis, and computer science—who are also experts in developing code and optimizing it for the LCF systems.**
**Large projects are assigned liaisons to maximize opportunities for success on the leadership computing resources.**

- Expertise

## Partnership with projects on LCF resources

- Partnership with projects on LCF resources

**Porting/tuning**
**Library utilization**
**Compiler flags**

**Optimizations**
**Choice of tools**

**Code design**
**and **
**algorithmic developments**
**@**
**scale**

**Shared R&D staff**

**Pilot**

**End Stations**

**Projects**

**Level of integration**

## Liaison model helps maximize science

**Project**

**End-to-End**

**Visualization**

**Computing**

## Producing new insights for RF heating of ITER plasmas

**Fully 3-dimensional simulations of plasma shed new light on the behavior of superheated ionic gas in the multibillion-dollar ITER fusion reactor.**

***“******Until recently, we were limited to two-dimensional simulations. The larger computer [Jaguar] has allowed us to achieve three-dimensional images and validate the code with observations.” ***– Fred Jaeger, ORNL

**Hot spots by antenna surface**

**Radial wave propagation**

**3D simulations reveal new insights:**
**“****hot spots” near antenna surface,**
**“****parasitic” draining of heat to the plasma surface in smaller reactors.**
**Work pushing the boundaries of the system (22,500 processor cores, 87.5 TF) and demonstrating**
**radial wave propagation and rapid absorption,**
**efficient plasma heating.**
**AORSA’s predictive capability can be coupled with Jaguar power to enhance fusion reactor design and operation for an unlimited clean energy source.**

## First-ever control runs of CCSM 3.5 at groundbreaking speed
Major improvements in CCSM 3.5
Arctic and Antarctic sea ice: Will the Arctic be ice free in summer of 2050?
Surface hydrology of land, critical for predictions of drought
Positioned to test full carbon-nitrogen cycle_x001a_

**First-ever control runs of CCSM 3.5 at groundbreaking speed**
**Major improvements in CCSM 3.5**
**Arctic and Antarctic sea ice: Will the Arctic be ice free in summer of 2050?**
**Surface hydrology of land, critical for predictions of drought**
**Positioned to test full ****carbon-nitrogen cycle****_x001a_**

- Climate scientists on cloud 9 (or 3.5)

**“****[On Jaguar,] we got 100-year runs in 3 days. This was a significant upgrade of how we do science with this model. Forty years per day was out of our dreams.”– ***Peter Gent of NCAR, Chairman of CCSM Scientific Steering Committee, during keynote address at CCSM Workshop, June 19, 2007*

*“**The most impressive new result in 10 years.” **–* *Peter Gent, NCAR, on  El Nin**õ**/Southern Oscillation*

- Simulated time evolution of the atmospheric CO2 concentration originating from the land’s surface** **

## Turning vehicle exhaust into power

***Researchers simulate materials that turn heat into electricity***

**Waste heat claims 60% ****of the energy generated ****by an automobile engine.**
**Team led by Jihui Yang of ****General Motors simulates materials that turn that heat into electricity.**
**General Motors, largest-ever simulation—****1,000-plus atom supercell—made possible by NCCS leadership computing resources.**
**Exploring thermodynamic properties of promising lead-tellurium-based material.**

**“****Only at  a place like the LCF can such an expensive calculation be done. We’re very lucky that LCF has been very supportive.” – ***Jihui Yang, General Motors*

**Notes:**
Jihui Yang of General Motors is work with Changfeng Chen of the University of Nevada–Las Vegas to perform first-principles calculations of thermoelectric materials capable of turning waste heat into electricity. 

In particular, the project is using Jaguar to examine an especially promising lead-tellurium-based material, calculating the energetics, electronic structure, and thermodynamic properties. 

## Finding the right balance of plasma turbulence for fusion energy

| Plasma creates energy when hydrogen atoms collide | Resulting high-energyalpha particles heatthe plasma, but can be ejectedby turbulence of the gas. |
| --- | --- |
| Turbulenceis necessary for a tokamak reactor | The GYRO code computes“optimal turbulence,”finding the perfect balanceof heat and alpha-particleproduction and loss. |
| LCF liaison contributions | Doubled performance of GYRO application on Cray X1E.
More effective use of MPI communication; bug finds and fixes.
Imported new sparse solver for decreased memory size and B/W requirements. |

**“****	****I just want to repeat that Mark Fahey of ORNL has been a crucial person in this effort, especially for code optimization. He sees things we sometimes don't. I have nothing but great things to say about him.”–**** *****Jeff Candy, General Atomics***

**Cutting-edge research explores tokamak****plasmas**** **

## Discovering the elusive core-collapse supernova explosion mechanism

**Investigators achieved longer run simulations and, 0.8 seconds after explosion, saw the initial shock wave revived by turbulence of infalling material.**
**CHIMERA code to investigate multiple stellar models, the effects of both Newtonian and Einsteinian gravity, and the impact of recently discovered subatomic physics.**
**Simulations achieved a 256 x 256 spatial mesh (2- to 4-fold increase over the state of the art).**

**Researchers glean unprecedented insight into the shock waves that blow apart a 10- to 20-Solar-mass star.**

**The upgraded Jaguar allows researchers to double the time simulated to 0.8 second “post-bounce.” Petascale systems will allow longer simulations. Tens of seconds after the explosion, heavy elements such as uranium are produced by the fiery storm of the supernova.**

## High-fidelity modeling of ocean CO2 uptake

- Project looks into the fate of trapped heat and greenhouse gases

**Simulation promises to increase understanding of the ocean’s role in regulating climate,****as a repository for greenhouse gases**

- High-fidelity modeling of ocean CO2 uptake

**First-ever 100-year simulation of the ocean at a fine enough scale to include the relatively small, circular currents known as eddies. Until recently researchers lacked the computing ****power to simulate eddies directly on ****a global scale.**
**The most fine-grained, global-scale ****simulations ever of how the oceans work.**
**New knowledge of the currents and processes at work in the oceans.**
**Details of possible transport of gases and chemicals released into the ocean.**

## Gaining understanding of cause and effect of core plasma turbulence

- Small eddies created by plasma turbulence in cross section along with the magnetic field lines threading the simulated tokamak

***W. X. Wang, PPPL          and S. Klasky, ORNL***

**A team led by Dr. W. W. Lee is using NCCS supercomputers to explore heat and particle loss in tokamak reactors. **
**Tokamaks are doughnut-shaped devices that house the ionized gas responsible for sparking the fusion reaction necessary to produce the energy.**
**Temperature must be regulated to create a proper environment for reactions.**
**Device must be large enough to facilitate the reactions.**
**2006 allocation: 2 million hours on Jaguar and 225,000 hours on Phoenix.**
**2007 allocation: 6 million hours on Jaguar and 75,000 hours on Phoenix.**

## Researchers spin better pulsar explanation

***This visualization shows the***
***propagation of a stationary-***
***accretion-shock-instability wave ***
***in a core-collapse supernova. ***
***The leading edge of a spiral flow***
***near the surface of the supernova******shock is marked by the blue area***
***in the figure. It is accompanied by a second flow spinning in the opposite direction underneath. This second spinning flow is responsible for imparting the pulsar spin, according to three-dimensional simulations performed at Oak Ridge National Laboratory.***

***Image Courtesy ***
***of John Blondin***

**Pulsars are left over from core-collapse supernovas.**
- Conventional wisdom:** **Pulsar spin comes from the spin of the original star.
- Better explanation:** **The core-collapse shockwave creates two rotating flows, with pulsar spin created by the inner flow.
- Why it’s better:** **It explains the range of observed pulsar spins, while the conventional wisdom explains only the fastest spins.
**Three-dimensional simulations run on the Cray X1E (Phoenix).**
- Tony Mezzacappa**, ORNL, and **John Blondin**, North Carolina State, published their findings in the January 4, 2007 issue of *****Nature*****.**

## New results in flame stabilization in an auto-ignitive jet

**Fuel**
**400 K**

**Air**
**1100 K**

**Air**
**1100 K**

**Instantaneous***** *****OH radical concentration on a stoichiometric mixture fraction iso-surface shows flame liftoff**

**First fully resolved simulation of a 3D lifted flame in heated co-flow with detailed chemistry.**
**Lifted flames occur in diesel engines and gas turbine combustors.**
**Flame stabilized against fuel jet and recirculating ****hot gases.**
**Direct numerical simulation of a lifted flame in ****heated co-flow: **
**~1 billion grid points and 14 degrees of freedom per ****grid point,**
**H****2****/air detailed chemistry,**
**jet Reynolds number = 11,000,**
**largest DNS at the highest Reynolds number,**
**2.5 M hours on Jaguar at the LCF.**
**Simulation reveals source of stabilization:**
**upstream auto-ignition,**
**vorticity generation at flame base due to baroclinic torque.**

**Side view**

**Notes:**
Regular flame anchored to burner surface; “lifted” flame is not attached to anything (the question becomes “how do we stabilize lifted flame?”)
Coflow is heated air (seen in the picture on either side of the fuel)
This calculation was 9,000 cores of jaguar
Above the blue it is burning; baroclinic torque is from the difference in pressure: slightly lower temp in fuel (blue), higher temp of the burning part (1200K, green) , causes a vortex which we try to understand... can you predict the height of the flame?... where does it stabilize?... engineers want to know – if it is too close to burner, it will fail/melt; too far away it shows instability in location and possible extinction. 
Detailed chemistry (vs simple chemistry) 9 chemical radicals – more accurate model and parameter production
Predictive modeling capability for missile engines design, gas-turbine combustors, aircraft engines.
In the field there is currently no predictive capability... all experimental at the moment. 
Ex. BES and EERE interested in making diesel engines cleaner.
So, government and industry are interested in all kinds of engines: the common factor is that they all ‘burn’ something and the current design mode is experimentally based (e.g. build an engine, if it doesn’t work, build another one). This research is for the development of models (and inclusion of parameters there) which will allow engineers to model the flame, its chemistry and behavior. What we see here is the simulation (including various radicals/chemistry) which shows the height above the burner at which the flame exists, and that the flame is stable.

## Preparing for the futureApplication requirements: Process and actionable results

| LCF Application RequirementsCouncil (ARC) | Stood up in 2006
Established ARC charterand requirements management process |
| --- | --- |
| LCF elicits requirementsin many ways | INCITE proposals
Questionnaires devised by LCF staff
One-on-one interviews
Existing publications/documentation
Analyzing source code |
| Application categories analyzed | Science motivation and impact
Science quality and productivity
Application models, algorithms, software
Application footprint on platform
Data management and analysis
Early access science-at-scale scenarios |
| Results | First annual 100+ page application requirements document published:  September 2007
New methods for categorizing platforms and application attributes devisedand utilized in analysis: guiding tactical infrastructure purchase and deployment
Best practice: Process being embraced and emulated by others |

**Notes:**
Your INCITE proposal
An ARC questionnaire devised by NCCS staff (7 categories) 
Science motivation and impact
Science quality and productivity
Application models
Application algorithms
Application software
Application footprint on platform
Data management and analysis
A “code project” survey devised by ASCAC member Gordon Bell
An end-to-end survey devised by NCCS staff
Other sources of information
Numerous reference docs collected and archived
SC liaisons (familiar with application)
Interviews with INCITE project members
INCITE project publications
Analysis of your application source code

## InnovationFeedback loop for ensuringapplication readiness

- InnovationFeedback loop for ensuringapplication readiness

**decides**

**Technology**
**Council**

**Application **
**Requirements **
**Council**

**Resource Utilization Council**

**LCF User****Group**

**Scientific Computing Group**

| Requirements |
| --- |
| Tools
Libraries
Memory
Disk storage
Visualization
Networking
HPC platforms
Security |

| Resources |
| --- |
| Tools
Libraries
Disk storage
Visualization
Networking
HPC platforms
Facilities
Security |

**adjusts use**

**determines**

| Scientific Computing Group | Liaisons to application project teams |
| --- | --- |
| Application Requirements Council | Identification of applicationrequirements |
| Technology Council | Decide how to best meet future application resource needs |
| Resource Utilization Council | Takes into account Science Team time constraints, e.g., upcoming meeting |

## Contact

- Ricky A. Kendall
- Scientific Computing
- National Center for Computational Sciences
- (865) 576-6905
- kendallra@ornl.gov

-    Kendall_SciComp_SC07