Supporting Supporting

the

Warfighter arfighter arfighter

High

Performance Performance

Computing Computing

DoD HPC

MODERNIZATION

PROGRAM

Modernization Plan 2000
Front cover captions (clockwise starting from top right):

Quantum-mechanically derived electrostatic potential of the explosive TNT

corresponding to the 0.001 electrons/bohr3 isosurface of electron density.

The geometry of a Magnetically Insulated Line Oscillator (MILO) and the radial electric

field extremes in the extractor and waveguide regions of the device are shown in the

figure.

The V-22 rotor-wake system is visualized in a post process where filaments of particles

are released from inboard and outboard blade-tips every 25 time-steps.

Computed Structure of Zinc octabromo-meso-tetraphenylporphyrin.

Turning Maneuver Induced by a 10 Degree Rudder Deflection (Selected Views).

Frequency of gravity wave activity above the Alps for 1 October - 15 November 1999,

expressed as a percentage of time, at 10 km amsl (color shading). The calculations

are based on hourly output from COAMPS forecasts with a horizontal resolution of 5

km. The topography is shown in black contours and stippling, with the darkest

representing elevations above 3000 m.
Department of Defense

High Performance Computing

Modernization Program

Modernization Plan 2000

June 2000 June 2000


Executive Summary

High performance computing historically has played a major role in the ability of the United States

to develop and deploy superior weapons, warfighting capabilities, and mission support systems.

Under the auspices of the Deputy Undersecretary of Defense (Science and Technology), the High

Performance Computing Modernization Program continues to focus on acquiring, managing, and

sustaining modern commercial high performance computing resources in support of the defense

science and technology and test and evaluation communities.

The 2000 High Performance Computing Modernization Plan includes a program overview, several

recent examples of defense projects successfully employing high performance computing, and a

section on program accomplishments and future plans, with an emphasis on fiscal years 2000 and

2001. The overriding goal of the program continues to be the exploitation of high performance

computing technology for military advantage across the battlespace. The strategy to achieve this goal

remains that of acquiring and sustaining world-class high performance computing and network

capabilities in support of defense scientists and engineers. The program supports over 600 high

performance computing projects and the user base exceeds 5,000 scientists and engineers located at

more than 100 Department of Defense laboratories, test centers, contractor and academic sites.

Many of the leading edge, computationally intensive projects supported by the program are making

critical contributions to high-priority warfighting systems; others are addressing research challenges

fundamental to assuring defense technology competitiveness on the 21st century battlefield.

The program helps put advanced technology in the hands of U.S. forces more quickly, less

expensively, and with greater certainty of success. Today’s weapons programs, such as the V-22

Tiltrotor, Comanche Helicopter, the Medium Tactical Vehicle Replacement, and the Javelin Missile

program have benefited through innovative materials, advanced design concepts, improved and faster

modification designs, higher fidelity simulations and more efficient tests. Future weapons systems,

such as radio frequency weapons, are benefiting through basic research in plasma physics, molecular

engineering, high energy materials and advanced signal processing. Even though today’s program

has the resources to support only 50–60 percent of the program’s validated high performance

computing requirements, the projects that are supported are making a real difference in ensuring our

warfighters are the best equipped in the world.

The use of high performance computing technology is reducing the costs and time required for

systems analysis, design, development, test, and deployment; helping to avoid environmental damage;

and improving the integration and effectiveness of complex weapons systems. As DoD continues to

reform and reengineer its acquisition processes, high performance computing assets, along with highfidelity scalable models and simulations, are being used to reduce the number and cost of building

expensive prototypes. Increasingly, modeling and simulation are being used to explore more detailed

design options and to identify important testing priorities earlier in the design cycle. High

performance computing is now a key ingredient of the successful implementation of many major DoD

acquisition programs.

3

Contents

Executive Summary .......................................................................................................... 1

Program Overview ..................................................................... ..................................................................... 7

Introduction ...................................................................................................................... 7

Program Mission ............................................................................................................... 7

Program Vision ................................................................................................................. 7

Program Scope ............................................................................................................... 10

User Requirements.......................................................................................................... 10

Computational Technology Areas .................................................................................... 11

Program Goals ................................................................................................................ 16

Program Strategies ......................................................................................................... 16

Program Initiatives .......................................................................................................... 18

High Performance Computing Centers ...................................................................... 19

Major Shared Resource Centers ........................................................................... 19

Distributed Centers .............................................................................................. 20

Networking ................................................................................................................ 22

Defense Research and Engineering Network ........................................................ 22

Security ................................................................................................................ 24

Common High Performance Computing Software Support Initiative .......................... 25

HPC Contributions to DoD Mission HPC Contributions to DoD Mission Success — 2000.................... ....................29

Introduction .................................................................................................................... 29

Expanding Diagnostic Capabilities in Aerodynamic Designs ............................................ 30

High Performance Computing Makes Ground Vehicle Simulations Possible..................... 31

High Performance Aircraft, High Performance Solutions ................................................. 33

From the Sea to the Shore: Forecasting Surface Wave Conditions for the

Military Crisis in Kosovo ................................................................................................... 35
4

2000 HPC Modernization Plan

Program Accomplishments and F rogram Accomplishments and Future Plans e Plans................................... ................................... 37

High Performance Computing Centers ............................................................................ 37

Major Shared Resource Centers ................................................................................. 37

Major Shared Resource Centers, Fiscal Year 2000 ................................................ 37

Major Shared Resource Centers, Fiscal Year 2001 ................................................ 39

Distributed Centers .................................................................................................... 39

Distributed Centers, Fiscal Year 2000 ................................................................... 39

Distributed Centers, Fiscal Year 2001 ................................................................... 43

Networking...................................................................................................................... 43

Defense Research and Engineering Network.............................................................. 43

DREN, Fiscal Year 2000 ....................................................................................... 44

DREN, Fiscal Year 2001 ....................................................................................... 44

Security ..................................................................................................................... 45

Security, Fiscal Year 2000 ..................................................................................... 45

Common High Performance Computing Software Support Initiative ............................... 46

Software Support, Fiscal Year 2000 ...................................................................... 46

Software Support, Fiscal Year 2001 ...................................................................... 46

DoD High Performance Computing Challenge Projects................................................... 47

Program Oversight and Review ....................................................................................... 47

Acquisition ...................................................................................................................... 50

Accomplishments ................................................................................................ 50

Future Plans ......................................................................................................... 51

Program Funding ............................................................................................................ 51

References ................................................................................ ................................................................................ 55

Acronyms Acronyms ................................................................................. ................................................................................. 57
5

CONTENTS

Figures

Figure 1. High Performance Computing Centers .............................................................. 8

Figure 2. Integrated Program Strategy of the Department of Defense High Performance

Computing Modernization Program................................................................................... 9

Figure 3. Interrelationships Among the Operational Concepts, Capability and Technology

Objectives and the HPCMP Computational Technology Areas ......................................... 15

Figure 4. DREN Connections, Shared Resource Centers, and Other Network

Access Points ................................................................................................................. 23

Tables

Table 1. Examples of High Performance Computing Projects and Their Value to the

Warfighter ....................................................................................................................... 12

Table 2. Computational Technology Areas ...................................................................... 13

Table 3. Computational Technology Area Leaders .......................................................... 14

Table 4. Computational Technology Area Support at MSRCs .......................................... 20

Table 5. Common HPC Software Support Initiative Projects and Principal Investigators

(Fiscal Year 2000)............................................................................................................ 27

Table 6. Major Shared Resource Centers High Performance Computing Capability

(Fiscal Year 2000)............................................................................................................ 38

Table 7. Distributed Centers ........................................................................................... 40

Table 8. Distributed Centers High Performance Computing Capability

(Fiscal Year 2000)............................................................................................................ 41

Table 9. DoD High Performance Computing Challenge Projects (Fiscal Year 2000) ........ 48

Table 10. Department of Defense High Performance Computing Modernization

Program Funding (Fiscal Years 1997–2005) .................................................................... 52

Illustrations

7

The High Performance Computing Modernization

Program (HPCMP) was initiated in 1993 in response to

congressional direction to modernize the Department of

Defense (DoD) laboratories’ high performance computing

(HPC) capabilities. Early on, senior leaders recognized the

unique potential of this technology as critical to our nation’s

future defense. Today, the HPCMP fields world-class,

commercial high performance computing capability, available

to the full Science and Technology (S&T) and Test and

Evaluation (T&E) communities, that includes a balanced set

of major shared resource centers, smaller focused distributed

centers, wide area networking services, and HPC software

development support. Figure 1 shows the HPC center

locations and Figure 2 depicts the integrated program

strategy.

The HPCMP mission is to provide world-class,

commercial, high-end high performance computing capability

to the DoD science and technology and test and evaluation

communities, in order to allow our scientists and engineers to

incorporate technological advantage into superior weapons,

warfighting capabilities, and related support systems more

rapidly and affordably with reduced risks to human life and

optimized system performance.

The HPCMP vision is to enable the DoD to maintain its

technological supremacy over adversaries of the United States

in weapon systems design and to foster the flow of this

technology into warfighting support systems, by providing

world-class, commercial HPC capability to the science and

technology and test and evaluation communities.

Program Overview

Introduction

Program Mission

Program Vision
8

2000 HPC Modernization PlanFIGURE 1. High Performance Computing Centers
9

PROGRAM OVERVIEW

FIGURE 2. Integrated Program Strategy of the Department of Defense High Performance Computing Modernization Program
10

2000 HPC Modernization Plan

The HPCMP scope is bounded both in terms of the user

community it serves and the technological capability that it

delivers. By limiting the scope and by concentrating the

majority of resources at a small number of HPC centers, the

program has been able to provide world-class, commercial

computing capabilities that could not have efficiently been

obtained and sustained by individual Services or Agencies.

This sharing of resources reduces overall acquisition and

sustainment costs and fosters collaboration and cooperation

across the DoD S&T and T&E communities.

The scope of the user community is defined by the

Congress (Public Law 104-61, December 1, 1995, 109

Statute 665, Sec. 8073) to be

“(1) the DoD Science and Technology sites under the

cognizance of the DDR&E, (2) the DoD Test and Evaluation

centers under the Director, Test and Evaluation OUSD (A&T),

and (3) the Ballistic Missile Defense Organization.”

From a technology capability perspective, the program

maintains a strict focus on providing high performance

computing. The definition of “high performance computing”

changes as the technology continues to evolve. This

consistent evolution requires that the program periodically

reassess its acquisition plans to ensure that it is not acquiring

systems which have become “departmental” class.

Departmental class systems remain the responsibility of the

local organization. The rapid evolution of high performance

computing also requires that the program focus on delivering

improved capability early in its life cycle. This allows the DoD

to maintain the technological edge required to analyze,

design, produce and deploy advanced weapons systems and

capabilities to the warfighter — before similar computational

capabilities are available to our adversaries and economic

competitors.

From its inception, the HPCMP was designed to address

the continuing high-end computing needs of the DoD S&T

program. Congress later amended the program scope to

include the T&E community creating its current user base of

over 5,000 scientists and engineers located at more than 100

DoD laboratories, test centers, contractor and academic sites.

The program office, working with representatives of the user

community, identify and track requirements at the

computational project level.

Program Scope

User Requirements
11

PROGRAM OVERVIEW

The requirements process, developed and employed by

the program, includes an annual survey sent to users and

potential users throughout the DoD S&T and T&E

communities, coupled with visits to selected sites to discuss

requirements and to provide information to user groups. In

addition, the program obtains feedback through user

satisfaction surveys, annual user group meetings, and

advisory panels that work with program representatives to

establish policies and procedures at the HPC centers.

HPC systems performance metrics are tracked throughout

the year at the computational project level to ensure projects

receiving allocations are indeed able to use the resources

effectively and efficiently. This process provides feedback to

the Services and Agencies prior to allocating resources the

following fiscal year. The overall integrated process, linked at

the computational project level, furnishes accurate and timely

data on which to base program decisions.

Twenty to twenty-five percent of the program’s total

resources are dedicated each year to a set of DoD HPC

Challenge Projects. These computationally intensive, highpriority projects are selected annually through a rigorous

technical and mission relevance evaluation. The remaining

resources are allocated by Services and Agencies according to

their prioritized needs. Table 1 provides a few examples of

HPC projects and their value to the warfighting community.

The DoD HPCMP user community is organized around

ten broad computational technology areas (CTAs). Each

computational technology area has a designated leader who is

a prominent DoD scientist or engineer working within the

research disciplines included in his or her area. Table 2

provides a brief description of each CTA. Table 3 lists the

current CTA leaders.

Teams of leading defense scientists and engineers use

HPC resources to develop, advance, maintain, and use

hundreds of scientific algorithms, codes, models and

simulations needed to analyze, design, develop, test, evaluate

and deploy technologically superior weapons, warfighting

capabilities, and related support systems. Their efforts

produce and support key enabling technologies, capabilities,

and demonstrations articulated by the Defense Technology

User Requirements–

continued

Computational

Technology Areas
12

2000 HPC Modernization Plan

TABLE 1. Examples of High Performance Computing Projects and Their Value to the Warfighter

Projects Value to the Warfighter and Joint Warfighting

Capability Objectives Supported

3-D Computational Fluid Dynamics (CFD)

Modeling of the Chemical Oxygen-Iodine Laser

Provides the Air Force Space-based Laser (SBL) program with

data to assist in developing an advanced nozzle for increased

performance. Impacts the Airborne Laser (ABL) development

program by providing high quality, high fidelity simulations of

chemical laser devices in a timely manner. (Precision Force)

Applied CFD in Support of Aircraft-Store

Compatibility and Weapons Integration

Enables the performance of numerical simulations to determine

store loads, store separation characteristics, and trajectories.

Provides flight clearances and certification recommendations

that translate into an operational capability. (Joint Readiness

and Logistics, and Sustainment of Strategic Systems)

Coupled Environmental Model Prediction (CEMP) Enables development of coupled ocean/ice/air prediction

models to provide forecasts in the battle space environment.

(Joint Readiness and Logistics, and Sustainment of Strategic

Systems)

Development of Standards for Stand-Off

Distance and Blast Walls for Force Protection

Enables development of standards for standoff distances and

blast wall designs for survivable bases and facilities to meet

force protection needs. Enhances the design of barriers or blast

walls to shield or protect vulnerable compounds and structures

and advance protective construction technology. (Combating

Terrorism)

Global and Basin-Scale Ocean Modeling and

Prediction

Provides ocean models and capability to the Fleet Numerical

Meteorology and Oceanography Center that furnishes daily

atmospheric and oceanographic forecasts for the operational

Navy. (Joint Readiness and Logistics, and Sustainment of

Strategic Systems)

High-Resolution Adaptive Mesh Refinement

(AMR) Simulations of Confined Explosions

Provides input to military strategy for counter-proliferation

measures. Aids forensic investigations of terrorist bombing

incidents. Suggests methods of protecting civilians and US

troops. (Combating Terrorism)

Modeling Complex Projectile Target Interactions Increase ballistic protection of ground combat vehicles and

increase munitions lethality. (Precision Force)

Parallel Simulations of Weapon/Target

Interactions Using a Coupled CFD/Computational

Structural Dynamics (CSD) Methodology

Improves counter-terrorism and force-protection activities. Aids

in the forensic analysis following an attack. Supports better

targeting of weapons-of-mass-destruction facilities by improved

blast-damage assessment tools. (Force Protection)

Unsteady Aerodynamics of Advanced Guided

Munitions

Reduces design costs and provides a more detailed

understanding of the complex aerodynamics than can be

achieved through experiment and actual test firings. (Joint

Readiness and Logistics, and Sustainment of Strategic

Systems)

Virtual Prototyping of Radio Frequency (RF)

Weapons

Enables detailed modeling of directed energy radio frequency

(RF) weapons, which are expected to play a large role in

controlling the battlefield of the 21st century. (Precision Force)
13

PROGRAM OVERVIEW

TABLE 2. Computational Technology Areas

Computational Structural Mechanics (CSM)

provides high resolution, multi-dimensional modeling of

materials and structures subjected to a broad range of

static, dynamic, and impulsive loading conditions. Uses

include affects of explosions on various facilities,

underwater explosion and ship response, structural

acoustics, structural analysis, propulsion systems,

lethality and survivability (aircraft, ships, submarines,

tanks), theater missile defense, and real-time, largescale soldier and hardware in-the-loop vehicle

dynamics.

Signal/Image Processing (SIP) provides for extraction

and analysis of key real-time information from various

sensor outputs. Sensors include sonar, radar, visible

and infrared imagers, and signal intelligence and

navigation assets. Uses include intelligence,

surveillance and reconnaissance (ISR), avionics,

communications, smart munitions, and electronic

warfare. Functions include detecting, tracking,

classifying, and recognizing targets in the midst of noise

and jamming; generating high-resolution low-noise

imagery; and the compression of imagery for

communications and storage.

Computational Fluid Dynamics (CFD) provides the

accurate numerical solution of the equations describing

fluid and gas motion and fluid-dynamics research. Uses

include the design of complex combustion and

propulsion systems that are inaccessible or too costly to

prototype; the dynamics of submarines, subsonic,

transonic and supersonic air vehicles, pipe flows, air

circulation, missiles, and projectiles; and

magnetohydrodynamics for advanced power systems

and weapons effects.

Forces Modeling and Simulation (FMS)/C4I

integrates high speed command, control,

communications, computers and intelligence systems

(C4I) to manage a battle space; provides large-scale

simulations of complex military engagements to facilitate

mission rehearsal/training, mission planning, and postmission analysis; and advances digital library

technology for support of FMS/C4I research and

development activities. Uses span the design,

development, test, evaluation, and deployment of a

variety of warfighting and training systems.

Computational Chemistry and Materials Science

(CCM) predicts properties of chemical species and

applies this molecular understanding to development of

advanced materials. Uses include design of new

chemical compounds for fuels, lubricants, explosives,

rocket propellants and chemical defense agents. In

addition, advanced modeling techniques are used to

develop new high performance materials for electronics,

advanced sensors, engines, laser protection systems,

engine components, and biomedical applications.

Environmental Quality Modeling and Simulation

(EQM) provides high-resolution, three-dimensional

Navier-Stokes modeling of hydrodynamics and

contaminant transport through air, ground and aquatic

ecosystems. Uses include stewardship and

conservation of natural and cultural resources, prediction

of chemical, biochemical contaminant flows, design and

operation of installation restoration, integrated

management in support of environmental quality, noise

evaluation and abatement, and water quality models.

Computational Electromagnetics and Acoustics 

(CEA) provides high-resolution, multi-dimensional

solutions of Maxwell's equations and acoustic wave

equations. Uses include calculating fields about

antenna arrays; signatures of tactical ground, air, sea

and space vehicles; the signature of buried munitions;

performance/design factors for electromagnetic gun

technology; high power microwave performance;

modeling of acoustic fields for surveillance and

communication; seismic fields for mine detection; and

acoustic shock waves of explosions for anti-personnel

weapons.

Computational Electronics and Nanoelectronics 

(CEN) provides the design, modeling, and simulation of

complex electronic devices, integrated circuits, and

small components. Uses include lower costs, weights,

size and improved performance of electronics through

predictive high-fidelity modeling and simulation;

analog/digital high-frequency circuit and device

simulation and optimization; modeling and simulation of

micro-electromechanical devices, micro-resonators,

active and passive millimeter-wave circuits and

structures; electronic/photonic interconnect and

packaging analysis; and fault modeling.

Climate/Weather/Ocean Modeling and Simulation

(CWO) provides the numerical simulation and forecast

of the earth's climate as well as oceanic and

atmospheric variability. Uses include improved flight

safety; search-and-rescue mission planning;

propagation of weapons; aircraft and ship routing; antisubmarine and undersea warfare; enhanced capabilities

in adverse weather; capability to predict magnetic

storm-induced effects; and outages on command,

control, communications (C3), surveillance, and

navigation systems.

Integrated Modeling and Test Environments (IMT)

applies high performance computing software tools and

techniques with live tests and hardware-in-the-loop

simulations for test and evaluation of DoD weapons,

components, subsystems, and systems in virtual and

composite virtual/real environments. Uses include

digital scene generation, six degree-of-freedom

trajectory simulations, real-time test-data analysis and

display systems for test control and evaluation, and high

fidelity engineering and closed-loop engagement models

(one-on-one and few-on-few).
14

2000 HPC Modernization Plan

Objectives (DTOs). These enabling DTOs, in turn, support

Joint Vision 2020 and the eleven Joint Warfighting Capability

Objectives (JWCOs) promulgated by the Joint Requirements

Oversight Council of the Joint Chiefs of Staff.

While not all-inclusive, JWCOs provide focus, priority, and

a common reference point for much of the DoD’s research,

test and evaluation efforts. They are described in detail in the

annual Joint Warfighting Science and Technology Plan

(JWSTP).

Figure 3 depicts the interrelationships among Joint Vision

2020, the JWCOs, the DTOs, and the CTAs. In most cases,

each CTA supports multiple DTOs and JWCOs. Likewise,

individual DTOs and JWCOs are supported by multiple CTAs.

Computational

Technology Areas–

continued

TABLE 3. Computatioal Technology Area Leaders

Computational Technology Areas Leaders

Computational Structural Mechanics Dr. Raju Namburu

Army Research Laboratory, Aberdeen Proving Ground, MD

Computational Fluid Dynamics Dr. Jay Boris

Naval Research Laboratory, Washington, DC

Computational Chemistry and Materials Science Dr. Leslie Perkins

Air Force Research Laboratory, Propulsion Directorate,

Edwards Air Force Base (AFB), CA

Computational Electromagnetics and Acoustics Dr. Robert Peterkin

Air Force Research Laboratory, Directed Energy

Directorate, Kirtland AFB, NM

Climate, Weather, and Ocean Modeling and Simulation Dr. George Heburn

Naval Research Laboratory, John C. Stennis Space Center,

MS

Signal/Image Processing Dr. Richard Linderman

Air Force Research Laboratory, Information Directorate,

Rome, NY

Forces Modeling and Simulation/C4I Mr. Robert Wasilausky

Space and Naval Warfare Systems Center, San Diego, CA

Environmental Quality Modeling and Simulation Dr. Jeffrey Holland

Engineer Research and Development Center, Vicksburg,

MS

Computational Electronics and Nanoelectronics Dr. Barry Perlman

Army Communications-Electronics Command Research

Development and Engineering Center, Ft. Monmouth, NJ

Integrated Modeling and Test Environments Dr. Andrew Mark

Army Research Laboratory, Aberdeen Proving Ground, MD
15

PROGRAM OVERVIEW

FIGURE 3. Interrelationships Among the Operational Concepts, Capability and Technology Objecives and the

HPCMP Computational Technology Areas

Computational

Technology Areas

The program supports defense

science and technology and test

and evaluation efforts in ten

broad computational technology

areas:

• Computational Structural

Mechanics

• Computational Fluid Dynamics

• Computational Chemistry and

Materials Science

• Computational

Electromagnetics and

Acoustics

• Climate, Weather, and Ocean

Modeling and Simulation

• Signal/Image Processing

• Forces Modeling and

Simulation/C4I

• Environmental Quality

Modeling and Simulation

• Computational Electronics and

Nanoelectronics

• Integrated Modeling and Test

Environments

Joint Warfighting

Capability Objectives

The program supports 11 Joint

Warfighting Capability Objectives

defined by the Joint Chiefs of

Staff. These represent some of

the most critical capabilities for

maintaining the warfighting

advantage of U.S. forces:

• Information Superiority

• Precision Force

• Combat Identification

• Joint Theater Missile Defense

• Military Operations in

Urbanized Terrain

• Joint Readiness and Logistics

and Sustainment of Strategic

Systems

• Force Projection/Dominant

Maneuver

• Electronic Warfare

• Chemical/Biological Warfare

Defense and Protection and

Counter Weapons of

Mass Destruction

• Combating Terrorism

• Protection of Space Assets

Defense Technology

Objectives

The program supports most of

the 347 Defense Technology

Objectives (DTOs) as set forth in

the Defense Technology Area

Plan and the Joint Warfighting

Science and Technology Plan.

The DTOs are defined by

technology managers and

program managers. These

objectives identify specific

technology advancements to be

developed or demonstrated in the

following 11 categories:

• Air Platforms

• Chemical/Biological Defense

• Information Systems

Technology

• Ground and Sea Vehicles

• Materials/Processes

• Biomedical

• Sensors, Electronics, and

Battlespace Environment

• Space Platforms

• Human Systems

• Weapons

• Nuclear Technology

Joint Vision 2020

Emerging Operational

Concepts

The High Performance

Computing Modernization

Program supports the concepts

defined by the Joint Chiefs of

Staff's Joint Vision 2020:

• Dominant Maneuver

• Precision Engagement

• Full-Dimensional Protection

• Focused Logistics

• Information Superiority

• Technological Innovation

Joint

Vision

2020

Joint Warfighting

Capability Objectives

Defense Technology Objectives

Computational Technology Areas
16

2000 HPC Modernization Plan

The program provides an increasingly powerful, high-end,

high performance computing environment and has the

following goals:

(1) Continually acquire the best commercially available highend HPC systems.

(2) Acquire and develop common-use software tools and

programming environments.

(3) Expand and train the DoD high performance computing

user base.

(4) Link users and computer sites via high-speed networks,

facilitating the creation of collaborative work

environments.

(5) Exploit the best ideas, algorithms, and software tools

emerging from the national high performance computing

infrastructure.

The program implements the following strategies to

achieve the five program goals:

Goal 1 — Provide the best commercially available high-end

HPC capability.

• Provide a balanced set of commercially available

heterogeneous computing platforms and HPC systems to

meet the full range of DoD requirements and permit the

optimum mapping of requirements to system types.

• Build complete HPC environments, including large

computing systems, software, and support expertise at a

few DoD laboratories designated as major shared

resource centers (MSRCs) to support a wide community

of users.

• Place modest-sized HPC systems at selected distributed

centers (DCs) when that allows for either innovation or

mission support not provided by a major shared resource

center.

• Support efforts to build complementary HPC capabilities

and technologies throughout government, academia,

and industry that are applicable to defense S&T and T&E

requirements.

• Continually acquire and upgrade equipment and support

services to ensure that DoD HPC centers provide worldclass, commercial, state-of-the-art capabilities.

Program Strategies

Program Goals
17

PROGRAM OVERVIEW

Program Strategies–

continued

Goal 2 — Acquire and develop common-use software tools

and programming environments.

• Promote standards for defense research, development,

test, and evaluation software to ensure future transitions

and advancements in software technology are applied in

an efficient, cost-effective manner.

• Focus on applications software initiatives designed to

overcome technological inhibitors that may delay the

effective use of scalable high performance computers.

• Support software components used in specific

computational technology areas or by a subset of HPC

users across the community.

• Identify software suites to be shared across major shared

resource centers and selected distributed centers.

Goal 3 — Expand and train the DoD high performance

computing user base.

• Provide user education in scalable computer and

computational sciences, software applications and

optimization, and code conversion.

• Promote the formation of DoD-sponsored

interdisciplinary teams and collaborative groups to

determine how best to support each computational

technology area and to leverage academia and industry

expertise in HPC for the solution of DoD problems.

• Develop shared application software in support of highpriority and broadly-based computational technology

area needs.

• Aggressively transfer expertise and knowledge among the

DoD HPC user communities.

• Broaden the knowledge base of the DoD HPC user

community based on new applications and their

capabilities.

Goal 4 — Link users and computer sites via high-speed

networks, facilitating the creation of collaborative work

environments.

• Deploy robust and cost effective network connectivity

consistent with HPC traffic requirements among and

between DoD scientists and engineers and HPC assets,

and between the DoD and external research and test and

evaluation communities.
18

2000 HPC Modernization Plan

• Encourage remote usage where appropriate.

• Ensure users and the highest priority projects are well

supported.

Goal 5 — Exploit the best ideas, algorithms, and software

tools emerging from the national high performance

computing infrastructure.

• Ensure that users in the defense science and technology

and test and evaluation communities remain cognizant

of, interact with, and leverage internal Defense

Department initiatives.

• Ensure that defense science and technology and test and

evaluation users remain cognizant of, collaborate with,

and leverage other government, academia, industry, and

each other’s HPC efforts.

• Maintain cooperative contact with the appropriate

committees, subcommittees, working groups, and

advisory panels associated with the National Science and

Technology Committee structure.

• Maintain cooperative contact with the Department of

Energy Accelerated Strategic Computing Initiative.

• Provide HPC resources to support defense applications

that directly relate to dual-use technologies and National

Challenges, such as environment, medical, digital

libraries, and manufacturing processes and products.

• Ensure the availability of HPC resources to support

defense applications that address DoD HPC Challenge

Project needs.

• Leverage the nation’s HPC infrastructure to benefit the

warfighter.

Three initiatives implement the program strategies,

outlined previously, to achieve its goals. The initiatives are

HPC centers, Networking, and the Common High

Performance Computing Software Support Initiative (CHSSI).

The initiatives improve the ability of users to analyze, develop,

and deploy advanced weapons, warfighting capabilities, and

related support systems more rapidly and affordably with

reduced risk to human life and with optimal system

Program Strategies–

continued

Program Initiatives
19

PROGRAM OVERVIEW

performance. The initiatives align to program goals as

follows:

High Performance Computing Centers

Major Shared Resource Centers

The HPCMP operates four large major shared resource

centers to enable the DoD S&T and T&E communities to

effectively and efficiently use the full range of high-end HPC

resources. Each MSRC is operated and maintained by a

team of government and contractor personnel. The four

MSRCs are:

• Aeronautical Systems Center (ASC), Wright-Patterson

AFB, OH

• Army Research Laboratory (ARL), Aberdeen Proving

Ground, MD

• Army Engineer Research and Development Center

(ERDC), Vicksburg, MS

• Naval Oceanographic Office (NAVO), John C. Stennis

Space Center, MS

Each MSRC includes a robust complement of high-end

high performance computing and communications systems,

scientific visualization capabilities, peripheral and archival

mass storage devices, and support staff providing expertise in

the use of these assets. The MSRC emphasis is on supporting

large computational projects. As shown in Table 4, each

center’s hardware configurations, software, programming

environments, and training efforts are focused to best support

the needs of a particular set of CTAs designated for that

center.

Program Initiatives–

continued

HPC

Centers Networking SoftwareSupport

Goal 1 X

Goal 2 X X

Goal 3 X X X

Goal 4 X

Goal 5 X X X
20

2000 HPC Modernization Plan

A key element of support at each MSRC is the

Programming Environment and Training (PET) effort. This

effort, in addition to providing extensive training to meet a

wide variety of user needs, supports incorporation of state-ofthe-art system tools and HPC technology to facilitate the

effective and efficient use of the various HPC systems. This

support is accomplished through collaborative partnerships

established by each MSRC with several leading civilian HPC

centers and academic institutions. Such rapid collaboration

fosters the capture of innovation and draws additional high

performance computing technology and expertise into the

DoD. MSRCs also focus on training the DoD user base,

identifying HPC technology opportunities, and introducing

these opportunities into all HPC centers’ computing

environments.

Distributed Centers

Distributed centers (DCs) provide HPC capability to a

specified local and remote portion of the HPC user

community. Modest-sized systems are deployed to DCs

where there is a significant advantage to having a local HPC

Program Initiatives—

continued

TABLE 4. Computational Technology Area Support at MSRCs

Computational Technology Areas ASC ARL ERDC NAVO

Computational Structural Mechanics x x x

Computational Fluid Dynamics x x x x

Computational Chemistry and Materials Science x x

Computational Electromagnetics and Acoustics x x

Climate, Weather, and Ocean Modeling and Simulation x x

Signal/Image Processing x x

Forces Modeling and Simulation/C4I x x

Environmental Quality Modeling and Simulation x x

Computational Electronics and Nanoelectronics x

Integrated Modeling and Test Environments x
21

PROGRAM OVERVIEW

system and where there is a unique potential for advancing

DoD applications. DCs leverage HPC expertise or address

problems that cannot be readily solved at the MSRCs, such as

real-time or near real-time processing, embedded system

applications, and man-in-the-loop and hardware-in-the-loop

testing and evaluation. The centers are linked by high-speed

communications to the MSRCs and remote users. Thus, they

augment the MSRCs to form the total DoD HPCMP

computational capability. The seventeen DCs are:

• Air Armament Center (AAC), Eglin AFB, FL

• Air Force Flight Test Center (AFFTC), Edwards AFB, CA

• Air Force Research Laboratory, Information Directorate

(AFRL/IF), Rome, NY

• Air Force Research Laboratory, Sensors Directorate

(AFRL/SN), Wright-Patterson AFB, OH

• Arctic Region Supercomputing Center (ARSC), Fairbanks,

AK

• Army High Performance Computing Research Center

(AHPCRC),Minneapolis, MN

• Arnold Engineering Development Center (AEDC), Arnold

AFB, TN

• Joint National Test Facility (JNTF), Schriever AFB, CO

• Maui High Performance Computing Center (MHPCC),

Kihei, HI

• Naval Air Warfare Center Aircraft Division (NAWCAD),

Patuxent River, MD

• Naval Air Warfare Center Weapons Division (NAWCWD),

China Lake, CA

• Naval Research Laboratory (NRL-DC), Washington, DC

• Redstone Technical Test Center (RTTC), Huntsville, AL

• Space and Missile Defense Command (SMDC), Huntsville,

AL

• Space and Naval Warfare Systems Center (SSC-SD), San

Diego, CA

• Tank Automotive Research, Development and Engineering

Center (TARDEC), Warren, MI

• White Sands Missile Range (WSMR), White Sands Missile

Range, NM

Program Initiatives—

continued
22

2000 HPC Modernization Plan

Networking

Defense Research and Engineering Network

The Defense Research and Engineering Network (DREN)

is a Defense Department data communications network.

Figure 4 graphically displays the DREN connections for the

program’s high performance computing centers, and other

wide area network access points. The program’s users require

reliable, leading edge, high-speed access to HPC resources.

DREN satisfies this need by providing wide area networking

(WAN) services through a virtual private network using a

public Asynchronous Transfer Mode (ATM) communications

infrastructure supplied by American Telephone & Telegraph

(AT&T). High-speed remote connectivity is especially

important to our 3,500 remote users.

Providing wide area access over a commercial ATM

infrastructure has distinct advantages. Connecting and

disconnecting sites from the network and changing service

levels can be accomplished easily. Because DREN uses

commercial services, the most advanced networking

technology can be used as soon as it becomes available in the

commercial sector. Therefore, costs are minimized by

eliminating the need for a large staff to maintain and operate

legacy data communications systems.

With this arrangement, DREN provides HPC users with

transparent access to and from other commercial Internet

service providers while maintaining traditional federal peering

relationships with other government agencies. Peering allows

networks to exchange routing information directly, thereby

enabling the most efficient path for transmitting data. DREN

has also established important peering initiatives with new

federal network initiatives such as the Next Generation Internet

(NGI), the Internet 2 ABILENE project, and the National

Science Foundation’s very high performance Backbone

Network Service (vBNS).

DREN provides interoperable ATM and Internet Protocol

(IP) services for digital video, audio, imaging, and data.

Currently network access capabilities range from T3 (45

million bits per second) to OC-12c (622 million bits per

second) with OC-48c (2.4 billion bits per second) available by

2001. DREN fully supports evolving protocols such as

multicasting. State-of-the-art National Security Agency (NSA)

approved security technology encrypts and protects HPC

classified data transmissions across the WAN. As HPCMP

Program Initiatives—

continued
23

PROGRAM OVERVIEW

FIGURE 4. DREN Connections, Shared Resource Centers, and Other Network Access Points
24

2000 HPC Modernization Plan

networking requirements continue to evolve, DREN will

continue to provide the best commercially available support

for applications such as distributed computing, remote

training, distance learning, virtual workshops, and remote

interactive visualization.

Security

HPCMP security is implemented at both the HPC site

level, to protect HPC resources and users, and at the network

level for the HPC community.

The security posture of all HPC sites is verified before they

are authorized to connect to the DREN. While the HPCMP

retains overall security policy and oversight responsibilities,

security certification and accreditation are done at the

individual site level with accreditation delegated to the host

Services. Certifying user sites is also delegated to the host

Services while the HPCMP is responsible for certification of

HPC centers. HPC center certification is performed by a

security test and evaluation (ST&E) team led by the Defense

Information Systems Agency (DISA). The HPCMP also

performs Site Assistance Visits (SAVs) on an annual basis and

periodically sponsors independent National Security Agency

Security Assessments.

HPC centers and user sites that have classified computing

requirements are accredited and certified at the appropriate

classification level. The HPCMP requires that copies of

memorandums of agreement, in accordance with Department

of Defense Directive 5200.28, are on file before allowing

classified communications between user and provider sites.

All communications are accomplished using Type 1 National

Security Agency approved encryption devices.

For unclassified processing, the HPCMP requires strong

identification authentication to ensure that only authorized

users are allowed access to HPC resources. Strong

authentication is implemented using a combination of

Kerberos software, one-time passcodes (currently SecureID

cards), and Secure Shell software. This increases protection

from:

Program Initiatives—

continued
25

PROGRAM OVERVIEW

• disruption of service to authorized users;

• unauthorized disclosure, copying (theft), or alteration of

users’ data;

• unauthorized access to HPC centers’ computers; and

• misuse of HPC centers’ computers.

At the network level, the HPCMP monitors all HPC centers

and is preparing to monitor the DREN network access points

where connections to other networks are made. Monitoring is

done using network intrusion detection systems (NIDS)

running Department of Energy developed Joint Intrusion

Detection (JID) and other software and is performed by the

HPC Computer Emergency Response Team (CERT), which is

currently operated by the DISA Regional Network Operations

and Security Center in Columbus, OH. Access control lists on

network devices are used to further protect DREN users from

known sources of attack. The HPCMP also works closely with

other DoD organizations to ensure that the DREN and all

connected sites meet Service and Office of the Secretary of

Defense security regulations and requirements.

Common High Performance Computing

Software Support Initiative

The Common High Performance Computing Software

Support Initiative (CHSSI) provides efficient, scalable, portable

software codes, algorithms, tools, models and simulations

that run on a variety of HPC platforms and are needed by a

large number of S&T and T&E scientists and engineers.

CHSSI is organized around the ten CTAs and involves

approximately 5,000 scientists and engineers working in close

collaboration across government, industry and academia.

CHSSI teams include algorithm developers, applications

specialists, computational scientists, computer scientists and

engineers, and end users.

Developing software for scalable HPC systems remains a

technically challenging and labor intensive effort. CHSSI is

helping the DoD take advantage of existing and future

computing and communications capabilities by building

software with an emphasis on reusability, scalability, portability

and maintainability. In addition, CHSSI is producing a new

generation of world-class scientists and engineers trained in

Program Initiatives—

continued
26

2000 HPC Modernization Plan

scalable software techniques that will reduce the future costs

of doing business and increase our future defense capabilities.

Table 5 lists the current 35 CHSSI projects and principal

investigators across the CTAs.

Program Initiatives—

continued
27

PROGRAM OVERVIEW

TABLE 5. Common HPC Software Support Initiative Projects and Principal Investigators

(Fiscal Year 2000)

Project Principal

Investigator Organization

Computational Structural Mechanics

A Next Generation Scalable Finite Element

Software to Describe Fracture and Fragmentation

of Solids and Structures

Dr. A.M. Rajendran Army Research Laboratory, Aberdeen

Proving Ground, MD

Scalable Algorithms for Shock Physics Mr. Kent Kimsey Army Research Laboratory, Aberdeen

Proving Ground, MD

Computational Fluid Dynamics

FAST3D (Global Virtual-Cell

Embedding Gridding)

Dr. Jay Boris Naval Research Laboratory, Washington, DC

COBALT (Unstructured Gridding) Mr. Bill Strang Air Force Research Laboratory, WrightPatterson AFB, OH

FEFLO (Unstructured Gridding) Dr. Bill Sandberg Naval Research Laboratory, Washington, DC

OVERSET (Chimera Gridding) Dr. Bob Meakin Army-Aviation Missile Command, Ames

Research Center, Moffett Field, CA

Scalable CFD Simulation System for Complex

Bodies in Relative Motion

Mr. Jere Matty Arnold Engineering Development Center,

Arnold AFB, TN

SUPREM DSMC: Scalable, Parallel, Reacting,

Multidimensional Direct Simulation Monte Carlo

Flow Code

Dr. Ingrid Wysong Air Force Research Laboratory, Propulsion

Directorate, Edwards AFB, CA

Computational Chemistry and Materials Science

Car-Parinello Methods for Solids Dr. David Singh Naval Research Laboratory, Washington, DC

Quantum Chemistry Dr. Jerry Boatz Air Force Research Laboratory, Edwards

AFB, CA

Fast Parallel Total Energy Methods for Multiple

Length Scales

Dr. Dimitrios

Papaconstantopoulos

Naval Research Laboratory, Washington, DC

Molecular Dynamics for Energetic and NonEnergetic MaterialsDr. Leslie Perkins Air Force Research Laboratory, Propulsion

Directorate, Edwards AFB, CA

Classical Molecular Dynamics Dr. Ruth Pachter Air Force Research Laboratory, WrightPatterson AFB, OH

Computational Electromagnetics and Acoustics

Automatic Target Recognition and Scene

Generation

Dr. Jeff Hughes Air Force Research Laboratory, WrightPatterson AFB, OH

A Scalable, Dynamically Adaptive Mesh Software

Package for General Magnetohydrodynamic and

Space Weather Modeling

Dr. Sprio Antiochos Naval Research Laboratory, Washington, DC

MACH3: Portable, Scalable, Parallel-Computing

Magnetohydrodynamics Software for Multimaterials in Complex 3-D Geometry

Dr. Robert Peterkin Air Force Research Laboratory, Directed

Energy Directorate, Kirtland AFB, NM

Software Development for Electromagnetic

Sensing of Surface and Subsurface Targets:

Simulation and Signal Processing

Dr. Ronald Chase Army Research Laboratory, Adelphi, MD
28

2000 HPC Modernization Plan

TABLE 5—Continued. Common HPC Software Support Initiative Projects and Principal Investigators

(Fiscal Year 2000)

Project Principal

Investigator Organization

Climate, Weather, and Ocean Modeling and Simulation

Development of Two Complete Numerical Weather

Prediction Systems for Heterogeneous Scalable

Computing Environments

Dr. Thomas Rosmond Naval Research Laboratory, Monterey, CA

Satellite Radiance Variational Data Assimilation:

Code Migration to Scalable Architectures

Dr. Frank Ruggiero Air Force Research Laboratory, Space

Vehicles Directorate, Hanscom AFB, MA

Scalable Ocean Models with Data Assimilation Dr. Alan Wallcraft Naval Research Laboratory, John C. Stennis

Space Center

Scalable Weather Research and Forecast (WRF)

Model Development

Major Michael Farrar Air Force Weather Agency, Offutt AFB, NE

Signal/Image Processing

Efficient, Maintainable, Portable, and Scalable

HPC Codes for Image Fusion and Signal/Image

Processing

Dr. Richard Linderman Air Force Research Laboratory, Information

Directorate, Rome, NY

Infrared Search and Track Processing for Missile

Surveillance

Mr. Dennis Cottel Space and Naval Warfare Systems Center,

San Diego, CA

Scalable Algorithms for Sonar Beamforming Mr. Bob Bernecky Naval Undersea Warfare Center, Newport, RI

Acoustic Analysis Workbench Using Windows NT Dr. Keith Bromley Space and Naval Warfare Systems Center,

San Diego, CA

Forces Modeling and Simulation/C4I

Simulation and Analysis of Heterogeneous, Large

Scale Communication Networks for Multi-Element

Using Commercial-Off-The-Shelf (COTS) and

Government-Off-The-Shelf (GOTS) Simulators

Dr. Albert Legaspi Space and Naval Warfare Systems Center,

San Diego, CA

High Level Architecture Runtime Infrastructure Mr. Jeff Wallace Space and Naval Warfare Systems Center,

San Diego, CA

Environmental Quality Modeling and Simulation

Structured-Unstructured Modeling Dr. Robert Bernard Engineer Research and Development Center,

Vicksburg, MS

Resolved Transport Algorithm Dr. Mark Dortch Engineer Research and Development Center,

Vicksburg, MS

Scalable Parallel Implementation of Department of

Defense Groundwater Modeling System

Dr. Fred Tracy Engineer Research and Development Center,

Vicksburg, MS

Computational Electronics and Nanoelectronics

Efficient Numerical Solutions to Large Scale

Military Tactical Communications Problems via a

Scalable Time Domain Method (Sc-MRTD)

Dr. Barry Perlman Army Communications-Electronics Command,

Ft. Monmouth, NJ

Integrated Modeling and Test Environments

Simulation and Software Implementation of NonUniformity Correction (NUC) for Multi-Element

Infrared Scene Projector (IRSP) Arrays

Mr. Kenneth Lasueur Redstone Technical Test Center, Redstone

Arsenal, AL

Model-Based Test Data Validation Mr. Joseph Babilon Arnold Engineering Development Center,

Arnold AFB, TN

Real-Time Image-Based Sorting and Classifying

on Multiple Parallel Processors

Dr. David Marlin White Sands Missile Range, NM

High Performance Computing in Concurrent

Engineering, Modeling, and Testing

Dr. Andrew Mark Army Research Laboratory, Aberdeen Proving

Ground, MD
29

HPC Contributions to DoD Mission

Success — 2000

High performance computing enables detailed computerbased design, virtual prototyping, and improved numerical

modeling in order to significantly shorten the acquisition

cycle, reduce costs, and increase the effectiveness of DoD

weapons systems. High-end computational capabilities

impact all aspects of DoD research, development, test, and

evaluation — from basic research in the laboratories to testing

at the test centers. The extent of this impact is accelerating.

When compared with traditional theoretical and experimental

methods, computational modeling and simulation allow the

exploration of more options at significantly reduced cost. The

use of high performance computing, coupled with judiciously

chosen experimental data points, provides a powerful tool for

improving the development process of DoD warfighting and

support systems.

Many major accomplishments have already resulted from

the use of high-end computing in the DoD science and

technology and test and evaluation programs. In 1995,

1996, 1997, and 1998, the High Performance Computing

Modernization Program (HPCMP) published High

Performance Computing Contributions to DoD Mission

Success. Those publications described an extensive array of

success stories resulting from the use of HPCMP assets. The

HPCMP will publish another High Performance Computing

Technical Report Series in 2000. Included here are four

typical examples from the hundreds of projects being

accomplished on HPC provided resources. Applicable

references pertaining to these stories can be found at the back

of this plan.

• Expanding Diagnostic Capabilities in Aerodynamic

Designs

• High Performance Computing Makes Ground Vehicle

Simulations Possible

• High Performance Aircraft, High Performance Solutions

• From the Sea to the Shore: Forecasting Surface Wave

Conditions for the Military Crisis in Kosovo

Introduction Introduction
30

2000 HPC Modernization Plan

As the Department of Defense

implements its “Revolution in

Military Affairs” it is also

conducting a revolution in

business affairs, exploiting

information systems

technology advances that are

having a direct impact on the

range of acquisition lifecycle

activities. High fidelity

modeling and simulation

increasingly supports testing

of advanced weapons

systems, which historically

relied on techniques such as

live testing and wind tunnel

testing. Applying high

performance computing

resources to modeling of

aerodynamic designs

dramatically expands the

diagnostic capabilities

available to the Department’s

scientific and engineering

communities, and enables

them to quickly identify

design weaknesses, at

relatively low cost, without

endangering human life.

The Army’s RAH-66

Comanche helicopter

exhibited a tail-buffet

phenomenon in level forward

flight during flight and in

scale-model wind tunnel

tests. Dr. Robert Meakin of

the U.S. Army

Aeroflightdynamics

Directorate (AMRDEC),

Moffett Field, CA, studied the

effects of the tail-buffet on

the Comanche. He applied a

High Performance Computing

Modernization Program

Common HPC Software

Support Initiative software

application to study the

design of the fuselage and

rotor hub sections of the

aircraft. This application was

run on an Engineering

Research and Development

Center (Vicksburg, MS) IBMSP supercomputer. By

structuring grids in the shape

of the Comanche, Dr. Meakin

developed a numerical model

of the helicopter fuselage and

simulated the airflow around

it to identify the cause of the

problem.

His research showed that the

airflow surrounding the rotor

hub is forced towards the tail

section of the aircraft (see

figures below). The buffeting

adversely affects the control

and handling characteristics

of the aircraft and over time,

and can lead to structural

fatigue of the tail section.

Based on Dr. Meakin’s highresolution simulation

methodology, which

supplemented wind tunnel

and flight test data, design

modifications were made to

the Comanche.

Dr. Meakin’s approach

represents a low-risk and

relatively quick means of

investigating design problems

on aircraft and other vehicles

susceptible to unsteady flow.

Wind-tunnel tests are costly

and take time to schedule and

run. High fidelity simulations

are possible on high

performance computing

systems. These simulations

allow for “what if” design

tradeoff studies that are not

practical to set up and run in

wind tunnels and they allow

many more “simulations” in

the same amount of time.

Today’s supercomputers and

computational software

methods make these types of

simulations possible.

Expanding Diagnostic Capabilities in Aerodynamic Designs Expanding Diagnostic Capabilities in Aerodynamic Designs

R. L. Meakin

 Army Aeroflightdynamics Directorate (AMCOM), Moffet Field, CA

Unsteady flow over

an isolated

Comanche

helicopter fuselage

(top and side

views). Particles

released from the

base of the rotor

hub reveal vortex

shedding and

downstream

convection to the

vehicle tail section.
31

SUCCESS STORIES — 2000

Military ground vehicles play

a pivotal role on the modern

battlefield. Vehicles such as

the BMP-2 armored infantry

fighting vehicle and the M1

Abrams tank are essential to

conducting a land battle.

However, developing military

ground vehicles is time

consuming and costly. Using

high performance computing

resources to conduct

simulations shortens the

design-build-test-modify

cycle. High performance

computing provides the size,

processing speed, and

throughput necessary to run

these simulations. The

Department of Defense

(DoD), in association with

industry, has developed the

means and technology needed

to support simulation

research.

The US Army TankAutomotive Research,

Development and Engineering

Center (TARDEC) Distributed

Center, in Warren, Michigan,

built a ground vehicle model

and hardware/operator-in-theloop simulation capability.

This capability assesses the

performance of a ground

vehicle, evaluates design

modifications, and helps

investigate soldier/machine

interfaces. The figure below

shows the configuration of

the simulation. The TARDEC

Simulation Laboratory is also

developing and integrating

advances in multi-body

dynamics compositions,

vehicle-terrain interaction,

and synthetic environments

to create simulations for

engineering design analysis.

Engineers at the TARDEC

Simulation Laboratory have

successfully integrated a realtime operator-in-the-loop

Bradley tank model using the

in-house Symbolically

Optimized Vehicle Analysis

System multi-body dynamics

method. The TARDEC team

also integrated an Abrams

model using technology

developed through the

National Automotive Center

(NAC) sponsored Automotive

Research Center. The NAC

serves as the Army’s focal

point for the development of

dual-needs/dual-use

automotive technologies and

their application to military

ground vehicles. The

Automotive Research Center,

a collaboration of

government, industry, and

academia, develops simulator

technologies.

High Performance Computing Mak erformance Computing Makes Ground V es Ground Vehicle Simulations ehicle Simulations

Possible ossible

M. Brudnak, P. Nunez, and R. Romano

Tank-Automotive Research, Development and Engineering Center (TARDEC), Warren, MI and

Realtime Technologies, Inc

Conceptual overview of TARDEC’s Operator-in-the-loop distributed simulation
32

2000 HPC Modernization Plan

The TARDEC engineers used

their SGI Power Challenge

Array and the Origin 2000 to

run the Bradley and Abrams

tank models in a distributed

computational environment.

The models provide motion

cues to the Ride Motion

Simulator. When the

simulator is used with the

current Crew Station/Turret

Motion Base Simulator, the

soldier can drive or operate

the vehicle, and respond to

battlefield scenarios run

either locally or networked to

the Defense Simulation

Internet. The TARDEC team

can create an entire synthetic

environment including high

frequency motion cues, highresolution terrain, and

realistic video and audio

responses. These

engineering-high fidelity

models run at real-time

update rates on the SGI

Origin 2000 supercomputer

provided by the DoD High

Performance Computing

Modernization Program.

Using high performance

computing resources, the US

Army includes the soldier

early in the design cycle to

evaluate vehicle designs and

modifications without

investing heavily in the

production of hardware. This

greatly reduces the designbuild-test-modify cycle. It

also significantly increases

readiness by producing

systems that are more

effective in a shorter period

with less cost than was

previously possible.
33

SUCCESS STORIES — 2000

Materials science is a

cornerstone of weapons

systems design within the

Department of Defense. The

challenge has always been,

and continues to be, our

ability to have greater

granularity in the exploration

of new chemicals and

compounds that will form the

basis for national defense in

the 21st century. This is

particularly true in the

development of high

performance aircraft.

In order to achieve maximum

power and fuel efficiency,

aircraft engines must operate

at high temperatures, often

above the melting range of

the engine metal alloy. To

gain maximum power and fuel

efficiency, Thermal Barrier

Coatings (TBCs) must be

used to protect the

underlying metal superalloys

(which are high-strength

complex alloys resistant to

high temperature) from

melting. TBCs consist of a

thin ceramic layer with a

metal alloy bond-coat layer

between the ceramic and the

underlying engine superalloy.

With repeated thermal

cycling, these coatings tend

to crack and chip.

Under contract by the Air

Force, two researchers, E.A.

Jarvis and E.A. Carter, from

the University of California at

Los Angeles are modeling

surfaces and interfaces

present in a TBC system

using high performance

computing resources provided

by the High Performance

Computing Modernization

Program. They were able to

develop a new atomic-level

understanding of the causes

of chipping with increasing

Thermally Grown Oxide (TGO)

thickness. (The TGO is a

“layer” which grows between

the ceramic topcoat and the

metal alloy bond coat that

results from high

temperature oxygen ion

diffusion through the ceramic

topcoat. This TGO region

introduces strain on the

system.) The build up of a

TGO region may cause TBC

failure.

The team used Density

Functional Theory to obtain

properties of the system by

solving for the electron

High Performance Aircraft, High P erformance Aircraft, High Performance Solutions erformance Solutions

E.A. Jarvis and E.A. Carter

University of California at Los Angeles, Los Angeles, CA

Schematic cross-section of a thermal barrier coating
34

2000 HPC Modernization Plan

density. The periodic

boundary conditions allow

simulation of “infinite”

surfaces. The computational

work concentrates on the

valence or “outer” electrons.

The results will help the

Department improve the bond

coat layer by decreasing

thermal expansion mismatch

strain while promoting

adhesion of the TBC to the

engine alloy. This work could

lead to higher performance jet

engines with longer operating

cycles between maintenance

actions. That means more

capable aircraft with lower

maintenance costs.
35

SUCCESS STORIES — 2000

Environmental conditions

directly impact the success of

military missions around the

world. In particular, the Navy

relies heavily on

environmental forecasting

during the conduct of

amphibious landings.

Recently, events in Kosovo

prompted the Navy to rapidly

apply the high performance

computing capabilities of the

Naval Oceanographic Office

(NAVOCEANO) with those of

the Naval Research

Laboratory – Stennis Space

Center (NRL-SSC) to rapidly

coordinate the configuration

of a prototype numerical

model capable of predicting

wave products from the deep

water of the Adriatic Sea to

the shallow waters of coastal

Albania. The timely delivery

of tactically significant

information to the fleet was

possible only with high

performance computing

resources provided by the

HPCMP.

In response to military needs,

NAVOCEANO provided a 72-

hour wave height and

direction forecast at a 5-

minute resolution for the

Adriatic Sea. As the military

situation matured and the

likelihood of an amphibious

landing increased, the

information available from the

5-minute resolution Wave

Model (WAM) was insufficient

for near-shore operations.

Figure 1 shows a typical WAM

product delivered during the

Kosovo crises. Fortunately,

NAVOCEANO had been

developing the Steady State

Spectral Wave Model

(STWAVE) at the NAVO Major

Shared Resource Center

From the Sea to the Shor rom the Sea to the Shore: Forecasting Surface Wave

Conditions for the Military Crisis in K Conditions for the Military Crisis in Kosovo

M. Brooking, J. Dykes, G. Mason, P. Wilz, R. Allard, and R. Jensen

Warfighting Support Center, Naval Oceanographic Office (NAVO), Planning Systems Inc., and Naval

Research Laboratory (NRL), at Stennis Space Center, MS, and Engineer Research and Development

Center (ERDC), Vicks burg, MS

Figure 1. WAM

product delivered

during the Kosovo

Crisis

(MSRC) and had the program

configured to support final

operational evaluation. The

operational need in Kosovo

led the Navy to rapidly

commission the STWAVE as a

predictive model before it had

completed final Operational

Evaluation.

STWAVE provided wave

prediction information in the

shallow water zone between

2 to 20 meters that allowed

for detailed planning of

amphibious operations.

Figure 2 shows a typical

STWAVE product delivered

during the Kosovo crises.

Because of the HPCMP high

performance computational

resources available at the

NAVO MSRC, the waveforecast results were

provided to the fleet in a

timely manner.

Figure 2. STWAVE product delivered during the

Kosovo Crisis

37

Major Shared Resource Centers

The High Performance Computing Modernization

Program (HPCMP) completed key acquisitions to establish

four major shared resource centers (MSRCs) in fiscal year

1996. Integration contracts provide the centers’ hardware,

software, maintenance, operations, and support staff. Each

contract includes a base computational capability (referred to

as Performance Level I) with two options to upgrade the

capability as funds become available (referred to as

Performance Levels II and III). Since the first high

performance computing systems were fielded by the program

in 1993, the total peak computational high performance

computing capability at the MSRCs increased from 30

gigaflops to just over 3,481 gigaflops at the end of fiscal year

1999. During fiscal year 2000, the program will continue to

modernize the centers to sustain the Defense Department’s

world-class high performance computing infrastructure.

Approximately 2,800 peak gigaflops are expected to be added

during fiscal year 2000, for a total of 6.2 teraflops. (It should

be noted that “Peak Teraflops” represent a simple measure of

growth. It, however, does not consider architecture specific

features, nor does it provide a measure of actual performance

delivered). Table 6 shows the fiscal year 2000 high

performance computing systems’ capability planned for the

MSRCs.

Major Shared Resource Centers, Fiscal Year 2000

In fiscal year 2000, the program will sustain the centers’

high performance computing capabilities, completing the

Performance Level III upgrade that began in fiscal year 1999.

When concluded, the upgrade will have increased each

center’s capability at least tenfold over that in fiscal year 1996.

The program will continue to provide improved operating

environments, supporting tools, and software in the users’ ten

computational technology areas. The multi-year process of

building the centers’ wide area computing environment will

continue as well, with an emphasis on peripheral and archival

Program Accomplishments and Future

Plans

High Performance

Computing Centers
38

2000 HPC Modernization Plan

mass storage capabilities. The program will continue to

provide user training and interaction with other government

agencies, industry, and academic centers so that defense

scientists and engineers can take advantage of the latest high

performance computing expertise, capabilities, standards, and

infrastructure.

High Performance

Computing Centers–

continued

TABLE 6. Major Shared Resource Centers High Performance Computing Capability

(Fiscal Year 2000)

Major Shared Resource Center Processors Total Memory

(gigabytes)

Total Capability

(gigaflops)

Aeronautical Systems Center (ASC), WrightPatterson AFB, OH

IBM SP/SMP Power 3

SGI Origin 2000/195

SGI Origin 2000/195

IBM SP/135

Compaq ES-4 0, EV-6

 264

128

 384

 256

 64

 264

 128

 192

 264

 64

 211

51

 154

 126

 64

Army Research Laboratory (ARL), Aberdeen

Proving Ground, MD

Cray SV1

SGI Origin 2000/300

SGI Origin 2000/250

SGI Origin 2000/250 (Classified)

SGI Origin 2000/250 (Classified)

SGI Origin 2000/300 (Classified)

SGI Origin 2000/195 (Classified)

Sun E10000

Sun E10000 (Classified)

Cray SV1 (Classified)

32

 256

 32

 128

 64

 32

 32

 128

128

 16

28

 256

 32

 112

 32

 28

 32

 128

 128

 32

38

 153

 16

 64

32

 13

 19

102

 102

 19

Engineer Research and Development Center

(ERDC), Vicksburg, MS

Cray T3E-1200

IBM SMP Power 3

IBM SP/135

IBM SP/160

SGI Origin 2000/195

 544

 512

 256

 126

 128

 136

 256

 256

 63

 64

 653

 455

 138

 81

 50

Naval Oceanographic Office (NAVO), Stennis

Space Center, MS

Cray T3E-900

SGI Origin 2000/195+250

Cray SV1-2

Cray T90

Cray J916SE (Classified)

SGI Origin 2000/250 (Classified)

Sun E10000 32-way SMP

1,088

 256

 32

 16

 16

 8

 32

 400

 128

 32

 8

 4

 4

32

 979

 114

38

 29

 3

 4

 26

Note: Current as of July 2000.
39

PROGRAM ACCOMPLISHMENTS AND FUTURE PLANS

Major Shared Resource Centers, Fiscal Year 2001

The program will continue the upgrade of major shared

resource center capabilities in response to increasing user

requirements. Additional high performance computing

systems, mass storage, and scientific visualization capabilities

will be added to the centers. The multi-year process of

building the major shared resource centers’ wide area

computing environment will continue, with an emphasis on

wide-area sharing of storage resources. Programming

environment and training activities will continue to emphasize

distance learning and distributed training, ensuring that

defense scientists and engineers can take advantage of

expanded high performance computing capabilities as they

become available.

Distributed Centers

Since fiscal year 1993, the program has established and

upgraded the computational capabilities of distributed centers

(DCs). Distributed centers are selected annually for a onetime investment in high performance computing equipment.

Typically, the equipment is no longer considered high

performance computing class after three to four years. If a

DC is not selected for an upgrade in computational capability

during this time period, the center may be retired as an

HPCMP DC under the program’s sunset policy. In February

1999, four existing and two new DCs were selected to receive

a total of $18 million for the procurement of high

performance computing systems as part of the fiscal year

1999 distributed center selection process. In April 1999, the

office conducted a post-deployment evaluation and

assessment of DCs selected in fiscal year 1997 and reviewed

their progress towards meeting stated goals and milestones.

Distributed Centers, Fiscal Year 2000

Table 7 lists the program’s fiscal year 2000 DCs. Table 8

shows the fiscal year 2000 high performance computing

systems at the DCs. The total peak gigaflop computational

capability at the distributed centers is expected to reach just

over 4,500 gigaflops at the end of fiscal year 2000.

In December 1999, six organizations were selected to

receive a total investment of $19M for the procurement of

high performance computing systems. The six centers

receiving awards were: the Redstone Technical Test Center

(Huntsville, AL); the Naval Air Warfare Center, Aircraft Division

High Performance

Computing Centers–

continued
40

2000 HPC Modernization Plan

(Patuxent River, MD); the Naval Research Laboratory

(Washington, DC); the Air Force Flight Test Center (Edwards

AFB, CA); the Air Force Research Laboratory, Sensors

Directorate (Wright-Patterson AFB, OH); and the Maui High

Performance Computing Center (Kihei, HI). These centers will

acquire high performance computing systems during fiscal

year 2000.

The Redstone Technical Test Center will use its high

performance computing assets to expand and refine the

capability of its Dynamic Infrared Scene Projector. The Naval

Air Warfare Center, Aircraft Division will expand the

computational power of its modeling and simulation capability

High Performance

Computing Centers–

continued

Center Location Most Recent

Upgrade

Air Armament Center (AAC) Eglin AFB, FL 1999

Air Force Flight Test Center (AFFTC) Edwards AFB, CA 2000

Air Force Research Laboratory Information Directorate

(AFRL/IF) Rome, NY 1998

Air Force Research Laboratory Sensors Directorate

(AFRL/SN) Wright-Patterson AFB, OH 2000

Arctic Region Supercomputing Center (ARSC) Fairbanks, AK 1999

Army High Performance Computing Research Center

(AHPCRC) Minneapolis, MN 2000

Arnold Engineering Development Center (AEDC) Arnold AFB, TN 1999

Joint National Test Facility (JNTF) Schriever AFB, CO 1999

Maui High Performance Computing Center (MHPCC) Kihei, HI 2000

Naval Air Warfare Center Aircraft Division (NAWCAD) Patuxent River, MD 2000

Naval Air Warfare Center Weapons Division (NAWCWD) China Lake, CA 1999

Naval Research Laboratory (NRL-DC) Washington, DC 2000

Redstone Technical Test Center (RTTC) Huntsville, AL 2000

Space and Missile Defense Command (SMDC) Huntsville, AL 1996

Space and Naval Warfare Systems Center (SSC-SD) San Diego, CA 1999

Tank Automotive Research, Development and Engineering

Center (TARDEC) Warren, MI 1999

White Sands Missile Range (WSMR) White Sands Missile Range, NM 1998

TABLE 7. Distributed Centers
41

PROGRAM ACCOMPLISHMENTS AND FUTURE PLANS

TABLE 8. Distributed Centers High Performance Computing Capability

(Fiscal Year 2000)

Distributed Centers Processors Total Memory

(gigabytes)

Total Capability

(gigaflops)

Air Armament Center (AAC), Eglin AFB, FL

SGI Origin 2000

SGI Onyx (Classified)

 20

 32

 20

 8

 10

 13

Air Force Research Laboratory Information

Directorate (AFRL/IF), Rome, NY

Sky Excalibur 384 25 256

Arctic Region Supercomputing Center (ARSC),

Fairbanks, AK

Cray J932

Cray T3E

 12

 272

 8

68

 2

230

Army High Performance Computing Research

Center (AHPCRC), Minneapolis, MN

Cray T3E 1,296 259 1,303

Arnold Engineering Development Center

(AEDC), Arnold AFB, TN

HP V2500

HP V2500 (Classified)

SGI Origin 2000

64

64

 64

64

 32

 32

113

113

38

Joint National Test Facility (JNTF), Schriever AFB,

Colorado Springs, CO

SGI Onyx 2 (Classified)

SGI Origin 2000 (Classified)

88

 64

 44

 32

 83

 64

Maui High Performance Computing Center

(MHPCC), Kihei, HI

IBM P3/SMP (Classified)

IBM SP3/SMP

IBM P2SC

IBM SP2 (Classified)

IBM P2SC (Classified)

IBM P2SC

IBM SP2 (Classified)

IBM P3/SMP

IBM P2SC

IBM P2/SMP

IBM P3/SMP

IBM P3/SMP

IBM P2

 320

 160

 160

 128

32

 32

 48

 16

19

64

 16

 8

 16

 56

80

80

 40

 8

 20

 13

2

4

8

 8

 4

 2

 480

 142

 102

34

 15

 21

 13

 14

 9

29

 14

 7

 4

Note: Current as of July 2000.
42

2000 HPC Modernization Plan

TABLE 8—Continued. Distributed Centers High Performance Computing Capability

(Fiscal Year 2000)

Distributed Centers Processors Total Memory

(gigabytes)

Total Capability

(gigaflops)

Naval Air Warfare Center Aircraft Division

(NAWCAD), Patuxent River, MD

SGI Power Challenge (Classified)

SGI Onyx 2 (Classified)

 40

64

 12

24

 16

25

Naval Air Warfare Center Weapons Division

(NAWCWD), China Lake, CA

SGI Onyx 2 (Classified) 64 24 25

Naval Research Laboratory (NRL), Washington,

DC

HP Exemplar SPP-2000

SGI Origin 2000

Sun Ultra

 80

 128

 140

 20

 128

 31

 57

 77

 63

Redstone Technical Test Center (RTTC),

Redstone Arsenal, AL

SGI Origin 2000

SGI Origin 2000 (Classified)

 32

 32

 7

 7

 13

 13

Space and Missile Defense Command (SMDC),

Huntsville, AL

SGI Origin 2000

SGI Origin 2000

SGI Origin 2000

SGI Origin 2000

SGI Origin 2000

SGI Origin 2000 (Classified)

SGI Origin 2000 (Classified)

 128

 64

 64

 32

 16

 8

 8

 32

 22

 16

 8

 4

 2

 2

 52

 26

 26

 13

 6

 3

 3

Space and Naval Warfare Systems Center (SSCSD), San Diego, CA

HP V2500

HP V2500 (Classified)

 16

 16

24

24

28

 28

Tank-Automotive Research, Development and

Engineering Center (TARDEC), Warren, MI

SGI Power Challenge

SGI Onyx 2

 60

 32

 12

 31

 22

 16

White Sands Missile Range (WSMR), NM

SGI Origin 2000 128 48 75

Note: Current as of July 2000.
43

PROGRAM ACCOMPLISHMENTS AND FUTURE PLANS

at the site’s Air Combat Environment Test and Evaluation

Facility. The Naval Research Laboratory continues to serve as

the DoD high performance computing community’s

pathfinder, exploring, maturing, and stabilizing “leading edge”

technological advancements prior to their introduction to the

wider DoD high performance computing user community.

The Air Force Flight Test Center intends to use high

performance computing to develop high fidelity models to

simulate electronic warfare battlefields. The Air Force

Research Laboratory Sensors Directorate’s research in the

areas of signal and image processing (SIP) and automatic

target recognition (ATR) will be greatly enhanced with a stateof-the-art rugged and mobile high performance computing

system. Finally, the Maui High Performance Computing

Center will update and expand high performance computing

assets for users requiring classified computing resources.

Also, in Fiscal Year 2000, Congress provided additional

funding to the program to significantly expand the high

performance computing capability at the Army High

Performance Computing Research Center Distributed Center.

The AHPCRC system configuration shown at Table 8 is a

result of that funding. The system increased by almost 1, 000

processors and almost 1,000 gigaflops - making the center's

Cray T3E, one of the largest in the world.

Distributed Centers, Fiscal Year 2001

There currently is no call for proposals planned in fiscal

year 2001 due to funding constraints.

Defense Research and Engineering Network

During 1998, the Defense Research and Engineering

Network (DREN) completed its transition from an interim,

point-to-point T-3 (45 million bits per second) backbone to an

OC-3c (155 million bits per second) virtual private network

over a public commercial asynchronous transfer mode (ATM)

communications infrastructure. DREN has added another

major Network Access Point (NAP) at Chicago, IL in order to

exchange wide area network traffic more efficiently with other

congressionally funded initiatives such as Next Generation

Internet (NGI) and the Internet 2 ABILENE project. DREN

also participates in Next Generation Internet EXchanges

(NGIXs) and in other Metropolitan Area Exchanges (MAEs).

High Performance

Computing Centers–

continued

Networking
44

2000 HPC Modernization Plan

DREN is a major participant in enhancing wide area

networking between Alaska and Hawaii and the continental

United States.

The HPCMP co-chairs the Joint Engineering Taskforce

(JET). The JET, which replaced the Federal Networking

Council, is chartered by the National Science and Technology

Council’s Subcommittee on Computing Information and

Communications. The JET acts as a forum for network

engineering collaborations between federal agencies to meet

their research, education, and operational mission goals. The

JET also works to bridge the gap between advanced

networking technologies being developed and the eventual

acquisition of mature versions of these technologies from the

commercial sector for the Federal market.

DREN, Fiscal Year 2000

During fiscal year 2000, network connectivity will be

sustained, and options to increase the bandwidth for the

major shared resource centers and other selected sites will be

executed. This increased bandwidth will facilitate participation

in distributed computing environment testbeds. The DREN

plans to continue adding sites to its high-speed classified

virtual private network and will work towards establishing

mutual peering with other classified networks. In

collaboration with the Defense Information Systems Agency

(DISA), acquisition planning continues to ensure that DREN

services are available in the future. The program will continue

ongoing efforts to enhance the total bandwidth capability.

DREN, Fiscal Year 2001

During fiscal year 2001, network connectivity will be

sustained and additional options for increased bandwidth will

again be executed. DREN will continue to support the testbed

environments and facilitate the distributed computing

environments across the program’s MSRCs and selected DCs.

The long term objective of this effort is to optimize sharing of

valuable Department of Defense (DoD) HPC resources across

centers by supporting dynamic, real-time, ubiquitous

computing environments; providing better protection during

times of disaster or extended outages; and by eliminating

single points of failure. The very high-speed DREN will be the

key to success, since individual centers will need immediate,

bandwidth-intensive connectivity to move increasingly larger

data sets.

Networking–continued
45

PROGRAM ACCOMPLISHMENTS AND FUTURE PLANS

Security

Security at HPCMP sites and on the DREN continues to be

improved. During 1999, the HPCMP implemented the

Uniform Use-Access Policy for Unclassified DoD HPCMP

Resources to ensure that security requirements for obtaining

accounts at all HPC centers are adequate to protect HPC

resources. The program also instituted independent National

Security Agency Security Assessments at MSRCs and selected

DCs. These assessments provide an independent review and

verification of the sites’ security postures. Centralized network

intrusion detection systems (NIDS) coverage has been

expanded to all HPC centers and is being implemented at

access points where the DREN meets the Internet and other

networks. The HPC Computer Emergency Response Team

(CERT) is in place to support all HPCMP users. The HPCMP

entered into a joint project with the Department of Energy

(DOE) to improve the DOE-owned Joint Intrusion Detection

(JID) software used on our NIDS. HPCMP researchers at

Army Research Laboratory (ARL) modified the JID software so

that it can monitor high-speed ATM network traffic. The code

improvement increases the number of locations where

intrusion detection can be performed and allows the program

to monitor all DREN traffic. The DOE will distribute the

improved code to agencies across the government. The

HPCMP has also increased the level of security filtering done

across the DREN. In concert with the Services and DISA,

access control lists (ACLs) on devices at network access

points have been expanded to provide increased protection

from known sources of attack. HPCMP ACLs are coordinated

with the Services and DISA lists to ensure coverage that meets

all DoD requirements.

Security, Fiscal Year 2000

Our defense-in-depth strategy is protecting the program in

the face of increasing numbers of cyber space attacks. The

HPCMP has concentrated its security efforts, including

certification and accreditation (C&A), at the edges of the

network on the host systems. During fiscal year 2000, the

HPCMP will add to its security posture by rewriting its system

security authorization agreements and by performing a C&A

of the DREN. To increase the speed with which attacks on

HPCMP resources are identified, the HPCMP will evaluate and

prepare plans to field real-time monitoring solutions, which

augment existing capabilities. The joint project with DOE to

Networking–continued
46

2000 HPC Modernization Plan

improve the JID software will also continue in fiscal year 2000.

HPCMP researchers and contractors will improve the code so

that it effectively uses multiprocessor computers and is

capable of monitoring at speeds up to OC-12.

Several Common High Performance Computing Software

Support Initiative (CHSSI) teams released alpha and beta

software products in fiscal years 1997 and 1998. Evaluations

of these products show a significant improvement in capability

to efficiently exploit HPC scalable systems in support of a large

number of DoD user projects, including DoD Challenge

Projects. Several of these products are used in important

defense applications such as armor-anti-armor simulations,

blast-structure interactions, antenna design, and radar cross

section calculations for aircraft and ground vehicles. Many

products incorporate state-of-the-art user interface and

graphic capabilities that increase usability and improve

analyses by users. The program office promulgated a call for

proposals to DoD laboratories and test centers in fiscal year

1999. Numerous proposals were solicited and evaluated.

Sixteen new CHSSI projects were selected for initiation at the

start of fiscal year 2000.

Software Support, Fiscal Year 2000

The initiative will continue to develop, test, and release

software in fiscal year 2000 to take advantage of advances in

high-speed computing and communications technology,

especially in distributed computing, wide-area computing

environments, massively parallel high performance computing

architectures, and remote desktop collaboration. During

fiscal year 1999, the initiative increased its emphasis on strong

software engineering, reusability, and maintainability.

Additional projects will be initiated as ongoing projects are

completed.

Software Support, Fiscal Year 2001

Software development, testing, and deployment will

continue in fiscal year 2001. Many of the original CHSSI

projects will reach completion. The HPCMO will solicit and

evaluate proposals for new projects. These new projects will

allow the DoD to develop and transition additional key DoD

applications software to work effectively on new scalable

architectures.

Networking–continued

Common High

Performance

Computing Software

Support Initiative
47

PROGRAM ACCOMPLISHMENTS AND FUTURE PLANS

Approximately 25 percent of the program’s total

computational resources are allocated annually to highpriority Service and Agency projects with very large

computational requirements. These mission-critical,

computationally intensive DoD Challenge Projects are

approved by the Deputy Undersecretary of Defense (Science

and Technology) (DUSD(S&T)). Proposals are evaluated by

the DoD Challenge Projects Allocation Board with

representatives from each Service, selected Defense Agencies,

and the broader HPC community. The Board selected 34

projects for implementation in fiscal year 2000. Many of the

projects use multiple hardware platforms and in some cases,

multiple HPC centers. Computationally, these projects were

allocated an average of 46 gigaflops-years for the period

October 1999 through September 2000. Table 9 lists the

fiscal year 2000 DoD HPC Challenge Projects. A new set of

DoD HPC Challenge Projects will be selected during fiscal year

2000 for implementation in fiscal year 2001, continuing the

annual selection and evaluation process.

During fiscal year 1998, the HPCMP performed the Initial

Operational Test and Evaluation of its systems in compliance

with federal statutes and Defense Department regulations.

The Joint Interoperability Test Command (JITC) served as the

independent test activity and performed the initial operational

evaluation of the program. The command witnessed the

evaluation of two MSRCs, two DCs, and two CHSSI code

development projects as well as site acceptance testing of the

DREN.

In January 1999, the JITC issued an Independent

Evaluation Report stating the HPC centers and network are

operationally effective and operationally suitable. The report

stated,

“The HPCMP is operationally effective and suitable when

operated by its intended users in its operational

environment. Users generally agreed the HPCMP has

revolutionized their ability to conduct scientific and

engineering research. [emphasis added] They indicated

that problems, while needing attention, are minor to the

HPCMP’s overall contribution to their research efforts.”

DoD High

Performance

Computing Challenge

Projects

Program Oversight

and Review
48

2000 HPC Modernization Plan

TABLE 9. DoD High Performance Computing Challenge Projects (Fiscal Year 2000)

Project Sponsoring Organization Systems

Computational Structural Mechanics

Damage Simulations in Hard and Deeply Buried Targets

Due to Internal Blast and Shock Loading

Engineer Research and Development

Center

IBM SP

Cray T3E

SGI Origin 2000

Development of Standards for Stand-Off Distance and

Blast Walls for Force Protection

Engineer Research and Development

Center

IBM SP

Cray T3E

SGI Origin 2000

Modeling Complex Projectile Target Interactions Army Research Laboratory SGI Origin 2000

Cray J90

Sun E10000

Computational Fluid Dynamics

3-D Computational Fluid Dynamics (CFD) Modeling of the

Chemical Oxygen-Iodine Laser (COIL)

Air Force Research Laboratory SGI Origin 2000

Cray T3E

Analysis of Infrared Radiance Effects from DivertJet

Exhaust Flow Over the Theater High Altitude Area Defense

(THAAD) Seeker Window

Space and Missile Defense

Command

Cray T90

SGI Origin 2000

Applied CFD in Support of Aircraft-Store Compatibility

and Weapons Integration

Air Armament Center SGI Origin 2000

Automatic Aerodynamic Design for Complete Aircraft

Configurations Using an Adjoint Based Multiblock Method

Air Force Office of Scientific

Research

SGI Origin 2000

Contaminant Transport and Source Simulations for Urban

and Environmental Hazard Assessment

Naval Research Laboratory Convex Exemplar

IBM SP

SPP-2000

SGI Origin 2000

High-Resolution Adaptive Mesh Refinement (AMR)

Simulation of Confined Explosions

Defense Threat Reduction Agency Cray T3E

IBM SMP

IBM SP

Hybrid Particle Simulations of High Altitude Nuclear

Explosions in 3D

Defense Threat Reduction Agency SGI Origin 2000

Integrated High Performance Turbine Engine Technology

(IHPTET) Combustor Design Studies Using Large-Eddy

Simulation (LES)

Air Force Research Laboratory Cray T3E

SGI Origin 2000

Interdisciplinary Advanced Aerospace Vehicle Simulation Air Force Research Laboratory IBM SP

Cray T3E

SGI Origin 2000

Numerical Modeling of Wake Turbulence for Naval

Applications: Vortex Dynamics and Late-Wake Turbulence

in Stratification and Shear

Office of Naval Research Cray T3E

SGI Origin 2000

Parallel Simulations of Flow-Structure Interactions Office of Naval Research IBM SP

Cray T3E

Parallel Simulations of Reacting Turbulent Two-Phase

Flows

Army Research Office SGI Origin 2000

Cray T3E

Parallel Simulations of Weapons/Target Interactions Using

a Coupled CFD/Computational Structural Dynamics (CSD)

Methodology

Defense Threat Reduction Agency Cray T90

IBM SMP

SGI Origin 2000

Time Domain Computational Ship Hydrodynamics Office of Naval Research IBM SP

SGI Origin 2000

Cray T3E

Cray T90
49

PROGRAM ACCOMPLISHMENTS AND FUTURE PLANS

TABLE 9—Continued. DoD High Performance Computing Challenge Projects (Fiscal Year 2000)

Project Sponsoring Organization Systems

Computational Fluid Dynamics (Cont.)

Unsteady Aerodynamics of Advanced Guided Munitions Army Research Laboratory SGI Origin 2000

IBM SP

Unsteady Aerodynamics of Aircraft Maneuvering at High

Angles of Attack

Air Force Research Laboratory IBM SP

Cray T3E

Unsteady Hydrodynamics of the Maneuvering Submarine Office of Naval Research SGI Origin 2000

Cray T3E

Computational Chemistry and Materials Science

Computationally Assisted Development of High

Temperature Structural Materials

Air Force Research Laboratory IBM SP

Cray T3E

SGI Origin 2000

Computer Design of Materials from First Principles

Theory: A Supercomputer-Based Laboratory

Naval Research Laboratory IBM SP

IBM SMP

Multiscale Simulation of Nanotubes and Quantum

Structures

Office of Naval Research Cray T3E

New Materials Design Air Force Research Laboratory

Air Force Office of Scientific

Research

Cray T90

SGI Origin 2000

IBM SP

Theoretical Investigation of Gun Tube Erosion Related

Requirement

Army Research Laboratory SGI PCA

SGI Origin 2000

Computational Electromagnetics and Acoustics

Airborne Laser Challenge Project Air Force Research Laboratory Cray T3E

IBM SP

Evaluation and Design of Advanced Navy Topside

Communications Systems

Space and Naval Warfare Systems

Center, San Diego

IBM SP

Virtual Prototyping of Radio Frequency (RF) Weapons Air Force Research Laboratory Cray T90

IBM SP

SGI Origin 2000

Climate, Weather, and Ocean Modeling and Simulation

Coupled Environmental Model Prediction (CEMP) Naval Postgraduate School

Naval Research Laboratory

Cray T3E

SGI Origin 2000

Data Assimilation in High Resolution Numerical

Simulations of the Ocean Circulation

Office of Naval Research Cray T3E

Global and Basin Scale Ocean Modeling and Prediction Naval Research Laboratory Cray T3E

Signal/Image Processing

Automatic Target Recognition Performance Evaluation Air Force Research Laboratory SGI Origin 2000

Environmental Quality Modeling and Simulation

Quantification of the Impacts of Subsurface Heterogeneity

on Military Site Cleanup

Engineer Research and Development

Center

IBM SP

Cray T3E

Computational Electronics and Nanoelectronics

Atomistic Simulation of Micro-Electromechanical (MEMS)

Devices via the Coupling of Length Scales

Naval Research Laboratory IBM SP
50

2000 HPC Modernization Plan

Acquisition

 The results of the evaluation represent a significant step

for the program in the formal oversight process. The

Independent Evaluation Report for CHSSI projects was also

favorable. The annual test report to Congress by the Director,

Operational Test and Evaluation, contained no significant

findings about the program.

Following the test period, acquisition oversight of the

program was formally delegated from the Assistant Secretary

of Defense (Command, Control, Communications and

Intelligence) to the Deputy Undersecretary of Defense

(Science and Technology).

In December 1998, the Under Secretary of Defense for

Acquisition, Technology, and Logistics USD(AT&L) formed an

Integrated Product Team (IPT) under the leadership of the

DUSD(S&T). The IPT, after a year of review, reaffirmed the

critical impact high performance computing is having on the

DoD. Specifically, they concluded HPC is a critical enabling

tool that decreases the time to solution, enables new and

innovative approches, and allows the solution of highly

complex, previously unsolvable problems. They further noted

that the current HPCMP not only leverages common

resources for both the science and technolgy (S&T) and test

and evaluation (T&E) communities, it also promotes the

transition of technology from S&T to T&E.

During fiscal year 2000, the program will continue an

aggressive test program to ensure defense scientists and

engineers receive high performance computing assets that are

operationally effective and suitable for their work efforts. In

April 1999, the HPCMP carried out a post-deployment

evaluation of distributed centers selected for implementation

in fiscal year 1997. Representatives of the program office will

witness testing at selected MSRCs, all DCs selected for

implementation in fiscal year 2000, and beta tests under the

CHSSI. In March 2000, the program assessed the efficiency

and effectiveness of all program initiatives.

Accomplishments

We are entering a new round of acquisitions in the

program office. This gives us the opportunity to examine

what worked, what needs improvement, and what the current

DoD best practices for the program are as a whole. All of the

Program Oversight

and Review—

continued
51

PROGRAM ACCOMPLISHMENTS AND FUTURE PLANS

program elements that expire in the near term are developing

new strategies. These strategies include extending current

efforts through contract modifications, using existing contract

vehicles, and planning for new procurements. IPTs will assist

in the planning and integration of long-term program

initiatives and acquisitions.

Future Plans

Our future plans include developing strategies that make

information technology contracting more successful and not

just more streamlined. We are actively looking at

performance-based contracting, cost as an independent

variable, award-term contracting, and share-in-savings

concepts.

The HPCMP is projected to spend $1.24 billion for fiscal

years 2000–2005. Table 10 shows the funding profile by year

and major spending category, including actual budgets for

fiscal years 1997–1999 and the six-year projections from the

President’s fiscal year 2000 budget. Within the overall

program, the funding level currently supports only a portion of

the validated high performance computing requirements (e.g.,

in fiscal year 1999, the program provided approximately 30

percent of the computing capability needed by the user

community). Although there have been a number of budget

adjustments since the 1994 modernization plan, most of the

reductions for fiscal years 2000–2003 are the result of a single

$269 million reduction sustained by the program as part of

the final resolution of the Defense Program Objective

Memorandum for fiscal years 1997 through 2001.

The increase in fiscal year 2000 procurement from last

year’s modernization plan was provided by Congress in the

fiscal year 2000 Department of Defense Appropriations Act.

As a result, the program anticipates being able to support 50–

60% of the documented requirement in fiscal year 2000.

The modest budget increase in fiscal year 2001 Research,

Development, Test, and Evaluation funding from last year’s

plan supports sustainment funding for the Arctic Region

Supercomputing Center (Fairbanks, AK) and the Maui High

Performance Computing Center (Kihei, HI).

Acquisition—

continued

Program Funding
52

2000 HPC Modernization Plan

Comparison of the current budget profiles with earlier

budget projections, industry briefings, and acquisition

documents indicates that over 75 percent of the cumulative

reductions referenced above—an average of almost $100

million per year beginning in fiscal year 2000—will be

sustained by the program’s acquisition account as a direct

result of the $269 million reduction described earlier. As

Program Funding—

continued

TABLE 10. Department of Defense High Performance Computing Modernization Program Funding

(Fiscal Years 1997–2005)

stated in previous modernization plans, the lower budget is

sufficient to exercise the base capability as currently

programmed for Performance Level III of the contracts for the

MSRCs, but severely limits future capital investments.

A significant fraction of the projected user requirements

can not be adequately addressed by the current funding

profile. Therefore, as part of its strategic planning efforts, the

HPCMP staff continues to examine options to ensure that the

highest priority warfighter needs are met in light of these

shortfalls. An extensive HPC resource allocation process

ensures the program’s resources are allocated by the Services

and Defense Agencies to the highest priority projects of the

Defense Department.

1997 1998 1999 2000 2001 2002 2003 2004 2005

Procurement (Capital

Investments) 122.7 87.1 91.4 95.8 40.0 50.4 49.4 50.3 51.9

Research. Development, Test

and Evaluation

 Networking (Includes Security) 16.0 21.7 17.0 29.3 32.7 32.9 33.1 33.7 34.6

 Common High Performance

 Computing Software Support

 Initiative

21.1 20.9 23.6 21.6 22.3 20.7 20.7 21.8 22.3

 Major Shared Resource Centers 52.8 72.1 87.6 87.2 88.3 83.6 88.4 89.5 90.9

 Distributed Centers 28.5 24.3 23.4 26.2 20.7 0.8 0.8 0.8 0.8

PROGRAM TOTAL 241.1 226.1 243.0 260.1 204.0 188.4 192.4 196.1 200.5
53

PROGRAM ACCOMPLISHMENTS AND FUTURE PLANS

Other significant results of the out-year budget reductions

include scaling back the growth rate for network capability

and phasing out most sustainment funding for the DCs.

These reductions will have the greatest impact on new or just

emerging applications requiring either real-time or high-end

interactive support at sites remote to MSRCs. The continued

stable funding of the CHSSI and the MSRC sustainment

budget lines reflect the importance that the program places

on software, integration, and related shared services to its

HPC user community. These investments will continue to

guarantee that the program’s focus remains on enabling users

to address warfighter needs, not only by maintaining the

support infrastructure, but also by providing effective

mechanisms for continued leverage of both national and

industrial investments in HPC-based productivity.

55

References

Expanding Diagnostic Capabilities in Aerodynamic Designs

Meakin, R., “Unsteady Aerodynamic Simulation of Static and Moving Bodies Using Scalable

Computers”, AIAA-99-3302-CP, pp. 469–483, 14th AIAA Computational Fluid Dynamics

Conference, Norfolk, VA, June 1999.

Meakin, R., “On Adaptive Refinement and Overset Structured Grids”, AIAA-97-1858-CP, pp. 236–

249, 13th AIAA Computational Fluid Dynamics Conference, Snowmass, CO, June 1997.

High Performance Computing Makes Ground Vehicle Simulations Possible

David Lamb, Jim Overholt, Patrick Nunez, Greg Hudas, and Mark Brudnak, “Vehicle System

Modeling in SOVAS” in Proceedings of the 21st Army Science Conference, 1998, Norfolk, VA, June

16, 1998

57

Acronyms

6-DOF Six Degree-of-Freedom

AAC Air Armament Center

ACLs Access Control Lists

AEDC Arnold Engineering Development Center

AFB Air Force Base

AFIT Air Force Institute of Technology

AFRL/IF Air Force Research Laboratory Information Directorate

AFRL/SN Air Force Research Laboratory Sensors Directorate

AHPCRC Army High Performance Computing Research Center

AMCOM US Army Aeroflightdynamics Directorate

ANSI American National Standards Institute

ARC Automotive Research Center

ARL Army Research Laboratory

ARSC Arctic Region Supercomputing Center

ASC Aeronautical Systems Center

AT&T American Telephone and Telegraph

ATM Asynchronous Transfer Mode

CCM Computational Chemistry and Materials Science

CEA Computational Electromagnetics and Acoustics

CEMP Coupled Environmental Model Prediction

CEN Computational Electronics and Nanoelectronics

CERT Computer Emergency Response Team

CFD Computational Fluid Dynamics

CHSSI Common High Performance Computing Software Support Initiative

COIL Chemical Oxygen-Iodine Laser

COTS Commercial-off-the-Shelf

CSM Computational Structural Mechanics

CTA Computational Technology Area

CWO Climate/Weather/Ocean Modeling and Simulation

DCs Distributed Centers

DFT Density Functional Theory

DISA Defense Information Systems Agency

DoD Department of Defense

DREN Defense Research and Engineering Network
58

2000 HPC Modernixation Plan

DT&E Development Test and Evaluation

DTC Developmental Test Command

DTOs Defense Technology Objectives

EOSFEL Electro-Optical Sensor Flight Evaluation Laboratory

EQM Environmental Quality Modeling and Simulation

ERDC Engineer Research and Development Center

FMS Forces Modeling and Simulation/C4I

GOTS Government-off-the-Shelf

HPC High Performance Computing

HPCMP High Performance Computing Modernization Program

HWIL Hardware-in-the-Loop

IHPTET Integrated High Performance Turbine Engine Technology

IMT Integrated Modeling and Test Environments

ISAR Inverse Synthetic Aperture Radar

JET Joint Engineering Taskforce

JID Joint Intrusion Detection

JITC Joint Interoperability Test Command

JNTF Joint National Test Facility

JWCOs Joint Warfighting Capability Objectives

JWSTP Joint Warfighting Science and Technology Plan

LES Large-Scale Eddy Simulation

MAEs Metropolitan Area Exchanges

MEMS Micro Electro Mechanical Systems

MHPCC Maui High Performance Computing Center

MPI Message Passing Interface

MSRCs Major Shared Resource Centers

NAC National Automotive Center

NAP Network Access Point

NAVO Naval Oceanographic Office

NAWCAD Naval Air Warfare Center Aircraft Division

NAWCWD Naval Air Warfare Center Weapons Division

NGI Next Generation Internet

NGIXs Next Generation Internet Exchanges

NIDS Network Intrusion Detection Systems

NRL Naval Research Laboratory

NUC Non-Uniformity Correction

PET Programming Environment and Training

RCS Radar Cross Section
59

ACRONYMS

RTTC Redstone Technical Test Center

S&T Science and Technology

SAVs Site Assistance Visit

SIP Signal/Image Processing

SITREPs Situation Reports

SMDC Space and Missile Defense Command

SOVAS Symbolically Optimized Vehicle Analysis System

SSAAs System Security Authorization Agreements

SSC Space and Naval Warfare Systems Center

T&E Test and Evaluation

TARDEC Tank Automotive Research, Development and Engineering Center

TBCs Thermal Barrier Coatings

TGO Thermally Grown Oxide

THAAD Theater High Altitude Area Defense

vBNS very high performance Backbone Network Service

VPF Vector Product Format

VPG Virtual Proving Ground

WAN Wide Area Network

WRF Weather Research and Forecast

WSMR White Sands Missile Range


Department of Defense

High Performance Computing Modernization Office

1010 North Glebe Road, Suite 510

Arlington, VA 22201

Phone: (703) 812-8205

Facsimile: (703) 812-9701

http://www.hpcmo.hpc.mil