AD-A285 IIII 359IIlII 111111 1111 1111 iil ii

"* DEPARTMENT OF DEFENSE

HIGH PERFORMANCE

* COMPUTING MODERNIZATION.

PLAN •T1C

June 1994

* 40 A 94-31804

0•I H ilI i I II
REPORT DOCUMENTATION PAGE Ot= V 070" 1 U

fba si bX*W far 1W8 &Gd oaflma Wftioadwa is egaid ft"aap aa, ahdt awmgunada .am -a m a sammun &"imalma

inIags. tifunida.iacadnSmahu ~ a~t bra~. Vhhapm NmwamaSwi7Ms. DvcWafar " IamaaaOpuiatk and RepawW 1215 kislua

Dlvis H4ighmy. Staia 1204. Ahikatm. VA 222024302, WA to td Ojs Of MWaGWaN mWW BDdgg. P*woak R*Am Ptopm (07041 Uil). Wauau1 DC 20503.

1. AGENCY USE ONLY (Meave blnk) 2. REORT DATE 3. REPORT TYPE AND DATES COVERED

June 1994 Final

4. TITLE AND SUBTITLE 5. FUNDING NUMBERS

Department of Defense High Performance Computing Modernization

Plan Task Order T-A 15-1274

6. AUTHOR(S)

High Performance Computing Modernization Office

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT

NUMBER

Institute for Defense Analyses (IDA)

1801 N. Beauregard St.

Alexandria, VA 22311-1772

9. SPONSORLNG/MONITORLNG AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORINGAMONITORING AGENCY

High Performance Computing Modernization Office REPORT NUMBER

c/o User Technology Associates

1110 Glebe Road, Suite 650

Arlington, VA 22203

11. SUPPLEMENTARY NOTES

12a. DISTRIBUTION/AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE

Approved for public release, unlimited distribution: June 1994 A

13. ABSTRACT (Maximum 200 words)

In May 1992, the Secretary of Defense forwarded the DoD High Performance Computing (HPC)

Modernization Plan to Congress. That plan, prepared by the Office of the Director of Defense Research and

Engineering and published in March 1992, provided the rationale, process, timetable, and funding

requirements for high performance computing modernization supporting DoD research, development, test, and

evaluation. The purpose of this document is to update and revise the Modernization Plan to reflect the activities

of the past two years, and to present FY94, FY95, and outyear plans.

The DoD HPC Modernization program is structured to modernize the total high performance computational

capabilities of DoD research and development (R&D) to a level comparable to that available in the foremost

civilian and other Government agency R&D environments. The program is multifaceted - creating robustly

networked HPC environments at Major Shared Resource Centers, accommodating specialized HPC

requirements at Distributed Centers, adapting and developing scalable applications software, and educating

users as new architectures and concepts evolve. The program fosters meaningful interaction with the entire

HPC community to facilitate sharing knowledge, tools, and expertise.

14. SUBJECT TERMS 15. NUMBER OF PAGES

High Performance Computing, Scalability, Modernization, Networks, Computer 46

Architectures 16. PRICE CODE

17.SECURITYCLASSiFCATION 18. SECURITY CLASSiFICATION 19. SECURITY CLASSIFICATION 20. LIMITATION OF ABSTRACT

OF REPORT OF THIS PAGE OF ABSTRACT

Unclassified Unclassified Unclassified SAR

NSN 7540-01-290-5500 Standard Form 298 (Rev. 2-89)

Prescribed by ANSI Std. Z39-18

298-102
- DoD HPC Modernization Plan

Office of the Director of Defense Research and Engineering

Department of Defense Accesion For

NTIS CRA&I

June 1994 Urn:anouinced LI

Justific-Itbn;

.................................................................. 4

Dist iby to I

A , ,

Disti .. _LLc..

0N

S•i I I IIIi
PURPOSE

In May 1992, the Secretary of Defense forwarded the Department of Defense (DoD)

High Performance Computing (HPC) Modernization Plan to Congress. That plan,

prepared by the Office of the Director of Defense Research and Engineering (DDR&E) and

published in March 1992, provided the rationale, process, timetable, and funding

requirements for high performance computing modernization supporting DoD research,

development, test, and evaluation (RDT&E). The purpose of this document is to update

and revise the Modernization Plan to reflect the activities of the past two years, and to

present FY94, FY95, and outyear plans.

The DoD HPC Modernization program is structured to modernize the total high

performance computational capability of DoD research and development (R&D) to a level

comparable to that available in the foremost civilian and other Government agency R&D

environments. The program is multifaceted-creating robustly networked HPC

environments at Major Shared Resource Centers, accommodating specialized HPC

requirements at Distributed Centers, adapting and developing scalable applications

software, and educating users as new architectures and concepts evolve. The program

fosters meaningful interaction with the entire HPC community to facilitate sharing

knowledge, tools, and expertise.

SCOPE

This plan covers HPC in support of the DoD science and technology program under

the cognizance of the DDR&E. It also stimulates interaction and coordination with other

DoD and external R&D activities and promotes the incorporation of HPC technology into

DoD operational environments.

i1.
Executive Summary

The Department of Defense relies on technologically superior systems to counter

numerically larger forces, to reduce casualties and damage to the national infra-structure,

and to permit rapid, decisive action when necessary. While DoD is facing significant

reductions in force structure, the need to maintain technical superiority is more critical than

ever. Therefore it follows that it is necessary to sustain the investment in Science and

Technology (S&T) and the supporting key technology program-the DoD High 0

Performance Computing Modernization Program.

The success of the HPC Modernization Program will critically impact the conduct

of the Department's Science and Technology initiatives as it addresses the crucial next step

of integrating and deploying advances in computing and communications into DoD S&T

programs, and couples them with military systems design and development.

The DoD HPC Modernization Program is structured to modernize the total high

performance computational capability of DoD research and development (R&D) to a level

comparable to that available in the foremost civilian and other Government agency R&D

environments. This is in consonance with the Congressional direction to DoD to program

and budget for HPC modernization of its laboratories in recognition that the HPC resources

and environments available to defense scientists and engineers had fallen below the levels

available to their counterparts in the civilian sector.

The resultant DoD HPC Modernization Program is a coordinated, sustained

initiative to develop complete, balanced HPC environments based on user requirements.

The planned HPC capability is focused upon the establishment of networked Major Shared

Resource Centers (MSRCs) and Distributed Centers. The MSRCs provide complete HPC

environments including various types of computing systems, scientific visualization

capabilities, extensive peripheral and archival storage, and expertise in use of these

systems. The MSRCs support the wide variety of research and development problems

arising from the S&T program. The MSRCs address the computer and computational

sciences, and provide the education and training which allows all of the DoD laboratories to

be more productive in S&T research and development. In addition to the MSRCs, the

program will deploy systems at Distributed Centers to fulfill special R&D requirements 9

where deployments of the HPC systems are critical to success of R&D efforts.

Initiatives and mechanisms are being established to ensure that the modernization of

the DoD laboratories is interrelated, well formulated, and within the context of the national

HPC infrastructure. The program will take actions to encourage and facilitate interactions,

collaborations, and communications with the civilian HPC community and other 0

government agencies to share knowledge, tools, and expertise.

The program is built upon the foundations of proper planning and management.

The program vision, goals, strategies, and metrics are described herein in detail, and, with

the requirements analysis (summary herein), form the basis for modernization decisions

and acquisition plans/strategies. A Joint High Performance Computing Modernization

Office was formed to function in an intellectual and leadership role for defining the

direction, planning, and policy-making for DoD HPC Modernization. This office will

perform the necessary life cycle management functions and the acquisition streamlining

processes, as required, to implement and execute the program in an effective and cost

efficient manner.

Finally, the plan identifies the current and following five year funding requirements 0

to enable the ongoing DoD R&D activities to meet mission-driven requirements and to

fully engage the national HPC computational and networking trends.

iv

m III m II I I II I I I I I I
TABLE OF CONTENTS

Eaga

1. Introduction .................................................................................. 1

1.1 Mission Needs ...................................................................... 1

1.2 Background .......................................................................... 2

1.3 Program Formalization ............................................................. 4

2. Program Foundations .................................................................... 7

2.1 Vision ............................................................................... 7

2.2 Goals ................................................................................. 7

2.3 Strategies ............................................................................ 8

2.4 Metrics ............................................................................. 10

2.5 Outcomes .......................................... 10

3. Leveraging National HPC Infrastructure ............................................ 11

3.1 Program Relationships ........................................................... 11

3.2 The Federal HPCC Program ..................................................... 11

3.3 Defense HPC R&D Programs ................................................... 13

3.4 Other Government Agency Programs .......................................... 14

4. Requirements Analysis ................................................................ 15

4.1 Methodology ...................................................................... 15

4.2 Summary of Requirements Analysis ............................................ 19

5. Implementation Strategy ............................................................... 23

5.1 Consolidation of DoD's HPC Infrastructure ...................................... 23

5.2 FY93 Efforts ...................................................................... 23

5.3 FY94 Efforts ...................................................................... 25

5.4 Planned FY95 Efforts ............................................................. 27

5.5 Long Term Implementation Strategy (FY96-FY99) .............................. 29

5.6 Funding Needs .................................................................... 29

6. Acronyms and Definitions ............................................................ 31

i i m i II Im
0

List of Tables

Table I DoD S&T Sites .................................................................. 16

Table 2 Total Number of HPC S&T Users ...................... ................. 16

Table 3 Computational Technology Areas ................................................. 17

Table 4 Summary of Overall Unclassified Performance Requirements (GF-Yrs) .... 19

Table 5 Summary of Overall Classified Performance Requirements (GF-Yrs) ....... 20 0

Table 6 Summary of Planned FY94 HPC Systems ................................... 26

Table 7 Funding Profile ...................................................................... 29

List of Figures

Figure 1 Interim DREN Configuration - 3Q FY94 ...................................... 3

Figure 2 HPC Modernization Program Management Structure ........................ 5

Figure 3 DoD HPC Modernizastion Plan Roadmap .................................... 24

Figure 4 DoD HPC Laboratory Modernization ............................................. 28

0

vi

0
1. Introduction

In March 1992, the Office of the Director, Defense Research and Engineering

(DDR&E) published the Department of Defense (DoD) High Performance Computing

(HPC) Modernization Plan. The purpose of the plan was to provide the rationale, process,

timetable, and funding requirements for high performance computing modernization in

DoD research, development, test, and engineering under the cognizance of the DDR&E.

This document updates the DoD HPC Modernization Plan, reflecting the activities of the

program to date, and the plans necessary to meet the Congressional modernization

mandate. The document is structured to provide the reader with a review of DoD mission

needs, the program genesis, and the recent program realignment. The plan then addresses

the program foundations: vision, goals, strategies, and metrics. A summary of a recently

completed comprehensive DoD HPC requirements analysis is provided, followed by the

FY94, FY95, and outyear implementation plans to satisfy these requirements. Finally, the

funding required to address mission-driven requirements is presented.

1.1 Mission Needs

The mission of this program is to modernize the high performance computational

capability of DoD laboratories to a level comparable to that available in the foremost civilian

and other Government agency R&D environments. The program is multifaceted. It

involves creating HPC environments at major and distributed centers accessible to local and

remote scientists and engineers via high-speed networks. Providing for the adaptation of

existing and development of new scalable applications and algorithms to address growing science and technology requirements, along with continued education of users as new

architectures and concepts evolve, are integral components of the program. The program

pursues continuous interaction with the national HPC infrastructure, including academe,

industry, and other p.vermment agencies to facilitate the sharing of knowledge, tools, and

expertise.

High performance computing has been identified as a key enabling technology

essential to achieving the objectives of the Department's science and technology (S&T)

program. DoD intends to exploit the orders of magnitude improvements in computational

and communications capabilities as a result of advancement in hardware, architectural

designs, software, networking, and computational methods in support of DoD missions.

Many Defense-related problems in the areas of chemistry and materials science,

computational fluid dynamics of air, undersea, and surface vessels and their ordnance,

parametric weight/vulnerability reduction studies, automatic target recognition, submarine

hydrodynamics, ocean acoustics, high performance weapons design, and prediction and

dispersion of hazardous materials, require computational performance of trillions of

operations per second (teraops). These diverse S&T applications also require massive data

storage and scientific visualization capabilities to render the computational results in

graphical images that enable understanding of the computations. The reliance of S&T

research on computational science is increasing rapidly due to the greatly expanding use of

HPC in scientific research, modeling, and simulation. HPC greatly benefits DoD

I
laboratories by decreasing the need for prototype systems and limiting the amount of livefire tests. Solutions to critical problems leading to advancements in these areas are

fundamentally driven or limited by the HPC capability and environment within the Defense

laboratories. DoD S&T researchers require HPC environments populated with a variety of

system architectures and data storage to permit the highest fidelity and greatest speed in

analyzing problems of ever increasing size and complexity.

For these reasons, it is critical that DoD vigorously continue implementation of this

modernization plan. While some recent improvement has been achieved by the program, HPC systems currently available to most DoD users significantly lag behind those

commercially available. Current data storage and scientific visualization capabilities are

undersized and inadequate for today's research problems. Many science problems,

modeling and simulation applications require HPC systems capable of scaling to several

orde's of magnitude in performance beyond those being used by DoD laboratories today.

Software init.atives are needed in applications and algorithm development to enable use of

the latest scalable technologies. Placement of these new technologies in the laboratories

requires renewed emphasis on new software techniques, computational and computer science, and on education and training. All of these areas are fundamental to effective HPC

capability and must be modernized and upgraded in parallel to meet even today's

requirements of the DoD S&T community.

DoD will need to provide seamless HPC capabilities to scientists and engineers (S&E) throughout DoD regardless of location of resources. To support the requirements of

remote users, an advanced high-speed, robustly interconnected network linking DoD S&T

researchers to the advanced HPC systems is needed. This network capability will require continued advancement to remain consistent with the performance of the computing

systems, the growing data traffic requirements, and rapidly evolving network technology.

Finally, to fully achieve the Department's scientific computational needs,

applications, software tools, and algorithm development must be advanced and expanded to

take advantage of the emerging computing power. The DoD HPC Modernization Program

must stimulate the efforts of its own researchers and engage and leverage the efforts of

academe and industry to improve the ability to effectively address HPC software problems.

1.2 Background

In 1991 Congress noted that DoD laboratories lagged considerably behind the

Department of Energy (DOE), National Aeronautical and Space Agency (NASA), and

National Science Foundation (NSF) Supercomputer Centers in computing capability. This

prompted the enactment of language in the National Defense Authorization Act of FY92-93

directing DoD to present a five year master plan to Congress to modernize of the HPC

capabilities of the DoD laboratories. The DDR&E established the HPC Working Group

(HPCWG) to develop this plan. The HPCWG consisted of representatives from the

Services, Defense Nuclear Agency (DNA), Advanced Research Projects Agency (ARPA),

and the Office of the Assistant Secretary of Defense for Command, Control,

Communications, and Intelligence (ASD(C31)). The Secretary of Defense forwarded the

original DoD HPC Modernization Plan to Congress in May 1992. A FY93 Implementation Plan and the HPC Modernization Program were initiated using funds that Congress had

authorized and appropriated (FY92 ($42M) and FY93 ($45M)) for DoD laboratory HPC

modernization.

During FY93 a large general purpose vector supercomputer, a Cray C916/512, was

acquired and installed at the Army Corps of Engineers Waterways Experiment Station

2
(CEWF.S), Vicksburg, MS. In addition, the following six scalable parallel systems were

installed or upgraded:

Intel Paragon XP/S-15, Aeronautical Systems Center (ASC), Wright-Patterson

AFB, Ohio

lntel Paragon XP/S-25, Naval Command, Control and Ocean Surveillance

Center (NCCOSC), San Diego, California

IBM SPl, Maul HPC Center (MHPCC) (supporting the Air Force Maul

Optical Station (AMOS)), Maui, Hawaii

Kendall Square Research KSR 1-256, Army Research Laboratory (ARL), Aberdeen Proving Ground, Maryland

• TMC CM-5 enhanced, Naval Research Laboratory (NRL). Washington, DC

TMC CM-5 enhanced, Army High Performance Computing Research Center,

(AHPCRC) University of Minnesota, Minneapolis, Minnesota

In the area of connectivity, a high speed wide area network was established by integrating and supplementing existing Army and Air Force supercomputer networks to

cream the Interim Defense Research and Engineering Network (IDREN). This network,

illustrated in Figure 1, interconnects defense S&T users and DoD HPC resources and

provides gateway connectivity to the Intemet.

AriSe. JUoa'.. AHPCI•I: m.aa t.•b ,.,..,0]•

SNaUCk

-- \ .-i k

\ \ ,,wc.w,• ,co.,, NR

I SO Kbom NOAA

SLm w•no (Tl1

S40 taupe ('r31 • Nodo • •n•n• ....... Nol 04•EN mn•d Ho• Site • • sb

TO Im €IIQcomlMm¢I

Figure 1. Interim DREN Configuration - 3Q FY94

• ' ' ' . I I I I I
Finally, during FY93, various upgrades were made to computing systems at the

National Consortium for High Performance Computing (NCHPC) facilities. In return DoD

was given access to NCHPC's computing resources and a Memorandum of Agreement

(MOA) was put in place supporting DoD's HPC needs in applications software

development The agreement defines six DoD technology thrust areas:

Computational Structural Mechanics

"* Computational Fluid Mechanics

"* Computational Chemistry and Materials Science

"* Computational Electromagnetics and Acoustics

"* Climate / Weather / Ocean Modeling

"* Signal / Image Processing

NCHPC will collaborate with the Modernization Program to accelerate the

exploitation of scalable parallel systems to solve HPC problems in these technology areas.

The MOA provides the DoD S&T community a training and information distribution

infrastructure available through NCHPC and access for DoD researchers to NCHPC

computers. NCHPC's MOSAIC information system will be used by DoD to access these

NCHPC training and HPC resources, access to NCHPC HPC will facilitate DoD

researcher collaboration with NCHPC academics and will allow scaling up to much larger

scalable HPC in some cases.

1.3 Program Formalization

In recognition of the program's potential impact, magnitude of the effort, and the

need for a more formal management structure, the DDR&E appointed an Executive Director

and a Program Manager for the program in January, 1994. A HPC Modernization

(HPCM) Office was created and staffed with representatives from each Service to review

the program's vision and strategies to ensure that the program meets the laboratories' and

R&D centers' long-term needs. The resultant program addresses the total HPC

environment including both hardware and software issues. The HPCM office will

continue to function in an intellectual and leadership role by defiming the direction,

planning, and policy for DoD HPC modernization. This office also performs the necessary

life cycle management functions as required to implement and execute the program in an

effective and cost efficient manner.

The high level management structure for the HPC Modernization Program is

illustrated in Figure 2.

"4
iio * poicy &MC.ppbmR

*aem hquwmkm ____ Worwg __

6 es-pw...

NPC Advisory Paud *cvdwAet

vowgCOre 2. Modrniation sad ProramMangeen Staructure

AdmwIsiWPno5
00000000

6 

0
2. Program Foundations

0 DoD HPC modernization is fundamentally driven by Defense S&T community

requirements. This program will empower the DoD researcher to meet the computational

challenges of the research and development (R&D) mission. To provide an integrated cost

efficient solution, the modernization has been planned within the context of a set of

guiding principles and precepts which can be termed vision, goals, program strategy, and

metrics.

2.1 Vision

The HPC modernization vision is to raise DoD's R&D high performance computing

and communications capacity and capabilities to a level equal to or greater than that

available in the foremost academic research centers and industry, thereby enabling the

* United States to maintain its technological supremacy in weapon systems design and to

foster the flow of this technology into warfighting support systems and dual-use functions.

Successfully fulfilling this vision means every DoD S&T researcher will have ready

desktop access to the HPC tools necessary to complete the mission in a timely fashion.

This implies that DoD researchers have access via their desktop systems to a broad HPC

infrastructure which provides:

* Robust high-speed networking

* Remote classified and unclassified access through transparent network services

0 A variety of powerful and easy to use systems and applications software to

permit optimum mapping of system types to particular applications

* Massive, hierarchical data storage

* Immediate access to data

* Post-processing to facilitate meaningful data interpretation

2.2 Goals

* The DoD HPC modernization program will strategically locate, rapidly deploy, and

sustain the highest quality high performance computing environments and networks. The

emphasis is on leveraging the national trends fostered by existing DoD investments in the

Federal High Performance Computing and Communications (HPCC) Program, Defense

HPC R&D, and other related research, development, test, and evaluation (RDT&E) efforts.

It is a sustained program to fully engage and draw from the national effort.

7
Program initiatives are directed toward the following goals:

" Establish and improve the access to state of the art HPC capacity and capability 0

for DoD scientists, engineers, and analysts to enable weapons development

and more capable warfighting systems.

" Encourage development of software tools and applications to make it easier to

use and exploit the capabilities of HPC, and expand and educate the DoD HPC

user base.

" Engage, leverage, contribute to and be a major participant in the National HPC

Infrastructure and exploit its benefits for Defense R&D. This engagement

includes DoD laboratories, the Services' in-house basic research entities (Army

Research Office, Air Force Office of Scientific Research, and the Office of

Naval Research), the Advanced Research Projects Agency, industry, 0

universities, and other Government agencies such as the Department of

Energy, the National Science Foundation, and the National Aeronautics and

Space Administration.

2.3 Strategies •

True modernization of DoD's HPC capability and fulfillment of the above vision

and goals requires a program strategy that addresses all aspects of HPC. While advancing

the level of hardware performance is critical to success, the higher objective is to enable

better scientific research and technology development for superior weapons, warfighting

and related support systems.

The HPC assets acquired via this program will be available to scientists and

engineers supporting the DoD S&T program in the L _rvices and Defense agencies, as well

as collaborative researchers in academe and U.S. industry supporting the S&T program.

These HPC resources will be interconnected via the networking initiative being

implemented under the DoD HPC Modernization Program, the Defense Research and

Engineering Network (DREN). DREN will be interconnected to the Internet to allow DoD

scientists and engineers to properly and adequately interface to their counterparts and

colleagues in the civilian sector.

The strategy to improve availability and access to state of the art HPC capacity and

capability encompasses: •

" Providing a balanced set of HPC commercially available technologies to meet

the full range of DoD requirements and to permit optimum mapping of

requirements to system types.

" Building complete HPC environments at DoD laboratories and R&D centers 0

designated as HPC Major Shared Resource Centers (MSRCs) to support the

widest community of users. These MSRCs will include hardware, data

storage, scientific visualization capabilities, high speed interconnections,

supporting infrastructure, and expertise in computational and computer

sciences and HPC software.

* Supporting development efforts designed to provide teracomputing capabilities and technologies which are applicable to Defense S&T requirements.

8
0

Developing heterogeneous computing environments to allow the optimum

mapping of varied S&T problems onto appropriate types of HPC systems.

* Provisioning, where appropriate, special configurations, designs, or

architectures; embedded systems; classified systems; distributed data storage;

or scientific visualization platforms at laboratories designated as Distributed

Centers. These Distributed Centers will focus on and primarily address

smaller scale, intense, local mission needs and also make unused capacity and

* capability available to remote users. In addition, Distributed Centers will allow

leveraging of special collections of HPC expertise for the benefit of the entire

program.

Providing a focus on standards in DoD S&T software to ensure that future

transitions and advancements in software technology are applied in an efficient

* and cost effective manner.

The strategy to encourage the development of software tools and applications and

expand and educate the DoD HPC user base includes:

0 Focusing on applications oriented software initiatives designed to overcome the

* technological inhibitors which delay the effective use of many scalable parallel

high performance computers.

* Providing user education and training in computer and computational sciences,

software applications and optimization, and code conversion.

* Promoting the formation of DoD-sponsored inter-disciplinary teams and

collaboration groups to leverage academe and industry expertise in HPC for the

solution of DoD problems.

0 Deploying robust and cost effective network connectivity consistent with HPC

traffic requirements among and between DoD researchers and HPC assets, and

between the DoD and external R&D communities.

The strategy to engage, leverage, contribute to, and be a major participant in the

national HPC infrastructure includes:

" Ensuring the Defense laboratories remain cognizant of, interact with, and

• leverage internal DoD HPC initiatives such as the ARPA technology

development efforts and the algorithm and software research initiatives of the

Defense research offices.

" Ensuring that the Defense laboratories, remain cognizant of, collaborate with,

and leverage other government agencies' and each other's HPC efforts.

* Providing HPC environments to support defense applications which are

directly related to dual use technologies and National Challenges, e.g.,

environment, medical, automotive, and manufacturing processes and products.

"* Ensuring the availability of HPC environments to support Defense

• applications which address Grand Challenge needs, e.g., meteorological,

oceanographics, and environmental modeling, nature of new materials,

structure of biological and chemical phenomena, and energy efficient vehicles

and airplanes.

9
2.4 Metrics

The metrics for this program are closely tied to the program goals. The ultimate

measure of success will be improved quality of research within the S&T community, e.g.

more realistic models and quicker solutions to DoD S&T problems. DoD R&D HPC

capability and capacity will also be measured and compared relative to that available to the

non-Defense HPC community. The success of the program in meeting this goal will be

demonstrated by peer group evaluations.

In addition, the level to which DoD HPC capabilities satisfy the computational

requirements of the S&T program will be monitored not only for computational speed, but

also for all other HPC capabilities that are required for balanced HPC systems, such as

central memory, secondary storage, and archival storage. The program's success in

expanding and educating the DoD HPC user base will be measured _ dý)Iitatively based on

HPC use in Defense applications, while quantitative measures will : -sed on the increase

in the number of new HPC applications and users. Success will al.b,• . measured

qualitatively by user satisfaction with the training, ease of access, and ease of use of new or

scalable systems. The metrics associated with the outreach program to engage and leverage

the national HPC infrastructure will include the number of joint projects and collaborations

with academe and industry, cooperative agreements that have HPC as a key factor, and

evidence of transferred expertise and technologies to and from industry.

2.5 Outcomes

Deployment of modernized HPC provides an environment in which the DoD S&T

community makes great strides using HPC to solve mission problems. Additional HPC

resources allow researchers to tackle problems which are currently intractable. Improved 0

HPC capability shortens design cycles by reducing the need to rely on hand-crafted

prototypes and destructive testing. Robust, high-speed connectivity encourages daily

collaboration among separated and remote users. Massive on-line hierarchical data storage

coupled with interactive access to remote resources encourages users to ignore location and

start considering hardware and software fit to a particular problem. Finally, lessons

learned while executing the above provides important design feedback for future

generations of improved HPC.

The broad DoD HPC system which is the logical result of this program will be

widely recognized as a strategic resource. With these facilities, the DoD S&T community

will routinely conduct world class research, and the knowledge and insights gained will be

crucial to maintaining the technological edge the U.S. currently enjoys. In short, the

environment created by this program will provide all DoD researchers the resources needed

to successfully execute the S&T mission, and will empower each to access appropriate

resources as needed without leaving his or her desk or workstation.

0

10
3. Leveraging National HPC Infrastructure

0 3.1 Program Relationships

The overall Defense investment in HPC consists of three interrelated initiatives: 1)

participation in the Federal High Performance Computing and Communications (HPCC)

Program, 2) Defense-specific HPC R&D programs, and 3) the DoD HPC Modernization

Program. Each is a critically necessary element in the application of new HPC hardware

* and software technologies to Defense areas of science and technology that are essential to

the development of new military systems. All three programs involve extensive ongoing

research and development activities in academe and industry, as well as the Federal

laboratories.

The Federal HPCC Program is a national commitment to HPC R&D and application

* of HPC technology to support the National Information Infrastructure. For complete

details of the Federal HPCC Program see High Performance Computing and

Communications: Technology for the National Information Infrastructure, a report by the

Committee on Information and Communications, National Science and Technology

Council.

* The Defense-specific HPC R&D programs complement the Federal HPCC

Program. These technology development programs focus on fulfilling Defense HPC needs

such as providing tailored hardware or software elements, embedded systems, accelerators

for specific problem domains, and other challenging Defense-specific problems. Planning

for these programs is coordinated through the Office of the Director, Defense Research and

Engineering (DDR&E) and through the Joint Directors of Laboratories Project Reliance

* infrastructure.

The HPC Modernization Program is a sustained activity designed to deploy high

performance computing systems and networks at selected Defense laboratory locations to

provide complete HPC environments that will enable Defense scientists and engineers to

conduct important computational S&T in an effective and efficient fashion. The program

will leverage technology, applications, computer systems, and software products

developed directly or indirectly from DoD investments in the Federal HPCC Program,

Defense-specific HPC R&D, and other R&D efforts.

With the Federal HPCC Program and Defense HPC R&D programs providing the

techniques and the HPC Modernization Program providing the tools and environments, the

* Defense technology base programs will be significantly enhanced. In turn, these

technology base programs develop the Defense-related areas of S&T that are critical to the

development of new military systems.

3.2 The Federal HPCC Program

The -goal of the Federal HPCC Program is to accelerate the development of future

generations of high performance computers and networks and the use of these resources in

11

S. . .I I
the Federal Government and throughout the American economy. Scalable high

performance computers, advanced high speed computer communications networks, and

advanced software are critical components of a new National Information Infrastructure

(Nil). This infrastructure is essential to our national competitiveness and will enable us to

strengthen and improve the civil infrastructure, digital libraries, education and lifelong

learning, energy management, the environment, health care, manufacturing processes and

products, national security, and public access to government information.

Specific program goals include: 0

" Extend U.S. technological leadership in high performance computing and

computer communications.

" Provide wide dissemination and application of the technologies to speed the

pace of innovation and to improve the national economic competitiveness, 0

national security, education, health care, and the global environment.

" Provide a key part of the foundation for the National Information Infrastructure

and demonstrate selected Nil applications.

The goals will be realized through five equally important, integrated components 0

representing key areas of high performance computing and communications:

" High Performance Computing Systems (HPCS) to extend U.S technological

leadership in high performance computing through the development of scalable

computing systems, with associated software, capable of sustaining at least

one trillion operations per second (teraops) performance. 0

" National Research and Educational Network (NREN) to extend U.S.

technological leadership in computer communications by a program of research

and development that advances the leading edge of networking technology and

services.

" Advanced Software Technology and Algorithms (ASTA) to demonstrate 0

prototype solutions to Grand Challenge problems through the development of

advanced algorithms and software and the use of HPCC resources.

"* Information Infrastructure Technology and Applications (IfTA) to demonstrate

prototype solutions to National Challenge problems using HPCC enabling •

technologies.

" Basic Research and Human Resources (BRHR) to support research, training,

and education in computer science, computer engineering, and computational

science, and enhance the infrastructure through the addition of HPCC

resources. 0

HPCC program strategies include developing the networking technology required

for deployment of nation-wide gigabit speed networks through collaborations with

industry; demonstrating the productiveness of wide area gigabit networking to support and

enhance Grand Challenge applications collaborations; providing and encouraging

innovation in the use of high performance computing systems and network access 0

technologies for solving Grand Challenge and other applications by establishing

collaborations to provide and improve emerging software and algorithms; and creating an

infrastructure, including High Performance Computing research centers, networks, and

12
collaborations that encourage the diffusion and use of high performance computing and

communications technologies in U.S. research and industrial applications.

3.3 Defense HPC R&D Programs

The Advanced Research Projects Agency is the lead DoD agency for advanced

technology research and development and has the leadership responsibility for the Federal

HPCC Program within DoD. The ARPA program is developing the technologies needed to

enable a broad base of applications and users on new scalable computing systems. In

addition, the foundations for future generations of computing systems are being developed.

In the past year, the National Security Agency (NSA), has joined the HPCC

Program. The goal of NSA's HPCC Program is to accelerate the development and

application of the highest performance computing and communications technologies to meet

national security requirements.

The three Services each have basic research efforts and HPC-related areas to

address fundamental HPC research problems. These are addressed through the Army

Research Office, The Office of Naval Research, and the Air Force Office of Scientific

Research.

The Army Research Office (ARO) has efforts in computational methods performed

by three categories of investigations: (i) single investigators, (ii) block efforts and (iii)

Centers. The single investigator studies vary from scalable software tools for adaptive

scientific computation to studies in elastodynamics. The block efforts are funded primarily

by ARPA and deal with software development for scalable efforts.

The largest of the ARO-sponsored centers, the Army High Performance Computing

Research Center (AHPCRC) at the University of Minnesota, consists of three integrated

functions, namely, (i) interdisciplinary research into various aspects of HPC including

advanced algorithms, applications, implementation environments, and graphics and

visualization; (ii) evaluation of advanced computing systems; and (iii) an infrastructure

support program of structured technology transfer. A major emphasis is placed on human

resources with significant emphasis on the involvement of Historically Black Colleges and

Universities and Minority Institutions (HBCUs/MIs).

The Office of Naval Research is sponsoring a five year (FY93-97) multidisciplinary

basic research initiative in massive parallelism with the goal of concurrently accelerating

computational research capabilities in Navy-critical application domains while using the

characteristics of these applications to drive research in advanced system software support,

programming environments, and numerical methods. The initiative focuses on three

specific application domains chosen for their mission criticality and the spectrum of

challenges they offer to scalable parallel computing: large-scale ocean circulation

simulation, atomic and molecular interaction calculations for advanced materials, and solidphase propellant combustion. Many of these applications exhibit irregular, dynamic, or

adaptive behavior inadequately supported by existing parallel programming methodologies

resulting in codes that lack efficiency, scalability, or portability. Expected results include

efficient, scalable, portable codes representative of the three target application domains;

prototype compilers, programming tools, and system software; and innovative parallel

numerical algorithms. Transition plans include widespread distribution of prototype

software and algorithms and possible installation of prototype software at DoD HPC Major

Shared Resource Centers.

13

S
Also, the Office of Navh Research manages a development program to explore

HPC applications enabling solution of real Navy problems. For example, completed tasks

have investigated applications to (1) increase range/depth/azimuth resolution from passive 0

sonar arrays, (2) perform 2D and 3D visualization of geographic data on a massively parallel SIMD architecture, and (3) accomplish the parallelization of a C31 mission-planniig

tool onto a heterogeneous computer array. Additional work is ongoing to investigate a

number of Navy HPC applications.

Many of the future Air Force requirements for HPC require not only vast amounts

of computational power, but they also require the development of vastly improved

numerical methods to ensure that the application of this computational power achieves

accurate answers to pressing problems. The Air Force program, primarily through the Air

Force Office of Scientific Research (AFOSR) support in its external program to academia,

concentrates on the development of these novel methods. AFOSR concentrates on

supporting innovative methods and algorithmns which enable the improved modeling,

simulation, understanding, and control of complex physical phenomena.

The progress from AFOSR research efforts is utilized to impact critical applications

areas, such as computational plasma physics, computational electromagnetics,

computational fluid dynamics, and computational chemistry. In each of these application

areas, the Air Force, through the AFOSR basic research program and other technology

base support, manages an integrated extramural and in-house laboratory program to exploit

the potential of high performance computing.

3.4 Other Government Agency Programs

Many other government agencies, such as DOE, NASA, and NSF, have extensive

HPCC programs. DoD laboratories interact directly with these agencies and, in addition,

the HPC Modernization Program plans to interact more directly with these agencies within

its various program components. As the Modernization Program strives to provide a

complete HPC environment to the users in the DoD S&T community, it recognizes that

expertise in these various agencies can make vital contributions to that environment.

Indeed, collaborative interactions currently exist among the DOE National Laboratories, the

NSF HPC Centers, and DoD laboratories, including joint efforts in applications software

development. Several DoD laboratories are members of consortia involving Non-DoD

laboratories. The Modernization Program will leverage that expertise through a variety of

cooperative HPC programs.

14 0
4. Requirements Analysis

The DoD HPC Modernization Program is designed to meet user requirements. In

April 1994, a comprehensive Requirements Analysis was completed by the HPC

Modernization Office, resulting in a requirements database that is being maintained to

provide the primary basis for future program planning.

4.1 Methodology

This Requirements Analysis used information gathered via a survey and interview

process with the functional user community. It was accomplished by a tri-Service team of

eight scientists and engineers associated with HPC, with technical support from the

Institute for Defense Analyses.

Table I lists organizations and locations that were determined to have S&T HPC

requirements. The total numbers of HPC S&T users within the Services and participating Agencies are given in Table 2. Other agencies, such as the Defense Mapping Agency, have

expressed interest in stating future requirements and will be included as requirement are

periodically updated.

The analysis determined functional S&T requirements and all aspects of HPC

requirements that resulted from these functional S&T requirements, including hardware,

software, networking, and training. A major feature of the analysis was the exploration of

how computational requirements flow from the underlying S&T program goals, and, in

particular, to determine possible major accomplishments resulting from the exploitation of

HPC capabilities. An important aspect of this activity is the determination of the

impediments to routine use of new scalable HPC technologies by the DoD S&T program

and methods for removing these impediments.

The DoD computational S&T program was categorized into nine technology areas.

These areas are outlined, along with definitions of their scopes, in Table 3. The first six

areas represent collaboration groups between the DoD computational S&T community and

the National Consortium for High Performance Computing (NCHPC), designed to

accelerate the productive use of new scalable HPC technologies by the DoD S&T program. The additional three areas were identified in the Requirements Analysis as distinct major

DoD computational technology areas.

The quantitative and qualitative information was captured in a data base for display

and analysis and to enable the program to assess the evolving requirements. Overall HPC

performance requirements were aggregated in units of gigaflops-years (GF-yrs), i.e., one

GF-yr is the computational capacity represented by a one GF processor computing for one

year. In cases where user requirements were specified in terms of specific HPC systems,

these requirements were converted to GF-yrs using vendor-provided theoretical peak

performance for each of those systems. Recognizing that actual efficiencies vary widely

from system to system and from application to application, use of theoretical peak

performances to make the conversion to GF-yrs is considered to be more consistent and

less arbitrary than development of subjective efficiencies for typical DoD applications.

15
Table 1. DoD S&T Sites

Organization Locations Organization Locations

Aberdeen Proving Grounds Naval Surface Warfare Carderock, MD

(APG), MD Center Dahlgren, VA

Army Reearch Adelphi, MD China Lake, CA

Laboratory Fort Monmouth, NJ Naval Air Warfare Patuxent River, MD S

Watertown, MA Center Warminster, PA

White Sands, NM Naval Command, San Diego, CA

Control and Ocean

Surveillance Center

Tank-Automotive Warren. MI Naval Research Monterey. CA

Research, Develop- Laboratory Stennis Space Center, MS

ment and Engineering Washington, D.C.

Center

Champaign, IL Naval Undersea Warfare Newport, RI

Army Corps of Ft. Belvoir, VA Center

Engineers Hanover, NH Office of Naval Research Arlington, VA

Vicksburg, MS

Missile Command Huntsville, AL Air Force Office of Boiling AFB, DC

Army Medical Research APG, MD Scientific Research USAF Academy. CO

Development, Frederick. MD Eglin AFB, FL

Acquisition and Walter Reed, MD Wright Laboratory Tyndall AFB, FL

Logistics Command WPAFB, OH

(Provisional) .

Army Research Office Research Triangle Park, NC Griffiss AFB, NY

Aviation and Troop Moffett Field. CA Rome Laboratory Hanscom AFB, MA

Command St. Louis, MO Edwards AFB, CA

Concepts Analysis Bethesda, MD Phillips Laboratory Hanscom AFB, MA

Agency Kirtland AFB, NM

Army Materiel Systems APG, MD [Armstrong Brooks AFB, TX •

Analysis Activity _Laboratory Tyndall AFB, FL

Armament Research Dover, NJ

Development and Watervliet, NY

Engineering Command, [Defense Nuclear Agency jAlexandria, VA

Edgewood Research, APO, MD

Development, and •

Engineering Center

Table 2. Total Number of HPC S&T Users

Organization User Groups Users

Army 60 1,595

Navy 93 757

Air Force 76 584

DNA 4 330

Toal DoD 233 3,266

16

,, i II I i I !S
Table 3. Computational Technology Areas

Area Description

1. Computational Structural Mechanics - High resolution multi-dimensional modeling of materials

and structures subjected to a broad range of impulsive

loading ranging from weak to intense.

DoD application areas include conventional underwater

explosion and ship response, structural acoustics, coupled

field problems, space debris, propulsion systems,

structural analysis, total weapon simulation,

lethality/survivability of weapon systems (aircraft, ships,

submarines, tanks), theater missile defense lethality

analysis, and optimization techniques.

2. Computational Fluid Dynamics Detailed, three-dimensional, high resolution Navier-Stokes

and large eddy simulation (LES) modeling of fluid flow for

transport and tactical aircraft at transonic through high

supersonic velocities, projectiles and missiles at subsonic

through hypersonic velocities, and simulated nuclear and

conventional weapon blast wave impact on tactical

vehicles and structures.

" Transient detailed reactive flows in two and three

dimensions for engine and combustor design, chemicalacoustic instability and noise control, gun tube propellant

modeling which includes multiphase flow, combustion of

solid and liquid propellants, and detonation and explosion

dynamics for safety, survivability, and lethality.

" Detailed, three-dimensional transient hydrodynamic and

airwake flow about ships, submarines, and torpedoes

including the multiphase flows associated with cavitation,

wavebreaking, and underwater explosions.

" Detailed, medium and high resolution two- and threedimensional incompressible flow modeling for

investigating navigable waterway/harbor design,

vessel/fluid and structure/fluid interactions, ship-to-shore

logistics, hurricane protection, and hydrologic systems

impacts on battlefield mobility.

" Area includes atmospheric modeling, contaminant

transport modeling for battlefield and environmental

applications, and modeling of the cardiovascular system in

humans, including blood flow through the heart.

17
Table 3. Computational Technology Areas (Continued)

Area Description

3. Computational Chemistry and Prediction using advanced computational chemical and

Materials Science molecular design tools to obtain accurate values of

detectable spectroscopic properties, evaluate mechanical

properties of proposed advanced materials, and evaluate the

catalytic performance of zeolites and other catalytic materials for destruction of hazardous materials, such as

halons.

• Modeling of molecular and electronic structures, reaction

pathways and dynamics, and intramolecular and

intermolecular interactions of complex chemical systems.

* Global optimization of extended molecular systems.

4. Computational Electromagnetics and High resolution multi-dimensional solutions of Maxwell's

Acoustics equations to define the magnetic fields about shipboard

antenna arrays, and the electromagnetic (EM) signatureq of

tactical ground and air vehicles, electromagnetic

performance and design factors for EM gun technology,

and electromagnetic signatures of buried munitions.

High resolution multi-dimensional solutions of the

acoustic wave equations in solids, liquids, and gases to

model the acoustic fields of bodies of water for underwater

surveillance and communications, the seismic fields of

soil for mine detection, and the acoustic shock waves of

explosions for anti-personnel weapons.

5. Climate&Weather/Ocean Modeling The modeling of the earth's climate and weather aimed at

improved prediction capability.

The modeling of various properties of the ocean (e.g.,

temperature, salinity, currents) to improve the processing

gain for acoustic anti-submarine warfare (ASW).

6. Signal/Image Processing • The extraction of useful information from sensor outputs

(sonar, radar, imaging, signal intelligence (SIGINT),

navigation), in real time.

• Usually such processors are aboard deployable military

systems and hence require ruggedized packaging and

minimum size, weight, and power.

- Also the research, evaluation, and off-line test data

processing for these applications.

7. Forces Modeling and Simulation/C4l - The use of command, control, communications,

computers, and intelligence (C41) resources to manage a

battle space.

- Faster-than-real-time large-scale simulations of complex

military engagements to facilitate mission rehearsal,

mission planning, and post-mission analysis.

- Collaborative planning/replanning to support real-time

operation decision-making.

18
Table 3. Computational Technology Areas (Concluded)

Area Description

8. Environmental Quality Modeling and • High resolution three-dimensional Navier-Stokes modeling

Simulation of hydrodynamics and contaminant and multi-constituent

fate/transport through the aquatic and terrestrial ecosystem

and wetland subsystems, and their coupled hydrogeologic

pathways, and their interconnections with numerous

biological species.

Used for stewardship and conservation of natural and

cultural resources, optimal design and operation of

installation restoration, and enhancement alternatives and

development of short- and long-term strategies for

integrated management in support of installation

environmental quality.

"* Work in the area of noise evaluation and abatement as well

as water quality models.

9. Computational Electronics and Nano- • Simulation of electronic devices in circuitry, including

electronics fault simulation.

"* Monte Carlo and nonlinear quantum transport

characterization in nanoelectronic devices.

"* Electronic structure calculations oi novel materials for

electronics applications.

4.2 Summary of Requirements Analysis

The DoD science and technology program has a large number of functional

requirements which can be satisfied by rapid increases in HPC capabilities. These

functional S&T requirements have been identified both as a set of grand challenge problems

spanning the nine computational science technology areas and as the S&T goals of each

individual DoD laboratory location. These functional S&T requirements result in overall

HPC performance requirements for FY94, FY96, and FY99 as summarized by Service in

Tables 4 and 5.

Table 4. Summary of Overall Unclassified Performance

Requirements (GF-Yrs)

ORGANIZATION FY94 FY96 FY99

TOTAL ARMY 297 3,829 27,119

TOTAL NAVY 138 1,196 '5,679

TOTAL AIR FORCE 180 2,032 19,668

TOTAL DNA 1 1 2

TOTAL 616 7,058 62,468

19
Table 5. Summary of Overall Classified Performance

Requirements (GF-Yrs)

ORGANIZATION FY94 FY96 FY99

TOTAL ARMY 21 266 1,735

TOTAL NAVY 5 25 158

TOTAL AIR FORCE 16 266 3,021

TOTAL DNA 1 0 0 0

TOTAL 42 557 4,914

Each Service has rapidly increasing requirements for HPC capability in its S&T

program over this five-year time period. As modeling and simulation activities play an

increasingly larger role in DoD's R&D program, driven by the need to maintain

technological superiority in this downsizing era, it becomes crucial that advances in HPC

capabilities are rapidly made available to the DoD S&T computational community. As the

tables indicate, the computational S&T community has significant requirements for and is

ready to use teracomputing capability and beyond within the next five years to advance the

S&T programs identified in this report so !hat DoD can maintain its superiority in

warfighting systems.

The Requirements Analysis has confirmed the need for a full range of HPC system

sizes, with users citing requirements from workstations, to clusters, to scalable parallel

systems, to conventional supercomputers to accomplish their S&T computational mission.

To use the high-end systems to their maximum effectiveness, many users require a variety

of individual and clustered systems to pre- and post-process large production computations

performed on the high-end systems. There is a common need for scientific visualization

systems to allow scientists and engineers to make effective use of computational results.

Providing a range of system sizes ensures ý'.at scientific and engineering problems of

varying sizes can be accomplished on the most appropriate system, thus preserving highend system resources for the largest problems.

It is widely accepted that no single architecture or system type will fulfill all of the

computational requirements of the DoD S&T program. Systems that provide significant

growth capability are essential to satisfying the rapidly growing computational S&T

requirements identified in the Requirements Analysis.

A comparison between Tables 4 and 5 clearly shows that the bulk of the HPC

requirements of the DoD S&T program is unclassified. The classified requirement is

probably understated, however, due to the scarcity of available classified HPC resources at

this time. Many scientists and engineers tend to compute problems that are unclassified and

use those results to extrapolate, when necessary, to actual systems whose attributes are

classified. Even though overall performance requirements for classified S&T computing

are a relatively small fraction of the total requirement, those classified requirements

represent a key ability for DoD to respond quickly in any time of crisis to developmental

and operational needs.

In addition to computational capabilities, other key hardware requirements such as

central memory, secondary storage, and archival storage were gathered and analyzed. A

requirement for very large central memory, secondary, and archival storage was validated

for any mode of operation of a large HPC system, whether it be to provide maximum

throughput for a large number of DoD S&T users representing a broad mix of jobs, or to

facilitate a limited number of very large grand challenge DoD S&T applications. Archival

20

i l I I I I
storage requirements will quickly expand to petabytes (1015 bytes) as more complex

computational problems are attacked, with resultant increase in output file size. Automated

* storage capabilities are essential to ensure timely access to data for the large and growing

computational S&T user base. Anticipated leaps in computational power will be essentially

wasted if these central memory, secondary storage, and archival storage capabilities do not

keep pace.

Software requirements were identified as the one key element in ensuring that DoD

* computational scientists and engineers would be capable of taking full advantage of the

vastly increased computational capabilities of new scalable architectures. It is essential that

mature operating systems and system tools be made available as soon as possible in a new

system's life cycle so that its capabilities can be fully exploited. In addition, robust

parallelization tools will help ensure compatibility of applications codes across different

computational platforms. These parallelization tools include new standard compilers,

• debuggers, performance monitors, and other parallelization aids. One promising method of

accelerating the transition of existing DoD applications codes to new scalable architectures

is teaming computer scientists with an expertise in parallel programming with DoD

computational scientists who are knowledgeable about the computational problems being addressed. In general, most computational scientists express a strong desire not to be

computer scientists, but to spend their time doing S&T in the field of their training. This

* teaming would enable the development of expertise in the exploitation of scalable systems

within the DoD S&T program, and could be facilitated by placing graduate and postdoctoral

students and other temporary personnel with such expertise from academic computer

science departments into DoD laboratories.

Networking is a critical ingredient in the overall ability to ensure accessibility to

* state-of-the-art HPC capabilities across the entire DoD computational S&T program. It is

clear th7e. every DoD S&T site will not be hosting a major HPC system and most sites will

necessarily access the high-end systems remotely. The data gathered in the requirements

survey indicate high performance and comprehensive networking capabilities are required

to support effective, remote use of HPC resources. A prevalent and preferred mode of

remote processing is to return all or most output data files to the local S&T site, enabling,

for example, postprocessing and visualization rendering. These output data files are

typically quite large and create large network throughput demands if timely transfers are to

be achieved. Additionally, an eventual goal of real-time, interactive visualization performed

on remote major resource center assets will require gigabit/second network bandwidth end

to end. Postprocessing output data files at local S&T sites further introduces a requirement

for large capacity archival storage resources at these local sites. A distributed data storage

* architecture, enabled by an effective high performance network, has the benefit of reducing

long-term storage demands at the Major Shared Resource Centers.

To take advantage of projected Defense Research and Engineering Network

(DREN) capabilities to meet these large networking requirements between Major Shared

Resource Centers, Distributed Centers, collaboration partners, and user sites, each site

* must establish and support a local networking infrastructure capable of interfacing with

DREN wide area services and be appropriately scaled to handle DREN-supported data

rates. These local networking capabilities are the responsibility of each local site and failure

to support an adequate local infrastructure will impede its scientists and engineers in their

efforts to exploit DoD HPC resources to their full potential.

* A final and critical aspect of networking is a classified capability. Classified S&T

computing represents a key DoD capability to respond quickly in times of crisis. Current

encryption technology severely lags network bandwidth performance and hampers the

21
ability of the DoD Modernization Program to provide remote, high performance classified

HPC capability. Until this encryption performance deficiency is resolved, there will

continue to be critical classified HPC requirements which can only be met in the near-term

with local classified resources. As part of the DREN initiative, developments in advanced 0

encryption technology will be closely monitored and earliest possible deployment of

approved products are planned.

Training and education requirements likewise focus on efficient use of new scalable

HPC architectures. These requirements include specific HPC systems and efficient parallel

programming techniques in general. Long-term, full-time programs will be necessary to

provide training and education in the use and programming of scalable systems.

Requirements also exist, but to a lesser extent, for systems software such as Unix, Fortran,

Ada, and C, and for effective use of applications codes. For all training and education, a

two-tiered system seems most effective: an introductory level for novices and an advanced

level for more experienced users. Most users prefer training that is provided in their local

areas, but recognize the need for a critical mass to ensure the cost effectiveness of that

training.

The overall conclusion is that a complete HPC environment must be provided to

take full advantage of HPC capabilities within the DoD S&T program. This environment

must include a variety of HPC systems to provide the proper computational platforms for

which computational problems will run most efficiently. Raw computational capability 0

must be balanced by a compatible central memory, secondary storage, and archival storage

capability. Systems software and efficient applications software must be available to allow

the DoD scientist and engineer to compute effectively on HPC platforms and thus perform

the DoD S&T mission. Special programs to accelerate the porting and development of

applications software to fully utilize new scalable architectures are essential for the S&T

computational program to stay abreast of new HPC capabilities. A reliable high-speed 0

network is the crucial link between computational scientists and engineers at DoD

laboratories and the high-end HPC resources located at DoD Major Shared Resource

(Centers. Training and education is essential to keep the user base technically current.

22

i i a i l l i i0
0

5. Implementation Strategy

0 5.1 Consolidation of DoD's HPC Infrastructure

The planned HPC infrastructure will consist of both centralized HPC resources

located at MSRCs and decentralized HPC resources located at Distributed HPC Centers.

As discussed earlier, these resources will be available to S&T researchers and engineers

throughout the DoD laboratories and R&D centers via DREN. The roadmap/timetable for

* advancing the capabilities of this infrastructure is depicted in Figure 3.

Efforts conducted in FY93 and FY94 represent initial steps in advancing the HPC

capabilities in the infrastructure. Providing a large Cray C90 system at CEWES in FY93

represented the first step in establishing a MSRC. Providing other laboratories and centers

with new scalable systems and upgrades to existing scalable systems in FY93 was the first

* step in establishing Distributed HPC Centers. Establishing the Interim DREN by

integrating the existing Army Supercomputer Network and the Air Force Supercomputer

Network, and linking them to Navy and DNA centers, initiated the sharing of these HPC

resources across all the Services and participating agencies. This action began the

replacement of individual Services' and Agency's HPC centers with a consolidated DoD

HPC infrastructure.

In FY94 these actions are continued with the initiation of an additional MSRC and

several new Distributed HPC Centers, as well as upgrades to the first MSRC and some of

the existing Distributed Centers. The major efforts to initiate the long term strategy will be

conducted in FY95 and continued in FY96 and beyond. These actions are described in

more detail in the following sections.

5.2 FY93 Efforts

Section 1.2 outlined the actions taken to provide additional HPC capabilities to the

DoD S&T community as approved in the FY93 Implementation Plan using the combined

* funds provided by Congress in FY92 and FY93. Those actions have made significant

enhancements to the DoD S&T program and have begun to satisfy the DoD S&T

computational requirements as summarized in Chapter 4.

The addition of the Cray C916/512 and the conversion of the existing Y-MP8 at

CEWES to DoD operations represents approximately a doubling of the vector HPC

* capability previously available to the DoD S&T program. While its overall HPC

performance (16 GF-yrs) is only a small fraction of the 616 GF-yrs unclassified HPC

requirements for FY94 stated in Table 4, it has allowed DoD researchers to attack

significantly larger problems than they could previously. One crucial limitation has been

the amount of central memory, secondary storage, and archival storage available on that

system.

23

0
LS

0 C

UE

A~ I-

_j 0.

(A4-i

cll

- ~~ ~ u00 ~ c PC3- (

0W o

91 USDA maluc20

24 ~
The set of scalable parallel systems acquired or enhanced as part of the FY93

Implementation Plan represents, in the aggregate, over 200 GF-yrs of overall HPC

performance available to DoD S&T users. This capability is a significant fraction of the

total HPC requirements for FY94 (Tables 4 and 5). In addition, the Intel Paragon XP/S-25

operated at NCCOSC in a classified mode provides a substantial part of the 42 GF-yrs of

classified S&T requirements, but not all classified users are capable of taking advantage of

that particular architecture. Many of the scalable parallel systems implemented in FY93

lack adequate system peripherals and central memory to allow efficient use by multiple

• users.

Interim DREN has proven to be highly successful in allowing users at DoD user

sites to access the HPC systems implemented in FY93. Key limitations include insufficient

bandwidth to move large files created by some user applications and the slower

transmission speeds available to classified users who must depend on end-to-end

* encryption devices.

The MOA with NCHPC has recently begun to bear fruit for DoD S&T users. The

NCHPC collaboration groups have held initial meetings and begun to formulate plans for

scalable software development efforts in each area. One indirect benefit of the NCHPC

collaboration groups has been the incentive for internal DoD researchers within each

* technology group to meet and collaborate in ways that have not occurred previously.

Several DoD research projects have made use of NCHPC computational resources that

have allowed scaling problems to larger sizes than would have been possible on DoD HPC

systems alone. To date, this added capability represents only a tiny fraction of the total

DoD HPC S&T requirements, but further exploitation of NCHPC assets, based on the FY

92/93 investment, will allow unique opportunities for specialized DoD applications.

5.3 FY94 Efforts

An Implementation Plan has been developed in close coordination with the Services

and S&T user community to apply available FY94 funds to immediate upgrades and

• expansions, to alleviate current saturation, and to make existing systems more effective.

The DoD S&T community has significant requirements to solve research problems in the

memory and compute intensive domain for which high performance computing is required.

Table 6 indicates the HPC Center deployments that represent critical S&T requirements that

can be addressed immediately through FY94 acquisitions. Deployments to each of these

centers were judged using the criteria of S&T requirements, economic soundness

* management commitment, and information infrastructure. Execution of the plan will

involve the use of both new and previously competed contracts and be in full compliance

with Congressional direction for FY94. The Implementation Plan contains a system

summary, rationale, acquisition plan, and funding allocation for each of the planned FY94

acquisitions. A similar Implementation Plan will be issued early each FY to provide like

details for that FY's program execution.

25

0. . . ..... . . . l ll l l l •l llll
Table 6. Summary of Planned FY94 HPC Systems

SITE SYSTEM FY94 ACTIONS

CEWES C90 Upgrades to central memory, secondary storage, and archival

storage

NAVO C90 New Cray C91611024

Army TBD New scalable HPC system

Research Lab _ _ _ _ _ _ _ _ _ _ _ _

ASC Paraon Upgrade to distributed memory and secondary storage

AFDTC Cray T3D New Cray T3D scalable system

MHPCC IBM SP2 Upgrade of performance capability from 60 to 100 GF for

unclassified system; upgrades to distributed memory and

secondary storage

NCCOSC Paragon Upgrade of distributed memory and secondary storage;

addition of file archiving system

SPP-1 Upgrade of distributed memory and other peripherals;

additional processors

Naval CM-5 Upgrade to distributed memory and secondary storage;

Research Lab additional processors

_ TBD New scalable HPC system

Rome Lab TBD New scalable HPC system

In addition to addressing some specific local requirements detailed in the

Implementation Plan, these FY94 actions address a broader base of requirements as

summarized in Chapter 4 and as determined by initial experiences with systems

implemented in FY93. Several existing systems, including the C90 at CEWES, the IBM

SP2 at the MHPCC, the Paragons at NCCOSC and ASC, and the CM-5 at NRL, are

having either central memory or peripheral storage upgraded to make those systems more

efficient and effective for current and projected user bases. In each case, user requirements

and experiences dictate that these enhancements are essential to continued efficient

operations. In each of these cases, the initial FY92/93 investment provided the basic

system, but it was realized at that time that FY94 upgrades would have to be made to

complete a balanced system.

New capacity and capability is also a component of the FY94 actions. The addition

of a new Cray C916/1024 and the conversion of the existing Y-MP/8 to classified DoD use

at the Naval Oceanographic Office (NAVO) will further enhance DoD vector HPC

capability, alleviate some of the oversubscription of existing systems at CEWES, allow

more extensive "grand challenge" computations on these systems, and provide the first

DoD vector HPC classified capability. All of these improvements in the ability to satisfy

current user requirements are urgently needed.

New scalable parallel systems at AFDTC, ARL, NRL, and Rome Laboratory, with

computational enhancements planned to existing scalable parallel systems at the MHPCC

and NCCOSC, will add over 100 GF-yrs to the overall HPC performance available to DoD

S&T users. When the enhancements are combined with the capabilities provided in FY93

and the new vector HPC capabilities at NAVO in FY94, the total performance capability

available to DoD HPC researchers will be approximately 400 GF-yrs. This total represents

60 per cent of the total FY94 S&T requirements given in Tables 4 and 5.

26

i I I I i
5.4 Planned FY95 Efforts

To fulfill the goal of providing HPC capacity and capability, and to address the

software issues in training and education required to effectively use the HPC resources,

the program will continue its plans to establish DoD Major Shared Resource Centers

(MSRCs) located within the Services and participating Agencies. The MSRCs will provide

complete HPC environments and include various types of computing systems, scientific

visualization capabilities, extensive peripheral and archival storage, and expertise in use of

these systems. The MSRCs will support the wide variety of research and development

problems arising from the science and technology programs supporting DoD's weapons

development and warfighting support systems. The MSRCs will provide the computer

and computational sciences expertise to allow all of the DoD laboratories to advance their

capability in science and technology. The types of computer systems in the MSRCs will be

determined by user requirements and may differ from one MSRC to another.

Actions to establish MSRCs are:

The program has selected four sites to become MSRCs. Criteria for selection

were based on R&D synergism, infrastructure, affordability, management

commitment, and experience in serving remote users. The selected sites for

MSRCs are:

-- Army Corps of Engineers Waterways Experiment Station, Vicksburg, MS

-- Naval Oceanographic Office, Stennis Space Center, MS

* -- Army Research Laboratory, Aberdeen Proving Ground, MD

-- Aeronautical Systems Center, Wright-Patterson Air Force Base, OH

In FY95 the program office is planning a full and open competition for

acquisition of HPC capabilities, based on requirements of the users, to

0 populate the MSRCs. The program anticipates multi-awards to teams

including academe and industry, led by system integrators.

In addition to MSRCs, the program will continue to deploy systems to Distributed

HPC Centers to fulfill special R&D requirements where the HPC systems are critical to

success of S&T efforts and where special collections of HPC expertise can be leveraged for

•9 benefit of the entire program. Multiple systems will be acquired using several competitive

acquisition vehicles.

Figure 4 depicts the DoD HPC laboratory modernization of both MSRCs and

Distributed Centers. All functions are performed by the DoD HPC MSRCs. The shaded

functions in the figure represent those functions expected of the Distributed HPC Centers.

S In FY95 the program will initiate a full and open competition for acquisition of

network services to provide OC3-level service (155 Mbps) to selected sites and minimum

T3 (45 Mbps) service to all other DoD sites participating in the program. Award of a

contract for DREN network services is planned for the third quarter of FY95. These

services will replace the Interim DREN, which as described earlier, is comprised of

dedicated leased circuits. Anticipated benefits include higher performance and improved

cost effectiveness.

27

"0 •I|
DoD HPC Laboratory Modernization

DoD Distributed DoD HPC MSRCs HPC Centers

Army Corps of Engineers Waterways Expewment Station

Naval Oceanographic Offce Stennis

Army Research Laboratory ,t Planned*

Wright-Patterson Force Base

Supporting National HPC Infrastructure AHPCRC AFDTC

Academe Industry Other NAHPCC Ro-e Lab

Government NRL

Agenaces I _ _

*Requirements at other sites are being considered for establishing additional

Distributed HPC Centers

Figure .oD .PC 4. M..... Storage Lbrty Viz

C TR&D Applications . ....

0 Parallel I ICode/

N [ Education --- " Prog ram" -J- l-ove rsio

Figure 4. DoD HPC Laboratory Modernization

The program will leverage existing resources and university relationships for HPC•

capability, training, applications and algorithm development, and software tools. It will

continue to coordinate with NCHPC to take advantage of available training through the

NCHPC members, continue software collaborations, and make use of the NCHPC

software repository and NCHPC scalable hardware. Finally, the program is also planning

to implement an internal DoD scalable applications software initiative to enable DoD

researchers to begin to overcome the software impediments to effective use of scalable

HPC on DoD mission problems.

The summaries of classified and unclassified requirements in Tables 4 and 5

indicate an order of magnitude growth from FY94 to FY96. The planned FY95 activities,

particularly those establishing and equipping the MSRCs, must put in place a large share of

HPC systems and environments targeted to meet those FY96 needs. The total requirement

of over seven TF-yrs will be difficult to meet with planned FY95 activities alone. Current

projections indicate that the initial acquisitions for the MSRCs in FY95 could add several

hundred additional GF-yrs to the available DoD capability, resulting in a total capability

approaching one TF-yr.

28

;' •" • I I P
5.5 Long Term Implementation Strategy (FY96-FY99)

"" Continue to evolve and modernize MSRCs.

"" Continue to acquire special systems for Distributed HPC Centers with critical

requirements.

"* Acquire adequate data storage capabilities at appropriate distributed centers.

* * Evolve the network to gigabit per second (and beyond) performance levels to be

consistent with the performance levels of the HPC systems, and as user

requirements dictate.

"* Develop mechanisms for creation of interdisciplinary teaming which leverages

academe and industry expertise in HPC.

"* Promote software tooling and scalable application and algorithm development

initiatives.

If it is difficult to meet the FY96 S&T computational requirements, it is even more

difficult to satisfy the projected FY99 computational requirements of 70 TF-yrs. These

* planned out-year actions are likely to result in a total DoD S&T performance capability of

20-30 TF-yrs. This level of capability would represent slightly less than half of the

projected computational requirements.

5.6 Funding Needs

Table 7. Funding Profile ($M)

FY94 FY95 FY96 FY97 FY98 FY99

MSRC Systems1 73.2 80.0 80.0 80.0 95.0 100.0

Distributed Center SystemsI 49.1 40.0 40.0 40.0 55.0 55.0

Software, Appi Support, Education 0.82 20.0 25.0 30.0 40.0 50.0

and Training Initiatives

Netwokding 10.8 20.0 30.0 40.0 45.0 50.0

Program Management and 3.0 3.0 3.2 3.4 3.5 3.8

* Administration

Sustainment of Operations 6.9 20.0 32.9 44.0 49.8 55.0

TOTAL 143.83 183.0 211.1 237.4 288.3 313.8

1These systems recognize that there are specialized requirements both mission and contractual (such

as DNA's contractual requirements) which must be met by the HPC MP and provisions will be

made to meet these needs

2 Includes $0.8M COTS software

3 Available funds after undistributed cuts ($146.IM appropriated)

29
6. Acronyms and Definitions

AFB Air Force Base

AFDTC Air Force Development Test Center

AHPCRC Army High Performance Computing Research Center

algorithm procedure designed to solve a problem

AMOS Air Force Maui Optical Station

APG Aberdeen Proving Ground

ARL Army Research Laboratory

ARPA Advanced Research Projects Agency

ASC Aeronautical Systems Center

ASD Assistant Secretary of Defense

ASW anti-submarine warfare

CAA Concepts Analysis Agency

C31 Command, Control, Communications, and Intelligence

C41 Command, Control, Communications, Computers and

Intelligence

CEWES Army Corps of Engineers Waterways Experiment Station

Computational Science The systematic application of computing systems and

Engineering computational solution techniques to mathematical models

formulated to describe and simulate phenomena of scientific

and engineering interest.

CPU central processing unit

DDR&E Director of Defense Research and Engineering

DNA Defense Nuclear Agency

DoD Department of Defense

DOE Department of Energy

DREN Defense Research and Engineering Network

FFRDC Federally Funded Research and Development Centers

31
FY fiscal year

GF gigaflops

gflop gigaflops

Grand Challenge A fundamental problem in science and engineering with

broad economic and scientific impact, whose solution can be

advanced by applying HPC techniques and resources.

HBCU Historically Black Colleges and Universities

Heterogeneous System A distributed system that contains more than one kind of

computer.

HPC High Performance Computing 0

Covers the full range of advanced computing systems including workstations, networks of workstations with

servers, scalable parallel systems, vector parallel systems,

and more specialized systems. Scalable input/output

interfaces, mass storage systems, and archival storage are 0

components of these systems. Included also are system

software and software development environments that enable

users to view their workstations and the rest of their

computing environments as a unified system.

HPCC High Performance Computing and Communications 0

HPCM High Performance Computing Modernization

HPCWG High Performance Computing Working Group

IDREN Interim Defense Research and Engineering Network 0

Internet The global collection of interconnected multi protocol

computer networks.

LES large eddy simulation

MHPCC Maui High Performance Computing Center

MI Minority Institutions

MOA Memorandum of Agreement

MSRC Major Shared Resource Center

NAVO Naval Oceanographic Office

NASA National Aeronautics and Space Administration

NCCOSC' Naval Command, Control and Ocean Surveillance Center

32

Si I0 I I I I 
NCHPC National Consortium for High Performance Computing

NRL Naval Research Laboratory

NSF National Science Foundation

NSWC Naval SurfaceWarfare Center

NUWC Naval Undersea Warfare Center

OGA other government agency

ONR Office of Naval Research

R&D Research and Development

RDT&E Research, Development, Test, and Evaluation

S&E scientists and engineers

SRC Shared Resource Center

S&T science and technology

TBD to be determined

TF teraflops

WPAFB Wright-Patterson Air Force Base

33