# Presentation: 028976

## 30 Years of Navy Modeling and Supercomputers: an Anecdotal History

- Tom Rosmond
- Marine Meteorology Division
- Naval Research Laboratory
- Monterey, California USA

## Outline

- Before my Time: 1961-1974
- My Early Years: 1974-1980
- The Golden Years of NWP: late 1970’s-early 1980’s
- Up to the Present
- Conclusions and Thoughts for the Future

## Before my Time: 1961-1974

- 1959: Capt Wolff sets up shop on NPS Campus, Monterey
-     CDC 1604: Seymour Cray’s first design with CDC (first supercomputer?)
- 1961: Establishment of Fleet Numerical Weather Facility (FNWF)
- 1964: Routine dissemination of numerical products
-     Successive correction analyses of SLP, upper air heights
-     NH Barotropic and thickness advection models

- THE BEGINNING

## Before my Time: 1961-1974

- 1967: FNWC procurement of CDC 6500 to replace 1604
- Dual-processors, comparable to 386 microprocessor in FP performance
- Supercomputer of time was Seymour Cray’s CDC 6600 (NMC in 1966)
- 1969:  Second 6500 acquired
- 1970:  Northern Hemisphere Primitive Equation (NHPE) model operational   on 4 processors.
- World’s first multi-processor production code

- CDC 6500’s and NHPE

## Before my Time: 1961-1974

- Developed by P. Kessel and F. Winninghoff
- Processors shared model data through extended core storage (ECS)
- OS modifications to allow processor synchronization
- Similarities to both shared memory (OpenMP) and distributed memory (MPI) programming models
- Parallel efficiency = ~ 75%
- An impressive achievement that is certainly underappreciated
- An example of the struggle it was to fit models onto these early systems

- 4-processor 6500 NHPE

## My Early Years: 1974-1980

- 1My organization: EPRF     NEPRF     NOARL     NRL-Monterey

- 1974: I joined the Environmental Prediction Research Facility (EPRF)1
- Little involvement in FNWC2 model development
- Access of FNWC computer systems (6500’s, 1604)
- 1976: NEPRF Numerical Modeling Department formed at request of FNWC CO Capt. R. Hughes
- He realized that FNWC was unable to maintain R & D continuity needed for NWP system development.
- Need for a global NWP capability was major motivation, and he was committed to getting computer system to support it.
- 1976: FNWC acquired CDC CYBER 175 (similar to CDC 7600)

- 2Fleet Numerical: FNWF     FNWC     FNOC      FNMOC

## My Early Years: 1974-1980

- NEPRF Global NWP System Development

- 1976: Navy Operational Global Atmospheric Prediction System (NOGAPS)
- UCLA General Circulation Model
- Barnes Successive Corrections Global Analysis
- Variational Initialization with Balance Equation Constraint
- 1977: We also had spectral model dynamical core
- UCLA GCM physics
- Sat on shelf for several years in favor of UCLA GCM based system
- 1980: Prototype NOGAPS running on CYBER 175

## My Early Years: 1974-1980

- 1977: Benchmarking for FNWC procurement
- NCAR Cray-1, first system outside Los Alamos?
- UCLA GCM 2.40 x 3.00, 6 levels
- First introduction to vector programming
- Crude compiler + non-vectorizable code = poor performance
- Overly conservative target performance,  CYBER 203 could compete
- Allowed subsequent CDC success with CYBER 205
- 1980: CYBER 203 delivered to FNOC
- Heroic work by CDC to get UCLA model to run fast
- 1982: CYBER 203 replaced with CYBER 205

## The Golden Years of NWP1st half: late 70’s – mid 80’s

- Supercomputers were cheapest computers you could buy (price/performance)
- Cray: Cray-1, XMP, etc
- CDC: CYBER 203/205, ETA-10
- IBM: 360-195VP
- Development of global NWP forecast systems
- Dominance of spectral models
- Increased realization of importance of data assimilation
- Establishment of ECMWF
- “Raised the stakes” in operational NWP
- Accelerated progress
- Provided gold standard

## The Golden Years of NWPNavy operational models

- NOGAPS
- UCLA-GCM - 2.40x3.00 x L9 : 1-pipe/8Mbyte C205 : ~25min/fcstday
- Spectral – T47L18 : 2-pipe/32Mbyte C205 : ~6min/fcstday
- Spectral – T79L18 : 2-pipe/32Mbyte C205 : ~25min/fcstday (32bit)
- Spectral – T79L24 : 4-pipe/64Mbyte C205 : ~12min/fcstday (32bit)
- NORAPS (developed by Rich Hodur, NEPRF)
- Globally relocatable regional model
- 6-8 areas run operationally
- Varying resolution, domain and grid sizes
- e.g. ran over South Atlantic during Falklands war (1980)

## Some special comments about the CYBER 205

- CDC aggressively pursued meteorology market, both operations and research
- FNOC, NMC, UKMO, GSFC, GFDL, etc
- CRAY systems were better general purpose, but 205 excelled on our applications
- Initially 205 was difficult to use, compiler/user software was rudimentary, *but*
- Language extension showed how machine worked
- Rich array of exotic vector hardware instructions
- System software matured
- 32 bit/64 bit floating point support
- Easy mixing of Fortran and explicit vector instructions, we could get “close to the hardware”
- Spectacular percentage of peak performance possible

## The Golden Years of NWP2nd half: mid 80’s – early 90’s

- Introduction of multi-processor systems
- Divide and conquer
- X/MP, Y/MP, ETA-10
- Multi-tasking/vectorization programming model (parallel/vector)
- Spelled the end of single processor supercomputers, e.g. CYBER 205
- Important changes in technology
- Price/performance advantage of supercomputers ending
- Introduction of desktop workstations
- Many people didn’t need supercomputers, just cheap computing
- Shrinking supercomputer market

## Up to the Present:early 1990’s

- 1990: Introduction of CRAY C90
- The “best” supercomputer ever?
- Run by a higher percentage of NWP operational/research centers than any system before or since.
- Parallel/vector programming model very user friendly
- Easy to get high percentage of peak performance
- 1991: 8 processor C90 at FNMOC
- NOGAPS: T159L24 – 6 processors : 10 min/fcstday
- 1996: 8 and16 processor C90’s at FNMOC
- NOGAPS: T159L24 – 12 processors : 6 min/fcstday
- 1997: COAMPSTM  replaced NORAPS as Navy’s regional forecast system

## Up to the PresentMid to Late 1990’s

- Beginnings of dramatic changes in supercomputer industry
- Scalable commodity based architectures appearing, e.g. CRAY T3E
- Powerful workstations replacing supercomputers for many applications
- End of domination by American supercomputer vendors
- CDC/ETA were long gone
- CRAY sold to SGI
- MTTB (mean time to bankruptcy) very short for new companies
- Rise of Japanese vendors to dominate NWP marketplace, at least outside U.S.
- Fujitsu
- NEC

## Up to the Present2000 to today

- Proliferation of commodity based, scalable architectures for NWP applications
- Many T3E’s still in use
- SGI:  Origin 2000, Origin 3000
- IBM: SP3, SP4
- Linux clusters are also viable alternatives, especially when price/performance is overriding issue
- But, rumors of the demise of vector architectures are exaggerated
- 2000: NEC only vendor with traditional vector architecture, SX-6, but
- 2002: Resurrected CRAY, Inc introduced X-1
- Price/performance vs ultimate performance is central question concerning long-term prospects of these systems

## Up to the Present2000 to today

- 2001: FNMOC replaced C90’s with O3000’s
- 1152 total processors
- NOGAPS: T239L30 – 120 processors: 6 min/fcstday
- COAMPSTM : 8-11 areas – 40-180 processors: 10-30 min/fcstday
- 2003: Operational NAVDAS/NOGAPS (3DVAR): 60 processors
- 2004: Direct radiance retrievals with NAVDAS
- 2004: Currently under development at NRL, Monterey
- T479L54 semi-Lagrangian NOGAPS
- NAVDAS-AR: 4DVAR extension of NAVDAS
- COAMPSTM/WRF: more areas, higher resolution
- Clearly computational requirements are never satisfied!!!

## Conclusions and thoughts for the future

- Vector architectures will continue to be viable candidates for NWP applications
- Sharing some features with commodity systems, e.g. caches
- When ultimate performance overrides price/performance
- Will be part of heterogeneous computing environments
- Scientific computing ( and therefore NWP) is now niche market
- Be thankful that video game applications share many of our requirements
- Consumer based industry has driven some hardware costs to astonishingly low levels, e.g. hard drives
- We must work with vendors to ensure that our requirements are not forgotten
- Fortran compilers
- High-performance interconnects