# Presentation: 537899

## NMC/NCEP Supercomputers and the Models that Consumed Them:  A sordid tale of codependenceGeoff DiMegoKen Campana and Joe Irwin12 May 2004

**where the nation’s climate and weather services begin**

## Forecast Skill Trend

_**Forecast Skill Trend**_

## How We Got There

- Finer horizontal & vertical resolution
- More accurate formulations with fewer approximations / simplifications
- Better physics i.e. the representation of small-scale processes like radiation, clouds, precipitation & turbulent exchange of heat, moisture and momentum
- More accurate analysis and 4-dimensional data assimilation producing improved initial conditions
- Increase in available observations
- I’ll say very little about 2-5 since I’m concentrating on the connection to computers, but all four of them were enabled by increases in data processing and computer power.

## Peak Performance Trend

## The Pre-DiMego Years

## Beginnings & First Transitions

- Vilhelm Bjerknes

- who, in 1904, declared the problem of  prognosis as the integration of the equations of motion for the atmosphere ...

- that future states of the atmosphere are dictated by its detailed initial state and known boundary conditions. His students at the Bergen School were strongly influenced by Bjerknes and would become future giants of our discipline: C.-G. Rossby, A. Eliasen and R. Fjortoft. (see Kalnay Cambridge Press)

## First Attempt

- L. F. Richardson

- who, in 1922, laid out a numerical model and performed the first computations (by hand). Lacking tools needed

- to perform enough calculations precisely enough,

- the forecast was a bust, in part, due to too long of a time step and the accumulation of truncation errors.

## 1906: De Forest Audion Vacuum Tube

- through _FORTY YEARS_ of countless developments, leads to ...

## 1946  ENIAC

- Jules Charney, R. Fjortoft and John von Neumann made the first 48 hour barotropic filtered model forecast on ENIAC in 1949

- Electrical Numerical Integrator And Calculator

## 1952: IBM Unveils the IBM-701

- 1952: IBM 701 - The Company's First Fully Electronic Computer
- Thomas Watson, Sr. (seated) the CEO of IBM in 1952 using the IBM 701 computer, the company's first fully electronic model. This computer also had the ability to read/write magnetic tape, but at this stage it still relied mainly on punched cards for I/O.  A total of _nineteen_ 701's were installed during its three year lifespan at a monthly lease rate of about $15,000.
- 1954: Joint Numerical Weather Prediction Unit is formed ... for which

- 1 Kflops Class I

## Weather Bureau Buys Last IBM-701

- Number Shipped to               Date    Note
-  1  IBM World Hqtrs, NY   Dec. 1952
-  2  Univ of Cal. Los Alamos   Mar. 1953  (a)
-  3  Lockheed Aircraft, CA     Apr. 1953  (b)
-  4  Nat’l Security Agy, DC    Apr. 1953
-  5  Douglas Aircraft, CA      May  1953  (c)
-  6  General Electric, OH      May  1953
-  7  Convair, Fort Worth       Jul. 1953
-  8  U.S. Navy, Inyokern, CA   Aug. 1953  (d)
-  9  United Aircraft, CT       Sep. 1953
- 10  North American Av., CA    Oct. 1953  (e)
- 11  Rand Corp., CA            Oct. 1953  (f)
- 12  Boeing Corp., WA          Nov. 1953  (g)
- 13  Univ of Cal. Los Alamos   Dec. 1953
- 14  Douglas Aircraft, CA      Jan. 1954  (h)
- 15  Naval Aviation Supply, PA Feb. 1954
- 16  Univ of Cal. Livermore    Apr. 1954
- 17  General Motors Corp., MI  Apr. 1954
- 18  Lockheed Aircraft, CA     Jun. 1954  (b)
- _19  U.S. Weather Bureau, DC   Feb  1955  (i)_

- (a) Used for hydrodynamics calculations
- (b) Mathematics Analysis Department used the 701 for problems in aircraft design, such as aerodynamic performance and stability, thermal dynamics and structural and flight dynamics
- (c) Arrived on May 23, 1953, aboard a DC-6A aircraft. The 701 was used to get the DC-7 into production months ahead of schedule. It solved engineering & scientific problems on Douglas commercial aircraft,including DC-6B, DC-7, DC-7C & development of DC-8
- (d) Used to calculate rocket and missile performance and to simulate flight conditions of these devices at the U.S. Naval Ordnance Test Station China Lake
- (e) Handled engineering problems, from basic configuration selection through aerodynamic and structural design to the analysis of flight test data
- (f) Used to solve wide variety of problems in economics, mathematics, aircraft, missiles, electronics, nuclear energy and social sciences
- (g) Used to assist engineers and designers in solving problems in aerodynamics, stress and structural development, and flight testing of supersonic and jet aircraft and guided missiles
- (h) Solved engineering problems on U.S. Navy A3D Skywarrior, A4D Skyhawk and F4D Skyray programs, and USAF C-133 and RB-66 programs
- _(i) Produced from spare parts_

## Fir$t Computer Card$

- In 1890, Herman Hollerith made the size of the data-processsing card
- (later _commonly known as the IBM card_) equal to the dollar bill of that time - allowing the reuse of existing filing bins and similar equipment. Shown is a 1930 Silver Certificate that still had the same size.  In 1929 the dollar bill was reduced by 20% in both dimensions.

## 1955-56 - First Production Suite 

- _Baroclinic 300 km / 3-layer CONUS to 36 hr_
- _1500z run started 2100z_ finished 2230z
- _Thermotropic 375 km / 2 layer No. Am to 36 hr_
- _0300z run started 1000z_ finished 1230z
- _Barotropic  600 km / 1 layer ext No. America to 72 hr__ _
- 0300z run started 1230z finished 1300z
- The 6-7 hour lag between ob time and model start time highlights the need for more rapid transmission and processing of observational data
- LEGEND: _green = new_,  blue = upgrade, black = original, and red = models in development that are drivers for the next computer acquisition

## 1955 Model Domains

- Baroclinic CONUS

- Thermotropic No. America

- Barotropic ext No. America

## 1956 Model Guidance

## 1956 Verifying Analysis

## Analysis Assemblyline

_**Analysis Assemblyline**_

## 1957 USWB Buys IBM-704

**First to use core memory which was fast,reliable but initially $$$**

**One of ~4000 Logic Modules**

- 8 Kflops  Class I

## 1957 IBM-704

-   Chief architect was Gene Amdahl
-   IBM's first commercial machine with _floating-point hardware_
-   First "modern" machine with: index registers, random access (core) memory, magnetic tape and card input and output, _line_ _printer output_ - console had programmable lights & sense switches but there was still no operating system
-   First widely-used machine to use a high-level programming language - FORTRAN
-   FORTRAN (FORmula TRANslation) was written by one of the IBM 704 designers - first compiler was so efficient, it took 15 years for it to be replaced, could use magnetic tape for intermediate storage (i.e. before availability of fast affordable disk), but could only read 72 of the 80 columns on card (hardware limitation)

## 1957 IBM-704 Line Printers

- What would become the ubiquitous IBM line printer was a modified 407 accounting machine printer, which ran at 150 lines per minute. The printers were able to print lines of up to 132 characters on 17-inch-wide paper

## 1000 mb prog from grid-print “contouring”

## 1957-62  Production Suite 

- Baroclinic _replaced in 1958 by 2-layer CONUS_
- 1500z run started 2100z finished 2230z
- Thermotropic 375 km / 2 layer No. America
- 0300z run started 1000z finished 1230z
- Barotropic  upgraded in 1957 to 381 km using 1977 point No. Hemisphere octagon
- _First objective analysis_ and better numerics
- According to Shuman (1989) this is the workhorse model of the day
- Cressman filtered baroclinic and Reed surface

## 1947 - First Transistor 

- The first transistor (TRANSfer resISTOR) was invented at Bell Laboratories on December 16, 1947 by William Shockley (seated), John Bardeen (left) and Walter Brattain (right).

## Tubes to Transistors IBM-704 to IBM-7090

- A _1960_ successor to the tube-based IBM 704, the IBM 709 introduced distinct input-output channels, that allowed input output to proceed in parallel with computation.  At government request IBM built a transistorized version,
- first designated to be the 709T.  This machine, renumbered 7090, became
- the prime IBM scientific computer until the 360-series arrived in 1966.

## “BIG IRON”

- IBM's "big iron" Data Processing Division was formed in 1956 to focus on the development and marketing of mainframe products. Within a short time, the company had developed:
- the first commercial airline reservation system (SABRE), using 1958’s IBM 7090 data processing system
- the Stretch supercomputer (IBM 7030) delivered to Los Alamos in 1961 which was 200 times faster than the IBM 701, 40 times faster than the IBM 709 and 7 times faster than the IBM 7090
- the stretch became the IBM 7094 and was "basis" for CDC 6600
- Most of the old big-iron machines were scrapped long ago for precious metals content - check out the Stanford site: http://www.thegalleryofoldiron.com/

## 1960 USWB Buys an IBM 7090

- 67 Kflops Class I/II

## 1963 USWB Buys an IBM 7094

- 100 Kflops  Class II

## 1962-65 NMC Production Suite 

- 2-layer Baroclinic _replaced in 1962 by Cressman filtered 381 km/3-layer N.Hemis octogon to 36 hr_
- Barotropic  381 km No. Hemis to 72 hr
- Thermotropic_ replaced in 1962 by Reed Surface Model 381 km / 2 layer N.Hemisphere to 36 hr_
- All models initialized with objective analysis
- Development of hemispheric and regional Primitive Equation models awaiting bigger computers to run them on

## 1964 - CDC 6600

-  Designed by Seymour Cray with solid state components using a sophisticated Freon cooling system and 60-bit word (vs 36-bit IBM).
-  First “supercomputer” performing 3 mips with 10 ns clockspeed Fastest computer in the world until 1968 surpassed by CDC 7600
-  Speed, in part, came from the computer ́s distributed design, which had 10 small computers, known as peripheral processors, funneling data to a large central processing unit using reduced instruction set (RISC) many years before such a term had been invented
-   First to have a CRT console (CDC checkout engineers created
- computer games such as Baseball, Lunar Lander, and Space Wars, which became incentives for getting the machines operational. These are thought to be the first computer games that used monitors.)

## 1966 USWB Buys a CDC 6600

- 3 Mflops Class III

- NCEP converts from machine language to FORTRAN – first MAJOR conversion effort!

## 1966-73 NMC Production Suite 

- Cressman 3-layer & Reed Surface _replaced in 1966 by Shuman-Hovermale  PE model 381 km / 6 layer N.Hemis 53x57 to 36 hr_
- Barotropic 381 km N. Hemis to 72 hr
- _In 1970 Barotropic extension of PE 84-144 hrs_
- _In 1971 Limited-area Fine Mesh of Howcroft 190.5km /  6 layer North America to 24 hr_
- _TDL Wave model 381 km N.Hemis to 36 hr_
- Development of global analysis and modeling systems awaiting bigger computers to run them on

## Well...this is another FINE MESH...

## Some Firsts & Lasts

- Cressman makes changes on a Friday
- Codes bomb all weekend – this is the LAST time operational codes are changed on a Friday
- Soon after, FIRST meeting of _CAFTI_ (Committee for Analysis and Forecast Technique Implementation) to “manage” change

## Transistors to Integrated Circuits

- Jack Kilby, an engineer at Texas Instruments, completed his first device (pictured) on September 12, 1958 which was actually constructed on germanium rather than silicon, as he could not find a suitable piece of silicon at the time. But the integrated circuit was fully functional, and TI officially announced it in January 1959.

- 1958: The first Integrated Circuit is invented by Jack Kilby of Texas Instruments

## Cost Drop With Technology Advance 

## 1971 IBM 360-195

## 1972 NWS Buys IBM 360-195

- ESSA’s NWS buys 2 IBM 360-195’s and gets a third from GFDL in 1973 - first interactive development via IBM’s TSO

- 18 Mflops Class IV

## 1974-78 NMC Production Suite 

- _Flattery Hough Analysis for PE (‘74)_
- _Global PE of Stackpole 2.5 deg / 8 layer analysis cycle every 12 hours_
- LFM extended from 24 to 36 (‘75) to 48 hr (‘76)
- Global PE 2.5 deg / 9 layer analysis cycle 12 hrly (’76)
- _Vanderman PE 2.5 deg / 3 layer global to 242hr__ __(‘77)_
- PE 190.5km / 7 layer N.Hemis (129x129) to 84hr (‘78)
- Barotropic _extension 381 km N.Hemis 84-240 hr (’78)_
- _Movable Fine Mesh of Hovermale for hurricanes 60 km / 10 layers 50x50 window to 48 hr (‘78)_
- _Final analysis using__ multivariate OI of Bergman __(’78)_
- Development of global spectral model, normal mode initialization, NGM etc await bigger computers

## 1979-82  NMC Production Suite 

- LFM-II 127 km / 7 layer No. America to 48 hr (‘79)
- PE extension with 4th order finite differencing     381 km / 7 layer N.Hemis  60-144 hr (‘80)
- Barotropic extended to 240 hours (10 days) (‘80)
- _Global Spectral model of Sela R-30 / 12 layer global to 48 hr (‘80)_
- Spectral _extension  R-24/12 layer N.Hemis 48-144hr_
- LFM-II with 4th order finite differencing 190.5 km/7layer No. America to 48 hr (‘81)
- Development of NGM, optimum interpolation (OI),  Medium Range Forecast (MRF) await next machine

## Global Spectral Model Forecast

## Personal Reflection

- ONLY jobs using no more than 256K bytes of memory and no more than 1 minute of cpu time were run on IBM 360-195’s during the day - all other jobs ran overnight
- Graphics jobs went to microfilm, took 3 days
- I spent a month with NMCiDAS graphics code to get it to fit and drive new plotter to provide quick same day real-time graphics

## Personal Reflection

- NMC Director Bill Bonner trades away responsibility for facsimile transmission and comm’s in exchange for control over the

- central computer (this was a _really_ good deal!)

## 1981 NMC Buys a CDC CYBER 205

- First _parallel_ machine with two “pipes”, second machine in 1987. MAJOR conversion effort - needed long vectors to get max speed.  IBM 360’s kept as front-ends - eventually replaced by HDSs.

**400 Mflops x 2 Class V**

## 1983-89 NMC Production Suite 

- Global Spectral model R-40 / 12 layer to 240 hr (‘83)
- Global Spectral upgrade and MRF R-40 / 18 layer to 240 hr (‘85)
- _Nested Grid Model of Philips + Regional OI  80 km / 16 layer No. America to 48 hr (‘85)_
- _NOAA Ocean Wave (NOW) 2.5 deg global to 72hr (’85)_
- Global Spectral model upgrade T-80 / 18 layer to  240 hr (‘87 with arrival of 2nd Cyber 205)
- MFM hurricane model _replaced by Quasi-Lagrangian Model of Mathur  40 km / 10 layer window (111x111) to  72 hr (‘88)_
- _Major physics upgrade to NGM 80 km / 16 layers expanded No. America to 48 hr (’88)__ _
- Development of ensembles, Eta model, RDAS/ EDAS, Rapid Update Cycle (RUC)

## Personal Reflection

- Regional OI was _first_ NMC analysis to use the _significant_ level data from radiosondes
- Parallel runs became possible
- Plentiful disk space on IBM 360-195’s allowed an ONLINE 30-day rotating archive of production & parallel runs
- First Monthly Performance Summary

## 1990 NMC Buys a Cray Y-MP

_**1990 NMC Buys a Cray Y-MP**_

- Scripts converted to UNIX – model conversions less of an issue due to Cray’s excellent FORTRAN compiler.   Front ends replaced by J916’s

- 2.6 Gflops Class VI

## 1990-93 NMC Production Suite 

- Global Spectral model T-126 / 18 layer to 240 hr (‘91)
- _Ensemble Spectral of Toth & Kalnay (3 members) T-62 / 18 layer global to 240 hr (‘92)_
- _LFM__ replaced by early Eta Model of Janjic & Mesinger 80 km / 38 layers with step mountains No. America to 48 hr (’93)_
- Global Spectral model upgrade T-126 / 28 layer to  240 hr (‘93)
- NGM 2-grid upgrade_ with RDAS _80 km / 16 layers on greatly expanded No. America to 48 hrs (’91)_ __FROZEN__ _
- Development of  Meso Eta, 3DVAR + EDAS, Rapid Update Cycle await next machine

## Eta QPF Skill Trend

## 1994 NMC Buys a Cray C9016

- Awesome machine that would eventually succumb to a fire in a power supply that was extinguished with a corrosive fire retardant.

- 15.3 Gflops  Class VII

## 1994-95 NMC Production Suite 

- _Rapid Update Cycle (RUC) 60 km / 25 hybrid sigma/ isentropic layers to 12 hr every 3 hr (‘94)_
- NOW _replaced by WAM wave model 2.5 deg global to 72 hr (‘94)_
- ENS Spectral upgrade (17 members) T-62 / 18 layer global to 384 hr (‘94)
- QLM hurricane _replaced by GFDL model 1/6 deg / 18 layer movable window to 78 hr (’95)_
- _Meso Eta 29 km / 50 levels__ __extended CONUS to 51 hr  at 0300 and 1500 (‘95) _
- Early Eta upgrade 48 km/38 layer with partial cycled EDAS over No.America to 48hr (’95)

## 1996-99 NCEP Production Suite 

- Early Eta upgraded with coupled Land Surface Model 48 km / 38 layer No.Amer to 48 hr (’96)
- _Regional Spectral Model of Juang 10 km / 28 layer for Hawaii to 48 hr (’97)_
- _Sea-Ice Drift for Arctic and Antarctic to 384 hr (‘97)_
- Early Eta and Meso Eta combined into 4/day with _3DVAR of Parrish & fully cycled EDAS of Rogers_ 32 km /45 layer No.America to 48 hr (’98)
- RUC-II upgrade to 40 km / 40 hybrid layer CONUS to 12 hr every 3 hr (‘98)
- Development of  10 km Meso Eta, Short Range Ensemble Forecast (SREF)

## 1999 NCEP Buys IBM SP2

-  The Procurement from HELL -- I was on the team!
-  Originally planned to put at NASA, but had to put it in FOB-4 

-  Power supply fell through the floor

-  Found asbestos under the raised floor tiles so put in new floor, then ...
-  While installing equipment, they found asbestos above ceiling tiles (in air movers) and had to major abatement.
-  Resulted in move of IBM to Bowie facility (good news)
- Cray fire on day of move (bad news)

## 1999 NCEP Puts IBM SP2 in Bowie

- After SP2 is upgraded, becomes one of the most stable & capable machines in NCEP’s experience.

- 0.7-2.5 Tflops
- Class VIII

## Personal Reflection

- First time Production was separated from Development – nearly guarantees that at least half the machine will be available for development – previously there was always time when Prod used more (sometimes much more) than 50% of computer.  At ECMWF, only 20% of machine used for their Production!
- IBM’s reliability (helped by failover capability to the Development half) allows hourly RUC. In the past, for example, Cray Preventive Maintenance took 3 hours every Thursday when the Meso Eta would routinely be dropped.

## Record of On-Time Delivery & Receipt (within 15 min) of NCEP Guidance

## 2000-01 NCEP Production Suite 

- Global Spectral upgrade T-170 / 42 layer global to 240 hr (‘00)
- ENS Spectral upgrade (23 members) T-126/28 layer for 00-84 hrs, T-62/18 layer to 84-384 hr (‘00)
- WAM replaced by NOAA WaveWatch 3 of Tolman 1.25 deg x 1 deg global to 126 hr (’00)
- Eta upgrade to 22 km /50 layer No.America to 60 hr [48 hr at off-times] (’00)
- GFDL hurricane adds URI coupled ocean in Atl basin 1/6 deg / 18 layer movable window (4 storms) to 126 hr (’01)

## 2001-02 NCEP Production Suite 

- _Global Seasonal Forecast Model to produce 7 month guidance (‘01)_
- SREF (10 members Eta & RSM) 48 km No. Amer. To 63 hours (’01)
- HiResWindow nested Eta 10 km / 60 layer 1/3 CONUS & Alaska 1/day to 48 hr (’01)
- _Meso Eta 4/day upgrade with new cloud scheme  to 12 km / 60 layer No. Amer to 84 hr (ontime) 48 hr offtime (‘01)_
- Princeton Ocean Model (POM) 10-20 km / 19 layer ocean off east coast to 48 hr (’02)
- Global Forecast System (GFS) upgraded T-254 / 64 layer 00-72hr, T-170 / 42 layer 72-120hr & T-62 / 28 layer 120-384 hr 4-per-day (’02)

## Personal Reflections

- Blizzard BUST after IBM SP2 dedication January 2000 should warn folks from ever being overly optimistic-especially in public
- Forecasters were incredulous when 22 km Eta Model guidance overpredicted heavy QPF- early intolerance of any over-forecasting has disappeared – now all they want is perfection all the time everywhere

## Keeping Up With The Curve

**Procurements every 3 years – updates every 18-24 months**

## 2002 NCEP Buys IBM SP2 Upgrade 

- Housed in IBM facility in Gaithersburg

- Class IX

## 2002-04 NCEP Production Suite 

- _Meso Eta 4/day upgrade 12 km / 60 layer No. Amer to 84 hr at offtime (’03)__  __with earlier delivery (’04)__ _
- _SREF upgrade (__15__ members Eta, RSM, Eta-Kain Fritsch) 48 km No. America to 63 hr (’03)_
- HiResWindow nested _Nonhydrostatic Mesoscale Model (NMM) of Janjic_ 8 km / 60 layer 1/3 CONUS & Alaska 1/day to 48 hr (’03)
- _Fire Weather / IMET Support nested NMM selectable window 1/12 CONUS 8 km / 60 layer to 48 hr  48 km No. America to 63 hr (’03)_
- ENS spectral upgrade (45 members) T-126 / 28 layer 00-180hr T-62 / 28 layer 180-384 hr (’04)
- NOAA Wave Watch 3 upgrade 1.25 deg x 1 deg global to 180 hr (’04)

## New Resolution HiResWindow Fixed-Domain Nested Runs

- R_outine_ runs made at the _same_ time _every_ day
- 00Z : Alaska-10 & Hawaii-8
- 06Z : Western-8 & Puerto Rico-8
- 12Z : Central-8 & Hawaii-8
- 18Z : Eastern-8 & Puerto Rico-8
- This gives everyone a _daily_ high resolution run when <2 hurricane runs need to be made

- http://www.emc.ncep.noaa.gov/mmb/mmbpll/nestpage/

## 26 Selectable 8 km Domains For Fire Weather / IMET Support Identical To  4 km Homeland Security Domains

## 12 km Meso Eta vs 8 km NMM Winds

## 2004+ NCEP Production Suite 

## Personal Perspective & Prospective

- We’ve come an awfully long way!
- Future will involve transition to WRF, ESMF, more Climate to contend with, more ensembles to contend with, more obs to contend with (always a good thing), more collaborators to contend with, more fire-drills than ever to contend with, but ...
- We’re extremely adaptive, resilient & resourceful so my view of the future is very optimistic no matter what they throw at us.
- Stay tuned

## Mesoscale Modeler’s Haikucourtesy of Bruce Webster

- Out of memory.
- We wish to hold the whole sky,
- but we never will.

## Management Haiku courtesy of Bruce Webster

- I’d be overwhelmed,
- but that would be just one more
- thing I’d have to do!