# Presentation: 663354

## Meteorological Development Laboratory’sNational Digital Forecast Database Extensible Markup Language Product

- John Schattel
- 27 February 2004

## Overview

- NDFD XML Background
- XML overview
- Motivation for NDFD XML
- NDFD XML Data Model
- NDFD XML High Level Data Flow
- Where NDFD XML data comes from
- Client/Server architecture
- User Interface (NDFDgen)
- Functional Analysis
- Accessing grid cube data
- Creating XML products

## XML Overview

**<dwml xmlns:xsd="http://www.w3.org/2001/XMLSchema" **
**              ****xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" **
**   ****<head>**
**      ****<product concise-name="time-series">**
**          ****<creation-date>2004-02-13T21:29:11Z</creation-date>**
**      ****</product>**
**      ****<source>**
**          ****<production-center>MDL</production-center>**
**      ****</source>**
**   ****</head>**
**   ****<data>**
**      ****<parameters>**
**         ****<temperature units="Fahrenheit">**
**            ****<value>45</value>**
**            ****<value>31</value>**
**            ****<value>29</value>**
**            ****<value>37</value>**
**            ****<value>41</value>**
**            ****<value>41</value>**
**            ****<value xsi:nil="true" />**
**         ****</temperature>**
**      ****</parameters>**
**   ****</data>**
**</dwml>**

- XML Schema +
- SAX, DOM,
- XSLT

- TODAY: HIGH TEMP OF 45 F.
- TOMORROW: HIGH TEMP OF 31 F
- SUNDAY:  HIGH TEMP OF 29 F
- MONDAY:  HIGH TEMP OF 37 F
- TUESDAY:  HIGH TEMP OF 41 F
- WEDNESDAY: HIGH TEMP OF 41 F

**Extensible Markup Language (XML) *****n*****. 1. Internet standard for data exchange.**
**2. Language to create other languages. 3.  A way to provide NDFD data  **

- a meteogram

- - OR -

## XML on the Move

- Data
- Consumer
- (Client)

- Internet

- Data
- Provider
- (Server)

- SOAP Response

- SOAP  Request

- SOAP Response

- SOAP  Request

- UDDI
- (data broker)

- WSDL

- Data

- Data
- Provider

- Transport
- (HTTP)

- Messaging
- (SOAP)

- Description
- (WSDL/Schema)

- Discovery
- (UDDI)

** ****Communicating Process**

- WSDL

## Background

*NWS Strategic Plan for FY 2003 – FY 2008*

- “...evolving our services from a text-based paradigm to one based on making NWS information available quickly, efficiently, and in convenient and understandable forms...” and “...by taking advantage of existing and emerging technologies to disseminate this information...”

*NWS Policy on Partnership in the Provision of Weather, Water, Climate and Related Environmental Information*

- “... NWS will provide ... data in forms convenient to additional processing by others. NWS will make its data and products available in Internet-accessible form ...”

## Background

- Convenient for additional processing
- XML is ideal for machine to machine access through web services
- XML tools (SAX, DOM, etc.) facilitate value added applications
- Internet-accessible
- Available on weather.gov (retail)
- Value added version on user sites (whole sale)
- Data brokers (UDDI and WSDL) help users find NDFD data

## Background

- Make data available efficiently
- User only gets the piece of database they need
- Minimizes developer spin up -- use familiar tools
- Exploiting existing technology
- XML is an existing and widely used technology
- De Facto standard for internet data exchange
- Infrastructure exist (UDDI, parsers, etc.)

## Data Model

- <dwml>
-      <head>
-           <product />
-           <source />
-      </head>
-      <data>
-           <location />
-           <time-layout />
-           <parameters />
-      </data>
- </dwml>

- more follows = **...**
- zero or one = **?**
- zero or more = *****
- one or more = **+**
- choose one = **{ }**

***version***

**...**

- Element

- Attribute

**...**

**...**

**...**

**...**

**...**

## Data Model

- <dwml>
-      <head>
-           **<product />**
-           <source />
-      </head>
-      <data>
-           <location />
-           <time-layout />
-           <parameters />
-      </data>
- </dwml>

- more follows = **...**
- zero or one = **?**
- zero or more = *****
- one or more = **+**
- choose one = **{ }**

***refresh-frequency***

## Data Model

- <dwml>
-      <head>
-           <product />
-           **<source />**
-      </head>
-      <data>
-           <location />
-           <time-layout />
-           <parameters />
-      </data>
- </dwml>

- more follows = **...**
- zero or one = **?**
- zero or more = *****
- one or more = **+**
- choose one = **{ }**

## Data Model

***latitude***
***longitude***
***summarization ?***

- <dwml>
-      <head>
-           <product />
-           <source />
-      </head>
-      <data>
-           **<location />**
-           <time-layout />
-           <parameters />
-      </data>
- </dwml>

- more follows = **...**
- zero or one = **?**
- zero or more = *****
- one or more = **+**
- choose one = **{ }**

- _[Example](file:///samples/location.txt)_

## Data Model

- <dwml>
-      <head>
-           <product />
-           <source />
-      </head>
-      <data>
-           <location />
-           **<time-layout />**
-           <parameters />
-      </data>
- </dwml>

***time-coordinate***
***summarization***

***period-name ?***

**+**

*****

- more follows = **...**
- zero or one = **?**
- zero or more = *****
- one or more = **+**
- choose one = **{ }**

- _[Example](file:///samples/time.txt)_

## Data Model

- <dwml>
-      <head>
-           <product />
-           <source />
-      </head>
-      <data>
-           <location />
-           <time-layout />
-           **<parameters />**
-      </data>
- </dwml>

***type***
***units***
***time-layout***

**...**

***type***
***units***
***time-layout***

*****

*****

*****

*****

*****

***type***
***units***
***time-layout***

***type***
***units***
***time-layout***

***type***
***units***
***time-layout***

**...**

**...**

- more follows = **...**
- zero or one = **?**
- zero or more = *****
- one or more = **+**
- choose one = **{ }**

**applicable-location**

## Data Model

- <dwml>
-      <head>
-           <product />
-           <source />
-      </head>
-      <data>
-           <location />
-           <time-layout />
-           **<parameters />**
-      </data>
- </dwml>

***type***
***units***
***time-layout***

***type***
***units***
***time-layout***

*****

- more follows = **...**
- zero or one = **?**
- zero or more = *****
- one or more = **+**
- choose one = **{ }**

- _[Example](file:///samples/icons.txt)_

**...**

**?**

## Data Model

- <dwml>
-      <head>
-           <product />
-           <source />
-      </head>
-      <data>
-           <location />
-           <time-layout />
-           **<parameters />**
-      </data>
- </dwml>

***type***
***units***
***time-layout***

- more follows = **...**
- zero or one = **?**
- zero or more = *****
- one or more = **+**
- choose one = **{ }**

- _[Example](file:///samples/parameters.txt)_

- NOTE:  This structure applies to precipitation, probability-of-precipitation, wind-speed, direction,
-              cloud amount and humidity

**applicable-location**

## Data Model

- <dwml>
-      <head>
-           <product />
-           <source />
-      </head>
-      <data>
-           <location />
-           <time-layout />
-           **<parameters />**
-      </data>
- </dwml>

***coverage***
***intensity***
**additive**
**qualifier**
**weather-type**

***units***

**+**

***time-layout***

******* **(max of 5)

- more follows = **...**
- zero or one = **?**
- zero or more = *****
- one or more = **+**
- choose one = **{ }**

- _[Example](file:///samples/weather.txt)_

**applicable-location**

## Data Model

- <dwml>
-      <head>
-           <product />
-           <source />
-      </head>
-      <data>
-           <location />
-           <time-layout />
-           **<parameters />**
-      </data>
- </dwml>

- more follows = **...**
- zero or one = **?**
- zero or more = *****
- one or more = **+**
- choose one = **{ }**

- _[Example](file:///samples/wave.txt)_

**+**

***time-layout***

***type***
***units***

**+**

**?**

**applicable-location**

## NDFD Data Flow

## High Level Data Flow

- SOAP
- Client

- Internet

- Load
- Balancer

- Web
- Server

- Data
- Server

- Load
- Balancer

- Web
- Server

- Web
- Server

- Data
- Server

- NDFD XML
- Server
- (SOAP)

- Data
- Probe

- SOAP Response

- SOAP  Request

- SOAP Response

- SOAP  Request

- _SOAP Request_
-      Desired Function
-      Function Parameters
-           latitude (float)
-           longitude (float)
-           product (string)
-           beginTime (dateTime)
-           endTime (dateTime)
-           ndfdElements (array of booleans)

- _SOAP Response_
-      ndfdXml (string)
-              or
-      SOAPFault (string)

- _Data Probe Call_
-      cubeName (string)
-      -DP
-     -pnt
-      latitude (float)
-      longitude (float)
-      beginTime (string)
-      endTime (string)
-      units (string)

- _Data Probe Return_
-      ndfdData (string)

- Data Probe
- Call

- Data Probe Return

- weather.gov

- WSDL

- N
- D
- F
- D

## NDFD XML Server

- Function Parameters
-           latitude (float)
-           longitude (float)
-           product (string)
-           beginTime (dateTime)
-           endTime (dateTime)
-           ndfdElements (array of booleans)

## NDFD XML Server

- ndfd_soap_perf.log

- Write Timing
- Data

- Invoke NuSOAP
- Server

- 1.Calls NDFDgen
- 2.Formats SOAP response containing NDFD XML
- 3.Sends SOAP response to client
- 4.Destroy SOAP Server

- no

- yes

## NDFDgen

- Start

- Determine
- Which
- Elements

**latitude**

**longitude**

**Product**

**ndfdElements**

**startTime**

**endTime**

- MaxT, [F], 200402232000, 200402250000, 41.236
- MaxT, [F], 200402232000, 200402260000, 40.156
- MaxT, [F], 200402232000, 200402270000, 39.313
- MaxT, [F], 200402232000, 200402280000, 49.234
- MaxT, [F], 200402232000, 200402290000, 60.373
- MaxT, [F], 200402232000, 200403010000, 65.233
- MaxT, [F], 200402232000, 200403020000, 56.251

- archive

- I1

- I2

- I3

- P1

## NDFDgen

- Determine
- Local Time
- Offset

- Format
- <location>
- Element

- Format
- <time-layout>
- Element

- Time Zone

- Daylight
- Saving
- (yes/no)

**latitude**

**longitude**

***time-coordinate***
***summarization***

***period-name ?***

**+**

*****

***latitude***
***longitude***
***summarization ?***

- P2

- P4

- P5

- P3

## NDFDgen

- Format
- <parameter>
- Element

- Return
- NDFD
- XML

**applicable-location**

***type***
***units***
***time-layout***

- P6

- O1

## Testing

- Test data is a mix of live and regression data
- Validate output NDFD XML against schema
- Work with recreation.gov to test web services
- Performance testing

## Plans

- Incorporate additional NDFD elements
- Introduce experimental product (NWSI 10-102)
- Partnership with recreation.gov (1 April 2004)
- Work performance enhancements
- Provide data in Geography Markup Language
- Expand to other databases (MOS, Guidance, etc.)
- Support WMO XML activities

## Issues

- Add start and end times to probe
- Expand performance assessment

## Acronyms

- DOM – Document Object Model
- NDFD - National Digital Forecast Database
- SAX – Simple API for XML
- SOAP - Simple Object Access Protocol
- UDDI - Universal Description, Discovery, and Integration
- WSDL - Web Service Description Language
- XML - Extensible Markup Language
- XSLT – Extensible Stylesheet Language Transformations