# Presentation: 215448

## EARTH SCIENCE MARKUP LANGUAGETutorial 

- “Define Once Use Anywhere”
- INFORMATION TECHNOLOGY AND SYSTEMS CENTER
- UNIVERSITY OF ALABAMA IN HUNTSVILLE

## Tutorial Overview 

- Introduction to ESML
- Data/Application Interoperability Problem
- Interchange Technology Solution via ESML
- Writing ESML Description Files using the Editor
- ASCII Format
- Binary
- HDF-EOS
- Writing ESML reader using Python

## ESML Vision

## Earth Science Data Characteristics

- Different formats, types and structures (18 and counting for Atmospheric Science alone!)
- Different states of processing  ( raw, calibrated, derived, modeled or interpreted )
- Enormous volumes
**Heterogeneity leads to Data usability problem**

- HDF

- HDF-EOS

- netCDF

- ASCII

- Binary

- GRIB

## Interoperability: Accessing Heterogeneous Data

- Difficult to implement and enforce
- Can’t anticipate all needs
- Some data can’t be modeled or is lost in translation
- The cost of converting legacy data
- A better approach:  Interchange Technologies
- Earth Science Markup Language

**ESML **
**LIBRARY**

**APPLICATION**

**DATA **
**FORMAT 1**

**DATA **
**FORMAT 2**

**DATA **
**FORMAT 3**

***The ESML Solution***

**ESML**
**FILE**

**ESML**
**FILE**

**ESML**
**FILE**

## What is ESML?

- It is a **specialized markup language** for Earth Science metadata based on XML - NOT another data format.
- It is a machine-readable and -interpretable representation of the **structure **of any data file, regardless of data format
- ESML description files contain **external metadata** that can be generated by either data producer or data consumer (at collection, data set, and/or granule level)
- ESML provides the **benefits of a standard, self-describing data format** (like HDF, HDF-EOS, netCDF, geoTIFF, ...) without the cost of data conversion
- ESML is the basis for **core Interchange Technology** that allows data/application interoperability
- ESML **complements and extends data catalogs** such as FGDC and GCMD by providing the use/access information those directories lack.

## ESML Components

***ESML Schema*** defines syntactic  metadata that describe the structure of the file in machine-readable and interpretable terms and are the grammar that are used to generate the ESML description file

- <a:ESML >
-   <SyntacticMetaData>
-     <Ascii>
-       <Structure instances="1">
-         <Header name="SizeX" format="%d" symbol=“true" />
-         <Header name="SizeY" format="%d“ symbol=“true" />
-         <Array occurs=“$SizeX">
-           <Array occurs=“$SizeY">
-             <Field name="BrightnessTemp" format="%d"/>
-           </Array>
-         </Array>
-       </Structure>
-     </Ascii>
-   </SyntacticMetaData>
- </a:ESML>

-   4
-   5
-   1   2   3   4   5
-   6   7   8   9 10
- 11 12 13 14 15
- 16 17 18 19 20

**SIMPLE ASCII**
**DATA FILE**

**ESML MARKUP**
**FOR THE DATA FILE**

***ESML Description Files*** specify the structure of the data file format in terms of bits and bytes

***ESML Library*** is the middleware that applications use to parse an ESML Description File and retrieve data

## Components of the ESML Interchange Technology

- DATAFORMAT1

- DATAFORMAT2

- DATAFORMAT3

- OTHER FORMATS

- ESMLFILE

- ESMLFILE

- ESMLFILE

- ESMLSCHEMA

- ESML LIBRARY

- OTHER
- APPLICATIONS

- ESMLEDITOR

- ESMLDATA
- BROWSER

- ADaM DATA
- MINING
- SYSTEM

- External description file for
- dataset or formats

- Rules that govern the
- description of the data files

- Library parses and interprets
- the description file and figures
- out how to read the data

## Components of the ESML Interchange Technology

- DATAFORMAT1

- DATAFORMAT2

- DATAFORMAT3

- OTHER FORMATS

- ESMLFILE

- ESMLFILE

- ESMLFILE

- ESMLSCHEMA

- ESML LIBRARY

- ESMLDATA
- BROWSER

- ADaM DATA
- MINING
- SYSTEM

- OTHER
- APPLICATIONS

- ESMLEDITOR

- ESMLFILE

- ESMLSCHEMA

- INTERCHANGE
- TECHNOLOGY

- These three key components allow applications to use data in a wide variety
- of formats

## Interchange Technology for Data Users and Application Developers

- DATAFORMAT1

- DATAFORMAT2

- DATAFORMAT3

- OTHER
- FORMATS

- ESMLFILE

- ESMLFILE

- ESMLFILE

- ESMLSCHEMA

- ESMLDATA
- BROWSER

- ADaM DATA
- MINING
- SYSTEM

- OTHER
- APPLICATIONS

- ESMLEDITOR

- ESMLFILE

- ESMLSCHEMA

- INTERCHANGE
- TECHNOLOGY

- DATA PRODUCERS
- OR CONSUMERS

- APPLICATION
- DEVELOPERS

- ESML can be used by both
- scientists and application
- developers

- ESML LIBRARY

## Advantages of using ESML

- Scientist (Data Producer/Consumer)
- ESML will let them use virtually any data format in their applications
- ESML files are external description files that can be easily created, modified and viewed by any text editor
- ESML has a few simple concepts which can be used to describe numerous data sets
- An ESML file can be seen as a set of instructions to the application on how to read and understand a data file
- If the format of the data changes for whatever reason (e.g., new version of data set) no software changes are required, just a new ESML file.
- Does that mean a scientist has to write an ESML file for every data file?
- No, in fact the beauty of ESML is that it allows scientist to write ONE ESML file to describe MANY data files that are structurally and semantically similar

## Advantages of using ESML

- Data Archiving Centers (Data Producers)
- ESML files can be used to store not only the *structural* but also embed *semantic* information about the data sets
- Since ESML files are independent separate files, they can be generated on the fly utilizing metadata databases as datasets are ordered
- Centers can archive data in its native formats and not have to store them in any “selected” format
- Centers can now also “ESMLize” all their legacy datasets with minimal efforts
- The existing legacy datasets now become a more valuable data resource for scientists, because they can be used more efficiently and effectively
- Application Developers
- By using the ESML library, developers can build “ESML enabled” applications!
- ONE single reader component can read all the various data formats instead of having separate reader module for different formats

## ESML IN ACTION:Collocation Algorithm 

- ESML
- file

- ESML
- file

- ESML
- file

**ESML Library**

- Collocation Algorithm

- MODIS

- CERES

- MISR/
- Others

_*Scientists can:*_
*Select a variety of data in different formats for the collocation analysis *

_*Purpose:*_
*To study the relationship between shortwave flux and cloud/aerosol properties*
*Important for climate change studies*

- ESML IN ACTION:Collocation Algorithm

- Analysis

## ESML IN ACTION: Ingest surface skin temperature data in Numerical Models 

- GOES - McIDAS
- Reanalysis Data - GRIB
- MM5 Model - MM5 Binary
- AVHRR - HDF
- MODIS - EOS-HDF

- ESML IN ACTION: Ingest surface skin temperature data in Numerical Models

**Reanalysis**
**GRIB files**

**MM5**

**GOES**

**ESML**
**FILE**

**ESML**
**FILE**

**ESML**
**FILE**

**ESML **
**LIBRARY**

**APPLICATION**

## ESML-Enabled Generic Subsetter

- ESML
- file

- ESML
- file

- ESML
- file

**ESML Library**

- Subsetting Algorithm

- HDF-EOS

- Binary/
- ASCII

- Other
- Formats

- Network

- Subsetted Data

## Prototype: Smart Subsetter

- To demonstrate a smart application using ESML and ontologies, a subsetting prototype is being developed
- Subsetting is a frequent preprocessing step used by scientists to reduce the size and complexity of the data
- The subsetting prototype parses the ***semantic tags ******embedded in the ESML Description File***
- The subsetting prototype then ***uses the linked ontologies****** to decipher meaning of thes******e tags ******to make useful decisions***
- Components of this Prototype:
- Simple ontologies describing “Subsetting” and “Dataset”
- ESML Description Files
- Reasoning System from JTP from Stanford University used as an inference engine

- Dataset Ontology

- Subsetting Ontology

## Smart Applications/Services using ESML and Ontologies 

- ESMLSCHEMA

- ESML Schema’s focus is on providing structural data interoperability between data/application
- However, ESML ***allows embedding semantic terms*** for data fields in the Description File to provide a complete structural and semantic description of the data
- Various science communities can create their own ontologies (for example, SWEET) and link them with ESML Description Files for their data
- Application developers can add semantic parsers on top of the core ESML Library to build ***“smart” applications or services ***

- ESMLSchema

- ESMLSCHEMA

- Ontologies

- Ontologies

*Structural *
*Information*

*Semantic*
*Information*

## ESML Schema 

- ESML schema defines*** Syntactic***  metadata that describe the structure of the file in machine-readable and -interpretable terms

## Writing an ESML File (1)

- <a:ESML >
-   <SyntacticMetaData>
-     <Ascii>
-       <Structure instances="1">
-         <Header name="SizeX" format="%d" />
-         <Header name="SizeY" format="%d" />
-         <Array occurs=“4">
-           <Array occurs=“5">
-             <Field name="BrightnessTemp" format="%d"/>
-           </Array>
-         </Array>
-       </Structure>
-     </Ascii>
-   </SyntacticMetaData>
- </a:ESML>

-   4
-   5
-   1   2   3   4   5
-   6   7   8   9 10
- 11 12 13 14 15
- 16 17 18 19 20

- SIMPLE ASCII
- DATA FILE

- ESML MARKUP
- FOR THE DATA FILE

- The next slides will set
- describe how to write an ESML file for a simple ASCII file described below

## Writing an ESML File (2)

-   <SyntacticMetaData>
-     <Ascii>
-       <Structure instances="1">
-         <Header name="SizeX" format="%d" />
-         <Header name="SizeY" format="%d" />
-         <Array occurs=“4">
-           <Array occurs=“5">
-             <Field name="BrightnessTemp" format="%d"/>
-           </Array>
-         </Array>
-       </Structure>
-     </Ascii>
-   </SyntacticMetaData>
- </a:ESML>

- DESCRIBING ONLY THE
- STRUCTURE

-   4
-   5
-   1   2   3   4   5
-   6   7   8   9 10
- 11 12 13 14 15
- 16 17 18 19 20

- SIMPLE ASCII
- DATA FILE

## Writing an ESML File (3)

-   <SyntacticMetaData>
-     <Ascii>
-       <Structure instances="1">
-         <Header name="SizeX" format="%d" />
-         <Header name="SizeY" format="%d" />
-         <Array occurs=“4">
-           <Array occurs=“5">
-             <Field name="BrightnessTemp" format="%d"/>
-           </Array>
-         </Array>
-       </Structure>
-     </Ascii>
-   </SyntacticMetaData>
- </a:ESML>

- DESCRIBE THE
- FORMAT

-   4
-   5
-   1   2   3   4   5
-   6   7   8   9 10
- 11 12 13 14 15
- 16 17 18 19 20

- SIMPLE ASCII
- DATA FILE

## Writing an ESML File (4)

-   <SyntacticMetaData>
-     <Ascii>
-       <Structure instances="1">
-         <Header name="SizeX" format="%d" />
-         <Header name="SizeY" format="%d" />
-         <Array occurs=“4">
-           <Array occurs=“5">
-             <Field name="BrightnessTemp" format="%d"/>
-           </Array>
-         </Array>
-       </Structure>
-     </Ascii>
-   </SyntacticMetaData>
- </a:ESML>

- ENTIRE FILE CONTENTS
- INTO 1 LOGICAL
- STRUCTURE

-   4
-   5
-   1   2   3   4   5
-   6   7   8   9 10
- 11 12 13 14 15
- 16 17 18 19 20

- SIMPLE ASCII
- DATA FILE

## Writing an ESML File (5)

-   <SyntacticMetaData>
-     <Ascii>
-       <Structure instances="1">
-         <Header name="SizeX" format="%d" />
-         <Header name="SizeY" format="%d" />
-         <Array occurs=“4">
-           <Array occurs=“5">
-             <Field name="BrightnessTemp" format="%d"/>
-           </Array>
-         </Array>
-       </Structure>
-     </Ascii>
-   </SyntacticMetaData>
- </a:ESML>

- DEFINE THE FIRST
- FIELD IN THE FILE:
- HEADER INFORMATION

-   4
-   5
-   1   2   3   4   5
-   6   7   8   9 10
- 11 12 13 14 15
- 16 17 18 19 20

- SIMPLE ASCII
- DATA FILE

## Writing an ESML File (6)

-   <SyntacticMetaData>
-     <Ascii>
-       <Structure instances="1">
-         <Header name="SizeX" format="%d" />
-         <Header name="SizeY" format="%d" />
-         <Array occurs=“4">
-           <Array occurs=“5">
-             <Field name="BrightnessTemp" format="%d"/>
-           </Array>
-         </Array>
-       </Structure>
-     </Ascii>
-   </SyntacticMetaData>
- </a:ESML>

- DEFINE THE SECOND
- FIELD IN THE FILE:
- HEADER INFORMATION

-   4
-   5
-   1   2   3   4   5
-   6   7   8   9 10
- 11 12 13 14 15
- 16 17 18 19 20

- SIMPLE ASCII
- DATA FILE

## Writing an ESML File (7)

-   <SyntacticMetaData>
-     <Ascii>
-       <Structure instances="1">
-         <Header name="SizeX" format="%d" />
-         <Header name="SizeY" format="%d" />
-         <Array occurs=“4">
-           <Array occurs=“5">
-             <Field name="BrightnessTemp" format="%d"/>
-           </Array>
-         </Array>
-       </Structure>
-     </Ascii>
-   </SyntacticMetaData>
- </a:ESML>

- DEFINE THE  DATA
- FIELD IN THE FILE:
- PROVIDE SIZE AND
- FORMAT INFORMATION

- SIMPLE ASCII
- DATA FILE

-   4
-   5
-   1   2   3   4   5
-   6   7   8   9 10
- 11 12 13 14 15
- 16 17 18 19 20

## Writing an ESML File (8)

-   <SyntacticMetaData>
-     <Ascii>
-       <Structure instances="1">
-         <Header name="SizeX" format="%d" />
-         <Header name="SizeY" format="%d" />
-         <Array occurs=“4">
-           <Array occurs=“5">
-             <Field name="BrightnessTemp" format="%d"/>
-           </Array>
-         </Array>
-       </Structure>
-     </Ascii>
-   </SyntacticMetaData>
- </a:ESML>

- CLOSE ALL THE
- TAGS: ESML FILE
- IS READY

-   4
-   5
-   1   2   3   4   5
-   6   7   8   9 10
- 11 12 13 14 15
- 16 17 18 19 20

- SIMPLE ASCII
- DATA FILE

## Another Possible ESML Description (9)

-   <SyntacticMetaData>
-     <Ascii>
-       <Structure instances="1">
-         <Header name="SizeX" format="%d" symbol=“true" />
-         <Header name="SizeY" format="%d“ symbol=“true" />
-         <Array occurs=“$SizeX">
-           <Array occurs=“$SizeY">
-             <Field name="BrightnessTemp" format="%d"/>
-           </Array>
-         </Array>
-       </Structure>
-     </Ascii>
-   </SyntacticMetaData>
- </a:ESML>

- USE HEADER
- INFORMATION

-   4
-   5
-   1   2   3   4   5
-   6   7   8   9 10
- 11 12 13 14 15
- 16 17 18 19 20

- SIMPLE ASCII
- DATA FILE

## Using The ESML Editor

- Writing an ESML file using the ESML Editor
- Simple ASCII file
- Simple Binary file
- HDF-EOS file
- Utilizing the ESML descriptions to browse the data files using the ESML Data Browser
- Overview of the ESML library
- Writing a simple reader
- Writing a generic reader

## ESML Editor

- ESML Schema Window
- Displays the ESML Schema in tree structure
- ESML File Window
- Displays the user generated ESML description
- ESML Attribute Window
- Provides the means to set the attributes for the selected ESML element

## ESML Editor

- 100% Java version prototype
- Utilizes Expert System principles to give users correct options
- Hides the XML tags from the users
- Future Features:
-  Allow text editing of the XML tags also
-  Incorporate feedback from users

## Writing an ESML File: Simple ASCII File (1)

- ASCII data file:
-   4
-   5
-   1   2   3   4   5
-   6   7   8   9 10
- 11 12 13 14 15
- 16 17 18 19 20

## Writing an ESML File: Simple ASCII File (2)

- Understand the structure of the data file to be described by ESML file
- Open ESML editor (ready to create new ESML file) and describe this structure using the rules established in the ESML schema
- Add “SyntacticMetadata” as child of “ESML” and modify its attributes if available
- Add “Ascii” as child of “SyntacticMetadata” and modify its attributes if available

## Writing an ESML File: Simple ASCII File (3)

- Add “AsciiStructure” as child of “Ascii” and modify its attribute
- Add “Field” element as one child of “AsciiStructure” and modify its attributes if available. This element is used to describe the first datum in the data file
- Add ‘semantic information’ to describe this “Field”
- Add “Field” element as one child of “AsciiStructure” and modify its attributes if available. This element is used to describe the second datum in the data file
- Add “Array” element as one child of “AsciiStructure” to describe the outer dimension of the 2D array, and modify its attributes if available

## Writing an ESML File: Simple ASCII File (4)

- Add “Array” element as child of “Array” to describe the inner dimension of the 2D array, and modify its attributes if available
- Add “Field” element as child of “Array” to describe the individual element and modify its attributes if available

## Writing an ESML File: Simple Binary File (1)

- The structure and content are the same as those of ASCII file that we just worked on, but stored in Binary format
- Open ESML editor
- Add “SyntacticMetadata” as child of “ESML” and modify its attributes if available
- Add “Binary” as child of “SyntacticMetadata” and modify its attributes if available

## Writing an ESML File: Simple Binary File (2)

- Add “BinaryStructure” as child of “Binary” and modify its attribute
- Add “Field” element as one child of “BinaryStructure” and modify its attributes if available. This element is used to describe the first datum in the data file
- Add “Field” element as one child of “BinaryStructure” and modify its attributes if available. This element is used to describe the second datum in the data file
- Add “Array” element as one child of “BinaryStructure” to describe the outer dimension of the 2D array, and modify its attributes if available

## Writing an ESML File: Simple Binary File (3)

- Add “Array” element as child of “Array” to describe the inner dimension of the 2D array, and modify its attributes if available
- Add “Field” element as child of “Array” to describe the individual element and modify its attributes if available

## Writing an ESML File: HDF-EOS File (1)

- Since HDF-EOS is self-describing format and metadata is contained in the data file, ESML file for HDF-EOS data file is simple
- Here are the steps to create ESML file for HDF-EOS:
- Open ESML editor
- Add “SyntacticMetadata” as child of “ESML” and modify its attributes if available and needed
- Add “HdfEos” as child of “SyntacticMetadata”
- You are done!!!

## Writing an ESML File: HDF-EOS File (2)

- HDF-EOS format has rules that Latitude and Longitude fields have to have field names of “Latitude” and “Longitude”, respectively
- If a HDF-EOS file does not follow this rule, ESML file has to be more detailed
- Here are the follow-on from the ESML file before:
- Add “HdfEosStructure” as child of “HdfEos” and modify its attributes if available and needed

## Writing an ESML File: HDF-EOS File (3)

- Add “Field” as child of “HdfEosStructure”
- Add ‘semantic information’ as child to describe this “Field” element. If this field is latitude, then add a child element “Latitude” to indicate that this field is latitude field. If longitude field, add “Longitude” child element to describe the “Field” element.

## Design follows the layered cake approach where: 
Lowest (core) level provides the basic functionality of reading the structural metadata from the ESML file and returning data to the user
Additional software layers can be added to provide other functionalities such as using semantics from an ontology to “use” the data intelligently
Includes plug-in modules for each individual format, allowing packaging of libraries
Provides a simple API for easy addition of new formats as plug-in modules
Provides a more intuitive user API based on the analogy of file access in a directory structure
Provides the library source code via Source Forge repository

- Lowest (core) level provides the basic functionality of reading the structural metadata from the ESML file and returning data to the user
- Additional software layers can be added to provide other functionalities such as using semantics from an ontology to “use” the data intelligently
- Includes plug-in modules for each individual format, allowing packaging of libraries
- Provides a simple API for easy addition of new formats as plug-in modules
- Provides a more intuitive user API based on the analogy of file access in a directory structure
- Provides the library source code via Source Forge repository

- ESML Library

## ESML v3.0 Library Layered Design

- The **core ESML library** provides the basic functionality of reading structural metadata from the ESML file and returning data to the user
- Intuitive **user API** based on the analogy of file access in a directory structure
**Plug-in modules** for each individual format allow flexible packaging of libraries
- Simple **Plug-in API** for easy addition of new formats
**Additional software** can be easily added to provide other functions such as using semantics from an ontology to “use” the data intelligently

## PyESML Example 

- List all the
- API functions

- Open a data
- file using ESML

- List all the
- structures

- List all the
- fields

- Get the data
- for a field

- Print the data

## Current Status

- ESML data formats
- Currently supported
- ASCII, Binary, HDF-EOS, netCDF, Grib
- In work
- HDF5
- ESML Library
- Currently available
- C++ for Windows and Linux, Python plugin
- In work
- IDL plugin
- ESML Editor application

## For More Information

- URL: **esml.itsc.uah.edu **
- Become a member and post ESML related news items on the website
- Schema and related documents available to all
- Download the latest products
- Source code available via Source Forge
- Join the ESML mailing list