# Presentation: 262540

## ESML, Subsetting, Mining Tools

- Sara Graves
- Rahul Ramachandran
- Information Technology and Systems Center (ITSC)
- University of Alabama in Huntsville (UAH)
- www.itsc.uah.edu

- MODIS Science Team Meeting
- July 24, 2002

**Notes:**
My name is Matt Smith
Many of you have heard parts of this talk before, I presented a talk 3 years ago about a dataset-independent subsetter.  But this time I want to explain something that some people have missed before.

## Tools Encompassing All Phases of Scientific Analysis 

- Science Data Usability
- Data/Application Interoperability
- Earth Science Markup Language (ESML)
- Science Data Preprocessing
- Subsetting
- Various Subsetting Tools such as HEW
- Science Data Analysis
- Data Mining
- Algorithm Development and Mining (ADaM) System
- Mission/Project/Field Campaign Coordination
- Electronic Collaboration

## Science Data Usability

- _[http://esml.itsc.uah.edu](http://esml.itsc.uah.edu/)_

## Earth Science Data Characteristics

- Different formats, types and structures (18 and counting for Atmospheric Science alone!)
- Different states of processing  ( raw, calibrated, derived, modeled or interpreted )
- Enormous volumes
**Heterogeneity leads to Data usability problem**

- HDF

- HDF-EOS

- netCDF

- ASCII

- Binary

- GRIB

## Data Usability Problem

**DATA **
**FORMAT 1**

**DATA **
**FORMAT 2**

**DATA **
**FORMAT 3**

**APPLICATION**

**READER 1**

**READER 2**

**FORMAT**
**CONVERTER**

- Requires specialized code for every format
- Difficult to assimilate new data types
- Makes applications tightly coupled to data
-  One possible solution - enforce a Standard Data Format
-  Not practical, especially for legacy datasets

## ESML Solution

- ESML (external metadata) files containing the structural description of the data format
- Applications utilize these descriptions to figure out how to read the data  files resulting in data interoperability for applications

**ESML **
**LIBRARY**

**APPLICATION**

**ESML**
**FILE**

**ESML**
**FILE**

**ESML**
**FILE**

**DATA **
**FORMAT 1**

**DATA **
**FORMAT 2**

**DATA **
**FORMAT 3**

## What is ESML?

- It is a specialized markup language for Earth Science metadata based on XML
- It is a machine-readable and -interpretable representation of the structure and content of any data file, regardless of data format
- ESML description files contain external metadata that can be generated by either data producer or data consumer (at collection, data set, and/or granule level)
- ESML provides the benefits of a standard, self-describing data format (like HDF, HDF-EOS, netCDF, geoTIFF, ...) without the cost of data conversion
- ESML is an Interchange Technology that allows data/application interoperability

## ESML Tools/Products Availablehttp://esml.itsc.uah.edu 

| Tools/Products | Features |
| --- | --- |
| ESML Schema | Defines ASCII, Binary (McIDAS), HDF-EOS, GRIB formats (more formats to follow)
Provides preprocessing, wild card, symbols and semantic tagging capabilities |
| ESML Library
(C++/Java JNI ) | WINDOWS and LINUX
URL access
Handles ASCII, Binary (McIDAS), HDF-EOS, GRIB files
Handles preprocessing, wild card and  symbols  |
| ESML Editor | Ability to write and validate ESML descriptions |
| ESML Data Browser | Ability to browse data files using the ESML description files |

## MODIS/CERES Collocation Application

- ESML
- file

- ESML
- file

- ESML
- file

**ESML Library**

- Collocation Algorithm

- MODIS

- CERES

- MISR/
- Others

- Network

- Analysis

_*Scientists can:*_
*Select remote files across the network*
*Select fields by modifying semantic tags in the ESML file *

_*Purpose:*_
*To study the relationship between shortwave flux and cloud/aerosol properties*
*Important for climate change studies*

## Science Data Preprocessing

- _[http://subset.org](http://subset.org/)_

## Currently Available/Planned Subsetting Applications

- HEW Subsetting
- Complete System 				(available)
- Subsetting Engine Only			(available)
- Subsetting Center 				(available)
- SPOT  - Subsettability Checker			(available)
- HEW Integration with ECS 			(in work)
- Remote Subsetting Service 			(planned)
- Subsetting as a Web Service 			(planned)
- Customized Subsetting
- MODIS tools				(available)
- Coarse-grain SSM/I Subsetter			(available)
- General Purpose Customizable Subsetting
- Based on ADaM Data Mining Engine		(available)
- Subsetting Tool using ESML			(in work)

## Tools developed for MODIS Scientists

*modland* – subsetter for MODIS gridded data
-  *stitcher* – pieces together 2 or 4 contiguous   	MODIS tiles
-  MODIS – Atmosphere
*modair* - specialized subsetter for MODIS swaths

## Slide 13

## HEW integration with ECS

*ECS*

* *

*EDG System*

* *

**EDG**

** **

**ECS**

** **

**Subsetter**

** **

- Input

- data

- Output

- data

*Order *

*submission *

*(HTML)*

* *

*Data order*

* *

*and reply*

* *

*Subset ODL*

* *

*and reply*

* *

*Subsetting System*

* *

*Output data *

*(Reingested)*

* *

- 1

- 2

- 3

- 4

- 5

- 6

- 7

*UAH/ITSC-written subsetting and interface software*
*Ongoing testing with ECS 6a.05 and EDG 3.4 at NSIDC, LP DAAC, GDAAC*
*Enhancements for DAACs may be made*

## ESML enabled generic Subsetter

- ESML
- file

- ESML
- file

- ESML
- file

**ESML Library**

- Subsetting Algorithm

- HDF-EOS

- Binary/
- ASCII

- Other
- Formats

- Network

- Subsetted Data

*For HDF-EOS data ****not**** formatted for subsetting*_* with the HDF-EOS library:*_ *ESML file can be used to correct the semantic tag required to subset HDF-EOS data without the need to recreate the data file*

## Science Data Analysis

- _[http://datamining.itsc.uah.edu](http://datamining.itsc.uah.edu/)_

## Data Mining

- Data Mining is the task of discovering interesting patterns/anomalies and extracting novel information from large amounts of data
- Data Mining is an interdisciplinary field drawing from areas such as statistics, machine learning, pattern recognition and others

## Iterative Nature of the Data Mining Process

- DATA

- PREPROCESSING

***CLEANING***
***And***
***INTEGRATION ***

- MINING

***SELECTION***
***And***
***TRANSFORMATION ***

- DISCOVERY

- KNOWLEDGE

***EVALUATION***
***And***
***PRESENTATION ***

## ADaM Engine Architecture

**Preprocessed**
**Data**

**Data**

**Translated**
**Data**

**Patterns/**
**Models**

**Results**

- Preprocessing

- Analysis

- Clustering
-     K Means
-     Isodata
-     Maximum
- Pattern Recognition
-     Bayes Classifier
-     Min. Dist. Classifier
- Image Analysis
-     Boundary Detection
-     Cooccurrence Matrix
-     Dilation and Erosion
-     Histogram Operations
-     Polygon Circumscript
-     Spatial Filtering
-     Texture Operations
- Genetic Algorithms
- Neural Networks
- Others...

- Selection and Sampling
-     Subsetting
-     Subsampling
-     Select by Value
-     Coincidence Search
- Grid Manipulation
-     Grid Creation
-     Bin Aggregate
-     Bin Select
-     Grid Aggregate
-     Grid Select
-     Find Holes
- Image Processing
-     Cropping
-     Inversion
-     Thresholding
- Others...

**Processing**

## Reasons for Building a Data Mining Environment

- Provide scientists with the capabilities to iterate
- Allow the flexibility of creative scientific analysis
- Provide data mining benefits of
- Automation of the analysis process
- Reduction of data volume
- Provide a framework to allow a well defined structure for the entire analysis process
- Provide a suite of mining algorithms for creative analysis
- Provide capabilities to add “science algorithms” to the framework

##  ADaM : Mining Environment for Scientific Data

-  **ADaM : Mining Environment for Scientific Data**

*The system provides knowledge discovery, feature detection and  content-based searching for data values, as well as for metadata.*
*contains over 120 different operations *
*Operations vary from specialized science data-set specific algorithms to various digital image processing techniques, processing modules for automatic pattern recognition, machine perception, neural networks, genetic algorithms and others*

## Extensibility of ADaM

**ADaM Mining Engine**

**Analysis**
**Modules**

**Input**
**Modules**

**Output**
**Modules**

## Reasons for using ADaM for Scientific Data Analysis

- Provide scientists with the capabilities to iterate
- Allow the flexibility of creative scientific analysis
- Is a powerful tool for research and analysis given the volume of science data
- Extremely useful when manual examination of data is impossible
- Allows scientists to add problem specific algorithms to the ADaM toolkit
- Minimizes scientists’ data handling to allow them to maximize research time
- Reduces “reinventing the wheel”

## Mission/Project/Field Campaign Coordination

- Electronic Collaboration

## Strategic and Tactical Coordination

- Data acquisition and integration from multiple platforms, instruments and agencies for quick exploitation
- Intra-project  communications before, during, and after CAMEX campaigns

***Technologies to coordinate complex projects***

## CAMEX-4 Coordination:pre-flight

- CAMEX-4 Coordination:pre-flight

## CAMEX-4 Coordination:in flight

- CAMEX-4 Coordination:in flight

## CAMEX-4 Coordination:post-flight

- CAMEX-4 Coordination:post-flight