Data Mining for CEOP Data
October 9, 2002

Shin-ichi Sobue/NASDA1

Purpose of Data Mining“The extraction of hidden predictive
information from large databases.”

Hidden!? Data2

Example - Medical Research
July 18, 2002
Healthy Lifestyle May Help Cut
Alzheimer’s Risk, Scientists Find
Study of Alzheimer’s - 1,449 people for 21 years.
High cholesterol and high blood pressure raise therisk of developing Alzheimer’s more than carrying
ApoE-e4, a gene variation considered the mostimportant genetic risk factor for the disease.
High cholesterol and high blood pressure!
3

Business Example - Beer and Diapers

On Thursdays
Diapers --> Beer
1. Displays of Beer and Diapers close
together (near the registers).
2. ?
4

Preprocessing: Removing Seasonality• Remove seasonality to see events(anomalies) of interest.
– Monthly Z Score• Subtract monthly mean and divide bymonthly standard deviation– 12 month moving average– Discrete Fourier Transform– Singular Value Decomposition5

Preprocessing: Removing Seasonality

6

Time Series Dataz Global snapshots of values for a number of variables

on land surfaces or water.
z Monthly values over a range of 10 to 50 years.
z Gridded values (NPP 0.5° x 0.5°, SST 1° x 1°)
NPP.
PressureNPP.

Pressure.

Precipitation

Precipitation

SSTSST

Latitudegrid cellLongitude

Time7zone

Data Clustering• Cluster Formation– Find regions of the land or ocean which have similar
time series behavior.
• Teleconnections– Teleconnections - time series behavior is (very)
similar over widely separated points on the Earth.
• Interested in relationships between regions, not “points.”• For ocean, clustering based on SST (Sea SurfaceTemperature), SLP (Sea Level Pressure), etc.
• For land, clustering based on NPP or other variables,e.g., precipitation, temperature.
8

K-means Clustering• The K-means algorithm– “K” is number of (nonoverlapping) clusters– “means” - center of the cluster is the mean ormedian of the “nearness”of the points in its
cluster, where “nearness” is defined by asimilarity function (Pearson’s correlation
coefficient).

9

K-Means Clustering of Raw NPP and Raw SST

10Discovering new Teleconnections• Find land and sea clusters that are highly correlated, toidentify new potential teleconnection patterns.
– Produced 100 clusters for the land (NPP) and 100 clusters
for the sea (SST).
– Calculate correlations between sea and land clusters.
– Next page shows diagram of sea cluster 19 and land clusters
56 and 58 (correlations of 0.56 and 0.50).

• Sea cluster 19 is highly correlated (-0.77) with thePacific Decadal Oscillation
11

One Sea Cluster and Associated Land Clusters

12Discovery of Ocean Climate Indices••••••••••••

1. SOI ( Southern Oscillation Index)
2. NAO (North Atlantic Oscillation)
3. AO (Artic Oscillation)
4. PDO (Pacific Decadel Oscillation)
5. QBO (Quasi-Biennial Oscillation Index )6 .CTI (Cold Tongue Index)
7. WP (Western Pacific)
8. ANOM12 (Normalized version of NINO12)
9. ANOM3 (Normalized version of ANOM3)
10. ANOM4 (Normalized version of NINO4)
11. ANOM34 (Normalized version NINO34)(Note: 1, 6, and 8-11 are El Nino related indices)13

Discovery of Ocean Climate Indices• SST clusters are potential Ocean ClimateIndices (OCIs).
• Determine if the clusters match known OCIs.
• Evaluate the influence of clusters (potentialOCIs) on land points.
• For clusters that don’t match know OCIs,
consider them to be potential OCIs andconduct further analysis to see if they areuseful and interesting.
14

Shared Nearest Neighbor (SNN) Clustering• Find the nearest neighbors of each data
point.
– In this case data points are time series.
– Examine the similarity between pairs ofpoints in terms of how many nearestneighbors two points share.
• Eliminate noise, which are points with lowdensity.
• Build clusters around the core points,
which are points with high density.
15

SST Clusters
1 07 SNN Clu s te rs fo r De tre nd e d Mo nth ly Z S ST (1 9 58 -1 9 98 )90

60

la titude

30

0-30

-60

-90
-180

-150

-120

-90

-60

-30

0longitude
163060

90

1 20

1 50

1 80

SST Clusters that Correspond to El Nino Climate Indices
EL Nin o Re la ted S S T Clus ters
90

NiñoRegion
60

la titude

30

07578 67 94

RangeLongitude

Range
Latitude

1+2(94)90°W-80°W

10°S-0°3(67)150°W-90°W

5°S-5°N

3.4(78) 170°W-120°W

5°S-5°N

4(75) 160°E-150°W

5°S-5°N

El Nino Regions Definedby Earth Scientists-30-60-90-180 -15 0 -12 0-90-60-3 0030
longitude6090

171 201 50

1 80

SNN clusters of SST that arehighly correlated with El Nino
indices, ~ 0.93 correlation.

ADaM
Algorithm Development and Mining System
Results
Results

Translated
Data

Data
Data

Preprocessed
Preprocessed
Data
Data

Patterns/
Patterns/
ModelsModelsProcessing
Input
HDF
HDF-EOSGIF PIP-2
SSM/I Pathfinder
SSM/I TDR
SSM/I NESDIS Lvl 1B
SSM/I MSFC
Brightness Temp
US RainLandsat
ASCII GrassVectors (ASCII Text)
Intergraph Raster
Others...

Preprocessing
Selection and Sampling
Subsetting
Subsampling
Select by Value
Coincidence Search
Grid Manipulation
Grid Creation
Bin Aggregate
Bin SelectGrid Aggregate
Grid SelectFind Holes
Image Processing
Cropping
Inversion
Thresholding
Others...

Analysis

OutputClustering
K Means
Isodata
Maximum
Pattern Recognition
Bayes Classifier
Min. Dist. Classifier
Image Analysis
Boundary Detection
Cooccurrence Matrix
Dilation and Erosion
HistogramOperationsPolygon
Circumscript
Spatial Filtering
Texture Operations
Genetic Algorithms
Neural Networks
Others...

GIF Images
HDF-EOSHDF Raster Images
HDF SDS
Polygons (ASCII, DXF)
SSM/I MSFC
Brightness Temp
TIFF Images
Others...

18

ADaM - Input
Data Readers and Writers• Binary
• GIF
• HDF
• HDF-EOS• TIFF• ASCII
19

ADaM - Preprocessing• Grid Operations• Subsetting• Image Processing
Collage
Crop
DilateEqualize
ErodeGabor
FilterInvertOverlay
Pulse Coupled Neural Network
Quantize
RotateSpatial Filters
Statistics
20

ADaM - Analysis
1. Genetic Algorithms
2. Pattern Recognition• Bayes Classifier• Isodata clustering operation• K Means• Max/Min Operation• Multiple
PrototypeMinimumDistanceClassifier• Decision Tree Classifier• Oblique Decision Tree
• Recursively Splitting Neural Network• Minimum Distance Classifier
21

ADaM - Plan Builder Client

22

Space Time Toolkit

23Space Time Toolkit
Can visually integrate virtually any
geographic data set, regardless ofdifferences in their spatial andtemporal representation.
Low-level sensor data
Gridded products
Vector based GISdata
3D model output,etc.
24

“Types” of Data Mining• Relationship Mining - mining for various types of relationships
includes most types of data mining.
– Cause or Indicator Relationships - Find the actual causes of an “event”(e.g. determine if land clearing has disturbed hydrologic runoff and/orflood frequency).
– Effect Relationships - The analysis of events having regional climate
impact (e.g., volcanic eruptions, desert flash floods).
– Correlation Relationships - Study of earthquakes (e.g. the space-timecorrelations involving triggered slips, foreshocks and aftershocks).
– Linkage Relationships - An event at one time and place is related to an
event at another time and place (e.g., El Nino and drought in Indonesia).
Feature Transformation– Prediction Relationships - Predict storm tracts and changes in intensity.
25

“Types” of Data Mining (continued)• Exploratory Pattern Mining - Look for unexpected
spatial/temporal patterns long-term data that cover alonger period (e.g. three to five years),• Complex Process Characterization - Develop predictive
statistical models that can be applied to areas such as
seismic activity or the spreading of fire.
Physical/mechanistic models that can be applied to areas
such as earthquake fault modeling, interactions between
crust/mantle and stress transfer.

26

WTF-CEOP Activities - next 6 months (1/2)
1. WTF-CEOP website: list information on research in data
mining of Earth Science data.
2. WTF-CEOP website: list examples of data mining.
Example of data mining query:••

Investigation: It has been shown that warming in the equatorial Pacific
Ocean in the El Nino area affects the strength of winds at the 200 mb
level in the tropical Atlantic. During drought years in West Africa,winds increase from west to east at the 200 mb level, in the Central
Atlantic.
Query: The user wants to find the time periods where the SST anomaly
in the El Nino region, is less than -1°C and winds, at 200 mb, in theCentral Atlantic, are greater than -5.00 m/s. For those time periods
the user then wants to study the precipitation and vegetation data, for
the western Sahel.
27

WTF-CEOP Activities - next 6 months (2/2)
3. Survey CEOP scientists for what type of data mining would
be useful to them. Add these queries to item 2 above.
4. WTF-CEOP website: develop an on-line library ofexamples of visualization techniques for 3D/4D data anddata mining.
5. Prepare a CEOP database testbed (has the same softwareenvironment as CEOP). This database testbed will be
used to adapt and test CEOS data mining algorithms
before they are installed on CEOP.
6. Identify “candidate” CEOP algorithms - algorithms thathave been developed by (NASA, ESA, etc.) scientists thatcan be used by CEOP. NASDA will install and test thealgorithms on the CEOP database testbed.
7. Survey COTS Data Mining software.
28