# Presentation: 637159

## Remote Imageryand Supercomputing:A Match Made “OnEarth”

- Lucian Plesea
- Jet Propulsion Laboratory

## Introduction

- OnEarth is the name of the public JPL Web Map Service (WMS) server, hosting a half arc-second, global earth image mosaic.
- Has 3600 times more pixels than previous global earth images, at about 15m per pixel
- This mosaic contains about three trillion pixels, more than five Terabytes of data.
- Creating such a large mosaic was only possible by using large parallel computers.

## WMS Global Mosaic

## Atacama Desert, Chile, South America

- Atacama Desert

## Atacama Desert, South America, IR View

## Atacama Desert, South America, Default View (IR and Visual)

## Atacama Desert, South America, Visual View

## Coral Atolls, Maldives, Indian Ocean

## Belcher Islands, Hudson Bay, Canada, North America

## Brandberg Formation and Sand Storm, Namibia, Africa

## Bosphorus Strait, Istanbul, Eurasia

## Polders, Netherlands, Europe

## Etna Volcano, Italy, Europe

## Fuji-San, Japan, Asia

## Grand Canyon Fire of 2000, USA, North America

## Glacier, Canada, North America

## Fog, Iceland, Atlantic Ocean

## Shrimp Farms, Manila Bay, Philippines, Asia

## Namib Desert, Namibia, Africa

## Rock and Sand, Namib Desert, Namibia, Africa

## Algae and Mud, Queens Channel, North Australia

## Paris, France, Europe

## Everglades Park, Florida, USA, North America

## Zion Park, Utah, USA, North America

## The Project

- NASA is leading an effort to increase the accessibility of remote imagery.
- The availability of a recent, high resolution, global coverage map of the earth was seen as an important component of this effort.
- The project started in earnest in Jan 2003, with an expected completion date of Dec 2003
- Release One on-line since April 2004

## WMS Access: Custom Images

## Project Components

- Dataset: GeoCover, orthorectified Landsat 7
- Storage: RASCHAL, a 40TB storage cluster
- Data Format: Journaling Image File Format
- Application: Parallel Image Mosaic Engine
- Data Access: Network Image Protocol
- Computation Resource: IPG
- Mosaic: Release 1
- Mosaic availability: WMS

## Dataset

- NASA purchased global orthorectified Landsat 7 as a part of the Scientific Data Purchase Phase II.
- Expected 8000 scenes, with a final delivery date of July 2003. Currently 10700 scenes have been received at JPL
- Each scene has 9 spectral bands, in UTM coordinates, 500-550 Megabytes.

## For this project, and a few others, a very large storage system was necessary.
Raid Again Storage using Commodity Hardware And Linux (Raschal), is a 40TB NAS storage cluster, built in-house.
RASCHAL became operational April 2003, and has been in continuous use since then.
OnEarth uses about 15TBytes

- Raid Again Storage using Commodity Hardware And Linux (Raschal), is a 40TB NAS storage cluster, built in-house.
- RASCHAL became operational April 2003, and has been in continuous use since then.
- OnEarth uses about 15TBytes

- Storage

## Storage

## Data Format

- A Journaling Image File Format is used extensively, for both the input scenes and the output mosaic.
- It is a tiled, multispectral and multi-resolution file format that supports lossless and lossy compression at the tile level.
- A level of indirection in data access, adding journaling features which ensures file consistency

## Data Format

- Input images have been converted to this format, using bz2 (block arithmetic) lossless compression.
- Lower resolution input images are pre-built, for lower resolution test runs.
- The output mosaic itself is stored using libz compression, less efficient but faster.
- Files larger than 1Tbyte have been generated during this project

## Components

- Dataset: GeoCover, orthorectified Landsat 7
- Storage: RASCHAL, a 40TB storage cluster
- Data Format: Journaling Image File Format
- Application: Parallel image mosaic engine
- Data Access: Network Image Protocol
- Computation Resource: IPG
- Mosaic: Release 1
- Mosaic access: WMS

## Application

- An image mosaic builder for very large geographic datasets, applying in a single pass the coordinate transformation, color correction and scene blending.
- Unique capabilities include UTM to geographical reprojection , blend mask driven data selection and feathering, and per band first order polynomial color correction.
- Implemented as a chain of custom SGI Image Library Operators.

## Application

- The input scenes are first analyzed, data location and the statistical distribution of each spectral band are determined. This data can saved for later use.
- The data saved in the analysis stage can be  used to generate a global spectral matching solution.
- Blend masks are generated, derived from the location information.

## Input Scenes

- Each band stored in a separate file, as large as 17,000x17,000
- Images are in UTM projection, in 60 different zones.
- Image swath has different orientations, depending on altitude

## Blend Mask

## Application

- Many algorithms for the spectral matching are possible, a few have been tried.
- The current approach uses the MODIS derived, “Blue Marble” 1 Km image brightness to guide the Landsat 7 panchromatic band brightness.
- The other 8 bands are using the coefficients from the panchromatic band, followed by a recursive filtering to reduce neighbor scene differences.

## Grayscale Blue Marble

## Panchromatic Band

## Application

- The mosaic engine works on a small area of the output at a time, using only a few scenes.
- The matching correction is applied if available, then a geometrical transformation matches the scene projection and resolution to the output.
- The images are now in a common projection and can be combined, using a blend mask weighted sum.

## Application

- Since the mosaic engine only operates on a small area of output at a time, checkpoint and restart capabilities are built in.
- Use of the Journaling File Format allows for multiple processing cluster to generate output for the same file at the same time.
- The mosaic builder code is using a shared memory architecture, and uses the SGI Image Vision Library.

## Data Access

- Computation resources are located remote from the storage resources, direct access to the data is not possible.
- An image specific data access subsystem allows small regions of the input and output images to be transferred independently. The Network Image Protocol is used, separating the location and specific file format from the application.

## Data Access

- Data tiles are transferred in either raw or compressed format, under user control.
- A single plain text setup file provides access to all input files, in this case, more than 70,000.
- The Network Image Protocol was implemented as an Image Format Library module, no significant changes were required to the mosaic engine

## Computation Resource

- The mosaic engine is a shared memory application, it benefits from faster CPUs and a very large memory footprint.
- Chapman, a 1024 CPU Origin 3000, with 600MHz CPUs and 256MB per CPU was used.
- This machine is located at AMES, and connected to JPL via a 15MB/sec link, the NREN network component of the IPG

## Computation Resource

- Four groups of 32 CPUs were running simultaneously to produce the mosaic, using about 50000 CPU-hours total.
- This architecture provided a balance of data access and computational loads, achieving a CPU load between 30 and 60 percent.
- Peak data transfer rates of 12MB/sec were seen, with an average of 2.5MB/sec data read and 1.25MB/sec data write

## Components

- Dataset: GeoCover, orthorectified Landsat 7
- Storage: RASCHAL, a 40TB storage cluster
- Data Format: Journaling Image File Format
- Application: Parallel image mosaic engine
- Data Access: Network Image Protocol
- Computation Resource: IPG
- Mosaic: Release 1
- Mosaic access: WMS

## WMS Global Mosaic

- There are nine separate images, at three different resolutions. Coverage area is 180W to 180E and S85 to N85.
- 600 GB, panchromatic band, 0.5 arc-second (15m), 2,592,000x1,224,000, or 3 trillion pixel.
- 50 GB, thermal band, high and low gain, 2 arc-second per pixel (60m).
- 900GB, three visual and three near IR bands, stored at 1 arc-second (30m).

## Access: WMS

- Access to the mosaic is best done via the WMS server. A prototype client is available on http://OnEarth.jpl.nasa.gov
- The server is implemented as a CGI application, and uses the same technologies as the mosaic application.
- Server provides color selection, pan-sharpening, multiple projections.
- Image control using Styled Layer Descriptor.

## To Be Continued

- Pending:
- Web Site Refresh
- Packaging for delivery to interested parties
- Possible Further Development
- Improve and maintain the mosaic
- SRTM Elevation
- Web Terrain Service

## Slide 51

## Credits

- Dave Curkendall, shares the blame for the whole thing
- Kacie Shelton, the tape and data shepherd
- Jimi Patel, built most of RASCHAL
- Richard Schreyer, summer student, wrote the current WMS parser engine.
- George Percivall, manages the WMS Global Mosaic project for NASA

## Sponsors

- WMS Global Mosaic
- Geospatial Interoperability Office, YO
- Image Access Technologies:
- ESTO-CT
- Computing Resources:
- AMES CNIS Program