# Presentation: 664222

## Automated Digital Terrain Model Generation from HiRISE & LROC Imagery

- Michael Broxton (PI)
- CMU/NASA Ames
- Laurence Edwards (Co-I)
- NASA Ames Research Center
- Ross Beyer (Co-I)
- SETI Institute/NASA Ames
- AISR PI Meeting May 5th, 2008

## Preparing for the Lunar and Mars Data Firehose

- Data volumes from HiRISE & LRO are substantially larger than any previous mission. (note the log scale in the figure to the right...)
- Currently, critical efforts such as building the Clementine or Lunar Orbiter mosaics or updates to the lunar control network take years to complete.
- Human intensive processes need to be automated so that data can be processed more efficiently.

- Source:  B. A. Archinal, L. R. Gaddis, R. L. Kirk, T. M. Hare, and M. R. Rosiek. _Urgent Processing and Geodetic Control of Lunar Data_.
- Workshop on Science Associated with the Lunar Exploration Architecture,  2007.

- With today’s technology, we can process more data for less.

## The Ames Stereo Pipeline

- Compute a large number of stereo pairs of reasonable quality with no human intervention.
- Our international partners (ESA and JAXA) have/are also developing automated DEM generation pipelines.

- Better DEM coverage will be help...
- Studies of planetary morphology
- Landing Site Selection & Analysis
- Simulation & Training Efforts
- Terrain Relative Navigation
- ... to name a few.

- Upcoming State of the Art

- Current State of the Art

- Photogrammetric Workstation w/ Human Operator (days to weeks)

- Automated stereo matching w/ human quality control.  (minutes to days)

- Problem: Given multiple images from different viewing angles, compute the 3D terrain.

- Higher throughput due to automation means more, higher resolution DEMs!

## Data Flow Through the Stereo Pipeline

## Built on the NASA Vision Workbench

- Overview
- Modular, extensible, C++ machine vision and image processing library (Linux, OS-X, Win32)
- Developed as a framework for unifying image processing work at NASA Ames
- Designed for easy, expressive coding of efficient image processing algorithms.
- Modules
- Core (abstract datatypes & utilities)
- Camera (models & calibration)
- Cartography (geospatial images)
- GPU (HW accelerated processing)
- HDR (high-dynamic range images)
- Interest Point (tracking & matching)
- Mosaic (composite & blend huge images)
- Stereo Processing (high-quality DEMs & 3D models)

- Stereo Pipeline

- Mission Specific Code

- Vision Workbench

- Image

- Image Processing

- FileIO

- ISIS File I/O

- InterestPoint

- Image Alignment

- Stereo

- Dense Stereo Correlation

- Stereo Camera Geometry

- Cartography

- DEM Generation

- Georeferenced File I/O

- Camera

- VW Camera Models

- ISIS Camera Models

- Image File I/O

- _[http://ti.arc.nasa.gov/visionworkbench/](http://ti.arc.nasa.gov/visionworkbench/)_

- The Stereo Pipeline is a relatively thin application built upon the open source Vision Workbench toolkit.

## Geo-Referenced Data Products

## Processing Really, REALLY Large Images

- The Vision Workbench handles arbitrarily large images via intelligent caching and a flexible abstraction of an image called an “image view.”
- Image operations are evaluated lazily, allowing for optimization down the line.
- Processing occurs one tile at a time, and is usually driven by the output operation (i.e. writing a tile to disk).
- DiskImageView, BlockCacheView, ImageViewRef, block_rasterize(), and blocked-savvy write-image()/FileIO
- Scalable performance on multi-threaded machines (soon to include Columbia)
- Thread and ThreadPool/WorkQueue objects
- Specifically targeting the stereo correlator, outlier rejection, and stereo intersection algorithms.

- HiRISE
- (20,000x40,000)

- Apollo Metric Camera
- (16,000x16,000)

- CTX
- (5064x
- 16000)

- LROC
- (10000x50000)

- HRSC
- (5184x
- 16000)

- MOC-NA
- (2048x4800)

- MER
- (1024x1024)

- Nominal Resolutions for Various Imagers.  All sizes given in pixels.

*New!*

## Improved Stereo Matching

- Discrete Correlation

- Sub-pixel Correlation

- 3. Consistency Checks
- Left/Right Cross Check
- Median Filtering
- Other methods to be added soon:
- Epipolar, photometric, continuity/smoothness constraints.
- Robust Cost Function

- Template Region
- (from Left Image)

- Right Image

- Search Area

- 1. Discrete Correlation
- Find the integer offset (disparity) that minimizes the sum of absolute difference between template region and the right image.
- For speed:
- Coarse-to-fine processing.
- Disparity search sub-regioning1
- Box filter-optimized correlator.

- 2. Sub-pixel Refinement
- Fit a 2D convex quadratic surface to the nine nearest points in correlation fitness space.

- Candidate Disparity(dx, dy)

- 1. Changming Sun. _Rectangular Subregioning and 3-D Maximum-Surface Techniques for Fast Stereo Matching_. In Proceedings of the IEEE Workshop on Stereo and Multi-Baseline Vision (2001)

## Integration with USGS ISIS

- The USGS Integrated System for Imagers and Spectrometers
- Serves as a central repository for authoritative spacecraft telemetry and instrument geometry for orbital imagers.
- Camera models (geometric & radiometric)
- SPICE Kernels
- 3D terrain models (MOLA, ULCN 2005)
- A collection of useful tools for making maps
- Orthoprojection onto 3D terrain
- Map projection/mosaicing
- ISIS has a heritage of use by the planetary science community
- ISIS 3.0 (just released) is a complete rewrite in C++

- Added a File I/O driver for reading/writing ISIS CUBE files
- Allows the Stereo Pipeline to take advantage of the close association between ISIS files and their associated SPICE kernels and camera models.
- Added a wrapper for the ISIS camera models to the VW Camera module
- Takes advantage of ISIS’s high quality, peer-reviewed camera models.  (Useful in many algorithms! Stereo reconstruction, ortho-projection, mosaicking, etc.)

- 3D Surface Reconstruction from Apollo Metric Camera Frames AS15-M-0081 & AS15-M-0082

*New!*

## Upcoming Work in 2009

- Year 2 of AISR-funded Research
- First public open-source release of the Stereo Pipeline as an additional install on top of ISIS.
- Operational stereo reconstruction capability (and training) for HiRISE team.
- Formal validation of HiRISE DEMs with USGS.
- Software Release Strategy
- NASA Open Source Agreement (NOSA), an OSI-approved, unrestrictive open source license
- VW version 2.0 alpha (including stereo algorithms) already available on-line:
- _[http://ti.arc.nasa.gov/visionworkbench/](http://ti.arc.nasa.gov/visionworkbench/)_

- Lunar Mapping and Modeling Program (Constellation, LPRP)
- USGS, JPL, GSFC, and Ames are joining forces to support the Constellation program’s Lunar mapping and modeling needs.
- Systematic stereo processing of Apollo Metric Camera Data (pilot project underway) - stereo coverage of ~20% of the lunar equatorial region.
- Preparation for the arrival of LROC data.
- MRO CTX Instrument Team
- (Co-I Larry Edwards)
- “On-demand” DEM generation service
- MSL Mission Planning

## Education & Outreach

- We are in the midst of a major geospatial information revolution.  Map data is becoming universally and easily accessible on-line.
- We should leverage open standards and the Geobrowser platforms as tools for communicating with colleagues (and the public).
- These tools will help to sift through the incredible volume of new lunar data.

- Unprecedented Data Availability & Fusion

- _[http://moon.google.com](http://moon.google.com/)_

- Geospatial Image
- Browsing/Indexing

- Upcoming State of the Art

- Current State of the Art

- Access to data in archival form through GIS interfaces. (cumbersome, though powerful)

- Geobrowsers, web interfaces, and extensible API’s.  (efficient & intuitive, powerful in its own way)

## Google Moon

- Visible Base Map

- Elevation Map

- Geological and Topo-
- graphical Charts

- Apollo Traverse Maps

- Apollo Placemarks

- Apollo Panoramas

- http://moon.google.com

## Questions?

- Michael Broxton
- Intelligent Robotics Group (IRG)
- NASA Ames Research Center
- _michael.broxton@nasa.gov_

## Backup: Pinhole Camera: Epipolar Lines

- Provided that you have prior knowledge of the intrinsic and extrinsic camera parameters, the epipolar constraint dictates that, given a pixel in one image, you need only search along a straight line to find the matching pixel in a second image.

## Linear Pushbroom Camera Model

## Linear Pushbroom Camera: Epipolar Hyperbolas

- Due to the orthographic projection of the y-axis in linear pushbroom imagery, the epiploar constraint between linear pushbroom images is a hyperbola.

## Linear Pushbroom Camera: Epipolar Hyperbolas

- Without the Epipolar constraint, correlation can be very expensive.
- Our solution: Pre-align the images and search in a small 2D area.
- Linear 3x3 projective matrix
- This is accomplished using feature-based image registration and RANSAC.