Data-Centric Analysis of Science Return for Human-Directed Robotic
Geology

###### Second Year Annual Report

Sponsor: Applied Information Systems Research Program,

National Aeronautics and Space Administration

###### Project Number NAG5-11981

Geb Thomas, PI

Nathalie Cabrol, NASA Ames

Robert Anderson, JPL

Contact Information

Geb Thomas

##### Department of Mechanical and Industrial Engineering

The University of Iowa

Iowa City, IA 52246

Phone: (319) 335-5936

Fax: (319) 335-5669

URL: http://grok.ecn.uiowa.edu

##### June 11, 2004 

## Introduction

This work develops the relationships between the data collected by a
robot on another planet and the geomorphologic conclusions made by
geologists at ground control (Figure 1).  These models will be developed
analytically in the first year, verified with post-hoc analysis of a
robotic field test in the second year, and confirmed in the third year
by using the model to predict what information the science team will
collect and what information the science team will find most useful in
forming scientific hypotheses. The model will form the basis for a
series of data products including software that enables the science team
to interactively explore a dataset\'s strengths and limitations relative
to specific scientific conclusions. Other products of the research
include recommendations for science operations and mission planning and
performance guidelines for artificial intelligence algorithms on-board
planetary robots. The proposal addresses the Applied Information Systems
Research Program by increasing the scientific return on research with a
new approach to modeling information use, exploiting modeling and
analysis advances in information technology for the benefit of Space
Science, and promoting strong interdisciplinary collaborations among
interface developers, mission planners and geologists.  This work
generates innovative concepts that will provide the foundation for new
robot designs and mission concepts, and will facilitate the integrated
and systematic investigation of Mars and other planetary programs
involving rovers.

![](media/image1.wmf){width="5.038888888888889in"
height="1.8777777777777778in"}

## Review of progress made on plans from last year's report

Progress to date is proceeding as scheduled. The following tasks are
taken verbatim from last year's report in the section titled "Plans for
Next Year." Progress on each task is individually addressed.

**Year 2003 2004**

**Task** J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D

1.6
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--

2.1 \-\-\-\-\-\--

2.2 \-\-\-\-\-\-\--

2.3 (est) \-\-\--

2.4 \-\-\-\-\-\-\-\-\-\-\-\-\--

2.5 \-\-\-\-\-\--

6.  \-\-\-\-\-\--

7.  \--

8.  \--

9.  \--

**1.6 Refine model, build tools to visualize model parameters, publish
the model and supporting data on a web site, conference proceedings and
journal articles (All, a continuing effort).** In the last year,
particular attention was paid to the measurement of the roundness and
sphericity of sedimentary grains, which turned out to be helpful in the
analysis of the MER missions, as well as the analysis of slope and
obstacle size. New analysis techniques were developed which
significantly increased the resolution of our principle dependent
variable for measuring scientific performance: hypotheses and
observations. In 1999 our analysis of the behavior of 15 scientists
during 3 days of field testing yielded 24 specific scientific
observations and hypotheses. In our test this year, our analysis of
19,000 lines of transcripts from 3 geologists yielded 422 specific
hypotheses and observations. We expect that this capability to resolve
the scientific process with greater resolution will enable greater model
refinement in the future. During this period we have submitted 2 journal
articles and 3 conferences papers. Two more journal articles and 2 more
conference papers are presently being revised for submission based on
the results of last year's work.

> **2.1 Analyze sediment experiments. Determine if follow-on is
> necessary or useful. Publish results. (Thomas, 1 month).** The
> analysis of the sediment experiments was successfully completed and
> has been submitted for journal publication and has been presented at
> the annual IEEE Systems, Man and Cybernetics Conference. The
> experiments provided three useful results. 1) Geologists may reliably
> estimate the length of features (specifically, rock size) larger than
> 2 pixels in the image. 2) Geologists are not very consistent in their
> estimation of the roundness and sphericity of rocks and although the
> image resolution may bias the estimates towards perceiving the rocks
> as rounder and more spherical than physical estimates of the same
> specimens, the bias is negligibly small compared to the uncertainty
> and inconsistency in the geologist estimates. This result suggests
> that geologists may need to receive specialized training to improve
> the consistency of their estimates of roundness and sphericity if
> subtle distinctions in these values are to be used as important
> geological evidence. 3) Observer estimates of the distribution of
> rocks of different size in the same image was surprisingly inaccurate.
> This specific task may be performed more reliably by machine. However,
> we are aware of no algorithms that can reliably distinguish individual
> rocks to perform this calculation, although several are in
> development.
>
> **2.2 Complete Investigate mid-terrain feature analysis (Thomas, 3
> months)** This experiment was successfully completed and is currently
> being prepared for publication. The results suggest that observers
> consistently overestimate the size of objects beyond 5 m of the rover,
> even when a three dimensional model and a physical mock-up of the
> remote scene is available to the observer. Observers consistently
> overestimate the size of targets by as much as 200%. However, we also
> discovered that a relatively simply estimation strategy based on the
> position of the horizon in the image can significantly reduce the
> magnitude of errors.
>
> **2.3 Develop modeling showing implications from small-scale analysis
> to large-scale analysis. (Thomas, 3 months).** This experiment was
> also successfully completed and has been submitted for publication.
> The results suggest that observers significantly overestimate the
> slope of terrain near the rover. We successfully modified an existing
> psychological model for slope perception which seems to account for
> and predict the magnitude of this misestimation, based on the angle of
> regard of the rover's cameras. This knowledge may be used to correct
> for perceptual biases introduced by geologist analysts and mission
> planners, or may be used to better train these professionals with the
> objective of eliminating the bias.
>
> **2.4 Identify appropriate field test in which we can observe
> geologists interpreting geomorphology of a region (Thomas, 1 month).**
> Working with Bob Anderson, we decided to conduct the field test in a
> desert region near Grey Mountain, Arizona, a few hundred yards from
> where the Mars Explorer Rover Operational Readiness Tests were
> conducted.
>
> **2.5 Prepare data recording equipment (audiotape and lapel
> microphones, video monitoring and background computer logging
> technology) for tracking scientists' behavior, use of data during a
> simulated mission, and scientific hypothesis formation. (Thomas, 2
> months).** Two students from Iowa prepared a set of lapel microphones
> and video cameras and equipped the mission control room for the test.
> The equipment worked more effectively than in past missions and
> transcripts were successfully made for nearly all remarks made by the
> geologists during the mission. However, the data logging software on
> the computers had to be turned off because it slowed computer
> performance significantly after approximately ½ hour. The audio signal
> on the outdoor video recorder was largely lost because of wind noise.
>
> **2.6 Participate in the experiment (All, 1 month).** The field
> experiment was successfully conducted in October, 2003.
>
> **2.7 Create audiotape transcripts. Analyze videotapes to connect
> audio with computer and data source. Also make a video demo of
> critical incidents (Thomas, 6 months).** Approximately 19,005 lines of
> transcript were created from the tapes. These transcripts were
> analyzed by hand by two analysts and the results compared. The
> inter-analyst reliability was above 95% for most analyses. Because the
> audio signal was lost from the outdoor microphones, critical incidents
> were identified from the lapel microphone data.
>
> **2.8 Take geologists to test site to compare conclusions made during
> the field test with live, on-site interpretation. Note misperceptions
> and suspected reasons for misperceptions (Cabrol and Anderson, 2
> months).** This portion of the field test was completed, yielding 20
> specific analysis errors, which are exhaustively analyzed in the
> doctoral dissertation prepared and defended by Iowa graduate student
> Jacob Wagner.
>
> **2.9 Compare data to hypothesis cycle (from transcripts and critical
> incidence tapes) with model to check accuracy of the model (Cabrol and
> Anderson, 2 months)**. The laboratory experiment would have predicted
> two of the errors observed in the experiment: the misestimation of the
> scale of the remote environment on the first day and the misperception
> of a streambed for an Aeolian (wind-blown) channel. However, most of
> the other errors would not have been predicted by existing models of
> robotic geology, indicating the need for greater model refinement.
>
> **2.10 Prepare recommendations with robotic geology operations (All, 1
> month).** These recommendations are included in Wagner's dissertation,
> but will be discussed in detail at the next team meeting in July.
>
> **2.11 Share results with science and engineering team (All, 1
> month)**. This activity will also take place in July, although the
> dissertation has already been formally published.

## Challenges and Opportunities:

> Last year we expressed concern that building the whole model of
> robotic geology at once would be unwieldy. We decided instead to
> develop some low-level predictions, specifically with analyzing images
> of small-grain sediment, slopes and obstacles. Apparently this new
> strategy is effective. The sediment analysis proved useful to analysts
> working on the current Mars mission. The analysis of slopes and
> obstacles predicted errors observed in the field experiment, although
> the field experiment was clearly not designed to create such errors.
> However, the fact that only 5% of the errors observed in our field
> experiment were predicted by our laboratory research suggests that
> there are still many discoveries ahead.

As expected, there were no more operational readiness tests for the MER
mission, so we conducted our own field test. The limitation of this
approach is that we replaced a rover with a tripod and camera. We don't
believe that this change significantly reduced the validity of the
results from the geological analysis and the behavior of the scientists.
However, we will team with Carnegie Mellon on an ASTEP proposal this
Fall to complete a field test with an active rover.

## Plans for Next Year

From the original proposal, the plans for Year 3 are as follows:

### Year 3

**3.1 Identify appropriate field test for stage 3 (Cabrol and
Anderson).** This task has been completed. We will participate in the
exploration of the Atacama Desert.

**3.2 Pre-visit test site and make predictions regarding most useful
data based on model and mission operations recommendations (Cabrol or
Anderson and Thomas).** This task is no longer necessary, since we are
no longer attempting to develop a comprehensive model of robotic
geology. However, we will ask the scientists to make specific
observations at the end of each of the 10 mission days, some of which
will relate to predictions made by our existing models. These
observations will be compared against direct measurements made in the
field.

**3.3 Record audio, video and computer logs during field test (whoever
was not at field site previously, for double blind protocol).** This
task will be conducted during the field test in October. We hope to
collaborate with ethnographers working both in the mission control room
and in the field.

**3.4 Make transcripts and use video and computer logs to connect
observations to data (Thomas).** This task will continue following the
techniques used during our last field test.

**3.5 Take geologists in field test to the field site to record any
misperceptions they had and the reasons for the misperceptions (Cabrol
and Anderson).** Unfortunately, this will not be possible during the
2004-2005 field season because it would violate the protocol established
in the ASTEP mission design, which includes this step at the end of the
2005-2006 field season. We hope to complete this step during the 2005-06
season, if funds are available.

**3.6 Compare transcripts and field observations to predictions.
Determine if predictions were accurate and, if not, why not (All).**
This step will be conducted as originally conceived.

**3.7 Use mission data to refine model (All).** This step will be
conducted as originally conceived.

**3.8 Refine mission operations recommendations (All).** This step will
be conducted as originally conceived.

**3.9 Prepare a report suggesting key areas in which artificial
intelligence could increase mission effectiveness by recognizing
important data, comparing artificial intelligence performance with human
performance on similar task and examining implications of differences
(All).** This step will be conducted as originally conceived.

3.10 Share results with science team (All). This step will be conducted
as originally conceived.

## Publications resulting from this effort

1.  **Xiang, Z., Thomas, G.**, Cabrol, N., Grin, E., and Anderson, R.C.,
    "Slope perception from monoscopic field images: applications to
    mobile robot navigation," *IEEE Systems, Man and Cybernetics, Part A
    (submitted April, 2004).*

2.  **Wagner, J., Thomas, G., Glasgow, J.**, Cabrol, N., Grin, E., and
    Anderson, R.C., "The Accuracy of Sediment Size, Shape, and
    Distribution Measurements from Robotic Geological Images,"
    (submitted in January 2004), *IEEE Transactions on Systems, Man and
    Cybernetics, Part A.*

3.  **Wagner, J., Thomas, G., Glasgow, J.**, Anderson, R.C., Cabrol, N.,
    and Grin, E. (2004), Error-associated behaviors and error rates for
    robotic geology, Human Factors and Ergonomics Society 48th Annual
    Meeting, New Orleans, LA, September 20-24.

4.  **G. Thomas**, Engineering Robotic Geology for Mars Exploration,
    Institute of Industrial Engineers Annual Conference, March 2004,
    Houston, TX.

5.  **J. Wagner, G. Thomas**, "Assessing Geologic Image Interpretations
    Errors Occurring in Extraterrestrial Robotic Exploration," IEEE
    Systems, Man and Cybernetics, October 6-8, 2003, Washington, D.C.