Data-Centric Analysis of Science Return for Human-Directed Robotic
Geology

###### Final Report

Sponsor: Applied Information Systems Research Program,

National Aeronautics and Space Administration

###### Project Number NAG5-11981

Geb Thomas, PI

Nathalie Cabrol, NASA Ames

Robert Anderson, JPL

Contact Information

Geb Thomas

##### Department of Mechanical and Industrial Engineering

The University of Iowa

Iowa City, IA 52246

Phone: (319) 335-5936

Fax: (319) 335-5669

URL: http://grok.ecn.uiowa.edu

##### January 23, 2006 

## Introduction

This work developed the relationships between the data collected by a
robot on another planet and the geomorphologic conclusions made by
geologists at ground control (Figure 1).  These models were developed
analytically in the first year, verified with post-hoc analysis of a
robotic field test in the second year, and confirmed in the third year
by using the model to predict what information the science team will
collect and what information the science team will find most useful in
forming scientific hypotheses. The model formed the basis for a series
of data products including software that enables the science team to
interactively explore a dataset\'s strengths and limitations relative to
specific scientific conclusions. Other products of the research include
recommendations for science operations and mission planning and
performance guidelines for artificial intelligence algorithms on-board
planetary robots. The proposal addressed the Applied Information Systems
Research Program by increasing the scientific return on research with a
new approach to modeling information use, exploiting modeling and
analysis advances in information technology for the benefit of Space
Science, and promoting strong interdisciplinary collaborations among
interface developers, mission planners and geologists. 

![](media/image1.wmf){width="5.038888888888889in"
height="1.8777777777777778in"}

The principle outcomes of the project were a series of papers that
demonstrated the following findings:

-   Changes in science team effectiveness correlates with the number of
    comments made by the science team about data provided by the rover.
    This finding is described in Pudenz, E., Glasgow, J., Thomas, G.,
    Coppin, P., Wettergreen, D., Cabrol, N. Searching for a Quantitative
    Proxy for Rover Science Effectiveness, Proceedings of the 2006
    Conference on Human-Robot Interaction, March 2-4, 2006, Salt Lake
    City, Utah.

-   During a large field test, we observed that although the science
    team requested a large amount of data in the form of three
    dimensional panoramic images, they didn't seem to use this
    three-dimensional data in many of their scientific conclusions. We
    recommended that they reduce the frequency with which they request
    such data and instead request other types of data from the rover.
    Initial indications suggest that this further improved the team's
    scientific effectiveness, although the final analysis is still being
    conducted. The evidence for this finding is presented in Glasgow,
    Justin, Erin Pudenz, Geb Thomas, Nathalie Cabrol, Peter Coppin and
    David Wettergreen (2005), Observations of a Science Team during an
    Advanced Planetary Rover Prototype Field Test, Ro-MAN Conference,
    August 13-15, Nashville, TN, 2005.

-   Novice rover operators have difficulty perceiving the height of
    distant objects from monoscopic, panoramic images provided by a
    rover, even when the rover operator is provided with substantial
    context aids including a physical model of the local environment.
    This finding was demonstrated in both a laboratory environment and
    in a large-scale field test. The evidence for this finding is
    presented in Kanduri, A.K., Thomas, G., Cabrol, N., Grin, E. and
    Anderson, R.C. (2005), \"The (In) Accuracy of Novice Rover Operators
    Perception of Obstacle Height from Monoscopic Images,\" Systems, Man
    and Cybernetics A 35(4), 505-512.

-   Operators have difficulty estimating the slope of terrain near the
    rover. This was demonstrated with both controlled laboratory
    experiments and in field tests. The evidence for this is currently
    under review in a paper tentatively titled Xiang, Z., Thomas, G.,
    Cabrol, N., Grin, E., and Anderson, R.C., "Slope perception from
    monoscopic field images: applications to mobile robot navigation." A
    copy of the working paper is attached here as Appendix A.

-   Another series of laboratory studies measured the limits of
    resolution for a geologist to analyze sedimentary grains with a
    rover-mounted camera. These limits are currently being applied to
    analysis conducted as part of the MER mission. Wagner, J., Thomas,
    G., Glasgow, J., Cabrol, N., Grin, E., and Anderson, R.C., "The
    Accuracy of Sediment Size, Shape, and Distribution Measurements from
    Robotic Geological Images." This article is also under review, but a
    copy of the working paper is appended here.

-   A series of images and associated geologist analysis was created for
    estimating rock roundness and grain distribution. Two separate
    research groups have used these images as part of studies
    investigating how to automate this process. The standard datasets
    allow this research to be conducted and compared to human
    performance on the same task. This archive is online at
    http://grok.ccad.uiowa.edu/\~jacob/Mars/images.html

## Review of progress made since last year's report

The project generally proceeded as scheduled. The following tasks are
taken verbatim from last year's report in the section titled "Plans for
Next Year." Progress on each task is individually addressed.

**Year 2003 2004**

**Task** J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D

1.6
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--

2.1 \-\-\-\-\-\--

2.2 \-\-\-\-\-\-\--

2.3 (est) \-\-\--

2.4 \-\-\-\-\-\-\-\-\-\-\-\-\--

2.5 \-\-\-\-\-\--

6.  \-\-\-\-\-\--

7.  \--

8.  \--

9.  \--

### Year 3

**3.1 Identify appropriate field test for stage 3 (Cabrol and
Anderson).** This task is been completed. We participated in the
exploration of the Atacama Desert as part of the Life in the Atacama
project led by Carnegie Mellon University.

**3.2 Pre-visit test site and make predictions regarding most useful
data based on model and mission operations recommendations (Cabrol or
Anderson and Thomas).** In year two, we determined that this task was
not necessary, since we are no longer attempting to develop a
comprehensive model of robotic geology. However, we did ask the
scientists to make specific observations at the end of each of the 10
mission days, some of which will relate to predictions made by our
existing models. These observations were compared against direct
measurements made in the field.

**3.3 Record audio, video and computer logs during field test (whoever
was not at field site previously, for double blind protocol).** During
the 2004 field test we recorded 265 audio tapes, each lasting
approximately 3 hours. In addition we collected the server logs, the
rover command logs, and a complete archive of all the data analyzed
during the missions.

**3.4 Make transcripts and use video and computer logs to connect
observations to data (Thomas).** . From the 265 tapes, 61 were selected
for transcription, from which 22,492 lines of transcript were collected.

**3.5 Take geologists in field test to the field site to record any
misperceptions they had and the reasons for the misperceptions (Cabrol
and Anderson).** In year 2, we decided that this would not be possible
during the 2004-2005 field season because it would violate the protocol
established in the ASTEP mission design, which includes this step at the
end of the 2005-2006 field season. However, a follow-on proposal
provided funds to continue this research through the AISR project (Award
Number NNG05GA51G) and we completed the field trip in January 2006. Four
members of the LITA science team traveled with six members of our
research team to the regions of the desert explored by the rover. We
prepared lists of hypotheses for the scientists at each of the locations
visited by the rover. The analysis of this data is ongoing as part of
the work with the new grant.

**3.6 Compare transcripts and field observations to predictions.
Determine if predictions were accurate and, if not, why not (All).** The
analysis of transcripts and field observations is nearly complete for
the 2004 season. Of particular interest is the validation of the
difficulties scientists face in estimating the scale of objects seen in
images provided by the rover. This is particularly evident in objects
visible in the panoramic image that are more than approximately 20m from
the rover.

**3.7 Use mission data to refine model (All).** This work is still
on-going. Several are being prepared for submission or are in the review
process.

**3.8 Refine mission operations recommendations (All).** Our analysis at
the end of the 2004 season suggested that the science team had requested
a large amount of data, particularly from the stereoscopic panoramic
cameras, that did not contribute directly to the science results. We
recommended that the science team reduce the number of stereoscopic
images that they requested. In the 2005 season, the team chose to
download other data than stereoscopic panoramic images. We also
recommended that the science team increase the formality with which they
reported their analysis each day and made several suggestions regarding
the manner in which the findings were checked (ground-truthed). Many of
these suggestions were implemented by the 2005 field season.

**3.9 Prepare a report suggesting key areas in which artificial
intelligence could increase mission effectiveness by recognizing
important data, comparing artificial intelligence performance with human
performance on similar task and examining implications of differences
(All).** This task is currently being pursued as part of the follow-on
study. Our current approach emphasizes the analysis of panoramic images
to determine which areas are of greatest interest to the scientists. The
rover team has also expressed interest in analyzing which features and
feature characteristics in a panorama are of greatest interest to the
scientists. JPL and CMU have both expressed interest in developing
image-processing techniques to analyze rock size and density from the
test set collected in year 1 of this project.

3.10 Share results with science team (All). This step will be conducted
as originally conceived.

## Challenges and Opportunities:

Last year we discussed the following as a challenge/opportunity:

> As expected, there were no more operational readiness tests for the
> MER mission, so we conducted our own field test. The limitation of
> this approach is that we replaced a rover with a tripod and camera. We
> don't believe that this change significantly reduced the validity of
> the results from the geological analysis and the behavior of the
> scientists. However, we will team with Carnegie Mellon on an ASTEP
> proposal this Fall to complete a field test with an active rover.

Now that we have both experiences to compare, it is clear that there are
advantages and disadvantages to both. The principle advantage of
participating in a live field test is the face validity and the
opportunity to observe challenges as they unfold. The limitation to the
approach is the lack of experimental control and the large number of
confounding variables that can make it difficult to specifically
identify the cause of a particular effect.

One of these challenges is the limited number of participants in the
study, which reduces our ability to generalize findings from a full
field test. The principle question in this case is: was the effect
caused by the system or is it a consequence of the individual analyst?

Ultimately, we conclude that an appropriate balance must be struck
between the field studies and laboratory studies to confirm each finding
with appropriate statistical rigor.

## Publications and Presentations resulting from this effort

1.  Kanduri, A.K., Thomas, G., Cabrol, N., Grin, E. and Anderson, R.C.
    (2005), \"The (In) Accuracy of Novice Rover Operators Perception of
    Obstacle Height from Monoscopic Images,\" *Systems, Man and
    Cybernetics A 35(4), 505-512.*

2.  Pudenz, E., Glasgow, J., Thomas, G., Coppin, P., Wettergreen, D.,
    Cabrol, N. Searching for a Quantitative Proxy for Rover Science
    Effectiveness, Proceedings of the 2006 Conference on Human-Robot
    Interaction, March 2-4, 2006, Salt Lake City, Utah.

3.  Glasgow, Justin, Erin Pudenz, Geb Thomas, Nathalie Cabrol, Peter
    Coppin and David Wettergreen (2005), Observations of a Science Team
    during an Advanced Planetary Rover Prototype Field Test, Ro-MAN
    Conference, August 13-15, Nashville, TN, 2005.

4.  Thomas, G., Coppin, P., Cabrol, N., Wettergreen, D., Pudenz, E.,
    Glasgow, J. (2005), Collaborative Virtual Environments for Control
    of Planetary Exploration Rovers, Special Session on Human Robot
    Interaction, Human Computer Interaction / Virtual Reality Conference
    2005, July 22-27, 2005, Las Vegas, NV.

5.  Thomas, G., J. Wagner, Z. Xiang, A. Kanduri, and J. Glasgow (2004),
    Analytical Rover Operations Development, IEEE Systems, Man and
    Cybernetics Conference, The Hague, The Netherlands.

6.  Wagner, J., Thomas, G., Glasgow, J., Anderson, R.C., Cabrol, N., and
    Grin, E. (2004), Error-associated behaviors and error rates for
    robotic geology, Human Factors and Ergonomics Society 48th Annual
    Meeting, New Orleans, LA, September 20-24.

7.  G. Thomas, Engineering Robotic Geology for Mars Exploration, IIE
    Annual Conference, March 2004, Houston, TX.

8.  Keynote speech, "Science with Robots on Earth, the Moon and Mars,"
    ASME Student Leadership Training Seminar, Iowa City, IA 9/24/05.

9.  Robotic Space Exploration and the GROK Lab a presentation for the
    Engineering Alumni, 6/04, The University of Iowa

## Appendix A

**Slope Perception from Monoscopic Field Images: Applications to Mobile
Robot Navigation \[Working Document\]**

Zhen Xiang, Geb Thomas, Kristopher Thornburg,

Nathalie Cabrol, Edmond Grin, Robert C. Anderson

A submission to the International Journal of Human-Computer Studies

This work was conducted in the

Graphical Representation of Knowledge Laboratory at the University of
Iowa

4225 Seamans Center for the Engineering Arts and Sciences

Iowa City, Iowa 52242 USA

SUMMARY

When remotely navigating a mobile robot, operators must estimate the
slope of local terrain in order to avoid areas that are too steep to
climb or that slope so steeply downward that the operator would lose
control of the rover. Although many rovers are equipped with sensor
systems to aid the operator in this task, it is sometimes necessary to
estimate slopes from two-dimensional images, either when planning
operations, or when the operator wishes to monitor the results of a
sensor system. This experiment compares the operator's estimates of the
slope in Martian terrain with the actual slope determined from
three-dimensional data. The 10 participants overestimated the slope of
the indicated regions by an average of 19 degrees (standard deviation 16
degrees). An analytic model of the error, based on psychophysical
analysis, accurately predicts the average magnitude of the errors.
Implementation of this model eliminates an average amount of participant
error. However, the large estimate variance within and between
participants and images still poses a challenge for veridical slope
estimation.

INTRODUCTION

Sheridan's Supervisory Control Theory (Sheridan (1992)) outlines how
operators control most semi-autonomous mobile robots, or rovers. The
control architecture explains that an operator controls a
semi-autonomous machine by specifying goals that the robot can execute
on its own. The theory describes five roles the operator must fulfill:
plan, teach, monitor, intervene, and learn. Crucial to safely
controlling a rover, the operator must conduct the plan and monitor
roles completely, consistently, and accurately.

Planning (Sheridan (1992)) requires the operator to understand the
mission goals, the rover's current state, its capabilities and
limitations, and its physical environment. The operator must then
conceptualize a task sequence that the rover can understand and execute
to achieve the mission goals. Ideally, the task sequence would also
optimize certain mission goals such as completing the task in the
minimum time, moving the furthest distance with the least energy, or
minimizing risk to the rover.

Monitoring (Sheridan (1992)) requires the operator to continuously
compare the current state of the rover with expectations, searching for
failures or process variations. Effective monitoring requires the
operator to accurately predict how the rover should perform the
specified tasks, interpret the information coming back from the rover,
and detect and track small variations between the predicted and actual
states. The operator's understanding of the rover, its behavior, and the
environment should be detailed enough to differentiate between
significant and insignificant differences in expected and actual
performance. Although both planning and monitoring require a precise
understanding of the geometry of the rover's environment, people have
trouble perceiving this geometry with the information that a rover
generally provides (Lewis, Wang, Hughes, and Liu (2003), Casper and
Murphy (2003), Drury (2003)).

A thorough understanding of the local geometry is required for the
operator to perform the plan and the monitor roles. When planning the
operator determines favorable paths for navigating the rover around
obstacles or dangerously steep slopes. The final mission plan must
accommodate any environmental constraints. Monitoring requires an
accurate understanding of the environment's geometry, aiding the
operator in developing a precise expectation of the rover's behavior as
it traverses the environment. Operators cannot sufficiently complete the
plan and monitor roles because of difficulty perceiving environmental
geometry as presented by the rover (Lewis et al. (2003), Casper and
Murphy (2003), Drury (2003), Perrone (1980)). In order to design a
system to aid operators in completing their roles adequately and
efficiently, we must first understand the difficulty experienced by
operators.

Common difficulties operators have when navigating a rover include
disorientation, overestimation of geographical feature size, problems
with determining rover orientation, and problems with distance and slope
estimations. In short, many operators describe the navigation of a rover
as "looking through a soda straw (Steinfeld (2004))." Some systems
currently use 3D or stereoscopic displays, however these systems can
lead to operator disorientation (Steinfeld (2004)). Casper and Murphy
(2003) describe operators having difficulty trying to determine if the
rover was in the upright position while van Erp (1998) showed that
operators consistently overestimate distances while navigating a vehicle
from a video monitor. A major limiting factor in navigation is operator
overestimation of height in monoscopic images (Kanduri (2005)). Noticing
this effect, Woods, Tittle, Feil, and Roesler (2004)) highlight examples
in which operators cannot determine whether a rover can pass through an
opening or over an obstacle. Proffitt, Bhalla, Gossweiler, and Midgett
(1995) and Proffitt (2001) have shown that people are notoriously poor
at determining the angle of an incline in virtual as well as real
environments. From these examples, it is evident that rover operators
have little understanding of the rover's geographical environment.

In an effort to increase the operator's understanding of the rover's
geographical environment, Lewis et al. (2003) developed a gravity
referenced view (GRV) system, which rotates the camera view of the rover
along one axis with respect to gravity. This system allows the operator
to visualize the current roll of the rover, improving the accuracy of
determinations of the rover's ability to traverse the environment,
therefore avoiding some rollover accidents. In the experiments conducted
by Lewis et al. (2003), the GRV system significantly reduces but does
not eliminate the number of rollover accidents. This system improves the
monitor role of the operator, making it slightly easier for operators to
estimate the rover's current orientation and environmental position. The
operator still requires assistance in making geographic determinations
during the plan role.

To make the role of plan manageable for the operator, an aid must
present the operator with information about the geographical environment
ahead. This understanding relies on an unbiased perception of the
terrain or slope over which the rover must navigate. People cannot
naturally make accurate estimations of slope; therefore, the system must
accurately present slope information to ensure satisfactory completion
of the plan role. Effectively designing an interface to present this
information requires an understanding of how operators currently
perceive slope, and how the perceived slope will be used in a
navigational decision.

BACKGROUND

With respect to the observer's viewpoint along the z-axis, a slope has
two components: pitch and roll (Figure 1). Pitch is defined as the
surface's rotation away from vertical about the x-axis, which is
equivalent to the angle between the y-z projection and the y-axis. Roll
is the rotation from the vertical about the z-axis, which is equivalent
to the angle between the x-y projection and the y-axis (Proffitt et al.
(1995)). Most slopes in a natural terrain have non-zero pitch and roll.
Methods similar to the ideas employed by Lewis et al. (2003) aid
operators in determining roll but not pitch, so this paper exclusively
considers the accurate perception of pitch.

![](media/image2.wmf)θ~r~, with respect to gravity. In general, this
angle could affect the both the rover's pitch and roll but only rover
pitch is considered here. Many rovers use a camera mounted on a pan and
tilt gimbal, so that the camera can be directed towards different
positions in the rover's environment. In Figure 2, the camera aims
downward at an angle t, which we refer to as the camera tilt. Horizontal
displacement of the camera constitutes the camera's pan. The camera in
Figure 2 is imaging a region in front of the rover. This region has an
overall slope tendency and a local region that has a separate slope
tendency, defined by θ, which is the angle between the normal of the
sloping surface and the camera's direct line of sight. The operator's
task is to estimate the slope of the local region with respect to
gravity. This slope estimate depends on the rover's angle with respect
to gravity, the camera tilt, and the operator's estimate of θ.
Generally, an accelerometer fixed to the rover's body measures angle
with respect to gravity. Position encoders on the tilt and pan gimbal
record the camera position.

![](media/image3.jpeg){width="6.002777777777778in"
height="2.423611111111111in"}

*Figure 2: Illustration of the task geometry involved in estimating a
local terrain slope with an image taken by a rover equipped with a
camera on a pan and tilt gimbal.*

It is not clear, however, how the operator incorporates the numerical
estimates of robot tilt and camera tilt in his or her estimation of
slope. For example, the operator may assume that the rover is level
(i.e., θ~r~ = 0) and estimate the camera pitch (t + θ~r~) from the
texture gradient of the general terrain in the scene. Such an assumption
may be incorrect or prone to greater error than relying on the numerical
values from the accelerometer and gimbal angle encoder. Sparse research
covers how these values aid in improving slope estimations; however, we
chose to explore the accuracy of slope estimates without this data to
account for a worst-case scenario.

Many rover designers have circumvented operators' imprecise slope
perception by incorporating hardware and software that directly estimate
local terrain slopes with three-dimensional imaging systems, such as
stereoscopic cameras or laser rangefinders. The operator can use this
information to provide quantitative estimates of the slope, or to
generate a display of safe versus unsafe navigation regions. Because of
the limits of sensor resolution, such solutions are generally only
possible in the immediate vicinity of the rover. The operator must still
estimate slope from images when on-board measurement systems are not
available, when planning rover operations beyond the range of the
on-board sensors, or when visually confirming the results of the
automatically generated slope estimates. Work described here
investigates the operator's ability to make these judgments without a
three-dimensional map.

Existing perceptual models, developed with geometric situations much
less complex than those investigated here, suggest that people's
estimates of slope tend to bias in the direction towards the projection
plane (Epstein and Park (1964)). This bias is evident in laboratory
studies of optical slant, in which research participants estimate the
relative angle between their viewpoint and a stimulus surface (e.g.
Gibson (1950), Smith (1959)). Research in geographical slant perception,
in which research participants estimate the absolute slope of terrain
when they are directly perceiving it or when they are looking at images
of natural and virtual environments (Proffitt et al. (1995),
Creem-Regehr et al. (2003)) shows the same bias. This existing research
suggests that observers misestimate the angle of regard or projection
plane of the visual information. The research reported here considers
the hypothesis that slope perception is inaccurate in the visual
conditions provided by a mobile rover interface. More importantly, it is
the first to demonstrate that this inaccuracy is large enough to limit
effective rover operation.

There are several ways that an operator may improve performance over the
conditions tested here. It is likely that training and experience with a
rover would improve an operator's ability to perceive slope, so long as
the training includes effective feedback. Without appropriate feedback,
such as seeing the rover directly, an operator might form and maintain a
false perception of scale, for example, imagining that the rover is
larger than it is. Without seeing the rover approach and climb a steep
slope, the operator might easily misconstrue how steep a slope the rover
could climb. An operator might use familiar objects in the imagery to
understand the scale of the environment. The operator might use
landmarks to reconstruct the geometry of the terrain from different
viewing positions and thus have an independent method for evaluating the
environment's geometry. The presence of familiar objects or landmarks
could create a perceptual feedback loop that might provide the feedback
necessary to refine slope estimation with controlled experience with a
rover. In unfamiliar environments, however, performance may not improve
beyond the unacceptable levels reported here.

Before developing a strategy to mitigate perceptual challenges, it is
necessary to investigate the perceptual mechanisms that may be the
source of the inaccuracy, and to provide some baseline performance
measure with which to understand the significance of the problem. For
this, we consider two perceptual theories proposed by Perrone.

[Perrone's Two Models of Optical Slant Perception]{.underline}

Perrone (1980) analyzed the texture pattern in visual images and
concluded that a veridical perception of optical slant is possible if
the observer can perceive direction and accurately estimate the length
of the vector extending from the viewing position to perpendicularly
intersect the extended plane of the slanted surface (line OA in Figure
3). In practical situations, this critical location (position A in
Figure 3) is not directly perceivable by the viewer because of the
limited field of view, or frame. In natural environments, this position
may not be physically observable because the observer's position and the
slanted surface are often located so that this important position does
not lie on the limited extent of the physical surface. When the critical
position is not observable, Perrone argues, observers form an assumption
about the missing information. Perrone analyzes the consequences of two
different assumptions that observers could make.

![](media/image4.png){width="5.53125in" height="3.122916666666667in"}

*Figure 3: Geometrical relationships for Perrone's Perpendicular Model.
The observer at position O views the slanted surface, DR. The observer's
vertical field of view is ZV. See text for definitions of other
variables.*

Perrone's first model (Figure 3), which we refer to as Perrone's
Perpendicular Model, proposes that observers erroneously take the
slanted surface to be perpendicular to the ray along the bottom of the
image (OQ). Perrone's rationale for this counter-intuitive hypothesis is
that the normal visual environment contains "an abundance of information
about the direction of the perpendicular from the eye to the surface we
are viewing." When the image frame removes this normally available
information, the observer assumes that the point of intersection (A) is
the bottom of the image (Q). With this assumption, the gradient is
proportional to the angle of the optical slant, sin θ. In other words,
Perrone proposed that in the absence of the direct perception of the
true perpendicular, observers assume that the vector OQ is perpendicular
to the slope surface. Consequently, they perceive the slope to lie in
the direction Q'C'R' rather than QCR. As a result of this assumption,
the perceived optical slant, β (\<C'OQ'), differs from the actual
optical slant, θ (\<COA). The two parameters are related by
$\beta = \text{arccos}(1 + \text{tan}v\text{tan}\theta)^{\frac{- 1}{2}}$,
where v is half of the visual angle vertically subtending the visible
part of the slope surface. Despite the basis of Perrone's Perpendicular
Model on optical slant, it extends to geographic slant. The next section
considers the introduction of a variable for camera tilt, which aids in
extending the model to geographic slant. In Perrone (1980), this model
is shown to be consistent with Gibson's experiments with optical slant
perception.

Perrone's second model, which we call Perrone's Reference Model,
proposes that observers incorrectly assume that the line from observing
point O to the bottom edge of the visible surface is parallel to the
ground plane. The rationale for this hypothesis is that, in natural
environments, the horizontal information is generally clear from the
position of the visual horizon. When visual cues are impoverished,
people are very poor at estimating the location of the horizontal
direction, despite available posterial cues.

According to Perrone's Reference Model, the observer sees the surface in
the correct relative position, but misestimates the reference. If the
observer takes the reference plane to be perpendicular to ground, the
error in the slant estimate is equal to error in the reference plane
position. Consequently, the perceived slant, φ, is given by: θ = φ − ν,
for slopes tilting away from the projection plane, as illustrated in
Figure 4.

![](media/image5.jpeg){width="5.982677165354331in"
height="3.353543307086614in"}

*Figure 4: Geometric Relationships for Perrone's Reference Model.*

Perrone compared the minimum absolute value estimate for the
perpendicular and reference models using previously published optical
slant estimates; the fit was within 3.25 degrees. Later, Perrone and
Wenderoth (1981) followed by Perrone (1982) proposed a more general
model based on the idea that the direction from the observing point to
the nearest part of the stimulus is perceived to be the true
straight-ahead direction. The model may also apply to more general
geometric cases, such as when the stimulus tilts forward relative to the
visual line of sight. However, this revised model is essentially
equivalent to Perrone's Reference Model, in which slopes generally tilt
away from the projection plane, which is usually the case when judging
potentially navigable terrain in natural environments. In addition, the
later model considers the convergent angles of a regular texture pattern
on the stimulus surface, reducing its applicability in natural
environments. Perrone's models are the only models found to date of
human slope perception published in peer-reviewed literature.

This paper seeks to measure slope perception error for rover navigation
in natural, desert-like environments to determine whether observers tend
to overestimate slopes. This paper also seeks to determine which of
Perrone's models may be adapted to this domain, given that the problem
of estimating slopes from images of natural environments considered here
is somewhat more complex than the geometric situation considered by
Perrone. Perrone's work relied on perceptual experiments in which a
participant viewed a tilted planar surface through a round aperture. One
complexity is that a natural visual scene may contain many regions with
different slopes. A second complexity is that the observer can not use
the perception of gravity or isokinetic feedback to relate the optical
slant to an absolute reference frame, a potentially large source of
error. Instead, the observer must use the observed natural terrain to
estimate the direction of gravity with respect to the viewed scene. This
is analogous to estimating both the robot tilt and the camera tilt from
the visual field of view. Previous perceptual experiments controlled or
eliminated these factors (Perrone (1980), Proffitt et al. (1995), Gibson
(1950), Smith (1959)). Consequently, it is not clear that the previous
laboratory studies will directly apply to the problem of robot
navigation.

[Extending Perrone's Models to Rover Slope Estimation]{.underline}

To adapt Perrone's model to slope perception from field images, several
additional complexities must be accounted for. First, both of Perrone's
models assume that the observer views the physical stimulus directly. In
this case, the perceived texture gradient is related to the observer's
position relative to the textured surface. When viewing an image of a
slope, the perceived texture gradient is a function of both the camera's
position with respect to the surface and the observer's position with
respect to the viewed image. This arrangement weakens the observer's
ability to use his or her knowledge of body position relative to the
surface and gravity. It also introduces the possibility of a distorted
perception of the image if the observer's line-of-sight is not
perpendicular to the image plane. Because we are interested in modeling
the perceptual limitations of a rover operator when viewing a rover
image, we do not attempt to restrict the viewer's head position or
adjust the position of the image to match the camera tilt.

The second complexity arises from the fact that Perrone's models assume
that a single sloping region fills the entire effective field of view,
in which case the center of the slope and the center of the image are at
the same position. When a region of slope is a subportion of the image,
the center of the slope and the center of the image are different. To
account for this, we calculate the geometric center of the slope and use
this position. Consequently, there is a small offset term for the region
offset added to the overall tilt of the image. Thus, we redefine t in
Perrone's model to be t', as illustrated in Figure 5.

A third complexity is the adjustment for half the vertical field of
view, v, used by Perrone, which is also based on the assumption that the
slope occupies the entire field of view. Instead, we use v', which is
half of the vertical field of view of the sloping subregion of the
image, which is also illustrated in Figure 5.

![](media/image6.jpeg){width="6.001388888888889in"
height="2.373611111111111in"}

*Figure 5: Geometric considerations for extending the models.*

The fourth complexity is related to the observer's need to estimate the
absolute slope, rather than the relative slope modeled by Perrone. This
is problematic because all the cues related to the direction of gravity
are indirect. If the general tendency of the terrain is perpendicular to
gravity, the global horizon and texture gradient provide information
about the direction of gravity. If the global terrain is not
perpendicular to gravity, however, the horizon and texture gradient will
not reflect the direction of gravity, only the general trend of the
landscape. For the sake of this study, we choose to address this based
on Perrone's two models and our experiment instruction. For the
Reference Model, only the direct estimation of the tilt of local terrain
is required. For the Perpendicular Model the observer must perceive the
direction of gravity in the image in order to estimate the camera tilt.
We presume that the observers estimate the camera tilt from indirect
cues, such as the texture gradient of the whole image and the position
of the rocks and sediment, for example. It is not clear exactly what
cues contribute to the perception of the camera tilt angle, but the
images clearly provide some indication. This assertion leads to the
assertion that the observer can also estimate the tilt of the camera,
also through indirect cues.

If the observer can estimate the camera tilt, then he or she can
estimate the absolute slope of a region by transforming the optical
slant to global coordinates. For Perrone's Perpendicular Model (Figure
3), the real slope QCR tilts up to Q'C'R'. In this situation, α is the
slope perceived by observers and α = 90 °-- t' -- v'. This is the
Modified Perpendicular Model. For Perrone's Reference Model (Figure 4),
OQ is assumed to be parallel to the ground, α is the perceived slope,
and α = t' + s~0~ +v', where S~o~ is as indicated in Figure 3. This is
the Modified Reference Model.

Our objective is to measure the accuracy of a novice observer's estimate
of slope in conditions similar to those experienced by a rover operator
and to determine whether an observer perceives slopes in an image of a
natural environment in the manner predicted by the Modified
Perpendicular Model or the Modified Reference Model. Specifically, we
hypothesize that observers will perceive the slope of areas within a
region according to either: (a) 90°-- t' -- v', or (b) t' + s~0~ +v'.

**METHOD**

[Participants]{.underline}

Ten participants (3 females, 7 males) aged between 18 and 60 years were
recruited locally; 8 were University of Iowa students. All the subjects
have normal or corrected to normal vision.

[Stimuli]{.underline}

The stimuli were taken from the Mars Pathfinder mission data archive
(NASA (2003)), which includes stereoscopic image pairs taken by the Mars
Pathfinder mission lander and a three-dimensional terrain model of the
terrain within 2-15 m of the lander generated from these images (Jet
Propulsion Laboratory (2003)). The terrain model uses a set of over 90
images as texture maps. Each of these 256×248 pixel images subtends a
14.0°x13.6° field of view with respect to the lander camera viewpoint.
About 120 points in each image are tied to three-dimensional coordinates
in the terrain model. The experimental stimuli consist of a subset of 15
images from the full dataset. Images were selected based on the
following criteria: (1) selected images should not contain views of the
lander, (2) selected images should display a range of slope angles at
varying distances, and (3) the selected slopes should be planar and the
uncertainty of the coefficients of regression for a plane passing
through six selected points on the slope should determine pitch within
0.5 degrees.

Estimates of the selected slopes in the image subset were calculated
from the three-dimensional terrain model. To estimate a slope within an
image, we first selected three points from the terrain model
corresponding to locations in the sloping region in the image. An
algorithm then considered all combinations of groups of six points in
the three-dimensional data close to the three selected points. The set
of six points having the largest R^2^ value when fitted to a plane was
taken to be the best group of points to represent the slope. This
procedure resulted in six points associated with each selected slope,
along with an estimate of the slope of the corresponding planar surface
passing through the three-dimensional position of the points. All slopes
in stimulus set have a confidence interval of +/- 0.5 degrees at the 70%
confidence level. A consequence of a defining a mathematically
unambiguous sloping region was that the points are not uniformly spaced,
though it seems appropriate for natural environments with uncontrolled
geometry.

Two images were prepared for each slope in the stimulus set. The first
image was the original image annotated with a small, red, filled circle
that indicate the geometric center of the points used to estimate the
slope. The second image included the blue circles placed at the position
of each of the sample points, plus the red circle placed at the center
of the chosen points (Figure 6). These annotations served to direct the
participant's attention to the correct region of the image and specify
exactly which points were considered to be included in the slope.

![](media/image7.png){width="2.6666666666666665in"
height="2.5833333333333335in"}

*Figure 6: A sample of a fully annotated stimulus. The double-outlined
circle (which was red) indicates the geometric center of the slope; the
other outlined circles (which were blue) indicate the points used to
indicate the local region of interest.*

[Apparatus]{.underline}

A laptop with a 15'' screen displayed the images. Participants reported
their slope estimates with a slope indicator, a flat circular plate
covered with a random texture pattern mounted on a pan and tilt gimbal
that tilted in both directions about the center of the plate.

![](media/image8.jpeg){width="2.7916666666666665in"
height="3.7180555555555554in"}

*Figure 7: The slope indicator. The circular top surface rotates about a
pivot point below its center.*

[Design]{.underline}

Each subject estimated the slope of all 15 stimuli three times. Each
stimuli set was presented in a separate, randomized block. For each
trial the pitch and roll angles of the slope indicator were recorded.
The dependent variable is the estimated pitch angle, defined below.

[Procedure]{.underline}

Participants adjusted their seat and the laptop screen angle to a
comfortable working position. For each stimulus, first the image
annotated with the location of all the points in the region was
displayed, followed by the image with just the annotation of the
geometric center of the slope. Participants could freely alternate
between the two views, but their final slope estimate was made from the
second annotated image. For each stimulus, the participants indicated
their estimate of the local slope on the slope indicator so that the
slope indicator surface would pass parallel to the plane formed by the
slope.

Participants were instructed to imagine that their eyes were at the
position of the camera and to indicate their perception of the absolute
slope of the indicated region by adjusting the angle of the tilt
indicator. They were also asked not to assume that the reference frame
provided by the table top was parallel to the terrain, but to
exclusively rely on their perception of the top of the slope indicator
to indicate their perception of the local slope.

**RESULTS**

A total of 450 estimates were collected. Participants significantly
overestimated pitch with a mean error of 19° (SD 16°, t(449) = 24.68, p
\< .0005). Figure 9 presents the pitch estimate error as a function of
stimulus image.

![](media/image9.wmf){width="4.854166666666667in"
height="2.8020833333333335in"}

*Figure 8: Pitch Estimates vs. stimulus image. The error bars indicate
one standard deviation of the combined participant estimates.*

For each image, the observed pitch errors (estimate minus actual) were
compared with the errors predicted by the Modified Perpendicular and
Reference Models. Figure 9 present the average and predicted errors of
the Modified Perpendicular and Reference Models of each stimulus. A
paired t-test indicates that the errors predicted by the models are
significantly different (t(14)= 63.08, p\<0.0005). The average error
predicted by the Modified Reference Model is 47.9 degrees less than
those predicted by the Modified Perpendicular Model. The observer pitch
estimate errors and the Modified Reference Model predictions are not
significantly different (t(449)=-1.5, p=0.933).

> ![](media/image10.wmf){width="5.135416666666667in" height="3.125in"}
>
> *Figure 9: The estimated error predicted by both models compared to
> the error observed in the experiment.*

The average standard deviation of the subject pitch errors was 14
degrees. The average standard deviation of each individual subject on
each individual image was only 4 degrees. A two-way analysis of variance
of the pitch error as a function of subject, image and subject-by-image
interaction indicated that 26%, 22% and 36% of the remaining variability
is accounted for by the subjects, images and subject-by-image
interaction, respectively.

DISCUSSION

The data suggest that veridical, absolute slope estimation from
monoscopic images may pose a significant challenge for a rover operator.
The average pitch overestimation of 19 degrees is quite large compared
to the size of slopes that may be safely traversed by rovers. If a rover
operator overestimates the magnitude of an upward slope, he or she will
likely navigate the rover to avoid the slope, which can potentially lead
to choosing inefficient paths and missing opportunities to explore
upslope areas that the rover is capable of navigating. The bias also
suggests that operators may underestimate the magnitude of down slopes,
or estimate steep downward slopes to be level or even sloping upward. In
this case, the operator might direct a rover down a potentially
dangerous slope, leading to a loss of navigation control. This condition
represents a serious potential rover hazard.

The data do not support the Modified Perpendicular Model, which proposes
that observers assume that viewing direction to the lowest point in the
projection plane is perpendicular to the ground between the bottom of
the projection plane and the position of interest. The data do support
the Modified Reference Model, which proposes that observers assume that
the viewing direction to the lowest point in the projection plane is
parallel to the ground. The between-stimulus differences between the
observer's pitch estimate error and the Modified Reference Model suggest
that other factors, such as the size of the slope, or its position
relative to the center of the image, may contribute to this slope
estimation bias.

The Modified Reference Model provides an opportunity to mitigate this
overestimation bias by correcting operator estimates. The model predicts
pitch bias as a function of the camera pointing angle and the actual
slope. During rover operations, the true camera angles are known, but
the true terrain pitch is generally not known. If the model uses the
estimated pitch from the rover operators as an approximation of the true
pitch, it will predict a bias, which may then be subtracted from the
operator bias.

![](media/image11.wmf){width="5.40625in" height="3.21875in"}

*Figure 10: Observer estimates corrected by the error predicted by the
Modified Reference Model*

Figure 10 presents the errors of observer estimates and the same
estimates corrected by the Modified Reference Model, following the above
procedure. A paired t-test comparison of the errors of each observation
with their corresponding corrected estimate indicates that the corrected
estimates had significantly less error than the original estimates
(t(449)= 50.59, p\<0.0005), with a 95% confidence interval of (19.538°,
21.117°). The amount of correction is very close to the scale of
original estimation error. A paired t-test between the corrected
estimates and the real slopes indicated that they were not significantly
different (t(449) = 1.5, p=0.134). The 95% confidence interval for the
error of corrected estimates is (-0.373, 2.784). Consequently, when
averaged across all the images, the error reduction is about 89%-98%.
For each single image, the model correction reduced the error by 40% -
99%, except for image 3, where the correction makes the estimation worse
by 125.9%. No obvious feature of image 3 suggests why it would be
estimated differently than the other images.

Further model refinements designed to reduce the between-stimulus
differences may not be as important as reducing the variance in observer
estimates. The standard deviation of 16 degrees is too large for safe
rover navigation, even if the estimates are unbiased, because the
occasional large misestimation could lead to catastrophic mission
failure. Compared to most rover navigation constraints, even an unbiased
perception of slope with such a large uncertainty would be inappropriate
for reliable navigation. Consequently, the most pressing research
question is to understand how to increase the precision with which rover
operators estimate slope. By further refining the model based on
geometric considerations, we could hope to eliminate as much as 22% of
the variation in the errors associated with image differences. However,
it may be more beneficial to study how operator pitch estimation
strategies differ, because 36% of the error variation is associated with
the subject-by-image interaction. Without specialized training, our
results suggest that participants are consistent in their estimates, as
indicated by the 4 degree standard deviation of pitch error. With
training, an operator may become even more consistent. An effective
error range for slope estimation in rover operations would be on the
order of 5%, or a standard deviation of approximately 1 degree for a 20
degree slope. Increasing the estimation precision by a factor of
approximately 4 would significantly reduce the rover's system complexity
by eliminating some of the need to carry sensors to detect terrain slope
and would increase the operator's ability to reliably supervise rover
operations.

CONCLUSIONS

Observers overestimate the absolute pitch angles of a sloping region in
the Mars Pathfinder dataset by 17.6 to 20.6 degrees at the 95%
confidence level. The bias may be corrected, on average, with a model
that presumes that observers take the ray from the camera's focal point
to the bottom of the locally sloping region to be parallel to the
horizon. This model may be used to correct for observer slope pitch
estimate biases during rover operations. However, the between-observer
and within-observer pitch estimate variation still pose significant
challenges for a rover operator attempting to estimate local slope from
a monoscopic image. The data and literature suggest that humans cannot
make accurate slope estimations in two- or three-dimensional images. In
order for an operator to be able to independently assess the quality of
three dimensional terrain information presented by a rover's
terrain-mapping subsystems, or to plan rover operations beyond the range
of such subsystems, it is necessary refine appropriate technologies and
training techniques. Devising a combination of training and technology
to improve rover operator's ability to accurately perceive local slope
would increase the operator's effectiveness in navigating the rover
reliably and safely.

ACKNOWLEDGEMENTS

The authors would like to thank the members of the GROK Laboratory at
the University of Iowa for their constructive insight and the research
participants who participated in this study.

BIBLIOGRAPHY

Apostolopoulos, D. (2001). Analytical Configuration of Wheeled Robotic
Locomotion. Ph.D. Dissertation, Robotics Institute, Carnegie Mellon
University.

Aviles, W.A., Hughes, T.W., Everett, H.R., Umeda, A.Y., Martin, S.W.,
Koyamatus, A.H., Solorzano, M.R., Laird, R.T. and McArthur S.P. (1990).
Issues in Mobile Robotics: The Unmanned Ground Vehicle Program
TeleOperated Vehicle (TOV). Proceedings of the SPIE Conference on Mobile
Robots V, Boston, MA.

Casper, J. and Murphy, R. (2003). Human-Robot Interactions During the
Robot-Assisted Urban Search and Rescue Response at the World Trade
Center. IEEE Transactions on Systems, Man, and Cybernetics-Part
B:Cybernetics, 33(3), 367-385.

Creem-Regehr, S.H., Gooch, A.A., Sahm, C.S. and Thompson, W.B. (2003).
Perceiving virtual geographical slant: action influences perception.
Retrieved January 27, 2004 from
<http://www.psych.utah.edu/~sc4002/pubs.htm>.

Drury, J., Scholtz, J. and Yanco, H. (2003). Awareness in Human-Robot
Interactions. IEEE International Conference on Systems, Man and
Cybernetics.

Epstein, W. and Park, J. (1964). Gibson's psychophysical hypothesis.
Psychological Bulletin, 62 (3), 180-196.

Fong, T., Thorpe, C. and Baur, C. (2001). Advanced interfaces for
vehicle teleoperation: collaborative control, sensor fusion displays,
and remote driving tools. Autonomous Robots, 11 (1), 77-85.

Gibson, J.J. (1950). The perception of visual surfaces. American Journal
of Psychology, 63, 367-384.

Heath-Pastore, T. (1994). Improved Operator Awareness of Teleoperated
Land Attitude. NCCOSC Technical Report 1659.

Jet Propulsion Laboratory. (2004). Spacecraft: surface operations:
rover. Retrieved March 5, 2004 from
http://marsrovers.jpl.nasa.gov/mission/spacecraft_rover_wheels.html.

Jet Propulsion Laboratory, Intelligent Mechanisms Group at NASA Ames and
Fourth Planet, Corp. (1997). High Resolution Model. Retrieved November
11, 2003 from <http://marsprogram.jpl.nasa.gov/MPF/vrml/pathvrml.html>.

Kanduri, A.K., Thomas, G., Cabrol, N., Grin, E. and Anderson, R.C.
(2005), \"The (In) Accuracy of Novice Rover Operators Perception of
Obstacle Height from Monoscopic Images,\" Systems, Man and Cybernetics A
35(4), 505-512.

Lewis, M., Wang J., Hughes, S. and Liu X. (2003). Experiments with
Attitude: Attitude Displays for Teleoperation. IEEE International
Conference on Systems, Man and Cybernetics.

Matthies, L., Gat, E., Harrison, R., Wilcox, B., Volpe, R., and Litwin,
T. (1995). Mars Microrover Navigation: Performance Evaluation and
Enhancement. Autonomous Robots, 2 (4), 291-311.

McGovern, D.E. (1990). Experiences and Results in Teleoperation of Land
Vehicles. Sandia Report, SAND90-0299, Albuquerque, NM.

Murray, D. and Little, J. (2000). Using Real-Time Stereo Vision for
Mobile Robot Navigation. Autonomous Robots, 8, 161-171.

NASA. (2003). Planetary Data System Database, Mars Pathfinder IMP
imager, Presidential Panorama. Retrieved November 11, 2003 from
<http://stardev.jpl.nasa.gov/pds/index.jsp>.

Perrone, J.A. (1980). Slant underestimation: a model based on the size
of the viewing aperture. Perception, 9, 285-302.

Perrone, J.A. (1982). Visual slant underestimation: a general model.
Perception, 11, 641-654.

Perrone, J.A. and Wenderoth, P.M. (1981). Visual slant underestimation.
In S.R. Ellis, Ed. Pictorial Communication in Virtual and Real
Environments, 496-503. London: Taylor & Francis.

Proffitt, D.R., S.H.Creem, and W.D. Zosh (2001), "Seeing mountains in
mole hills: geographical-slant perception," *Psyhological Science*,
12(5):418-23.

Proffitt, D.R., Bhalla, M., Gossweiler, R. and Midgett, J. (1995).
Perceiving geographical slant. Psychonomic Bulletin & Review, 2 (4),
409-428.

Sheridan, T. (1992). Telerobotics, automation, and human supervisory
control, Cambridge, MA: MIT Press.

Smith, A.H. (1959). Outline convergence versus closure in the perception
of slant. Perceptual and Motor Skills, 9, 259-266.

Steinfeld, A. (2004). Interface Lessons for Fully and Semi-Autonomous
Mobile Robots. IEEE International Conference on Robotics & Automation,
2752-2757.

Thomas, G., Reagan, M., Bettis III, E.A., Cabrol, N., Rathe, A.
**(1999).** Analysis of science team activities during the 1999
Marsokhod rover field experiment: implications for automated planetary
surface exploration. Journal of Geophysical Research -- Planets, 106
(E4), 7775-7783.

Urmson, C., Dias, B., and Simmons, R. (2002). Stereo Vision Based
Navigation for Sun-Synchronous Exploration. IEEE International
Conference on Intelligent Robots and System, 1, 805-810.

van Erp, J. (1998). Trade-offs Between Spatial and Temporal Resolution
in Driving Unmanned Ground Vehicles. Proceedings of the Human Factors
and Ergonomics Society 42^nd^ Annual Meeting, 1550-1554.

Woods, D.D., Tittle, J., Feil, M. and Roesler A. (2004). Envisioning
human-robot coordination in future operations. IEEE Trans. Systems, Man
and Cybernetics Part C, 34 (2), 210-218.

**FOOTNOTES**

This material is based upon work supported by NASA under award No:
NAG5-11981.

Zhen Xiang, Geb Thomas, and Kristopher Thornburg are with the Department
of Mechanical and Industrial Engineering at the University of Iowa, Iowa
City, IA, 52242 (email: \[Zhen-Xiang \| Geb-Thomas \|
Kristopher-Thornburg\]@uiowa.edu.

Nathalie Cabrol and Edmond Grin are with NASA Ames/SETI, Moffett Field,
CA.

Robert Anderson is with the Jet Propulsion Laboratory, Pasadena, CA.

FIGURE CAPTIONS

Figure 1: Definition of pitch and roll. N is the normal of the terrain
surface.

Figure 2: Illustration of the task geometry involved in estimating a
local terrain slope with an image taken by a rover equipped with a
camera on a pan and tilt gimbal.

Figure 3: Geometrical relationships for Perrone's Perpendicular Model.
The observer at position O views the slanted surface, DR. The observer's
vertical field of view is ZV. See text for definitions of other
variables.

Figure 4: Geometric Relationships for Perrone's Reference Model.

Figure 5: Geometric considerations for extending the models.

Figure 6: A sample of a fully annotated stimulus. The outlined circle
(which was red) indicates the geometric center of the slope; the other
circles (which were blue) indicate the points used to indicate the local
region of interest.

Figure 7: The slope indicator. The circular top surface rotates about a
pivot point below its center.

Figure 8: Pitch Estimates vs. stimulus image. The error bars indicate
one standard deviation of the combined participant estimates.

Figure 9: The estimated error predicted by both models compared to the
error observed in the experiment.

Figure 10: Observer estimates corrected by the Modified Reference Model.

## Appendix B

# The Accuracy of Sediment Size, Shape, and Distribution Measurements

# from Robotic Geological Images

# 

A Working Paper

Jacob Wagner^1^, Geb Thomas^1^, Justin Glasgow^1^, Nathalie Cabrol^2^,
Robert C. Anderson^3^ and Edmond Grin^2^

^1^ The University of Iowa, Iowa City, IA

^2^ SETI Institute/NASA Ames Research Center, Moffett Field, CA

^2^ The Jet Propulsion Laboratory, Pasadena, CA

### Abstract

*Quantifying and determining the source of system error within a complex
human-machine system is an important step in understanding and improving
the system performance. This work addresses the accuracy of geologic
image interpretations within the context of robotic exploration.
Experiments assessed the accuracy of geologists performing the tasks of
length measurement, shape classification, and grain distribution in
person versus an image of the grains. The inaccuracy in edge location
introduced by the camera ranged between 2 and 4 pixels. The inaccuracy
of length measurements was 2.33 pixels with a standard deviation of
2.399. The accuracy of grain distribution measurements was 42.5% of the
actual bin size with a standard deviation of 322%. The accuracy of shape
classifications was 0.151 categories for sphericity classifications and
0.114 categories for roundness classifications. The shape classification
experiment suggest lower-than-expected intra- and inter- geologist
consistency.*

Keywords: man-machine system, error, robot exploration, geology, image
analysis

### Introduction

Robotic geology, the vicarious study of the geology through a remotely
operated robot, enables geologists to work in hazardous and distant
environments of great scientific significance, such as volcanoes,
underwater, and other planets. Generally, geologists direct the robot to
collect images, chemical spectra, and other science data. Often the
scientists work closely with an engineering team that monitors the
robot's health and generates the command sequences to control the robot.

The traditional path for robot design and development has been for
engineers working with scientists to design and build the robot, then
test the robot in a field trial. Such trials have been conducted at the
Kilauea volcano\[1\]; Mt. Spur, Alaska \[2, 3\]; under the Arctic
glacial ice \[4\]; Dinosaur Tracks, Arizona \[5\]; the Atacama Desert in
Chile \[6-8\], and Black Rock Summit, Nevada \[8b\]. In these tests, a
remote robot is directed by a team of geologists and engineers to make
scientific observations of the field site terrain. Typically, the
centerpiece of the operator interface has been the generation and
presentation of photorealistic virtual environments representing the
robot\'s immediate surroundings \[5, 9\]. Throughout these tests, there
has been significant growth and sophistication in the robot systems and
a corresponding quality increase in the scientific results. Through
these efforts the field has grown from an early exploration and
investigation phase, in which different robot and interface designs were
rapidly developed and tested, into a more considered phase in which many
of the principle challenges have been identified. The remaining work is
to isolate these challenges and develop strategies to mitigate them.

A number of studies have been conducted to determine the needs of the
geologists while working through a rover. The earliest investigations
were conducted by McGreevy \[10, 11\], who studied geologists wearing
helmet-mounted displays while they explored a Mars-like environment.
These studies emphasized the types of words that geologists used and the
effect of the limited vision on their navigation and behavior. More
recent work conducted by Clancey \[12\] emphasizes the ethnographic
study of geologists as they explored the Haughton Crater in the Canadian
High Arctic. Clancey's work emphasizes the team coordination, tools, and
behaviors of the scientists simulating daily activities that might occur
during a manned Mars mission.

Robot/human interaction researchers working in related areas, such as
rovers for disaster response and military operations have also
identified a number of factors that should be considered in the context
of designing rovers to be more useful for remote exploration. Kaber,
Onal and Endsley \[13\], explored the effect of different levels of
autonomy on the situational awareness of a robot operator. Woods et al.
\[14\] describe both the difficulty of coordinating the users (the
scientists) and the system builders (the engineers). They also discuss
the problems of perception and navigation with a remote robot. Olivares,
Zhou, and Adams \[15\] explore the mechanics of the user interface and
explore different ways to present information to the robot controller.
Wang \[16\] is developing alternative strategies for presenting
graphical information to help a driver control a robot and search a
disaster site by modifying an action-game software engine. Fong, Thorpe,
and Baur \[17\] explore ways to fuse information from different sensors
to support operator control of remote robots.

Each of these works explores interesting questions related to operating
a robot. However, if the problem of controlling a robot is divided into
two parts: enabling the operator to move the robot and avoid obstacles,
and enabling the operator to perform the task for which the robot has
been built, most of this research is pursuing the first goal. Indeed,
there are many challenges associated with providing the appropriate
information to the operator so that he or she may accurately perceive
the hazards and navigation challenges of the remote environment and move
without colliding with obstacles or becoming lost. In this work,
however, we chose to pursue the second objective, trying to understand
how the rover might affect the geologists' interpretation of the remote
environment.

One of the principle challenges in designing a science rover is
balancing the weight, power and communication constraints to keep the
robot safe and provide the scientists with the greatest possible science
return. For the designing engineers, it is a difficult technical matter
to design the robot system, but a rich theoretical framework can provide
great insight into the various design tradeoffs that must be considered
for the mechanical design, camera design, communication system design,
and control system design. The tradeoffs are much less clear when the
designer considers the science systems. In this domain, there are few
theories that relate scientific benefit to power, imaging systems, and
other robot parameters. Quantitative design decisions require
quantitative understanding of the effect of different designs on the
scientific utility of the rover, particularly at the level of detailed,
specific, rover parameters, such as the height of the camera and camera
field of view. To date developers have relied on experience and
analogies to determine the best value for these parameters. However, to
advance the field and optimize the selection of these parameters, models
are needed to formalize the effect of these parameters on scientific
success.

The geological evaluation of Mars involves the use of many different
remote data collection systems. These systems require the geologists to
change their approach to field geology. Instead of directly interacting
with the environment, their interactions with the environment are
constrained by the parameters of the data collection systems. The
question addressed by this research was how the geologic interpretations
change when the rover was inserted between the geologist and the
environment.

This research focuses on sedimentology, the study of sediments. The
fine-scale appearance of rocks and soils can provide indications of how
they were formed. For example, the size and shape of grains can show how
they were transported and deposited. Often geologists in the field use a
hand lens to closely examine these features, which is one reason why the
microscopic imager is part of the MER payload. Sedimentology is
particularly useful for determining whether water, wind, or both have
been present or are present in the environment. The influence of
remotely interpreting these features rather than directly observering
them is therefore closely related to the scientific results
effectiveness of the mission design. Consequently, this study's
objectives were to assess the accuracy of measurements made using
sediment images and determine if differences exist between the
interpretations made using the images and interpretations made directly
from the sediments.

Two experiments tested different sedimentology measurements commonly
made by geologists. The first experiment assessed the consistency of
shape classifications. To perform a shape classification, a geologist
compares a rock (or image of a rock) with a classification table
consisting of photographs or illustrations of rocks with different
shapes. The shape experiment measured the difference between a
geologists' classifications when looking at a physical sample compared
with looking at an image of the same sample. The second experiment
assessed the accuracy of grain distribution measurements taken from
images. Grain distribution is a complex, objective measure, which
determines the percentage of the total number of grains within specific
size ranges.

The camera used in this experiment was a EOS D30 possessing a 2160 X
1440 pixel CMOS imaging chip fitted with either a Sigma 300mm telemacro
lens or a Canon 52mm zoom lens. Experiments conducted with the camera on
a set of prepared visual targets indicated that the blurring caused by
the imaging system in the conditions used in this experiment was on the
order of 2 pixels.

### Grain Shape

A grain's shape can be described using two parameters: the roundness and
the sphericity. Roundness measures how smooth the edges and surface of a
grain are. Sphericity measures how closely a grain resembles a sphere.
Both measurements are made subjectively with the geologists relying on
their background for proper classification. For this experiment, the
sphericity and roundness were assessed using Croft's \[18\] visual
measure. The visual measure consists of a 6x6 matrix with one dimension
representing sphericity and the other roundness. The two axes are
considered orthogonal. For each axis, the categories are given verbal
descriptions. For roundness, the categories are: 1) very-angular, 2)
angular, 3) sub-angular, 4) sub-rounded, 5) rounded, and 6)
well-rounded. For sphericity, the categories are: 1) very-spherical, 2)
spherical, 3) sub-spherical, 4) sub-flat, 5) flat, and 6) very-flat. For
the experiment, the verbal descriptions were mapped to integer values
(1-6 for both axes).

Images were taken using three camera resolutions (550 pix/in, 100
pix/in, and 50 pix/in) and the grains were selected from a range of
sizes (10 -- 300 pixels). An effort was made to select grains from a
wide range of shapes. In all, 195 grains were selected, imaged,
numbered, and stored. Six subjects (all geologists) participated in the
experiment. Before each experiment, the subjects were asked if they were
familiar with this classification scheme and all subjects had previous
used the scheme. Three of the subjects classified all 195 grains and
three of the subjects classified a subset consisting of 120 grains. Each
grain was classified by five of the six subjects.

Because shape classification is a subjective measure, there is no gold
standard to assess the accuracy of the classification. For this
experiment, the geologists classified both the images of the grains and
the actual sediments. For both the sphericity and roundness
classification, the accuracy was determined by subtracting the
classification of the physical specimen from the classification of the
imaged specimen. For sphericity the average difference was --0.20
categories with a standard deviation of 1.27 categories. This indicates
the geologist found the image specimens to be slightly more spherical
than the actual specimens. For the roundness, the average difference was
0.46 categories with a standard deviation of 1.38 categories. This
indicates the image grains were found to be slightly more round than the
physical specimens. A regression analysis was done to determine if
camera resolution and grain length were significant factors for any
differences. Camera resolution was found to be significant for both the
difference in roundness (p \< 0.0001) and the difference in sphericity
(p \< 0.0001). For the roundness regression, the intercept was
significant (p = 0.0248), but for the sphericity regression, the
intercept was not significant (p = 0.4103). Equations 2 and 3 list the
regression models determined from the analysis. The models indicate as
the resolution increases, the sphericity error decreased and roundness
error increased. As the resolution increases, imaged grains are
interpreted as being more round and more spherical.

Roundness error = 0.1305 + 0.0011×camera resolution (2)

Sphericity error = -0.00083×camera resolution (3)

Information theory can be used to estimate the total system error
introduced in robotic geology \[19\]. Information theory measures the
transfer of information through a communication system in the presence
of system noise, in this case both human and machine. The theory
measures both input and output signals in terms of the number of bits it
would take to minimally encode the relevant information. The number of
input bits minus the number of output bits determines the information
lost. The calculation is based on a 6X6 confusion matrix. To generate a
confusion matrix, we selected a pair of subjects and placed the
categorizations made by the first subject along each row and the
categorizations made by the second subject along each column. If the
first subject coded 5 rocks with the value 1, and the second subject
categorized 4 of the same rocks with the value 1, but one of rocks with
the value 2, then the first row of the confusion matrix would be \[4 1 0
0 0 0\]. The confusion matrix allows the calculation of both the input
and output signals, according to the following formulae:

$H(S) = - \sum_{s = 1}^{n}{p(S)\text{log}_{2}(p(S))}$ (4)

$H(R) = - \sum_{s = 1}^{n}{p(R|S)\text{log}(p(R|S)}$ (5)

where H(S) is the information in the signal (in bits), H(R) is the
information in the response (in bits), p(S) is the probability of a
particular signal, and p(R\|S) is the probability of a particular
response given a particular signal.

We constructed a total of 218 confusion matrices for the categorizations
of all combinations of each pair of participants and the following
categorizations: physical roundness versus physical roundness, physical
sphericity versus physical sphericity, imaged roundness versus imaged
roundness, imaged sphericity versus imaged sphericity, physical
roundness versus imaged roundness and physical sphericity versus imaged
sphericity. We then averaged the amount of information in the original
classifications for each comparison and the average amount of
information in the response. For the response classification averages
involving the same classification material, we did not include
contributions of participants with themselves, because these
contributions would represent the perfect transmission of information.
Table 1 summarizes the results.

Table 1: Average information communicated among

[subject ratings of roundness and sphericity of imaged and physical rock
specimens]{.underline}

+-------------------+---+----------------+---+------------+---+----+---+------------+
| Comparison        |   | Average Info   |   | Average    |   | N  |   | Standard   |
|                   |   | in Stimulus    |   | Info       |   |    |   |            |
|                   |   |                |   | Co         |   |    |   | Deviation  |
|                   |   |                |   | mmunicated |   |    |   |            |
+-------------------+---+----------------+---+------------+---+----+---+------------+
| Physical          | 2 |                | 1 |            | 3 |    | 0 |            |
| Roundness         | . |                | . |            | 0 |    | . |            |
|                   | 3 |                | 1 |            |   |    | 4 |            |
+-------------------+---+----------------+---+------------+---+----+---+------------+
| Physical          | 1 |                | 0 |            | 3 |    | 0 |            |
| Sphericity        | . |                | . |            | 0 |    | . |            |
|                   | 9 |                | 8 |            |   |    | 5 |            |
+-------------------+---+----------------+---+------------+---+----+---+------------+
| Imaged Roundness  | 2 |                | 0 |            | 3 |    | 0 |            |
|                   | . |                | . |            | 0 |    | . |            |
|                   | 1 |                | 7 |            |   |    | 4 |            |
+-------------------+---+----------------+---+------------+---+----+---+------------+
| Imaged Sphericity | 2 |                | 0 |            | 3 |    | 0 |            |
|                   | . |                | . |            | 0 |    | . |            |
|                   | 0 |                | 6 |            |   |    | 5 |            |
+-------------------+---+----------------+---+------------+---+----+---+------------+
| Physical v Imaged | 2 |                | 0 |            | 3 |    | 0 |            |
| Roundness         | . |                | . |            | 6 |    | . |            |
|                   | 3 |                | 8 |            |   |    | 5 |            |
+-------------------+---+----------------+---+------------+---+----+---+------------+
| Physical v Imaged | 1 |                | 0 |            | 3 |    | 0 |            |
| Sphericity        | . |                | . |            | 6 |    | . |            |
|                   | 9 |                | 5 |            |   |    | 5 |            |
+-------------------+---+----------------+---+------------+---+----+---+------------+

The value of 2.3 bits of information in the physical roundness of rocks
and 1.9 bits of information in the physical sphericity of rocks
indicates that geologists generally believed that there was a wide
distribution in the rocks provided for study, since if the rocks were
evenly distributed across the 6 classification levels, the information
in the signal would be at a maximum of log~2~(6) = 2.58 bits. However,
the average information communicated suggests the geologists were not
very consistent in their rock categorization. The value of 1.1 and .8
bits of communicated information suggests that if one geologist
classifies a physical specimen of a rock, only approximately 1 bit of
information would be transmitted to another geologists. Since one bit of
information was used, the information could be most efficiently coded
into just two categories. This result suggests that instead of six
levels of categorization for roundness and sphericity, only two may be
required to accurately convey the geologists' interpretations.

The image roundness and image sphericity indicate that although the
information containing the image inputs signals is similar, much less
information is conveyed among the geologists by their categorizations,
particularly for estimates of roundness, which drops from 1.1 bits of
information communicated for physical specimens to .7 bits of
information for imaged specimens. The table indicates a smaller
information drop for judgments of sphericity: from .8 bits (physical) to
.6 bits (imaged). This indicates that the geologists disagreed more
about roundness and sphericity categorizations made from physical
specimens than they did for imaged specimens.

The best measure of the difference between physical and imaged judgments
may be made from the last two rows of the table, which compare average
information transmitted between geologists when comparing physical and
imaged specimens. This measure includes measurements of each subject
when viewing the same physical and imaged specimens, which would
presumably allow more information to be transferred, compared to the
other measures. However, the information transferred is just .8 and .5
bits for the roundness and sphericity judgments, respectively. This
indicates that there is very little agreement in the categorizations of
the rocks made by the geologists when comparing categorizations of the
physical versus imaged specimens. Comparing the between subject
information loss with the between presentation style loss of
information, suggests that between 1.1 and 1.4 bits of information lost
is caused by categorization disagreements among the geologists looking
at either physical or imaged specimens, but when comparing across
modalities, 1.4-1.5 bits of information are lost. This leads us to the
conclusion that although imaging the rocks increases the confusion among
the geologists, the baseline confusion among the geologists working with
physical samples dominates the information loss problem.

### Grain Distribution

###  Natural processes sort sediment grains. The extent of the sorting can be measured by determining the distribution of grain sizes in a sediment sample. In traditional field geology, the distribution is measured by collecting a sample and then sorting it using a set of sieves. A distribution can then be determined based on the percentage of the total sample's mass each sieve contains. Collecting and returning a sample is currently not possible in robotic geology; therefore, sediment distributions are measured from images. The current method for imaged distributions is to compute the percentage of the total image area covered by grains within a specified size range. This experiment determined the difference between the distribution measured using images and the distribution measured using the physical sample.

Eighteen sample images of pre-measured distributions were prepared using
three different camera resolutions (550 pix/in, 100 pix/in, and 50
pix/in) and three different median grain sizes (10 pixels, 30 pixels,
and 60 pixels). This allowed for two images in each size range for each
camera resolution. Each distribution contained grains from six different
size ranges.

Ten subjects (3 geologists and 7 students) participated in the
experiment. Each subject did not analyze all the images, but only a
subset of the images. Each subset contained images from all three camera
resolutions and all three size ranges. The number of images in each
subset varied depending on the availability of the subjects. Sixty-three
separate analyses were collected with each of the 18 images being
analyzed three times and nine of the image being analyzed four times. In
each analysis, the subjects determined the percentage of the total area
covered by grains from each size range. To reduce confusion, the size
ranges were presented to the subjects with the measurements in pixels
instead of inches. The subjects used Photoshop to display and analyze
the images. Many subjects used the grid feature to either divide the
image into 100 uniformly sized regions or to produce a grid where each
square represented the sieve's screen size.

Two different measurements were used to determine the accuracy of the
grain distribution. The first measurement was the absolute value of the
difference between the bin size determined from the image and the bin
size measured with the sieves. The second measurement was the difference
between the bin size measured from the image and the bin size measured
with the sieves divided by the bin size measured with the sieves. The
first statistic assesses the inaccuracy of each bin measurement in terms
of the total image area. The second statistic is the percent error for
each bin measurement.

The average absolute error was 11.11 percent of the total image area
with a standard deviation of 14.91 percent of the total image area. The
errors ranged from 69.85 to 0.040 percent of the total area. A
regression analysis was performed to determine if any factors produced
significantly different absolute errors. Three factors were used in the
analysis: the camera resolution, the relative size ranking of the bin in
each image (bin 1 is the smallest and bin 6 is the largest), and the
relative ranking of the actual bin size in pixels (12 bins with bin 1
being the smallest and bin 12 being the largest). Only the relative
ranking within each distribution and the actual size were significant
factors for the absolute error. Tukey's post-hoc analysis was done to
determine the significant differences within each of these factors.
Figure 4 shows the significant differences for the relative size ranking
in each image. The interesting part of this analysis is that the
smallest and largest bin sizes have significantly larger errors than the
bins in the middle of the distribution. Tukey's post-hoc analysis
indicated there was no pattern to the significant differences for the
actual size rankings.

The average percent error for each bin was 42.5 % with a standard
deviation of 321.9 %. The errors ranged from 4990 % to -- 100 %. To
determine the factors affecting the percent error a regression analysis
was performed using the same three factors used in the regression for
absolute error. The only significant factor was the relative size within
each distribution. Figure 5 shows these differences. Again, the smallest
and largest bins had the largest amount of error.

![](media/image12.png){width="4.445833333333334in"
height="3.4270833333333335in"}

*Figure 4. The box indicates the median and quartile values for the
absolute difference in percent of the total image area for each bin in
the distribution. The X's indicate the mean value for each bin. The bins
were number from smallest to largest for each image. The bars to the
left of the boxes indicate bins where there was no significant
difference between median values.*

![](media/image13.png){width="4.060416666666667in"
height="3.2847222222222223in"}

*Figure 5: The box indicates the median and quartile values for the
percent error for each bin in the distribution. The X's indicate the
mean value for each bin. The bins were number from smallest to largest
for each image. The bars to the left of the boxes indicate bins where
there was no significant difference between median values.*

### Conclusion

This work attempts to bridge the gap between the scientific
interpretations made by geologists and the designers of the information
collection systems. By experimentally assessing the impact of system
parameters on specific scientific judgments, this work helps designers
understand what system parameters affect scientific results, and shows
scientists how their judgments may be affected by the system they are
using.

The size experiment indicates that the camera resolution is directly
related to the accuracy of a size measurement and that the participants
in this experiment were reasonably accurate in determining the size of
the objects they measured, so long as the object was larger than a few
pixels. This unsurprising finding simply indicates that if a scientific
question can be related to the accuracy of a length measurement, the
camera system can be relatively easily defined around this measurement
need.

The shape experiment showed that there was agreement between shape
classification using the images and classification using physical
specimens, but the information that a classification conveyed between
geologists was rather low. Although it is difficult to precisely
determine how the properties of the imaging system affect these
judgments, the results suggest that the imaging system decreases the
agreement of rock classifications. Considering the broader problem of
rock classification, however, the results suggest that the primary issue
is learning how to improve the classification agreement between
geologists for the same physical specimen, and then addressing how the
imaging system affects these agreements. A critical component of such a
study would include the identification of within-subject classification
for the same specimen. If an individual geologist is not consistent in
classification, there is little hope that inter-geologist classification
would be consistent. We are currently conducting a follow-up experiment
to determine the size of this effect. It seems reasonable to conclude
that although artificial intelligence algorithms or special-purpose
instruments could be developed for robotic geology for this task, the
benefit may be limited by the imprecise collective understanding and
interpretation of the shape and roundness of physical specimens.

The distribution experiment indicates that the task simply can not be
performed well with sample images. Clearly, this is an area where
greater research is required, since measuring grain distributions is a
common task in field geology. Based on the length experiment, the
problem does not seem to be one of camera spatial resolution, although
we can not rule out the possibility of differentiating individual rocks
based on color. However, studying the images used suggests that the
rocks are reasonably distinct except when they become very small
relative to the pixel size. The inaccuracy may be related to the
assumption that a visual image can be used to relate the distribution of
grains determined by their mass, a measurement that implies three
dimensional geometry and knowledge of rock density. If that is the case,
the robot must be equipped with some instrument that can directly
measure the features of interest. A second possibility is that the task
is simply too tedious for people to reliably conduct. In this case, an
artificial intelligence algorithm may be more effective than a human
analyst. In either case, the experiment suggests that the current
technique is not appropriate and another technique should be developed.

Taken together these experiments provide new insight into the
relationship between the robot and the geologist. Each experiment
provides a quantitative measurement of scientist performance on several
tasks specific to robotic geology. For length measurements, the primary
limitation is the camera resolution. For shape, the primary limitation
is lack of consensus among the geologists. For distribution, the task
appears to be inappropriate either because of the underlying assumptions
or because the task is too tedious to be reliably conducted by people.
This work provides a new dimension to the challenge of building
effective rovers and, in addition to suggesting some specific
refinements of current techniques and technologies, provides an
experimental path for future insight and discovery.

**References:**

\[1\] C. Stoker and B. Hine, \"Telepresence control of mobile robots:
Kilauea Marsokhod Experiment,\" presented at American Institute of
Aeronautics and Astronautics, Reno, NV, 1996.

\[2\] J. E. Bares and D. S. Wettergreen, \"Lessons from the Development
and Deployment of Dante II,\" presented at 1997 Field and Service
Robotics Conference, Canberra, Australia, 1997.

\[3\] T. Fong, P. Henning, D. Wettergreen, E. Nygren, B. Hine, P.
Hontalas, and C. Fedor, \"Operator Interfaces and Network-Based
Participation for Dante II,\" presented at SAE 25th International
Conference on Environmental Systems, San Diego, CA, 1995.

\[4\] B. Hine, C. Stoker, M. Sims, D. Rasmussen, P. Hontalas, T. W.
Fong, J. Steele, D. Barch, D. Anderson, E. Miles, and E. Nygren, \"The
Application of Telepresence and Virtual Reality of Subsea Exploration,\"
presented at The 2nd Workshop on Mobile Robots for Subsea Environments,
Proc. ROV\'94, Monterey, CA, 1994.

\[5\] D. Christian, D. Wettergreen, M. Bualat, K. Schwher, D. Tucker,
and E. Zbinden, \"Field Experiments with the Ames Marsokhod Rover,\"
presented at 1997 Field and Service Robotics Conference, Canberra,
Australia, 1997.

\[6\] N. A. Cabrol, G. Chong Diaz, J. Dohm, M. Pereira Arredondo, G.
Dunfield, V. Gulick, A. Jensen Iglesia, R. Keaton, C. Herrera Lamelli,
R. Landheim, P. Lee, L. Pedersen, T. Roush, K. Schwher, C. Stoker, and
A. Zent, \"Atacama I: Science Results of the 1997 Nomad Rover Field Test
in the Atacama Desert, Chile,\" presented at Lunar and Planetary Science
Conference, Houston, TX, 1998.

\[7\] N. A. Cabrol, G. Chong Dias, G. Dunfield, J. Dohm, M. Pereira
Arredondo, V. Gulick, A. Jensen Iglesia, R. Keaton, C. Herrera Lamelli,
R. Landheim, P. Lee, L. Pedersen, T. Roush, K. Schwher, C. Stoker, and
A. Zent, \"Atacama II: Nomad Rover Sample 1-250697 and Implications for
Fossil Characterization during Mars Exploration,\" presented at Lunar
and Planetary Science Conference, Houston, TX, 1998.

\[8\] N. A. Cabrol, P. Lee, G. Chong Dias, L. Pedersen, J. Dohm, M.
Pereira Arredondo, G. Dunfield, V. Gulick, A. Jensen Iglesia, R. Keaton,
C. Herrera Lamelli, R. Landheim, T. Roush, K. Schwher, C. Stoker, and A.
Zent, \"Atacama III: Meteorite Search during the Nomad Field Tests:
Perspectives on Automated Field Operations by Teleoperated Vehicles in
Extreme Environments,\" presented at Lunar and Planetary Sceience
Conference, Houston, TX, 1998.

\[8b\] Arvidson, R. E., S. W. Squyres, E. T. Baumgartner, P. S.
Schenker, C. S. Niebur, K. W. Larsen, F. P. Seelos IV, N. O. Snider, and
B. L. Jolliff. FIDO prototype Mars rover field trials, Black Rock
Summit, Nevada, as test of the ability of robotic mobility systems to
conduct field science, J. Geophys. Res.,  107(E11), 8003,
10.1029/2000JE001464, 2002.

\[9\] N. I. Durlach, A. S. C. A. N. R. C. Mavor, R. Committee on Virtual
Reality, and Development, *Virtual reality : scientific and
technological challenges*. Washington, D.C.: National Academy Press,
1995.

\[10\] M. W. McGreevy, \"The Presence of Field Geologists in Mars-Like
Terrain,\" *Presence*, vol. 1, pp. 375-403, 1992.

\[11\] M. W. McGreevy, \"An Ethnographic Object-Oriented Analysis of
Explorer Presence in a Volcanic Terrain Environment,\" NASA Ames
Research Center, Moffett Field, CA NASA Technical Memorandum 108823, May
1994 1994.

\[12\] W. J. Clancy, \"Field Science Ethnography: Methods for Systematic
Observation on an Arctic Expedition,\" *Field Methods*, vol. 13, pp.
223-243, 2001.

\[13\] D. B. Kaber, E. Onal, and M. R. Endsley, \"Design of automation
for telerobots and the effect on performance, operator situation
awareness, and subjective workload,\" *Human Factors and Ergonomics in
Manufacturing*, vol. 10, pp. 409-430, 2000.

\[14\] D. D. Woods, J. Tittle, M. Feil, and A. Roesler, \"Envisioning
Human-Robot Coordination in Future Operations,\" *IEEE Transactions on
Systems, Man, and Cybernetics*, in press.

\[15\] R. Olivares, C. Zhou, B. Bodenheimer, and J. A. Adams,
\"Interface Evaluation for Mobile Robot Teleoperation,\" presented at
ACM Southeast Conference, Savannah, GA, 2003.

\[16\] J. Wang, M. Lewis, and J. Gennari, \"USAR: A game based
simulation for teleoperation,\" presented at 47th Annual Meeting of the
Human Factors and Ergonomics Society, Denver, CO, 2003.

\[17\] T. Fong, C. Thorpe, and C. Baur, \"Advanced interfaces for
vehicle teleoperation: Collaborative control, sensor fusion displays,
and remote driving tools,\" *Autonomous Robots*, vol. 11, pp. 77-85,
2001.

\[18\] R. S. Crofts, \"A Visual Measure of Shingle Particle Form for use
in the Field,\" *Journal of Sedimentary Petrology*, vol. 44, pp.
931-934, 1974.

\[19\] C. E. Shannon, \"A mathematical theory of communication,\" *Bell
Systems Technical Journal*, vol. 27, pp. 379-423,623-656, 1948.

**Acknowledgements**

The material is based upon work supported by NASA under award No:
NAG5-11981.