Intelligent Agent for Science Data Processing

  -------------------------------------- --------------------------------
  Wanlin Pang\                           Keith Golden\
  *UARC/NASA Ames\                       *Google, Inc.*
  Mail-Stop 269\                         
  Moffett Field, CA 94035\               
  <wpang@email.arc.nasa.gov>\            
  Phone: (650) 604-2584\                 
  Fax: (650) 604-4036*                   

  -------------------------------------- --------------------------------

Abstract

*Managing and processing data is a significant challenge in all NASA
enterprises. Satellites, unmanned space craft, planetary rovers and
observatories, for all their complexity, can all be viewed as remote
sensors; their sole purpose is to gather data, which are then processed,
delivered to the scientific community and the public, and archived.
Given the large number of socially and economically important
information that can be derived from the data, the complexity of the
data processing needed to derive them and the many terabytes of data
that must be processed each day, there are great challenges and
opportunities in processing the data in a timely manner, and a need for
more effective automation. Our approach to providing this automation is
to cast it as an AI planning problem: we represent data-processing
operations as planner actions and desired data products as planner
goals, and use a planner to generate data-flow programs that produce the
requested data products.*

*The data production problem consists of converting an initial set of
low-level data products into higher-level data products that can be used
for science or decision support. For example, in Earth-science data
processing, the data products of interest are geospatial data measuring
specific variables of the Earth system, such as precipitation,
vegetation productivity and fire risk. Higher-level data products may be
altered versions lower-level data products, or they may be entirely new
products that estimate unknown Earth system variables, such as soil
moisture, based on known variables, such as precipitation. These
variables are estimated by running one or more computational models,
such as simulation codes. The models can be precisely characterized in
terms of their input and output requirements, which makes them
straightforward to represent in an AI planning system. However, there
are significant differences between the data production problem and more
traditional AI planning domains, calling for different techniques.*

*Notable features of data processing domains include large dynamic
universes, incomplete information and uncertainty. There are petabytes
of data available, with new data becoming available all the time, and
new data products are generated in the course of fulfilling the user's
goal --- data products that could be used to fulfill subsequent goals.
There is also considerable uncertainty --- uncertainty of the time that
particular data will be available, or whether the data will arrive at
all, uncertainty in the quality of data, even uncertainty as to whether
a given processing algorithm will succeed.*

*We have developed a planner-based agent, called IMAGEbot, to automate
data production. The data production problem may be viewed as a planning
problem in which the initial state describes the current set of
available data products, and whose goal state describes the properties
of the desired high level data products. Planner operators correspond to
data transformation and generation tools. IMAGEbot takes data product
requests as high-level goals and executes the commands needed to produce
the requested data products.*

*We adopt a planning approach somewhat similar to Graphplan, consisting
of a Graphplan-style reachability analysis and a constraint-based
search. However, the large universe of the data production problem makes
the grounded planning graph of Graphplan inapplicable; instead, we
choose a lifted representation where actions and plans contain
variables. Because of the lifted representation, and the uncertain and
dynamic nature of the data production problem, the reachability analysis
and search cannot be separated; instead, IMAGEbot interleaves planning,
constraint reasoning and execution.*

*In contrast to the traditional approach of automating data processing
by means of scripts, our agent-based approach provides the following
advantages: i) Plug-and-play design. New data sources, data processing
operations or other capabilities can be added or removed, while the
system is running, without the need to modify existing scripts. ii)
Fault tolerance. Failures or unexpected events can be detected and
recovered from to automatically. Anticipating all possible failures in a
script or program is notoriously difficult. iii) Flexibility. The agent
is not limited to a fixed set of goals. It is capable of using available
resources to achieve requests that may not have been anticipated at
design time.*

*IMAGEbot has been demonstrated in the Terrestrial Observation and
Prediction System (TOPS), an ecological forecasting system that
assimilates data from Earth orbiting satellites and ground weather
stations to model and forecast conditions on the surface, such as soil
moisture, vegetation growth and plant stress. The planner identifies the
appropriate input files and sequences of operations needed to satisfy a
data request, executes those operations on a remote TOPS server, and
displays the results, quickly and reliably.*

*Whereas TOPS is concerned with geospatial data measuring specific
variables of the Earth system, such as precipitation, vegetation
productivity and fire risk, the agent based approach is applicable to
other types of data as well. Our current effort focuses on extending
IMAGEbot functionalities and applying it to other areas, such as space
science data processing, automated data processing for mission
operations, automated on-orbit data processing, just to name a few.*

.