**ASPEN Evaluation Report**

Jeff Robinson

Commerce One

e-Government Solutions, Inc.

November 8, 2001

# Brief Description

The **G**oal **O**riented **C**ommand Generation (GOC) team has been
tasked with developing the **S**olid State Recorder **PL**ayback
**A**utomation **T**ool (SPLAT) to automate the scheduling of **S**olid
**S**tate **R**ecorder (SSR) data playbacks for the TERRA Spacecraft as
the first phase of the GOC project. The **A**utomated **S**cheduling and
**P**lanning **EN**vironment (ASPEN) was selected as the scheduling
engine for GOC project, and SPLAT tool. The goals of the first phase of
the GOC project (SPLAT Tool) were to gain an understanding of the ASPEN
system and to demonstrate its usefulness in a real-world prototype.

The purpose of this document is to describe briefly the operation of the
ASPEN system, document the lessons learned during development, describe
the deficiencies identified during development of the SPLAT tool, and
provide general thoughts on the ASPEN package.

# What is ASPEN?

The **A**utomated **S**cheduling and **P**lanning **EN**vironment
(ASPEN) is an object-oriented planning and scheduling system written in
C++ and developed by the **A**rtificial **I**ntelligence **G**roup (AIG)
at NASA's **J**et **P**ropulsion **L**aboratory (JPL).

ASPEN provides an expressive modeling language, the **A**spen
**M**odeling **L**anguage (AML), through which the user defines the
spacecraft operation as a series of Activities, States and Resources:
Activities represent an action or step in a plan (An example of an
activity from the SPLAT tool is an ASTER buffer playback). States
represent the state of a subsystem or resource over time, and Resources
describe or profile a physical resource over time. Note that ASPEN
supports three different types of resources, they are: atomic (on/off),
depletable (diminishing capacity), and non-depletable (fixed capacity).

The AML describing spacecraft operation is specified in model (.mdl)
files that contain activity (activities.mdl), state
(state-variables.mdl), and resource (resources.mdl) definitions. These
model files contain activity definitions (operations, reservations),
temporal constraints amongst activities, resource capacities, state
values and state transitions affecting the activities.

Whereas the model files define activity, resource, and state transition
behavior, the instance files define specific instances of activities or
resources to be placed on the individual timelines. In other words the
instance files contain specific instances in time of the activities
defined in the model files. Instances files are created for all resource
and state timelines as well as for activities to be placed on the
timelines.

Once the model and instance files have been created, they are loaded
into ASPEN either on the command line or via the Java GUI client. In
either case, the loaded model and initial schedule files can be
processed into a schedule using the forward dispatch command to place
the activities out on the timeline and the repair command to iteratively
repair any schedule conflicts.

Repair can take on many forms in ASPEN. The schedule can be manually
repaired through operator intervention (by adding, deleting or moving
activities). Or, the schedule may be repaired by through one of the many
repair algorithms available in ASPEN. These repair algorithms attempt to
fix schedule conflicts by iteratively selecting random conflicts and
attempting to repair the problems using a predetermined set of heuristic
functions. Note that advanced users can create their own heuristic
functions but this requires an intimate knowledge of the ASPEN software
architecture.

Schedules generated by ASPEN can be saved, reloaded, modified, and
stored to disk in several different formats. The activity, resource,
state and timeline instances can be saved to a flat ASCII file using the
general "save schedule" option. Saving the schedule in this way causes
the ASPEN server to store the currently loaded schedule in AML to a flat
ASCII file (schedule.ini). ASPEN also allows the user to save schedules
or portions of schedules in a user specified format to flat ASCII files
via the "Save commands" command. This method allows users to customize
the schedule output, printing out only what they need and in a format
they specify. The desired activities, resources, etc. are formatted
according to the layout specified in the command format file
(commands.fmt). Each entry in the command format file contains an
activity name followed by information describing how to format the
activity. Note that the command format file is provided to ASPEN at
server startup or during a project load.

For a more detailed description of ASPEN, please refer to the ASPEN
User's Guide Version 2.0 or one of the many papers describing ASPEN on
the ASPEN web site at

<http://www-aig.jpl.nasa.gov/public/planning/aspen/aspen_index.html>.

# Continuous Planning with ASPEN

CASPER (**C**ontinuous **A**ctivity **S**cheduling **P**lanning
**E**xecution and **R**eplanning) provides a real-time continuous
planning, scheduling, and re-planning extension to ASPEN. CASPER uses
iterative repair to support continuous modification and updating of a
current working plan in light of changing operating context.

We, however, have not had an opportunity to exercise the CASPER features
other than to execute the rover demos supplied with ASPEN which use
CASPER to plan and re-plan rover missions.

The main obstacle to using the CASPER extension is its lack of
documentation. In order to develop a model for CASPER, one would need to
examine the rover models and adapt them for the specific need. CASPER is
one of the many ASPEN features/extensions that we feel would prove
invaluable provided sufficient documentation describing its usage was
available.

For more information on CASPER, please refer to the CASPER Web page.

<http://www-aig.jpl.nasa.gov/public/planning/casper/>

# Build Problems

ASPEN is delivered in a **t**ape **ar**chive (tar) file containing the
source code and binaries for Solaris. For our use though, the out of the
box ASPEN was not sufficient. We needed to integrate ASPEN into SPLAT
and thus needed the ability to build the ASPEN binaries locally. Relying
on JPL to make source code modifications and waiting for a new delivery
of ASPEN each time modifications were needed was deemed unacceptable.

ASPEN can be built with either the Solaris C++ or the GNU C++ compiler.
We chose to build ASPEN with the GNU compiler because all of the tools
required to build the distribution with the GNU C++ compiler were
freeware. Had we opted to use the Solaris Compiler, a third party
commercial Standard Template Library (STL) from ObjectSpace would have
been required at an additional fee. Refer to the ObjectSpace web site
for more information (<http://www.objectspace.com/products/toolkits/>).

Building ASPEN with the GNU C++ compiler yielded a wealth of Standard
Template Library (STL) errors. The issue was a difference between JPL's
version of the GNU compiler and our version of the GNU compiler. JPL
builds ASPEN with GNU C++ version 2.8.1.3. Our development system
contains GNU C++ version 2.95.3. The template errors reported on our
system were reported as warnings to JPL. These errors/warnings are
related to how the ASPEN code was being "plugged" into the template
library code. In short, the templates were expecting "char \*" arguments
but were being passed "const char\*" arguments. Fixing this problem
required modifications to many of the ASPEN source modules and several
widely used header files.

It is important to note that we worked closely with JPL developers on
this issue and the source code modifications necessary to fix the above
errors/warnings have been integrated in the JPL ASPEN code base. What
does this mean? Well it means that future users who wish to build a GNU
compiled version of ASPEN in their local environment shouldn't encounter
the above mentioned template errors.

However it is important to note that our project was one of the few to
build ASPEN outside of the JPL environment. Be aware that building ASPEN
on another system or using a different version of the GNU compiler may
raise other problems not found during our build.

# Model Creation

Model creation for ASPEN is advertised as being simple, straightforward,
and one of the advantages to using ASPEN. It is true that simple models,
those that use only the basic features of ASPEN, are relatively easy to
construct. We were able to develop a simple model for a single TERRA
instrument (MODIS) and buffer within a relatively short period of time
(1 week). However when we added the more complex features of our model,
the basic features were no longer sufficient. We needed to use some of
the more advanced features in ASPEN and the AML to develop our model and
this proved difficult due to a lack of documentation and would not have
been possible without the direct assistance of the JPL developers.

An example of the problems we encountered was adding support to our
model for scheduling the start time of playbacks based on start times of
the contact windows. The start of playback always occurs at some offset
from the start of a k-band contact window. The exact offset is
determined by the start times of the k-band contact and s-band contact.
If the two contact start periods are the same, the start of the playback
is offset 120 seconds from the start of the k-band contact. If the start
times are different the start of the playback is offset 30 seconds from
the start of the k-band contact window. In order to support this
variable offset in our model, we had to use two ASPEN constructs that
are not well documented. The first is external dependencies. This
mechanism allows an activity to grab information from timelines or
activities other than itself. In our playback activity, we needed to
test the k-band and s-band activity timelines to determine the state of
the contact (in or out). In order to grab the values from the external
timelines we used another lightly documented feature of ASPEN parameter
functions. The parameter function read the k and s band timelines and
returned the state settings at a specified time. Developing the
parameter function and understanding how external model dependencies
worked would have proven difficult or impossible to implement without
the assistance of the JPL developers.

JPL does provide a series of example models with the ASPEN distribution
as examples. However, these models are lightly documented at best and we
were unable to compile and test most of the models. The bottom line is
that simple models such as the EO1 example provided with the ASPEN
distribution are relatively easy to create. They have a few activities
and don't really exercise the more advanced and powerful features so
they can be easily generated with only a limited knowledge of AML and
the ASPEN system. However the more complex the model, the more the
advanced features are needed, and the bigger the need for a working
knowledge of the ASPEN system internal.

We believe that additional documentation describing the AML in greater
detail including the features such as parameter functions, external
dependencies, user defined timelines, etc. would go a long way to
clearing up the problems we encountered during development of the Terra
Model.

# Resource Precision Issues

One of the biggest problems and most troubling issues encountered with
ASPEN was the relatively limited range of values assignable to
depletable resources. ASPEN limits integer resources to the following
range: --2^30^ to 2^30^ -- 1(-1073741824 to 1073741823). In order to
properly model the SSR buffers, our model needed to support filling and
depleting very large buffers down to the bit level. The four SSR buffers
contained in our model have capacities ranging from 1.5 to 90 Gbits
(Billion) and modeling them to the bit level exceeds the valid range of
values for integer resources.

# Resource Reservation Issues

The ASPEN method of handling resource reservations on timelines is brute
force and was not acceptable for our model. ASPEN handles resource
reservations by applying the full reservation value to the timeline at
the beginning of an activity. This is fine for activities where the
actual reservation is not based on activity duration and occurs at the
beginning of the activity. However for activities such buffer filling
and playback, the resource reservation doesn't occur all at once rather
the reservation increases or decreases linearly over time from the
beginning to the end of the activity.

To solve this problem, activities in the TERRA SSR model that require
linear modeling of resources have been decomposed into a series of
shorter duration activities. Instead of having the reservation slammed
at the beginning of a single long activity, many reservations are
applied, one for each of the shorter duration activities, causing a
stair step affect and giving the appearance of a somewhat linear
application of reservations for buffer depletion and playback
activities. While not perfect, this method provided a quick solution to
the problem and works well enough for the TERRA SSR model.

It is important to note that should the stair step method not be
acceptable, one would need to create a new timeline type that linearly
applies the reservation over the activity duration.

# Floating Point Resource Issues

In our search for a solution to the resource precision issue it was
thought that we could model the buffers to the bit level by using
floating-point numbers and maintaining a portion of the precision beyond
the decimal point. However, we quickly found out that floating-point
resources are not directly available in ASPEN. The default timeline
implementations in ASPEN support only integer or state values
(enumerated types), no floating-point values. In order to use
floating-point values on a resource timeline, one must use the
Generalized Time Line (GTL) mechanism in ASPEN. The GTL mechanism allows
one to create a new type of timeline that can represent different types
of data (i.e. floating point values).

Our attempts at using the GTL mechanism to create floating point
timelines were not fruitful. We were able to use the GTL mechanism to
create the floating-point resources easily enough (with the assistance
of JPL). However tests of the model with the floating-point (GTL)
timelines were extremely slow during dispatch and repair and were deemed
unacceptable.

Note that according to the JPL folks, this is a known problem and is
being addressed. However no timetable was given as to when this will be
completed. We were also warned that the GTL mechanism is relatively new
to ASPEN and has only been lightly tested.

# Socket Interface Issues

Integrating ASPEN into the SPLAT Tool required the creation of a socket
client within the SPLAT source code. Through this client socket
connection, the SPLAT tool commands ASPEN server operation. During the
development of this socket client interface, we encountered a series of
problems with the ASPEN socket interface. The most glaring of which is
the fragility of the socket interface. Passing incorrect arguments
(either number or type) or not listening (reading) for return results
can lead to disaster (server crashes and core dumps).

Another problem encountered during development of the SPLAT socket
client for ASPEN was the shortage and quality of documentation
describing the socket interface and the command set. The only
documentation pertaining to the socket interface other than that
available in the code is a text file (socketServerAPI.txt) provided with
the ASPEN distribution. However we found this document to be misleading
and incomplete. If you should need to command ASPEN through the socket
interface use the socketServerAPI.txt descriptions as a reference only.
Refer to the source code to determine the command numbers and arguments.

# ASPEN Performance

The performance of the ASPEN system (CPU and memory use) is largely a
function of the number of activities within a schedule and it depends on
the detail of the plan and the time covered by the plan. What we have
found is that for the SPLAT Tool, loading and dispatching the schedule
onto the timelines takes \~30 seconds while individual schedule repair
iterations range from 20 -- 40 seconds in length. The SSR Model used for
the SPLAT Tool typically has 900 -- 1000 activities when scheduled and
covers an entire days operation for the spacecraft. One should expect
the time needed to schedule and dispatch to increase with the number of
activities and the length of the schedule. However one can always
improve response time by looking at a more abstract plan (limit the
detailing of activities), or at a smaller window of time.

According to JPL, the current bottleneck in ASPEN is memory. They
noticed that page swapping significantly increased scheduling time. For
one of their implementations, schedules with up to 9000 activities were
not uncommon. They noticed that the ASPEN process began swapping for
schedules larger than 6000 activities. Timing tests of the larger
schedules indicated that the 6000 activity plan took about 12 minutes to
schedule and the 9000 activity plan about 40 minutes. So beware very
large schedules may cause ASPEN to swap leading to degraded performance.

Note: SPLAT timing tests were generated with an ASPEN server executing
on a Sun Ultra10 Workstation (440 MHz UltraSPARC-II Processor) with 256
MegaBytes (MB) of Random Access Memory (RAM) and running the Solaris 8
OS. JPL timing numbers are for an ASPEN server running on a Sun Ultra 60
(450 MHz UltraSPARC-II Processor) with 1 GigaByte (GB) of RAM.

# Software Maturity Issues

Another disturbing feature of the ASPEN system is the lack of software
maturity. During testing and development, we experienced a large number
of core dumps and our fair share of ungraceful server exits.

These crashes appear random in nature and often times are not
reproducible. Some of these crashes could be due to the fact that our
model is exercising little used and new functionality in ASPEN. However
one would expect that a mature software product would be completely
tested and debugged prior to release.

Our assessment of the ASPEN system is that it is a mature prototype.

# Final Thoughts

The ASPEN system appears to be a very powerful and highly flexible
system. However we feel that this system lacks maturity and robustness.
The maturity issue continually raised its head during integration and
testing of the SPLAT Tool. We were constantly encountering random system
crashes and glitches that caused the scheduling engine to crash and exit
ungracefully.

It is important to note the use of ASPEN for SPLAT has only scratched
the surface with respect to what can be accomplished with ASPEN. ASPEN
allows the flexibility of user defined scheduling functions, user
defined (custom) timelines, schedule scoring, custom parameter
functions, custom heuristic functions, etc. However, utilizing the
advanced features in ASPEN requires an intimate knowledge of the
software structure and a clear understanding of the advanced features.
Unfortunately documentation both in-line and hardcopy for ASPEN is
sparse at best. The Users Guide lacks descriptions of how to use the
advanced features and is sometimes misleading or incorrect. No
programmer documentation exists for the system and the source level code
is terrible.

In order for ASPEN to be a more widely used and accepted system, User
level documentation must be completely revamped with descriptions of all
system features. Additionally, the source code documentation must be
beefed up, and a Programmers Guide describing how to implement custom
versions of the advanced features within ASPEN must be developed.