DSS Assimilation Guidebook Summarized

Verne Kaupp, University of Missouri

Charles Hutchinson, University of Arizona

March 2004

# [Introduction]{.smallcaps}

The purpose of this document is to summarize the general procedure as
developed and reported in the guidebook[^1] for DSS assimilation for
infusing---assimilating---NASA Earth Science data, mission products,
technology and/or science results into a priority National Application
decision support system (DSS) owned by a partnering Federal agency.

The goals for assimilating NASA results into a DSS owned by a partnering
Federal agency include:

-   Exploiting their significant value for the decision-making process
    at work for sustainable national applications,

-   Increasing the return to society for its investment in the NASA
    Earth Science Enterprise,

-   Improving the information available on which management and policy
    decisions are based, and

-   Enhancing the accuracy, speed, and scale of applicability of the
    outputs from the Nation's DSSs.

The general procedure and basic considerations for assimilating NASA
results into a partnering agency's DSSs are laid out in this booklet.
The general procedure starts with establishing a need and ends with
transferring a DSS enhanced with NASA results to its operational support
of decision making.

# Scope

This booklet is designed to guide the senior program manager through the
steps to be followed in working with a Federal partnering agency to
enhance it's DSS to work with NASA results.

It summarizes the procedure in a straightforward way without discussing
the underlying theoretical and practical considerations. Detail
information is therefore lacking as well. All such additional
information is contained in the guidebook (Footnote 1).

# Background

NASA recognizes the importance of partnering with other Federal agencies
to make systematic connections between EOS geospatial data, science
models, and their applications in the relevant decision-making systems
(DMSs) that serve our Nation's policy and decision makers. These
partnerships focus on innovative approaches for using Earth science
information and knowledge to provide or enhance decision-support
capability. This enhancement of capability supports myriad operational
applications, as well as policy discussions, and is used to address a
variety of national priorities that range from agricultural
competitiveness to homeland security and their resultant socioeconomic
impacts.

NASA is committed to understanding the Earth system and improving
predictions or forecasts of climate, weather, and natural hazards.
Employing 18 Earth observing satellites routinely making measurements
with over 80 onboard remote sensing instruments, NASA has defined two
strategic objectives: (1) Science---to observe, analyze, and model the
Earth system to discover how it is changing and the consequences for
life on Earth; and (2) Applications---to expand and accelerate the
realization of economic and societal benefits from Earth science
information and technology; with Earth System Science and Earth Science
Applications as corresponding themes.

Through the Earth Science Applications Program, NASA is working with
agencies and organizations that have the requisite infrastructure to
apply NASA's Earth science results in order to help manage coastal
environments, water resources, and aviation safety; monitor air quality,
forest fires, and the impacts of infectious diseases and invasive
species and conduct hurricane forecasting, disaster preparedness and
response efforts.

# [Decision-Making Considerations]{.smallcaps}

Decisions are the "finished products" of a *decision-making system
(DMS).* In a DMS, the user is the decision maker and the information
processing system is the *decision support system (DSS)*. Taken
together, the DSS and its user constitute a DMS.

Figure 1 illustrates a generalized model for a DMS containing five
fundamental parts: (1) the *User* who formulates the problem and
produces a decision from the DSS output and other pertinent information,
(2) the *Input Interface* which is designed to facilitate HCI
(human-computer interaction), (3) the *Problem Processing System (the
PPS)* that takes an input problem statement and information from the
*Knowledge System (KS)* to produce information that supports (enhances,
or makes possible) a decision process, (4) the *KS* that contains the
body of knowledge available about a problem domain (the *KS* has the
databases, background -- historical information, problem domain
knowledge, algorithms, etc.), and (5) the *Output Interface* that allows
the user to retrieve the processed information in a format that is
understandable/visualizable.

Each DSS is unique but, from a conceptual viewpoint, the four blocks in
the figure containing (1) Input interface, (2) PPS, (3) KS, and (4)
Output interface often represent the minimum configuration for a DSS.

![](media/image1.wmf){width="3.375in" height="1.2041666666666666in"}

![](media/image2.wmf){width="3.375in" height="1.7548611111111112in"}

Figure 2 illustrates the Knowledge System (KS). The KS is where the
complete domain of knowledge about a particular problem, or class of
problems resides. It is in the KS where NASA will focus the majority of
DSS assimilation and enhancement work (as shown in the figure) whereas
the partnering agency, in upgrading its DSS to the enhanced operational
State 2 will concentrate its efforts on the interfaces and PPS.

Note that this means that in order to provide a DSS enhancement, NASA's
assimilation result must function within the partnering agency's DMS.
This burdens the NASA effort with understanding the high level needs and
requirements -- the operating environment -- and the low level status
for both the DSS and the DMS. The type of low level status information
needed include but is not limited to such as (1) data types and sources,
(2) data formats, (3) spatial, spectral, and temporal resolution, (4)
spatial and temporal coverage, (5) data cost, (6) availability and
timeliness of data, (7) volume of data to process, (8) geospatial,
numerical, and thematic accuracy, (9) etc. The high level considerations
include knowing (1) the framework, (2) the knowledge type, (3) the
management paradigm, and (4) the decision-making typology. These high
level elements are briefly discussed in the following.

***Framework**---*Use of a "framework" or matrix describing management
decision-making activities is common in the literature. Table 1 proposes
such a decision support framework matrix for use in this DSS work. The
degree of problem complexity is listed across the top and the degree of
structure -- with the continuum of structure being quantized into the
three choices listed -- Is along the left. Cell values are particular
decision types. Note that it is the decision *context* that is
unstructured, not the DSS itself.

The degree of problem complexity ranges from modeled (already done) to
novel (may not be amenable to modeling or may not yet be modeled). The
more structured a problem, the more amenable it is to solution by
computerized algorithms in a DSS. The simplest case would be the
Structured/Modeled Problem noted as a *routine* decision type.

At the other extreme, the more unstructured a problem, the more a
decision must rely on human judgment and intuition. Thus, the
Unstructured/Novel Problem is noted as requiring a *custom tailored*
solution. In the middle, a semi-structured problem has both structured
and unstructured phases and elements. Clearly, as the decision making
system is developed, the intent is to make it less idiosyncratic and
more consistent -- to move decisions from the bottom right toward the
upper left of the framework.

Table 1. A decision support framework

  ----------------- ------------------ ------------------ ------------------
                    Modeled Problems   Complex Problems   Novel Problems

  Structured        1\. Routine        2                  3

  Semi-structured   4                  5                  6

  Unstructured      7                  8                  9\. Custom
                                                          tailored
  ----------------- ------------------ ------------------ ------------------

***Types of Knowledge**---*Different types of knowledge are recognized
in the literature: explicit vs. tacit, declarative vs. procedural,
esoteric vs. exoteric, and deep vs. shallow. Each represents a different
management challenge. Briefly, the types of knowledge are:

-   *[Tacit knowledge]{.underline}*---is contained within a person's
    head, and is difficult to express,

-   *[Explicit knowledge]{.underline}*---can be readily articulated,
    written down, and shared,

-   *[Declarative knowledge]{.underline}*---consists of facts about the
    state of the world,

-   *[Procedural knowledge]{.underline}*---involves "how to" do
    something,

-   *[Esoteric knowledge]{.underline}*---is highly specialized, and
    applicable to narrow domains,

-   *[Exoteric knowledge]{.underline}---*is applicable to broad domains,
    and might be considered to be "common sense",

-   *[Deep knowledge]{.underline}*---consists of formal theories of
    behavior of phenomena,

-   *[Shallow knowledge]{.underline}---*is found in social domains and
    is less well organized and codified than scientific domains.

***Knowledge Management Paradigms**---*The following three different
perspectives on knowledge management embrace the spectrum of choice:

-   *[Functional paradigm]{.underline}---*is that most often adopted by
    those in practice, especially in the software industry,

-   *[Interpretive paradigm]{.underline}*---consists of fostering
    communications between individuals, sharing and enriching
    interpretations, and coordinating actions,

-   *[Critical paradigm]{.underline}*---is concerned primarily with
    social conflict and antagonistic relationship amongst various
    stakeholders and special interest groups.

***Typology**---*Over the past 50 years, decision-making and
decision-making environments have been extensively studied. In the
classical literature, there are *five types of decision making systems*
or inquiring organizations: *Leibnizian (Type 1), Lockean (Type 2),
Hegelian (Type 3), Kantian (Type 4),* and *Singerian (Type 5)*, each
based on the philosophies of their respective namesakes. We find that
decision making systems can be classified analogously and, thus, we will
adopt these definitions for characterizing DSSs but will use the more
generic labels, *Type 1 -- Type 5*. A brief description of each follows:

-   *[Type 1---Analytic -- Deductive Approach]{.underline}---*creates
    knowledge by using *formal logic* and *mathematical analysis* to
    make inferences about cause-and-effect relationships, and maintains
    that everything needed to solve a problem is already contained
    within its boundaries. As closed systems, they have access only to
    knowledge generated internally.

-   *[Type 2---Consensual -- Inductive Approach]{.underline}---*creates
    knowledge from empirical information gathered from external
    observations and used inductively to build a representation of the
    world with a set of labels (or properties) assigned to the
    observations. The decision style is clearly group-oriented and open.

-   *[Type 3---Empirical -- Theoretical Approach]{.underline}---*
    recognizes that there may be *many different perspectives on a
    problem*, or at least many different ways of modeling it. The
    perspectives are all, however, *based on the belief that problems
    can be modeled analytically*. It finally *chooses the model which
    best explains the data*.

-   *[Type 4---Conflict -- Synthesis Approach]{.underline}---*is based
    on the belief that the most effective way to create knowledge is by
    observing a debate between two diametrically opposed viewpoints
    about a topic, a *thesis* and an *antithesis,* from which a
    *synthesis* is constructed as the worldview.

-   *[Type 5---Multiple Perspectives -- Holistic
    Approach]{.underline}---*views the world as a holistic system, in
    which everything is connected to everything else. Solving complex
    problems may require knowledge from any source and knowledgeable
    people in any discipline or profession. The multiple perspectives
    approach does not end with the technical, organizational, and
    personal perspectives. It also explicitly brings *ethics and
    aesthetics* into play.

# [Earth Science Enterprise Components]{.smallcaps}

Figure 3 illustrates the Earth Science Enterprise system components and
the general scope of the assimilation process available from a
NASA-centric position. NASA ***missions*** carry Earth observing
instruments into space. The instruments produce experimental (proof of
concept) or systematic (extended) ***observations*** of the Earth that
are molded into ***geophysical parameters*** to answer specific science
questions about Earth system processes (e.g., the forces that drive
climate). By understanding processes, it is possible to develop
***models*** that can be used to ***predict*** future conditions (e.g.,
El Niño seasonal climate forecasts). The outputs of models can be used
to make ***decisions*** (e.g., what price to pay for corn futures),
increasingly through the use of formal ***decision support systems***
(DSS). Clearly, there will be instances where NASA observations may be
directly incorporated in an existing DSS (e.g., SRTM topography to
assist in flood planning). However, it is likely that science outputs --
models that allow future scenarios to be evaluated -- will assume
increasing importance as we seek to increase economic efficiency and to
protect lives and property.

# [Approach]{.smallcaps}

The general procedure for assimilation and infusion of NASA Earth
Science Enterprise (ESE) science results, data, and information into a
decision support system (DSS) owned by another Federal agency or
national organization is illustrated in Figure 4.

In the most straightforward approach, the general procedure breaks down
into a process with four essential steps. This booklet uses those four
steps to present the general procedure.

**To be provided**

**Figure 3: Integrated Earth Observation System Components (Coin
Chart)**

The generalized approach flows from Step 1 to Step 4. However, it is not
a requirement that the process be strictly serial, it is expected that
some degree of parallel activity will result in cost savings and reduced
time to complete the enhancement of a DSS.

## The First Step

The first step is to conduct the set of tasks shown as the preliminary
work on the left side of Figure 2 and thereby develop the basis for an
agreement with another Federal agency to upgrade its DSS. ***The first
task of Step 1*** is to identify a DSS that might benefit from
assimilation of NASA science results, data and/or technology and that is
currently functioning in its operational state (known as State 1) to
serve the decision needs of a potential partnering Federal agency.

***The second task of Step 1*** is to form an Assimilation team to
assist the program manager in assessing the political environment and
policy setting of the Decision-Making System (DMS) within which the DSS
functions and to coordinate and conduct all subsequent tasks. The
Assimilation team should be drawn from the policy and technical staff of
the partnering agency, appropriate NASA Center(s), appropriate
universities, and the NASA Earth Science Enterprise. Broad participation
on the Assimilation team is necessary to ensure credible results
achieved in a process acceptable to all parties.

The Assimilation team will work with the Earth Science Applications
manager to (1) document its role throughout the assimilation process,
(2) develop the documentation and reporting protocols and requirements,
(3) determine the metrics and performance objectives that will underpin
the zeroth-order feasibility assessment and risk/benefit analysis, (4)
establish the basis for determining the acceptability of NASA
technology, data and information products, (5) determine the criteria
for deciding when the assimilation is completed, and (6) conduct all
subsequent tasks.

***The third task of Step 1*** is to conduct a "zeroth-order" analysis,
or feasibility assessment, of the DSS and the decision-making system
(DMS) it serves. This should be a relatively rapid appraisal of the DSS
and its DMS. This feasibility assessment forms the basis of a potential
agreement between NASA and the partnering agency. It lays the groundwork
for assimilation and establishes the basic underlying framework of the
work to be undertaken. It assesses the practicality of infusing NASA
results, data and information into the DSS. It establishes the potential
for improving the DSS output, and the probable value of this
enhancement. It catalogues the DMS needs and reviews the owner agency's
plans (if any) for upgrading the DSS. The feasibility assessment
determines the current DSS status from the joint perspective of the
agency and NASA, documents its position within the National
Application's DMS, catalogues the DMS's needs, and identifies the
management or policy level of the DMS.

***The fourth task of Step 1*** is to determine the potential value,
benefits, and risks to both parties that might arise from engaging in an
infusion process to upgrade a specific DSS. This is done in a
preliminary risk assessment that is a critical and structured process
that is intended to identify both potential benefits and undesirable
outcomes that might accompany assimilation. It starts with the results
from the zeroth-order feasibility assessment task. It requires
identifying the possible NASA contributions. It involves estimating the
enhancement potential of the DSS. It estimates the risks, both direct
and unintended, to both parties from an infusion of NASA results, data
and information.

The results of this analysis form the basis by which an agreement is
developed between NASA and the host agency establishing their shared
mutual commitment and laying out a clear set of objectives for the
planned DSS upgrade.

***The fifth task of Step 1*** is to negotiate an agreement with the DSS
owner. The purpose of the agreement is to formalize the proposed joint
activity between NASA and the partner agency. The agreement should
describe the general approach to be followed and the intentions and
expectations of each of the partnering agencies. The zeroth-order
feasibility assessment, the risk analysis, and the benefit opportunities
provide crucial information on which to base the agreement. The
agreement should have at least the following six elements:

-   *An Agreement in principle*---first, and most obviously, the
    agreement should state the willingness of the two parties to
    cooperate in the upgrade and enhancement of the candidate DSS,

-   *A statement of objectives*---a clear description of the overall
    objectives of the assimilation activity should be stated,

-   *A set of metrics*---a set of specific criteria that have been
    derived from the objectives to establish how the assimilation
    activity will be assessed and to define how the infusion process
    will be terminated should be stated,

-   *An approach to follow*---a concise description of the activities to
    be undertaken and an explicit requirement that NASA science results,
    data, or technology will be used should be included,

-   *Identification of key personnel*---a list and description of the
    roles of key, or essential personnel expected to be involved in each
    of the activities should be included,

-   *A statement of commitments*---a brief description of the levels of
    resources that might be committed by each partner to the agreement
    should be stated.

## The Second Step

The second step is to initiate the assimilation process by formally
characterizing the current operational state (State 1) of a candidate
DSS and benchmarking its performance. On Figure 2, this is represented
in part by the second vertical column from the left consisting of four
blocks. ***The first task of Step 2*** is to start the assimilation and
benchmark processes. The terms of the agreement reached in the First
Step usually are broad, setting forth top level conditions, only.
Details are worked out as the assimilation activity proceeds. From the
terms of the agreement, goals, objectives, scope, and resources all need
to be determined. In addition, the issues of how to deal with
disengagement of NASA at the end of the assimilation period, how to deal
with the transferability of NASA assimilation/input, what the long term
relationship between agencies is to be, and who will be responsible for
long term operational sustainability need to be resolved.

![](media/image3.png){width="5.990944881889764in"
height="4.509055118110236in"}

Figure 5 illustrates the benchmarking process. At its most basic,
benchmarking is a management tool. It has four essential
phases---planning, development, synthesis, and enhancement---that follow
one another in succession with essential documentation and reporting at
each phase. As it applies here, benchmarking will be used to assess the
performance improvement in a DSS after upgrade to function at State 2
compared to a baseline set of indicators representing its State 1/0
characteristics. Once initiated, the benchmarking process encapsulates
all work from that point forward until a DSS has been upgraded to
function at State 2 and is considered operational.

Benchmarking involves teams -- teams of experts, facilitators, and
organizations. These are required to provide information, guidance, and
resources: to provide information about the state-of-the-art at which a
DSS is to function; and to provide the underlying science and data
output expectations of the decision-making environment it serves.

Observe from the figure that for DSS benchmarking, the basic phases of
the process are:

-   *[Identify and select]{.underline}*---the benchmark partners (that
    is, the participating agencies) and the process or DSS to be
    benchmarked. This phase is concerned with identifying and selecting
    the benchmarking partners, selecting the process to be
    followed---specifically, the type(s) of benchmarking to be
    employed---for assimilation, and initiating the oversight and
    documentation requirements. Phase 1 *planning* thus leads to the
    second phase.

-   *[Characterize, assess and observe]{.underline}*---the DSS and the
    associated benchmarking partners and create
    benchmarking/assimilation teams. This phase is concerned with
    formally and completely characterizing the baseline status of a DSS
    (either State 1 or State 0). It is also concerned with formally
    identifying the requirements for State 2 to fulfill and its expected
    outcomes. This phase is further concerned with ensuring that the
    plans for upgrading the DSS are established and the design for
    obtaining State 2 is developed. Finally, performance metrics for the
    upgraded DSS are developed and established. Phase 2 *development* of
    plans for creating a State 2 DSS leads to the third phase.

-   *[Assimilate and analyze]{.underline}*---involves measuring the
    performance of both the baseline DSS and its upgrade to State 2,
    comparing performance indicators, evaluating the results obtained,
    collecting metrics, and performing a risk-benefit assessment. Phase
    3 *synthesis* leads to the fourth phase.

-   *[Adapt and make it operational]{.underline}*---is the point at
    which a system is either certified for operational status or the
    process is re-initiated for additional enhancements. If the system
    meets standards and is certified for operational status, the
    benchmarking process will be completed with formal submission of all
    required documentation. If additional improvements in the DSS are
    required at this phase, the benchmarking rols full circle through
    *enhancement* back to the first phase.

Benchmarking is a formal tool for process direction, control, oversight,
and documentation, from start to finish. It is critical to success of
the assimilation process. It provides a formal means by which system
changes---improvements---between the current and enhanced state of a DSS
are measured and ultimately evaluated. Benchmarking must be a team
effort. By the completion of this task, a Benchmarking team must be
established to guide the rest of the assimilation effort.

***The second task of Step 2*** is to form the Benchmarking team. The
Assimilation team with the program manager will identify the
Benchmarking team members and will establish the roles and
responsibilities of the various team members and their organizations.
The team will be drawn from the technical staff of the partner agency,
appropriate NASA Center(s), appropriate universities, and the NASA Earth
Science Enterprise. The Assimilation and Benchmarking teams have
different roles in the assimilation process---the Assimilation team is
system oriented while the Benchmarking team is process focused---but may
include some of the same people as members.

Upon being established, among other tasks, the Benchmarking team will:

-   document the role it will play throughout the assimilation process,

-   develop the documentation and reporting protocols and requirements,

-   determine and agree upon the metrics and performance objectives that
    will underpin measuring the change from the current, or baseline
    state to enhanced operational status,

-   establish the basis for determining the acceptability of NASA
    technology, data, and information products, and

-   determine the criteria for completing the assimilation effort.

***The third task of Step 2*** is for the Benchmarking and Assimilation
teams to formally, completely characterize the operational functionality
of both the DSS that is being upgraded and the DMS within which it
functions. This involves at least two levels of assessment. The upper
level of assessment requires classification of the decision-making
system (DMS) and its environment within which the DSS is embedded. The
lower level requires a clear statement of the objectives the DSS is
intended to satisfy. The following are key considerations in
understanding the objectives of a DSS: (1) Classification---an upper
level assessment that baselines the decision-support environment and
defines its relationship with the DSS, (2) Environment---an automated,
digital information processing system, (3) Information Products---what
is produced for the decision-maker, and (4) Operational Plans---plans
for enhancing the DSS to better serve the decision process.

![](media/image4.png){width="3.6251968503937007in"
height="1.6874015748031497in"}

Figure 6 illustrates a straight-forward A-B-C-D upper level
classification scheme for a decision-making system and it's DSS that
will be used to characterize all States of a DMS. It provides an
explicit representation of a classification based on the scheme
presented above in the section on Decision-Making Considerations. It
encapsulates the problem framework, types of knowledge, knowledge
management paradigm, and the typology of the decision-making system
making it simple and straight forward to classify. A description of each
category follows:

-   A is an assessment of the problem framework and describes the degree
    of complexity and structure. The more structured a problem, the more
    amenable it is to solution by computerized algorithms in a DSS. The
    more unstructured a problem, the more a decision must rely on the
    judgment and intuition outside the DSS. An important implication of
    this for assimilation is that new data and models introduced may
    increase the structure of the decision-making context while
    expanding the capability of the DSS and thus relieving the
    decision-maker of intuition/experience/expert judgment as the
    driving decision rationale,

-   B identifies the type(s) of knowledge used. Each imposes conditions
    on a DSS and each represents a different management challenge,

-   C describes the knowledge management paradigm(s) in use. Each
    presents unique challenges that must be met,

-   D provides a typology of the decision-making system/environment.
    Being able to accurately categorize a decision-making
    system/environment by type identifies how a DSS must be structured
    for success---arbitrarily changing type to accommodate NASA Earth
    science models and data can be avoided.

It is this A-B-C-D classification of a decision-making system that will
guide the upgrade assimilation of a DSS to State 2, will guide our
system engineering support, and will help to achieve the greatest degree
of benefit while minimizing risk.

![](media/image5.wmf)As an example of the use of this classification
scheme, Figure 7 illustrates a possible State 1 classification of a DMS.
This example classification shows the system to function within the
decision support framework as a semi-structured/complex problem \[A =
5\], to require declarative, procedural, esoteric, and deep knowledge
\[B = 3,4,5,7\], to operate in the functional knowledge management
paradigm \[C = 1\], and to be embedded in a consensual and empirical
decision-making environment \[D = 2,3\].

![](media/image6.wmf)Figure 8 illustrates the classification scheme that
might result from upgrading the DMS/DSS to an enhanced State 2. Note
that the decision support framework changed to be a
semi-structured/modeled and semi-structured/complex problem \[A = 4,5\].
This suggests that part of the problem has been constrained in degree of
complexity---part remains complex---thereby making a part of the
resultant DSS function at a more routine level and thus making it more
amenable to algorithmic solution. Note also that the types of knowledge
required have been reduced to just declarative and deep \[B = 3,7\] and
that the system still functions in a functional knowledge management
paradigm \[C = 1\]. Dropping the requirement for procedural knowledge
eliminates the need to deal with the problem of "how to" do something
and dropping the need for esoteric knowledge removes the use of
knowledge applicable to only a narrow domain. Note that State 2 is now a
straight-forward empirical decision-making system \[D = 3\]. These
changes are significant, suggesting that the assimilation work has
changed the DSS to be a more computerized algorithm/modeled solution.
These changes have a number of important ramifications for the
upgrade/enhancement to State 2. Among these are the changes that offer
the prospects of: (1) enhanced repeatability of results, (2) increased
accuracy, (3) increased throughput speed, (4) reduced costs, and (5)
increased socio-economic benefit.

Altogether, these changes are positive and should result in an
assimilation effort that is satisfying to both NASA and the partnering
agency.

The final act of Step 2 is to baseline the current state. In either
State 1 (a DSS that is fully functional and routinely providing support
to its DMS) or State 0 (a DSS that is either planned or under
construction but not yet operational), the DSS will be characterized
component-by-component and this assessment will be recorded as a formal
part of the benchmarking process. Each DSS is likely to be unique, but
since each is a system that uses domain information from a resident
knowledge system[^2] (KS) to process an input signal into decision
support information products, a listing of candidate relevant
characteristics to baseline include: (1) current input and output
requirements being fulfilled, (2) current science models and algorithms
being used, functions, internal algorithms, processing steps, and data
flows, (3) hardware and software, (4) people involved, level of training
required, specialized skills needed, (5) current outputs, and (6)
current level of performance.

***The fourth task of Step 2*** is to formally determine and clearly
state the objectives that the DSS, functioning in State 1 or existing in
State 0 while being developed, is intended to satisfy. The original
objectives may be institutionalized in a mission statement, but with
time "requirements creep" may result in very real, but unstated enhanced
objectives. Thus, it is important to come to an explicit agreement
regarding the current, actual objectives; and especially to understand
how those are expected to change if the DSS is upgraded.

Several key considerations for understanding the decision-support
objectives of a candidate DSS include:

-   *[Classification]{.underline}*---Classify the environment of the
    decision-making system via the variables presented in Figure 5 and
    discussed above to baseline the decision-support environment and to
    establish the functional relationship between the DSS and the DMS
    within it operates,

-   *[Environment]{.underline}*---Determine that the DSS is implemented
    in an automated, digital environment and is, at the very least, an
    automated, digital information processing system to establish its
    suitability to be enhanced by assimilation of NASA results,

-   *[Information Products]{.underline}*---Determine the kinds of
    information products produced by the DSS and how they are used by
    the DMS,

-   *[Operational Plans]{.underline}*---Obtain and examine the partner
    agency's operational plan for the DSS and establish current
    requirements to be met in upgrading the system.

It is not required that a DSS be operational (functioning in State 1)
for it to be a candidate for assimilation of NASA results. In fact,
flexibility and the opportunity for introducing new science models and
data products may be greater for DSSs in the planning or prototype
development stages (existing in State 0) than for established DSSs.
Being aware of the DSSs development state will help to clarify the
opportunities for upgrade enhancements.

***The fifth task of Step 2*** is to benchmark the current state of the
DSS being considered for enhancement. This involves characterizing the
system component-by-component and formally recording the assessment as
previously discussed.

If a *State 1* DSS is proposed for enhancement, it is already
functioning in an operational context and being used by decision makers
to achieve their policy and management needs. In this case, the actual
operational performance characteristics can be observed, measured, and
codified, but it will be necessary to measure these without disrupting
ongoing activities.

If a *State 0* DSS is proposed for enhancement, it may be only in the
planning stages, or it may only exist as a prototype under development.
Benchmarking a State 0 DSS is more uncertain or nebulous, without fully
knowing what the actual behavior and performance of the proposed system
will be. The State 0 DSS offers the opportunity for early NASA
involvement in the design process and may provide the way for a more
complete upgrade to better satisfying requirements.

Since each DSS encountered is likely to be unique, example relevant DSS
characteristics to examine, not an exhaustive listing, are listed for
illustration purposes:

-   *[Current inputs]{.underline}*---Identifying the science models and
    data inputs used in the State 1/0 DSS will establish the current
    input characteristics, may lead to identifying which NASA models or
    data products should be examined for possible use in the enhanced
    DSS, and may show needs that can be filled by new inputs. Specific
    input characteristics and requirements should be evaluated, such as
    the following:

    -   data types and sources,

    -   data formats,

    -   resolution -- spatial, temporal, spectral,

    -   data cost,

    -   availability and latency of data,

    -   volume of data to process,

    -   accuracy -- geospatial, numerical, thematic,

    -   etc.

-   *[Current functions, internal algorithms, processing steps, data
    flows]{.underline}*---Knowing the structure and function of all of
    the elements of a particular DSS will be necessary to baseline its
    performance characteristics.

-   *[Hardware and software used]{.underline}*---A complete
    characterization of the functional environment within a DSS is
    required in order to benchmark the current state,

-   *[People involved, levels of training, specialized
    skills]{.underline}*---It is necessary to understand the nature of
    the workforce that is charged with operating the DSS in order to
    identify opportunities and constrains for improvements.

-   *[Current outputs]{.underline}*---The output of a DSS is information
    that provides advice to a human decision maker. This advice may be a
    single coherent, "boiled down" prediction or recommendation, or it
    may be a set of discrete pieces of information to be evaluated and
    combined with other data and information in the decision process.
    Outputs may be textual, numerical, graphical, or pictorial, or a
    combination of these. Since each DSS is likely to be unique, an
    example listing of relevant DSS characteristics to examine, not an
    exhaustive listing, are provided for illustration purposes in the
    following:

    -   output data types,

    -   data formats,

    -   resolution -- geospatial, temporal, thematic,

    -   spatial and temporal coverage,

    -   production cost,

    -   availability, timeliness of the production output,

    -   volume of output information to interpret,

    -   accuracy -- locational, numeric, thematic,

    -   etc.

-   *[Current performance]{.underline}*---Performance is the ultimate
    level of benchmarking. There is a wide range of benchmarking
    performance measures, some of which are described below for
    illustration purposes. The actual performance metrics to be used
    will be established by the Assimilation and Benchmarking teams.

    -   range of services provided,

    -   processing speed,

    -   processing costs,

    -   trained staff requirements,

    -   latency of products,

    -   accuracy of products -- location, numeric, thematic,

    -   bottlenecks,

    -   etc.

## Third Step

The third step has three major goals: (1) identify possible NASA
results, (2) develop an enhanced DSS, and (3) transition it to
operations. In the normal case, it may require significant resources and
a major effort expended over multiple fiscal years to complete the work
of this step.

Figure 4 captures the work of this step in the block entitled *Determine
NASA Assimilation Components* and the four surrounding blocks. As can be
seen from the figure, these three goals complete the part of the general
procedure wherein State 2 is developed and transitioned to operations.

An explicit model is offered in Figure 9 for the major effort captured
as tasks 2 -- 7 of this step. The figure graphically illustrates the
interrelations and flow of work and information captured in these tasks.
It is intended to help illustrate the text of each task. Figure 9
presents one model; many different approaches are possible.

***The first task of Step 3*** is to prepare a development plan for the
overall effort. NASA will be primarily responsible for the first goal
stated. The partnering Federal agency will (in the usual case) be
responsible for the second and third goals and NASA's role for these
goals will be primarily that of consultation. Effective planning,
including detailed verification and validation (V & V) and benchmarking
plans, is crucial to ultimate success of this venture.

***The second task of Step 3*** is to determine the DMS/DSS needs and
requirements that are driving the effort to upgrade the DSS to an
enhanced operational state. The foundations for developing requirements
should be outlined in the agreement that has been negotiated in the
first step. The objectives and metrics contained there should form the
basis for developing a requirements document.

There exist various levels of needs and requirements and of upgrade
planning by an owner/user agency, various degrees of documentation, and
numerous ways for obtaining needed information. Among these are the
following three, illustrative of the continuum of possibilities with
variations on these themes certain:

-   *[Formal upgrade plans and documentation of requirements and needs
    exist]{.underline}*---Obtain existing documentation of needs and
    requirements, and of the current upgrade plans for the DSS from the
    owner/user agency,

-   *[Informal plans for upgrade and anecdotal requirements
    exist]{.underline}*---Anecdotal evidence of needs and requirements
    and for upgrade planning by the owner/user agency should be
    documented by the Assimilation and Benchmarking teams and this
    documentation should become the basis for developing a requirements
    document.

-   *[NASA-generated interest for upgrade with no basis of
    requirements]{.underline}*---The DSS owner/user agency may have no
    current plans to upgrade and may be interested in upgrading only
    because NASA offers significant additional capabilities. The
    requirements and needs in this case may be completely ad hoc, based
    solely on the potential augmentation offered by NASA inputs. The ad
    hoc evidence should be documented by the Assimilation and
    Benchmarking teams and used to baseline the requirements and needs
    and to stimulate upgrade planning.

***The third task of Step 3*** is to prepare a formal requirements
document. The bulwark of effective systems engineering is the
requirements/needs document. The requirements should be categorized by
engineering versus scientific themes. Engineering themes will be those
which improve throughput, reduce cost, increase capacity, etc.
Scientific themes will be those which augment capability, introduce
prediction, improve forecasts, reduce uncertainty, improve calibration,
etc. The engineering themes will probably be the responsibility of the
partnering Federal agency while the scientific themes may be NASA's
responsibility.

***The fourth task of Step 3*** is to brief the program manager. The
Assimilation and Benchmarking teams should brief all findings to the
appropriate NASA program manager and present the requirements and needs
document as evidence. In most cases, the program manager will be
involved throughout the assimilation process and this step will be a
formality. It is nevertheless an important step because it documents the
set of requirements and needs and formally sets in motion the process to
find appropriate NASA solutions.

***The fifth task of Step 3*** is to identify existing results. The goal
of this task is to uncover both engineering and scientific
capabilities---data, science models, technology, systems engineering
enhancements---for NASA to *consider* for the State 2 upgrade. The
requirements/needs document will form the basis for this effort. The
program manager, with the Assimilation and Benchmarking teams will
establish a process for casting a net NASA-wide to find assimilation
augmentation possibilities meeting the requirements/needs.

In some cases, this will be straight forward with obvious solutions. In
others, solutions will not be obvious and a process for identifying
candidate solutions is needed. Numerous ways exist for doing this. One
that can be very effective when potential solutions are not obvious is
to hold a joint workshop with appropriate attendees from NASA, the
DSS/DMS owner agency, downstream users and potential users, and
representatives of the pertinent scientific communities. The findings of
the workshop will identify suitable candidate solutions. At this point,
these are only potential solutions, solutions that may work, but it is
not certain how they will perform in an operational setting.

***The sixth task of Step 3*** is to determine from the list of
candidates a set of specific NASA recommended solutions for the planned
upgrade to better serve the decision process. This will require a number
of issues to be examined before a candidate solution is moved forward to
become a part of the solution set that is recommended to the partnering
agency.

The Assimilation and Benchmarking teams, together with experts appointed
by the program manager should review in the context of (1) an
operational setting, (2) the requirements or needs it satisfies (cf.,
completing task 3 of this step), and (3) the baseline DSS/DMS
characterization resulting from completing the third task of Step 2, the
suitability of each candidate to become a solution.

A clear statement of the DSS objectives a particular solution is
intended to satisfy should be crafted. In particular and working from
the previous Step 2 baseline characterization of State 1, the following
items need to be considered:

-   *Classification*---an upper level assessment of how this solution
    functioning in State 2 will augment capability while being an
    extension, yet compatible with the assessment made in the third task
    of Step 2 \[See the text related to Figure 6: Decision-Making System
    Classification.\],

-   *Environment*---determination of either the interface requirements
    to the existing automated, digital information processing system
    \[cf., task 3, Step 2\] or the requirements for a new system to meet
    the HCI needs of the augmented system,

-   *Information products*---expectations of the specific products that
    will be produced for the decision maker and how they differ from the
    current set,

-   *Operational plans*---identifying the needs for enhancing the
    operational environment within which the DSS functions.

```{=html}
<!-- -->
```
-   *DMS issues*

    -   *How will the DMS change*---Will this change the environment
        within which the DSS functions,

    -   *How may the DSS/DMS classification* *change at State 2*---Will
        the problem framework, types of knowledge, knowledge management
        paradigm, or type of decision making system change,

    -   *Policy Issues---*Unique requirements for each DSS,

    -   *Management Issues---*Unique requirements for each DSS,

-   *DSS issues*

    -   *Scientific issues*---The impact on the Knowledge system of a
        recommended change \[cf., Figure 1: Knowledge System\],

        -   *Data suitability and availability for an operational
            need*---Does the data improve the DSS outputs and are the
            data routinely available to meet this need?

        -   *Mission longevity/successor missions*---Will necessary data
            continue to the available for as long as the DMS needs?

        -   *Calibration/accuracy/errors*---Will this change/improve the
            DSS output accuracy and reliability?

        -   *Model forecasting capability*---Does the forecasting
            capability of this model meet DMS operational needs?

        -   *Model suitability/scalability*---Is the model suitable for
            the DMS operational needs and is it scalable for that
            environment?

        -   *Etc*.

    -   *Engineering issues*---The impact on the [Precision
        Processing]{.mark} System of changes required by changes in the
        Knowledge System \[cf., Figure 2: Decision Making System\],

        -   *Throughput analysis*---Does performance resulting from this
            change meet the operational needs for the application?

        -   *Cost/benefit analysis*---Does the change improve this
            ratio?

        -   *Throughput capacity*---Is the information processing system
            suitable to accept this change?

        -   *Etc*.

***The seventh task of Step 3*** is to propose NASA solutions to the
DSS/DMS owner agency. The program manager, with the Assimilation and
Benchmarking teams, will prepare a proposal of recommended NASA
solutions for the design of an enhanced DSS. They will then brief the
DSS owner agency and decision-making constituency (the DMS agency) on
NASA's recommendations.

***The eighth task of Step 3,*** upon acceptance of NASA's
recommendations by the owner agency, is to develop the State 2 DSS.
Referring again to Figure 4, note that the upper right-most block,
*Develop enhanced DSS,* is tan to signify that in general this effort
will be managed, funded, and implemented by the DSS owner agency. NASA's
role in this particular effort will vary from case to case, but in
general it is recommended to be limited to that of consultant.

There exists no single solution as to how to go about developing the
system known as State 2. Accordingly, we recognize this and since it
almost certainly is the owner agency's responsibility, we merely
acknowledge that NASA will have a role, to be negotiated, and that this
part of the development process will be lengthy and require significant
resources.

***The ninth task of Step 3*** is to conduct verification and validation
(V & V) on the State 2 system components incorporating the NASA suite of
solutions. Verification is a life cycle process to ensure the products
being developed meet the stated specifications (functional, performance,
and design). Validation is a process to ensure the completed products
(software, algorithm, or model) effectively serve the functional
requirements. V & V must occur for all potential enhancements to a DSS.
The responsibility for V & V is usually dispersed, residing often with
the original product, etc., developers. As noted in task 1 of this step,
a detailed V & V must be developed and implemented.

***The tenth task of Step 3*** is to perform DSS demonstrations. This is
the part of the effort at which a system is tested in its operational
environment. A major part of this effort is to verify that the system is
sufficiently robust to meet the most severe DMS, or other user needs.

***The eleventh task of Step 3*** is to transition the upgraded, State
2, DSS to operations and to conduct benchmark testing.

## Fourth Step

The fourth step is to measure the improvements achieved in the
assimilation process and to transition the enhanced DSS to operations.
Refer back to Figure 4 one last time. The bottom three blocks represents
the end of the assimilation activity for a particular DSS. These are (1)
*Examine enhanced DSS*, (2) *Measure improvements*, and (3) *Complete
benchmark process*. The first, second, and third tasks, respectively, of
this task treat those blocks. ***The first task of Step 4*** is to
benchmark the State 2 DSS. This requires complete characterization of
State 2 just as was done with the baseline system at State 1/0. The
appropriate parts of Step 2, not repeated here, are required to be
applied to characterization of State 2.

***The second task of Step 4*** is to apply the metrics previously
designed and established to the task of assessing the DSS State 2. The
same measures that were applied to benchmarking State 1/0 are to be
applied to State 2.

***The third task of Step 4*** is to complete the benchmark process.
This is achieved by:

-   Comparing the characterization of State 2 (enhanced operational
    status) to State 1/0 (baseline status),

-   Measuring the changes obtained in moving from the baseline DSS to a
    system functioning at the enhanced operational status of State 2,

-   Applying the metrics and assessing the performance objectives
    previously designed.

The difference in system performance between States 1 (or 0) and 2
should demonstrate improvements in one or more of the metrics that were
selected as performance measures at the beginning of the benchmarking
and improvement planning phases. Again, some of these may relate to
straightforward measures of speed and accuracy. However, others should
describe changes in the fundamental nature of the decision support
system itself. These might be movement from an unstructured to a
structured problem-solving environment that allows increased automation
of the decision making process. They might also include increased use of
quantitative data that also permit movement toward automation, but may
also increase reliability or replicability, or reduce levels of
uncertainty. Ultimately, these measures may prove to be at least as
compelling for the partner agency as improvements in simple measures,
like accuracy.

***The fourth task of Step 4*** is to document the results achieved. The
entire process must be documented. Part of this is in keeping with good
practice. In the assimilation process, however, it assumes a larger
role. These will be the results that are reviewed by partner agency
managers to make decisions that may have profound influence on their
"corporate ways of doing business," and will thus be closely scrutinized
at a number of levels. Moreover, when the partner agency assumes
operational responsibility, this will serve to document the ways in
which State 2 evolved and how it functions, and also to demonstrate the
improvements that it offers over State 1.

***The fifth task of Step 4*** is to transfer responsibility to the
owner agency for operations, maintenance, and performance of the DSS and
the NASA solutions that have been incorporated. In the general case,
this will be the point at which NASA disengages from the assimilation
project and the DSS begins to function as a routine component of the
over decision making system.

The owner agency plays an active role throughout the process. However,
it is likely that their participation will be highly selective in terms
of the staff involved, and that the individuals involved will be those
who are particularly skilled -- and interested -- in the activity. Thus,
in the process of transferring operational responsibility, training of
rank-and-file agency staff becomes critically important. Before the
system enters operations, the workforce needs to be trained in how to
run the system. The DSS owner agency needs to be trained to properly
interpret the information product outputs. And, the DMS elements, the
decision-makers, need to be informed as to the new range of outcomes now
possible and the new limitations of the system.

Transition to *Operational Status* represents successful installation of
a State 2 DSS and completion of the assimilation and development
process. Until and unless the agency assumes full responsibility for
operation and disseminates the DSS to all those divisions who have need
of the tool, the project is incomplete. This represents the ultimate
goal of the activity and it must be done by the partner agency.

# Next Steps

From NASA's perspective at this point, a project should be completed
that assimilates---infuses---NASA Earth Science data, mission products,
technology and/or science results into a priority National Application
DSS owned by a partnering Federal Agency. The enhanced DSS functioning
at State 2 should have been transitioned to operations. A final
risk/benefits analysis should have been conducted. The improvements
achieved should have been measured. The assimilation/benchmarking
process should have been completed.

The next steps involve capturing the myriad documentation from the
completed project, preparing final reports, and planning for the future.
All of the documentation of the benchmarked process should be acquired,
reviewed and analyzed, and a *'Success Story',* a *'Lessons Learned'*
report, and a *'Final Report'* should be prepared. The *'Success Story'*
should report the degree to which the original goals have been achieved.
These should include the following points:

-   The significance and value provided to the decision-making process
    through application of NASA results in an enhanced DSS for a
    sustainable national application,

-   The return to society for its investment in the NASA Earth Science
    Enterprise achieved in this completed DSS enhancement,

-   The degree of improvement and the significance for management and
    policy decisions that is provided by the upgrade in information
    available from the enhanced DSS,

-   The improvements in the accuracy, speed, and scale of applicability
    of the outputs from the enhanced DSS and,

-   The path that this NASA product followed (missions, to models, to
    decision support) so that the process can be fully understood and
    the value of the investments made at each step can be explained.

The *'Lessons Learned'* report should include all successes and failures
that have bearing on future assimilation/infusion efforts. The *Final
Report'* should provide traditional information sufficient to properly
complete the documentation requirements of the project.

Future planning is the next and final act. This project is completed,
but it is highly likely -- and desirable -- that a future
assimilation/infusion process is planned that will incorporate
additional/new NASA results as the DSS is upgraded to a new operational
State 3. The next steps, then, are to (1) obtain the documentation from
this completed effort, (2) analyze the process, its achievements, and
its failures as they might relate to the future upgrade, (3) review the
'Lessons Learned' documentation for specific considerations that might
apply to the new effort, (4) start the process over from the beginning
(from the Second Task of Step 1 of this procedure) for the State 3
assimilation/infusion upgrade.

# Conclusions

This document summarizes the general DSS assimilation guidebook[^3]. It
provides a straightforward tour through the four essential steps
required to complete a successful assimilation of NASA results into a
partnering agency's DSS. The tour follows a path through the steps, from
the starting requirements for establishing a need for DSS enhancement to
transferring an enhanced DSS to its operational environment. This
particular tour makes it appear that the path is linear and that
successive steps must follow in order, which is a consequence of this
summary providing order to a complex undertaking. Clearly, when it makes
sense to conduct tasks out of order and/or in parallel, then that is the
correct path to follow.

This general DSS assimilation procedure is founded on proven theoretical
considerations. It is currently being tested on several DSS assimilation
projects.

[^1]: Kaupp, V., C. Hutchinson, S. Drake, W. van Leeuwen, and D. Tralli,
    *Assimilation of NASA Earth Science Results and Data in National
    Decision Support Systems: A Guidebook,* NAG-5 -12482 and
    NAG-5-12484.

[^2]: Knowledge System (KS)---is the core of a decision-making system
    containing the complete domain of information about the decision
    environment.

[^3]: Kaupp, V., C. Hutchinson, S. Drake, W. van Leeuwen, and D. Tralli,
    *Assimilation of NASA Earth Science Results and Data in National
    Decision Support Systems: A Guidebook,* NAG-5 -12482 and
    NAG-5-12484.