**Assimilation of NASA Earth Science Results and Data in National
Decision Support Systems:**

**A Guidebook**

**Verne Kaupp**

**University of Missouri**

**Charles Hutchinson**

**Sam Drake**

**Willem van Leeuwen**

**University of Arizona**

**David Tralli**

**Jet Propulsion Laboratory**

**October 15, 2003**

# TABLE OF CONTENTS

[TABLE OF CONTENTS 2](#table-of-contents)

[EXECUTIVE SUMMARY 4](#executive-summary)

[INTRODUCTION 6](#introduction)

[Purpose of this Document 6](#purpose-of-this-document)

[The Approach Summarized 8](#the-approach-summarized)

[Organization of this Document 9](#organization-of-this-document)

[APPROACH 11](#approach)

[I---FRAMING THE PROBLEM 13](#iframing-the-problem)

[DECISION-MAKING CONCEPTS 13](#decision-making-concepts)

[Decision Making Systems 14](#decision-making-systems)

[Framework 16](#framework)

[Types of Knowledge 17](#types-of-knowledge)

[Knowledge Management Paradigms 17](#knowledge-management-paradigms)

[Classification 17](#classification)

[Example Classifications 18](#example-classifications)

[BENCHMARKING CONCEPTS 20](#benchmarking-concepts)

[Decision Support Systems (DSSs) and Benchmarking concepts
20](#decision-support-systems-dsss-and-benchmarking-concepts)

[Types of benchmarking activities 20](#types-of-benchmarking-activities)

[The Benchmarking Process 21](#the-benchmarking-process)

[II---ESTABLISHING AGENCY RELATIONSHIPS
24](#iiestablishing-agency-relationships)

[Identify DSS 25](#identify-dss)

[Form assimilation team 25](#form-assimilation-team)

[Conduct zeroth-order assessment 26](#conduct-zeroth-order-assessment)

[Conduct risk analysis 26](#conduct-risk-analysis)

[Obtain agreement 27](#obtain-agreement)

[III---CHARACTERIZING CURRENT DSS 28](#iiicharacterizing-current-dss)

[Start benchmarking process 28](#start-benchmarking-process)

[Form benchmark team 29](#form-benchmark-team)

[Characterize the DSS 30](#characterize-the-dss)

[Characterize DMS environment 30](#characterize-dms-environment)

[Determine DSS objectives 30](#determine-dss-objectives)

[State 0 or State 1 31](#state-0-or-state-1)

[State 1 31](#state-1)

[State 0 32](#state-0)

[Benchmark the current state 32](#benchmark-the-current-state)

[IV---ASSIMILATING NASA PRODUCTS 34](#ivassimilating-nasa-products)

[Plan for DSS enhancement for State 2
34](#plan-for-dss-enhancement-for-state-2)

[Determine needs/requirements for DSS upgrade/enhancement
35](#determine-needsrequirements-for-dss-upgradeenhancement)

[Prepare requirements document 36](#prepare-requirements-document)

[Brief program manager 36](#brief-program-manager)

[Solicit existing results 36](#solicit-existing-results)

[Determine suitable NASA capabilities
36](#determine-suitable-nasa-capabilities)

[Propose assimilation plans 37](#propose-assimilation-plans)

[Develop enhanced DSS---State 2 37](#develop-enhanced-dssstate-2)

[V---MEASURING IMPROVEMENTS AND MOVING TO OPERATIONS
39](#vmeasuring-improvements-and-moving-to-operations)

[Complete benchmark process 39](#complete-benchmark-process)

[Document results 39](#document-results)

[Transfer responsibility 40](#transfer-responsibility)

[REFERENCES 41](#references)

[APPENDIX A: Decision Making Systems
42](#appendix-a-decision-making-systems)

[APPENDIX B: Decision Support Systems Framework
47](#appendix-b-decision-support-systems-framework)

[APPENDIX C: Types of Knowledge 48](#appendix-c-types-of-knowledge)

[APPENDIX D: Knowledge Management Paradigms
49](#appendix-d-knowledge-management-paradigms)

[APPENDIX E: Decision-Support Systems Classification
50](#appendix-e-decision-support-systems-classification)

[APPENDIX F: Benchmarking Tasks 52](#appendix-f-benchmarking-tasks)

[APPENDIX G: Usability and Performance Metrics
53](#appendix-g-usability-and-performance-metrics)

[APPENDIX H: Benchmarking Code of Conduct
54](#appendix-h-benchmarking-code-of-conduct)

[APPENDIX I: Risk analysis and benefit assessment
57](#appendix-i-risk-analysis-and-benefit-assessment)

#   {#section .unnumbered}

# EXECUTIVE SUMMARY

The purpose of this document is to outline a process for
infusing---assimilating---NASA geospatial data, mission products, and/or
science results into a priority National Application decision support
system (DSS) owned by a partnering Federal agency.

A National Application (1) is identified as a national priority by the
Executive and/or Legislative branches, (2) is relevant to national
program(s) of one or more Federal agencies, (3) has validated
requirements (by other agencies) with the potential to be served by
Earth science and remote sensing research and development results, and
(4) has significant societal and/or economic value in terms of clearly
defined metrics, such as quality of life improvements, potential lives
saved, and economic or resource savings.

The first set of tasks in the process is intended to develop a working
relationship with the host agency, conduct a feasibility assessment, and
establish an agreement. The process begins with identification of a
suitable DSS within a partner agency that might benefit from
assimilation of NASA science results, data or technology. The types of
benefits that might be realized by this assimilation are identified in a
"zeroth" order, or feasibility assessment of the system. This is
accompanied by a preliminary risk assessment that is intended to
identify potential benefits and undesirable outcomes as well that might
accompany assimilation. The products form the basis by which an
agreement is developed between NASA and the host agency that establishes
their shared mutual commitment and lays out a clear set of objectives.
Throughout the assimilation activity, work is conducted by teams that
have representatives from both the NASA "side" and the host agency.

The second set of tasks is intended to characterize and benchmark the
existing (State 1) or planned (State 0) DSS of the host agency. This
will include a description of processes, inputs, outputs, processing
requirements, speeds, accuracies and the types of information that would
be expected in the description of any system. In addition, however,
there will be a more theoretical description of the structure and
operation of the whole decision making system in which the DSS is
embedded, and the kinds of knowledge on which it is based. In the first
case, some decision making systems are highly structured, mechanistic,
and are driven by quantitative data. Others are flexible, largely
unstructured, and accommodate qualitative data. Clearly, there is a wide
array of options between these two extremes and most decision making
systems are amalgams of different types of DSSs. However, as a general
principle, the evolution of DSSs is to move along a path from
unstructured to structured problem solving, and to replace qualitative
data with quantitative so that the problem can be solved in a more
systematic and reliable way. Seasonal climate forecasting is a good
example of this evolution in which, initially, forecasts were based on
past patterns, hunches, and obscure observations of questionable value
(e.g., the behavior of certain plants and animals noted in *The
Farmers'Almanac*). Now, forecasts are based on models of physical
processes that are built on sound theory, that are driven by a web of
observations made from the ground, from satellites, and buoys that
describe temperature, vertical structure, winds, circulation of the
atmosphere and ocean, and topography of the sea surface. Thus, in
addition to considering improvements in obvious measures such as speed
and accuracy, improvements may also be measured in the degree to which
problem solving moves from unstructured to structured, or the degree to
which it considers measures such as replicability/reliability, and the
degree to which risk or uncertainty is reduced.

In keeping with the partnering nature of the whole activity, a
benchmarking team that measures current performance of the system at
State 1 (or State 0) is assembled from representatives from the host
agency, appropriate NASA centers, universities, and the private sector,
as appropriate. Because of its supporting role to the Applications
Division in verification and validation, the Stennis Space Center is
expected to be involved in most benchmarking activities.

The third set of tasks is the assimilation itself, in which the existing
system (State 0 or State 1) is modified by adding new data, algorithms,
or other NASA-derived products. This activity is undertaken by the
assimilation team. While the benchmarking and assimilation teams may
share some members, and the results of the benchmarking activity may
help shape the assimilation objectives, the function of the two teams is
clearly different. In particular, the benchmarking team must be viewed
by all as independent and its analysis transparent to ensure its
credibility.

As with benchmarking, the assimilation team is a partnership with
representatives drawn from the host agency, appropriate NASA centers,
universities, and the private sector, as appropriate. The first task of
the team is to develop an assimilation plan that clearly establishes
performance objectives, how they will be achieved, roles and
responsibilities of the partners, and what the expected outcomes might
be. The assimilation process is open in that changes are made to the
system, performance is assessed, and further changes are made in an
optimization loop until performance objectives are met or approximated.

The fourth set of tasks completes the exercise. It involves (1)
benchmarking of the enhanced system (State 2), (2) a comparison of
difference in performance -- including risk reduction -- between States
1 and 2, (3) preparation of a final report, and (4) "operationalization"
of the enhanced DSS. The benchmarking team will again perform an
assessment of the State 2 DSS. The performance of the system will be
compared with the assessment from State 1. Results will be captured in a
final report that summarizes the differences in performance,
particularly in risk reduction. The final step of the process is the
"hand-off' in which the host agency formally implements the enhanced
system and assumes operational responsibility. This final step is the
ultimate measure of success.

# INTRODUCTION

## Purpose of this Document

The purpose of this document is to outline a process for
infusing---assimilating---NASA Earth science data, mission products,
and/or science results into a priority National Application decision
support system (DSS) owned by a partnering Federal agency.

NASA recognizes the importance of partnering with other Federal agencies
to make systematic connections between Earth Observing System (EOS)
geospatial data, science models, and their applications in the relevant
decision-making systems (DMSs) that serve our Nation's policy and
decision makers. These partnerships focus on innovative approaches for
using Earth science information and knowledge to provide or enhance
decision-support capability. This enhancement of capability supports
myriad operational applications, as well as policy discussions, and is
used to address a variety of national priorities that range from
agricultural efficiency to homeland security and their resultant
socioeconomic impacts.

NASA is committed to understanding the Earth system and improving
predictions or forecasts of climate, weather, and natural hazards.
Employing over 15 earth observing satellites routinely making
measurements with over 80 onboard remote sensing instruments, NASA has
defined two strategic objectives: (1) Science---to observe, analyze, and
model the Earth system to discover how it is changing and the
consequences for life on Earth; and (2) Applications---to expand and
accelerate the realization of economic and societal benefits from Earth
science information and technology; with Earth System Science and Earth
Science Applications as corresponding themes.

Through the Earth Science Applications Program, NASA is working with
agencies and organizations that have the mandate and requisite
infrastructure to apply NASA's Earth science results in order to help
manage coastal environments, water resources, and aviation safety;
monitor air quality, forest fires, and the impacts of infectious
diseases and invasive species and conduct hurricane forecasting,
disaster preparedness and response efforts.

**Figure 1. Earth Science Enterprise Components**

**TBD**

Figure 1 illustrates the Earth Science Enterprise system components and
the general scope of the assimilation process available from a
NASA-centric position. NASA ***missions*** carry Earth observing
instruments into space. The instruments produce experimental (proof of
concept) or systematic (extended) ***observations*** of the Earth that
are molded into ***geophysical parameters*** to answer specific science
questions about Earth system processes (e.g., the forces that drive
climate). By understanding processes, it is possible to develop
***models*** that can be used to ***predict*** future conditions (e.g.,
El Niño seasonal climate forecasts). The outputs of models can be used
to make ***decisions*** (e.g., what price to pay for corn futures),
increasingly through the use of formal ***decision support systems***
(DSS). Clearly, there will be instances where NASA observations may be
directly incorporated in an existing DSS (e.g., SRTM topography to
assist in flood planning). However, it is likely that science outputs --
models that allow future scenarios to be evaluated -- will assume
increasing importance as we seek to increase economic efficiency and to
protect lives and property.

The purpose of this document is to outline a general procedure by which
NASA data, science results, and technology can be incorporated --
assimilated -- into an existing DSS. More importantly, it suggests a
process by which the performance of that DSS can be measured before and
after the assimilation itself. The intent is to allow the partner agency
to evaluate the improvements that can be realized by adopting NASA
products and thus ultimately to improve the quality of service that they
provide their clients.

## The Approach Summarized

The following listing given here for reference is a high-level
representation of the approach to assimilation developed and the
necessary tasks to conduct as presented in this document:

-   First set of tasks---Establish the need and develop relationships

    -   Identify DSS

    -   Form assimilation team

    -   Conduct "zeroth" order, feasibility assessment

    -   Conduct risk analysis

    -   Obtain agreement

-   Second set of tasks---Characterize the current DSS status

    -   Start benchmark process

    -   Form benchmark team

    -   Characterize the DSS current status

    -   Determine DSS objectives

    -   Benchmark the current state

-   Third set of tasks---Identify possible NASA solutions and develop
    enhanced DSS

    -   Plan for DSS enhancement for State 2

    -   Determine needs/requirements for DSS upgrade/enhancement

    -   Prepare requirements document

    -   Brief program manager

    -   Solicit existing results

    -   Determine suitable NASA capabilities

    -   Propose assimilation plans

    -   Develop enhanced DSS---State 2

    -   Conduct verification & validation

    -   Perform DSS demonstrations

    -   Transition/benchmark testing

-   Fourth set of tasks---Measure improvements and transition to
    operations

    -   Benchmark the State 2 DSS

    -   Evaluate metrics

    -   Complete benchmark process

    -   Document results

    -   Transfer responsibility

## Organization of this Document

This Document is organized into five sections, with appendices providing
greater detail in select areas. The first section is a preamble in which
we present an overview of decision making systems and benchmarking
concepts. For decision making systems we draw upon the past 50 years of
literature in areas ranging from management decision making to
artificial intelligence and knowledge management. This section is
intended to provide the context or background for the four sets of tasks
that are the focus of the assimilation activity.

In the second section, we describe the set of tasks (the first set of
tasks listed above) that are intended to develop a working relationship
with the host agency and establish an agreement. These tasks start with
identifying (a) candidates among existing DSSs, or (b) priority
situations where candidate DSSs might soon emerge. Once a DSS has been
identified as a candidate for enhancement using NASA products, the next
tasks are to perform a "zeroth" order, feasibility assessment and to
identify the potential risks and benefits of the proposed assimilation.
Once these have been completed, the results are shared with the Federal
agency partner. The final objective in this section is to reach
agreement on direction and scope and to prepare a formal memoranda
describing the agreement, respective responsibilities, expected outcomes
and levels of commitment.

In the third section, we describe the set of tasks (the second set of
tasks, from above) that are intended to characterize and benchmark the
existing (State 1) or planned (State 0) DSS of the host agency. These
are carried out within the theoretical framework laid out in section
one. Critical at this point is development of a process that is accepted
and trusted by all stakeholders. Equally important is development of
mutually acceptable metrics to quantify the benefits and assess
performance gains realized in upgrading a DSS from State 1 to State 2.

In the fourth section, we describe the conceptual tasks (third set)
involved in the assimilation process itself. A key set of tasks involve
establishing the needs and requirements within the decision-making
environment driving the host agency to consider upgrading the DSS to
assimilate NASA products. A major objective of this section is to
describe the ways in which the NASA science results, data or technology
are "assimilated" into an improved DSS (State 2). Again, as in all
steps, there is participation by all those who have a stake in the DSS
-- NASA centers, agency personnel, universities, and the private sector,
as appropriate.

In the fifth section, we describe the final set of tasks (fourth set) in
the assimilation process. These start with benchmarking State 2 (as in
the third section). With this completed, the next tasks are to compare
the performances of States 1 and 2 and measure differences. The final
set of tasks are to document results and share findings with the partner
agency. At the completion of the assimilation project, the partner
agency assumes operational responsibility for the new DSS and embeds it
within appropriate divisions.

Throughout our discussion of concepts and the processes of benchmarking
and assimilation, we will give examples of how they can be applied in
hypothetical cases. The intent is to explain the value of this approach
in developing an understanding of DSSs and the ways in which their
performance will be assessed before and after the planned NASA
contributions.

# APPROACH

The approach to assimilation and infusion of NASA Earth Science
Enterprise (ESE) science results, data, and information into a decision
support system (DSS) owned by another Federal agency or national
organization is illustrated in Figure 2.

For a given National Application, the goal of preliminary work---left
side of the figure---is to obtain an agreement for an infusion process
between NASA and a DSS owner. This effort should normally be led by the
responsible NASA Headquarters (HQ) program manager with strong support
from the appropriate NASA centers and the assimilation team. For a given
National Application, the goal of the assimilation work is to expand and
accelerate the realization of economic and societal benefits from ESE
results, data and information.

The starting point is to identify a suitable DSS in a Federal agency or
national organization and to identify the individual who will serve as
initial point of contact for this evaluation. The next steps are to
establish an assimilation team and to conduct a zeroth-order,
feasibility assessment of the DSS to determine its current status from
the joint perspective of the agency and NASA, understand its position
within the National Application's decision-making environment, catalog
the decision-making environment's needs it satisfies and to what level,
and to review the owner's plans (if any) for upgrading it. The third
step is to determine the potential value, benefits and risks to both
parties from engaging in an infusion process to upgrade a specific DSS.
This will require identifying the possible ESE contributions, estimating
the enhancement potential of the DSS, and identifying the risks --
direct, indirect, and unintended -- to both parties from an infusion of
ESE results, data and information. The last step in this preliminary
work is to negotiate and agreement with the DSS owner.

From the terms of this agreement, the first two steps are to initiate
the DSS assimilation effort and to start the formal benchmarking
process. Drawing upon the information and work output from preliminary
work, (left side of Figure 2), the next step is to examine the current
state of the DSS. If it is in operation, it is in State 1. If it is in
development and soon to be in operation, it is in State 0. In either
case, the work objectives here are to completely characterize the
present DSS. Next, it is necessary to examine NASA's science models,
data and information products to identify the recommended assimilation
component. The next steps would be to work with the partnering agency to
develop the enhanced DSS with capabilities augmented by NASA input and
to completely characterize the resultant, state 2 DSS. For benchmarking,
it is important to measure the improvements and augmentation obtained by
comparing State 2 with the earlier State 1 (or 0, as the case may be)
apply appropriate metrics, determine the lessons learned, and prepare a
final report and documentation.

# I---FRAMING THE PROBLEM {#iframing-the-problem .Section}

# DECISION-MAKING CONCEPTS

Decision-making has been studied for more than half of the past century
and there exists a rich literature from various disciplines on the
subject. We draw upon that literature in developing a theoretical basis
for classifying and understanding the high priority national application
decision-making systems or environments and their DSSs that we expect to
encounter in this assimilation work. This understanding will provide a
theoretical basis for the benefit/risk assessment undertaken during the
zeroth-order assessment, will identify critical requirements that must
be met, and will guide identifying potential assimilation prospects when
mining the storehouse of the NASA Earth Science Enterprise for models,
data, information products, and technology. A theoretical review of the
decision-making environment is fundamental to the risk/benefit
assessment, key to achieving an agreement between NASA and a partnering
organization by having a shared understanding, crucial to upgrade
planning within an appropriate architecture, and vital in the
operational scenario. In turn, these requirements, risks and
assimilation activities will be captured in an overarching risk
management/analysis effort (see Appendix I).

Figure 3 illustrates a straight-forward A-B-C-D classification scheme
for a decision-making system that will be used to characterize States 1
and 2.

![](media/image1.wmf)

It is this A-B-C-D classification of a decision-making system that will
guide the upgrade assimilation of a DSS to State 2, will guide our
system engineering support, and will help to achieve the greatest degree
of benefit while minimizing risk. To see how this applies, we first
discuss the concept of a decision-making system and then we discuss each
classification element.

## Decision Making Systems

Following Bonczek, Holsapple, and Whinston (1981), we say that decisions
are the "finished products" of a *decision-making system (DMS).* In a
DMS, the user is the decision maker and the computer and processor
constitute an information processing system we call the *decision
support system (DSS)*. Taken together, the DSS and its user constitute a
DMS. Appendix A contains additional information on this topic.

Figure 4 is adapted from Bonczek, et al (1981). The figure illustrates a
generalized model for a decision-making system. From the figure it is
seen that there are five fundamental parts to a generalized DMS.

1.  The User formulates the problem and receives the decision-making
    system output.

2.  The Input Interface is designed to facilitate HCI.

3.  The Problem Processing System (the PPS)---discussed in Appendix
    A---takes an input problem statement and information from the
    Knowledge System (KS) to produce information that supports
    (enhances, or makes possible) a decision process. The PPS has one or
    more of the seven abilities required for decision-making (cf.,
    Appendix A).

4.  The KS contains the body of knowledge available about a problem
    domain. It has the data bases, background (historical) information,
    problem domain knowledge, algorithms, etc.

5.  ![](media/image2.wmf){width="4.375in"
    height="1.5611111111111111in"}The Output Interface allows the user
    to retrieve the processed information in a format that is
    understandable/visualizable

When service of a National Application requires, the DMS activates its
DSSs to obtain necessary information. The available DSSs use the problem
domain information contained in the KS(s) and cause the PPS(s) to
process appropriate models and algorithms to produce outcome decision
support information. These outcomes are passed to the DMS. The decision
maker considers the information provided and makes an appropriate
decision.

![](media/image3.wmf){width="4.75in" height="2.470138888888889in"}

The KS is where NASA will focus its assimilation efforts. Upgrading the
PPS of a particular DSS and the DMS in general is most likely where the
national application partner will focus its efforts. Note, then, that to
provide DSS enhancement NASA's assimilation result must function within
the partnering agency's DMS. This burdens the NASA effort with
understanding the DMS; its needs and requirements, the elements listed
as Figure 3 (its framework, its knowledge type and management paradigm,
its decision-making typology), and its operating environment. The
elements listed in Figure 3 (above) are briefly discussed in the
following sections.

## Framework

Gorry and Scott Morton (1971) produced a "framework" or matrix
describing management decision-making activities which we adapt---see
Table 1---for use in this DSS work. Degree of problem complexity is
listed across the top and degree of structure is along the left. Cell
values are particular decision types. The degree of problem complexity
ranges from modeled (already done) to novel (may not be amenable to
modeling). The more structured a problem, the more amenable it is to
solution by computerized algorithms in a DSS. The simplest case would be
the Structured/Modeled Problem noted as routine. Conversely, the more
unstructured a problem, the more a decision must rely on judgment and
intuition outside the DSS. Thus, the Unstructured/Novel Problem is noted
as requiring custom tailoring for solution. A semi-structured problem
has structured and unstructured phases or elements. (Note that it is the
decision *context* that is unstructured, not the DSS itself. Appendix B
contains additional information on this topic.) Clearly, as the decision
making system is developed, the intent is to make it less idiosyncratic
and more consistent -- to move decisions from the bottom right toward
the upper left of the framework.

Table 1. A decision support framework

  ------------------ ------------------ ----------------- -----------------
                     Modeled Problems   Complex Problems  Novel Problems

  Structured         1\. Routine        2                 3

  Semistructured     4                  5                 6

  Unstructured       7                  8                 9\. Custom
                                                          tailored
  ------------------ ------------------ ----------------- -----------------

## Types of Knowledge

Several different types of knowledge are recognized in the literature:
explicit vs. tacit, procedural vs. declarative, esoteric vs. exoteric,
and shallow vs. deep. The pairings of data types make up Table 2,
included here merely to show the pairs side-by-side. Each represents a
different management challenge. Appendix C contains more information on
this topic.

## Knowledge Management Paradigms

Schultze (in press) describes three different perspectives on knowledge
management: the functional, the interpretive, and the critical. Appendix
D contains additional information.

As applied in this assimilation work, these are defined as:

-   ***Functional paradigm*** is that most often adopted by those in
    practice, especially in the software industry.

-   ***Interpretive paradigm*** consists of fostering communications
    between individuals, sharing and enriching interpretations, and
    coordinating actions.

-   ***Critical paradigm*** is concerned primarily with social conflict
    and antagonistic relationship amongst various stakeholders and
    special interest groups.

## Classification

Over the past 50 years, decision-making and decision-making environments
have been extensively studied. In the classical literature, there are
*five types of decision making systems* or inquiring organizations:
*Leibnizian (Type 1), Lockean (Type 2), Hegelian (Type 3), Kantian (Type
4), and Singerian (Type 5)*, each based on the philosophies of their
respective namesakes. In this work we will adopt these definitions for
characterizing DSSs but will use the more generic labels, *Type 1 --
Type 5*. Table 3 summarizes them. Additional discussion about these
classifications is provided in Appendix E.

Throu![](media/image4.wmf){width="4.0625984251968505in"
height="1.0515748031496064in"}ghout the assimilation work, these five
types of decision-making systems will be defined as:

> **Type 1---Analytic -- Deductive Approach---**creates knowledge by
> using *formal logic and mathematical analysis* to make inferences
> about cause-and-effect relationships, and maintains that everything
> needed is already contained within its boundaries. As closed systems,
> they have access only to knowledge generated internally.
>
> **Type 2---Consensual -- Inductive Approach---**creates knowledge from
> empirical information gathered from external observations and used
> inductively to build a representation of the world with a set of
> labels (or properties) assigned to the observations. The decision
> style is clearly group-oriented and open.
>
> **Type 3---Empirical -- Theoretical Approach---**recognizes that there
> may be *many different perspectives on a problem*, or at least many
> different ways of modeling it. The perspectives are all, however,
> *based on the belief that problems can be modeled analytically*. It
> finally *chooses the model which best explains the data*.
>
> **Type 4---Conflict -- Synthesis Approach---**is based on the belief
> that the most effective way to create knowledge is by observing a
> debate between two diametrically opposed viewpoints about a topic, a
> *thesis* and an *antithesis,* from which a *synthesis* is constructed
> as the worldview.
>
> **Type 5---Multiple Perspectives -- Holistic Approach**---views the
> world as a holistic system, in which everything is connected to
> everything else. Solving complex problems may require knowledge from
> any source and knowledgeable people in any discipline or profession.
> The multiple perspectives approach does not end with the technical,
> organizational, and personal perspectives. It also explicitly brings
> *ethics and aesthetics* into play.

## Example Classifications

Figure 6 illustrates a possible State 1 classification of a DSS. The
problem framework is noted to be a semi-structured/complex problem
\[5\]. State 1 is noted to require declarative, procedural, esoteric,
and deep knowledge \[3,4,5,7\] in a functional knowledge management
paradigm \[1\]. Finally, the classification tells us that the
decision-making environment is both consensual and empirical \[2,3\].

![](media/image5.wmf)

Figure 7 illustrates the classification scheme that might result from
upgrading this same DSS to an enhanced State 2. Note that the problem
framework changes to a semi-structured/modeled and
semi-structured/complex problem \[4,5\]. This suggests that part of the
problem has been constrained in degree of complexity---part remains
complex---thereby making a part of the resultant DSS function at a more
routine level and thus be more amenable to algorithmic solution. Note
also that the types of knowledge required have been reduced to just
declarative and deep \[3,7\], still in a functional knowledge management
paradigm \[1\]. Eliminating the need for procedural knowledge gets rid
of the need to deal with the "how to" do something while eliminating the
need for esoteric knowledge removes the requirement for knowledge
applicable only to a narrow domain. State 2 has been upgraded to be a
straight-forward empirical decision-making system \[3\]. These changes
are significant. They suggest that the assimilation work resulting in an
enhanced State 2 have improved the DSS to be much more amenable to
computerized algorithm/model solution. This has a number of important
ramifications, among which are the facts that it offers the prospects
of:

1.  enhanced repeatability of results,

2.  increased accuracy,

3.  increased throughput speed,

4.  reduced costs, and

5.  increased socio-economic benefit.

![](media/image6.wmf)

The intent in the development of improved decision making systems is to
make them more structured, reliable and accurate. Part of this can be
achieved with better data. But part of it arises from a better
understanding of the part of nature that is being managed or predicted.
Thus, the continuous unfolding of science allows us to understand the
processes that determine how systems operate and help constrain the
"problem space." The example of seasonal climate forecasting is a good
example of this evolution in which, initially, forecasts were based on
past patterns, hunches, and obscure observations of questionable value
(e.g., the behavior of certain plants and animals noted in *The Farmers'
Almanac*). Now, forecasts are based on process-based physical models
that are built on sound theory, that are driven by a web of observations
made from the ground, from satellites and buoys that describe
temperature, vertical structure, winds, circulation of the atmosphere
and ocean, and topography of the sea surface.

# BENCHMARKING CONCEPTS

## Decision Support Systems (DSSs) and Benchmarking concepts 

At its most basic, benchmarking is a management tool. It has four
essential facets---planning, development, synthesis, and
enhancement---that follow one another in succession with essential
documentation and reporting at each stage. As it applies here,
benchmarking will be used to assess the performance improvement, or
enhancement of a DSS compared to a baseline set of indicators. In the
normal case, we will be benchmarking the enhancements realized in a DSS
operating at State 2 compared against the baseline performance of either
State 0 or State 1, as appropriate. A complete listing of benchmarking
activities is contained in Appendix F. Some candidate metrics are
contained in Appendix G.

Benchmarking gives structure to the process by which we enhance a DSS
and upgrade it to function at State 2. It involves identifying concerns,
problems and opportunities; assessing or measuring performance; drawing
conclusions based on the information and findings; and stimulating DSS
changes and improvements (Andersen and Pettersen, 1996). Benchmarking
can be applied to components of a DSS, to a complete DSS, or to finished
products of a DSS as well, throughout the assimilation lifecycle.

Benchmarking involves teams, teams of experts, facilitators, and
organizations. These teams are required to provide information,
guidance, and resources; to provide information about the
state-of-the-art at which a DSS is to function; and to provide the
underlying science and data output expectations of the decision-making
environment it serves. The benchmarking process will require a level of
cooperation and trust (See Appendix H) that sets this activity apart
from other information gathering activities (such as pilot studies, for
example). After completing a benchmarking activity, results may be
presented in the form of recommendations or may require changing the
system. Another benchmarking process loop can be initiated to enhance a
DSS with the assimilation of new products.

## Types of benchmarking activities

One way of categorizing benchmarking activities is to look at *what* is
being compared. In this case, three types of benchmarking can be
defined:

-   Performance benchmarking---is the comparison of performance metrics
    to determine the improvements represented by State 2 over the
    baseline DSS performance,

-   Process benchmarking---is the comparison of methods and practices
    for a DSS operating at State 2 and the value of the outcomes for
    servicing the decision-making environment compared with the baseline
    set,

-   Strategic benchmarking---is the comparison of the improved impacts
    provided by a DSS at State 2 compared with the baseline
    achievements.

Another way of categorizing benchmarking activities is to look at *what*
a DSS is *compared against*. In this case, four types of benchmarking
can be defined:

-   Internal Benchmarking---is the comparison in which one overarching
    organization compares its DSS(s) with similar activities in
    different locations, departments, operating units, and states,

-   Competitive Benchmarking---is the comparison of performance between
    the DSSs of direct competitors providing products (i.e. DSS) to the
    same decision makers or customers,

-   Functional benchmarking---is the comparison of processes and
    functions against non-competitor organizations within the same
    industry or technological area.

-   Generic benchmarking---is the comparison of one's own processes or
    DSSs against organizations with the state-of-the-art DSSs, products
    and services.

## The Benchmarking Process

Observe from Figure 8 that for DSS benchmarking, the basic phases of the
process are:

-   **Identify and select** the benchmark partners (e.g. NASA and USDA)
    and the process or DSS to be benchmarked,

-   **Characterize, assess and observe** the DSS and the associated
    benchmarking partners, and set up a benchmarking team,

-   **Assimilate** data, science or technology **and analyze** the
    causes for the performance differences between the baseline
    (State 1) and enhanced DSS (State 2)

-   **Adapt** and implement improvements and changes based on the
    benchmarking analysis **and make it operational**

As is apparent from the figure, benchmarking is a tool providing process
direction, control, oversight, and documentation, from start to finish.
Referring back to Figure 2, the benchmarking process is initiated after
an agreement has been reached between NASA and a DSS-owner and it
encapsulates all work from that point until a DSS has been upgraded to
function at State 2 and is considered to be operational. Thus, the top
box -- 1 -- of Figure 8 is the *Identify and Select* part of
benchmarking. At this point, the DSS is selected and a preliminary,
zeroth-order assessment will be completed, an Agreement has been
enacted, and the terms of the Agreement establish the broad outlines of
what is to be done, who is to do what, and what resources will be
provided. The top box is concerned with identifying and selecting the
benchmarking partners, selecting the process to be
followed---specifically, the type(s) of benchmarking to be
employed---for assimilation, and initiating the oversight and
documentation requirements. This is, thus, *planning* and that leads to
the second box.

The second box -- 2 -- of Figure 8 is the *Characterize, Assess and
Observe* part of benchmarking. It is involved with the work of
characterizing a DSS, the Decision-Making System in which it functions,
determining the requirements that the system fulfills and the outcomes
it offers the decision-maker(s). Specifically, it is here at which the
baseline status of a DSS (State 0 or State 1) is formally and completely
characterized and the requirements for State 2 to fulfill, its expected
outcomes are formally identified,
![](media/image7.png){width="5.990944881889764in"
height="4.509055118110236in"}plans for upgrading the DSS are
established, and the design for obtaining State 2 is developed. A
benchmarking team, representing the benchmarking partners, is formed to
develop and implement performance metrics for the DSS. Thus, the
*development* of plans for creating a State 2 DSS leads to the third
box.

The third box -- 3 -- of the figure is the *Assimilate and Analyze* part
of benchmarking. It involves measuring the performance of both the
baseline DSS and its upgrade to State 2, comparing performance
indicators, evaluating the results obtained, collecting metrics, and
running a risk-benefit assessment. On completing this effort, then the
process is leads through *synthesis* to the fourth box.

The fourth box -- 4 -- of the figure is the *Adapt and Operationalize*
part of benchmarking. It is the point at which a system is either
certified for operational status or the process is re-initiated for
additional enhancements. If the system meets standards and is certified
for operational status, the benchmarking process will be completed when
the documentation formally required have been provided. If it is decided
to re-initiate the assimilation process for additional improvements in
the DSS, then the benchmarking process rolls through *enhancement* back
to box 1.

At all phases of benchmarking, there are formal requirements established
for documentation. At a minimum, it is expected that a report will be
prepared documenting the work of each phase and a final report will
encapsulate the entire effort. In addition, it is recommended that the
following reports be completed as part of this process:

-   Lessons learned,

-   Implementation plan,

-   Risk-benefit analyses (multiple stages),

-   Metrics, and

-   Performance indicators.

The benchmarking process can be repeated and recalibrated when new
enhancements in the form of data, models or technology become available.
The benchmarking process itself can then also be improved based on
experiences from completed benchmarking studies. A listing of all the
activities and topics associated with these four phases of the
benchmarking process are listed in Appendix F. The topics and activities
listed for each of the four phases are all important steps to accomplish
the benchmarking of a DSS. However, some activities may not apply for a
particular DSS, or the steps may be executed in a different
chronological order as circumstances require.

# II---ESTABLISHING AGENCY RELATIONSHIPS {#iiestablishing-agency-relationships .Section}

Figure 9 adds important information to the over-arching diagram of
Figure 2. As was noted in the APPROACH for a given National Application,
the goal on the left side of the figure is successful negotiation of an
agreement for an infusion process with a partnering Federal agency DSS
owner. This effort will typically be led by the responsible NASA HQ
program manager with strong support from the appropriate NASA centers
and the rest of the assimilation team. It is color-coded red to signify
that this effort is a necessary preliminary step and to emphasize that
it is NASA responsibility.

Note from the figure that this effort starts with identification of a
DSS and creation of an assimilation team, requires a zero-order
assessment of the DSS, continues on with a risk analysis, and is
completed with successful negotiation of an agreement. Note further that
at each step in the process information obtained is passed to the right
in order to baseline the work to be conducted there. In the following,
we discuss each step of this process.

## Identify DSS

We note that Federal agencies aligned with each of the 12 National
Applications have been identified, and one or more candidate DSSs have
been tentatively identified. In the general case, however, the starting
point for this evaluation is identification of both a suitable DSS in a
Federal agency or national organization and the individual who will
serve as initial point of contact. There are several key considerations
involved in identifying candidate DSSs:

-   **Environment---**a DSS is expected to be implemented in an
    automated, digital environment. While it is understood that there
    are likely to be components of a DMS that are not automated, there
    should be significant components that are,

```{=html}
<!-- -->
```
-   **Applicability---**a DSS should be amenable to infusion of NASA
    science results, data, or technology. In this, they must key, to
    some degree, on the general outputs of EOS**---**weather, climate
    and natural hazards predictions**---**and/or the data and technology
    that support them. Some linkages may be quite direct, such as the
    use of NASA data and models for predicting hurricane landfall in
    order to develop evacuation plans. Others may be less direct, such
    as the use of seasonal El Niño forecasts to predict Hanta virus
    outbreaks, and

-   **Development plans---**the partner agency should have a development
    plan for the DSS. This is not a strict requirement, but it is
    helpful in illustrating the commitment of the agency both to the
    system itself, and to the notion of system evolution. Not
    infrequently, if a plan is in place and all the preceding
    requirements are met, there is already a tacit agreement with NASA
    to contribute to system upgrade.

The final steps for the NASA program manager are to identify a
point-of-contact (POC) within the partner agency and to form an
Assimilation Team.. While there should be an administrative POC to
ensure agency participation, there must also be a technical POC who is
involved in the development, management and operation of the DSS.

## Form assimilation team

When preliminary work progresses to the point at which a zeroth-order
assessment is needed, it is suggested that the NASA program manager
create an Assimilation Team. The program manager will identify the
individuals, the roles, and the responsibilities of the various team
members and their organizations. The team may be drawn from the
technical staff of the partner agency, appropriate NASA Center(s),
appropriate university PIs, and Stennis Space Center (SSC), Earth
Science Applications Directorate (ESAD). The assimilation team is
expected to be essentially policy oriented, but it should have
sufficient technical strength that a zeroth-order assessment can be
conducted.

The purposes of the assimilation team are to 1) assist the program
manager in assessing the political environment and policy setting of the
DMS within which the DSS functions, 2) perform zeroth-order technical
assessment of the DSS, 3) assist in conducting a risk assessment and
analyze the benefits potentially arising from an assimilation of NASA
inputs, and to 4) help in defining the terms and conditions of an
agreement between NASA and the DSS-owner. Broad participation on the
benchmarking team is necessary to ensure that credible results are
achieved in a manner that is acceptable to all parties.

Upon being established, among other tasks, the benchmark team will:

-   document the role it will play throughout the assimilation process,

```{=html}
<!-- -->
```
-   develop the documentation and reporting protocols and requirements,

```{=html}
<!-- -->
```
-   determine and agree upon the metrics and performance objectives that
    will underpin the zeroth-order assessment and risk/benefit analysis,

-   establish the basis for determining the acceptability of NASA
    technology, data, and information products,

-   determine the criteria for deciding when the assimilation effort is
    completed,

The next activity, following this, will be conducting a zeroth-order DSS
assessment.

## Conduct zeroth-order assessment

The next step is to conduct a zeroth-order, feasibility assessment of
the DSS to determine its current status from the joint perspective of
the agency and NASA, understand its position within the National
Application's decision-making environment, catalog the decision-making
environment's needs and level it satisfies, and to review the owner's
plans (if any) for upgrading it. The fruits of this assessment feed the
risk/benefit analysis and thus underpin the basis for reaching agreement
to engage in the assimilation process.

The zeroth-order assessment is a relatively rapid appraisal of the
decision-making environment served by the candidate DSS, its top-level
information needs and requirements, and the current operating state and
characteristics of the candidate DSS that can be discovered prior to
having an agreement for cooperation written out. It includes a review of
the partner's upgrade plans for the DSS, and a determination of the
practicality of infusing NASA results, data and information into it, the
potential for improving its output, and the probable value of this
enhancement.

The zero-order assessment will form the basis of the dialog between NASA
and the partner agency, and should constitute the core of whatever
agreement is ultimately signed. In effect, the zeroth-order assessment
lays the groundwork for assimilation and will act as a framework
proposal to Agency personnel, laying out the basics of the assimilation
activity to be undertaken.

## Conduct risk analysis

The third step is to determine the potential value, benefits and risks
to both parties from engaging in an infusion process to upgrade a
specific DSS. The analysis will start with the results from the
zeroth-order feasibility assessment, will require identifying the
possible ESE contributions, will involve estimating the enhancement
potential of the DSS, and will estimate the risks, both direct and
unintended, to both parties from an infusion of ESE results, data and
information. It is this analysis wherein the value of assimilation can
be documented and it is expected that the results of conducting this
risk analysis will form the basis for an agreement.

This is a critical and structured process. Detailed descriptions of the
concepts, methods, and tools that will be used for a risk/benefit
analysis at this preliminary state and throughout the benchmarking
process are found in Appendix I.

## Obtain agreement

The last step in this preliminary work is to negotiate an agreement with
the DSS owner.

The purpose of the agreement is to formalize the joint activity between
NASA and the partner agency. It is also intended to describe the general
path to be pursued, as well as to document the intentions and
expectations of each of the partners. The zeroth-order assessment, the
risk analysis, and benefit assessment will provide the information
required to open a full dialog with the partner agency on the bounds of
the agreement with NASA.

Details of the agreement should be developed by the appropriate NASA
Program Manager and his/her counterpart in the cooperating agency. Given
that the agreement will involve the commitment of agency resources, it
should be signed by administration at the next higher level, at least.
The agreement should have six elements, at minimum.

-   **Agreement in principle---**first, and most obviously, the
    agreement should state the willingness of the two parties to
    cooperate in the enhancement of the identified DSS,

```{=html}
<!-- -->
```
-   **Objectives---**a clear description of the overall objectives of
    the assimilation activity (e.g., reduction in time and cost) should
    be stated,

-   **Metrics---**a set of more specific criteria (i.e., metrics)
    derived from the objectives should be included to establish how the
    assimilation activity will be assessed,

-   **Approach---**a concise description of the activities to be
    undertaken should be included, including an explicit requirement
    that NASA science results, data, or technology will be used,.

-   **Personnel---**a listing and description of the roles of key, or
    essential personnel expected to be involved in each of the
    activities should be included, and

-   **Commitments---**finally, a brief description of the levels of
    resources that might be committed by each partner to the general
    activity should be included.

If there are other concerns that emerge in the negotiation, they should
be noted in the agreement that is reached.

# III---CHARACTERIZING CURRENT DSS {#iiicharacterizing-current-dss .Section}

The right-hand side of Figure 9 (cf., Section II) outlines the
assimilation process in an over-arching diagram of the work to be done.
The work of the right-hand-side starts with the agreement between
agencies negotiated through the preliminary work illustrated as the
left-hand side of the figure. From the terms of the agreement reached,
the first two activities shown start the assimilation work and launch a
benchmarking process within which the work is expected to be
encapsulated. The third block, drawing from the information previously
obtained (from the left-hand side efforts), requires complete
characterization of the current state of the DSS in order to baseline
its operational status. If it is in operation, it is in State 1. If it
is in development and soon to be in operation, it is in State 0. In
either case, the work objectives here are to completely characterize the
present DSS. Next, it is necessary to examine NASA's science models,
data and information products to identify the recommended assimilation
components.

The next steps are to work with the partnering agency to develop an
enhanced DSS with capabilities augmented by NASA input and to completely
characterize the resultant, state 2 DSS. For benchmarking, it is
essential to measure the improvements and augmentation obtained by
comparing State 2 with the baseline State 1 (or 0, as the case may be),
to apply appropriate metrics, to determine the lessons learned, and to
prepare a final report and documentation

The process ends with an enhanced DSS completely characterized,
installed, and in operation with operational performance improvements
measured and, as required by the benchmarking process, with performance
enhancements documented and reports prepared. These are all discussed in
greater detail in the following.

# Start benchmarking process

It is assumed that the agreement reached between agencies is broad,
setting forth top-level conditions, only. Details are worked out as the
assimilation activity proceeds. From the terms of the agreement, goals,
objectives, scope, and resources all need to be determined. In addition,
the issues of how to deal with disengagement of NASA at the end of the
assimilation period, how to deal with the transferability of NASA
assimilation/input, what the long term relationship between agencies is
to be, and who will be responsible for long term operational
sustainability need to be resolved.

The terms of the agreement reached will set forth broad goals to be
achieved. NASA and the partnering agency assimilation team will agree
upon the objectives to be completed in pursuit of those goals and will
establish a schedule with key milestones to do that. The assimilation
team will also agree upon how the technology, data, and information
products are transferred for use in the operational environment and how
the enhanced, operational DSS and its needs for continuation of NASA
input will be sustained. A budget will be prepared and resources made
available. The work will be initiated and the benchmarking process will
be started.

As is apparent from Figure 8, benchmarking is a tool for process
direction, control, oversight, and documentation, from start to finish.
It is critical to success of the assimilation process. It provides a
formal means by which system changes**---**improvements**---**between
States 0/1 and 2 are measured and ultimately evaluated.

In starting the benchmarking process, two essential efforts are
required:

-   creating and establishing the benchmark team, and

```{=html}
<!-- -->
```
-   determining the various criteria and metrics to measure achievement
    and success.

## Form benchmark team

As noted above and documented in Appendices F, G and H, benchmarking
must be a team effort. The team will be drawn from the technical staff
of the partner agency, appropriate NASA Center(s), appropriate
university PIs, and SSC, ESAD. With special responsibility to support
the Applications Division, ESAD will take a leadership responsibility in
verification and validation.

The assimilation team will establish the benchmark team(s). It will
identify the individuals, the roles, and the responsibilities of the
various team members and their organizations. The benchmark team(s) are
expected to include NASA, partnering agency, university, and contractor
personnel, as appropriate. The assimilation and benchmark teams have
different functions---the assimilation team is policy oriented while the
benchmark team is technically focused---but may be include the same
people. In the most common case, it is expected that the benchmark team
will contain some members of the assimilation team together with some
individuals added specifically for their technical expertise.

The purpose of the benchmarking team is to document performance of the
DSS in its initial State (0 or 1), and its improved State 2 in which
NASA science results, data and/or technology have been infused or
assimilated. Broad participation on the benchmarking team is necessary
to ensure that results are credible, but also achieved in a manner that
is transparent to all parties.

Upon being established, among other tasks, the benchmark team will:

-   document the role it will play throughout the assimilation process,

-   develop the documentation and reporting protocols and requirements,

-   determine and agree upon the metrics and performance objectives that
    will underpin measuring the change from baseline to enhanced
    operational status,

-   establish the basis for determining the acceptability of NASA
    technology, data, and information products,

-   determine the criteria for deciding when the assimilation effort is
    completed,

The next activity will be to formally and completely characterize the
DSS to be upgraded. This will involve several disparate efforts.

# Characterize the DSS

## Characterize DMS environment

As previously noted (cf., Section I), decisions are the "finished
products" of a *decision-making system (DMS).* In a DMS, the user is the
decision maker and his/her problem specific, computer implemented tools
constitute an information processing system we call the *decision
support system (DSS)*. The DMS may be composed of one or many DSSs.

Characterization of the DMS environment will include elucidation of the
organizational decision making framework, its philosophical type
(Kantian, Singerian, etc.), its problem solving approach, knowledge
management paradigm, types of knowledge used, and the degree of decision
context structure exhibited (cf., Figure 3, Section I).

The characterization component of the first-order assessment is intended
to avoid any potential cross-cultural pitfalls that may arise between
NASA and its partner Agencies due to unawareness of their respective
characteristics, and to provide useful qualitative information to the
benchmarking and evaluation processes. For example, it will be important
for evaluation to know whether DSS enhancement increases the structure
of the decision context, perhaps by replacing some tacit or intuitive
knowledge formerly used by decision makers with newly available explicit
knowledge about factors influencing decisions.

While conflicts of culture are not likely to be a stumbling block with
adequate awareness, it is conceivable that ignorance of the fundamental
personality characteristics of an Agency decision making group could
undermine the effectiveness of DSS enhancement, or make some efforts
futile. For instance, if all parties were not aware that a particular
DSS was housed in a Type 2 organization using a consensual-inductive
approach to problem solving, and pursued DSS "enhancements" more suited
to an empirical-theoretical problem solving approach, much effort might
produce indifferent results.

## Determine DSS objectives

Although it will be outlined in the agreement between NASA and the
partner agency and should be formalized by the assimilation/benchmarking
teams, the benchmarking activity must begin with a clear statement of
the objectives the DSS is intended to serve. It is not unusual for any
decision making system to be subject to "mission creep" or drift over
time. In such a case, the original objective(s) the DSS is intended to
serve may be institutionalized in a mission statement, but with time it
may be asked to serve new, important functions for which it was not
originally intended and which are very real but now unstated. Thus, at
the outset, it is necessary to come to a common and explicit statement
regarding the current objectives; and especially to know how those
objectives are expected to change if the DSS is upgraded.

There are several key considerations involved in understanding the
decision-support objectives of a candidate DSS:

-   **Classification---**a DSS functions within a DMS. Classify the
    environment of the DSS via the variables presented in Figure 3 (cf.,
    Setion I) to baseline the decision-support environment and to
    establish the relationship with its DMS,

```{=html}
<!-- -->
```
-   **Environment---**a DSS is expected to be implemented in an
    automated, digital environment or, at least, to be an automated,
    digital information processing system, itself,

-   **Information Products---**a DSS is expected to produce information
    products for the decision-maker. What kind of products are produced
    and how are they used, and

-   **Operational plans---**the partner agency should have an
    operational plan for the DSS and it would be very useful if the
    agency also had identified the requirements to be met in upgrading
    the system to function at State 2.

## State 0 or State 1

Subsequent to the zeroth-order assessment described in Section II, a
more thorough and detailed assessment of the selected DSS of interest
and its institutional context will be carried out to provide a basis for
assimilation activities. This may be thought of as a first-order
assessment of the current, existing state of the DSS before enhancement,
and includes a characterization of the overall decision making system
and environment in which the DSS functions. A primary component of the
first-order assessment is consideration of whether the current DSS is in
State 1, functionally operational, or State 0, in the planning and
prototype stages of development, and what implications this has for the
assimilation process.

It is not required that a DSS be operational for it to be a candidate
for assimilation. In fact, flexibility and the opportunity for
introducing new science and data products may be greater for DSSs in the
planning or prototype stages than for established DSSs. Being aware of
the subject's developmental state will help clarify the opportunities
and constraints presented.

Also bearing on the choices made throughout the assimilation process is
an assessment of the complete decision making environment, to gain
awareness of the cultural type of inquiring organization involved, its
problem solving approach, the types of knowledge employed, and other
factors described in Section I and Appendices A through E.

### State 1

If the DSS proposed for enhancement is already functioning in an
operational context and being used by decision makers in their work, it
is said to be in State 1, the baseline condition. Actual performance of
the DSS should be measured for benchmarking purposes, and its behavior
observed as part of an operational DMS. The two basic implications of a
State 1 DSS are that the actual operational performance characteristics
can be observed, measured, and codified, and that it will be necessary
to measure these characteristics without disrupting ongoing activities.

### State 0

A State 0 DSS may be an acceptable candidate for assimilation if it
shows high potential for emerging as a critical tool for use in a
priority National Application. A DSS in an initial State 0 condition is
not yet fully developed for operational use. It may be only in the
planning stages, or it may only exist as a prototype. Working with a
State 0 DSS implies that assimilation may be more uncertain or nebulous
at first, without fully knowing what the actual behavior and performance
of the proposed system will be. However, the State 0 DSS may offer more
opportunity to satisfy the requirements of the ultimate DMS, the
environment it serves within, and the decision makers' information needs
by early NASA involvement in the DSS design process.

## Benchmark the current state 

The DSS itself, as part of the current decision making system, will be
characterized component-by-component and this performance assessment
will be recorded as a formal part of the benchmarking process. Relevant
characteristics of a DSS to examine are included the following example.
It must be noted that these are only offered for illustration. Each DSS
is likely to be unique and have different critical features that must be
accounted for but may not be included here.

-   ***Current inputs***

Knowing the current science models and data inputs used by the DSS will
establish current input characteristics and will directly suggest which
NASA models or data products might be used for enhancement, or show
needs that may be filled by assimilation of new inputs. Specific
characteristics of current input material should be evaluated, such as:

-   data types and sources

-   data formats

-   spatial resolution

-   temporal resolution

-   spectral resolution

-   spatial and temporal coverage

-   data cost

-   availability, timeliness

-   volume of data to process

-   accuracy -- locational, numerical, thematic

```{=html}
<!-- -->
```
-   Current functions, internal algorithms, processing steps, data flows

Knowing the structure and function of all of the elements of a
particular DSS will be necessary to baseline its performance
characteristics. Knowing this will also suggest enhancement and system's
engineering opportunities. Typically, this will lead to optimizing and
other modifications of software. Upgrade enhancements and performance
optimization can be oriented around particular modules of code.

-   ***Hardware and software used***

A complete characterization of the hard- and software environment will
be performed. The intent is to understand the performance
characteristics observed and to identify bottlenecks in the DSS
information processing system. It will also be possible to identify
opportunities and constraints that the current environment might pose
for any specific improvements.

-   ***People involved, levels of training required, specialized
    skills***

As with hard- and software, it is necessary to understand the nature of
the workforce that is charged with operating the DSS. The profile of the
workforce will also provide opportunities and constraints for any
specific improvements.

-   ***Current outputs***

The output of a DSS is advice to a human decision maker. This advice may
be a single coherent, "boiled down" prediction or recommendation, or it
may be a set of discrete pieces of information to be evaluated and
combined with other data and information in the decision process.
Outputs may be verbal, numerical, graphical or pictorial, or a
combination. In any case, for purposes of benchmarking and assimilation
specific characteristics of the output(s) will be observed, analogous to
those of current inputs:

-   output data types

-   data formats

-   spatial resolution

-   temporal resolution

-   spectral resolution

-   spatial and temporal coverage

-   production cost

-   availability, timeliness

-   volume of output to interpret

-   accuracy -- location, numeric, thematic

```{=html}
<!-- -->
```
-   ***Current performance***

Performance, of course, is the ultimate level of benchmarking. There is
a wide range of benchmarking measures, some are described below and
others are suggested in Appendix G. These are only intended to serve as
examples. The ultimate performance metrics will be established by the
assimilation/benchmark team(s).

-   range of services provided

-   processing speed

-   processing costs

-   trained people required

-   timeliness of products

-   accuracy of products -- location, numeric, thematic

-   bottlenecks

# IV---ASSIMILATING NASA PRODUCTS {#ivassimilating-nasa-products .Section}

# Plan for DSS enhancement for State 2

Referring to Figure 9, the central box on the right-hand-side is
identified as *Determine NASA assimilation components.* That is the work
at hand---finding appropriate NASA inputs, including data, science
models, and/or technology to offer for DSS enhancement. Figure 10
illustrates one way to do this. The final products of Figure 9 are
potential NASA results to offer for a DSS upgrade development plan.

As described in the figure, the process starts with the original
agreement, the baseline assessment of the current DSS, and with existing
owner agency plans for upgrading to State 2. The next steps are to
identify the DMS/DSS requirements underlying the upgrade plans and to
prepare a requirements document. Next, the assimilation/benchmark team
will brief the NASA program manager of the DSS needs and begin the
search for existing NASA results to offer the upgrade/enhancement
process. The program manager will solicit results---either through a
workshop or some other mechanism---from the various components of NASA.
Potential solutions, upon being identified, will be made available to
the assimilation/benchmark team for incorporation in an enhanced State 2
development plan. This output product will become input to the box
entitled *Develop enhanced DSS* in Figure 9. It should be noted that
while the process of Figure 10 is serial, it may be more efficient for
specific DSSs to work several issues in parallel via a joint workshop
arrangement.

## Determine needs/requirements for DSS upgrade/enhancement

The foundations for developing requirements will be outlined in the
agreement that is negotiated between NASA and the host agency. The
objectives and metrics that are contained in that agreement will form
the basis for developing a requirements document.

A key point of assimilation is that of knowing the DMS driving
requirements for upgrading a DSS to operate at State 2. Why is it
desirable, or necessary to upgrade?

The first step is to determine this. There exist numerous ways for
obtaining this information. Among these are the following three,
provided here in the attempt to cover the spectrum of possibilities
(variations on these themes will certainly occur):

-   The DSS owner/user agency may have well-established plans for
    upgrading the DSS to State 2. The underlying DMS requirements and
    needs that are driving the upgrade will probably be documented. This
    documentation should be available to the assimilation/benchmark
    team,

```{=html}
<!-- -->
```
-   The DSS owner/user agency may have informal plans for upgrading. The
    underlying requirements/needs may be anecdotal and may or may not be
    documented. The anecdotal evidence should be documented by the DSS
    owner/user agency and that documentation made available to the team.

-   The DSS owner/user agency may be interested in upgrading only if
    NASA offers significant additional capabilities. The
    requirements/needs in this case may be completely ad hoc, based
    solely on the potential augmentation offered by NASA inputs. The ad
    hoc evidence should be documented by the agency and made available
    to the team.

After obtaining access to the requirements/needs driving the upgrade
from a DMS perspective, the assimilation/benchmark team should review
the documentation and categorize the needs by engineering and scientific
themes. Engineering themes will be those which improve throughput,
reduce cost, increase capacity, etc. Scientific themes will be those
which augment capability, introduce prediction, increase the period of
forecasting, reduce uncertainty, improve calibration, etc.

Depending upon the scope of requirements/needs identified, it may be
useful to conduct a joint workshop involving appropriate representative
of the DSS/DMS agencies, downstream users, representatives of the
scientific community, NASA scientists and engineers, etc., to stimulate
possibilities for assimilation of NASA capabilities. The goal is to
identify the best set of requirements/needs possible and there are
numerous ways to achieve this, with the workshop idea being one.

## Prepare requirements document

The bulwark of systems engineering is the requirements/needs document.
It is recommended that a formal requirements/needs document be prepared
by the assimilation/benchmark team.

## Brief program manager

As a serial progression of events, the assimilation/benchmark team
should next brief all findings to the appropriate NASA program manager
and present the requirements/needs document as evidence. In most cases,
the program manager will be involved throughout the assimilation process
and this step will be a formality. It is nevertheless an important step
because it documents this set of requirements/needs for which NASA
capabilities will be sought.

## Solicit existing results

The goal here is to uncover both engineering and scientific
capabilities---data, science models, technology, systems
engineering---for NASA to *consider* for the State 2 upgrade. The
program manager, with the assimilation/benchmark team will establish a
process for casting a net NASA-wide to find assimilation augmentation
possibilities. Again, numerous ways exist for doing this, one of which
is to hold a joint workshop with NASA representatives identified by
review of the requirements/needs documentation. Attendees at this
workshop will probably involve in addition to appropriate NASA
personnel, representatives of the DSS/DMS agency, downstream users and
potential users, and representatives of the scientific community.

## Determine suitable NASA capabilities

It is here where NASA focuses on specific solutions to offer the
assimilation process. The assimilation/benchmark team, together with
others identified by the program manager should review the possibilities
identified so far for their suitability to meet the requirements/needs
in an operational setting.

A number of issues need to be addressed before a particular solution is
carried forward. Among these are the following ones, arranged somewhat
by theme:

-   Scientific issues

    -   Data suitability and availability for an operational need---Do
        the data improve the DSS outputs and are the data routinely
        available to meet this need?

    -   Mission longevity/successor missions---Will necessary data
        continue to be available for as long as the DMS needs?

    -   Calibration/accuracy/errors---Will this change/improve the DSS
        output accuracy and reliability?

    -   Model forecasting capability---Does the forecasting capability
        of this model meet DMS operational needs?

    -   Model suitability/scalability---Is the model suitable for the
        DMS operational needs and is it scalable for that environment?

    -   Etc.

```{=html}
<!-- -->
```
-   Engineering issues

    -   Throughput analysis---Does performance resulting from this
        change meet the operational needs for the application?

    -   Cost/benefit analysis---Does the change improve this ratio?

    -   Throughput capacity---Is the information processing system
        suitable to accept this change?

    -   Etc.

## Propose assimilation plans

The recommendations coming from this process will be brought forward and
made available to the DSS/DMS agency for consideration as augmentations
of State 2. However, as has been noted above, many agencies with DSSs
that are suitable for assimilation of NASA science results, data, or
technology already have formal or informal relationships to carry these
activities out and the activities listed above have been performed to
varying degrees.

# Develop enhanced DSS---State 2

Referring again to Figure 9, the upper right-most block is *Develop
enhanced DSS*. That is the next step in this process and it is where
this effort bears fruit. Note that the block is tan and that in general
this effort will be managed, funded, and implemented by the DSS owner
agency. NASA's role in this particular effort will vary from case to
case, but in general it is recommended to be limited to that of
consultant. Figure 10 is a detailed look at one way in which this might
be done

Note from the figure that we refer back to the original agreement, the
original State 2 upgrade plans, and the recommendations coming from the
previous section---noted as the top three blocks in Figure 11---to start
the actual development process. Note further that the next lower line of
blocks---all tan in the figure---require iteration for
refinement/optimizations, changing State 2 plans, and obtaining DMS
concurrence. On receiving DMS concurrence, the actual development
starts.

The central five blocks---green in the figure---and the green
*Refinement/Optimization* block are the core of the actual State 2
development. The uppermost of these is *Develop Enhanced State 2*. This
block, together with the various loop back capabilities for
refinement/optimization and/or changes in capabilities is where State 2
is created

![](media/image8.wmf){width="6.0in" height="4.388188976377953in"}

The next lower block (*Verification & Validation)*, together again with
loop back capabilities, is forV&V (verification and validation). V&V is
a key strength of the Stennis Space Center and it is recommended that
appropriate representatives either conduct the V&V or assist in some
other material way.

Next in the chain is *DSS Demonstrations* which, with the loop backs is
where the system is made operational. A major part of this effort is to
verify that the system is sufficiently robust to meet the most severe
DMS or other user needs.

The following block, *Transition/Benchmark Test*, is where the new State
2 DSS is being readied for transition to operational status and made
ready for benchmark testing and comparison against the baseline State 1.

# V---MEASURING IMPROVEMENTS AND MOVING TO OPERATIONS {#vmeasuring-improvements-and-moving-to-operations .Section}

# Complete benchmark process

Refer back to Figure 9 one last time. The bottom three blocks represent
the end of the assimilation activity for a particular DSS.

The rightmost of these, *Examine enhanced DSS*, requires complete
characterization of State 2 just as was done with the baseline system at
State 1/0. As the effort to characterize State 2 is similar to that
previously described for the baseline characterization, we do not repeat
the text here.

The leftmost of these, *Measure improvements*, requires measuring and
comparing the States. Refer back to the discussion about initiating
benchmarking and how the assimilation/benchmark team will develop the
various criteria and metrics to measure achievement and success. The
same measures that were applied to benchmarking State 1 are applied to
State 2.

Finally, but certainly not least, is the very last block, *Complete
benchmark process*. It is here that the assimilation and benchmark teams
complete the process:

-   Compare State 2 to State 1/0,

-   Apply the metrics and performance objectives previously designed to
    measure the change from baseline to enhanced operational status.

The difference in system performance between States 1 (or 0) and 2
should demonstrate improvements in one or more of the metrics that were
selected as performance measures at the beginning of the benchmarking
and improvement planning phases. Again, some of these may relate to
straightforward measures of speed and accuracy. However, others should
describe changes in the fundamental nature of the decision support
system itself. These might be movement from an unstructured to a
structured problem-solving environment that allows increased automation
of the decision making process. They might also include increased use of
quantitative data that also permit movement toward automation, but may
also increase reliability or replicability, or reduce levels of
uncertainty. Ultimately, these measures may prove to be at least as
compelling for the partner agency as improvements in simple measure,
like accuracy.

# Document results

The entire process must be documented. Part of this is in keeping with
good practice. In the assimilation process, however, it assumes perhaps
a larger role. These will be the results that are reviewed by partner
agency managers to make decisions that may have profound influence on
their "corporate ways of doing business," and will thus be closely
scrutinized at a number of levels. Moreover, when the partner agency
assumes operational responsibility, this will serve to document the ways
in which State 2 evolved and how it functions, and also to demonstrate
the improvements that it offers over State 1.

# Transfer responsibility

The partner agency plays an active role throughout the process. However,
it is likely that their participation will be highly selective in terms
of the staff involved, and that the individuals involved will be those
who are particularly skilled -- and interested -- in the activity. Thus,
in the process of transferring operational responsibility, training of
rank-and-file agency staff becomes critically important. Before the
system enters operations, the workforce needs to be trained in how to
run the system. The DSS owner agency needs to be trained to properly
interpret the information product outputs. And, the DMS elements, the
decision-makers, need to be informed as to the range of outcomes now
possible and the limitations of the system.

The bottom green block, *State 2: Operational Status*, represents
successful installation of a State 2 DSS and completion of the
assimilation and development process. Until and unless the agency
assumes full responsibility for operation and disseminates the DSS to
all those divisions who have need of the tool, the project is
incomplete. This represents the ultimate goal of the activity and it
must be done by the partner agency.

# REFERENCES {#references .Section}

Andersen, B. and Pettersen P-G. (1996) *The Benchmarking Handbook:
Step-by-Step Instructions*. Chapman and Hall, London, UK

Anthony R.N., (1965) *Planning and Control Systems: A Framework for
Analysis*, Harvard University Graduate School of Business
Administration, Boston.

*Benchmarking Code of Conduct*
<http://professionals.pr.doe.gov/ma5/MA-5Web.nsf/Business/Code+of+Conduct?OpenDocument>

Blattberg R.C. and S.J. Hoch, (1990) Database models and managerial
intuition: 50% model+50% manager. *Management Science*, 1990, vol 36,
pp. 887-899.

Bonczek R.H., C.W. Holsapple and A.B. Whinston, (1981) *Foundations of
Decision Support Systems*, Academic Press, New York.

Churchman C.W., (1971) *The Design of Inquiring Systems: Basic Concepts
of Systems and Organization*, Basic Books, New York, NY.

Courtney J.F., (2001) Decision making and knowledge management in
inquiring organizations: Toward a new decision-making paradigm for DSS,
*Decision Support Systems*, 2001, vol 31, pp. 17-38.

Gorry G.A. and M.S. Scott Morton, (1971) A framework for management
information systems, *Sloan Management Review*, 1971, vol 13, no. 1.

Hoch S.J. and D.A. Schkade , (1996) A psychological approach to decision
support systems, *Management Science*, 1996, vol 42, pp. 51-64.

Jones D.R. and D. Brown, (2002) The division of labor between human and
computer in the presence of decision support system advice, *Decision
Support Systems*, 2002, vol 33, no. 4, pp. 375-388.

Kleinmuntz B., (1990) Why we still use our heads instead of formulas:
toward an integrative approach, *Psychological Bulletin*, 1990, vol 107,
pp. 296-310.

Lucertini, M., Nicolo, F., Telmon, D., (1995) Integration of
Benchmarking and Benchmarking of Integration. *Int. J. Production
Economics*, 1995, vol 38, pp. 59-71.

Mitroff I.I. and H.A. Linstone, (1993) *The Unbounded Mind: Breaking the
Chains of Traditional Business Thinking*, Oxford Univ. Press, New York.

Rittel H.W.J. and M.M. Webber, (1973) Dilemmas in a general theory of
planning, *Policy Sciences*, 1973, vol 4, pp. 155-169.

Schultze U., (2000) A confessional account of an ethnography about
knowledge work, *Management Information Systems Quarterly,* 2000, vol
24, no 1, pp. 1-39..

Shim J.P., M. Warkentin, J.F. Courtney, D.J. Power, R. Sharda and C.
Carlsson, (2002) Past, present and future of decision support
technology, *Decision Support Systems,* 2002, vol. 33, pp. 111-126.

Simon H.A., (1960) *The New Science of Management Decision*, Harper
Brothers, New York.

Spendolini, Michael J. (1992) *The Benchmarking Book*. Amacom, New York,
NY.

Whitecotton S.M., E. Sanders and K. Norris, 1998, Improving bond rating
accuracy with a combination of human intuition and mechanical decision
aids. *Organizational Behavior and Human Decision Processes,* 1938, vol.
76, pp. 325-348.

# APPENDIX A: Decision Making Systems

Following Bonczek, Holsapple, and Whinston (1981), we say that
decisions, in the guise of information, are the "finished products" of a
human-machine information-processing system. We refer to
information-processing systems that yield that finished product as
*decision-making systems (DMSs).* In a DMS, the human---the *user*---is
the decision maker and the machine---the *decision support system
(DSS)*---is an information processing system. Taken together, the DSS
and its user constitute a DMS.

Decision-making has three basic, independent aspects. In the language of
Management Information Science, these are: 1) *Power*[^1]*,* 2)
*Perception*[^2]*,* and 3) *Design*[^3]*.* These three aspects imply
four additional facets, each of which is described as a unique
combination of the original three. These are: 1) *Adaptation*[^4]*,* 2)
*Analysis*[^5]*,* 3) *Idealism*[^6]*,* and 4) *Implementation*[^7]*.*
Figure 1 illustrates these concepts diagrammatically \[Note that Figure
1 is not a Venn diagram.\]

As illustrated in the figure, each of the basic attributes of
decision-making fulfills a dual function. *Power* provides the forces
needed as a basis for the realization of *ideals, values,* or
*preferences* and the forces needed as a basis for *implementation.
Perception* provides empirical information needed as a basis of
*valuation* and as a basis of *knowledge, beliefs* and *expectations.
Design* formulates models that provide a basis for *implementation* and
a basis for *analysis.* The four secondary facets of decision-making
also fulfill dual functions. *Adaptation* maintains equilibrium among
the other six factors and in turn is constrained by them. *Analysis* is
the continuing adjustment between perceptions and formulations, between
information and models. *Idealism* is the ability to hold a vision while
applying power, forcing a vision into reality. *Implementation* is the
ability to execute a plan via directives, the potential for
implementation.

Figure 2 is adapted from Bonczek, et al (1981). The figure illustrates a
generalized model for a decision-making system for this work. From the
figure it is seen that there are five fundamental parts to a generalized
DMS. 1). The User formulates the problem and receives the
decision-making system output. The User is the decision maker. 2). The
Input Interface is designed to facilitate man/machine communication so
that the human can state a problem in such a way that the machine is
able to process the request. 3). The Problem Processing System (the PPS)
takes an input problem statement and information from the Knowledge
System (KS) to produce information that supports (enhances, or makes
possible) a decision process. The PPS has one or more the seven
abilities required for decision-making. 4). The Knowledge System (KS)
contains the body of knowledge available about a problem domain. It has
the data bases, background information, problem domain knowledge,
algorithms, etc. 5). the Output Interface exists between man and machine
allowing the human to retrieve understandable processed information.

![](media/image9.jpeg){width="5.38267716535433in"
height="1.5992125984251968in"}

Figure 3 shows the various components and their interrelationships of a
Problem Processing System (PPS). From the diagram, note that an input
causes information to be acquired from the KS, the problem to be
processed and the decision-making support information to be output for
consideration by the User. Note how the seven abilities of
decision-making are illustrated in the figure. Note also that a DMS may
incorporate a number of DSS and that a particular DSS will incorporate
only the abilities necessary to fulfill its function.

Figure 4 illustrates the Knowledge System (KS). The KS is where the
complete domain of knowledge about a particular problem, or class of
problems resides. It is in the KS where we will focus the majority of
enhancement work for assimilation of ESE data, science results, and
mission products whereas the partnering agency, in upgrading its DSS to
State 2 will concentrate its efforts on the interfaces and PPS.

This scheme can be used to analyze a particular existing DMS into its
logical parts. One or more of the parts can be a target for enhancement,
but it should be recognized that enhancement of one may necessitate
changes to all.

The knowledge system stores data and, often, a repertoire of models to
be used by the problem processing system. The problem processing system
may have models embedded within it or be programmed to "know" how to use
the knowledge system to formulate models. Its main function is to
interpret the syntax of requests coming from the user through the input
interface, implement the appropriate operations on granules of
information in the knowledge system, and return comprehensible advice to
the user through the output interface. Most commonly, the interfaces are
designed to receive and transmit a combination of near-spoken language
syntax and numerical information.

Knowing the structure and function of all of a particular subject DSS's
elements will be necessary to plan the assimilation strategy. Evaluation
of the parts and their relationships will reveal the most advantageous
points to focus enhancement efforts.

A Federal agency DSS may be developed and used by numerous individuals
in various groupings. It should be noted that there is a class of *group
decision support systems* which are applied specifically to the problem
of aiding a group collectively to come to a decision. Usually the group
is heterogeneous in opinion and stake, and the emphasis of the *group
DSS* is on establishing and mediating the process of decision making.
Most likely, the assimilation activity will not be devoted to this type
of DSS, but rather to the traditional type emphasizing computer
processing of data to provide advice to one or more decision makers
individually, in effect.

But the constellation of possible decision making systems which might
use the DSS should be considered as diverse and multifaceted, not
homogeneous. This presents opportunities for enhancement of the DSS to
tailor it to the users, add functionality or improve performance as
defined by any of the stakeholders.

# APPENDIX B: Decision Support Systems Framework

Anthony (1965) described management activities as consisting of
*strategic planning* (executive decisions regarding overall mission and
goals), *managerial control* (middle management guiding the organization
to goals), and *operational control* (first line supervisors directing
specific tasks). Simon (1960) described decision problems as existing on
a continuum from programmed (routine, repetitive, well-structured,
easily solved) to non-programmed (new, novel, ill-structured, difficult
to solve).

Gorry and Scott Morton (1971) integrated Anthony\'s (1965) categories of
management activity and Simon\'s (1960) description of decision types to
produce a "framework" or matrix with operational control, management
control, strategic planning across the top, and *structured,
semi-structured, unstructured* down the side (Figure 1). Cell values are
particular decision types. A semi-structured problem has structured and
unstructured phases or elements. (Note that it is the decision *context*
that is unstructured, not the DSS itself.)

As a framework for describing problems to be handled in a decision
support system, consider replacing the concepts of *managerial control,
operational control*, and *strategic planning* with *modeled problems,
complex problems,* and *novel problem* as provided in Table 1.

Table 1. A decision support framework

  ----------------- ----------------- ----------------- -----------------
                    Modeled Problems  Complex Problems  Novel Problems

  Structured        Routine                             

  Semistructured                                        

  Unstructured                                          Custom tailored
  ----------------- ----------------- ----------------- -----------------

Turban (xxxx) suggests that management information systems (MIS), or
simple models largely internal to an organization, are sufficient to
support structured decisions, while more elaborate and comprehensive
DSSs are applied to semi-structured or unstructured problems. The more
structured a problem, the more amenable it is to solution by
computerized algorithms in a DSS. Conversely, the more unstructured a
problem, the more a decision must rely on the judgment and intuition of
a human being outside the DSS. *An important implication of this for
assimilation is that new data and models can be introduced to explicitly
increase the structure of the decision making context in an Agency,
expanding the capability of the DSS while relieving the decision maker
of the need to guess.* This deliberate increase in structure may be the
essence of DSS enhancement.

# APPENDIX C: Types of Knowledge

Several different types of knowledge are recognized in the literature.
Each represents a different knowledge management challenge.

-   ***Tacit knowledge*** is that which is contained within a person\'s
    head, and is difficult or impossible to express, write down and
    codify. Examples of tacit knowledge would be how to close a deal
    with a particular type of client, or how to develop an effective
    advertising campaign. A good deal of the knowledge management
    literature deals with the creation of organizational policies and
    cultures that encourage sharing of tacit knowledge.

-   ***Explicit knowledge***, on the other hand, is that which can be
    readily articulated, written down, codified and shared. Standard
    operating procedures, and instructions on how to bake quiche are
    examples of explicit knowledge.

-   ***Declarative knowledge*** is distinguished from procedural
    knowledge in the expert systems literature. Declarative knowledge,
    (= data or information) consists of facts or observations about the
    state of the world, such as a patient\'s temperature, or a business
    firm\'s current P/E ratio.

-   ***Procedural knowledge*** is closer to what most authors would
    consider knowledge, as it involves \"how to\" do something, such as
    how to diagnose hepatitis B, or how to analyze a financial
    statement.

-   ***Esoteric knowledge*** is that which is highly specialized,
    formalized, and applicable to narrow domains; in short, that which
    is found in most scientific disciplines. It is of limited value in
    solving unstructured, complex management problems.

-   ***Exoteric knowledge*** is applicable to broad domains, and in some
    cases might be considered \"common sense.\" It is applicable to
    complex, unstructured problems.

-   ***Deep knowledge*** and shallow knowledge are also distinguished in
    the expert systems literature. Deep knowledge is usually related to
    relatively well-structured scientific and technical domains, and
    consists of formal theories of behavior of phenomena in those
    domains.

-   ***Shallow knowledge*** is often that in social domains where
    theories and understanding are usually less well organized and
    codified than in scientific domains.

# APPENDIX D: Knowledge Management Paradigms

Schultze (in press) describes three different perspectives on knowledge
management: the functional, the interpretive, and the critical.

The ***functional paradigm*** is that most often adopted by those in
practice, especially in the software industry. Here knowledge management
is defined as the way that organizations create, capture, store, re-use
and protect knowledge to achieve organizational objectives. This
reflects a realist ontology, the belief that the world is factual, and
that the facts can be known and captured, codified and shared. The most
prevalent technology within the realm of the functional paradigm is a
repository of one form or another (including the tacit repositories in
people\'s minds).

The ***interpretive paradigm*** is founded on the belief that social
reality is socially constructed, and attention is directed to
interpretation, distributed cognition, communications, and social
processes. Knowing and knowledge are inseparable from action. Knowledge
is viewed both as action and object; that is, as both procedural and
declarative. Organizational knowledge is viewed as existing in a
\"collective mind,\" developed through interpretation, communication,
and shared meanings. Organizational knowledge is in a constant state of
flux as new experiences are evaluated and shared. Knowledge management
in this environment consists of fostering communications between
individuals, sharing and enriching interpretations, and coordinating
actions.

The ***critical paradigm*** is concerned primarily with social conflict
and antagonistic relationships. Various stakeholders and special
interest groups take positions and form strategies that produce
differences and conflict among them. Marxist labor processes, class
struggles, and a radical humanist perspective are aspects of the
critical perspective. Knowledge management in this perspective is viewed
in a rather pejorative way as the exploitation of workers by owners, who
seek to extract knowledge from them and commoditize it.

# APPENDIX E: Decision-Support Systems Classification

Our goal is to help a partnering agency in its upgrade plans for a DSS.
To do this, it is vital to know what kind of environment for decision
making will exist for the DSS. Over the past 50 years, decision-making
and decision-making environments have been extensively studied. Mitroff
and Linstone (1993) propose that inquiring organizations are learning
organizations patterned after Churchman\'s (1971) inquiring systems.
Courtney (2001) suggests that decision making systems can be classified
analogously, and presents the following description. In the classical
literature, there are *five types of decision making systems* or
inquiring organizations: *Leibnizian (Type 1), Lockean (Type 2),
Hegelian (Type 3), Kantian (Type 4), and Singerian (Type 5)*, each based
on the philosophies of their respective namesakes. In this work we will
adopt these definitions for characterizing DSSs but will use the more
generic labels, *Type 1 -- Type 5*.

**Type 1---Analytic -- Deductive Approach---**Churchman\'s (1971)
*Leibnizian* inquiring system is a closed, deductive system with a set
of *built-in elementary axioms* that are used along with formal logic
and analysis to generate more general fact nets or tautologies. The
*Type 1* organization creates knowledge by using *formal logic and
mathematical analysis* to make inferences about cause-and-effect
relationships, and maintains that everything needed is already contained
within its boundaries. Type 1 systems are created in a recursive,
self-generating, closed and autonomous manner. As closed systems, they
have access only to knowledge generated internally.

Decision-making procedures in Type 1 organizations exhibit a strict,
formal, bureaucratic, \"by the book\" approach. Missions, policies,
goals, and standard operating procedures serve as Type 1 axioms.
\"Truth\" is determined in a procedural manner, with focus on structural
concerns, and with error detection and correction being a direct
consequence of comparing inputs with the accepted \"axioms\" of the
system. Mathematical models, especially optimization models that attempt
to get at the *one \"best\" answer*, would be widely utilized. Such an
approach to decision making is only well-suited to structured, routine
problems.

**Type 2---Consensual -- Inductive Approach---**Mitroff and Linstone
(1993) refer to the *Lockean* inquiring system as being inductive and
consensual. The Type 2 organization creates knowledge from empirical
information gathered from external observations and used inductively to
build a representation of the world with a set of labels (or properties)
assigned to the observations. Communication and consensus are hallmarks
of this approach, and agreement by the Type 2 community on the
properties (labels) to be assigned to an observation is the guarantor of
the system.

A community of Type 2 inquirers learns by *observing the world, sharing
observations, and creating a consensus* about what has been observed.
Organizational knowledge is created through observation, interpretation,
communication, and the development of shared meaning. The
organization\'s members must feel free to observe and express opinions.
Moreover, they must have a common language and mindset, which permits
effective communication. The decision style is clearly group-oriented
and open. Input is sought from a variety of sources, communication is
encouraged, and consensus is sought.

**Type 3---Empirical -- Theoretical Approach---**the *Kantian* approach
recognizes that there may be *many different perspectives on a problem*,
or at least many different ways of modeling it. The perspectives are
all, however, *based on the belief that problems can be modeled
analytically*. The Type 3 inquirer constructs various models which
attempt to interpret and explain observations about a decision
situation. Each model has some \"goodness of fit\" measure, such as a
standard error or variance. An executive routine is capable of invoking
a particular type of modeling process, and observing its behavior. It
can turn off models that are not performing well. It finally *chooses
the model which best explains the data*.

Mitroff and Linstone (1993) believe this approach is suitable for
problems of moderate complexity. There is little or no emphasis placed
on human interpretation of the problem, nor of human involvement. The
problem is attacked strictly from a technical perspective.

**Type 4---Conflict -- Synthesis Approach---** the *Hegelian* inquirer,
is based on the belief that the most effective way to create knowledge
is by observing a debate between two diametrically opposed viewpoints
about a topic, a *thesis* and an *antithesis,* from which a *synthesis*
is constructed as the worldview.

The Type 4 decision style requires *synthesizing* a solution from
*conflict* between two opposing perspectives and the debate between
parties holding opposing views on a problem. The knowledge to be managed
in this environment consists of the information that the thesis and
antithesis attempt to interpret, the thesis and antithesis themselves,
the debate and, of course, the synthesis.

**Type 5---Multiple Perspectives -- Holistic Approach---** the
*Singerian* inquirer views the world as a holistic system, in which
everything is connected to everything else. From this Type 5
perspective, problems and knowledge domains (disciplines) are
inseparable. The artificial division of knowledge into disciplines and
the reduction of complex problems into simple components inhibit the
solution to complex problems. Solving complex problems may require
knowledge from any source and knowledgeable people in any discipline or
profession. The Type 5 approach, *holistic* with *multiple perspectives*
(*technical; organizational and social; personal and individual*),
embraces the other thinking styles (Types 1 -- 4), using any or all of
them where appropriate in decision-making processes. The multiple
perspectives approach does not end with the technical, organizational,
and personal perspectives. It also explicitly brings *ethics and
aesthetics* into play.

#  APPENDIX F: Benchmarking Tasks

List of topics and activities associated with the 4 phases of the
benchmarking process:

1.Identify and select the benchmarking partners and the process or DSS
to be benchmarked

> 1.1. Select the DSS, DSS-module or process to be benchmarked, based on
> the development strategy of the 'owner' of the DSS.
>
> 1.2. Establish Critical Success factors (cooperative agreement,
> development plan)
>
> 1.3. Identify Benchmarking partners: NASA, Federal agencies, Natl.
> Labs, University collaborators
>
> 1.4. Conduct zero-order assessment of the DSS
>
> 1.5. Identify and qualify sources of data, models, and technology
>
> 1.6. Conduct risk analysis and benefit assessment

2.Characterize and observe the DSS and assess how the benchmarking
partners can contribute best to this DSS

> 2.1. Understand and document the DSS to be benchmarked
>
> 2.2. Assess the information needs of the DSS
>
> 2.3. Select methods and tools for providing science, data and
> technology inputs
>
> 2.4. Form a benchmarking team of experts
>
> 2.5. Develop an enhancement plan
>
> 2.6. Establish performance measures for the DSS (e.g. quality, time,
> cost, usefulness)
>
> 2.7. Carry out information collection and debrief
>
> 2.8. Set up a benchmarking plan

3.Assimilate data, science or technology in the DSS and analyze the
causes for the performance differences between the baseline (State 1)
and improved DSS (State 2)

> 3.1. Sort and organize the collected information and data for baseline
> and enhanced DSS
>
> 3.2. Quality control assessment of the information and data
>
> 3.3. Test and evaluate the interface between algorithms, technology
> and observations and a DSS
>
> 3.4. Identify difference in performance levels
>
> 3.5. Identify causes for the performance differences
>
> 3.6. Involve users and experts in evaluation
>
> 3.7. Conduct risk analysis and benefit assessment

4.Adapt and implement improvements and changes based on the benchmarking
analysis

> 4.1. Identify improvements and opportunities based on identified
> enhancements
>
> 4.2. Set targets for the improvements
>
> 4.3. Develop an implementation plan, implement improvements, monitor
> the progress of the implementation
>
> 4.4. Lessons learned
>
> 4.5. Write a final report

# APPENDIX G: Usability and Performance Metrics

Usability and performance metrics are critical factors for the
adaptation and assimilation of enhancements to a DSS. Both qualitative
and quantitative information will be gathered to benchmark a DSS. This
will involve the use of web statistics, questionnaires and interviews
and other risk assessment tools.

Success factors and metrics for benchmarking a DSS (Lucertiniet al.,
1995)

-   Usability of the DSS (state1 -- state 2)

    -   Ease of use (GUI)

    -   Learning curve (training)

    -   Workload (more data, better models)

    -   User needs (consistent data)

    -   User tasks (interactive)

    -   Documentation

-   Performance of the DSS (state1 -- state 2)

    -   Cost effectiveness,

    -   User needs (consistency, timeliness)

    -   Organizational needs (IT, expertise)

    -   User performance (accuracy, timeliness)

    -   Organizational performance

    -   Bottlenecks (network)

# APPENDIX H: Benchmarking Code of Conduct

**Benchmarking Code of Conduct**\
by [The International Benchmarking Clearinghouse](http://www.apqc.org/)

[http://professionals.pr.doe.gov/ma5/MA5Web.nsf/Business/Code+of+Conduct?OpenDocument](http://professionals.pr.doe.gov/ma5/MA-5Web.nsf/Business/Code+of+Conduct?OpenDocument)

Principle of Legality

-   If there is any potential question on the legality of an activity,
    consult with your corporate counsel.

-   Avoid discussions or actions that could lead to or imply an interest
    in restraint of trade, market and/or customer allocation schemes,
    price fixing, dealing arrangements, bid rigging, or bribery. Don\'t
    discuss costs with competitors if costs are an element of pricing.

-   Refrain from the acquisition of trade secrets from another by any
    means that could be interpreted as improper including the breach or
    inducement of a breach of any duty to maintain secrecy. Do not
    disclose or use any trade secret that may have been obtained through
    improper means or that was disclosed by another in violation of duty
    to maintain its secrecy or limit its use.

-   Do not, as a consultant or client, extend benchmarking study
    findings to another company without first ensuring that the data is
    appropriately blinded and anonymous so that the participants\'
    identities are protected.

Principle of Exchange

-   Be willing to provide the same type and level of information that
    you request from your benchmarking partner to your benchmarking
    partner.

-   Communicate fully and early in the relationship to clarify
    expectations, avoid misunderstanding, and establish mutual interest
    in the benchmarking exchange.

-   Be honest and complete.

Principle of Confidentiality

-   Treat benchmarking interchange as confidential to the individuals
    and companies involved. Information must not be communicated outside
    the partnering organizations without the prior consent of the
    benchmarking partner who shared the information.

-   A company\'s participation in a study is confidential and should not
    be communicated externally without their prior permission.

Principle of Use

-   Use information obtained through benchmarking only for purposes
    stated to the benchmarking partner.

-   The use or communication of a benchmarking partner\'s name with the
    data obtained or practices observed requires the prior permission of
    that partner.

-   Contact lists or other contact information provided by the
    International Benchmarking Clearinghouse in any form may not be used
    for purposes other than benchmarking and networking.

Principle of Contact

-   Respect the corporate culture of partner companies and work within
    mutually agreed procedures.

-   Use benchmarking contacts, designated by the partner company, if
    that is their preferred procedure.

-   Obtain mutual agreement with the designated benchmarking contact on
    any hand-off of communication or responsibility to other parties.

-   Obtain an individual\'s permission before providing his or her name
    in response to a contact request.

-   Avoid communicating a contact\'s name in an open forum without the
    contact\'s prior permission.

Principle of Preparation

-   Demonstrate commitment to the efficiency and effectiveness of
    benchmarking by being prepared prior to making an initial
    benchmarking contact.

-   Make the most of your benchmarking partner\'s time by being fully
    prepared for each exchange.

-   Help your benchmarking partners prepare by providing them with a
    questionnaire and agenda prior to benchmarking visits.

Principle of Completion

-   Follow through with each commitment made to your benchmarking
    partner in a timely manner.

-   Complete each benchmarking study to the satisfaction of all
    benchmarking partners as mutually agreed.

Principle of Understanding and Action

-   Understand how your benchmarking partner would like to be treated.

-   Treat your benchmarking partner in the way that your benchmarking
    partner would want to be treated.

-   Understand how your benchmarking partner would like to have the
    information he or she provides handled and used, and handle and use
    it in that manner.\
    **NOTE:** Identification of firms, organizations contacts/visited is
    prohibited without advance approval from the organization.

# APPENDIX I: Risk analysis and benefit assessment

The lifecycle of a DSS consists of three principal phases --
formulation, implementation, and enhanced operations -- separated by
corresponding transitions at states 0, 1, and 2. Formulation phase
considers architectural and preliminary design options for the DSS,
responsive to agency development plans. Implementation includes detailed
design, verification and validation (V&V), and operations. Enhanced
operations encompass a revisit of architecture, design, implementation,
and V&V that includes the assimilation of NASA components (i.e. data,
science results, models and technology). While states provide discrete
reference points for benchmarking the performance of a DSS, the
continuous lifecycle is the basis for DSS maturity assessment and risk
analysis.

Each DSS state is defined by a set of requirements, identified from
corresponding development, operational implementation and upgrade plans
from agency partners. The maturation between states is defined in terms
of requirements attainment. For example, between states 0 and 1, a DSS
maturity assessment would capture the DSS readiness for implementation,
whereas an assessment between states 1 and 2 would indicate the ability
to address enhanced operational requirements in view of NASA
assimilation products. The maturity in each case is given by the
percentage of state 1 and state 2 requirements met, respectively; this
figure is established qualitatively by NASA and/or agency partner
consensus, and quantitatively if modeling, simulation, stochastic
methods and closed-form approaches are available and feasible. With the
incorporation of budget plans and actual costs, and milestone schedules,
the maturity assessment is not unlike earned value management in the
ability to represent development and implementation progress towards
meeting requirements and providing DSS deliverables.

It is assumed that risks to meeting requirements are either outlined or
described in corresponding agency plans or can be expertly elicited.
These risks are associated with a diversity of factors such as
technology, data delivery, organization, resources and infrastructure.
Each risk element can be assigned a likelihood of occurrence and an
impact on each of the requirements. For example, the unavailability of
lidar images of a collapsed urban corridor within 2 days of a natural
disaster may have a 90% likelihood of occurrence with a 30% impact on
the ability to estimate property losses using HAZUS for disaster
response, but no impact on the use of HAZUS for studying scenario
earthquakes for setting property and casualty reinsurance rates next
year. By capturing risk elements in each lifecycle phase, the maturation
of a DSS can be equated with risk retirement. Further, the attendant
work breakdown structure (WBS) for DSS development, implementation and
enhancement through assimilation can be represented by activities that
retire the risks identified in meeting requirements. Each WBS element
has a corresponding cost or resource allocation and schedule. This, in
turn, allows cost-benefit analysis for selection of an optimal WBS
yielding a mutually agreed upon risk profile, where benefit is a direct
measure of requirements attainment. Note that this is not a
return-on-investment analysis that relies on dubious estimates of
economic or dollar benefit against investments; rather, benefit is
traceable to requirements in the units of those requirements and the
cost is the direct mitigation cost from the WBS.

The figure below is a sample graphical user interface snapshot from the
lifecycle risk management decision-support software tool -- *Defect
Detection and Prevention* (DDP) \[1, 2\] developed by NASA for complex
space systems to assess and visualize risk in the planning, formulation
and implementation of projects. Further, DDP is being used to assess the
maturity if technologies for infusion into space systems -- technology
infusion for systems thus the analog to data assimilation to DSSs. The
figure shows a residual risk profile. A risk profile is a histogram of
risk elements where the height of each risk column is the aggregate
impact of that particular risk across all requirements. A residual risk
profile would indicate the impact of risk mitigations on the baseline
(i.e. previously benchmark) risk profile. The example is taken from the
application of DDP (for illustration purposes only) to an intermodal
transportation project under the joint DOT-NASA program; the project
looked into the use of remote sensing products for development of an
Intermodal Connector Analysis Tool (ICAT) \[3\], a potential DSS for
DOT. Thirteen risk elements are indicated in the figure, with the
residual risks between two phases or approaches (i.e. WBS structures)
for use of remote sensing in the ICAT.

![](media/image10.png){width="5.9944444444444445in"
height="4.794444444444444in"}

With the identification of requirements, risks and selection of a risk
retirement WBS, the NASA assimilation with a DSS can be managed on the
basis of risk analysis. At each state transition, the benchmarking
itself can be represented by a \[residual\] risk profiles from states 0
to 1 and 2. The requirements and risks at each state also differ and
hence influence the shape change of the profiles. However, there is
consistency in units (i.e. metrics) that allows comparison from one
state to another and communication to stakeholders of the objectives of
the risk analysis -- namely, phased requirements attainment and actually
pushing-back on these requirements based on cost-benefit analysis and
available resources.

It is clear that risk profiles are valuable in making decisions and
allocating resources, encompassing technical to programmatic
considerations. Risk profiles also are useful in indicating development
or implementation maturity along the lifecycle. With the inclusion of
likelihoods of occurrence, tree structures, weighting schemes, and
consensus (i.e. NASA and agency partner) on risk impacts and mitigation
effectives, risk assessments recognize the inherent uncertainty in the
results due to uncertainties in the underlying data, modeling and
simulation, expert opinion and operational environments. The
requirements, risks and mitigations typically will be captured as tree
structures, with principal and subordinate elements to appropriate
levels of breadth and depth. The scope of risk profiles will vary
depending on the DSS, and can range from expert elicitation methods to
analysis methods (e.g. HAZUS in the abovementioned lidar scenario). Risk
analysis as suggested herein offers a uniform architecture for working
with agency partners on DSS benchmarking and enhancement.

\[1\] Feather, MS, SL Cornford, and M Gibbel, "Scalable Mechanisms for
Requirements Interaction Management", *in Proc 4th IEEE Intl. Conf. on
Requirements Engineering, Schaumburg, Illinois, 19-23 June 2000. IEEE
Computer Society.*

\[2\] Cornford, SL., "Managing Risk as a Resource using the Defect
Detection and Prevention process," *International Conference on
Probabilistic Safety Assessment and Management*, September 13-18, 1998.

\[3\] Lyte, WF (Tetra Tech, Inc.), N. Bryant (JPL), etc., Application of
High Resolution Commercial Satellite Imagery to Analysis of Federal
Highway System Intermodal Connectors Along the Alameda Corridor, for
Office of the U.S. Secretary of Transportation, Office of Intermodalism,
and U.S. Department of Transportation Office of Research and Special
Programs, 2002.

[^1]: *Power* is the ability to govern and eliminate that which is
    unresponsive.

[^2]: *Perception* is the ability to observe, to gather information.

[^3]: *Design* is the ability to formulate, to create models.

[^4]: *Adaptation* is the core or essence of decision making.

[^5]: *Analysis* is information gathering and processing.

[^6]: *Idealism* is holding to a vision.

[^7]: *Implementation* is executing a plan.