Technical Report

report number (CMU/SEI-96-TR-xxx)

esc number (ESC-TR-96-xxx)

12/11/96

RISQ: A Web-Based Tool for Referencing Information on Software Quality

  ---------------------------------------------------- ---------------------------------------------------------------
  ![](media/image1.emf){width="0.6381944444444444in"   
  height="0.40208333333333335in"}                      

  ---------------------------------------------------- ---------------------------------------------------------------

Charles B. Weinstock

Dolores R. Wallace

Dependable System Upgrade Initiative

Unlimited distribution subject to the copyright

**Software Engineering Institute**

Carnegie Mellon University

Pittsburgh, PA 15213

This report was prepared for the

SEI Joint Program Office

HQ ESC/ENS

5 Eglin Street

Hanscom AFB, MA 01731-2116

The ideas and findings in this report should not be construed as an
official DoD position. It is published in the interest of scientific and
technical information exchange.

FOR THE COMMANDER

(signature on file)

Thomas R. Miller, Lt Col, USAF

SEI Joint Program Office

This work is sponsored by the U.S. Department of Defense.

Copyright © 6/24/96 by Carnegie Mellon University.

Permission to reproduce this document and to prepare derivative works
from this document for internal use is granted, provided the copyright
and ÒNo WarrantyÓ statements are included with all reproductions and
derivative works.

Requests for permission to reproduce this document or to prepare
derivative works of this document for external and commercial use should
be addressed to the SEI Licensing Agent.

NO WARRANTY

THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE
MATERIAL IS FURNISHED ON AN ÒAS-ISÓ BASIS. CARNEGIE MELLON UNIVERSITY
MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY
MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR
MERCHANTIBILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE
MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY
KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT
INFRINGEMENT.

This work was created in the performance of Federal Government Contract
Number F19628-95-C-0003 with Carnegie Mellon University for the
operation of the Software Engineering Institute, a federally funded
research and development center. The Government of the United States has
a royalty-free government-purpose license to use, duplicate, or disclose
the work, in whole or in part and in any manner, and to have or permit
others to do so, for government purposes pursuant to the copyright
license under the clause at 52.227-7013.

This document is available through Research Access, Inc., 800 Vinial
Street, Pittsburgh, PA 15212.

Phone: 1-800-685-6510. FAX: (412) 321-2994. RAI also maintains a World
Wide Web home page. The URL is http://www.rai.com

Copies of this document are available through the National Technical
Information Service (NTIS). For information on ordering, please contact
NTIS directly: National Technical Information Service, U.S. Department
of Commerce, Springfield, VA 22161. Phone: (703) 487-4600.

This document is also available through the Defense Technical
Information Center (DTIC). DTIC provides access to and transfer of
scientific and technical information for DoD personnel, DoD contractors
and potential contractors, and other U.S. Government agency personnel
and their contractors. To obtain a copy, please contact DTIC directly:
Defense Technical Information Center, Attn: FDRA, Cameron Station,
Alexandria, VA 22304-6145. Phone: (703) 274-7633.

Use of any trademarks in this report is not intended in any way to
infringe on the rights of the trademark holder.

# Abstract

World Wide Web (Web) technology has provided access to enormous amounts
of information on-line. As such it has a tremendous potential for use as
an aid to technology transfer. However, locating relevant information is
often difficult partially because of the volume of information
available. This report describes a new search engine which is designed
to allow the user to do efficient searches for information within a
specific domain. The initial application of the engine is in the domain
of high integrity software systems and is called RISQ which stands for
"Reference Information for Software Quality." The search engine allows
searches by taxonomy based keywords, other keywords, and artifact type.
Artifacts can range from simple abstracts, documents and software to
video, audio and on-line interactive demonstrations of software tools.
This report discusses the philosophy of the system, the acceptance
criteria for artifacts, and provides instructions for using the RISQ
system.

# Keywords

high integrity software, search, search engine, quality, software,
taxonomy, World Wide Web

# Table of Contents

**Abstract**

Keywords

Table of Contents

Introduction

History of RISQ

Philosophy

General Philosophy

Searching

Taxonomy based search

Keyword based search

Artifact type

Artifact Submission

Artifact Acceptance Criteria

Requirements

Articles

Tool Artifact

Using the RISQ System

Finding the RISQ System

Setting the Selection Criteria

Selecting Artifact Types

Setting User Defined Keywords

Taxonomy Based Search

Retrieving Artifacts

Submitting an Artifact

Maintaining the RISQ Database

Add Artifact

Modify Artifact

Reviewing Submissions

Tool Demonstration Artifacts

Slide-Show

Live Demonstration

Canned Demonstration

The Demonstration Evaluation Form

Summary

Acknowledgments

# Introduction

This document describes the Reference Information for Software Quality
(RISQ) system developed by members of the National Institute of
Standards and Technology (NIST) and Software Engineering Institute (SEI)
technical staff[^1].

The RISQ system is an important part of CHISSA, the Center for High
Integrity Software Systems Assurance. CHISSA was organized by NIST's
Computer Systems Laboratory in 1994 as a collaborative approach for
government, industry, and academia to make available the technology
which is necessary for assuring high integrity software in an ever
growing number of applications.

Software is a major factor in corporate operation, consumer goods,
military systems, environmental and energy services, communications,
health care, and government operations. High integrity software, that
is, software that can and must be trusted to work dependably, is a
necessity for the ability of United States industries and government to
function. Good dependability techniques developed in laboratories have
not made it into use by development organizations. Conversely, very real
problems faced by development organizations are not being addressed by
the research community. A major goal of CHISSA is to make it easier for
researchers to collaborate with developers to

1\. see how research results perform in practice,

2\. improve the dependability of the resulting applications, and

3\. direct the researchers\' efforts more towards helping the developers
with their real problems.

The RISQ facility helps achieve these goals by making available a wide
variety of artifacts[^2] related to software quality in a highly
organized manner. The overarching goal is to help to transition
technologies so that the best of them become standard practices when
developing software systems with a requirement for high integrity.

The current RISQ facility is located on the World Wide Web ( Web) at
hissa.ncsl.nist.gov/risq/. By locating the facility on the Web, we
achieve the goal of making the material available to a large and
geographically diverse set of potential users.

# History of RISQ

When CHISSA was first established, a Call for White Papers was issued to
collect ideas on how to shape the organization to best serve the high
integrity software systems community. There were over 100 responses to
the Call, and it soon became apparent that an important role for CHISSA
would be to provide a central location for industry, government, and
academia to exchange information about tools and techniques available
for achieving high integrity software systems.

Originally a CHISSA demonstration facility, to be hosted at NIST, was
proposed. This facility would act as a repository for demonstrations of
high integrity software systems technologies, and would behave much like
a library. Patrons would visit the facility to view the demonstrations
and leave with relevant literature.

It soon became apparent that this approach was not practical. Not only
would it require a physical plant on the NIST campus, but it would also
require NIST personnel to conduct the demonstrations. This was clearly
not feasible from a budgetary point of view. Worse, it would require the
patron to journey to the NIST campus, something that would limit the
effectiveness of the facility in terms of its ability to serve a large
number of people.

The second approach to a CHISSA demonstration facility was prototyped on
the Web in 1995. This facility would consist of canned and live
demonstrations, all hosted on a NIST Web site. The canned demonstration
would be a series of Web pages with clickable images to simulate the
actual operation of the system being demonstrated. A live demonstration
would take advantage of a combination of the Web and windowing systems
to actually allow the user to experiment with the real tool, or at least
a special modified version of the real tool.

The prototype of this demonstration facility worked well. However a
useful facility would consist of more than tool demonstrations. It
became clear that a facility of any size would need to be distributed
over many sites. This would allow, for instance, home organizations to
install and maintain their own demonstrations on differing types of
computers and greatly ease the work load on NIST personnel. As a result
of this thinking the current RISQ system was prototyped in early 1996 by
Tony Cincotta of NIST and the current version was implemented later that
same year.

# Philosophy

## General Philosophy

The designers of the RISQ system took the point of view that generality
is not always the best way to proceed. Since the domain of interest is
relatively well defined, the system has been organized to make searching
within that specific domain more efficient for the user.

Another key part of the philosophy behind RISQ is that the best way to
populate it, given the limited resources available to CHISSA, is to
allow the community to submit suggested artifacts for inclusion.
Although the RISQ administrators will have the final say as to what gets
added to the database, allowing users to nominate artifacts provides a
tremendous amount of leverage.

A third part of the philosophy is that users should have confidence in
artifacts located through the RISQ system. This is embodied in the
published acceptance criteria.

## Searching

Objects maintained by the RISQ database (i.e., RISQ artifacts) are
classified according to three axes:

1.  Taxonomy category

2.  Keywords

3.  Artifact type

### Taxonomy based search

Ordinary search engines are either free form or keyword based. The user
is left with the quandary of how to carefully choose keywords to elicit
the appropriate artifacts. The RISQ system has taken a different
approach. It presents the user with a taxonomy of high integrity terms.
Every artifact will fit into one or more of the categories in the
taxonomy. The result is that the user does not have to guess at what the
appropriate search terms might be. Figure 1 shows the current RISQ
taxonomy.

The user may select as many terms as desired to restrict the number of
items located. As with any search engine, selecting incompatible terms
will result in no items being returned. Selected terms are combined
according to the following rules:

-   In general, terms are combined using the *and* relationship. Thus if
    the terms Fault Tolerance and Covert Channel Analysis are both
    selected, only artifacts that are classified in both categories are
    returned.

-   Selecting a parent node selects all of the child nodes using the
    *or* relationship. Thus selecting Dynamic Analysis (under
    Verification and Test) will cause artifacts classified under either
    Prototyping or Simulation to be returned. Note that this is not
    equivalent to selecting prototyping and simulation individually. In
    this latter case an artifact would have to be classified under both
    terms to be returned. In the former case it would only have to be
    classified under one of the terms.

### Keyword based search

Any artifact can be located by selecting an appropriate set of terms
from the taxonomy described above. As the database of artifacts gets
more fully populated, restricting the search to manageable sets of
artifacts will become increasingly difficult. To help with this problem,
and to allow searching for keywords that are not in the taxonomy, the
RISQ system also allows searches based on user defined keywords.

Searching using user defined keywords is somewhat problematic in that
the user must guess at what keywords will elicit the appropriate
artifacts. If the person entering the artifact into the database has a
different world view than the person doing the search, important
artifacts might not be retrieved. In the future we plan to develop the
capability to automatically search for synonyms of the user defined
keywords.

### Artifact type

The third way in which searches may be restricted is by selecting
specific artifact types. Currently defined artifact types include:

Abstract A text artifact that is an extended abstract describing some
object that is *not* in the RISQ database (e.g., a paper published in a
journal.)

Audio Artifacts that are aural in nature. This could be a .WAV file with
relevant information, or a live lecture on a high integrity topic.

Code Source program artifacts that implement some high integrity
function.

Data Artifacts consisting of real-world data which can be studied to
glean additional information about how systems behave.

Design A design document related to high integrity systems.

Document A complete document related to high integrity systems. This
could be an HTML document, a text file, a PDF file suitable for viewing
with Acrobat, or any other format that can be read on-line.

Image A picture or figure related to high integrity systems.

PostScript A document in PostScript form. These documents typically must
be printed to be read.

Requirement A requirements document.

Resource A resource or on-line service that cannot otherwise be
categorized. This often will be the location on the web of additional
high integrity artifacts.

Taxonomy Alternative taxonomies relevant to high integrity systems.

Test Testing materials.

Tool A tool. This may be just a demonstration (live or canned), or it
may be the actual tool itself.

Tutorial A document artifact that is a tutorial covering some aspect of
high integrity software systems.

Video An on-line video relevant to high integrity systems.

If the user does not specify an artifact type, the RISQ system defaults
to type document.

All of these search mechanisms can be combined. For example the taxonomy
search can pick "Hazard Analysis" and if the artifact type is "tool",
any tools in the RISQ database that address hazard analysis will be
retrieved.

## Artifact Submission

One of the key philosophies underlying the RISQ System is that, for the
repository to grow, we will need the assistance of the user community to
populate it. Using an almost identical facility to what the RISQ
administrators use to add artifacts, any user of the RISQ System is able
to nominate an artifact for inclusion in the database. The
administrators are notified of this nomination and have an opportunity
to review the artifact to see if it meets the appropriate acceptance
criteria (as discussed below). The administrator may reject the
submission, accept it as is, or edit it.

## Artifact Acceptance Criteria

In order to ensure that the information provided by the RISQ system is
of maximum usefulness, the methods, tools and techniques described in
RISQ artifacts must represent accepted practice or new technology, and
meet certain criteria reflecting professional standards for publication.

All artifacts must be related to high integrity software. Any of the
requirements discussed below may be waived at the discretion of the RISQ
administrator.

### Requirements

There are two overarching requirements to consider when looking at any
submission to the RISQ system:

1.  The credibility of the RISQ system must be maintained.

2.  The RISQ system may not endorse commercial products, while at the
    same time it must assure equal opportunity for technology to be
    exposed to industry.

The RISQ administrators will enforce acceptance criteria for each
artifact type. A preliminary set of criteria are identified for document
(including those in abstract or postscript form) and tool artifacts.
Acceptance criteria will eventually be developed for all artifact types.
The RISQ/CHISSA project manager will review recommendations from the
RISQ administrator and has final authority for accepting or rejecting
submissions for this facility.

### Articles

An article is an artifact of the type abstract, document or PostScript.
An article submitted to RISQ must have had a technical peer review
(e.g., a review process for a published journal article; a refereed
conference paper) or in the case of an abstract, the item referenced by
the abstract must meet this criterion. It must be associated with the
RISQ taxonomy (i.e., it must address one or more topics identified in
the taxonomy).

The article may describe a software development or assurance method, a
software tool associated with the method or an experiment with the
method. It may contain case studies of using the method, or may provide
a tutorial for using the method. Articles may also be surveys of a
specific subject area (e.g., software safety, testing).

### Tool Artifact

A software tool may implement a software development or assurance
(verification, testing, any diagnostic) method or a new approach or
algorithm to solve a specific problem. It must be associated with the
taxonomy of the RISQ (i.e., it must address one or more topics
identified in the taxonomy.) Current acceptance criteria are:

-   The tool must be supported by a technical paper that has had a
    technical peer review (e.g., refereed conference ). The paper must
    describe the tool and the technology on which it is based. Normally,
    both the tool artifact and the supporting technical paper document
    artifact should be submitted to the RISQ system.

-   The submitter of the tool must supply evidence that the tool will
    completely execute in its demonstration mode.

-   The demonstration consists of a tutorial accompanying each
    demonstrated function of the tool (see tool installation
    instructions.) The submitter must provide on-line user documentation
    separate from the demonstration.

-   In general, tool demonstrations normally execute on the submitter\'s
    server.

-   The submitter is responsible for modifying the tool to be able to
    execute under control of the RISQ, which executes via the Web. The
    submitter should contact the RISQ administrator for instructions on
    how to modify the tool in order for it to be accessed via the RISQ
    search facility.

# Using the RISQ System

## Finding the RISQ System

The RISQ system is on the World Wide at hissa.ncsl.nist.gov/risq/. The
initial page is presented as shown in Figure 2.

The page is divided into three logical sections. Section one gives
access to the RISQ system's search facility. To enter it push the button
marked Access Resource Center. Section two gives access to the user
submission facility. To submit an artifact push the Submit an Artifact
button. Finally, the third section is a source of help, and general
information about CHISSA.

## Setting the Selection Criteria

When the user pushes the Access Resource Center button, a screen similar
to that shown in Figure 3 appears. There are three ways to limit a RISQ
system search, and thus there are three major portions of interest in
the RISQ Query Selection Criteria screen; a section for selecting
artifact types, a section for user defined keywords, and a section for
taxonomy based searches.

### Selecting Artifact Types

The search can be restricted to any of a number of artifact types. To
select a particular artifact type, click on the box next to that
artifact type. A check mark will appear next to those artifact types
selected. To deselect an artifact type, click on it to remove the check
mark. Checking the "all" artifact type is equivalent to individually
checking each of the other artifact types.

### Setting User Defined Keywords

To provide a user defined keyword phrase, type it in the box above the
button labeled "Keyword". Then push the button. As keywords are defined
they will appear, with a check box, under the button. To undefine a user
keyword, uncheck the box next to it. Note that case is not important in
user defined keywords. For purposes of searching, "Fault" is exactly the
same as "fault".

### Taxonomy Based Search

To select taxonomy terms, click on them in the tree. Initially the tree
is shown as "High Integrity" plus a set of five buttons---one for each
of the areas covered by the taxonomy. Clicking on "High Integrity" will
select all terms in the taxonomy. (Indeed, clicking on "High Integrity"
and selecting "All" for the artifact type will result in the search
engine returning all artifacts in the database.) Clicking on one of the
five buttons will result in a taxonomy tree being displayed for that
particular area.

Within a tree, clicking on a term will select that term. Clicking on a
child of an already selected term will deselect the original term and
select the child instead. Likewise, clicking on a parent will replace
any previously selected children with the parent. So in the example
shown in Figure 4 if "Hazard Analysis" had been previously selected,
clicking on "FTA" would replace "Hazard Analysis" with "FTA". Then
clicking on "Safety" would replace "FTA" with "Safety".

In the example the artifact types chosen are Abstract, Document,
PostScript, and Tool. The user defined keyword is software hazard
analysis, and the taxonomy based search will cover the whole Safety
taxonomy.

## Retrieving Artifacts

Once the selection criteria are set, clicking on the button labeled
"Find Matching Artifacts" will result in a screen similar to that of
Figure 5. This screen gives a capsule summary of the section criteria
used, and shows the retrieved artifacts, if any. If you are unhappy with
what was retrieved, the selection criteria may be modified by clicking
on the "Change Selection Criteria" button.

Otherwise the retrieved artifacts may be inspected. Clicking on the
underlined text will produce a RISQ abstract similar to that shown in
Figure 6. Clicking on a button like the one under the underlined title
(or the one shown on the abstract page), will take you to the artifact
itself. In this case clicking on the button will take you to the
document shown in Figure 7.

## Submitting an Artifact

The RISQ system provides a means for the user to submit artifacts for
possible inclusion in the database. To submit an artifact the submitter
provides information about the artifact, including its taxonomy
classifications, any user defined keywords, and the artifact types
involved. This information is stored for later retrieval by the RISQ
administrators who will make the ultimate decision as to whether the
artifact will be added to the database.

The process begins when the user pushes the "Submit an Artifact" button
on the RISQ home page (Figure 2). This causes a page very similar to the
search selection page of Figure 3 to be shown. In fact, except for the
title, the button labels, and the instructions, the page is identical.
In particular the taxonomy, user defined keyword, and artifact type
sections behave identically.

Figure 8 shows the screen as it looks after pushing the "Submit an
Artifact" button on the RISQ home page. The submitter uses this page to
classify the artifact as precisely as possible. Note that it is
permissible to have one description apply to multiple artifacts (e.g., a
document and a tool may share the same description.) In this case only
one entry will appear in the database, but it will be displayed with
multiple buttons---one for each of the artifact types.

Once the user has set the search terms for the artifacts, he pushes the
"Submit Artifact" button and a page similar to Figure 9 appears. This
page includes a capsule summary of the terms selected, along with
additional fields to be filled in. The first field is the e-mail address
of the submitter. This is important as it is the only way that the RISQ
administrators can get in touch with the submitter. The next field is
for the title of the artifact. Following that field, there will be a
locator field for each of the artifact types selected. In this case
there is only the one locator---for the document artifact type. This
field will typically be filled in with a

URL that points to a location where the artifact may be found. Finally,
there is a field for a short abstract that describes the artifact. Once
the user is satisfied with his submission, a click on the "Submit"
button saves it away and notifies the RISQ administrator that the
submission is ready for them to look at.RISQ administrator that the
submission is ready for them to look at.

#  

# Maintaining the RISQ Database

For the RISQ administrators, maintaining the RISQ database is done
almost the same as browsing the RISQ system. Maintenance mode is
accessed via the URL: http://hissa.ncsl.nist.gov/maintain.html. Figure
10 shows that page, which is divided into two sections. The first
section, with the button label "Access RISQ Maintenance Center," allows
the administrator full access to the database. The second is used to
review and approve or reject submissions by others.

Clicking on the first button will cause the familiar selection criteria
screen (Figure 3, Figure 8) to appear. The only difference is that there
are now two buttons: "Add Artifact" and "Modify Artifact".

## Add Artifact

To add an artifact to the database, the search terms must first be
selected using the Maintenance Selection Criteria form. Once the terms
are completely defined, the "Add Artifact" button is pushed resulting in
a screen that looks similar to Figure 9. The only differences are the
lack of a field for an e-mail address, and the addition of a field for a
password. The rest of the form is identical and is filled out in the
same manner.

Once the form has been filled out (including the current password),
pressing the "Add" button will cause the artifact to be added to the
database.

## Modify Artifact

To modify an artifact in the database, it must first be located. This is
done exactly as if the artifact were being searched for---the search
terms are set and the "Modify Artifact" button is pushed resulting in a
screen similar to Figure 11.

This screen is almost identical to Figure 5 with the addition of two
buttons "Modify" and "Delete". Pressing the "Delete" button will give
the maintainer a chance to review the artifact and then delete it.
Pressing the "Modify" button will present the maintainer with a filled
out form similar to Figure 9. This form can be edited and then submitted
to complete the modification process. Of course a password is necessary
to complete either of the modify or delete operations.

## Reviewing Submissions

The other function of maintenance mode is to allow the RISQ
administrators to review artifacts submitted by others and to add them
to the database with a minimal amount of effort. This process is started
when the "Review Submissions" button is pressed on the maintenance home
page.

The RISQ Submission Review page is shown in Figure 12. Every artifact
submitted is shown in a list. The maintainer can select one of the
artifacts and either delete it, or install it. In either event the
maintainer will be given a chance to review the artifact (in a form
similar to that shown in Figure 9) before proceeding. Either installing
or deleting a submitted artifact will result in it's being deleted from
the Submission Review page.

## Tool Demonstration Artifacts

We expect that making tool demonstrations available through the RISQ
system will be particularly important. There are several sorts of
demonstrations that are suitable for inclusion in the RISQ system. We
have identified three of them:

A slide-show a sequence of Web pages, containing text, graphics, and
perhaps even audio or video clips. These pages are linked together to
tell a story in much the same way a speaker would use overhead
transparencies.

A canned demo a sequence of Web pages, containing images captured from
the tool being demoed. Using the Web's image map facility, the viewer is
able to manipulate the images in much the same manner as he would
manipulate the real tool.

A ÒliveÓ demo a combination of a live demonstration of a tool on the
userÕs screen, and a set of Web pages which guide the user through the
demonstration.

Each of these is implemented in a different manner and we will treat
each of them in turn. The key to creating a successful demonstration for
use in the RISQ system is a carefully thought out storyboard. The
storyboard has its roots in the field of animation. It is a sequence of
pictures depicting the key events in the story. It is used by animators
to lay out how the story will progress linearly through time. This
ensures that the foils created by multiple animators can be assembled
into a coherent story. Since a good demonstration tells a story, the
storyboard is equally important to the RISQ system demonstration
designer.

The storyboard will be different depending on the type of demonstrations
being given. For the slide-show it will most likely be based on a
traditional slide presentation. For either of the other two kinds of
demonstrations, the storyboard will consist of a tree (or even graph) of
paths through the operation of the tool.

### Slide-Show

Given that a presentation already exists, creating a slide-show for the
RISQ system is easy. In its simplest form the slide-show consists of
on-line copies of the slides already in the presentation. However, to
make this demonstration useful, it is important to realize that the
slides are not enough. In a verbal presentation, it is usually the case
that the slides do not tell the whole story. Rather, it is the
combination of the slides and presenter that tell the story. For a
slide-show to work on the RISQ system, the words that the presenter adds
to the slides must be captured and displayed. One approach to doing this
is to attach notes at the bottom of each slide as shown in Figure 13.

For the slide-show to work, the viewer must have a way of progressing
through it. The buttons at the bottom of the slide accomplish this,
allowing the viewer to move forward, backward, or to a ÒhomeÓ page.

To take full advantage of the capabilities of the Web, the slides can be
embellished with links to other relevant items. For instance, the slide
in Figure 13 has a reference to POSIX. This reference could easily be
made to be a link to an on-line version of the POSIX standard, if it is
available. Also, due to the nature of the Web, the slide-show can become
multimedia, including both sound and full-motion video.

To make a slide-show presentation suitable for inclusion in the RISQ
system, we ask that:

-   The slide-show should be self-contained. While references to other
    documents are fine, it is helpful if all of the parts of the main
    presentation are available at a single location.

-   The first ÒslideÓ should include a brief abstract of the
    presentation, along with author contact information. In addition to
    any other links, it should include a link to the RISQ system
    demonstration evaluation form (to be discussed).

-   Similarly, the last ÒslideÓ should contain contact information, and
    a link to the RISQ system demonstration evaluation form.

### Live Demonstration

We discuss the creation of a live demonstration first, because in many
ways it is easier to create than a canned demonstration. These
instructions apply to live demonstrations of X Window System based
applications.

The first step in developing a live demonstration is to determine what
you would like to show the user. This will usually result in a sequence
of actions for the user to perform, and a set of descriptions of what
the user is seeing on the screen. The user actions, and the descriptions
should be captured on a set of Web pages that take the user through the
demonstration step-by-step.

For example, Figure 15 shows a sample screen, while Figure 14 shows the
accompanying Web page which describes it. As with the slide-show, the
Web pages can be embellished with links to other sites, as well as audio
and video segments.

For a live demonstration to be suitable for inclusion in the RISQ
system, we ask that

-   The demonstration should be self-contained. While references to
    other documents are fine, it is helpful if all of the parts of the
    main presentation are available at a single location.

-   The first Web page should include a brief abstract of the
    presentation, along with author contact information. In addition to
    any other links, it should include a link to the RISQ system
    demonstration evaluation form (to be discussed). It should also
    include a ÒformÓ allowing the user to set the location of his X
    display.

-   Similarly, the last Web page should contain contact information, and
    a link to the RISQ system demonstration evaluation form.

-   Since this is a live demonstration, you must take careful pains to
    ensure that the user viewing the demonstration cannot cause damage
    to the system running it.

The actual implementation of the live demo is very server and tool
specific.

### Canned Demonstration

The canned demonstration is a bit of a hybrid. It can be thought of as a
slide-show with some live demonstration capabilities added.

The first step in developing a canned demonstration, as for the live
demonstration, is to determine what you would like to show the user.
This will usually result in a sequence of actions for the user to
perform, and a set of descriptions of what the user is seeing on the
screen. The user actions, and the descriptions should be captured on a
set of Web pages that take the user through the demonstration
step-by-step.

The above paragraph is virtually identical to the one for the live
demonstration. The difference is that the canned demo relies on the
creation of a clickable image, using the Web's image map ability. Figure
16 shows a portion of a Web page from a canned demo. It happens to
correspond to the pages show in Figure 15 and Figure 14. As with the
slide-show, the Web pages can be embellished with links to other sites,
as well as audio and video segments.

For a canned demonstration to be suitable for inclusion in the RISQ
system, we ask that

-   The demonstration should be self-contained. While references to
    other documents are fine, it is helpful if all of the parts of the
    main presentation are available at a single location.

-   The first Web page should include a brief abstract of the
    presentation, along with author contact information. In addition to
    any other links, it should include a link to the RISQ system
    demonstration evaluation form (to be discussed).

-   Similarly, the last Web page should contain contact information, and
    a link to the RISQ system demonstration evaluation form.

### The Demonstration Evaluation Form

In order to evaluate the success of a particular demo, and the success
of the RISQ system, we need to receive feedback from our visitors. The
demonstration evaluation form is one mechanism to help accomplish this.
To create a demonstration evaluation form for a demo, you must modify a
copy of the file Òeval.htmlÓ which is available from the RISQ system in
the templates directory. Specifically you must:

-   Replace all instances of MYDEMO with the name of your demonstration.

-   Replaces MYADDRESS@MYHOST.DOM with your e-mail address. This will
    allow you to receive copies of user evaluations.

-   Modify the questions asked as necessary. The program that processes
    the form will take all fields presented to it and format it for
    e-mailing, so you can modify or add to the fields listed at will.
    Please do not remove any of the HIDDEN fields.

The modified version of the demonstration form is the one which your
demo should point to.

# Summary

World Wide Web (Web) technology has provided access to enormous amounts
of information on-line. As such it has a tremendous potential for use as
an aid to technology transfer. However, locating relevant information is
often difficult partially because of the volume of information
available. This report has described a new search engine which is
designed to allow the user to do efficient searches for information
within a specific domain. The initial implementation of the engine is in
the domain of high integrity software systems and is called RISQ which
stands for "Reference Information for Software Quality."

The RISQ system is an important part of CHISSA, the Center for High
Integrity Software Systems Assurance. CHISSA was organized by NIST's
Computer Systems Laboratory in 1994 as a collaborative approach for
government, industry, and academia to make available the technology
which is necessary for assuring high integrity software in an ever
growing number of applications.

The RISQ facility makes available a wide variety of artifacts related to
software quality in a highly organized manner. The overarching goal is
to help to transition technologies so that the best of them become
standard practices when developing software systems with a requirement
for high integrity.

The current RISQ facility is located on the World Wide Web ( Web) at
hissa.ncsl.nist.gov/risq/. By locating the facility on the Web, we
achieve the goal of making the material available to a large and
geographically diverse set of potential users.

While the initial database of artifacts is small, the system contains a
"submit artifact" capability to enable the user community to nominate
artifacts for inclusion. The RISQ administrators will verify that these
nominations meet the acceptance criteria.

Future work on the RISQ system may include:

-   refining the user features and taxonomy based upon user feedback

-   increasing the population of artifacts, and

-   developing a more efficient custom keyword search capability.

# Acknowledgments

The taxonomy created for the RISQ system represents a distillation of
taxonomies from a number of different sources. The resulting taxonomy is
a joint creation of the authors and Laura Ippolito of NIST, and Dr.
Marvin Zelkowitz of NIST and the University of Maryland, both of whom
have provided significant advice on the system design and implementation
as well. The original prototype of the RISQ system was designed and
implemented by Tony Cincotta of NIST. Many of the ideas incorporated
into the current system are based on his original design. Kathy Harvill
provided database population assistance.

[^1]: The Software Engineering Institute is sponsored by the U.S.
    Department of Defense.

[^2]: An artifact might be a document (in any of several forms), a tool
    capable of being demonstrated on-line, a program library, or even a
    video clip. The current implementation of RISQ allows for 15
    different artifact types. They will be fully described elsewhere in
    this document.