The **IUSR** Project: **I**ndustry **US**ability **R**eport

-   [White Paper in Word97 format](../WhitePaper.doc)

-   [White Paper in RTF format](../WhitePaper.rtf)

# Contents

[Introduction 1](#introduction)

[Purpose and Goals of the IUSR Project
2](#purpose-and-goals-of-the-iusr-project)

[Scope of the IUSR Project 2](#scope-of-the-iusr-project)

[Reporting Format 2](#reporting-format)

[Why a Common Format is Needed 2](#why-a-common-format-is-needed)

[Scope of the Reporting Format 2](#scope-of-the-reporting-format)

[Who Performs the Test 3](#who-performs-the-test)

[How Test Results Will Be Used 4](#how-test-results-will-be-used)

[The Pilot Study 4](#the-pilot-study)

[Pilot Duration 4](#pilot-duration)

[Pilot Description 4](#pilot-description)

[Usage and Liability Issues 5](#usage-and-liability-issues)

[History of the IUSR Project 5](#history-of-the-iusr-project)

[Relationship to Other Standards Organizations
5](#relationship-to-other-standards-organizations)

[Future Directions 6](#future-directions)

*This document is an overview of the **I**ndustry **US**ability
**R**eport (**IUSR**)) Project. The IUSR Project is designed to help
potential purchasers of software obtain information about the usability
of supplier products. There are two parts to the IUSR project:*

1.  *a proposed format for sharing usability information, and*

2.  *a pilot study which will allow both producer and consumer companies
    to test the effectiveness of using usability test results as
    procurement criteria and to verify the usefulness of the reporting
    format.*

# Introduction

Many factors affect a corporation's decision about which software
products to purchase. One key factor is the software's *usability*. In
simple terms, usability reflects:

1.  how easy the software is to learn,

2.  how easy the software is to use

3.  how productively users will work

4.  the amount of support users will need

Software developers employ a variety of techniques to ensure software
usability. In general terms, these techniques involve studying the users
to develop an understanding of their needs and iteratively refining
versions of the software based on *usability testing* results.

## Purpose and Goals of the IUSR Project

In making purchase decisions, companies and organizations have
traditionally had little indication of how usable a product would be or
how much training and support its users would need. The situation has
made it difficult to compare products, to plan for support, or estimate
total cost of ownership.

The goals of the IUSR Project are to:

-   Encourage software suppliers and consumer organizations to work
    together to understand user needs and tasks.

-   Develop a common usability reporting format for sharing usability
    data with consumer organizations.

-   Conduct a pilot plan to determine how well the usability reporting
    format works and to determine the value of using this format in
    software procurement.

## Scope of the IUSR Project

Usability testing can be valuable for a wide range of products, however
the IUSR Project is initially focused only on software. We do recognize
that the usability of hardware (printers, copiers, fax machines, etc.)
is important and often tightly integrated with software. The initial
focus on software was intended to narrow the focus of the initial
project so that a pilot study could be conducted. Extending the scope of
the reporting standard to include hardware and other products should be
addressed later in the project.

# Reporting Format

The IUSR Project is developing the initial version of a common format
for reporting usability test results, referred to as the Common Industry
Format (CIF). The intended reader is a usability or human factors
professional. The reporting format identifies the minimum format of
shared usability information to allow consumer organizations to evaluate
test results or replicate the tests if desired. Organizations that
participate in using the format may choose to provide more than the
minimum format.

## Why a Common Format is Needed

A format providing common information is needed because there are many
possible ways to report usability results. The purchase team may require
an evaluation of the validity and relevance of any test that it uses to
support its decision-making. A common format for reporting the test and
its results will facilitate evaluation of the test and the
interpretation of its results. It should also reduce misinterpretation
of the test results. A common format for reporting usability tests will
therefore benefit both suppliers and consumers of software products.

##  Scope of the Reporting Format

The detailed instructions for the report format are given in the
document entitled "Common Industry Format for Usability Test Reports".

The report format covers such topics as:

### The description of the product

As some products have several releases and versions, the product
description should include this information. The description should
describe basic functionality of the product and the intended users of
the product.

### The goals of the test

User testing may be performed to accomplish a variety of goals,
including problem identification or diagnosis, design alternatives
comparison, or to complete a summative test. The goal(s) of the reported
test should be clearly stated.

### The test participants

This section should include information on the number of users who
participated and the criteria by which they were selected.

### The tasks the users were asked to perform

This section should list the specific tasks that participants were asked
to perform during the study.

### The experimental design of the test

This section should explain the logical configuration of the test
conditions, including independent variables, what comparisons, if any,
are intended between groups, and how conditions which might contaminate
the results are brought under control.

### The method or process by which the test was conducted

This section should report the sequence of events that was actually
employed to instruct the test users, how well they followed it, any
intervention such as coaching, and materials used to give instructions
or ask questions of them.

### The usability measures and data collection methods

Usability measures may include objective measures of effectiveness,
efficiency, and how much effort is required to learn to use the product
successfully. Subjective data on user satisfaction may also be
collected.

### The numerical results

This section should report the data analysis procedures and summary
data. The results may contain summary statistics such as the mean,
range, standard deviation, and standard error of the estimate.

## Who Performs the Test 

The usability test may be performed by the software supplier using its
own usability group or by contracting with an independent testing
facility or a consumer organization. The consumer organization
interested in purchasing this software may accept the test results or
may replicate the test.

In many cases, the supplier organization will provide the results of the
last usability test conducted in the course of software development to
the consumer organization. This allows organizations that have a
usability testing program in place, to participate without incurring the
expense of additional usability testing. Proposed changes in the product
design that may occur as a result of the usability test may be listed in
the report documentation. This reduces the need of having to perform
additional usability test solely for the purpose of delivering the
report to the consumer organizations. However, proposed changes listed
in the report can in no way be taken as a guarantee that a supplier
organization will make those changes in the final shipping product.

## How Test Results Will Be Used

The procurement process for acquiring software often involves an
organization's purchasing department. Few purchasing departments have
the skills needed to evaluate usability test results. For this reason,
it is recommended that the consumer organization's usability group human
factors group should interpret the test results. This group will be
responsible for interpreting the test results and integrating their
evaluation into the procurement process.

On the supplier side, the sales process is typically the responsibility
of a sales and marketing organization which is not likely to have skills
in usability testing. It is recommended that the sales organization will
work with the supplier's usability or human factors group to identify an
appropriate study and package the results in the common format.

Because usability test data may be subject to misinterpretation if
incorrectly reported or published, it is assumed that in many cases the
results will be shared under a non-disclosure agreement between supplier
and prospective purchaser. Supplier organizations should treat both
large and small consumers equally in allowing access to this information
so as not to provide larger organizations with unfair advantage in
competitive situations. Specific negotiations regarding non-disclosure
are left to the individual organizations.

# The Pilot Study 

A Pilot will be conducted to determine the value of incorporating
usability results into decision-making for software purchases, and to
refine the procedures and reporting format. The initial focus will be on
software for upgrades, new products, or custom applications.

## Pilot Duration 

The Pilot will begin in 1999 and run for 2-1/2 years

## Pilot Description 

Suppliers and consumers who wish to participate in the formal pilot
study will enter into a Cooperative Research Agreement
([CRADA](http://www-i.nist.gov/r%2Bdhelpdesk/ag1.htm)) with the National
Institute of Standards and Technology (NIST). This mechanism will
facilitate sharing results from the pilot studies to refine the common
usability reporting format and the associated metrics. NIST will host
periodic workshops to discuss and consider refining the Pilot program.

During the formal pilot, pairs of Consumers and Suppliers who have
entered into a Cooperative Research Agreement with NIST will form the
teams that will work together on agreed upon software products. Each
pair of companies will commit to the following:

The Consumer Company will

1.  Consult with the Supplier to select a target software product with a
    minimum two-month advance notice

2.  Request usability test data from the Supplier on the target product.

3.  Apply the reported results in making decisions for software
    purchases.

4.  Gather internal data on overhead, productivity, and user
    satisfaction, and record it in a common format in the NIST database.

The Supplier Company will

1.  Conduct a usability test of the targeted software product.

2.  Report the test and its results in the format specified by the
    Workshop

3.  Put a nondisclosure agreement in place for sharing the usability
    test results with the Consumer corporation.

Interested companies are invited to participate in the pilot. Additional
information on the pilot study will be found in the document entitled
"Pilot Testing of the Common Industry Format for Usability Reporting"
(To be announced: approx 30 August 1999).

Companies may, of course, use the common reporting format on their own.
We recommend that consumer organizations wishing to use usability test
information as a factor in purchasing decisions develop a plan to
determine if there are measurable benefits in doing so. We encourage
companies to contact us for guidance on implementing plans in their
organizations. In addition, we encourage companies to contribute data
from efforts in their companies to a database maintained at NIST. This
data will be periodically reviewed to determine the effects, if any, of
these efforts.

# Usage and Liability Issues

The IUSR Project common reporting format is intended to simplify
information exchange between business organizations. There is no
certification that information supplied by the supplier is correct; nor
is there any certification regarding the effectiveness of any usability
efforts during the software development process. The working group
responsible for the creation of this document and accompanying common
usability reporting format is not liable should any of the information
be inaccurate. There is no liability on the part of the National
Institute of Standards and Technology (NIST) who provide a repository
for the common reporting format and the accompanying documentation and
any companies who participate in this program. (See accompanying [Letter
of Intent](../LetterOfIntent.html)).

# History of the IUSR Project

The IUSR Project is being conducted by a group of representatives from
both supplier and consumer companies hosted by the National Institute
for Standards and Technology (NIST).

During 1998, the IUSR Project held two workshops. The purpose of these
workshops was to bring together a group of usability experts from both
supplier and consumer corporations who would be able to:

-   Define a minimal common specification for reporting \"procurement\"
    usability tests.

-   Design a pilot study to evaluate the costs and benefits of including
    usability test results in software procurement.

Organizations who are producers or consumers of software products (many
organizations are both) are invited to participate in future workshops,
the pilot study and to comment on the project.

# Relationship to Other Standards Organizations

The IUSR Project is currently an independent effort, not connected with
any other standards body. We are working with representatives from other
standards bodies such as ISO, IEEE, and Human Factors and Ergonomics
Society to minimize overlap in our respective efforts and to leverage
each other's work. The SIGCHI liaison to Standards is a member of our
workshop. The USER Project common reporting format is consistent with
ISO 9241-11 and ISO 13407. We view this work as an implementation of
that ISO work.

We are currently investigating approaches to turning over a later
version of our work to a standards body when we have adequate validation
of the usefulness of this work.

# Future Directions

This first version of the IUSR Project will be expanded and refined if
there is sufficient interest on the part of the business community.
Participants in the pilot project will contribute results to a database
maintained at NIST. This information will be used to refine the product.
We encourage organizations that would like to participate or comment to
contact the NIST representative at <iuwinfo@nist.gov>