**Common Industry Specification for Usability --**

**Requirements**

**Draft Version 0.85**

**21-Jul-06**

Produced by the NIST Industry USability Reporting project:
[www.nist.gov/iusr](http://www.nist.gov/iusr)\
If you have any comments or questions about this document, please
contact: <iusr@nist.gov>

Editor

Nigel Bevan

Serco Usability Services

Contents

[1 Scope 5](#scope)

[1.1 Content of the CISU-R 5](#content-of-the-cisu-r)

[1.2 How the CISU-R can be used 5](#how-the-cisu-r-can-be-used)

[1.3 Who can use a specification of requirements based on the CISU-R
6](#who-can-use-a-specification-of-requirements-based-on-the-cisu-r)

[2 Conformance 6](#conformance)

[3 References 7](#references)

[4 Definitions 8](#definitions)

[4.1 usability 8](#usability)

[4.2 effectiveness 8](#effectiveness)

[4.3 efficiency 8](#efficiency)

[4.4 requirements 8](#requirements)

[4.5 satisfaction 8](#satisfaction)

[4.6 context of use 9](#context-of-use)

[4.7 measure (noun) 9](#measure-noun)

[4.8 measure (verb) 9](#measure-verb)

[4.9 measurement 9](#measurement)

[4.10 user 9](#user)

[4.11 user group 9](#user-group)

[4.12 scenarios of use 9](#scenarios-of-use)

[4.13 stakeholder 10](#stakeholder)

[4.14 goal 10](#goal)

[4.15 task 10](#task)

[5 Purpose 10](#purpose)

[5.1 General 10](#general)

[5.2 Iterative development of requirements
10](#development-of-requirements)

[5.3 Role in user centered design 11](#role-in-user-centered-design)

[5.4 Uses of the CISU-R 12](#uses-of-the-cisu-r)

[5.4.1 General 12](#general-1)

[5.4.2 Communication among members of the development team
12](#communication-among-members-of-the-development-team)

[5.4.3 Communication between customer and supplier
13](#communication-between-customer-and-supplier)

[5.4.4 Specification of requirements 13](#specification-of-requirements)

[5.4.5 Testing and verification 13](#testing-and-verification)

[5.5 High level and detailed usability requirements
13](#high-level-and-detailed-usability-requirements)

[5.6 Relationship to ISO 13407 14](#relationship-to-iso-13407)

[6 Usability requirements specification
14](#usability-requirements-specification)

[6.1 Overview 14](#overview)

[6.2 Context of use and scenarios 15](#context-of-use-1)

[6.2.1 Stakeholders 15](#stakeholders)

[6.2.2 Users 15](#users)

[6.2.3 Goals 16](#goals)

[6.2.4 Equipment: technical environment
16](#equipment-technical-environment)

[6.2.5 Physical and social environments
16](#physical-and-social-environments)

[6.2.6 Scenarios of use 16](#scenarios-of-use-1)

[6.2.7 Training 17](#training)

[6.2.8 Usability Metrics 17](#usability-metrics)

[6.3 Usability measures 17](#usability-measures)

[6.3.1 General 17](#general-2)

[6.3.2 Measures for scenarios of use 17](#measures-for-scenarios-of-use)

[6.4 Requirements for testing 18](#methods-for-testing)

[6.5 Role of usability requirements in development
19](#role-of-usability-requirements-in-development)

[6.5.1 Traceability 19](#traceability)

[6.5.2 Document control 19](#document-control)

[6.5.3 Detailed usability requirements
19](#detailed-usability-requirements)

**Introduction**

The purpose of the Common Industry Specification for Usability -
Requirements (CISU-R) is to provide a structure for:

-   Defining usability requirements in sufficient detail to make an
    effective contribution to design and development,

-   Defining usability criteria that can be empirically validated
    subsequently if needed.

This specification defines the content of usability requirements, rather
than requiring any specific process by which they are gathered.

This specification supports pro-active collaboration between a supplier
and customer to identify how a product can be effective, efficient and
satisfying, and can thus provide added value for end-users and their
work.

This specification complements the Common Industry Format for usability
test reports (CIF). If requirements specified using the CISU-R are
subsequently tested, the results can be reported in the CIF format. The
CIF format ensures that the results are documented in sufficient detail
to enable a potential acquirer to assess the validity of the data and
the extent to which the results are relevant to their needs.

The CIF and CISU-R take a broad approach to usability based on ISO
9241-11: the effectiveness, efficiency and satisfaction with which the
intended users can achieve their tasks in the intended context of
product use. The value of specifying these high level requirements is
that they relate closely to business requirements for successful use of
a product and increased productivity.

Other, more detailed usability requirements (for example for features of
the user interface) can be specified to complement the CISU-R
requirements.

Common Industry Specification for Usability - Requirements

# Scope

## Content of the CISU-R

The CISU-R specifies the information to be provided for the
specification of requirements for usability as defined in ISO 9241-11:
effectiveness, efficiency and satisfaction in a specified context of
use.

A specification of requirements based on the CISU-R has 3 parts (see
6.1):

a)  The **context of use**: intended users, their goals and tasks,
    associated equipment (including hardware, software and materials),
    the physical and social environment in which the product will be
    used, and examples of scenarios of use.

b)  **Usability measures**: effectiveness, efficiency and satisfaction
    measures for the main scenarios of use with target values where
    feasible.

c)  The **test method**: the method to be used to test whether the
    usability requirements have been met, and the context in which the
    measurements will be made. This provides a basis for testing and
    verification.

NOTE 1. Product requirements can be subdivided into functional and
non-functional (quality) requirements. ISO 9126 subdivides software
quality requirements into quality in use (the user\'s experience of the
quality of the system) and detailed quality requirements (see 5.5).

NOTE 2. Examples of other types of more detailed usability requirements
are given in informative Annex C.

NOTE 3. Examples of metrics for more detailed usability requirements can
be found in ISO/IEC 9126 parts 2 and 3.

The CISU-R does not cover specification of other requirements (such as
other non-functional (quality) requirements, non-usability functional
requirements, process requirement, business requirements etc).

The CISU-R does not prescribe specific measures nor does it prescribe
any specific development process.

## How the CISU-R can be used

The information in the CISU-R supports communication between the parties
involved to obtain a better understanding of the usability requirements
(see 5.4). The communication may be:

-   Among members of the development team, to specify requirements for
    use by the development team. (In this case the usability
    requirements may be incorporated into existing documents rather than
    forming a stand-alone document.)

-   Between the customer and supplier of a custom product, to define
    specific customer requirements.

-   Between a range of potential customers and a supplier of an off the
    shelf product, to define diverse requirements.

The specification can be used for either software or hardware products.

The CISU-R can be used to specify usability requirements as part of the
user centered development process described in ISO 13407 (see 5.3).

NOTE To ensure that the requirements are correct, other user centered
design activities (such as competitor analysis, focus groups or paper
prototyping) can be used early in the development process to obtain
feedback from users to iteratively refine requirements.

A specification of requirements using the CISU-R that includes usability
measures can be tested by carrying out a summative usability evaluation
to determine whether a product meets the specified usability criteria.
The results can be documented in the Common Industry Format for
usability test reports.

## Who can use a specification of requirements based on the CISU-R

A specification of requirements based on the CISU-R may for example be
used by individuals with the following roles to accomplish the following
objectives:

-   Usability professionals and designers in supplier organizations to
    decide whether usability requirements specified by a customer are
    realistic, and to plan how to ensure that a product meets these
    requirements.

-   Other technical professionals and managers in supplier organizations
    to decide whether to accept usability requirements.

-   Usability professionals in customer organizations to include
    usability in a Request for Proposals.

-   Other technical professionals and managers in customer organizations
    to assess if the usability requirements specified by supplier
    organization meet the needs of the customer organization.

# Conformance

A usability requirements specification conforming to the CISU-R may
either consist of information incorporated into existing documents
within an organization or may be produced as a stand-alone document
using the format specified in Annex A.

A usability requirements specification conforms to the CISU-R if it
complies with all the requirements (stated as "**shall**") in clause 6.
The recommendations (stated as "should") should be implemented whenever
appropriate. There are three levels of compliance to the CISU-R (as
summarized in Table 1 Levels of compliance).

**Level 1: Context of use**

Context of use provides information on intended use. To achieve Level 1
compliance, usability requirements **shall** specify

a)  The stakeholders

b)  The intended user groups.

c)  The main goals for each user group.

d)  The intended computing or other technical environment.

e)  The intended physical and social environments.

f)  Scenarios of use specifying how users carry out their tasks in a
    specified context.

g)  Any prerequisite documentation or training materials.

The following information should be provided:

a)  Which measures in Level 2 could be used to measure usability.

**Level 2: Usability measures.**

Usability measures provide evaluation criteria for defined scenarios of
use of the product, against which the usability of the product can be
evaluated. For Level 2 compliance, appropriate measures **shall** be
specified that may include:

a)  Performance measures, such as effectiveness and efficiency of
    achieving user goals.

b)  Satisfaction measures using an established questionnaire.

**Level 3: Test method.**

The test method specifies how it is planned to evaluate that the
usability requirements have been met.

  ------------- -------------------- ------------------ ------------------
  .             Part 1:\             Part 2:\           Part 3:\
                Context of use       Usability measures Test method

  Clause        6.2                  6.3                6.4

  Level 1                                              

  Level 2                                             

  Level 3                                             
  ------------- -------------------- ------------------ ------------------

**Table 1 Levels of compliance**

NOTE. Several versions of requirements may be produced with increasing
detail, as requirements are typically developed in an iterative fashion,
see 5.2.

Any statement of conformance **shall** indicate which categories of
information have been documented.

EXAMPLE 1 Level 1: The context of use requirements conform to ISO XXXXX.

EXAMPLE 2 Level 2: The requirements for usability measures conform to
ISO XXXXX.

EXAMPLE 3 Level 3: The usability testing requirements conform to ISO
XXXXX.

# References

ISO/IEC DTR 9126-2(2001): Software Engineering - Product quality - Part
2: External metrics

ISO/IEC DTR 9126-3(2001): Software engineering -- Product quality - Part
3: Internal metrics

ISO/IEC DTR 9126-4: (2001): Software engineering -- Product quality -
Part 4: Quality in use metrics

ISO 9241-11: 1998, Ergonomic requirements for office work with visual
display terminals (VDTs) -- Guidance on usability

ISO 13407 (1999) User centered design process for interactive systems.

ISO/IEC CD 25030: Software Product Quality Requirements and Evaluation
(SQuaRE) - Quality requirements and guide

ISO/IEC DIS 25062 Software engineering \-- Software product Quality
Requirements and Evaluation (SQuaRE) \-- Common Industry format for
usability test reports

# Definitions

For the purposes of this document, the following definitions apply.

## usability

the extent to which a product can be used by specified users to achieve
specified goals with effectiveness, efficiency and satisfaction in a
specified context of use

\[ISO 9241-11:1998\]

## effectiveness

the accuracy and completeness with which users achieve specified goals

\[ISO 9241-11:1998\]

## efficiency 

the resources expended in relation to the accuracy and completeness with
which users achieve goals

\[ISO 9241-11:1998\]

## requirements

expression of a perceived need that something be accomplished or
realized

NOTE The requirements may be specified as part of a contract, or
specified by the development organization, as when a product is
developed for unspecified users, such as consumer software, or the
requirements may be more general, as when a user evaluates products for
comparison and selection purpose.

\[ISO/IEC FCD 25030\]

## satisfaction

freedom from discomfort, and positive attitudes towards the use of the
product

\[ISO 9241-11:1998\]

## context of use

the users, tasks, equipment (hardware, software and materials), and the
physical and social environments in which a product is used

\[ISO 9241-11:1998\]

NOTE 1 The physical environment includes the workplace design
(furniture, posture and location) and the visual, auditory, thermal and
atmospheric conditions, and any health and safety hazards.

NOTE 2 The social environment includes social and organizational issues
including the work organization and structure, availability of
assistance and interruptions, presence of other people, job design and
autonomy.

## measure (noun)

variable to which a value is assigned as the result of measurement

NOTE The term "measures" is used to refer collectively to base measures,
derived measures, and indicators.

\[ISO/IEC 15939:2002\]

## measure (verb) 

make a measurement

\[ISO/IEC 14598-1:1999\]

## measurement

set of operations having the object of determining a value of a measure

\[ISO/IEC 15939:2002, based on the definition in International
Vocabulary of Basic and General Terms in Metrology, 1993\]

## user 

the person who interacts with the product

\[ISO 9241-11:1998\]

## user group

subset of intended users that are differentiated from other intended
users by factors such as age, culture, knowledge, skill, expertise, role
or responsibility that is likely to influence usability

NOTE : This may include current users, potential users, users with
disabilities, expected future users, users of the task output, and staff
who support or maintain the product.

## scenarios of use

how users carry out their tasks in a specified context.

## stakeholder

a party having a right, share or claim in a system or in its possession
of characteristics that meet that party's needs and expectations

\[ISO 15288:2002\]

## goal

an intended outcome

\[ISO 9241-11:1998\]

## task

the activities required to achieve a goal

NOTE 1: These activities can be physical or cognitive.

NOTE 2:Job responsibilities can determine goals and tasks.

\[ISO 9241-11:1998\]

# Purpose

## General

The purpose of the Common Industry Format for Usability - Requirements
Information (CISU-R) is to provide a structure for:

-   Defining usability requirements in sufficient detail to make an
    effective contribution to design and development.

-   Defining usability criteria that can be empirically validated.

Usability requirements specify the effectiveness, efficiency and
satisfaction with which the intended users can achieve their tasks in
the intended context of product use. As a tool for communication, the
CISU-R benefits both customer and supplier organizations.

##  Development of requirements

The CISU-R facilitates the iterative development of requirements through
the three levels of conformance:

-   First specifying the expected context of use: (level 1 conformance).
    This provides information on intended use that can be communicated
    to developers or customers.

-   Next performance and satisfaction measures and criteria can be
    specified for defined scenarios of use (level 2 conformance).

-   Finally the procedure to be used to test the requirements can be
    specified (level 3 conformance).

![](media/image1.jpeg){width="7.001388888888889in"
height="4.004166666666666in"}

As more information is gathered about requirements, the usability
requirements for the key user groups can be documented with increased
precision and specificity. After iterative rounds of discussion and
revision, the parties involved can finalize and approve the usability
requirements for the product. These requirements defined upfront then
form the basis for subsequent product development and usability
validation testing.

## Role in user centered design

For usability requirements specified according to the CISU-R to make an
effective contribution to design and development. The requirements need
to:

a)  Be based on accurate information about actual or potential users and
    their tasks

b)  Feed into other user-centered activities.

This document does not prescribe any specific user centered design
process, but it complies with the relevant process in ISO 13407.
Specifying and evaluating usability requirements are part of the wider
user centered design process as described in ISO 13407, and the process
of specifying quality requirements described in ISO/IEC 25030.

The following two activities that are part of a user centered design
process are necessary to ensure that appropriate information is included
in the usability requirements:

1.  Identify stakeholders

> All the usability requirements need to be covered by the complementary
> knowledge of the set of stakeholders. Stakeholders such as customers,
> users, suppliers and developers who are familiar with the targeted
> system or software can be identified and encouraged to participate in
> the requirements process.

2.  Elicit information from stakeholders about the context of use
    (users, tasks and environments) and quality requirements.

A typical user-centered development process that included level 2
compliance might include the following subsequent activities:

3.  Define scenarios of use.

4.  Document the context of use and scenarios of use in CISU-R format
    (level 1 compliance).

5.  Carry out a usability test of an existing system to obtain baseline
    measures of effectiveness, efficiency and satisfaction in these
    scenarios.

6.  Use the baseline data as a basis for establishing quantitative
    requirements for effectiveness, efficiency and satisfaction in these
    scenarios (level 2 compliance).

7.  Optionally specify how it is planned to test whether the
    requirements have been met (level 3 compliance).

8.  Refine the design by evaluating users carrying out tasks using
    low-fidelity prototypes.

9.  Evaluate prototypes of the working system.

10. Test the usability of the final system to check whether the
    requirements for effectiveness, efficiency and satisfaction have
    been met. The results can be documented using the CIF format.

Depending on the development environment, requirements may for example
be:

-   Iteratively elaborated as more information is obtained from
    usability activities such as paper prototyping during development;

-   Agreed by all parties before development commences, and subsequently
    only modified by mutual agreement.

However, the requirements for effectiveness, efficiency and satisfaction
in particular contexts of use are less likely to change, as they are
closely linked to meeting business and user needs.

## Uses of the CISU-R

### General

The CISU-R can be used initially to support communication between the
parties involved to obtain a better understanding of the requirements.
When the requirements are more completely defined it can be used as a
formal specification of requirements. These requirements can
subsequently be tested and verified.

### Communication among members of the development team

The CISU-R facilitates communication among the members of the
development or supplier organization. It is important that all concerned
groups in the supplier organization understand the usability
requirements before design begins. Benefits include reducing risk of
product failure, reducing the development effort, providing a basis for
estimating costs and schedules, serving as a basis for enhancement
through iteratively refining the product, and providing a means of
tracking evolving requirements.

a)  **Reducing risk of product failure.** Specifying performance and
    satisfaction criteria (level 2 conformance) derived from existing or
    competitor systems greatly reduces the risk of product failure as a
    result of releasing a product that is inferior to existing or
    competitor systems.

b)  **Reducing the development effort.** The CISU-R provides a mechanism
    for the various concerned groups in the customer's organization to
    consider all of the requirements before design begins and reduces
    later redesign, recoding, and retesting. Review of the requirements
    specified in the CISU-R can reveal misunderstandings and
    inconsistencies early in the development cycle when these issues are
    easier to correct.

c)  **Providing a basis for estimating costs**. Identifying usability
    requirements reduces the risk of unplanned re-work later in the
    development process, helping to ensure the cost estimates are more
    accurate.

d)  **Serving as a basis for enhancement and iteratively refining the
    product.** The CISU-R provides a foundation for continued production
    evaluation and refinement. Moreover, the CISU-R provides a means of
    tracking evolving requirements.

e)  **Tracking evolving requirements** by providing a format to document
    usability requirements.

### Communication between customer and supplier

A customer organization can specify usability requirements to accurately
describe what they need. In this scenario the CISU-R helps supplier
organizations understand what the customer wants and supports the
pro-active collaboration between a supplier and customer. It is
important for suppliers and customers to co-establish an upfront
understanding of the expected added value that the planned product will
provide for users and their work. The CISU-R provides a shared framework
for suppliers and customers to articulate, co-define, and iteratively
refine usability requirements as part of a collaboration process.

### Specification of requirements

When the product requirements are a matter for agreement between the
supplier and customer, the customer organization can specify:

-   The intended context of use (level 1 conformance).

-   The user performance and satisfaction criteria (level 2
    conformance).

-   The test procedure (level 3 conformance).

The CISU-R provides a baseline against which compliance can be measured.

### Testing and verification

As a tool to support testing and verification, the requirements
specified in the CISU-R can be tested and the results reported using the
Common Industry Format (CIF) for Usability Test Reports. Organizations
can develop their test and verification plans directly from the CISU-R.

## High level and detailed usability requirements

The CISU-R enables usability to be specified as an outcome of
interaction, measured as user performance and satisfaction. ISO 9126-1
defines this as a high level quality objective: the user's experience of
the "quality in use", which is determined not only by the features of
the user interface, but also by the extent to which the functionality
supports the user's tasks and by other non-functional requirements such
as acceptable computer performance and reliability.

More detailed requirements for features of the user interface also need
to be specified (see Annex C), but are not in themselves sufficient to
guarantee usability. The value of specifying the high level CISU-R
usability requirements is that they relate closely to business
requirements for successful use of a product and increased productivity.

## Relationship to ISO 13407

Specifying usability requirements according to this specification is
consistent with the recommendations of ISO 13407.

NOTE: The CISU-R directly supports the following objectives in ISO
13407:1999 7.3.2:

> "The specification of user and organizational requirements should
>
> a\) identify the range of relevant users and other personnel in the
> design,"
>
> "b) provide a clear statement of the human-centered design goals,"
>
> "c) set appropriate priorities for the different requirements,"
>
> "d) provide measurable criteria against which the emerging design can
> be tested,"
>
> "g) be adequately documented."

# Usability requirements specification 

## Overview

**Context of use (6.2):** The description of the context of use
**shall** include:

-   Stakeholders

-   Users

-   Goals

-   Equipment

-   Physical and social environments

-   Scenarios of use for the most important goals

EXAMPLE 1. Important goals may include those infrequent but critical or
exceptional tasks

**Usability measures (6.3)**: For a quantitative specification, the
usability measures **shall** include:

-   For the most important goals: Scenarios of use and Measures for each
    scenario.

-   At least one target value for each scenario.

> If there is any learning materials or training, measures should be
> given for the ease of learning of the training materials. For example,
> "Given 2 hours of training with the online training program, the user
> will be able to complete the item identification task under 1 minute
> and with fewer than 2 errors", or "After reading the owner's manual a
> user will be able to access and listen to a cellphone message upon the
> first try"

**Test method (6.4):** If requirements for testing are included, the
format in Annex B **shall** be used.

**User centered design (6.2 6.5):** Specifying and evaluating usability
requirements should form part of the wider user centered design process.

## Context of use 

### Stakeholders

The following information **shall** be provided:

a)  A list of all identified stakeholders who have a legitimate interest
    in the system throughout its life cycle.

```{=html}
<!-- -->
```
d)  The identified stakeholders roles and interests in regards to the
    product.

EXAMPLE 1. Role: marketing product manager; interest: business needs
solutions

EXAMPLE 2 Role: user/administrative assistant; interest facilitation of
administrative duties

e)  Whether or not a stakeholder is taken into consideration when
    identifying stakeholder requirements. If a stakeholder is not taken
    into consideration the rationale should be documented.

NOTE For usability requirements to be valid, information about the
context of use (users, tasks and environments) and requirements for
usability needs to be elicited from the stakeholders. Some stakeholders
needs are not stated explicitly, but implied from the context where the
product is to be used. Often stakeholders needs are identified from
analyzing business processes or how existing products are used.

In many situations needs only become evident when the customer or user
first tries to apply the product to their business process or task. Low
fidelity prototypes may be used to obtain feedback and refine the
requirements (see 5.3).

### Users 

The following information **shall** be provided:

a)  A list of expected user groups

```{=html}
<!-- -->
```
f)  Key characteristics and capabilities of each user group for which
    requirements are provided.

    Characteristics should be chosen to be relevant to the product's
    usability; they should be sufficiently detailed to ensure that test
    participants can be selected who are representative of the actual
    users.

    NOTE: Relevant factors may include experience in using computers,
    using similar products, familiarity with the task, frequency of
    usage, expertise, training, culture, nationality, age, special
    needs.

Requirements should be provided for the primary user groups, e.g. the
most frequent or critical groups for business purposes. Consideration
should be given to those critical users with disabilities.

### Goals

The main goals for each user group **shall** be listed, without
reference to any specific means of achieving them. The goal should be an
intended outcome of value to the user or business, for example:
accurately completing a particular form, locating the most relevant
information, or successfully setting up a PC.

NOTE: In some cases it may be possible to identify and list all the
usual goals. In other situations the list may be limited to the most
important tasks, representative tasks or those most frequently
encountered.

### Equipment: technical environment 

Any relevant aspects of the intended computing or other technical
environment **shall** be specified. Examples (particularly applicable to
software products) include:

a)  Hardware configuration: e.g., processor speed, memory size, network,
    storage, input and output devices.

```{=html}
<!-- -->
```
g)  Screen type (CRT or LCD), resolution and color depth. If relevant,
    also include display (monitor) size and whether multiple monitors
    are required.

h)  Print media size and print resolution.

i)  If visual interface elements (such as fonts) can vary in size, the
    size(s) used.

j)  Software configuration: e.g., Browser version, OS version,
    middleware, database if applicable.

k)  Assistive technologies available to users with disabilities.

l)  Documentation and support materials.

Examples specific to a hardware product might include:

m)  Wireless signal environment.

n)  Quality of connecting cables.

NOTE: These are only examples. For custom products the factors may be
quite precise, for consumer products they may specify an acceptable
range.

### Physical and social environments

Any aspects of the expected physical and social environments that may
influence usability **shall** be specified:

a)  Physical environment in which the product will be used, including
    location and relevant physical conditions, such as temperature or
    lighting.

```{=html}
<!-- -->
```
o)  Organizational environment, including group work dynamics, time
    pressures, supervision and support.

p)  Any physical or financial risks as well as health and safety issues.

### Scenarios of use

Scenarios specify how users carry out their tasks in a specified
context: they are examples of a user\'s activities, goals and
motivations for using the system in a specific situation.

### Training

If any documentation, training materials or course has to be studied
before the product is used, the following information **shall** be
specified:

a)  Which expected user groups are supported, and what previous
    knowledge or experience is required.

```{=html}
<!-- -->
```
q)  Which goals and tasks are included.

### Usability Metrics

The following information **should** be provided:

a)  The type of metrics (e.g., task completion rate, time on task, or
    subjective "ease of use" score).

b)  The units of the usability metrics (e.g., percentage of completed
    goals, mean time in seconds, or a five point scale).

## Usability measures

### General

Defining usability measures is optional, but to achieve level 2
compliance, appropriate measures **shall** be identified, and at least
one target value **shall** be given.

### Measures for scenarios of use

Select which tasks to elaborate with performance and satisfaction goals,
for example those that are most frequent or most critical to the
business or user. Indicate their relative importance.

Note. A common way to establish usability requirements is to first
measure the effectiveness, efficiency and satisfaction for the same
goals using an existing or competitor system, and then to set the
requirements for the new system to be as good as or better than the
existing system.

The goals that will form the basis of the requirements **shall** be
specified, with scenarios of usage. For each goal to be elaborated,
define one or more specific scenarios of usage that identify the
characteristics of the user and usage environment and the purpose for
carrying out the task. There may be separate scenarios for first use
(after any necessary training) and experienced use.

If any documentation, training materials or course has to be studied
before the product is used, measures should be given for the usability
of the training materials in at least one training scenario
(effectiveness, efficiency and satisfaction when completing training
goals).

For each scenario, appropriate measures **shall** be identified. These
may include:

a)  Effectiveness: the unassisted completion rate. The completion rate
    is the percentage of participants who completely and correctly
    achieve each goal. If goals can be partially achieved (e.g., by
    incomplete or sub-optimum results) then it may also be useful to set
    requirements for partial goal achievement, scored on a scale of 0 to
    100% based on specified criteria related to the value of a partial
    result (see A.4.1.1).

> EXAMPLE: The percentage of customers who can successfully complete a
> transaction on a web site, or the percentage of users who can
> successfully record an hour long TV program with a Video Cassette
> Recorder (VCR).
>
> NOTE: Although the requirements for effectiveness are based on
> error-free completion of the task, keeping track of the errors is
> particularly helpful in determining what aspects of the product need
> improvement.

r)  Efficiency: the mean time taken to successfully complete each goal.

s)  Satisfaction: mean score using an established questionnaire (see
    A.4.3).

t)  Relative user efficiency: the mean time taken by users who
    successfully achieve a goal divided by the time taken by an expert.

> NOTE: task time values are useful when making comparisons between
> products, while the relative user efficiency values are easier to
> interpret in isolation.

u)  Completion rate/mean time-on-task.

> NOTE: specifies the percentage of users who were successful (or
> percentage goal achievement) for every unit of time (see A.4.2.3).

For measures of effectiveness, efficiency and satisfaction, either
target values **shall** be given, or a note provided explaining that:

-   a target value will be provided later, or

-   why a target value cannot be established (for example for a
    completely new product), or

-   why no target value is needed (for example the measure is of low
    importance to the user or organization).

The relative importance of the measures for each goal should be stated.

Target values are the minimum acceptable value derived from the value
for an existing or competitor system value used for the same tasks and
goals, or based on expert judgment. Additional target values better than
the minimum acceptable value may also be specified.

Each target value may be given as:

-   a definite requirement, or

-   a provisional requirement subject to further negotiation, or

-   an objective for guidance.

## Methods for testing

Annex B provides the requirements for specifying the method to be used
to test whether the usability requirements have been met, and the
context in which the measurements will be made. If this information is
provided (level 3 conformance), the format in Annex B **shall** be used.

NOTE The performance and satisfaction criteria specified in 6.4 are the
values required in the intended context of use. The accuracy and
precision with which they are measured will be determined by the context
of measurement (the number of and types of users, way in which the tasks
are simulated, and the simulated working environment).

## Role of usability requirements in development

### Traceability 

The following information should be documented to assist the development
organization in maintaining and refining the requirements:

a)  The source of the requirements.

```{=html}
<!-- -->
```
v)  How the usability requirements support the business objectives for
    the product.

w)  Why the goals were selected.

    EXAMPLE The most frequent goals, the most troublesome goals, most
    critical to accomplishing business goals; most frequently requested
    requirement.

x)  The source of each goal.

    EXAMPLE Observation of customers using similar products, product
    marketing specifications

y)  Who requested this requirement.

z)  Whether the requirement is essential or desirable.

### Document control

The following information should be included to track different versions
of the requirements document each time a new version is issued:

a)  Version number

b)  Date of change

c)  Nature of change

d)  Person making and/or approving the change

### Detailed usability requirements

Detailed design requirements for the user interface (such as those in
Annex C) can be derived from the high level CIF requirements. The
development of these detailed requirements may be supported by the use
of standards such as ISO 9241 or ISO 9126.

Annex A:

###### Usability requirements format

This annex provides a format that may be used to document the usability
requirements. The format is consistent with the Common Industry Format
for Usability Test Reports (CIF).

####### Title page

The title page **shall** contain:

a)  Identify requirements as: Common Industry Format for Usability -
    Requirements Information v0.80.

```{=html}
<!-- -->
```
a)  Name the product and version that the requirements are for.

b)  Date the requirements were prepared.

c)  Company Name.

d)  Contact details for questions and/or clarifications.

The following information should be provided:

e)  Who produced the requirements.

####### Executive summary

The Executive Summary **shall** provide a high level overview of the
requirements. The intent of this section is to provide information for
people who may not read the technical body of this document.

-   This section **shall** begin on a new page and end with a page break
    to facilitate its use as a stand-alone summary.

-   The purpose of the requirements **shall** be explained. The level of
    conformance **shall** be stated.

-   A high level overview of the requirements **shall** be provided that
    includes:

    -   Name, version and brief description of the product.

    -   Summary of the type of users, tasks, any associated equipment
        (hardware, software, and materials), and the physical and social
        environments in which the product is intended to be used.

    -   A list of the task scenarios, with criteria for effectiveness,
        efficiency and satisfaction (for level 2 and 3 conformance).

-   The following information should be provided:

    -   Reason for and nature of the requirements.

####### Product details

######## Product description

The following information **shall** be provided:

a)  Formal product name and version.

```{=html}
<!-- -->
```
f)  The parts of the product for which requirements are being provided.

g)  The user groups for which the product is intended.

h)  Whether the product is intended to be "walk up and use", or whether
    any documentation, training materials or course has to be studied
    before the product is used.

The following information should be provided:

a)  The type of user work that is supported by the product.

```{=html}
<!-- -->
```
i)  Brief description of the environment in which the product should be
    used.

j)  Support for any groups with special needs.

k)  Reason for focusing on requirements for any subset of the product or
    usage environments.

######## Product objectives 

The following information **shall** be provided:

a)  The overall objectives for the product and specific objectives for
    any subset of usage.

```{=html}
<!-- -->
```
l)  Any known or intended functions and components that support key
    objectives.

####### Context of use and scenarios

Provide the information specified in 6.3, using the headings given in
6.3

####### Usability measures

######## Measures and target values for scenarios of use

Specify the goals that will form the basis of the requirements, with
scenarios of usage. Explain why these goals and scenarios were selected.

If any documentation, training materials or course has to be studied
before the product is used, measures should be given for the usability
of the training materials in at least one training scenario
(effectiveness, efficiency and satisfaction when completing training
goals).

For each scenario provide measures with target values that shall include
either:

-   target values for effectiveness, efficiency and satisfaction (as
    specified in 6.4.2), or

-   an explanation that a target value will be provided later, or

-   an explanation of why a target value cannot be established (for
    example for a completely new product).

    Each target value may be given as:

-   a definite requirement, or

-   a provisional requirement subject to further negotiation, or

-   an objective for guidance.

The relative importance of the measures for each goal should be stated.

######## Requirements for testing

The information specified in Annex B may also be provided for level 3
conformance.

Normative Annex B:

###### Usability test requirements format

This annex provides a format for specifying the method to be used to
test whether the usability requirements have been met. The format is
consistent with the Common Industry Format for Usability Test Reports
(CIF), and the results of the test can subsequently be documented using
the CIF.

Sufficient information **shall** be provided to enable the requirements
to be tested.

####### Users

The following information **shall** be provided:

a)  The total number of participants required.

```{=html}
<!-- -->
```
m)  Segmentation of user groups tested, if more than one.

n)  How participants are to be selected; so that they have the essential
    characteristics.

Characteristics should be chosen to be relevant to the product's
usability; they should be sufficiently detailed to ensure that the
participants are representative of the actual users.

The following information should be provided:

a)  Any expected differences between the participant sample and the user
    population.

    EXAMPLE Actual users will attend a training course which will have
    to be simulated.

```{=html}
<!-- -->
```
o)  Description of any groups with special needs.

####### Context of product use in the test

######## Goals

The following information **shall** be provided:

a)  Task scenarios to be tested.

```{=html}
<!-- -->
```
p)  Criteria for successful completion of each goal.

q)  Any task data to be given to the participants.

######## Test facility 

The following information should be provided:

a)  The setting, and type of space in which the evaluation will be
    conducted.

    EXAMPLES Usability lab, cubicle office, meeting room, home office,
    home family room, manufacturing floor

```{=html}
<!-- -->
```
r)  How relevant aspects of the intended usage environment will be
    simulated

######## Participant's computing environment 

The computing environment to be used **shall** be specified, including:

a)  Computer configuration, including model, OS version, required
    libraries or settings.

```{=html}
<!-- -->
```
s)  If used, browser name and version; relevant plug-in names and
    versions.

t)  Any relevant physical connectors or wireless connections.

######## Display devices 

The display devices to be used **shall** be specified:

a)  If screen-based, screen size, resolution, and color setting.

```{=html}
<!-- -->
```
u)  If print-based, the media size and print resolution.

v)  If visual interface elements, such as fonts, can vary in size,
    specify the size(s) to be used.

######## Satisfaction questionnaire

The following information **shall** be provided:

a)  The questionnaire to be used

####### Test procedure

The following information **shall** be provided:

a)  If more than one condition is being tested, the logical design of
    the evaluation.

```{=html}
<!-- -->
```
w)  The measures to be taken.

######## Procedure

The following information **shall** be provided:

a)  Operational definitions of measures.

```{=html}
<!-- -->
```
x)  Operational definitions of any independent variables or control
    variables.

y)  Maximum time limits on goals.

z)  Policies and procedures for interaction between tester(s) and
    subjects.

######## Participant general instructions 

The following information **shall** be provided:

a)  Instructions to be given to the participants (here or in an
    Appendix).

```{=html}
<!-- -->
```
a)  Instructions on how participants are to interact with any other
    persons present, including how they are to ask for assistance or
    interact with other participants, if applicable.

######## Participant task instructions

The following information **shall** be provided:

a)  Task instruction summary.

####### Usability metrics

The following information **shall** be provided:

a)  Metrics to be used for effectiveness.

```{=html}
<!-- -->
```
b)  Metrics to be used for efficiency.

c)  Metrics to be used for satisfaction.

Effectiveness and efficiency results **shall** be tested, even when they
are difficult to interpret within the specified context of use. In this
case, the requirements **shall** specify why the supplier does not
consider the metrics meaningful.

EXAMPLE Suppose that the context of use for the product includes real
time, open-ended interaction between close associates. In this case,
Time-On-Task may not be meaningfully interpreted as a measure of
efficiency, because for many users, time spent on this task is "time
well spent".

If it is necessary to provide participants with assists, efficiency and
effectiveness metrics **shall** be provided for both unassisted and
assisted conditions, and the number and type of assists **shall** be
included as part of the requirements results.

######## Effectiveness

Effectiveness relates the goals of using the product to the accuracy and
completeness with which these goals can be achieved. Common measures of
effectiveness include task completion rate, frequency of errors,
frequency of assists to the participant from the testers. It does not
take account of how the goals were achieved, only the extent to which
they were achieved. Efficiency relates the level of effectiveness
achieved to the quantity of resources expended.

######### Task completion rate

The completion rate is the percentage of participants who completely and
correctly achieve each goal. If goals can be partially achieved (e.g.,
by incomplete or sub-optimum results) then it may also be useful to set
requirements for partial goal achievement, scored on a scale of 0 to
100% based on specified criteria related to the value of a partial
result. For example, a spell-checking task might involve identifying and
correcting 10 spelling errors and the completion rate might be
calculated based on the percent of errors corrected. Another method for
calculating completion rate is weighting; e.g., spelling errors in the
title page of the document are judged to be twice as important as errors
in the main body of text. The rationale for choosing a particular method
of partial goal analysis should be stated, if such results are included
in the requirements.

The following measure **shall** be taken:

a)  The percentage of participants who completely and correctly achieve
    each goal.

######### Errors

Errors are instances where test participants did not complete the task
successfully, or had to attempt portions of the task more than once.
Scoring of data should include classifying errors according to some
taxonomy, such as in reference \[1\].

######### Assists

When participants cannot proceed on a task, the requirements
administrator sometimes gives direct procedural help in order to allow
the requirements to proceed. This type of tester intervention is called
an *assist* for the purposes of this requirements.

The following measure **shall** be taken:

a)  The unassisted completion rate (i.e. the rate achieved without
    intervention from the testers) as well as the assisted rate (i.e.
    the rate achieved with tester intervention) where these two metrics
    differ.

For example, if a participant received an assist on Task A, that
participant should not be included among those successfully completing
the task when calculating the unassisted completion rate for that task.
However, if the participant went on to successfully complete the task
following the assist, he could be included in the assisted Task A
completion rate. When assists are allowed or provided, the number and
type of assists must be included as part of the requirements results.

In some usability tests, participants are instructed to use support
tools such as online help or documentation, which are part of the
product, when they cannot complete goals on their own. Accesses to
product features which provide information and help are *not* considered
assists for the purposes of this requirements. It may, however, be
desirable to measure the frequency of accesses to different product
support features, especially if they factor into participants' ability
to use products independently.

######## Efficiency

######### Measures

Efficiency relates the level of effectiveness achieved to the quantity
of resources expended. Efficiency is generally assessed by the mean time
taken to achieve the task. Efficiency may also relate to other resources
(e.g. total cost of usage). A common measure of efficiency is time on
task.

The following measures **shall** be taken:

a)  The mean time taken to complete each task, together with the range
    and standard deviation of times across participants.

######### Relative user efficiency

The mean time taken by users who successfully achieve a goal divided by
the time taken by an expert.

######### Completion rate/mean time-on-task

The measure Completion Rate / Mean Time-On-Task is the core measure of
efficiency. It specifies the percentage of users who were successful (or
percentage goal achievement) for every unit of time. This formula shows
that as the time on task decreases, one would expect users to be more
successful. A very efficient product has a high percentage of successful
users in a small amount of time. This allows customers to compare fast
error-prone interfaces (e.g., command lines with wildcards to delete
files) to slow easy interfaces (e.g., using a mouse and keyboard to drag
each file to the trash).

######## Satisfaction

Satisfaction describes a user's subjective response when using the
product. User satisfaction may be an important correlate of motivation
to use a product and may affect performance in some cases.
Questionnaires to measure satisfaction and associated attitudes are
commonly built using Likert and semantic differential scales.

A variety of instruments are available for measuring user satisfaction
of interactive products, and many companies create their own. Whether an
external, standardized instrument is used or a customized instrument is
created, subjective rating dimensions such as Satisfaction, Usefulness,
and Ease of Use should be considered for inclusion, as these will be of
general interest to customer organizations.

A number of questionnaires are available that are widely used. They
include: ASQ \[5\], CUSI \[6\], PSSUQ \[6\], QUIS \[3\], SUMI \[4\], and
SUS \[7\]). While each offers unique perspectives on subjective measures
of product usability, most include measurements of Satisfaction,
Usefulness, and Ease of Use.

Suppliers may choose to use validated published satisfaction measures or
may submit satisfaction metrics they have developed themselves.

####### Statistical significance

The CIF (ISO 25062) specifies the statistical data to be given when
reporting results.

A requirement may be included to demonstrate a relationship with
previous test results, for example:

-   The test results for specified measures are at least as good as for
    the previous results (with 95% confidence).

-   The test results for specified measures are better than the previous
    results (with 80% confidence).

####### Appendices

The following information may be provided:

a)  Custom questionnaires, if used.

b)  Participant general Instructions (if not in the body of the
    requirements).

c)  Participant task Instructions.

Informative Annex B

###### Examples of other types of user and usability requirements

Other, more detailed usability requirements can be specified to
complement the CISU-R requirements (see 5.5). Examples include:

####### Design guidance

a)  Design principles

```{=html}
<!-- -->
```
d)  Human Factors and ergonomics

e)  Style guides

f)  Standards

####### Usability features

a)  Accessibility

```{=html}
<!-- -->
```
g)  Understandability

h)  Learnability

i)  Operability

j)  Attractiveness

####### Content and functions

a)  Functionality

```{=html}
<!-- -->
```
k)  Accessibility

l)  Content

-   Complete, accurate, up to date, style

-   Effectiveness (for learning)

-   Trust

-   Platform independence

Informative Annex C

###### Example requirements CIF

Common Industry Format for Usability - Requirements Information v.0.80

***Web Software Delivery Release 1***

*Requirements produced by: Sherman Jones.*

May 2005

> Address inquiries to: Sherman Jones
>
> CIFWorks
>
> Phone: 650-555-1212
>
> Email: sherman.jones@CIFWorks.com
>
> Address: 150 New Parkway
>
> New Pines, CA 91011

####### Executive Summary

The purpose of this document is to specify the usability requirements
for the first release of Web Software Delivery in a format that allows
CIFWorks' customers to review and provide input. This document conforms
to Level 3 of the CISU-R levels of conformance. The objective of the Web
Software Delivery product is to move away from physical shipments of
CD's and DVD's towards web based delivery of CIFWorks' products. The key
tasks involved are downloading the software, creating DVDs from the
software and setting up a staged (does not use DVDs) installation from
the downloaded files.

The intended users of Web Software Delivery are system administrators.
System administrators are experienced computer professionals whose
primary roles are the installation, configuration and administration of
multi-user, production software packages. They will use the product in a
corporate computing environment on a computer with a recent version of
the browser, sufficient disk space to download the product and a high
speed (10mbs or faster) internet connection.

The table below lists the key tasks associated with Web Software
Delivery and the usability requirements for effectiveness, efficiency
and satisfaction. A usability test will be conducted on the released
product to measure if the requirements for Effectiveness, Efficiency and
Satisfaction have been met.

Effectiveness is measured as unassisted task completion rate. Unassisted
task completion rate is the percentage of users who complete the tasks
without assists from the usability test administrator. Efficiency is the
average time to complete the task. Satisfaction is measured by the
industry standard SUMI questionnaire that is given at the end of the
study.

  ------------------ ---------------- ----------------- -----------------
  Task               Effectiveness    Efficiency        Satisfaction

  Download SITE      90% unassisted   10 minutes of     Post test SUMI
  guard software     task completion  user time (not    Score of 50.
                     rate             machine time).    

  Burn SITE guard    90% unassisted   15 minutes of     Post test SUMI
  software to DVD    task completion  user time         Score of 50
                     rate                               

  Prepare SITE guard 70% unassisted   25 minutes of     Post test SUMI
  software for       task completion  user time         Score of 50
  staged install and rate                               
  launch installer                                      
  ------------------ ---------------- ----------------- -----------------

#######  Introduction

The purpose of the Common Industry Format for Usability - Requirements
Information (CISU-R) is to enable usability requirements to be
objectively defined in a format that can be empirically validated. The
CISU-R supports pro-active collaboration between a supplier and
customer, to identify how a product can provide added value for
end-users and their work.

The CISU-R complements the Common Industry Format for usability test
reports (CIF). When requirements specified in the CISU-R format are
subsequently tested, the results can be reported in the CIF format.

The CIF and CISU-R take a broad approach to usability based on ISO
9241-11: the effectiveness, efficiency and satisfaction with which the
intended users can achieve their tasks in the intended context of
product use.

More detailed requirements for features of the user interface also need
to be specified, but are not in themselves sufficient to guarantee
usability. The value of specifying the high level CISU-R usability
requirements is that they relate closely to business requirements for
successful use of a product and increased productivity.

####### Product Details

-   Name and Version: CIFWorks Web Software Delivery Release 1

-   Parts for which requirements provided: All parts of the software

-   Intended user groups/populations: System Administrator

-   The product should not require any special training or documentation
    for its target user audience (System Administrators experienced with
    server software installation)

-   Type of user work supported by product: Downloading and preparing
    CIFWorks' Site Guard software for installation.

-   Usage environment: Medium to large size corporate and public sector
    companies.

-   Support for groups with special needs: Product needs to be coded to
    standards of section 508 of the Americans with Disabilities Act.

-   Reason for focusing on requirements for any subset of the product or
    usage environments? Not Applicable

####### Product Objective

The objective of the product is to ultimately replace mailing DVD's as
the default mechanism for shipping CIFWorks' software.

####### Context of Use

######## Users

System Administrators

System Administrators are experienced computer professionals whose
primary roles are the installation, configuration and administration of
multi-user, production, software . These users are typically
administrators of other enterprise software products such as operating
systems, databases and web servers.

System Administrator's educational backgrounds are varied as some
receive formal academic training in computer science or computer systems
management while others receive training on the job or through vendor or
3^rd^ party courses.

The most important characteristic of a System Administrator is on the
job experience. There are typically two routes by which these users gain
experience as a System Administrator:

1.  They extend their background from being a PC administrator or power
    user into the role of a System Administrator by industry training,

2.  They grow their skills in an organization under the tutelage of a
    Senior System Administrator.

For Web Software Delivery we expect System Administrators to have 1-10
years of experience in their job. In addition to software installation
and configuration, their day to day tasks would include operations
(adding users, adding disk space, monitoring an existing system) and
troubleshooting. They may also perform planning operations for future
software and hardware acquisitions.

######## Users with Disabilities:

Users with vision, hearing, and motor disabilities should be able to
function as a System Administrator with appropriate assistive devices
(e.g. screen readers).

####### Goals

The general goal is to obtain the appropriate version of the CIFWorks
Site Guard software for the appropriate platform in a form ready for
installation. Obtaining the software completes the purchasing process
but may or may not lead immediately to installation. In some cases
hardware or software has to be configured before installation of the
software begins. Users may immediately burn the electronic version of
the software onto DVD's, however this is not required for installation.
Some users will only install the product from the downloaded files
(referred to as a "staged install"). Based on this there are three tasks
associated with electronic delivery.

1.  Finding and downloading the Site Guard software via Web Software
    Delivery

2.  Create DVD's from the downloaded Site Guard software and begin the
    install.

3.  Prepare the software for an electronic or "staged" install and begin
    the install

In the case of tasks 2 and 3, the tasks end when the user has started
the installation successfully.

####### Equipment

To use Web Software Delivery the following computing environment will be
required.

-   A computer capable of running the browsers listed below.

    -   Internet Explorer (6.0 and higher)

    -   Netscape (6.0 and higher)

    -   Mozilla Firefox 1.1 and higher

-   10 gigabytes of disk space to download the zip files required for
    installation and create the staged installation.

-   500 megabytes of RAM

-   A 10mbs or faster internet connection.

-   A SVGA compatible monitor.

-   A DVD Burner

####### Physical and Social Environment

-   The physical environment will be either a corporate/ public sector
    office or a home office with a high speed internet connection (10
    mbs or faster). The home office may be used to download the software
    and burn CD's after hours.

-   Typically system administrators multi-task. It is expected that
    using web software delivery will be mixed in with other support
    tasks they are completing.

-   There is typically a time lag between acquiring and installing the
    software and rolling out the software into production. Because of
    that acquiring and installing the software will likely be part of a
    planning exercise without mission criticality pressure. However due
    to hacker attacks or other security concerns there may be times when
    obtaining Site Guard may be urgent.

####### Scenarios of use

Scenario 1: Download the Site Guard software and burn it to DVD

Ann is the lead system administrator at Biagra a producer of genetically
engineered wheat products. Last month her company was hit by a denial of
service attack by a group of organic fundamentalists. It was a rough
month and Ann ended up shutting down the company website leading to
unhappy customers and employees. It took no time to get the PO approved
for Site Guard and now she is on the hot-seat to get it installed on all
the companies servers. She is glad she can get the software online since
she does not want to wait for a physical shipment. Since her company's
firewall is closed, she has to download the software at home and take it
into the office and install it on her 2 primary servers.

After dinner she turns on Toy Story 2 for the kids and sits down in the
study to download the software. She has the printed out email from
CIFWorks and uses it to login to the Web Delivery site. After login, she
sees the software she purchased and online instructions for how to
download it. She uses Mozilla for the download as it has a built in
download manager. She begins the first DVD download and then goes to
check on the kids. Occasionally she peeks in on the computer to see how
the download is going. She is happy to see that it is progressing
nicely, although it is still a bit slow. 10 minutes later the first DVD
is downloaded so she starts the second DVD download. 20 minutes later
she has put the kids to bed and comes back in to see that the second DVD
download is complete.

Going back to the CIFWorks website she launches a java utility that will
help her burn the DVD's. It asks her for her computer information and
then asks her to insert a DVD. While the DVD is burning she sees a
progress indicator. It says 20 minutes to go, so she surfs the internet
for a while. Twenty minutes later she starts the second DVD. At the end
of the evening she has two DVD's ready for installation in the morning.

Back at work, she takes the DVD's and pops them into her development
environment. The development environment is a clone of the production
web server that she uses to test out installations. She opens a Unix
Shell and launches the installer. She is glad to see that the
installation starts successfully.

Scenario 2: Download the Site Guard software and create a staged
installation

Winston is a Senior System Administrator for the University of
South-East North Dakota. He administers the systems that run the
school's websites. He takes pride in maintaining a safe system despite
the many hackers at the school. As part of his 2005 budget, he got
approval to purchase Site Guard and deploy it to the server farm that is
used for the university website.

Today he received the email from CIFWorks with his license number and
instructions for downloading the software. Winston always likes to
install software from a staged environment. This makes it much faster to
take a master copy of the software and install it on multiple machines
in his server farm. He has designated a system called "stage_guard" to
be the stage for Site Guard. He prints out the email (he can't access it
from stage_guard) and walks over to the stage_guard machine. He logs
onto the CIFWorks website. He sees the software that he needs to
download and begins the first download. While the first download is
completing he looks around on the website for instruction on building a
staged install. He sees a link to instructions for staged install on the
side of the web page. He clicks the link and starts reading the
instructions. The instructions say he can either do it manually or
download a script. Being security conscious he does not trust "foreign"
scripts so he does it the manual way.

He follows the instructions and begins creating the directories he will
need for the install. It takes a while to create all the directories, as
he has to create them using command line UNIX and then set privileges
for them. He checks back on the first download and sees it is complete.
He starts the second download. While that is working he completes the
directories he needs and focuses on unzipping the files. The
instructions say to use a zip utility to unpackage and put the files in
the proper directories. He appreciates the fact that the instructions
list the exact command to use with his zip utility. He unzips the files
from the first download and puts them into the appropriate directory.
After that is complete he checks back in with the download and sees the
second download is complete. He unzips that file and puts it into the
second directory. He goes back to the first directory and runs the
installer. He is glad to see it has come up and he can proceed with the
installation.

####### Performance and Satisfaction Criteria

######## Goals

There are three core tasks involved with Web Software Delivery.

1.  Download SITE guard software

2.  Burn SITE guard software to DVD and launch installer

3.  Prepare SITE guard software for staged install and launch installer

Task 1 is the highest priority since successful delivery of the software
is a prerequisite to the other tasks. Task 2 has second priority since
it is the recommended approach for installing the software. Task 3 is
lowest priority as it is an optional way of installation.

The scenarios of usage for each task are listed below with the target
values for each task on effectiveness, efficiency and satisfaction. The
target values for Tasks 1 and 2 effectiveness is set at 90% unassisted
task completion rate. Unassisted task completion rate (UATR) is the
percentage of users who can successfully complete a given task without
assistance from a test administrator. The UATR was set at 90% since it
represents the highest unassisted task completion rate that could
realistically be achieved given user variability in a particular
usability test.

The UATR for Task 3 (staged install) was set to 70%. Building a staged
install is a more complex approach to Web Software Delivery that
involves manual steps. CIFWorks will support this approach with
instructions but given the variability in user environments and skill we
expect the success rate to be lower. We consider this acceptable since
there will be a smaller percentage of users who use staged install than
DVD install. In addition we will have a help desk available to walk
users through this approach.

The target value for efficiency (task completion time) was set for all
tasks based on a marketing research survey done on Web Software
Delivery. All times listed are for user interaction only (not machine
time). Given the high variability of networks and machines we could not
accurately predict machine time for these tasks.

The target value for Satisfaction was based on the SUMI questionnaire.
SUMI is an industry standard tool for measuring user satisfaction. The
SUMI target of 50 would be calculated by aggregating the post test SUMI
survey of each individual participant. The value of 50 is CIFWorks'
benchmark for user satisfaction as measured by SUMI.

For all tasks the user is a System Administrator (as defined in Context
of Use: Users) with between 1-10 years of experience administering
server based software. Tasks 1 and 2 may be completed in either a home
office with a high speed internet connection or in an office setting.
Task 3 would be completed in an office setting.

For tasks 1 and 2 the effectiveness requirements are definitive (not
subject to negotiation) due to corporate directive at CIFWorks. The
efficiency and satisfaction metrics for Tasks 1 and 2 are provisional
requirements, which are open to negotiation with customers of the
software. All requirements for Task 3 are objectives for guidance
intended for the internal development team. We cannot predict or control
the breadth of environments at customers' sites so we are keeping the
requirements at a high level.

Task 1: Download SITE guard software

Effectiveness: 90% unassisted task completion rate.

Efficiency: 10 minutes of user time.

Satisfaction: Total SUMI score of 50.

Task 2: Burn SITE guard software to DVD and launch installer

Effectiveness: 90% unassisted task completion rate.

Efficiency: 15 minutes of user time.

Satisfaction: Total SUMI score of 50.

Task 3: Prepare SITE guard software for staged install and launch
installer

Effectiveness: 70% unassisted task completion rate.

Efficiency: 25 minutes of user time.

Satisfaction: Total SUS or SUMI score of 50.

####### Document Control

Changes

  -----------------------------------------------------------------------
  Date              Change                     Person making or approving
                                               change.
  ----------------- -------------------------- --------------------------
                                               

  4/24/04           Version 1.0 of Document    Document Created
                    Created                    
  -----------------------------------------------------------------------

Requirements Contributors

  -----------------------------------------------------------------------
  Name                         Position
  ---------------------------- ------------------------------------------
                               

  Sherman Jones                Senior Usability Engineer

  Fred Taylor                  Product Manager
  -----------------------------------------------------------------------

Requirements Reviewers

  -----------------------------------------------------------------------
  Name                         Position
  ---------------------------- ------------------------------------------
                               

  Ann Smith                    VP of UI

  Alexander Kushenko           Product Boss
  -----------------------------------------------------------------------

Supporting Documents

  -----------------------------------------------------------------------
  Name                         Owner
  ---------------------------- ------------------------------------------
                               

  None                         
  -----------------------------------------------------------------------

Annex A: Usability test requirements format

This annex provides a format for specifying the method to be used to
test whether the usability requirements have been met. The format is
similar to Common Industry Format for Usability Test Reports (CIF), and
the results of the test can subsequently be documented using the CIF.
Sufficient information **shall** be provided to enable the requirements
to be tested.

####### Users

> A minimum of 8 users should be tested. All users should System
> Administrators with the following minimum characteristics. 1 year of
> experience as a system administrator managing server software.

####### Goals

> Refer to Performance and Satisfaction Criteria in the body of the
> document for Tasks and Goals.

####### Test facility 

> The test should be conducted in a usability lab that consists of a
> user room and a control room. The user room will be an office where
> the participants conduct the tasks. The lab should be equipped with
> audio and video monitoring and recording devices, as well as a scan
> converter that allows the display on the participant's monitor to be
> video recorded.

####### Participant's computing environment 

> The participants computing environment should meet the minimum set
> forth in the section "Equipment" in the main body of this document.

####### Display devices 

> As mentioned in the "equipment" section of this document, a minimum of
> a color VGA monitor with capability of 800x600 resolution is required.

####### Experimental Design

> There will be one order that the tasks are presented in since the
> second and third tasks are dependent on the first one. The six
> dependent measures for this study should be:

-   Unassisted task completion rate (the percentage of participants who
    completed a task without assistance from the tester)

-   Assisted task completion rate (the cumulative percentage of people
    who completed a task with zero, one or two assists from the tester)

-   Task time

-   Number of errors

-   Number of assists

-   Self-report satisfaction via SUMI measures.

####### Procedure

A minimum of eight sessions, one for each participant, **shall** be
conducted. The participant will work in the user room, while the test
administrator will observe from the control room behind a one-way
mirror. A system of microphones and speakers will allow the
administrator and user to communicate. At the beginning of each session,
participants will be given a consent form and non-disclosure agreement.
After completing the forms, participants will be instructed on the
nature of the testing session and the software.

Data collection will begin as the participant works through the tasks.
As participants work, a time-stamped datalog will be created. The
datalog will note errors, assists, user comments, and other observations
of interest such as navigation strategies. Upon completion of the tasks,
participants will fill out the SUMI. Participants will be thanked for
their time, and given their compensation. Compensation for this session
should be approximately \$150 given the experience of the participant
and the length of the study (2-3 hours).

####### Participant general instructions 

> Participants should be told that the study session consists of two
> parts: working on tasks with the application, and filling out some
> post-task surveys. Participants **shall** be instructed to read each
> task out loud, and to work through the tasks, "as efficiently as
> possible." Participants **shall** be assured that it is the software,
> not them, that is being tested, and to feel free to be honest in their
> feedback of the software.

####### Participant task instructions

\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*

**Task Scenarios -- Web Software Delivery**

You (Sandy Lee) are a System Administrator working at a company called
Marsh International that manufactures fencing products. Your company is
located in Fremont, California.

The Director of Information Technology (Dilip) has just signed a deal
with CIFWorks to use Site Guard to protect your corporate internet site.
The deal also includes support and upgrades.

Dilip emails you the ordering document that details the contract with
CIFWorks. As part of the deal with CIFWorks, he tells you that the
software will be delivered electronically via a download.

Exercise 1: Download the software

Using the information in the email open on your desktop, download the
database software that your company has purchased. Keep in mind the
following.

1.  You want to download the product for Solaris, Spark 64.

2.  You want the latest version of the product.

3.  You only want to download the software that you have licensed. You
    don't want any extras for trial purposes.

4.  Here is your company information

> Marsh International
>
> 500 Woodsy Lane
>
> Fremont, CA 96009

5.  Here is your information

> Sandy Lee
>
> Senior System Administrator
>
> sandy.lee@marsh.com

Exercise 2: Make DVDs

Now that you have downloaded the software you decide to make a set of
DVDs of the software for safe keeping. Using the burnable DVDs we have
provided to you create a set of DVDs that will allow you to install the
software from these DVDs.

Exercise 3: Install the database onto a Sun Machine without using the
DVDs

You have a development machine you want to install Site Guard onto
(dev2). However, the machine does not have a DVD drive. Install the
software that you have downloaded onto the Sun Machine without using the
DVDs you have just created.

Tester Note: The task here ends when the user successfully starts the
installer assuming they have completed all other steps correctly. They
don't have to run through the install process, as we are not testing the
installer.

\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*

####### Usability metrics

######## Effectiveness 

> [Completion Rate]{.underline}: Both assisted and unassisted task
> completion rates should be calculated. Unassisted completion rate is
> defined as the percentage of participants who successfully complete
> each task without any assistance from the tester. Assisted task
> completion rate is defined as the percentage of participants who
> complete each task with two or fewer assists from the tester (note
> that the assisted task completion rate includes the unassisted task
> completion rate). Participants fail to complete tasks with assistance
> if they require 3 or more assists, exceed the maximum time limit for
> each task, or if they give up on the task.
>
> [Errors]{.underline}: A deviation from the prescribed method to
> complete a task that did not result in goal attainment **shall** be
> counted as an error. Consulting the on-line help **shall** not be
> recorded as an error.

[Assists]{.underline}: Any intervention by the tester for any of the
following reasons **shall** be counted as an assist.

-   Preventing the test participant from performing an action that would
    result in a condition from which the task could not be completed.

-   Providing instruction in order for the test participant to complete
    the task or end the task as a result of surpassing the amount of
    time allocated to complete the task (more than 4 times the benchmark
    time).

######## Efficiency 

> Efficiency relates the level of effectiveness achieved to the quantity
> of resources expended. Efficiency is generally assessed by the mean
> time taken to achieve the task. Efficiency may also relate to other
> resources (e.g. total cost of usage). A common measure of efficiency
> is time on task. For this study efficiency **shall** be measured as
> the average amount of time required to complete the tasks.

######## Satisfaction 

> Satisfaction describes a user's subjective response when using the
> product. User satisfaction may be an important correlate of motivation
> to use a product and may affect performance in some cases.
> Questionnaires to measure satisfaction and associated attitudes are
> commonly built using Liker and semantic differential scales.

[SUMI scores]{.underline}: SUMI is an instrument specifically designed
to measure user satisfaction with commercial software. It has been used
with hundreds of thousands of participants and enables comparison with
industry norms for satisfaction with usability on 6 dimensions: global
(overall, efficiency, affect, helpfulness, control, and learning. An
industry average SUMI score on any index is 50.

Informative Annex E

###### Bibliography

1.  Norman, D.A. (1983) Design Rules Based on Analyses of Human Error.
    Communications of the ACM, 26(4), 254-258.

```{=html}
<!-- -->
```
1.  Chin, J. P., Diehl, V. A., and Norman, K. (1988). Development of an
    instrument measuring user satisfaction of the human-computer
    interface. In the Proceedings of ACM CHI '88 (Washington D.C.),
    213-218.

2.  Kirakowski, J. (1996). The software usability measurement inventory:
    Background and usage. In Jordan, P., Thomas, B., and
    Weerdmeester, B. (Eds.), Usability Evaluation in Industry. UK:
    Taylor and Francis.  

3.  Lewis, J. R. (1991). Psychometric Evaluation of an After-Scenario
    Questionnaire for Computer Usability Studies: the ASQ. SIGCHI
    Bulletin, 23(1), 78-81.

4.  Lewis, J. R. (1995). IBM Computer Usability Satisfaction
    Questionnaires: Psychometric Evaluation and Instructions for Use.
    International Journal of Human-Computer Interaction, 7, 57-78.

5.  Brooke, J. (1996). SUS: A "quick and dirty" usability scale.
    Usability Evaluation in Industry. UK: Taylor and Francis.

6.  Rubin, J. (1994) Handbook of Usability Testing, How to Plan, Design,
    and Conduct Effective Tests. New York: John Wiley & Sons, Inc.

7.  Dumas, J. & Redish, G. (1993), A Practical Guide to Usability
    Testing. New Jersey: Ablex Publishing Corp.

8.  Nielsen, J. & Landauer, T. K. (1993) A mathematical model of the
    finding of usability problems. In: CHI \'93. Conference proceedings
    on Human factors in computing systems, 206-213