NISTIR 7432 

Common Industry Specification 

for Usability - Requirements

Information Access Division

Information Technology Laboratory

June 2007

NISTIR 7432 6/28/2007 
NISTIR 7432 

Common Industry Specification for 

Usability - Requirements

Industry Usability Reporting Project Working Group

Mary Theofanos, Chair, Working Group

U.S. Department of Commerce

Carlos M. Gutierrez, Secretary 

Technology Administration

Robert Cresanti, Under Secretary of Commerce for Technology 

National Institute of Standards and Technology

 William Jeffrey, Director 

NISTIR 7432 6/28/2007 
Industry Usability Reporting Project 

In October of 1997, the U.S. National Institute of Standards and Technology (NIST) initiated an effort to 

increase the visibility of software usability, the Industry USability Reporting (IUSR) Project1. Cooperating in 

the IUSR project are prominent suppliers of software and representatives from large consumer organizations. 

The IUSR project is a product of the Visualization and Usability Group, Information Access Division, of 

Information Technology Laboratory. The IUSR goals are to:

the 

 

• Encourage software supplier and purchaser organizations to work together to understand user needs 

and tasks; 

• Perform standardization activities; 

• Create extensions/variations to the Common Industry Format (CIF) in areas such as requirements, 

formative testing, testing with hardware. 

Common Industry Specification for Usability- Requirements Working Group Members 

Nigel Bevan nigelbevan.com 

Jennifer Duncan Boeing 

Keith Butler Microsoft Global Services Automation 

Steve Fadden Landmark College 

George Hackman Oracle 

Scott Isensee BMC 

Sharon Laskowski NIST 

Arnie Lund Microsoft 

Emile Morse NIST 

Whitney Quesenbery Whitney Interactive Design 

Jean Scholtz Pacific Northwest National Laboratory, previously at NIST 

Brian Stanton NIST 

Kent Sullivan Microsoft Corporation 

Mary Theofanos NIST 

Irene Wong Oracle 

Reviewers 

1 Any commercial product identified in this document is for the purpose of describing a standards environment. This 

identification does not imply any recommendation or endorsement by NIST. 

Maria Alvarez-Ryan 

Glen Anderson 

Claire Caudry 

Nigel Claridge 

Catherine Courage 

Joe Dumas 

Lada Gorlenko 

Susan Harker 

John Innes 

Caroline Jarrett 

Steven Liu 

Sulata Mojumder 
NISTIR 7432 6/28/2007 

Ginny Redish 

Ahmed Seffah 

David Travis 

Gerard van Os 

Gregg Vanderheiden 

Anna Wichansky
Reference number of working document: IUSR: CISU-R v0.90 

Date: 2007-03-26

Committee identification: Industry Usability Reporting Community 

Common Industry Specification for Usability – Requirements 

Document type: International standard

Document subtype: 

Document stage: (20) Preparation

Document language: E 

Draft Version 0.90 

Editor: Whitney Quesenbery
NIST/IUSR – CISU-R 

Introduction ........................................................................................................................................................ix

vi 

Contents Page

1. Scope ......................................................................................................................................................1

1.1 General ....................................................................................................................................................1

1.2 Relationship to other standards...........................................................................................................1

1.2.1. General ....................................................................................................................................................1

1.2.2. Relationship to ISO/IEC 25062 – Common industry format for usability test reports ....................1

1.2.3. Relationship to ISO 13407 - Human centred design processes for interactive systems ...............2

1.2.4. Relationship to ISO/IEC 9126 – Software Engineering – Product Quality .......................................2

1.2.5. Relationship to ISO/IEC 25030 – Quality requirements and guide ...................................................2

2. Conformance ..........................................................................................................................................2

2.1 General ....................................................................................................................................................2

2.2 Level 1 Compliance ...............................................................................................................................3

2.3 Level 2 Compliance ...............................................................................................................................3

2.4 Level 3 Compliance ...............................................................................................................................4

3. Normative references ............................................................................................................................4

4. Terms and definitions ...........................................................................................................................4

5. Purpose ...................................................................................................................................................6

5.1 General ....................................................................................................................................................6

5.2 Communication among stakeholders..................................................................................................7

5.3 Benefits of usability requirements .......................................................................................................7

5.4 Development of requirements ..............................................................................................................8

5.5 Role in user centered design................................................................................................................8

6. Usability requirements specification ...................................................................................................9

6.1 General ....................................................................................................................................................9

6.2 Context of use ......................................................................................................................................10

6.2.1. General ..................................................................................................................................................10

6.2.2. Stakeholders ........................................................................................................................................10

6.2.3. User groups ..........................................................................................................................................10

6.2.4. Goals .....................................................................................................................................................11

6.2.5. Technical environment (equipment) ..................................................................................................11

6.2.6. Physical and social environments .....................................................................................................11

6.2.7. Scenarios of use ..................................................................................................................................12

6.2.8. Training .................................................................................................................................................12

6.3 Usability performance and satisfaction criteria ...............................................................................12

6.3.1. Identifying usability criteria (Level 1) ................................................................................................12

6.3.2. Defining target values for criteria (Level 2) .......................................................................................13

6.3.3. Defining specific criterion values (Level 3) .......................................................................................13

6.3.4. Including use of learning materials in the criteria...........................................................................13

6.4 Methods for testing..............................................................................................................................14

6.4.1. Identifying test methods (Level 1)......................................................................................................14

6.4.2. Specifying a preliminary usability test method (Level 2) ................................................................14

6.4.3. Specifying a complete usability test method (Level 3) ....................................................................14

6.5 Traceability ...........................................................................................................................................15

6.6 Document control ................................................................................................................................15

Annex A (informative) Usability requirements format .................................................................................16

A.1 Introduction ..........................................................................................................................................16

A.2 Title page ..............................................................................................................................................16

A.3 Executive summary .............................................................................................................................16

A.4 Product details .....................................................................................................................................17
NIST/IUSR – CISU-R

vii

A.4.1. Product description............................................................................................................................. 17

A.4.2. Product objectives .............................................................................................................................. 17

A.5 Context of use and scenarios ............................................................................................................ 17

A.6 Usability performance and satisfaction criteria and values ........................................................... 17

A.7 Requirements for testing .................................................................................................................... 18

Annex B (Informative) Examples of performance and satisfaction measures .......................................... 19

B.1 Introduction .......................................................................................................................................... 19

B.2 Effectiveness ....................................................................................................................................... 19

B.2.1. Task completion rate........................................................................................................................... 19

B.2.2. Errors .................................................................................................................................................... 19

B.2.3. Assists .................................................................................................................................................. 20

B.3 Efficiency .............................................................................................................................................. 20

B.3.1. Measures .............................................................................................................................................. 20

B.3.2. Relative user efficiency ...................................................................................................................... 20

B.3.3. Completion rate/mean time-on-task .................................................................................................. 20

B.4 Satisfaction .......................................................................................................................................... 21

Annex C (normative) Specification of a usability test protocol ................................................................. 22

C.1 Introduction .......................................................................................................................................... 22

C.2 Users ..................................................................................................................................................... 22

C.3 Context of product use in the test ..................................................................................................... 22

C.3.1. Test facility ........................................................................................................................................... 22

C.3.2. Participant’s computing environment ............................................................................................... 23

C.3.3. Display devices.................................................................................................................................... 23

C.4 Test procedure..................................................................................................................................... 23

C.4.1. Scenarios to be tested ........................................................................................................................ 23

C.4.2. Participant general instructions ........................................................................................................ 23

C.4.3. Participant task instructions .............................................................................................................. 23

C.5 Performance and satisfaction metrics .............................................................................................. 24

C.5.1. Criteria and measurements ................................................................................................................ 24

C.5.2. Metrics for effectiveness, efficiency, and satisfaction .................................................................... 24

C.5.3. Statistical significance ........................................................................................................................ 24

C.6 Appendices .......................................................................................................................................... 24

Annex D (informative) An example of a user centered design process .................................................... 25

Annex E (Informative) Examples of other types of user and usability requirements ............................... 26

E.1 Introduction .......................................................................................................................................... 26

E.1.1. Design guidance .................................................................................................................................. 26

E.1.2. Usability features................................................................................................................................. 26

E.1.3. Content and functions ........................................................................................................................ 26

Annex F (Informative) An example of a Level 1 requirements document .................................................. 27

F.1 Title Page .............................................................................................................................................. 27

F.2 Executive Summary ............................................................................................................................ 27

F.3 Product Details .................................................................................................................................... 28

F.4 Product Objective ................................................................................................................................ 28

F.5 Context of Use ..................................................................................................................................... 28

F.5.1. Users ..................................................................................................................................................... 28

F.5.2. Goals ..................................................................................................................................................... 29

F.5.3. Equipment ............................................................................................................................................ 29

F.5.4. Physical and Social Environment ...................................................................................................... 29

F.5.5. Scenarios of use .................................................................................................................................. 30

F.6 Performance and Satisfaction Criteria .............................................................................................. 31

F.6.1. Goals ..................................................................................................................................................... 31

F.6.2. Criteria .................................................................................................................................................. 31

F.7 Methods for Testing ............................................................................................................................ 31

Annex G (Informative) An example of a Level 2 requirements document.................................................. 32

G.1 Title Page .............................................................................................................................................. 32

G.2 Executive Summary ............................................................................................................................ 32

G.3 Product Details .................................................................................................................................... 33
NIST/IUSR – CISU-R 

viii 

G.4 Product Objective ................................................................................................................................34

G.5 Context of Use......................................................................................................................................34

G.5.1. Users .....................................................................................................................................................34

G.5.2. Goals .....................................................................................................................................................34

G.5.3. Equipment ............................................................................................................................................35

G.5.4. Physical and Social Environment ......................................................................................................35

G.5.5. Scenarios of use ..................................................................................................................................35

G.6 Performance and Satisfaction Criteria ..............................................................................................36

G.6.1. Goals .....................................................................................................................................................36

G.6.2. Criteria ..................................................................................................................................................37

G.7 Methods for Testing ............................................................................................................................37

G.7.1. Users .....................................................................................................................................................37

G.7.2. Goals .....................................................................................................................................................37

G.7.3. Test facility ...........................................................................................................................................37

G.7.4. Participant’s computing environment ...............................................................................................38

G.7.5. Display devices ....................................................................................................................................38

G.7.6. Experimental Design ...........................................................................................................................38

Annex H (Informative) An example of a Level 3 requirements document ..................................................39

H.1 Title Page ..............................................................................................................................................39

H.2 Executive Summary .............................................................................................................................39

H.3 Product Details.....................................................................................................................................41

H.4 Product Objective ................................................................................................................................41

H.5 Context of Use......................................................................................................................................41

H.5.1. Users .....................................................................................................................................................41

H.6 Goals .....................................................................................................................................................42

H.7 Equipment ............................................................................................................................................42

H.8 Physical and Social Environment ......................................................................................................42

H.9 Scenarios of use ..................................................................................................................................42

H.10 Performance and Satisfaction Criteria ..............................................................................................44

H.10.1. Goals .....................................................................................................................................................44

H.10.2. Criteria ..................................................................................................................................................44

H.11 Annex A: Usability test requirements format ...................................................................................45

H.11.1. Users .....................................................................................................................................................45

H.11.2. Goals .....................................................................................................................................................45

H.11.3. Test facility ...........................................................................................................................................45

H.11.4. Participant’s computing environment ...............................................................................................46

H.11.5. Display devices ....................................................................................................................................46

H.11.6. Experimental Design ...........................................................................................................................46

H.11.7. Procedure .............................................................................................................................................46

H.11.8. Participant general instructions .........................................................................................................46

H.12 Usability metrics ..................................................................................................................................47

H.12.1. Effectiveness ........................................................................................................................................47

H.12.2. Efficiency ..............................................................................................................................................48

H.12.3. Satisfaction...........................................................................................................................................48

Bibliography ......................................................................................................................................................49
NIST/IUSR – CISU-R

ix

Introduction 

The Common Industry Specification for Usability - Requirements (CISU-R) helps usability professionals, 

product managers, and others working in product design and development to create usability requirements. It 

sets standards for specifying usability requirements, which include three types of information: 

− The context of use: the intended users, their goals and tasks, associated equipment, and the 

physical and social environment in which the product can be used. 

− Performance and satisfaction criteria: measures of usability for the product. 

− The test method and context of testing: the method to be used to test whether the usability 

requirements have been met and the context in which the measurements will be made. 

Underlying the specific goals of the standard is a deeper goal of creating useful and usable products that allow 

users to complete their tasks efficiently, effectively, and with satisfaction. To support this deeper goal, the 

CISU-R provides a structure for requirements to help stakeholders: 

− Document the context of use for a product, including definitions of the expected technical, physical, 

and social environments, user groups, goals for use of the product and scenarios of use, as defined in 

Clause 6.2. 

− Write usability requirements in sufficient detail to make an effective contribution to design and 

development. (A suggested outline for usability requirements is provided in Annex A.) 

− Relate usability requirements to stakeholder requirements (including user, customer, and business) 

for successful use of a product and increased productivity. 

− Define usability criteria that can be empirically validated. (Annex B provides examples of usability 

criteria.) 

− Define the method for testing the product against the criteria. (Annex C identifies the information to be 

specified for the test method.) 

− Create requirements that are useful throughout the product design and development process, 

providing input to the design process early in a project and adding more detailed information about 

criteria and methods as it is available. 

Usability requirements created with the CISU-R can be updated throughout a product’s lifecycle. A project 

may choose to create complete requirements, including all information specified in the CISU-R, or may adopt 

a level of conformance that meets stakeholder and business goals. 

The CISU-R can help integrate the creation of usability requirements into any design or development process. 

(Annex D contains an example of a user centered design process that incorporates the development of 

usability requirements under the CISU-R.) 

− Requirements can meet one of three levels of compliance with the CISU-R. Each level builds on the 

previous one, allowing the usability requirements to be developed over time, with increasing detail and 

precision. This approach allows the CISU-R to be used for all types of projects, from the smallest or 

most informal to complex or formally specified products. 

− Customers, users, and development teams can use the CISU-R as a communications tool to 

understand and specify requirements within an organization or as the basis for a contractual 

relationship between companies. The three levels of compliance provide the flexibility to match the 

level of detail and formality of the usability requirements to business needs. 
NIST/IUSR – CISU-R 

− Requirements may also include indirect measures of usability, such as post-release satisfaction tests, 

or business performance requirements, such as completed transactions. These criteria are not 

intended for evaluation using a summative usability test, as required by the third level of conformance, 

but can be useful in establishing business or performance requirements. 

− The CISU-R recognizes that usability requirements are just one type of product requirement. They 

complement functional, business, process, safety, and non-functional (quality) requirements. The 

information gathered in creating usability requirements can be used to help define other user 

requirements (for example, for features of the interface). (Annex E lists other types of user and 

usability requirements.) 

The CISU-R focuses on the information needed to create usability requirements that are useful for design and 

development, rather than dictating a specific process or user centered design activities. (Annex E contains an 

example of a user centered design process that incorporates the CISU-R.) 

− In situations where there are already processes and document formats in place, stakeholders can 

incorporate the CISU-R into that process by ensuring that the information specified here is included in 

any usability requirements. 

− If there is no established format, requirements can use the information specified in Annex A as a 

guide. (Examples of all three levels are included in Annexes F, G, and H as a reference.) 

The process of creating requirements with the CISU-R can be incorporated into many user centred design 

processes. For example, in ISO 13407, specifying requirements is one activity in an iterative process. The 

CISU-R provides details for how to complete this activity, as shown in Figure 1. 

Figure 1 — Requirements in the User-Centered Design Process 

To ensure that the requirements are correct, other user centered design activities (such as competitive 

analysis, interviews, surveys, focus groups, field studies, task analysis, benchmark usability tests, or paper 

prototyping) can be used early in the development process to obtain feedback from users to iteratively refine 

requirements. 

The CISU-R complements other user centered design standards. 

− It uses the definition of usability in ISO 9241-11: the effectiveness, efficiency, and satisfaction with 

which the intended users can achieve their tasks in the intended context of product use. 

x 
NIST/IUSR – CISU-R

xi

− Usability requirements created using the CISU-R are consistent with the recommendations of ISO 

13407. 

− Usability tests conducted to measure whether usability requirements have been met can be reported 

using the Common Industry Format (CIF) (ISO/IEC 25062). 

− The development of usability requirements may be supported by the use of standards such as ISO 

9241 or ISO 9126. 

NIST/IUSR: CISU-R

© ISO 2002 – All rights reserved 1

Common Industry Specification for Usability - Requirements

1. Scope 

1.1 General 

The CISU-R describes how to specify usability requirements for both hardware and software products. It is 

consistent with the definition of usability in ISO 9241-11: effectiveness, efficiency, and satisfaction in a 

specified context of use. 

Usability requirements based on the CISU-R have three parts: 

− The context of use: intended users, their goals and tasks, associated equipment, and the physical 

and social environment in which the product will be used, and examples of scenarios of use. 

− Performance and satisfaction criteria: measures of usability for the product. 

− The test method and context for testing: the means of determining whether the usability 

requirements have been met. 

These requirements can be specified to different levels of detail, to meet three levels of conformance to this 

standard, at different stages of a product design and development process. 

The CISU-R only applies to usability requirements, one of many types of requirements for hardware and 

software products. Other types of requirements include non-usability functional requirements, process 

requirement, business requirements, safety requirements, and non-functional (quality) requirements. 

Usability requirements specified with the CISU-R may be used alone, or may be incorporated into a broader 

requirements gathering process or other software or systems development process. 

The CISU-R does not prescribe any specific development process. The CISU-R does not prescribe specific 

measures for usability.The CISU-R does not prescribe any specific user centered design process, but it is 

consistent with the relevant process in ISO 13407. 

1.2 Relationship to other standards 

1.2.1. General 

The CISU-R can be used in conjunction with other related standards. 

1.2.2. Relationship to ISO/IEC 25062 – Common industry format for usability test reports 

As a tool to support testing and verification, the requirements specified in the CISU-R can be tested and the 

results reported using the ISO/IEC 25062 (Common Industry Format (CIF) for usability test reports). 

Organizations can develop their test and verification plans directly from the CISU-R. 
NIST/IUSR – CISU-R 

2 

1.2.3. Relationship to ISO 13407 - Human centred design processes for interactive systems 

IS0 13407 defines one of the stages of user centered design as the specification of requirements. Usability 

requirements created using the CISU-R are consistent with the recommendations of ISO 13407. 

The CISU-R directly supports the following objectives in ISO 13407:1999 7.3.2: 

 “The specification of user and organizational requirements should 

a) identify the range of relevant users and other personnel in the design, 

b) provide a clear statement of the human-centered design goals, 

c) set appropriate priorities for the different requirements, 

d) provide measurable criteria against which the emerging design can be tested, 

g) be adequately documented.” 

1.2.4. Relationship to ISO/IEC 9126 – Software Engineering – Product Quality 

Usability requirements, as defined in the CISU-R, are used in conjunction with other detailed design 

requirements, such as those in Annex B. The development of these detailed requirements may be supported 

by the use of standards such as ISO 9241 or ISO/IEC 9126 

The CISU-R enables usability to be specified as an outcome of interaction, measured as user performance 

and satisfaction. ISO/IEC 9126-1 defines this as a high level quality objective: the user’s experience of the 

“quality in use”, which is determined not only by the features of the user interface, but also by the extent to 

which the functionality supports the user’s tasks and by non-functional requirements such as acceptable 

computer performance and reliability. 

Examples of metrics for usability requirements can be found in ISO/IEC 9126 parts 2 and 3. 

1.2.5. Relationship to ISO/IEC 25030 – Quality requirements and guide 

Usability requirements in this document correspond to the quality in use requirements in ISO/IEC 25030. 

2. Conformance 

2.1 General 

A usability requirements specification conforming to the CISU-R may either consist of information incorporated 

into existing documents within an organization or may be produced as a stand-alone document, which may 

use the format in Annex A. 

A usability requirements specification conforms to the CISU-R if it meets the requirements (stated as “shall”) 

in Clause 6 for one of three levels of compliance. The recommendations (stated as “should”) should be 

implemented whenever appropriate. 

The three levels of compliance also allow conforming requirements to be written at an appropriate level of 

detail for each product and development process. Each of these levels builds on the previous level(s), 

allowing requirements to be specified iteratively, adding additional information as the requirements reach the 

next level of compliance, as shown in Figure 2. 

The three levels also allow the requirements to be specified iteratively, adding more detailed information about 

criteria and methods as it is available, and providing useful input throughout the product design and 

development process. Each level can be completed at an appropriate stage or checkpoint. 
NIST/IUSR – CISU-R

EXAMPLE 1 In a process based on ISO13407, requirements might be developed to Level 1 during the “Understand 

Context of Use” activity; to Level 2 during the “Specify Requirements” activity; and to Level 3 during the “Evaluate 

Designs” activity. 

EXAMPLE 2 Requirements might be developed to Level 1 during work on conceptual prototypes; to Level 2 as the 

user interface design is specified; and to Level 3 for summative testing of the product. 

Figure 2: Information included in requirements at each level of compliance 

Any statement of conformance shall indicate which level of compliance has been achieved. 

EXAMPLE This document conforms to Level X of the CISU-R. 

2.2 Level 1 Compliance 

Requirements meeting Level 1 compliance include a description of the context of use and identify both the 

types of criteria and methods to evaluate them that can be used to measure a product’s success in meeting 

the requirements. At Level 1, evaluation methods are identified as an input to planning, and will be specified in 

more detail in later levels. 

For Level 1 compliance, usability requirements shall:

− Specify all of the information describing the context of use (see Clause 6.2). 

− Identify relevant criteria for use of the product (see Clause 6.3.1). 

− Identify appropriate evaluation methods (see Clause 6.4.1). 

2.3 Level 2 Compliance 

Requirements meeting Level 2 compliance add more detailed information about performance and satisfaction 

criteria, specifying the criteria to be used and identifying target values (or a method for determining those 

values). At this level, requirements also specify the evaluation methods including a description of how the test 

will be conducted. 

3
NIST/IUSR – CISU-R 

4 

For Level 2 compliance, usability requirements shall:

− Specify all of the information describing the context of use (see Clause 6.2). 

− Specify evaluation criteria, and target values (see Clause 6.3.2). 

− Specify a usability evaluation method, and identify the context for evaluation (see Clause 6.4.2). 

2.4 Level 3 Compliance 

For Level 3 compliance, requirements include a specification of target values, that have been validated, for the 

performance and satisfaction criteria, and a complete protocol for the usability test method. 

NOTE 1 For some products, usability requirements may not reach Level 3 compliance until close to the end of the 

design and development process; for others, Level 3 compliance may be part of a contractual agreement. 

NOTE 2 The usability test specified in Level 3 may be reported using ISO/IEC 25062 (Common Industry Format (CIF) 

for usability test reports). If use of this report format is planned, the protocol should include a way to meet all requirements 

of this standard. 

For Level 3 compliance, usability requirements shall: 

− Specify all of the information describing the context of use (see Clause 6.2). 

− Specify target values for each criterion and user group (see Clause 6.3.3). 

− Specify a complete protocol for the usability test method, as specified in Annex C (see Clause 6.3.3). 

3. Normative references 

The following referenced documents are indispensable for the application of this document. For dated 

references, only the edition cited applies. For undated references, the latest edition of the referenced 

document (including any amendments) applies. 

ISO/IEC DTR 9126-2 (2001): Software Engineering - Product quality - Part 2: External metrics 

ISO/IEC DTR 9126-3 (2001): Software engineering – Product quality - Part 3: Internal metrics 

ISO/IEC DTR 9126-4 (2001): Software engineering – Product quality - Part 4: Quality in use metrics 

ISO 9241-11 (1998): Ergonomic requirements for office work with visual display terminals (VDTs) – Guidance 

on usability 

ISO 13407 (1999) Human centred design processes for interactive systems 

ISO/IEC CD 25030: Software Product Quality Requirements and Evaluation (SQuaRE) - Quality requirements 

and guide 

ISO/IEC DIS 25062: Software engineering -- Software product Quality Requirements and Evaluation 

(SQuaRE) -- Common Industry Format for usability test reports 

4. Terms and definitions 

This document uses the following definitions. 
NIST/IUSR – CISU-R

5

4.1 

usability 

the extent to which a product can be used by specified users to achieve specified goals with effectiveness, 

efficiency, and satisfaction in a specified context of use 

[ISO 9241-11:1998] 

4.2 

effectiveness 

the accuracy and completeness with which users achieve specified goals 

[ISO 9241-11:1998] 

4.3 

efficiency 

the resources expended in relation to the accuracy and completeness with which users achieve goals 

[ISO 9241-11:1998] 

4.4 

requirements 

expression of a perceived need that something be accomplished or realized 

[ISO/IEC FCD 25030] 

NOTE The requirements may be specified as part of a contract, or specified by the development organization, as 

when a product is developed for unspecified users, such as consumer software, or the requirements may be more 

general, as when a user evaluates products for comparison and selection purpose. 

4.5 

satisfaction 

freedom from discomfort, and positive attitudes towards the use of the product 

[ISO 9241-11:1998] 

4.6 

context of use 

the users, tasks, equipment (hardware, software, and materials), and the physical and social environments in 

which a product is used 

[ISO 9241-11:1998] 

NOTE 1 The physical environment includes the workplace design (furniture, posture, and location) and the visual, 

auditory, thermal, and atmospheric conditions, as well as any health and safety hazards. 

NOTE 2 The social environment includes social and organizational issues including the work organization and structure, 

availability of assistance, and interruptions, presence of other people, job design, and autonomy. 

4.7 

measure (noun) 

variable to which a value is assigned as the result of measurement 

[ISO/IEC 15939:2002] 

NOTE The term “measures” is used to refer collectively to base measures, derived measures, and indicators. 

4.8 

measure (verb) 

make a measurement 

[ISO/IEC 14598-1:1999] 

4.9 

measurement 

set of operations having the object of determining a value of a measure 

[ISO/IEC 15939:2002, based on the definition in International Vocabulary of Basic and General Terms in 

Metrology, 1993] 
NIST/IUSR – CISU-R 

6 

4.10 

user 

a person who interacts with the product. 

[ISO 9241-11:1998] 

NOTE In this standard users may include current users, potential users, users with disabilities, expected future users, 

users of the task output, and staff who support or maintain the product. 

4.11 

user group 

subset of intended users that are differentiated from other intended users by factors that are likely to influence 

usability, such as age, culture, knowledge, skill, expertise, role or responsibility. 

4.12 

scenarios of use 

descriptions of how users carry out their tasks in a specified context. 

4.13 

stakeholder 

a party having a right, share or claim in a system or in its possession of characteristics that meet that party’s 

needs and expectations. 

[ISO 15288:2002] 

NOTE Stakeholders as used in this standard includes users and user groups 

4.14 

goal 

an intended outcome 

[ISO 9241-11:1998] 

4.15 

task 

the activities required to achieve a goal 

[ISO 9241-11:1998] 

NOTE 1 These activities can be physical or cognitive. 

NOTE 2 Job responsibilities can determine goals and tasks. 

5. Purpose 

5.1 General 

The purpose of the Common Industry Specification for Usability - Requirements (CISU-R) is to provide a 

structure and format for defining usability requirements, which can be incorporated into any user centered 

design process. Usability requirements specify the effectiveness, efficiency, and satisfaction with which the 

intended users can achieve their tasks in the intended context of product use. 

Usability requirements developed with the CISU-R can be incorporated into more comprehensive 

requirements documents, or can be created as a stand-alone document. In either case, to make an effective 

contribution to design and development, the requirements can: 

− Be based on accurate information about actual or potential users and their tasks 

− Provide sufficient detail to contribute to other user-centered activities. 
NIST/IUSR – CISU-R

7

− Define usability criteria that can be empirically validated. 

5.2 Communication among stakeholders 

The information in the CISU-R supports communication among stakeholders of a product to obtain a better 

understanding of the usability requirements for a product. 

For example, the communication can be: 

− Among members of a development team, to specify requirements for use in creating or updating a 

product. 

− Between the customer and supplier of a custom product, to define specific customer requirements. 

− Between potential customers and a supplier of a commercially or publicly available product, to define 

diverse requirements. 

Use of the CISU-R to create usability requirements helps supplier organizations understand what the 

customer wants and supports the pro-active collaboration between a supplier and customer. The CISU-R 

provides a mechanism for all stakeholders, in either supplier or customer organizations, to consider the 

requirements before design begins, and can reduce the need for later redesign, recoding, and retesting. 

Review of the requirements specified in the CISU-R can reveal misunderstandings and inconsistencies early 

in the development cycle when these issues are easier to correct. 

Examples of how information in the CISU-R can be used include: 

− Customer organizations (or customers within an organization) can specify usability requirements to 

accurately describe what they need. 

− Usability professionals and designers can decide whether usability requirements specified by a 

customer are realistic, and to plan how to ensure that a product meets these requirements. 

− Other technical professionals and managers can decide whether to accept usability requirements. 

− Customer organizations can include usability requirements in a contract or request for proposals (or 

specify that they be created and approved during the course of the project.). 

− Stakeholders in customer organizations can assess whether the usability requirements meet their 

needs. 

5.3 Benefits of usability requirements 

Defining usability requirements, and using them as a measure of the success of a product can provide many 

benefits for a product. The ultimate benefit is in increased usability. Benefits for the product and development 

process include: 

− Increasing the likelihood of product success. Specifying performance and satisfaction criteria 

derived from existing or competitor systems increases the likelihood that a product will meet user 

needs, and greatly reduces the risk of product failure as a result of releasing an inferior product. 

− Reducing the development effort. The CISU-R provides a mechanism for the various concerned 

groups in the customer’s organization to consider all of the requirements before design begins and 

reduces later redesign, recoding, and retesting. Review of the requirements specified in the CISU-R 

can reveal misunderstandings and inconsistencies early in the development cycle when these issues 

are easier to correct. 

− Reducing later costs. Identifying usability requirements reduces the risk of unplanned re-work later 

in the development process, helping to ensure that the cost estimates are more accurate. 
NIST/IUSR – CISU-R 

8 

− Providing a baseline for usability criteria. When detailed, quantitative measures are included in the 

requirements, they provide a baseline against which the success of a product in complying with the 

requirements can be measured. 

− Serving as a basis for enhancement and iteratively refining the product. The CISU-R provides a 

foundation for continued production evaluation and refinement. 

− Tracking evolving requirements by providing a format to document usability requirements. 

5.4 Development of requirements 

The CISU-R facilitates the iterative development of requirements. The three parts of a requirement can be 

documented with increased precision as more information is gathered about the key user groups and their 

needs. 

Within each level, a product team might develop the three parts of a requirement in stages: 

− First specifying the expected context of use. This provides information on users, the environment, and 

the intended use of the product that are the basis for usability requirements. This information can be 

communicated to designers, developers or customers for review and confirmation. 

− Next performance and satisfaction criteria can be specified for defined scenarios of use. These criteria 

are used as input to the design process, and to create testable measures to validate whether a 

product meets the requirements. 

− Finally the procedure for a usability test for the requirements can be specified. 

After each iterative round of discussion and revision, the stakeholders can approve the usability requirements 

for the product. These requirements then form the basis for continued product development and testing. 

5.5 Role in user centered design 

Specifying and testing usability requirements are part of the wider user centered design process. Two 

activities that are part of a user centered design process are necessary to ensure that appropriate information 

is included in the usability requirements: 

− Identify stakeholders: All the usability requirements need to be based on knowledge from identified 

stakeholders, including all users groups as well as other interested parties. 

− Elicit information from stakeholders: Stakeholders such as customers, users, suppliers, and 

developers can be encouraged to participate in the requirements process, and provide information 

about the context of use (users, tasks, and environments), and quality requirements. 

To ensure that the requirements are correct, other user centered design activities (such as competitive 

analysis, interviews, surveys, focus groups, field studies, task analysis, benchmark usability testing or paper 

prototyping) can be used to obtain feedback from users to iteratively refine requirements. 

Usability requirements specified with the CISU-R are an important part of a product specification, and can be 

used with more detailed requirements for features of the user interface to guarantee usability. (see Annex B). 

The value of specifying the high level CISU-R usability requirements is that they relate closely to business 

requirements for successful use of a product and increased productivity. 
NIST/IUSR – CISU-R

9

6. Usability requirements specification 

6.1 General 

Usability requirements based on the CISU-R have three parts: the context of use, the performance and 

satisfaction criteria, and the test method. All usability requirements include all three parts, but there are three 

levels of compliance. Some projects may choose to develop requirements iteratively, reaching higher levels of 

compliance over time. As more information is gathered about requirements, the usability requirements for the 

key user groups can be documented with increased precision and specificity as the parties involved finalize 

and approve the usability requirements for the product. 

The context of use: a description of the intended users, their goals, associated equipment (including 

hardware, software, and materials), the physical and social environment in which the product will be used, and 

examples of scenarios of use. (see Clause 6.1) 

− A complete description of the context of use as specified in Section 6.2 shall be included in all 

usability requirements (Levels 1, 2, and 3). Additional detail may be added during the design and 

development process. 

Performance and satisfaction criteria: ways in which the usability of the product can be measured, 

specified for the main scenarios of use. They may include target values, or may specify how these values will 

be identified (see Clause 6.3). These criteria can be specified to three levels of completeness and precision: 

− Level 1: the requirements shall identify the criteria appropriate for successful use of the product, and 

the relative importance of each and identifying those that are practical to measure directly (see Clause 

6.3.1). 

− Level 2: the requirements shall specify target values for the criteria, or a range of acceptable values, 

(see Clause 6.3.2). 

− Level 3: the requirements shall include specific values for measures of user performance and 

satisfaction to be tested, or a method of determining those values (see Clause 6.3.3). 

The test method: how the product will be tested to determine whether the usability requirements have been 

met. The specification of the test method may include only an identification of the appropriate test methods, 

may describe a method and identify the test context, or may include a complete specification of a test method 

(see Clause 6.4). These criteria can be specified to three levels or detail: 

− Level 1: the requirements shall identify the types of methods that are appropriate for the product and 

criteria selected (see Clause 6.4.1). 

− Level 2: the requirements shall include a preliminary test method and the criteria to be evaluated 

(see Clause 6.4.2). 

− Level 3: the requirements shall include a complete test protocol for a user performance test, as 

specified in Annex C (see Clause 6.4.3). 

These levels are intended to ensure that usability requirements can be developed with the CISU-R for all 

types of projects, from the smallest or most informal to the most complex or formally specified products. The 

three levels of compliance allow usability requirements to be written to an appropriate level of completeness 

and precision for the product. At the same time, they encourage all projects to reach at least Level 1 

compliance. This level includes all three parts of a requirement (context of use, criteria, and test methods), but 

allows for less detailed criteria and test method specifications. Each level contributes to good usability. The 

CISU-R allows an appropriate level of compliance to be selected for each project. 

EXAMPLE 1 A project to update the shopping cart on an e-commerce web site might aim for Level 1 compliance with 

a usability requirement to increase successful checkouts (task completion). The planned evaluation methods might also 

include actual site performance as measured by server log analysis, a checklist of requirements, or usability testing. 
NIST/IUSR – CISU-R 

10 

EXAMPLE 2 A project to develop a customer service application might aim for Level 2 compliance with a usability 

requirement that at least 90 % customer service representatives who have completed their initial training can complete a 

new customer order accurately, to be tested with sample scenarios in a lab setting or under simulated field conditions. 

EXAMPLE 3 A project to develop a hand-help field service measurement tool might aim for Level 3 compliance by 

including a complete protocol for a summative usability test, and further refining the criteria and criterion values. . 

In addition, a usability requirements document should include information to assist in maintaining and refining 

the requirements. This information includes: 

a) Information to support the traceability of these requirements, including their source and process. 

b) Document version control information 

6.2 Context of use 

6.2.1. General 

For all levels of compliance with this standard, the context of use shall include descriptions of: 

a) Stakeholders 

b) User groups 

c) Goals and tasks 

d) Technical environment (equipment) 

e) Physical and social environments 

f) Scenarios of use for the most important goals 

NOTE Important goals can include infrequent but critical or exceptional tasks 

6.2.2. Stakeholders 

Usability requirements shall include a description of all groups of people who are identified as stakeholders, 

including: 

a) A list of all groups who have a legitimate interest in the product throughout its life cycle. 

b) Their roles and interests in the product. 

EXAMPLE 1. Role: marketing product manager; interest: business needs solutions 

EXAMPLE 2 Role: user/administrative assistant; interest facilitation of administrative duties 

c) If any of the stakeholders will not be considered in identifying requirements, the rationale for 

excluding them. 

NOTE Information about the context of use (users, tasks, and environments) and requirements for usability should be 

elicited from the stakeholders. Some stakeholders needs are not stated explicitly, but implied from the context where the 

product is to be used. Often stakeholders needs are identified from analyzing business processes or how existing products 

are used. 

6.2.3. User groups 

Usability requirements shall include a list of all anticipated user groups. 
NIST/IUSR – CISU-R

11

Requirements should be provided for the user groups who use a product most frequently or who are critical for 

business purposes. Key characteristics and capabilities of these primary user groups shall be defined. 

a) Characteristics should be relevant to the product and context of use. 

NOTE Relevant factors may include experience in using computers, use of similar products, familiarity 

with the task, frequency of usage, expertise, training, culture, nationality, or age.

b) Characteristics should be defined in enough detail that they can be used to select representative 

usability test participants. 

c) Characteristics should include identification of any disabilities that may affect the ability to use the 

product. 

NOTE Requirements may only become evident when a user first tries to apply the product to their business 

process or task. 

6.2.4. Goals 

The main goals for each user group shall be listed, without reference to any specific means of achieving 

them. The goals should be an intended outcome of value to the user or business, for example: accurately 

completing a particular form, locating the most relevant information, or successfully setting up a computer. 

NOTE It may be possible to identify and list all the user goals for some products. In other case, the list may be 

limited to the most important, frequent or representative goals. 

6.2.5. Technical environment (equipment) 

Any relevant aspects of the intended or anticipated computing or other technical environment shall be 

specified. 

Examples (particularly applicable to software products) include: 

a) Hardware configuration: e.g., processor speed, memory size, network, storage, input, and output 

devices. 

b) Screen type (CRT or LCD), resolution, and color depth. If relevant, also include display (monitor) size 

and whether multiple monitors are required. 

c) Print media size and print resolution. 

d) Whether visual interface elements (such as text) can vary in size and the size(s) available. 

e) Software configuration: e.g., browser version, operating system version, middleware, database. 

f) Assistive technologies available to users with disabilities. 

g) Documentation and support materials. 

EXAMPLE Configuration information specific to a hardware environment might include the wireless signal environment 

and the quality of connecting cables. 

NOTE These are only examples. For custom products the factors may be quite precise, for consumer products they 

may specify an acceptable range or anticipated environment. 

6.2.6. Physical and social environments 

Any aspects of the expected physical and social environments that can influence usability shall be specified: 
NIST/IUSR – CISU-R 

12 

a) Physical environment in which the product will be used, including location and relevant physical 

conditions, such as temperature or lighting. 

b) Organizational environment, including group work dynamics, time pressures, supervision, and 

support. 

c) Any physical, health or safety issues. 

d) Any possible financial or security risks 

6.2.7. Scenarios of use 

The goals and tasks that will form the basis of the requirements shall be illustrated with scenarios of use. 

These scenarios describe how users meet their goals. They are examples of users’ activities, motivations, and 

how they carry out their tasks, using the product in a specific situation. 

The scenarios should illustrate the goals to be met, and tasks to be carried out to meet those goals. The 

scenarios should be the most frequent or most critical to the business or user. Each scenario includes an 

identification of the user group, technical, physical, and social environment, the goals to be met or tasks to be 

carried out, and the relative importance of the scenario. 

There may be separate scenarios for first use (after any necessary training) and experienced use. 

6.2.8. Training 

If users are expected to study any documentation, training materials or courses before using the product, the 

following information shall be specified: 

a) Which user groups are expected to study the materials. 

b) What previous knowledge or experience is required. 

c) Which goals and tasks are included in the training materials. 

6.3 Usability performance and satisfaction criteria 

6.3.1. Identifying relevant usability criteria (Level 1) 

Performance and satisfaction criteria identify the measures for the usability of a product. The first level in 

specifying performance and satisfaction criteria is to identify the criteria relevant to the context of use for the 

product. These may have important design implications. 

The following information shall be provided for Level 1 compliance: 

a) The types of performance and satisfaction criteria appropriate for successful use of the product. 

− These might include performance criteria such as task completion rate or time on task, subjective 

“ease of use” scores, indirect measures such as number of steps in a task, or a list of features to 

support usability that can be evaluated with a checklist. 

− Different user groups or scenarios may have different criteria. 

b) The relative importance of each criteria to the success of the product 

Examples of performance and satisfaction criteria include: the unassisted completion rate (effectiveness), the 

mean time taken to successfully complete each goal (efficiency), satisfaction scores on a specific 

questionnaire, efficiency relative to a past benchmark or other product. Annex B provides additional 

information about selecting appropriate criteria and best practices for how they should be measured. 
NIST/IUSR – CISU-R

13

EXAMPLE An example Level 1 specification for usability criteria from Annex F.6 is “The performance criteria for Web 

Delivery will be the ability of appropriate users to complete all core tasks with no training or assistance except on-screen 

prompts. Satisfaction with Web Delivery, after completing these tasks must be at least as high as satisfaction scores for 

current delivery methods.” 

6.3.2. Defining target values for criteria (Level 2) 

The second step in specifying performance and satisfaction criteria is to provide target values for the criteria, 

or indicate a method for determining those values later in the project. 

Target values are the minimum acceptable value, or a range of acceptable values, for the criteria. They may 

be based on the value for an existing or competitor system used for the same tasks and goals, based on 

expert judgment or specified by stakeholders. Additional target values better than the minimum acceptable 

value may also be specified. 

For Level 2 compliance, the usability criteria shall include target values or a range of acceptable values for 

these criteria. Target values may be in actual numbers, percentages, average or means, a range of values, or 

a scale. they may be absolute, or relative to performance benchmarks 

Each target value may be given as: 

a) a definite requirement, or 

b) a provisional requirement subject to further negotiation, or 

c) an objective for guidance. 

For effectiveness, efficiency, and satisfaction criteria, either target values shall be given, or a note provided 

explaining that either: 

a) a target value will be provided later, or 

b) why a target value cannot be established (for example for a completely new product), or 

c) why no target value is needed (for example the measure is of low importance to the user or 

organization). 

EXAMPLE An example of a Level 2 specification for usability criteria is included in Annex G.6.2. It updates the criteria 

to add target values, and to further define the criteria and requirements for testing them. 

6.3.3. Defining specific criterion values (Level 3) 

For Level 3 compliance the requirements shall include specific values for each criterion. These may be 

validated by benchmark usability testing, business requirements, or other methods. 

NOTE One way to establish criteria values is to first measure the effectiveness, efficiency, and satisfaction for the 

same goals using an existing or competitor system, and then to set the requirements for the new system to be as good as 

or better than the existing system. 

EXAMPLE The example Level 3 specification for usability criteria in Annex H.10 adds detail on the specific values 

and how they were determined..

6.3.4. Including use of learning materials in the criteria 

If any documentation, training materials or course will be studied before the product is used, criteria for the 

usability of the training materials in at least one training scenario (effectiveness, efficiency, and satisfaction 

when completing training goals) should be included. 

EXAMPLE 1 Given 2 hours of training with the online training program, a user will be able to complete the item 

identification task under 1 minute and with fewer than 2 errors. 
NIST/IUSR – CISU-R 

14 

EXAMPLE 2 After reading the owner’s manual a user will be able to access and listen to a cellphone message on the 

first attempt. 

6.4 Methods for testing 

6.4.1. Identifying evaluation methods (Level 1) 

The final part of a usability requirement identifies methods to evaluate whether the criteria have been met and 

the context in which the criteria will be measured. 

The first level in specifying test methods is to identify methods that are appropriate for the product. For some 

products these methods may include checklists or other inspection methods, or indirect measures such as 

business metrics, but ideally, they include methods in which people who are representative of the user groups 

test the product. 

For Level 1 compliance, the usability requirements shall include a list of usability methods that can be used to 

determine whether the usability requirements have been met. 

NOTE An expert review against a checklist of requirements would be appropriate for requirements that specify 

the presence (or absence) of a feature or elements of an interactive function, such as an undo feature or conformance to 

other guidelines.. 

EXAMPLE An example Level 1 specification (from Annex F.7) is “The product will be evaluated against the 

performance and satisfaction criteria with a usability test, using test scenarios based on the three scenarios.” 

6.4.2. Specifying a preliminary usability evaluation method (Level 2) 

The second level in specifying test methods is to describe the method to be used to test whether the usability 

requirements have been met, and the context in which the measurements will be made. This level adds 

enough information to describe the basic test method to a usability professional, but is not a complete 

specification of the method. 

For Level 2 compliance, the preliminary evaluation method shall include a description of the methods used to 

measure the product on all of the criteria. 

If a usability test method is specified, the description shall include: 

a) Goals of the test, including task scenarios to be tested and criteria for successful completion of each 

goal (see Annex C.2.1.a and b). 

b) The user groups (identified in the context of use) to be included in the test (see Annex C.1.b). 

c) The type of test facility, including the setting and type of space, and how relevant aspects of the 

intended environment will be simulated (see Annex C2.2). 

d) Computing environment, which may be specific or anticipated (for example, specifying the “current 

version at the time of the test” (see C.2.3). 

e) A general description of the test procedure (see C.3.1). 

EXAMPLE An example Level 2 specification (from Annex G.7) adds information about the test facility, the computing 

environment and display devices, as well as a brief outline of the experimental design.

6.4.3. Specifying a complete usability test method (Level 3) 

For level 3 compliance, a full protocol for a usability test shall be specified, providing the information required 

in Annex C. 
NIST/IUSR – CISU-R

15

The specification in Annex C is consistent with ISO/IEC 25062 (Common Industry Format (CIF) for usability 

test reports), which should be used for reporting the results. ISO/IEC 25062 requires reporting on the results 

of measurements of efficiency, effectiveness and satisfaction, or an explanation of why any of these metrics 

are not meaningful for this product. 

NOTE The performance and satisfaction criteria specified in 6.3 establish the values required in the intended context of 

use. The accuracy and precision with which they are measured will be determined by the context of measurement (the 

number of and types of users, way in which the tasks are simulated, and the simulated working environment). 

EXAMPLE An example Level 3 specification of a complete test protocol is included in Annex H.11.

6.5 Traceability 

The following information should be documented to assist the development organization in maintaining and 

refining the requirements: 

a) The source of the requirements. 

b) How the usability requirements support the stakeholders’ business objectives for the product. 

c) Why the goals were selected. 

EXAMPLE The most frequent goals, the most troublesome goals, most critical to accomplishing business 

goals; most frequently requested requirement. 

d) The source of each goal. 

EXAMPLE Observation of customers using similar products, product marketing specifications 

e) Who requested this requirement. 

f) Whether the requirement is essential or desirable. 

6.6 Document control 

The following information should be included to track different versions of the requirements document: 

a) Version number 

b) Date of change 

c) Nature of change 

d) Person making the change 

e) Person approving the change 
NIST/IUSR – CISU-R 

16 

Annex A (informative) 

Usability requirements format 

A.1 Introduction 

This annex provides a list of elements that should be used in a usability requirements document, including 

document and product information elements. It may either be used as a template, or may be used as a 

checklist for creating a format for usability requirements created with the CISU-R. 

The format of the information in this annex is consistent with the CIF (ISO/IEC 25062). 

A.2 Title page 

The title page contains: 

a) Identification of the requirements as conforming to Common Industry Specifications for Usability - 

Requirements v0.88. 

b) The name of the product and version that the requirements are for. 

c) The date the requirements were prepared. 

d) The organization name. 

e) Contact details for questions and/or clarifications. 

f) Names of the people who produced the requirements. 

A.3 Executive summary 

The Executive Summary provides a high level overview of the requirements. The intent of this section is to 

provide information for people who may not read the technical body of this document. This section should 

begin on a new page and end with a page break to facilitate its use as a stand-alone summary. 

The executive summary: 

a) Explains the purpose of the requirements. 

b) States the level of conformance. 

c) Provides a high level overview of the requirements that includes: 

− Name, version, and brief description of the product. 

− Summary of the type of users, tasks, any associated equipment (hardware, software, and 

materials), and the physical and social environments in which the product is intended to be used. 

− A list of the task scenarios, with criteria for effectiveness, efficiency, and satisfaction (for level 2 

and 3 conformance). 

d) Explains the reason for and nature of the requirements. 
NIST/IUSR – CISU-R

17

A.4 Product details 

A.4.1. Product description 

A product description includes: 

a) The formal product name and version. 

b) The parts of the product for which requirements are being provided. 

c) The user groups for which the product is intended. 

d) Whether the product is intended to be “walk up and use”, or whether any documentation, training 

materials or course has to be studied before the product is used. 

e) The type of user work that is supported by the product. 

f) Description of the environment in which the product should be used. 

g) Support for any groups with special needs. 

 

A.4.2. Product objectives 

Product objectives include: 

a) The overall objectives for the product and specific objectives for any subset of usage. 

b) Any known or intended functions and components that support key objectives. 

A.5 Context of use and scenarios 

The context of use and scenarios include all of the information and headings specified in Clause 6.2: 

a) Stakeholders 

b) User groups 

c) Goals and tasks 

d) Technical environment (equipment) 

e) Physical and social environments 

f) Scenarios of use for the most important goals 

A.6 Usability performance and satisfaction criteria and values 

The usability performance and satisfaction criteria include the information specified in Clause 6.3, including 

a) The goals that will form the basis of the requirements, with scenarios of usage, with an explanation of 

why these goals and scenarios were selected. 

b) The relative importance of each goal. 

c) A definition of the performance and satisfaction criteria appropriate for the scenario and product. 
NIST/IUSR – CISU-R 

18 

d) Target values or a range of anticipated values for these criteria, which may be: 

− a definite requirement, or 

− a provisional requirement subject to further negotiation, or 

− an objective for guidance. 

Criteria may include target values for effectiveness, efficiency, and satisfaction, an explanation that a target 

value will be provided later, or an explanation of why a target value cannot be established (for example for a 

completely new product). 

If any documentation, training materials or course has to be studied before the product is used, measures 

should be given for the usability of the training materials in at least one training scenario (effectiveness, 

efficiency, and satisfaction when completing training goals). 

A.7 Requirements for testing 

If a protocol for a usability test is included, the information specified in Annex C may be used. 
NIST/IUSR – CISU-R

19

Annex B (Informative) 

Examples of performance and satisfaction measures 

B.1 Introduction 

This annex provides examples of performance and satisfaction measures, appropriate for use in usability 

requirements created with the CISU-R. Because the CISU-R uses the definition of usability from ISO 9241 

(see the definition in Clause 4.1), these examples focus on effectiveness, efficiency, and satisfaction 

measures. 

It is desirable to set criteria for user performance and satisfaction criteria that can be measured during 

development. This will give a high degree of confidence that the product will be usable in the tested context of 

use, which is important if poor user performance or satisfaction will endanger success of the project or 

product. Sometimes it will only be possible to measure whether the criteria have been achieved after the 

product is released (for example from observation of users, or from web server logs). This will provide 

feedback on the adequacy of earlier usability work, and the need for any improvement in later releases. 

If any documentation, training materials or course will be studied before the product is used, criteria for the 

usability of the training materials in at least one training scenario (effectiveness, efficiency, and satisfaction 

when completing training goals) should be included. 

Other types of usability criteria such as the degree of adherence to usability guidelines, the number of user 

actions, or an expert rating can also provide feedback earlier in design and development. However they do 

not give the same degree of confidence in final usability. 

B.2 Effectiveness 

Effectiveness relates the goals of using the product to the accuracy and completeness with which these goals 

can be achieved. Common measures of effectiveness include task completion rate, frequency of errors, 

frequency of assists to the participant from the testers. It does not take account of how the goals were 

achieved, only the extent to which they were achieved. 

B.2.1. Task completion rate 

The completion rate is the percentage of participants who completely and correctly achieve each goal. If goals 

can be partially achieved (e.g., by incomplete or sub-optimum results) then it may also be useful to set 

requirements for partial goal achievement, scored on a scale of 0 to 100 % based on specified criteria related 

to the value of a partial result. 

EXAMPLE 1 The percentage of customers who can successfully complete a transaction on a web site, or the 

percentage of users who can successfully record an hour long TV program with a Digital Video Recorder (DVR). 

EXAMPLE 2 A spell-checking task might involve identifying and correcting 10 spelling errors. The completion rate 

might be calculated based on the percent of errors corrected. Another method for calculating completion rate is weighting; 

e.g., spelling errors in the title page of the document are judged to be twice as important as errors in the main body of text.

The rationale for choosing a particular method of partial goal analysis should be stated, if such results are included in the 

requirements. 

B.2.2. Errors 

Although the requirements for effectiveness are based on error-free completion of the task, keeping track of 

the errors is particularly helpful in determining what aspects of the product need improvement. Errors are 

instances where test participants did not complete the task successfully, or had to attempt portions of the task 

more than once. Scoring of data should include a classification of types of errors. 
NIST/IUSR – CISU-R 

20 

B.2.3. Assists 

When participants cannot proceed on a task, the test administrator sometimes gives direct procedural help in 

order to allow the test to proceed. This type of intervention is called an assist for the purposes of this standard. 

If it is necessary to provide participants with assists, efficiency, and effectiveness metrics shall be provided for 

both unassisted and assisted conditions, and the number and type of assists shall be included as part of the 

test results. When assists are allowed or provided, the number and type of assists shall be included as part of 

the test results. 

− The unassisted completion rate shall be calculated as the percentage of participants who achieve a 

goal without intervention from the testers. 

− The assisted rate shall be calculated as the percentage of participants who achieve a goal only with 

tester intervention. 

EXAMPLE For example, if a participant received an assist on Task A, but went on to successfully complete the task 

following the assist, he could be included in the assisted Task A completion rate. 

In some usability tests, participants are instructed to use support tools such as online help or documentation, 

which are part of the product, when they cannot complete goals on their own. Accesses to product features 

which provide information and help are not considered assists for the purposes of this requirements. It may, 

however, be desirable to measure the frequency of accesses to different product support features, especially 

if they affect participants’ ability to use products independently. 

B.3 Efficiency 

B.3.1. Measures 

Efficiency relates the level of effectiveness achieved to the quantity of resources expended. Efficiency is 

generally assessed by the mean time taken to achieve the task. Efficiency may also relate to other resources 

(e.g. total cost of usage). A common measure of efficiency is time on task, which can be defined as the mean 

time taken to complete each task, together with the range and standard deviation of times across participants. 

B.3.2. Relative user efficiency 

Relative user efficiency is the mean time taken by users who successfully achieve a goal divided by the time 

taken by an expert. 

NOTE Task time values are useful when making comparisons between products, while the relative user efficiency 

values are easier to interpret in isolation because they compare performance in the usability test with what is possible by 

an expert. 

B.3.3. Completion rate/mean time-on-task 

Completion Rate divided by Mean Time-On-Task is the core measure of efficiency. It is the percentage of 

users who were successful (or percentage goal achievement) for every unit of time. 

This formula shows that as the time on task decreases, one would expect users to be more successful. A very 

efficient product has a high percentage of successful users in a small amount of time. This allows customers 

to compare fast error-prone interfaces to slow easy interfaces. 

EXAMPLE 1 An error-prone interface such as a command line interface using wildcards to delete groups of files would 

typically show a low completion rate divided by mean time-on-task. 

EXAMPLE 2 A slower, but easier interface such as using a mouse and keyboard to drag each file to the trash would 

typically show a high completion rate divided by mean time-on-task. 
NIST/IUSR – CISU-R

21

B.4 Satisfaction 

Satisfaction describes a user’s subjective response when using the product. User satisfaction may be related 

to whether a users wants use a product and may affect performance in some cases. Questionnaires to 

measure satisfaction and associated attitudes often use Likert or semantic differential scales. 

A variety of instruments are available for measuring user satisfaction of interactive products, and many 

organizations create their own. Some examples of standard questionnaires include: ASQ [5], CUSI [6], 

PSSUQ [6], QUIS [3], SUMI [4], and SUS [7]). 
NIST/IUSR – CISU-R 

22 

Annex C (normative) 

 Specification of a usability test protocol 

C.1 Introduction 

This annex provides the information necessary to specify the method and test protocol to test whether the 

usability requirements have been met. The elements specified are consistent with the ISO/IEC 25062, the 

Common Industry Format for Usability Test Reports (CIF), and the results of the test can subsequently be 

documented using the CIF (ISO/IEC 25062). 

For specifying a usability test method to Level 3 compliance, all of the information specified in this annex shall

be included. 

Sufficient information shall be provided to enable the requirements to be tested. 

C.2 Users 

The following information shall be provided: 

a) The total number of participants required. 

b) Segmentation of user groups tested, if more than one. 

c) How participants are to be selected so that they have the essential characteristics. 

Characteristics should be chosen to be relevant to the product’s usability. They should be sufficiently detailed 

to ensure that the participants are representative of the actual users. 

The following information should be provided: 

a) Any expected differences between the participant sample and the user population. 

EXAMPLE Actual users will attend a training course which will have to be simulated. 

b) Description of any groups with special needs. 

C.3 Context of product use in the test 

C.3.1. Test facility 

The following information should be provided: 

a) The setting, and type of space in which the evaluation will be conducted. 

EXAMPLES Usability lab, cubicle office, meeting room, home office, kitchen, manufacturing floor 

b) How relevant aspects of the intended environment will be simulated. 

c) How the test setting differs from the normal context of use. 
NIST/IUSR – CISU-R

23

C.3.2. Participant’s computing environment 

If the product runs on a computer or other device, the technical environment to be used shall be specified, 

including: 

a) Computer configuration, including model, operating system version, required libraries or settings. 

b) If used, browser name and version; relevant plug-in names and versions. 

c) Any relevant physical connectors or wireless connections. 

C.3.3. Display devices 

If a display device is part of the product, the characteristics of the display shall be specified: 

a) If screen-based, screen size, resolution, and color setting. 

b) If print-based, the media size and print resolution. 

c) If visual interface elements, such as fonts, can vary in size, specify the size(s) to be used. 

C.4 Test procedure 

C.4.1. Scenarios to be tested 

The following information shall be provided: 

a) If more than one condition is being tested, the logical design of the test. 

b) Scenarios to be tested. 

c) Criteria for successful completion of the goal of each scenario. 

d) Maximum time limits for completing a scenario or task. 

e) Policies and procedures for interaction between tester(s) and test participants. 

C.4.2. Participant general instructions 

The following information shall be provided (here, or in an annex): 

a) Orientation to the test context and consents to be given to the participants. 

b) General instructions to be given to the participants. 

c) Instructions on how participants are to interact with any other persons present, including how they are 

to ask for assistance or interact with other participants, if applicable. 

C.4.3. Participant task instructions 

The following information shall be provided: 

a) Task instructions to be given to the participants. 

b) Any data to be given to the participants to enable them to complete the scenarios. 
NIST/IUSR – CISU-R 

24 

C.5 Performance and satisfaction metrics 

C.5.1. Criteria and measurements 

The following information shall be provided: 

a) The performance criteria to be measured. 

b) How performance criteria will be measured. 

c) Definitions of any independent variables or control variables. 

C.5.2. Metrics for effectiveness, efficiency, and satisfaction 

The metrics to be measured and reported shall include either: 

a) At least one of the criteria for effectiveness, and efficiency, and 

b) The questionnaire or other method for measuring satisfaction, or 

c) An explanation of why any of these metrics are not meaningful for this product. 

EXAMPLE Suppose that the context of use for the product includes real time, open-ended interaction between 

close associates. In this case, Time-On-Task may not be meaningfully interpreted as a measure of efficiency, 

because for many users, time spent on this task is “time well spent”. 

C.5.3. Statistical significance 

The CIF (ISO/IEC 25062) specifies the statistical data to be given when reporting results. 

A requirement may be included to demonstrate a relationship with previous test results, for example: 

− The test results for specified measures are at least as good as for the previous results (with 95 % 

confidence). 

− The test results for specified measures are better than the previous results (with 80 % confidence). 

C.6 Appendices 

The following information may be provided: 

a) Custom questionnaires, if used. 

b) Participant general instructions (if not in the body of the requirements). 

c) Participant task Instructions. 
NIST/IUSR – CISU-R

25

Annex D (informative) 

An example of a user centered design process 

A typical user-centered development process that incorporates the development of usability requirements 

under the CISU-R might include the following subsequent activities: 

a) Define scenarios of use. 

b) Document the context of use and scenarios of use in CISU-R format (level 1 compliance). 

c) Carry out a usability test of an existing system to obtain baseline measures of effectiveness, efficiency 

and satisfaction in these scenarios. 

d) Use the baseline data as a basis for establishing quantitative requirements for effectiveness, 

efficiency and satisfaction in these scenarios (level 2 compliance). 

e) Specify methods to test whether the requirements have been met (level 3 compliance). 

f) Refine the design by testing with users carrying out tasks using low-fidelity prototypes. 

g) Test prototypes of the working system. 

h) Test the usability of the final system to check whether the requirements for effectiveness, efficiency, 

and satisfaction have been met. The results can be documented using the CIF (ISO/IEC 25062) 

format. 

Depending on the development environment, requirements may be iteratively elaborated as more information 

is obtained from usability activities such as paper prototyping during development, or they may be agreed by 

all parties before development commences, and subsequently only modified by mutual agreement. 

NOTE The requirements for effectiveness, efficiency, and satisfaction in particular contexts of use are less likely to 

change, as they are closely linked to meeting business and user needs. 
NIST/IUSR – CISU-R 

26 

Annex E (Informative) 

Examples of other types of user and usability requirements 

E.1 Introduction 

Other, more detailed usability requirements can be specified to complement the core CISU-R requirements in 

a user centered design process Examples include: 

E.1.1. Design guidance 

a) Design principles 

b) Human factors and ergonomics 

c) Style guides 

d) Standards 

E.1.2. Usability features 

a) Accessibility 

b) Understandability 

c) Learnability 

d) Operability 

e) Attractiveness 

E.1.3. Content and functions 

a) Functionality 

b) Content 

c) Complete, accurate, up to date, style 

d) Effectiveness (for learning) 

e) Trust 

f) Platform independence 
NIST/IUSR – CISU-R

27

Annex F (Informative) 

An example of a Level 1 requirements document 

F.1 Title Page 

Usability Requirements 

for Web Software Delivery Release 1 

CIFWorks 

15 May 2006 

Requirements produced by: Sherman Jones. 

Contributors: Fred Smith, Product Manager 

Reviewers: Ann Smith, VP of UI 

 Alexander Kushenko, Product Boss 

Created under the Common Industry Specification for Usability – Requirements (CISU-R) v0.88 

Address inquiries to: Sherman Jones, CIFWorks 

Phone: 650-555-1212 

Email: sherman.jones@CIFWorks.com 

Address: 150 New Parkway, New Pines, CA 91011 

Version History 

3 May 2006 1.0 Sherman Jones Created document 

15 May 2006 1.1 Sherman Jones Revised based on review comments 

Related Documents 

”New Product Initiation Plan” – 13 April 2006 – Alexander Kushenko 

F.2 Executive Summary 

The purpose of this document is to specify the usability requirements for the first release of Web Software 

Delivery in a format that allows CIFWorks’ customers to review and provide input. This document conforms to 

Level 1 of the CISU-R. 

The objective of the Web Software Delivery product is to move away from physical shipments of CD’s and 

DVD’s towards web based delivery of CIFWorks’ products. The key tasks involved are downloading the 

software, creating DVDs from the software and setting up a staged (does not use DVDs) installation from the 

downloaded files. 

The intended users of Web Software Delivery are system administrators. System administrators are 

experienced computer professionals whose primary roles are the installation, configuration, and administration 

of multi-user, production software packages. They will use the product in a corporate computing environment 

on a computer with a recent version of the browser, sufficient disk space to download the product and a high 

speed (10mbs or faster) internet connection. 

The key tasks, for which requirements scenarios will be developed are: 

− Download SITE guard software 

− Burn SITE guard software to DVD 
NIST/IUSR – CISU-R 

28 

− Prepare SITE guard software for staged install and launch installer 

The usability of this product will be evaluated in a usability test before release, measuring the following 

criteria. Target values will be set through benchmark usability testing of the current delivery methods, 

interviews with users to establish their requirements, and competitive research with competitive products. 

− Effectiveness: the unassisted task completion rate, measured as a percentage of total task attempts 

(priority 1) 

− Efficiency: the average time to complete the task, measured as the average number of minutes to 

complete each task (priority 2) 

− Satisfaction, using a standard questionnaire. This may be the SUMI, SUS or a customized 

questionnaire (priority 3) 

F.3 Product Details 

Name and Version: CIFWorks Web Software Delivery Release 1 

Parts for which requirements provided: All parts of the software 

Intended user groups: System Administrator, experienced with server software installation 

Training required: None (for intended user groups) 

Work supported by product: Downloading and preparing CIFWorks’ Site Guard software for 

installation. 

Usage environment: Medium to large size corporate and public sector companies. 

Users with special needs: Product will meet standards for Section 508 of the Rehabilitation Act 

F.4 Product Objective 

The objective of the product is to replace mailing DVD’s as the default mechanism for shipping CIFWorks’ 

software. 

F.5 Context of Use 

F.5.1. Users 

F.5.1.1. System Administrators 

System Administrators are experienced computer professionals whose primary roles are the installation, 

configuration, and administration of multi-user, production, software . These users are typically administrators 

of other enterprise software products such as operating systems, databases, and web servers. 

System Administrator’s educational backgrounds are varied as some receive formal academic training in 

computer science or computer systems management while others receive training on the job or through 

vendor or 3rd party courses. 

The most important characteristic of a System Administrator is on the job experience. There are typically two 

routes by which these users gain experience as a System Administrator: 
NIST/IUSR – CISU-R

29

g) They extend their background from being a PC administrator or power user into the role of a System 

Administrator by industry training, 

h) They grow their skills in an organization under the tutelage of a Senior System Administrator. 

For Web Software Delivery we expect System Administrators to have 1-10 years of experience in their job. In 

addition to software installation, and configuration, their day to day tasks would include operations (adding 

users, adding disk space, monitoring an existing system), and troubleshooting. They may also perform 

planning operations for future software and hardware acquisitions. 

F.5.1.2. Users with Disabilities: 

Users with vision, hearing, and motor disabilities should be able to function as a System Administrator with 

appropriate assistive devices (e.g. screen readers). 

F.5.2. Goals 

The general goal is to obtain the appropriate version of the CIFWorks Site Guard software for the appropriate 

platform in a form ready for installation. Obtaining the software completes the purchasing process but may or 

may not lead immediately to installation. In some cases hardware or software has to be configured before 

installation of the software begins. Users may immediately burn the electronic version of the software onto 

DVD’s, however this is not required for installation. Some users will only install the product from the 

downloaded files (referred to as a “staged install”). Based on this there are three tasks associated with 

electronic delivery. 

Task 1. Finding and downloading the Site Guard software via Web Software Delivery 

Task 2. Create DVD’s from the downloaded Site Guard software and begin the install. 

Task 3. Prepare the software for an electronic or “staged” install and begin the install 

F.5.3. Equipment 

To use Web Software Delivery the following computing environment will be required. 

− A computer capable of running the browsers listed below. 

− Internet Explorer (6.0 and higher), Netscape (6.0 and higher), or Mozilla Firefox 1.1 and higher 

− 10 gigabytes of disk space to download the zip files required for installation and create the staged 

installation. 

− 500 megabytes of RAM 

− A 10mbs or faster internet connection. 

− A SVGA compatible monitor. 

− A DVD Burner 

F.5.4. Physical and Social Environment 

− The physical environment will be either a corporate/ public sector office or a home office with a high 

speed internet connection (10 mbs or faster). The home office may be used to download the software 

and burn CD’s after hours. 

− Typically system administrators multi-task. It is expected that using web software delivery will be 

mixed in with other support tasks they are completing. 
NIST/IUSR – CISU-R 

30 

− There is typically a time lag between acquiring and installing the software, and rolling out the software 

into production. Because of that acquiring and installing the software will likely be part of a planning 

exercise without mission criticality pressure. However due to hacker attacks or other security 

concerns there may be times when obtaining Site Guard may be urgent. 

F.5.5. Scenarios of use 

Scenario 1: Download the Site Guard software and burn it to DVD 

Ann is the lead system administrator at Biagra a producer of genetically engineered wheat products. Last 

month her company was hit by a denial of service attack by a group of organic fundamentalists. It was a 

rough month and Ann ended up shutting down the company website leading to unhappy customers and 

employees. It took no time to get the PO approved for Site Guard and now she is on the hot-seat to get it 

installed on all the companies servers. She is glad she can get the software online since she does not 

want to wait for a physical shipment. Since her company’s firewall is closed, she has to download the 

software at home and take it into the office and install it on her 2 primary servers. 

After dinner she turns on Toy Story 2 for the kids and sits down in the study to download the software. 

She has the printed out email from CIFWorks and uses it to login to the Web Delivery site. After login, 

she sees the software she purchased and online instructions for how to download it. She uses Mozilla for 

the download as it has a built in download manager. She begins the first DVD download and then goes 

to check on the kids. Occasionally she peeks in on the computer to see how the download is going. She 

is happy to see that it is progressing nicely, although it is still a bit slow. 10 minutes later the first DVD is 

downloaded so she starts the second DVD download. 20 minutes later she has put the kids to bed and 

comes back in to see that the second DVD download is complete. 

Going back to the CIFWorks website she launches a utility that will help her burn the DVD’s. It asks her 

for her computer information and then asks her to insert a DVD. While the DVD is burning she sees a 

progress indicator. It says 20 minutes to go, so she surfs the internet for a while. Twenty minutes later 

she starts the second DVD. At the end of the evening she has two DVD’s ready for installation in the 

morning. 

Back at work, she takes the DVD’s and pops them into her development environment. The development 

environment is a clone of the production web server that she uses to test out installations. She opens a 

Unix Shell and launches the installer. She is glad to see that the installation starts successfully. 

Scenario 2: Download the Site Guard software and create a staged installation 

Winston is a Senior System Administrator for the University of South-East North Dakota. He administers 

the systems that run the school’s websites. He takes pride in maintaining a safe system despite the 

many hackers at the school. As part of his 2005 budget, he got approval to purchase Site Guard and 

deploy it to the server farm that is used for the university website. 

Today he received the email from CIFWorks with his license number and instructions for downloading the 

software. Winston always likes to install software from a staged environment. This makes it much faster 

to take a master copy of the software and install it on multiple machines in his server farm. He has 

designated a system called “stage_guard” to be the stage for Site Guard. He prints out the email (he 

can’t access it from stage_guard) and walks over to the stage_guard machine. He logs onto the 

CIFWorks website. He sees the software that he needs to download and begins the first download. 

While the first download is completing he looks around on the website for instruction on building a staged 

install. He sees a link to instructions for staged install on the side of the web page. He clicks the link and 

starts reading the instructions. The instructions say he can either do it manually or download a script. 

Being security conscious he does not trust “foreign” scripts so he does it the manual way. 

He follows the instructions and begins creating the directories he will need for the install. It takes a while 

to create all the directories, as he has to create them using command line UNIX and then set privileges 

for them. He checks back on the first download and sees it is complete. He starts the second download. 

While that is working he completes the directories he needs and focuses on unzipping the files. The 
NIST/IUSR – CISU-R

31

instructions say to use a zip utility to unpackage and put the files in the proper directories. He 

appreciates the fact that the instructions list the exact command to use with his zip utility. He unzips the 

files from the first download and puts them into the appropriate directory. After that is complete he 

checks back in with the download and sees the second download is complete. He unzips that file and 

puts it into the second directory. He goes back to the first directory and runs the installer. He is glad to 

see it has come up and he can proceed with the installation. 

F.6 Performance and Satisfaction Criteria 

F.6.1. Goals 

The goal for using Web Software Delivery is to find and download the software for the appropriate platform in 

a form ready for installation. This goal is completed when the actual installation begins, or is ready to begin. 

There are three core tasks to complete this goal with Web Software Delivery. 

1. Download SITE guard software 

This task has the highest priority since successful download of the software is a prerequisite 

to the other tasks. 

2. Burn SITE guard software to DVD and launch installer 

This task has second priority since it is the recommended approach for installing the software. 

3. Prepare SITE guard software for staged install and launch installer 

This task is lowest priority as it is an optional way of installation. 

F.6.2. Criteria 

The performance criteria for Web Delivery will be the ability of appropriate users to complete all core tasks 

with no training or assistance except on-screen prompts. Satisfaction with Web Delivery, after completing 

these tasks must be at least as high as satisfaction scores for current delivery methods. 

Target values for these criteria will be set through interviews with users to establish their requirements, results 

of formative usability testing during the design process, and competitive research with competitive products. 

F.7 Methods for Testing 

The product will be evaluated against the performance and satisfaction criteria with a usability test, using test 

scenarios based on the three scenarios. 
NIST/IUSR – CISU-R 

32 

Annex G (Informative) 

An example of a Level 2 requirements document 

G.1 Title Page 

Usability Requirements 

for Web Software Delivery Release 1 

CIFWorks 

25 August 2006 

Requirements produced by: Sherman Jones. 

Contributors: Fred Smith, Product Manager 

Reviewers: Ann Smith, VP of UI 

 Alexander Kushenko, Product Boss 

 James Dearing, Director of Marketing 

 Jennifer Taylor, Deployment Manager 

 John Smith, User Researcher 

 Jackie Mayer, Usability Lab Manager 

Created under the Common Industry Specification for Usability – Requirements (CISU-R) v0.88 

Address inquiries to: Sherman Jones, CIFWorks 

Phone: 650-555-1212 

Email: sherman.jones@CIFWorks.com 

Address: 150 New Parkway, New Pines, CA 91011 

Version History 

10 May 2006 1.0 Sherman Jones Created document 

15 May 2006 1.1 Sherman Jones Revised based on review (Level 1 complete) 

10 July 2006 2.0 Sherman Jones Added detail for criteria 

18 August 2006 2.1 John Smith Added preliminary test method 

25 August 2006 2.2 Sherman Jones Added target values (Level 2 complete) 

Related Documents 

”New Product Initiation Plan” – 13 April 2006 – Alexander Kushenko 

”Product competitive landscape review” – 8 July 2006 – James Dearing 

”Site visits and user observations report” – 5 August 2006 – John Smith 

”Product specifications document” – 10 August 2006 – Fred Smith 

”User interface specifications” – 15 August 2006 – Ann Smith 

G.2 Executive Summary 

The purpose of this document is to specify the usability requirements for the first release of Web Software 

Delivery in a format that allows CIFWorks’ customers to review and provide input. 

This document conforms to Level 2 of the CISU-R. It adds to the previous Level 1 document with: 
NIST/IUSR – CISU-R

33

− Complete definitions for performance and satisfaction criteria 

− Preliminary target values, to be confirmed with benchmark usability testing 

− Preliminary usability testing method to test the product against the criteria 

The objective of the Web Software Delivery product is to move away from physical shipments of CD’s and 

DVD’s towards web based delivery of CIFWorks’ products. The key tasks involved are downloading the 

software, creating DVDs from the software and setting up a staged (does not use DVDs) installation from the 

downloaded files. 

The intended users of Web Software Delivery are system administrators. System administrators are 

experienced computer professionals whose primary roles are the installation, configuration, and administration 

of multi-user, production software packages. They will use the product in a corporate computing environment 

on a computer with a recent version of the browser, sufficient disk space to download the product and a high 

speed (10mbs or faster) internet connection. 

The key tasks, for which requirements scenarios have been developed are: 

− Download SITE guard software 

− Burn SITE guard software to DVD 

− Prepare SITE guard software for staged install and launch installer 

The usability of this product will be evaluated in a usability test before release, measuring the following 

criteria. This document includes preliminary target values, which will be confirmed with a benchmark usability 

test before final testing of the product. 

− Effectiveness: the unassisted task completion rate, measured as a percentage of total task attempts 

(priority 1) 

− Efficiency: the average time to complete the task, measured as the average number of minutes to 

complete each task (priority 2) 

− Satisfaction, using a standard questionnaire. This may be the SUMI, SUS or a customized 

questionnaire (priority 3) 

G.3 Product Details 

Name and Version: CIFWorks Web Software Delivery Release 1 

Parts for which requirements provided: All parts of the software 

Intended user groups: System Administrator, experienced with server software installation 

Training required: None (for intended user groups) 

Work supported by product: Downloading and preparing CIFWorks’ Site Guard software for 

installation. 

Usage environment: Medium to large size corporate and public sector companies. 

Users with special needs: Product will meet standards for Section 508 of the Rehabilitation Act 
NIST/IUSR – CISU-R 

34 

G.4 Product Objective 

The objective of the product is to replace mailing DVD’s as the default mechanism for shipping CIFWorks’ 

software. 

G.5 Context of Use 

The context of use section of this document has not been changed from the Level 1 version (15 May 2006) 

G.5.1. Users 

G.5.1.1. System Administrators 

System Administrators are experienced computer professionals whose primary roles are the installation, 

configuration, and administration of multi-user, production, software . These users are typically administrators 

of other enterprise software products such as operating systems, databases, and web servers. 

System Administrator’s educational backgrounds are varied as some receive formal academic training in 

computer science or computer systems management while others receive training on the job or through 

vendor or 3rd party courses. 

The most important characteristic of a System Administrator is on the job experience. There are typically two 

routes by which these users gain experience as a System Administrator: 

− They extend their background from being a PC administrator or power user into the role of a System 

Administrator by industry training, 

− They grow their skills in an organization under the tutelage of a Senior System Administrator. 

For Web Software Delivery we expect System Administrators to have 1-10 years of experience in their job. In 

addition to software installation, and configuration, their day to day tasks would include operations (adding 

users, adding disk space, monitoring an existing system), and troubleshooting. They may also perform 

planning operations for future software and hardware acquisitions. 

G.5.1.2. Users with Disabilities: 

Users with vision, hearing, and motor disabilities should be able to function as a System Administrator with 

appropriate assistive devices (e.g. screen readers). 

G.5.2. Goals 

The general goal is to obtain the appropriate version of the CIFWorks Site Guard software for the appropriate 

platform in a form ready for installation. Obtaining the software completes the purchasing process but may or 

may not lead immediately to installation. In some cases hardware or software has to be configured before 

installation of the software begins. Users may immediately burn the electronic version of the software onto 

DVD’s, however this is not required for installation. Some users will only install the product from the 

downloaded files (referred to as a “staged install”). Based on this there are three tasks associated with 

electronic delivery. 

Task 1. Finding and downloading the Site Guard software via Web Software Delivery 

Task 2. Create DVD’s from the downloaded Site Guard software and begin the install. 

Task 3. Prepare the software for an electronic or “staged” install and begin the install 
NIST/IUSR – CISU-R

35

G.5.3. Equipment 

To use Web Software Delivery the following computing environment will be required. 

− A computer capable of running the browsers listed below. 

− Internet Explorer (6.0 and higher), Netscape (6.0 and higher), or Mozilla Firefox 1.1 and higher 

− 10 gigabytes of disk space to download the zip files required for installation and create the staged 

installation. 

− 500 megabytes of RAM 

− A 10mbs or faster internet connection. 

− A SVGA compatible monitor. 

− A DVD Burner 

G.5.4. Physical and Social Environment 

− The physical environment will be either a corporate/ public sector office or a home office with a high 

speed internet connection (10 mbs or faster). The home office may be used to download the software 

and burn CD’s after hours. 

− Typically system administrators multi-task. It is expected that using web software delivery will be 

mixed in with other support tasks they are completing. 

− There is typically a time lag between acquiring and installing the software, and rolling out the software 

into production. Because of that acquiring and installing the software will likely be part of a planning 

exercise without mission criticality pressure. However due to hacker attacks or other security 

concerns there may be times when obtaining Site Guard may be urgent. 

G.5.5. Scenarios of use 

Scenario 1: Download the Site Guard software and burn it to DVD 

Ann is the lead system administrator at Biagra a producer of genetically engineered wheat products. Last 

month her company was hit by a denial of service attack by a group of organic fundamentalists. It was a 

rough month and Ann ended up shutting down the company website leading to unhappy customers and 

employees. It took no time to get the PO approved for Site Guard and now she is on the hot-seat to get it 

installed on all the companies servers. She is glad she can get the software online since she does not 

want to wait for a physical shipment. Since her company’s firewall is closed, she has to download the 

software at home and take it into the office and install it on her 2 primary servers. 

After dinner she turns on Toy Story 2 for the kids and sits down in the study to download the software. 

She has the printed out email from CIFWorks and uses it to login to the Web Delivery site. After login, 

she sees the software she purchased and online instructions for how to download it. She uses Mozilla for 

the download as it has a built in download manager. She begins the first DVD download and then goes 

to check on the kids. Occasionally she peeks in on the computer to see how the download is going. She 

is happy to see that it is progressing nicely, although it is still a bit slow. 10 minutes later the first DVD is 

downloaded so she starts the second DVD download. 20 minutes later she has put the kids to bed and 

comes back in to see that the second DVD download is complete. 

Going back to the CIFWorks website she launches a utility that will help her burn the DVD’s. It asks her for her 

computer information and then asks her to insert a DVD. While the DVD is burning she sees a progress 

indicator. It says 20 minutes to go, so she surfs the internet for a while. Twenty minutes later she starts the 

second DVD. At the end of the evening she has two DVD’s ready for installation in the morning. 
NIST/IUSR – CISU-R 

36 

Back at work, she takes the DVD’s and pops them into her development environment. The development 

environment is a clone of the production web server that she uses to test out installations. She opens a 

Unix Shell and launches the installer. She is glad to see that the installation starts successfully. 

Scenario 2: Download the Site Guard software and create a staged installation 

Winston is a Senior System Administrator for the University of South-East North Dakota. He administers 

the systems that run the school’s websites. He takes pride in maintaining a safe system despite the 

many hackers at the school. As part of his 2005 budget, he got approval to purchase Site Guard and 

deploy it to the server farm that is used for the university website. 

Today he received the email from CIFWorks with his license number and instructions for downloading the 

software. Winston always likes to install software from a staged environment. This makes it much faster 

to take a master copy of the software and install it on multiple machines in his server farm. He has 

designated a system called “stage_guard” to be the stage for Site Guard. He prints out the email (he 

can’t access it from stage_guard) and walks over to the stage_guard machine. He logs onto the 

CIFWorks website. He sees the software that he needs to download and begins the first download. 

While the first download is completing he looks around on the website for instruction on building a staged 

install. He sees a link to instructions for staged install on the side of the web page. He clicks the link and 

starts reading the instructions. The instructions say he can either do it manually or download a script. 

Being security conscious he does not trust “foreign” scripts so he does it the manual way. 

He follows the instructions and begins creating the directories he will need for the install. It takes a while 

to create all the directories, as he has to create them using command line UNIX and then set privileges 

for them. He checks back on the first download and sees it is complete. He starts the second download. 

While that is working he completes the directories he needs and focuses on unzipping the files. The 

instructions say to use a zip utility to unpackage and put the files in the proper directories. He 

appreciates the fact that the instructions list the exact command to use with his zip utility. He unzips the 

files from the first download and puts them into the appropriate directory. After that is complete he 

checks back in with the download and sees the second download is complete. He unzips that file and 

puts it into the second directory. He goes back to the first directory and runs the installer. He is glad to 

see it has come up and he can proceed with the installation. 

G.6 Performance and Satisfaction Criteria 

The performance and satisfaction criteria have been updated in this version to add target values, and to 

further define the criteria and requirements for testing them. 

G.6.1. Goals 

The goal for using Web Software Delivery is to find and download the software for the appropriate platform in 

a form ready for installation. This goal is completed when the actual installation begins, or is ready to begin. 

There are three core tasks to complete this goal with Web Software Delivery. 

1. Download SITE guard software 

This task has the highest priority since successful delivery of the software is a prerequisite to 

the other tasks. 

2. Burn SITE guard software to DVD and launch installer 

This task has second priority since it is the recommended approach for installing the software. 

3. Prepare SITE guard software for staged install and launch installer 

This task is lowest priority as it is an optional way of installation. 
37

G.6.2. Criteria 

The table below lists the key tasks associated with Web Software Delivery and the usability requirements for 

effectiveness, efficiency, and satisfaction. These target values were determined through interviews and 

observations of typical users completing comparable tasks, in consultation with the deployment manager and 

marketing. They will be confirmed or updated with benchmark usability testing of the current delivery methods 

to ensure that these target criteria would result in an improvement over the current delivery methods. 

− Effectiveness: the unassisted task completion rate. 

Unassisted task completion rate is the percentage of users who complete the tasks without assists 

from the usability test administrator. This criteria is the first priority, as successful completion of each 

task is the most important requirement for users. 

− Efficiency: the average time to complete the task 

Efficiency will be measured as the average number of minutes to complete each task. Only user time 

will be measured. Machine time (for example, actual download time) will not be counted. This is the 

second priority, as the time taken by the machine is expected to be longer than the user time. 

− Satisfaction, using the SUMI questionnaire, administered after completing the three tasks. This is also 

a second priority because any lack of satisfaction will reduce users’ willingness to continue to use web 

delivery. 

Task 

Effectiveness: 

Unassisted task 

completion rate of at 

least

Efficiency: 

Maximum user time 

(not machine time) of

Satisfaction: 

Post-task SUMI score 

of at least:

Download SITE guard 

software 

90 % 10 minutes 

Burn SITE guard software to 

DVD 

80 % 15 minutes 

Prepare SITE guard 

software for staged install 

and launch installer 

70 % 30 minutes 

Product overall 50 

 Table 1: Target criteria values 

G.7 Methods for Testing 

The product will be evaluated against the performance and satisfaction criteria with a usability test, using test 

scenarios based on the three scenarios. 

G.7.1. Users 

A minimum of 8 users should test the product. All users should be System Administrators with at least 1 year 

of experience as a system administrator managing server software. 

G.7.2. Goals 

Refer to Performance and Satisfaction Criteria in the body of the document for Tasks and Goals. 

G.7.3. Test facility 

The test should be conducted in a usability lab that consists of a user room and a control room. The user room 

will be an office where the participants conduct the tasks. The lab should be equipped with audio and video 

monitoring and recording devices, as well as a scan converter that allows the display on the participant’s 

monitor to be video recorded. 
NIST/IUSR – CISU-R 

38 

G.7.4. Participant’s computing environment 

The participants computing environment should meet the minimum set forth in the section “Equipment” in the 

main body of this document. 

G.7.5. Display devices 

As mentioned in the “equipment” section of this document, a minimum of a color VGA monitor with capability 

of 800x600 resolution is required. 

G.7.6. Experimental Design 

Tasks will be presented to each participant in the same order because since the second and third tasks are 

dependent on the first one. The six dependent measures for this study should be: 

− Unassisted task completion rate (the percentage of participants who completed a task without 

assistance from the tester) 

− Assisted task completion rate (the cumulative percentage of people who completed a task with zero, 

one or two assists from the tester) 

− Task time 

− Number of errors 

− Number of assists 

− Self-report satisfaction via SUMI measures.
NIST/IUSR – CISU-R

39

Annex H (Informative) 

An example of a Level 3 requirements document 

H.1 Title Page 

Usability Requirements 

for Web Software Delivery Release 1 

CIFWorks 

18 November 2006 

Requirements produced by: Sherman Jones. 

Contributors: Fred Smith, Product Manager 

Reviewers: Ann Smith, VP of UI 

 Alexander Kushenko, Product Boss 

 James Dearing, Director of Marketing 

 Jennifer Taylor, Deployment Manager 

 John Smith, User Researcher 

 Jackie Mayer, Usability Lab Manager 

Created under the Common Industry Specification for Usability – Requirements (CISU-R) v0.88 

Address inquiries to: Sherman Jones, CIFWorks 

Phone: 650-555-1212 

Email: sherman.jones@CIFWorks.com 

Address: 150 New Parkway, New Pines, CA 91011 

Version History 

10 May 2006 1.0 Sherman Jones Created document 

15 May 2006 1.1 Sherman Jones Revised based on review (Level 1 complete) 

10 July 2006 2.0 Sherman Jones Added detail for criteria 

18 Aug 2006 2.1 John Smith Added preliminary test method 

25 Aug 2006 2.2 Sherman Jones Added target values (Level 2 complete) 

15 Oct 2006 3.1 Sherman Jones Adjusted target values to benchmark test 

18 Nov 2006 3.2 John Smith Finalized testing protocol (Level 3 complete) 

Related Documents

”New Product Initiation Plan” – 13 April 2006 – Alexander Kushenko 

”Product competitive landscape review” – 8 July 2006 – James Dearing 

”Site visits and user observations report” – 5 August 2006 – John Smith 

”Product specifications document” – 10 August 2006 – Fred Smith 

”User interface specifications” – 15 August 2006 – Ann Smith 

”Benchmarking usability test report” – 3 October 2006 – Jackie Mayer 

H.2 Executive Summary 

The purpose of this document is to specify the usability requirements for the first release of Web Software 

Delivery in a format that allows CIFWorks’ customers to review and provide input. 
NIST/IUSR – CISU-R 

40 

This document conforms to Level 3 of the CISU-R. It updates the previous Level 3 document with: 

− Updated target values for the performance and satisfaction criteria, as confirmed through benchmark 

usability testing 

− Complete usability testing protocol to test the product against the criteria 

The objective of the Web Software Delivery product is to move away from physical shipments of CD’s and 

DVD’s towards web based delivery of CIFWorks’ products. The key tasks involved are downloading the 

software, creating DVDs from the software and setting up a staged (does not use DVDs) installation from the 

downloaded files. 

The intended users of Web Software Delivery are system administrators. System administrators are 

experienced computer professionals whose primary roles are the installation, configuration, and administration 

of multi-user, production software packages. They will use the product in a corporate computing environment 

on a computer with a recent version of the browser, sufficient disk space to download the product and a high 

speed (10mbs or faster) internet connection. 

The key tasks, for which requirements scenarios have been developed are: 

− Download SITE guard software 

− Burn SITE guard software to DVD 

− Prepare SITE guard software for staged install and launch installer 

The usability of this product will be tested against the following criteria: 

− Effectiveness: the unassisted task completion rate, measured as a percentage of total task attempts 

(priority 1) 

− Efficiency: the average time to complete the task, measured as the average number of minutes to 

complete each task (priority 2) 

− Satisfaction, using the SUMI questionnaire (priority 3) 

The table below lists the key tasks associated with Web Software Delivery and the usability requirements for 

effectiveness, efficiency, and satisfaction. A usability test will be conducted on the released product to 

measure if the requirements criteria have been met. 

Task 

Effectiveness: 

Unassisted task completion 

rate of at least

Efficiency: 

Maximum user time (not 

machine time) of

Satisfaction: 

Post-task SUMI score 

of at least:

Download SITE guard 

software 

90 % 10 minutes 

Burn SITE guard software to 

DVD 

90 % 15 minutes 

Prepare SITE guard 

software for staged install 

and launch installer 

75 % 25 minutes 

Product overall scores 75 
NIST/IUSR – CISU-R

41

H.3 Product Details 

Name and Version: CIFWorks Web Software Delivery Release 1 

Parts for which requirements provided: All parts of the software 

Intended user groups: System Administrator, experienced with server software installation 

Training required: None (for intended user groups) 

Work supported by product: Downloading and preparing CIFWorks’ Site Guard software for 

installation. 

Usage environment: Medium to large size corporate and public sector companies. 

Users with special needs: Product will meet standards for Section 508 of the Rehabilitation Act 

H.4 Product Objective 

The objective of the product is to ultimately replace mailing DVD’s as the default mechanism for shipping 

CIFWorks’ software. 

H.5 Context of Use 

The context of use section of this document has not been changed from the Level 1 version (15 May 2006) 

H.5.1. Users 

System Administrators 

System Administrators are experienced computer professionals whose primary roles are the installation, 

configuration, and administration of multi-user, production, software . These users are typically administrators 

of other enterprise software products such as operating systems, databases, and web servers. 

System Administrator’s educational backgrounds are varied as some receive formal academic training in 

computer science or computer systems management while others receive training on the job or through 

vendor or 3rd party courses. 

The most important characteristic of a System Administrator is on the job experience. There are typically two 

routes by which these users gain experience as a System Administrator: 

− They extend their background from being a PC administrator or power user into the role of a System 

Administrator by industry training, 

− They grow their skills in an organization under the tutelage of a Senior System Administrator. 

For Web Software Delivery we expect System Administrators to have 1-10 years of experience in their job. In 

addition to software installation, and configuration, their day to day tasks would include operations (adding 

users, adding disk space, monitoring an existing system), and troubleshooting. They may also perform 

planning operations for future software and hardware acquisitions. 

H.5.1.1. Users with Disabilities: 

Users with vision, hearing, and motor disabilities should be able to function as a System Administrator with 

appropriate assistive devices (e.g. screen readers). 
NIST/IUSR – CISU-R 

42 

H.6 Goals 

The general goal is to obtain the appropriate version of the CIFWorks Site Guard software for the appropriate 

platform in a form ready for installation. Obtaining the software completes the purchasing process but may or 

may not lead immediately to installation. In some cases hardware or software has to be configured before 

installation of the software begins. Users may immediately burn the electronic version of the software onto 

DVD’s, however this is not required for installation. Some users will only install the product from the 

downloaded files (referred to as a “staged install”). Based on this there are three tasks associated with 

electronic delivery. 

1. Finding and downloading the Site Guard software via Web Software Delivery 

2. Create DVD’s from the downloaded Site Guard software and begin the install. 

3. Prepare the software for an electronic or “staged” install and begin the install 

In the case of tasks 2 and 3, the tasks end when the user has started the installation successfully. 

H.7 Equipment 

To use Web Software Delivery the following computing environment will be required. 

− A computer capable of running the browsers listed below. 

− Internet Explorer (6.0 and higher), Netscape (6.0 and higher), or Mozilla Firefox 1.1 and higher 

− 10 gigabytes of disk space to download the zip files required for installation and create the staged 

installation. 

− 500 megabytes of RAM 

− A 10mbs or faster internet connection. 

− A SVGA compatible monitor. 

− A DVD Burner 

H.8 Physical and Social Environment 

− The physical environment will be either a corporate/ public sector office or a home office with a high 

speed internet connection (10 mbs or faster). The home office may be used to download the software 

and burn CD’s after hours. 

− Typically system administrators multi-task. It is expected that using web software delivery will be 

mixed in with other support tasks they are completing. 

− There is typically a time lag between acquiring and installing the software, and rolling out the software 

into production. Because of that acquiring and installing the software will likely be part of a planning 

exercise without mission criticality pressure. However due to hacker attacks or other security 

concerns there may be times when obtaining Site Guard may be urgent. 

H.9 Scenarios of use 

Scenario 1: Download the Site Guard software and burn it to DVD 
NIST/IUSR – CISU-R

43

Ann is the lead system administrator at Biagra a producer of genetically engineered wheat products. Last 

month her company was hit by a denial of service attack by a group of organic fundamentalists. It was a 

rough month and Ann ended up shutting down the company website leading to unhappy customers and 

employees. It took no time to get the PO approved for Site Guard and now she is on the hot-seat to get it 

installed on all the companies servers. She is glad she can get the software online since she does not 

want to wait for a physical shipment. Since her company’s firewall is closed, she has to download the 

software at home and take it into the office and install it on her 2 primary servers. 

After dinner she turns on Toy Story 2 for the kids and sits down in the study to download the software. 

She has the printed out email from CIFWorks and uses it to login to the Web Delivery site. After login, 

she sees the software she purchased and online instructions for how to download it. She uses Mozilla for 

the download as it has a built in download manager. She begins the first DVD download and then goes 

to check on the kids. Occasionally she peeks in on the computer to see how the download is going. She 

is happy to see that it is progressing nicely, although it is still a bit slow. 10 minutes later the first DVD is 

downloaded so she starts the second DVD download. 20 minutes later she has put the kids to bed and 

comes back in to see that the second DVD download is complete. 

Going back to the CIFWorks website she launches a utility that will help her burn the DVD’s. It asks her 

for her computer information and then asks her to insert a DVD. While the DVD is burning she sees a 

progress indicator. It says 20 minutes to go, so she surfs the internet for a while. Twenty minutes later 

she starts the second DVD. At the end of the evening she has two DVD’s ready for installation in the 

morning. 

Back at work, she takes the DVD’s and pops them into her development environment. The development 

environment is a clone of the production web server that she uses to test out installations. She opens a 

Unix Shell and launches the installer. She is glad to see that the installation starts successfully. 

Scenario 2: Download the Site Guard software and create a staged installation 

Winston is a Senior System Administrator for the University of South-East North Dakota. He administers 

the systems that run the school’s websites. He takes pride in maintaining a safe system despite the 

many hackers at the school. As part of his 2005 budget, he got approval to purchase Site Guard and 

deploy it to the server farm that is used for the university website. 

Today he received the email from CIFWorks with his license number and instructions for downloading the 

software. Winston always likes to install software from a staged environment. This makes it much faster 

to take a master copy of the software and install it on multiple machines in his server farm. He has 

designated a system called “stage_guard” to be the stage for Site Guard. He prints out the email (he 

can’t access it from stage_guard) and walks over to the stage_guard machine. He logs onto the 

CIFWorks website. He sees the software that he needs to download and begins the first download. 

While the first download is completing he looks around on the website for instruction on building a staged 

install. He sees a link to instructions for staged install on the side of the web page. He clicks the link and 

starts reading the instructions. The instructions say he can either do it manually or download a script. 

Being security conscious he does not trust “foreign” scripts so he does it the manual way. 

He follows the instructions and begins creating the directories he will need for the install. It takes a while 

to create all the directories, as he has to create them using command line UNIX and then set privileges 

for them. He checks back on the first download and sees it is complete. He starts the second download. 

While that is working he completes the directories he needs and focuses on unzipping the files. The 

instructions say to use a zip utility to unpackage and put the files in the proper directories. He 

appreciates the fact that the instructions list the exact command to use with his zip utility. He unzips the 

files from the first download and puts them into the appropriate directory. After that is complete he 

checks back in with the download and sees the second download is complete. He unzips that file and 

puts it into the second directory. He goes back to the first directory and runs the installer. He is glad to 

see it has come up and he can proceed with the installation. 
NIST/IUSR – CISU-R 

44 

H.10 Performance and Satisfaction Criteria 

The performance and satisfaction criteria have been updated in this version to update target values, based on 

results of benchmark usability testing and additional user research via web surveys. 

H.10.1. Goals 

The goal for using Web Software Delivery is to find and download the software for the appropriate platform in 

a form ready for installation. This goal is completed when the actual installation begins, or is ready to begin. 

There are three core tasks to complete this goal with Web Software Delivery. 

1. Download SITE guard software 

This task has the highest priority since successful delivery of the software is a prerequisite to 

the other tasks. 

2. Burn SITE guard software to DVD and launch installer 

This task has second priority since it is the recommended approach for installing the software. 

3. Prepare SITE guard software for staged install and launch installer 

This task is lowest priority as it is an optional way of installation. 

H.10.2. Criteria 

The table below lists the key tasks associated with Web Software Delivery and the usability requirements for 

effectiveness, efficiency, and satisfaction. These target values were determined through interviews and 

observations of typical users completing comparable tasks, in consultation with the deployment manager and 

marketing, and updated based on benchmark usability testing of the current delivery methods to ensure that 

these target criteria would result in an improvement over the current delivery methods. 

− Effectiveness: the unassisted task completion rate. 

Unassisted task completion rate is the percentage of users who complete the tasks without assists 

from the usability test administrator. This criteria is the first priority, as successful completion of each 

task is the most important requirement for users. 

− Efficiency: the average time to complete the task 

Efficiency will be measured as the average number of minutes to complete each task. Only user time 

will be measured. Machine time (for example, actual download time) will not be counted. This is the 

second priority, as the time taken by the machine is expected to be longer than the user time. 

− Satisfaction, using the SUMI questionnaire, administered after completing the three tasks. This is also 

a second priority because any lack of satisfaction will reduce users’ willingness to continue to use web 

delivery. 

The scenarios of usage for each task are listed below with the target values for each task 

Task 

Effectiveness: 

Unassisted task 

completion rate of at 

least

Efficiency: 

Maximum user time 

(not machine time) of

Satisfaction: 

Post-task SUMI score 

of at least:

Download SITE guard 

software 

90 % 10 minutes 

Burn SITE guard software to 

DVD 

80 % 15 minutes 

Prepare SITE guard 

software for staged install 

and launch installer 

70 % 30 minutes 

Product overall 50 
NIST/IUSR – CISU-R

45

 Table 1: Target criteria values 

The unassisted task completion rate was set at 90 % since it represents the highest unassisted task 

completion rate that could realistically be achieved given user variability in a particular usability test. 

The unassisted task completion rate for Task 3 (staged install) was set to 70 %. Building a staged install is a 

more complex approach to Web Software Delivery that involves manual steps. CIFWorks will support this 

approach with instructions but given the variability in user environments and skill we expect the success rate 

to be lower. We consider this acceptable since there will be a smaller percentage of users who use staged 

install than DVD install. In addition we will have a help desk available to walk users through this approach. 

The target value for efficiency (task completion time) was set for all tasks based on a marketing research 

survey done on Web software delivery, and confirmed through benchmark usability testing. All times listed 

are for user interaction only (not machine time). Given the high variability of networks and machines we could 

not accurately predict machine time for these tasks. 

The target value for Satisfaction was based on the SUMI questionnaire. SUMI is an industry standard tool for 

measuring user satisfaction. The SUMI score will be calculated by aggregating the post test survey of each 

individual participant. The industry average for SUMI questionnaires is 50, but given the importance of the 

business goal of converting users to Web delivery, the value of 75 was chosen as CIFWorks’ benchmark for 

user satisfaction as measured by SUMI. 

For all tasks the user is a System Administrator (as defined in Context of Use: Users) with between 1-10 years 

of experience administering server based software. Tasks 1 and 2 may be completed in either a home office 

with a high speed internet connection or in an office setting. Task 3 would be completed in an office setting. 

For tasks 1 and 2 the effectiveness requirements are definitive (not subject to negotiation) due to corporate 

directive at CIFWorks. The efficiency and satisfaction metrics for Tasks 1 and 2 are provisional requirements, 

which are open to negotiation with customers of the software. All requirements for Task 3 are objectives for 

guidance intended for the internal development team. We cannot predict or control the breadth of 

environments at customers’ sites so we are keeping the requirements at a high level. 

H.11 Annex A: Usability test requirements format 

This annex provides a format for specifying the method to be used to test whether the usability requirements 

have been met. The format is similar to CIF (ISO/IEC 25062), and the results of the test can subsequently be 

documented using the CIF. 

H.11.1. Users 

A minimum of 8 users should test the product. All users should be System Administrators with at least 1 year 

of experience as a system administrator managing server software. 

H.11.2. Goals 

Refer to Performance and Satisfaction Criteria in the body of the document for Tasks and Goals. 

H.11.3. Test facility 

The test should be conducted in a usability lab that consists of a user room and a control room. The user room 

will be an office where the participants conduct the tasks. The lab should be equipped with audio and video 

monitoring and recording devices, as well as a scan converter that allows the display on the participant’s 

monitor to be video recorded. 
NIST/IUSR – CISU-R 

46 

H.11.4. Participant’s computing environment 

The participants computing environment should meet the minimum set forth in the section “Equipment” in the 

main body of this document. 

H.11.5. Display devices 

As mentioned in the “equipment” section of this document, a minimum of a color VGA monitor with capability 

of 800x600 resolution is required. 

H.11.6. Experimental Design 

There will be one order that the tasks are presented in since the second and third tasks are dependent on the 

first one. The six dependent measures for this study should be: 

− Unassisted task completion rate (the percentage of participants who completed a task without 

assistance from the tester) 

− Assisted task completion rate (the cumulative percentage of people who completed a task with zero, 

one or two assists from the tester) 

− Task time 

− Number of errors 

− Number of assists 

− Self-report satisfaction via SUMI measures.

H.11.7. Procedure 

A minimum of eight sessions, one for each participant, shall be conducted. The participant will work in the 

user room, while the test administrator will observe from the control room behind a one-way mirror. A system 

of microphones and speakers will allow the administrator and user to communicate. At the beginning of each 

session, participants will be given a consent form and non-disclosure agreement. After completing the forms, 

participants will be instructed on the nature of the testing session and the software. 

Data collection will begin as the participant works through the tasks. As participants work, a time-stamped 

datalog will be created. The datalog will note errors, assists, user comments, and other observations of 

interest such as navigation strategies. Upon completion of the tasks, participants will fill out the SUMI. 

Participants will be thanked for their time, and given their compensation. Compensation for this session 

should be approximately $150 given the experience of the participant and the length of the study (2-3 hours).

H.11.8. Participant general instructions 

Participants should be told that the study session consists of two parts: working on tasks with the application, 

and filling out some post-task surveys. Participants shall be instructed to read each task out loud, and to work 

through the tasks, “as efficiently as possible.” Participants shall be assured that it is the software, not them, 

that is being tested, and to feel free to be honest in their feedback of the software.

H.11.8.1. Participant task instructions 

************************************************************************************** 

Task Scenarios – Web Software Delivery 

You (Sandy Lee) are a System Administrator working at a company called Marsh International that 

manufactures fencing products. Your company is located in Fremont, California. 
NIST/IUSR – CISU-R

47

The Director of Information Technology (Dilip) has just signed a deal with CIFWorks to use Site Guard to 

protect your corporate internet site. The deal also includes support and upgrades. 

Dilip emails you the ordering document that details the contract with CIFWorks. As part of the deal with 

CIFWorks, he tells you that the software will be delivered electronically via a download. 

Exercise 1: Download the software 

Using the information in the email open on your desktop, download the database software that your company 

has purchased. Keep in mind the following. 

1. You want to download the product for Solaris, Spark 64. 

2. You want the latest version of the product. 

3. You only want to download the software that you have licensed. You don’t want any extras for trial 

purposes. 

4. Here is your company information 

Marsh International 

500 Woodsy Lane 

Fremont, CA 96009 

5. Here is your information 

Sandy Lee 

Senior System Administrator 

sandy.lee@marsh.com 

Exercise 2: Make DVDs 

Now that you have downloaded the software you decide to make a set of DVDs of the software for safe 

keeping. Using the burnable DVDs we have provided to you create a set of DVDs that will allow you to install 

the software from these DVDs. 

Exercise 3: Install the database onto a Sun Machine without using the DVDs 

You have a development machine you want to install Site Guard onto (dev2). However, the machine does not 

have a DVD drive. Install the software that you have downloaded onto the Sun Machine without using the 

DVDs you have just created. 

Tester Note: The task here ends when the user successfully starts the installer assuming they have completed 

all other steps correctly. They don’t have to run through the install process, as we are not testing the installer. 

H.12 Usability metrics 

H.12.1. Effectiveness 

Completion Rate: Both assisted and unassisted task completion rates should be calculated. Unassisted 

completion rate is defined as the percentage of participants who successfully complete each task without any 

assistance from the tester. Assisted task completion rate is defined as the percentage of participants who 

complete each task with two or fewer assists from the tester (note that the assisted task completion rate 

includes the unassisted task completion rate). Participants fail to complete tasks with assistance if they require 

3 or more assists, exceed the maximum time limit for each task, or if they give up on the task. 
NIST/IUSR – CISU-R 

48 

Errors: A deviation from the prescribed method to complete a task that did not result in goal attainment shall 

be counted as an error. Consulting the on-line help shall not be recorded as an error. 

Assists: Any intervention by the tester for any of the following reasons shall be counted as an assist. 

− Preventing the test participant from performing an action that would result in a condition from which 

the task could not be completed. 

− Providing instruction in order for the test participant to complete the task or end the task as a result of 

surpassing the amount of time allocated to complete the task (more than 4 times the benchmark 

time).

H.12.2. Efficiency 

Efficiency relates the level of effectiveness achieved to the quantity of resources expended. Efficiency is 

generally assessed by the mean time taken to achieve the task. Efficiency may also relate to other resources 

(e.g. total cost of usage). A common measure of efficiency is time on task. For this study efficiency shall be 

measured as the average amount of time required to complete the tasks.

H.12.3. Satisfaction 

Satisfaction describes a user’s subjective response when using the product. User satisfaction may be an 

important correlate of motivation to use a product and may affect performance in some cases. Questionnaires 

to measure satisfaction and associated attitudes are commonly built using Likert and semantic differential 

scales. 

SUMI scores: SUMI is an instrument specifically designed to measure user satisfaction with commercial 

software. It has been used with hundreds of thousands of participants and enables comparison with industry 

norms for satisfaction with usability on 6 dimensions: global (overall, efficiency, affect, helpfulness, control, 

and learning. An industry average SUMI score on any index is 50. 
NIST/IUSR – CISU-R

49

Bibliography 

[1] Brooke, J. (1996). SUS: A “quick and dirty” usability scale. Usability Evaluation in Industry. UK: Taylor and 

Francis 

[2] Chin, J. P., Diehl, V. A., and Norman, K. (1988). Development of an instrument measuring user 

satisfaction of the human-computer interface. In the Proceedings of ACM CHI ‘88 (Washington D.C.), 213-

218. 

[3] Dumas, J. & Redish, J. (1999) A Practical Guide to Usability Testing, revised edition. Bristol, England: 

Intellect.. 

[4] Kirakowski, J. (1996). The software usability measurement inventory: Background and usage. In Jordan, 

P., Thomas, B., and Weerdmeester, B. (Eds.), Usability Evaluation in Industry. UK: Taylor and Francis. 

[5] Lewis, J. R. (1991). Psychometric Evaluation of an After-Scenario Questionnaire for Computer Usability 

Studies: the ASQ. SIGCHI Bulletin, 23(1), 78-81. 

[6] Lewis, J. R. (1995). IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and 

Instructions for Use. International Journal of Human-Computer Interaction, 7, 57-78. 

[7] Nielsen, J. & Landauer, T. K. (1993) A mathematical model of the finding of usability problems. In: CHI 

'93. Conference proceedings on Human factors in computing systems, 206-213 

[8] Norman, D.A. (1983) Design Rules Based on Analyses of Human Error. Communications of the ACM, 

26(4), 254-258. 

[9] Rubin, J. (1994) Handbook of Usability Testing, How to Plan, Design, and Conduct Effective Tests. New 

York: John Wiley & Sons, Inc.