NIST Special Publication 500-316 

Framework for Cloud Usability
 

Brian Stanton 

Mary Theofanos 

Karuna P. Joshi 

This publication is available free of charge from: 

http://dx.doi.org/10.6028/NIST.SP.500-316 

NIST SP 500-316 12/02/2015 
 

 

 

 

 

 

NIST Special Publication 500-316 

Framework for Cloud Usability
 

Brian Stanton 

Information Access Division 

Information Technology Laboratory 

Mary Theofanos 

Office of Data and Informatics 

Materials Measurement Laboratory 

Karuna P. Joshi 

UMBC 

This publication is available free of charge from: 

http://dx.doi.org/10.6028/NIST.SP.500-316 

December 2015 

U.S. Department of Commerce 

Penny Pritzker, Secretary 

National Institute of Standards and Technology 

Willie May, Under Secretary of Commerce for Standards and Technology and Director 

NIST SP 500-316 12/02/2015 
 

 

 

 

 

Certain commercial entities, equipment, or materials may be identified in this 

document in order to describe an experimental procedure or concept adequately. 

Such identification is not intended to imply recommendation or endorsement by the 

National Institute of Standards and Technology, nor is it intended to imply that the 

entities, materials, or equipment are necessarily the best available for the purpose. 

National Institute of Standards and Technology Special Publication 500-316 

Natl. Inst. Stand. Technol. Spec. Publ. 500-316, 17 pages (December and 2015) 

CODEN: NSPUE2 

This publication is available free of charge from: 

http://dx.doi.org/10.6028/NIST.SP.500.316 

NIST SP 500-316 12/02/2015 
NIST SP 500 -316 Page ii 12/02/2015 
 

 

 

Table of Contents 

1. INTRODUCTION ........................................................................................................................................... 1
	

2. BACKGROUND.............................................................................................................................................. 2
	

2.1 CLOUD STANDARDS................................................................................................................................. 2
	

2.2 THE CLOUD USER EXPERIENCE ............................................................................................................... 4
	

3. PROPOSED FRAMEWORK FOR CLOUD USABILITY ......................................................................... 4
	

3.1 CLOUD USABILITY FRAMEWORK............................................................................................................. 5
	

3.1.1 Capable .......................................................................................................................................... 6
	

3.1.2 Personal ......................................................................................................................................... 7
	

3.1.3 Reliable.......................................................................................................................................... 8
	

3.1.4 Valuable......................................................................................................................................... 9
	

3.1.5 Secure ............................................................................................................................................ 9
	

4. CONCLUSION AND ONGOING WORK............................................................................................. 10
	

5. REFERENCES ............................................................................................................................................... 10
	

NIST SP 500-316 Page iii 12/02/2015 
 

 

List of Figures 

FIGURE 1 THE CLOUD USER EXPERIENCE 11 

FIGURE 2 THE USER CENTERED DESIGN PROCESS 12 

FIGURE 3 CLOUD USABILITY FRAMEWORK 13 

NIST SP 500-316 Page iv 12/02/2015 
 

 

 

EXECUTIVE SUMMARY
 

Organizations are increasingly adopting cloud-based services to meet their business needs. 

However, due to the complexity and diversity of cloud systems it is important to evaluate 

the user experience within a framework that encompasses the characteristics that define the 

user experience. In this paper, we propose a cloud usability framework to provide a structure 

to evaluate the key attributes of the cloud user experience. The framework includes five 

attributes and 21 elements that characterize this user experience. Generally these describe 

the consumer’s expectations of the cloud. 

NIST SP 500-316 Page v 12/02/2015 
 

1. INTRODUCTION 

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access 

to a shared pool of configurable computing resources (e.g., networks, servers, storage, 

applications, and services) that can be rapidly provisioned and released with minimal 

management effort or service provider interaction (this last part of the definition bear a 

strong resemblance to usability measures) [15]. Organizations are increasingly adopting 

cloud-based services to address their information technology (IT) needs for software, 

hardware or network bandwidth [5]. To date, the driving forces behind the cloud have been 

technological advances and business needs that often outpace the needs of the end-user. In 

order to better enable ready adoption (acceptance and utilization), users’ needs (and their 

goals) must be incorporated into the process of cloud implementation. Fundamentally, users’ 

needs and goals drive usability. 

Various cloud service providers employ different models and deployment types of cloud 

services and resulting in no consistency to the user experience. Moreover, cloud services 

deployed via the web or mobile device are limited by the capabilities of the web browser or 

mobile application interface and often provide user interfaces that are of lower quality than 

that provided by traditional client server applications on a desktop platform. It is being 

increasingly recognized by the cloud user community that there is a need for developing 

cloud Usability standards to ensure consistency among different cloud services purchased 

from different vendors [18]. Lack of a consistent user experience in an enterprise cloud 

environment can result in longer time to on-board an application into the enterprise cloud, 

more training requirements and more technical assistance for the End Users. This will affect 

the actual cost of the application to the organization. If an organization with hundreds of end 

users can reduce their training requirements by a day, they can improve their organizational 

productivity and realize cost savings. 

The cloud pushes us to examine the user experience beyond the standard desktop computing 

environment and models of well established usability principles, guidelines, and metrics. 

Thus we have developed a framework to provide a structure to evaluate the key attributes of 

the cloud user experience. The framework driven by user needs identifies five attributes 

further compartmented into elements and associated measures to ensure that critical usability 

areas are not overlooked. 

NIST SP 500-316 Page 1 12/02/2015 
NIST SP 500-316 Page 2 12/02/2015

2. BACKGROUND

The complexity and diversity of cloud systems has made it difficult to evaluate the user 

experience using typical usability evaluation techniques. 

2.1 CLOUD STANDARDS

The National Institute of Standards and Technology (NIST) cloud computing initiative was 

aimed at standardizing the definition of cloud computing, its various models and deployment 

strategies. NIST’s definition of the cloud mandates five essential characteristics [15] of ondemand self-service, broad network access, resource pooling, rapid elasticity and measured 

service. The NIST deployment models [15], that include Private, Public, Community and 

Hybrid cloud models, do not specifically address the attributes of human computer 

interaction or usability that make up the user experience of a cloud service.

The NIST cloud model enumerates three service models [15]. Usability elements to consider 

for each service model are different and depend on the level of control the end user and the 

consumer organization have over the cloud service. The service models include:

• Software as a Service (SaaS): provides consumers the use of applications in the cloud.

The consumer does not control or manage the applications or the environment in which

the application is running. In this model the end user and the consumer organization

generally have very limited capability of customizing the user interface of the cloud

service. For instance, Google Apps (http://www.google.com/enterprise/apps/business/)

that are being adopted as Enterprise Email applications in many organizations have the

same look and feel for all the consumers and limit customization of the user interface to

colors and company logos.

• Platform as a Service (PaaS): provides consumers with the ability to deploy applications

created or acquired by the consumer using programming languages, libraries, services,

and tools supported by the provider. The consumer does not manage or control the cloud

infrastructure, but has control over the deployed applications. In this model, the end user

and the consumer organization generally have capability of customizing the user

interface of the cloud service. However they are limited by the capability of the platform

and cannot use other open source languages, like C or Java, to develop custom

applications. For instance, Salesforce.com provides ‘Salesforce1 Platform’ application

[19] as a PaaS cloud model that allows end users to develop apps on the salesforce

platform. However, they are limited to the tools provided by Salesforce on that offering

and so might be limited in getting the same user interface as other applications in their

enterprise cloud that are purchased from other vendors.
NIST SP 500-316 Page 3 12/02/2015

• Infrastructure as a Service (IaaS): provides consumers with the ability to provision

processing, storage, networks, and other fundamental computing resources where the

consumer is able to deploy and run arbitrary software, which can include operating

systems and applications. The consumer does not manage or control the underlying cloud

infrastructure but has control over operating systems, storage, and deployed applications.

For instance, Amazon Web Services (http://aws.amazon.com/) provide an IaaS

environment to their consumers who can develop personal applications using languages

and tools of their choice. This model provides end users with the maximum flexibility of

customizing their cloud services and so facilitates development of a consistent user

interface. However, IaaS cloud services also require consumer organizations to develop

in-house expertise to be able to utilize the benefits of the cloud.

As these cloud computing environment characteristics and components demonstrate 

decisions about moving to the cloud can be overwhelming to organizations/consumers. [7]

Many frameworks and taxonomies have been developed to assist enterprises as they consider 

the use of the cloud. [6][2][17] These frameworks generally focus on technical capabilities 

and measure business-relevant attributes. Others provide mechanisms for comparing business 

services. For example, Garg et al [6] developed a framework that measures the quality of 

cloud services and ranks. Usability features have only been mentioned in passing, if at all.

While these frameworks may be useful for evaluating the cloud in regards to technical and 

business needs they are not sufficient to evaluate cloud usability. 

The study of the usability of cloud-based services has been limited so far. We believe an 

assumption of the cloud community appears to be that cloud environments are readily usable 

as they are an extension of the existing computing environment. However the complexity and 

diversity of cloud computing environments have demonstrated the necessity for a usability 

framework that encompasses the added dimensions and provides a structure such that key 

areas are not overlooked. Key areas of the framework include cloud usability requirements 

generation and evaluation. 

The usability community has a long history of using frameworks to evaluate usability of 

systems. From the International Organization for Standardization (ISO) usability measures 

of effectiveness, efficiency, and user satisfaction [8] that influenced the desktop computing 

evaluation models, to those developed for pervasive or ubiquitous computing [1][4][13][16], 

to those for mobile computing [3][14] each is tailored to accommodate the key differences 

that define the user’s computing environment
NIST SP 500-316 Page 4 12/02/2015

2.2 THE CLOUD USER EXPERIENCE

While users experience the cloud through a user-interface, organizations enable the end user 

experience by setting up the cloud service and its characteristics. The end user interacts with 

the cloud and its cloud based services through its applications (see Figure 1). These 

applications are effected by attributes of the cloud and the services and will exhibit behaviors 

that can be traced back to the cloud. Testing the applications reveal the usability of the 

cloud and cloud services that is available to the end user. 

3. PROPOSED FRAMEWORK FOR CLOUD USABILITY

Any usability framework should start with a good understanding of what usability is. For our 

purposes, we used the ISO definition of usability: 

Usability is the extent to which a system, product, or service can be used by specified 

users to achieve specified goals with effectiveness, efficiency, and satisfaction in a 

specified context of use. [8] 

Figure 1 The Cloud User Experience
NIST SP 500-316 Page 5 12/02/2015

Figure 2 The User Centered Design Process

The definition provided us with a roadmap for developing the framework. It first talks about 

users and their goals that have often been left out when considering the cloud. The definition 

then goes on to state three metrics: effectiveness, efficiency, and satisfaction. Finally, in the 

definition, states that the aforementioned all occur within a specified context of use. It is 

during the development of the context of use that users’ needs (and their goals) are first 

identified in the User Centered Design (UCD) process[8]. As can be seen in Figure 2, User 

Needs are central to developing a usable system, product, or service. It is these user needs 

that form the basis of our framework.

3.1 CLOUD USABILITY FRAMEWORK

Figure 3 is a pictorial representation of the NIST Cloud Usability Framework. This figure 

lists the various usability attributes that are of interest to cloud users.

Context	of	Use

User	&

Organizational

Requirements

Design	Solution

Evaluation

User	

Needs
NIST SP 500-316 Page 6 12/02/2015

Figure 3 Cloud Usability Framework

In a similar vein as the Cloud Services Measurement Initiative Consortium (CSMIC), who 

created a framework to establish measures for quality of cloud services (Service 

Measurement Index, SMI [2]), we independently created a framework that characterizes the 

usability attributes that the cloud should contain. The framework’s attributes and elements 

are based on other usability frameworks [16] and ISO usability standards. Consumers should 

expect the cloud to be Capable, Personal, Reliable, Valuable, and Secure. Figure 3 illustrates 

this framework.

3.1.1 Capable

Cloud consumers should expect their cloud services to have certain capabilities. The four 

main features that they should look for in a capable cloud service are:

1. Current: The cloud service should be based on the latest technology. The provider

should upgrade the service to stay current with the state-of-art of the technology. The

service provider should ensure that the cloud service stays compatible with the latest

release of hardware, Operating System (OS) and related software applications on which it

is dependent. Currently providers upgrade the technology of their services every 18-24

months. In the coming years, we may see a reduction in this time span.

Capable	

Current	

Device	Independence	

Platform	Independence	

Functions	

Personal	

Accessibility	

Personalization/	

Customization	

Data	Ownership	

Identity	Management	

(access)	

Control	

Reliable	

Available	

Responsive	

Consistent	

Transparency		

Secure	

Security	

Privacy	

Identity	Management	

(authorization)	

Trust	

Valuable	

Savings	

User	satisfaction	

Utility	

Consumer

(End user & organization)

expects the cloud to be:
NIST SP 500-316 Page 7 12/02/2015

2. Device Independence: For a cloud service to be capable, consumers should be able to

access the cloud service using any device type – fixed/mobile etc.

3. Platform Independence: Consumers who use IaaS services want to be able to define the

exact hardware specifications of the service they wish to purchase. For SaaS services,

consumers want the cloud service to be independent of the cloud hardware, Operating

System, etc. so that the service can function correctly on any device hardware and

software setup.

4. Cloud Functionality: Cloud services can be considered capable for the consumers if

they provide cloud functionality, like elasticity, scalability; rapid provisioning, etc., that

is not possible in other platforms.

3.1.2 Personal

Cloud Services should allow consumers (organizations and end users) to change the look and 

feel of the user interface and customize service functionality to suit their needs. At times, an 

organization’s IT policy may not allow the end users to personalize the user interface of any 

enterprise application; however, they may still customize the basic solution for the whole 

organization. There are five main elements that a User should expect under the Personal 

attribute:

1. Accessibility: It is the degree to which a product or system can be used by people with

the widest range of characteristics and capabilities to achieve a specified goal in a

specified context of use [11]. Cloud services should be accessible to consumers with a

variety of needs. For instance, many consumers in the United States mandate that the

solution should be compliant with the Americans with Disabilities Act (ADA).

2. Customization: Customization is adaptation of a software product to the needs of a

particular audience [12]. Cloud services should allow consumers to change their user

interface to suit their own needs. An organization may choose a standard customized

interface for all their end users to ensure lower maintenance needs.

3. Data Ownership: Consumers should have complete ownership over the data they store

in the cloud services they use. Users should be able to declare the policies on its usage (in

advertisements, other services etc.). For instance, if a consumer uploads pictures to a

cloud service, the consumer should be able to determine how that image is used or who

has access to it.

4. Identity Management (access): To ensure ease of use, multiple access authentications

will have to be implemented in a seamless manner so that the consumer is not aware of

the number of authentication/authorization steps they have to go through to access their

applications on the cloud.
NIST SP 500-316 Page 8 12/02/2015

5. Control: The consumer should have a sense of control over the functionality of the

cloud service. For instance, they should be able to determine what cookies are set by the

cloud service, or be able to switch off Global Positioning System (GPS) tracking if the

service is accessed as a mobile application.

3.1.3 Reliable 

Reliability is the ability of a system or component to perform its required functions under 

stated conditions for a specified period of time[10]. There are five main elements that cloud 

consumers look for under the Reliable attribute:

1. Available: Availability is defined as the ability of a service or service component to

perform its required function at an agreed instant or over an agreed period of time [9].

Cloud users should expect high availability and it is not unusual to find providers

advertising over 99 % availability.

2. Responsive: cloud consumers should want the cloud services to have high degree of

performance. The main performance measure that is of interest to the end user is the

response time, which is the time it takes the service to respond to a user’s

request/instruction.

3. Consistent: A cloud service should exhibit the same functionality under every situation.

The service should also have no conflicts with the user device type (e.g., different types

of desktops) accessing the service.

4. Transparency: The cloud service provider’s service policies and technology should be

transparent to the cloud consumer organization. Organizational users (like managers,

database administrators, etc.) should have access, as needed, to the cloud datacenter and

have details about the cloud platform’s capabilities and planned changes. This feature is

important for building trust between cloud providers and cloud consumers.

5. Error-Resistant: An error-resistant program is a piece of software which resists the

adverse effects of errors. In order to accomplish this goal, the program must possess the

capabilities to detect errors, to locate and contain the propagation of errors, and to

recover from the errors [20].
NIST SP 500-316 Page 9 12/02/2015

3.1.4 Valuable

Cloud consumers should expect cloud services to provide value to them and their 

organization. The four main features that can measure this value include –

1. Savings: By using the cloud service, users will be able to save on costs with respect to

running the same application on their personal infrastructure. In addition they will be able

to consolidate and save resources they might need if they ran the same application on

their personal infrastructure.

2. User satisfaction: User satisfaction measures the affective experience of the service. The

user satisfaction measure should be high for the user to continue consuming the service.

3. Utility: Cloud services will be valuable to the consumers if they can provide new features

that are not possible with any other IT setup. For instance, cloud services are proving to

be very valuable in:

a. Distributed computing with large datasets (Big Data applications) and

b. Mobile and pervasive computing.

3.1.5 Secure

Cloud consumers want their cloud services and data to be secure. The three main features 

that they are looking for in cloud services under this category are:

1. Security: The cloud service should be resistant to attacks from unauthorized users, other

cloud services, malicious software, and attacks on cloud hardware and the Internet.

Security attacks on a service by cloud based attackers compromise the functionality of

that service. While these attacks may not directly harm the user’s private data; they will

make the cloud service more vulnerable to Privacy attacks.

2. Privacy: The cloud service can prevent leakage of data that compromises end user’s

private data like personal information, financial accounts, geolocation (if not desired by

the user), etc.

Privacy Attacks compromise a cloud user’s personal information. These attacks may not

directly harm the cloud service’s security, but can affect its performance.

To ensure a cloud user’s privacy, cloud providers authenticate or validate a user’s

credentials. Tools, patches, utilities and applications are deployed by the cloud providers

to ensure the cloud user’s privacy.
NIST SP 500-316 Page 10 12/02/2015

3. Identity Management (Authorization): The cloud service should not allow

unauthorized users to access user data or execute any process. The organization will

designate an administrator who will be able to maintain the list of users and their

authorization levels.

4. Trust: The end user should have trust in the cloud. How it handles data, how it facilitates

the users’ tasks, and the general user experience all contribute to the feeling of trust in the

cloud.

4. CONCLUSION AND ONGOING WORK

We have presented a framework for evaluating the user experience of the cloud. The 

framework builds on techniques, metrics, and guidelines from desktop computing and more 

recent work in ubiquitous computing. The goal of the cloud usability framework is to provide 

a structure to ensure that key areas of evaluation and consumer needs are not overlooked.

While this framework is based on evaluations from desktop computing, a literature review 

and personal experience, we have plans to apply it to a cloud environment.

Finally we believe the framework is the foundation for developing usability metrics for 

organizations adopting the cloud. Thus our goal is to develop a method of measuring the 

elements in the framework and defining an associated set of usability metrics. 

5. REFERENCES

[1] Bellotti, V.; Back, M.; Edwards, W.K.; Grinter, R.E.; Henderson, A.; Lopes, C. 2002. Making sense of sensing systems: Five questions 

for designers and researchers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Changing our 

World, Changing Ourselves. New York: ACM Press, 415–422.

[2] Cloud Services Measurement Initiative Consortium, The Service Measurement Index. Retrieved September 6, 2013, from 

http://csmic.org/

[3] Coursaris, C.K., Kim, D.J., A Meta-Analytical review of empirical mobile usability studies, Journal of Usability Studies, Vol. 6, Issue 3, 

pp. 117-171, May 2011 

[4] Friedman, B. and Kahn, P.H. Jr. 2003. Human values, ethics, and design. In The Human-Computer Interaction Handbook. Mahwah, 

NJ: Lawrence Erlbaum, 1177–1201.

[5] Gartner Predicts Infrastructure Services Will Accelerate cloud Computing Growth (2013, February 19). Retrieved September 12, 2013 

from http://www.forbes.com/sites/louiscolumbus/2013/02/19/gartner-predicts-infrastructure-services-will-accelerate-cloudcomputing-growth/

[6] Garg, S.K., Versteeg, S., Buyya, R., SMI cloud: A Framework for Comparing and Ranking cloud Services, Utility and cloud 

Computing (UCC), 2011 Fourth IEEE International Conference on, vol., no., pp.210, 218, 5-8 Dec. 2011. doi: 10.1109/UCC.2011.36

[7] Goyal, P., Enterprise Usability of Cloud Computing Environments: Issues and Challenges. 2010 Workshops on Enabling 

Technologies: Infrastructure for Collaborative Enterprises. (2010), 54-59.

[8] ISO/IEC 9241 -210:2010 Ergonomics of human-system interaction -- Part 210: Human-centred design for interactive systems

[9] ISO/IEC 20000-1:2011 Information Technology Service Management

[10] ISO/IEC 24765:2010 Systems and software engineering Vocabulary

[11] ISO/IEC 25010:2011 Systems and software engineering--Systems and software Quality Requirements and Evaluation (SQuaRE)--

System and software quality models, 4.2.4 

[12] ISO/IEC 26514:2008 Systems and software engineering Requirements for designers and developers of user documentation
NIST SP 500-316 Page 11 12/02/2015

[13] Jameson, A. 2003. Adaptive interfaces and agents. In The Human-Computer Interaction Handbook. Mahwah, NJ: Lawrence 

Erlbaum, 316–318.

[14] Kwahk, J., & Han, S.H. (2002). A methodology for evaluating the usability of audiovisual consumer electronic products. Applied 

Ergonomics, 33, 419-431.

[15] Mell, P. & Grance, T. (2011) The NIST Definition of Cloud Computing, (Special Publication 800-145). Gaithersburg MD: National 

Institute of Standards and Technology.

[16] Scholtz, J. and Consolvo, S. 2004. Toward a framework for evaluating ubiquitous computing applications. Pervasive Computing, 3, 2, 

82–88.

[17] Vaquero, L.M., Rodero-Merino, L., Caceres, J., and Lindner, M., A break in the clouds: towards a cloud definition. SIGCOMM 

Comput. Commun. Rev. 39, 1, pp.50-55, Dec 2008, doi: 10.1145/1496091.1496100

[18] Singleton D., “It’s time for a cloud UI standard”, http://blog.softwareadvice.com/articles/enterprise/its-time-for-a-cloud-ui-standard1021412/, last retrieved 18 Nov 2013 

[19] Salesforce1 Platform, http://www.salesforce.com/platform/overview/, last retrieved 18 Nov 2013

[20] Yau, S.S., Cheung, R.C., and Cochrane, D.C., “An approach to error-resistant software design.” In the ICSE ’76 Proceedings of the 2nd

international conference on software engineering. Pages 429-436. IEEE Computer Society Press, Loss Alamitos CA USA 1976.