Special Publication 500-307

Cloud Computing 

Service Metrics 

Description

NIST Cloud Computing Reference Architecture 

and Taxonomy Working Group

NIST Cloud Computing Program

Information Technology Laboratory
ii

This page is left intentionally blank
iii

NIST Special Publication 500-307

Cloud Computing Service 

Metrics Description

NIST Cloud Computing Reference Architecture 

and Taxonomy Working Group

http://dx.doi.org/10.6028/NIST.SP.307

2015

U.S. Department of Commerce 

Penny Pritzker, Secretary

National Institute of Standards and Technology 

Willie May, Acting Under Secretary of Commerce for Standards and Technology and Acting Director 
iv

This page is left intentionally blank
v

Reports	on	Computer	Systems	Technology

The Information Technology Laboratory (ITL) at the National Institute of Standards and 

Technology (NIST) promotes the U.S. economy and public welfare by providing technical 

leadership for the Nation’s measurement and standards infrastructure. ITL develops tests, test 

methods, reference data, proof of concept implementations, and technical analyses to advance the 

development and productive use of information technology. ITL’s responsibilities include the 

development of management, administrative, technical, and physical standards and guidelines for 

the cost-effective security and privacy of other than national security-related information in 

Federal information systems. This Special Publication 500-series reports on ITL’s research, 

guidance, and outreach efforts in Information Technology and its collaborative activities with 

industry, government, and academic organizations.

National Institute of Standards and Technology Special Publication 

500-307

Natl. Inst. Stand. Technol. Spec. Publ. 500-307, 24 pages (2015)

DISCLAIMER 

This document has been prepared by the National Institute of Standards and Technology (NIST) 

and describes technical research in support of the NIST Cloud Computing Program.

Certain commercial entities, equipment, or material may be identified in this document in order to 

describe a concept adequately. Such identification is not intended to imply recommendation or 

endorsement by the National Institute of Standards and Technology, nor is it intended to imply 

that these entities, materials, or equipment are necessarily the best available for the purpose.
vi

Acknowledgements	

This document reflects the contributions and discussions by the members of the NIST Cloud 

Computing Reference Architecture and Taxonomy Working Group – Cloud Service Metrics Sub 

Group, lead by Frederic de Vaulx of Prometheus Computing, LLC under contract with the U.S.

Department of Commerce, National Institute of Standards and Technology, Information 

Technology Laboratory.

NIST SP 500-307 has been collaboratively authored by the NIST Cloud Service Metrics Sub 

Group.

NIST would like to acknowledge the specific contributions from the following contributors:

Jane Siegel (CMU)

Jacques Durand (Fujitsu)

John Calhoon (Microsoft)

Jeff Perdue (CMU)

Steve Woodward (Cloud Perspectives)

Alan Sill (TTU)

Ken E. Stavinoha (Solutions 

Architect. Cisco Systems)

Tom Rutt (Fujitsu)

Jenny Huang (AT&T)

Eric Roggenstroh (eGT for GSA)

Omar Fink (SAIC)

Steven J. McGee (SAW Concepts LLC)

Eric Simmon (NIST)

Scott Feuless (ISG/CSMIC)

Jim Watts (TSA)

Keyun Ruan (Chief Scientist, Espion Group, 

Ireland)

Jesus Luna (CSA)

William M. Fitzgerald (EMC Information 

Systems International, Ireland)

Kimberley Laris (Positive Assurance)

Massimiliano Rak (CerICT, Italy)

Neeraj Suri (Technical University of 

Darmstadt, Germany)

David Núñez (Universidad de Málaga, Spain)

Carmen Fernández-Gago (Universidad de 

Málaga, Spain)

Isaac Agudo (Universidad de Málaga, Spain)

The NIST editors for this document were: Frederic de Vaulx, Eric Simmon and Robert Bohn.
vii

Table	of	Contents

1 EXECUTIVE SUMMARY............................................................................................................................................................................................ 1

2 INTRODUCTION.......................................................................................................................................................................................................... 3

2.1 AUDIENCE................................................................................................................................................................ 3

2.2 BACKGROUND.......................................................................................................................................................... 3

3 DEFINITIONS ............................................................................................................................................................................................................... 4

3.1 ABSTRACT METRIC ................................................................................................................................................. 4

3.2 ABSTRACT METRIC DEFINITION ............................................................................................................................. 4

3.3 CLOUD SERVICE PROPERTY.................................................................................................................................... 5

3.4 CONCRETE METRIC DEFINITION ............................................................................................................................ 5

3.5 CONTEXT ................................................................................................................................................................. 5

3.6 MEASUREMENT ....................................................................................................................................................... 5

3.7 MEASUREMENT RESULT.......................................................................................................................................... 5

3.8 METRIC.................................................................................................................................................................... 5

3.9 OBSERVATION.......................................................................................................................................................... 6

3.10 UNIT OF MEASUREMENT....................................................................................................................................... 6

4 THE ROLE OF METROLOGY IN CLOUD SERVICES......................................................................................................................................... 6

4.1 THE CLOUD SERVICE TRIFECTA............................................................................................................................. 7

4.1.1 METRICS FOR SELECTING CLOUD SERVICES .......................................................................................................... 7

4.1.2 METRICS FOR SERVICE AGREEMENTS (SAS).......................................................................................................... 8

4.1.3 METRICS FOR SERVICE MEASUREMENT................................................................................................................. 8

4.2 OTHER METRICS ..................................................................................................................................................... 9

4.3 SCENARIO ................................................................................................................................................................ 9

5 CLOUD SERVICE METRIC MODEL..................................................................................................................................................................... 11

5.1 CLOUD SERVICE METRIC ECOSYSTEM................................................................................................................. 11

5.2 MODEL CHARACTERISTICS ................................................................................................................................... 12

5.2.1 CONSISTENT REPRESENTATION OF INFORMATION................................................................................................ 12

5.2.2 EXPLICIT RELATIONSHIPS.................................................................................................................................... 12

5.2.3 REPOSITORY OF DEFINITIONS .............................................................................................................................. 12

5.2.4 COMPARABILITY ................................................................................................................................................. 12

5.2.5 FLEXIBILITY AND ADAPTABILITY ........................................................................................................................ 12

5.2.6 COMPOSABILITY.................................................................................................................................................. 12

5.3 CLOUD SERVICE METRIC DIAGRAM..................................................................................................................... 12

5.4 CLOUD SERVICE METRIC ELEMENT DEFINITIONS............................................................................................... 13

5.4.1 ABSTRACTMETRIC CLASS ................................................................................................................................... 14

5.4.2 RULEDEFINITION CLASS ..................................................................................................................................... 15

5.4.3 PARAMETERDEFINITION CLASS........................................................................................................................... 16

5.4.4 METRIC CLASS .................................................................................................................................................... 16

5.4.5 METRICRULE CLASS ........................................................................................................................................... 17

5.4.6 METRICPARAMETER CLASS................................................................................................................................. 17

6 HOW TO USE THE CSM MODEL........................................................................................................................................................................... 19

7 MEASUREMENT UNCERTAINTY......................................................................................................................................................................... 20

8 OTHER CONSIDERATIONS.................................................................................................................................................................................... 21

8.1 METRICS USED FOR PROPERTY COMPOSITION..................................................................................................... 21

8.2 CALIBRATION & MEASUREMENT STANDARD (ETALON)...................................................................................... 21

9 CONCLUSION............................................................................................................................................................................................................. 22

ANNEX A - DEFINITIONS SURVEY................................................................................................................................................................................ 23

ANNEX B - REFERENCES ................................................................................................................................................................................................. 25
viii

List	of	Figures

Figure 1 Metric and Property .............................................................................................................................................. 7

Figure 2 Cloud Service Selection........................................................................................................................................ 8

Figure 3 Cloud Service Agreement..................................................................................................................................... 8

Figure 4 Cloud Service Objectives Monitoring .................................................................................................................. 9

Figure 5 Scenario and Metric............................................................................................................................................ 10

Figure 6 Cloud Service Metric Ecosystem Model............................................................................................................ 11

Figure 7 Cloud Service Metric (CSM).............................................................................................................................. 13

Figure 8 CSM Definitions Blocks..................................................................................................................................... 19

Figure 9 Metric Definition Process................................................................................................................................... 20
1

1 Executive	Summary

With cloud computing in the mainstream, there is a preponderance of cloud based services in the 

market and the choices for consumers increase daily. However, comparing the service offerings 

between cloud service providers is not a straightforward exercise. To be successful in procuring 

cloud services, one must have requirements that are clear, create service level agreements (SLA) 

which reflect these requirements and be measureable in order to validate the delivery of these 

services along with their performance and remedies. 

As part of the decision making framework for moving to the cloud, having data on measurable 

capabilities, for example - quality of service, availability and reliability, give the cloud service 

customer the tools and opportunity to make informed choices and to gain an understanding of the 

service being delivered. NIST’s definition of cloud computing describes a “Measured Service” as 

being one of the five essential characteristics of the cloud computing model. To describe a 

“measured service”, one needs to identify the cloud service properties that have to be measured 

and what their standards of measurement or metrics are. 

A metric provides knowledge about characteristics of a cloud property through both its definition 

(e.g. expression, unit, rules) and the values resulting from the observation of the property. For 

instance, a customer response time metric can be used to estimate a specific response time 

property (i.e. response time from customer to customer) of a cloud email service search feature. It 

also provides the necessary information that is needed for to reproduce and verify observations 

and measurement results.

In this context, the role of that metrics play is very important to support decision-making as well 

as:

• Selecting cloud services

• Defining and enforcing service agreements

• Monitoring cloud services

• Accounting and Auditing

Metrics for cloud computing services can be described using the model proposed in this 

document. The model represents the information needed to understand the targeted cloud 

property and which constraints should be applied during observation. The Cloud Service Metric 

model (CSM) describes the higher level concepts of the abstract metric definitions for a specific 

cloud service property; service uptime is a prime example. Definitions for abstract metrics 

contain parameters and rules to express a formal understanding the property of interest. The CSM 

model also contains concrete metric definitions that are based on abstract metric definitions. 

Concrete metric definitions add specific values to rules and parameters that make the metric 

usable for a given scenario.

A scenario represents a particular use case in which metrics play a role. Stakeholders need to 

have a way to understand, assess, compare, combine and make decisions about cloud service 

properties. This means that for a given scenario (e.g. choosing a cloud service or setup a service 
2

agreement), a stakeholder needs to be able to get information on cloud service properties, which 

when measured (observed) will guide the stakeholder along the proper course of action. The 

scenario and cloud service property will determine the metric (standard of measurement) to be 

used.
3

2 Introduction

2.1 Audience

This document proposes a framework that identifies and characterizes the information and 

relationships needed to describe and observe properties of cloud services that are representative, 

accurate and reproducible. This information can be used in a variety of ways including, collection, 

comparison, gap analysis, and assessment or description of metrics at the technical or business 

levels. These metrics can connect information intended for decision-making, for the service 

agreements between provider and customer, for the runtime performance measurement and the 

underlying properties within the providers system.

This document may be used as a source of information to better understand metrology within the 

context of cloud services, and as a framework to describe, collect and access information related 

to metrics. The measurement process and methodology necessary for performing the 

measurement of a given cloud property is not the focus of this document.

The targeted audience of this document includes but is not limited to:

• U.S. Government agencies

• Cloud service customers

• Cloud service auditors 

• Cloud service providers

2.2 Background

Cloud computing shifted the use of compute resources from asset-based physical resources to 

service-based virtual resources. NIST in its definition of cloud computing [2] describes a 

“Measured Service” as being one of the five essential characteristics of the cloud computing 

model. Providing data on measurable capabilities (such as; quality of service, security features, 

availability and reliability) gives the cloud service customer the opportunity to make informed

choices and to gain understanding of the state of the service being delivered. It also gives the 

cloud service provider the opportunity to present the properties of their cloud services to the 

cloud service customer.

However cloud metrology is not necessarily well understood. Common terminologies (i.e. the 

definition of measurement, metric, and related concepts) or sets of measurement artifacts (i.e. unit 

of measurement, metric) often have several definitions, which makes it very difficult for the cloud 

service customer to compare services or rely on third party tools to monitor the health of the 

service. It also makes it difficult for the provider to show that the service is performing correctly 

or to allow its service to enter into a complex cloud service chain or federation.

Organizations, like U.S. agencies, need a way to consistently define sets of metrics on which 

they can rely, trust and share. This has the net-effect of increasing the overall confidence in the 

results of measurements of selected cloud service properties. This effect also increases the 

support of the decision-making process during the different stages of the cloud service lifecycle.
4

It is critical to have the capacity to represent what needs to be measured, how the measurement 

results are used, and how they impact business and technical decisions.

Cloud metrology is vast and takes into account many different components including:

• The definition of metrics and their use.

• The definition of measurement processes and methods.

• The calibration of measurement tools.

• The measurement operations.

• The processing of measurement results and associated consequences.

This document’s primary focus is on the first item and introduces an approach to define and 

represent the concepts and uses of measurement within the context of cloud services and their 

underlying components.

3 Definitions

Currently, the terminology of cloud service measurements is not well defined. Different 

stakeholders in the Information and Communication Technology (ICT) community use the same 

terms with slightly different (or sometimes contradicting or overlapping) meanings. This may be 

due to wide variety of ICT’s technology domains (i.e. Software, Telecommunication, 

Manufacturing), each using its own language. It could also come from the lack of a common 

process to define new terminology. This leads to great confusion among cloud service providers, 

customers, carriers and other cloud stakeholders.

The use of well-defined and understood terms within a given domain will enable the stakeholders

to communicate more efficiently. It reduces the risk of the misinterpretation of information and 

facilitates the combination and comparison of information. 

To bring clarity to the vocabulary of cloud service measurements, some of the core terms used in 

the document are defined below. 

3.1 Abstract Metric

An abstract standard of measurement used to assess a property. The standard of measurement 

describes what the result of the measurement means, but not how the measurement was 

performed. The Abstract Metric is not used by itself, but is instantiated using a Metric.

3.2 Abstract Metric Definition

A collection of elements that defines the expression of a specific metric for a given metric 

category like a blueprint
5

3.3 Cloud Service Property

A property of a cloud service to be observed. A property may be expressed qualitatively or 

quantitatively.

3.4 Concrete Metric Definition

A collection of elements that complete an abstract metric definition by linking the metric to its 

primary abstract metric and assigning specific values to the rule(s) and parameter(s) defined in 

the abstract metric definition

3.5 Context

The circumstances that form the setting for an event, statement, or idea, in which the meaning of 

a metric can be fully understood and assessed.

3.6 Measurement

Set of operations having the object of assigning a Measurement Result.

Note: Based on the definition of Measurement in ISO/IEC 15939:2007 [6]. Also used here to 

describe an actual instance of execution of these operations leading to the production of a 

Measurement Result instance.

3.7 Measurement Result

Value that expresses a qualitative or quantitative assessment of a property of an entity.

Note: Based on the definition of Measurement Result in ISO/IEC 15939:2007 [6]

Note: The term measure is not used in this document. Measure is defined with so many 

divergent definitions it is difficult to use. Section 9 “Definitions Survey” shows a sample of the 

definitions related to “measure”.

3.8 Metric

A standard of measurement that defines the conditions and the rules for performing the 

measurement and for understanding the results of a measurement.

Note: A metric implements a particular abstract metric concept.

Note: A metric is to be applied in practice within a given context that requires specific properties

to be measured, at a given time(s) for a specific objective.
6

3.9 Observation

Measurement based on a metric, at a point in time, on a measurement target.

3.10 Unit of Measurement

Real scalar quantity, defined and adopted by convention, with which any other quantity of the 

same kind can be compared to express the ratio of the two quantities as a number [7].

Note: part of an Abstract Metric

4 The	Role	of	Metrology	in	Cloud	Services

Metrology – the science of measurement – is important for cloud computing not just for the 

measurement of properties of cloud services, but also to gain a common understanding of the 

properties themselves. 

Physical properties can be measured using a standardized metrology process. Software properties 

measurement has some associated standards like functional size measurement methods [3][4][5] 

that are not exactly at the level of physical metrology.

Metrics are used to understand a particular measurement (or type of measurement) of a cloud 

service property and to understand the property itself by providing a standard for describing a 

measurement and measurement result. 

Figure 1 shows the relationship between a property and a metric. Cloud services have properties 

that represent characteristics of the service. The understanding of these properties is very 

important to determine the service capabilities. One way to understand these properties is with 

metrics. The use of a metric through an observation results in measurement results to estimate the

property of an element. For instance a customer response time metric can be used to estimate a 

specific response time property (i.e. response time from customer to customer) of a cloud email 

service search feature. 

A metric provides knowledge about aspects of the property through its definition (e.g. expression, 

unit, rules). It also provides the necessary information for reproducibility and verification of 

observations and measurement results.
7

Figure 1 Metric and Property

In this manner cloud metrics help providers communicate the properties of their cloud services

that are measurable, help customers and providers agree on what will be provided, and allow 

cloud service features to be measured to ensure the agreement is met (and therefore the customers 

requirements are met).

Cloud system can leverage metrics – standards for measurements – for many different purposes. 

For instance metrics can be used at different layers of a cloud computing system (e.g. hardware

layers, logic layers, governance layers or service layers). They can also be used at different stages

of the cloud computing services life cycle (e.g. procurement, operation, audit and retirement).

4.1 The Cloud Service Trifecta

The use of metrics for cloud computing systems at the service interface can be broken down into 

three general areas, service selection, service agreement, and service verification. Metrics are 

essential, not just to understand each of these areas, but to connect these three distinct parts of the 

cloud procurement process. The three aspects of the trifecta are described below.

4.1.1 Metrics for Selecting Cloud Services

Metrics are essential at the stage of deciding what cloud offering should be best suited to meet the 

business and technical requirements. The customer of cloud services should be able to select and 

use metrics and their underlying measures to assess and decide which offering would be best. 

Solutions like the Service Measurement Index (SMI), [8] produced by the Cloud Services 

Measurement Initiative Consortium (CSMIC), could be used to determine which metrics are

relevant to the selection of a particular cloud offering.

Figure 2 shows how metrics are used to understand the factors and properties necessary for

distinguishing and deciding between two different cloud offerings. Such metrics may be used to 

provide data on actual cloud operations (e.g. performance, responsiveness, scalability, 

availability...) as produced by some independent auditing or monitoring of the provider when 

servicing its current customers. The use of these metrics may also result in an assessment on the 

readiness and ability of a cloud service provider to ensure some level of service quality prior to 

and independently from actual operations (e.g. various aspects of security, accessibility, customer 

support, financial flexibility).
8

Figure 2 Cloud Service Selection

4.1.2 Metrics for Service Agreements (SAs)

A Service Agreement (SA) represents a binding agreement between the provider and customer of 

a cloud service. Among the elements that it contains are the description of the service, the rights 

and responsibilities of both the provider and the customer and terms definition. It also contains 

essential information related to the measurement of different aspects of the cloud service (e.g., its 

business level objectives or its performance level). The definition and usage of appropriate 

metrics with their underlying measures are essential components of the Service Level Agreement 

(SLA) and Service Level Objectives (SLO), which are constituents of the SA. The references [9] 

and [10] describe, in detail, the importance of and need for metrics in SLAs. At this point, the 

metrics are used to set the boundaries and margins of errors the provider of the service abides by 

and sets their limitations. For instance these metrics could be used at runtime for service 

monitoring and balancing, or remediation (e.g., financial). Using a standardized set of metrics or 

metric templates in SAs makes it easier and quicker to define SLAs and SLOs, and to compare 

them with others.

Figure 3 illustrates the use of metrics to support an SLA document that defines the expectations 

of the two parties – cloud customer and cloud provider, allowing them to understand the

characteristics of the specific service (cloud offering) being provided.

Figure 3 Cloud Service Agreement

4.1.3 Metrics for Service Measurement

Once the customer purchases a cloud service, it is necessary to ensure the service level objectives 

are being met. If they are not met, a pre-determined remedy needs to be initiated.
9

Figure 4 illustrates the service being delivered to the cloud customer from the cloud provider. In 

this case, metrics are used when monitoring the service level objectives defined in the service 

agreements.

Figure 4 Cloud Service Objectives Monitoring

4.2 Other Metrics

Metrics can also be used internally to the cloud service itself. These metrics are more technical 

and used only by the cloud service provider to monitor and to understand the internal 

performance of their cloud system. References [11] and [12] discuss to specifications that show 

potential representations and usages of measurement concepts that can be used in cloud 

computing systems. Measurement results based on metrics for internal use may not be available 

to the cloud customer.

In addition, other parts of the cloud ecosystem can be influenced through the use of metrics like 

accounting, auditing and security. In the case of accounting, metrics are for instance linked to the 

amount of usage of a particular service. In the case of auditing, metrics are linked to the 

certification assessment of selected cloud service properties. Finally, cloud metrics can be used to 

negotiate/monitor the customer’s requirements based on the agreed SLA.

4.3 Scenario

Stakeholders need to have a way to understand, assess, compare, combine and make decisions 

about cloud service properties. This means that for a given instance, i.e. scenario (e.g. choosing a 

cloud service or setting up a service agreement), a stakeholder needs to be able to get information

on cloud service properties, which when measured (observed) will aid the stakeholder in selecting

the proper course of action. The scenario and cloud service property will determine the metric

(standard of measurement) to be used. The metric relies on the abstract metric definitions that are 

related to the selected cloud service property. The measurement (observation) of the cloud service 

property through the metric will result in measurement results.

Figure 5 shows the scenario concept:

• The Scenario represents a particular use case (business process decision making, 

application monitoring, Service Level Agreements, etc.).

• The Abstract Metric describes the base concept on which the Metric is based.
10

• The Metric adds the data necessary to use the abstract metric.

• The Measurement Result is data that results from making a measurement that follows a 

given metric.

More specifically, stakeholders (e.g. cloud customer or cloud provider) define the scenario for 

which the metric will be needed. The scenario represents:

• The expectations of an underlying business or operational process (e.g. SLA or 

Operation)

• How the metrics are used to assist in such a process

• What acceptable levels of the measured properties are 

The scenario also includes the way the selected metrics are applied – what resource or service 

they support, under which conditions are their evaluation triggered and the frequency of the 

evaluation.

Figure 5 Scenario and Metric

In other words, a metric is a standard set of procedures and rules that generates values for its 

associated abstract metric. In practice, the metric is applied within a given scenario that 

determines specific conditions, such as a specific resource(s) being measured, at a given time(s) 

for a specific objective.

Possible scenarios could be the application of an availability metric for a performance objective 

of 99% in an SLA scenario or the application of an accessibility metric for a usability objective of 

value “high” in a decision process scenario.
11

5 Cloud	Service Metric	Model

Understanding of the relationships between different data elements of cloud service metrology is 

very important in order to create meaningful and traceable metrics. This section introduces the 

Cloud Service Metric model (CSM), its general concept and a full element description of the

foundation diagram that describes a Metric definition.

5.1 Cloud Service Metric Ecosystem

As explained in the earlier Section 3, a metric is a fundamental concept that should provide 

information on how to understand a property being considered and how to estimate its value 

through observations.

The information that comprises the metric ecosystem can be broken down into these specific 

aspects:

• The description and definition of a standard of measurement (e.g. metric for customer 

response time) – CSM

• The addition of the context of the standard of measurement (e.g. objectives and 

applicability conditions of the customer response time metric) – CSM Context

• The use of the standard of measurement to define observations (e.g. the observation of 

response time property based on the customer response time metric) – CSM Observation

• The use of the standard of measurement in a scenario (e.g. the selection and use of the 

customer response time metric in an SLA scenario) – CSM Scenario

Figure 6 Cloud Service Metric Ecosystem Model

Figure 6 shows the breakdown of these different aspects and their relationships. The CSM box

contains the description and definition of the standard of measurement. The CSM can be 

enhanced with information from the CSM Context; it contains elements that describe the 

environment of a particular standard of measurement. The CSM Scenario relies on the CSM and 

the CSM Context and contains elements that describe use cases that rely on standards of 

measurement. The CSM Observation relies on the CSM and the CSM Context and contains 

elements that are used during measurement operations.

In this document, only the CSM concept model is explained. The remaining concepts that 

compose the CSM Ecosystem model are the topics of other documents.
12

5.2 Model Characteristics

The following are characteristics that were considered important when developing the CSM

model.

5.2.1 Consistent Representation of Information

Information related to metrics should be represented in a consistent, repeatable way in order to 

efficiently organize it, share it and use it.

5.2.2 Explicit Relationships

Concepts like metrics should be represented in such a way that the relationships among them, if 

any, are explicit. This clarifies the effects these concepts have on one another and their 

importance.

5.2.3 Repository of Definitions

There should be a way to organize metrics so they are reusable, searchable and derivable.

5.2.4 Comparability

The properties of the different concepts should allow its user to have enough information to 

efficiently compare them to find and understands either similarities or differences.

5.2.5 Flexibility and Adaptability

The concept model should be sufficiently flexible and adaptable to allow for easy integration with 

other metric models. These models could be complementary to the concept model (e.g. represent 

measurement methods and process).

5.2.6 Composability

The metrics should allow metrics definitions and instances to be reusable. Thus one should be 

able to use one or more metrics to build a composite metric. This metric that is composed of 

underlying metrics builds upon the information they contain. This results in metrics that could 

possibly be composed of underlying metrics of different kind (e.g. qualitative and quantitative). 

This consideration will be discussed in Section 8.

5.3 Cloud Service Metric Diagram

This subsection formally describes the CSM diagram, its elements, what these elements are 

composed of and the way they are connected to each other.
13

Figure 7 introduces the CSM concept as a UML class diagram [13]. The purpose of the CSM 

model is to capture the information needed to describe and understand a metric. The metric is 

used for gaining knowledge about, and measuring cloud service properties.

Figure 7 Cloud Service Metric (CSM)

5.4 Cloud Service Metric Element Definitions

The different elements of the CSM model are described below. In this section the use of the terms 

class, attribute and association and the model itself conform to the UML 2.0 specification. The 

classes are described in the following order:

• AbstracMetric

• RuleDefinition

• ParameterDefinition

• Metric

• MetricRule

• MetricParameter

Within each class, the attributes are described, then the associations between the class and other 

classes. The attributes are described following the order of the diagram.

MagicDraw, 1-1 /Users/fdevaulx/Documents/organized/cloud/cloud-metrics/csmm/cloud-metrics-3-2-10.xml cloud-metric-core Aug 13, 2014 8:4

-name

-referenceId

-unit

-scale

-expression

-definition

-note

AbstractMetric

-name

-referenceId

-note

Metric

-value

-note

MetricParameter

-value

-note

MetricRule

-name

-referenceId

-parameterType

-definition

-note

ParameterDefinition

-name

-referenceId

-definition

-note

RuleDefinition

-primaryAbstractMetric 1

0..* 0..* -metricParameters

0..* -parameterDefinitions -ruleDefinitions 0..*

-metricRules 0..*

-ruleDefinition 1

0..*

1 -parameterDefinition

0..*

-underlyingAbstractMetrics

0..*

-underlyingMetrics

0..*
14

5.4.1 AbstractMetric Class

The AbstractMetric class holds the basic information necessary to understand the measurement of 

a property to be observed, but does not include the additional information (e.g. context, target) to 

actually use the metric.

5.4.1.1 AbstractMetric Attributes

name 

The name of the AbstractMetric. 

(e.g. TimeDuration)

referenceId 

A unique identifier for the abstract metric defined by convention. 

(e.g. AM001)

unit 

The unit that will be associated with the AbstractMetric.

(e.g. second)

Note that not every AbstactMetric is associated with a scalar unit. For instance, AbstractMetrics 

whose scale is nominal or ordinal (i.e. qualitative) could be associated with a list of elements (e.g. 

low, medium, high for data sensitivity) or a more complex construct.

scale 

Information on how the measurement value can be interpreted and what sort of operations can be 

performed on it. It is based on the theory of scales of measurement [14].

The scale also reflects the kind of the AbstractMetric, i.e. qualitative or quantitative.

Quantitative - A metric that has values of numeric type, with the semantics of a quantity. 

The expression (or formula) that determines how such a value is calculated is of numeric 

output with a quantitative meaning (e.g. speed = distance / duration is of quantitative 

kind). The interval and ratio scales are viewed as quantitative.

Qualitative - A metric that has either nominal or ordinal values. When ordinal, the 

metric usually expresses a “score” (e.g. on a scale from 1 to 10). When nominal, it 

expresses a quality (e.g. “good”, “average”, “bad”). There is usually an expression (or 

formula) associated with each possible value, which is of qualitative nature. The nominal

and ordinal scales are viewed as qualitative

Allowed values:

Nominal – Qualitative

Ordinal – Qualitative

Interval – Quantitative

Ratio – Quantitative

expression 

The function used to assemble the underlying AbstractMetrics and the ParameterDefinitions that 

compose the AbstractMetric. RuleDefinitions can also be part of the expression. In its simplest

form, the expression is a literal but it can also be a more formal expression language.
15

(e.g. expression = Sum(ResponseTime)/n where “ResponseTime” is an underlying 

AbstractMetric element and “n” is a ParameterDefinition element)

definition

A formal description of the AbstractMetric.

note

Additional information or comments related to the AbstractMetric.

5.4.1.2 AbstractMetric Associations

ruleDefinitions 

An AbstractMetric may have zero or more RuleDefinions associated with it. RuleDefinitions may 

be part of the expression of an AbstractMetric to constrain it.

parameterDefinitions

An AbstractMetric may have zero or more ParameterDefinitions associated with it.

ParameterDefinitions may be part of the expression of an AbstractMetric.

underlyingAbstractMetrics

Shows any AbstractMetrics that are used as a base for the AbstractMetric being defined.

underlyingAbstractMetrics are part of the expression of an AbstractMetric.

5.4.2 RuleDefinition Class

A RuleDefinition element is used to further constrain some parts of an AbstractMetric element 

and indicate possible method(s) for measurement. For instance an 

“AvailabilityDuringBusinessHour” Metric element could be defined with a scope that constrains 

some piece of a generic “Availability” AbstractMetric element that limits the observation period 

to defined business hours.

5.4.2.1 RuleDefinition Attributes

name 

The name of the RuleDefinition.

(e.g. whenStart RuleDefinition for a TimeDuration AbstractMetric)

referenceId 

A unique identifier for the abstract metric defined by convention. 

(e.g. RD001)

definition

A formal description of the RuleDefinition element.

note

Additional information or comments related to the RuleDefinition.
16

5.4.3 ParameterDefinition Class

A ParameterDefinition element is used to define a parameter needed in the expression of an 

AbstractMetric. A ParameterDefinition may be used by more than one AbstractMetrics.

5.4.3.1 ParameterDefinition Attributes

name 

The name of the ParameterDefinition.

(e.g. measurementTimeframe)

referenceId 

A unique identifier for the ParameterDefinition defined by convention. 

(e.g. PD001)

parameterType

The type of the ParameterDefinition, the way it should be interpreted. 

(e.g. integer, string)

definition

A formal description of the ParameterDefinition.

note

Additional information or comments related to the ParameterDefinition.

5.4.4 Metric Class

Defines the concrete standard of measurement for a specific cloud service property, It is based on 

the AbstractMetric concept, adding the specific parameters, and rules which are required to use 

the AbstractMetric.

5.4.4.1 Metric Attributes

name 

The name of the Metric.

(e.g. CustomerResponseTime)

referenceId 

A unique identifier for the metric defined by convention.

note

Additional information or comments related to the Metric.

5.4.4.2 Metric Associations

primaryAbstractMetric
17

The primary AbstractMetric that the Metric implements.

metricRules

A Metric is associated with zero or more MetricRules. These MetricRules are one piece of the 

implementation of an AbstractMetric through its RuleDefinitions association.

metricParameters

A Metric is associated with zero or more MetricParamters. These MetricParameters are one piece 

of the implementation of an AbstractMetric through its ParameterDefinitions association.

underlyingMetrics

Shows any Metrics that are used as a base for the Metric being described.

5.4.5 MetricRule Class

The element that represents a concrete rule of the Metric based on information from the Metric’s 

primary AbstractMetric element.

5.4.5.1 MetricRule Attributes

value

The value of the rule defined by the associated RuleDefinition.

(e.g. value could be “scheduled maintenance” for the associated ruleDefinition 

observation_exclusion)

note

Additional information or comments related to the MetricRule.

5.4.5.2 MetricRule Associations

ruleDefinition

A MetricRule is dependent on a single RuleDefinition. This RuleDefinition is selected from the 

ruleDefinitions of the Metric’s primaryAbstractMetric element.

5.4.6 MetricParameter Class

The element that represents a concrete parameter of the Metric based on information from the 

Metric’s primary AbstractMetric element.

5.4.6.1 MetricParameter Attributes

value

The value of the parameter defined by the associated ParameterDefinition.

(e.g. value could be 30 for the associated parameterDefinition measurement_timeframe)

note

Additional information or comments related to the MetricParameter.
18

5.4.6.2 MetricParameter Associations

parameterDefinition

A MetricParameter is dependent on a single ParameterDefinition. This ParameterDefinition is 

selected from the parameterDefinitions of the Metric’s primaryAbstractMetric element.
19

6 How	to	Use	the	CSM	Model

The CSM model defines the fundamental elements needed to describe standards of measurement 

(i.e. metrics). These elements are organized into two parts. The first part represents the elements 

that make up the abstract definition of a particular metric (e.g. Service Availability). This is the 

abstracted model for a category of metrics. The second part represents the elements that make up 

a specific instantiation of this abstract metric.

Figure 8 CSM Definitions Blocks

Figure 8 displays another perspective of the CSM concept. This perspective doesn’t go into as 

much details for each element of the CSM as in Figure 7. Instead it displays the CSM at a higher 

viewpoint to show how the CSM can be organized into 2 sets of constituents, Abstract and 

Concrete Metric Definitions blocks. 

In block 1, for a given metric category, the abstract metric definition of a metric is composed of 

AbstractMetric(s) and associated ParameterDefinition(s) and RuleDefinition(s). This abstract 

metric definition represents the collection of elements that defines the expression of a specific 

metric for a given metric category like a blueprint. For instance in the availability metric category

which expresses the availability property of a cloud service, one can define several metrics like 

service uptime percentage or number of successful http requests. Each of these metrics can be 

expressed with a set of attributes like unit, scale, expression; a set of parameters and a set of rules 

and that constrain the definition. The rule definition and parameter definition elements describe 

what the rules and parameters are about, not what values they have. This allows one to add a rule 

for availability that expresses downtime exclusion without setting a specific value for that 

exclusion. This is the role of the concrete metric definition.

In block 2, the concrete metric definition of a metric is composed of a Metric and associated 

MetricParamter(s) and MetricRule(s). A concrete metric definition completes an abstract metric 
20

definition by linking the metric to its primary abstract metric and assigning specific values to the 

rule(s) and parameter(s) defined in the abstract metric definition block. This way multiple 

concrete metric definitions can leverage the same abstract metric definition.

Figure 9 displays the process followed to define metrics from a different viewpoint. The CSM 

model defines the core concepts and elements that constitute a standard of measurement. Specific 

instances of a subset of these elements (Figure 8 block 1) are then used to create an abstract 

metric definition. Then for a given abstract metric definition, implementation metrics are created 

using instances of another subset of the CSM elements (Figure 8 block 2).

Figure 9 Metric Definition Process

7 Measurement	Uncertainty

In metrology, the result of a measurement is not meaningful if a statement of the uncertainty of 

the measurement is not specified. This statement allows users to assess the quality of the 

measurement results and to build confidence to compare results and use them within the range of 

the measurement uncertainty.

The International Vocabulary of Metrology (VIM) [7] defines measurement uncertainty as 

A non-negative parameter characterizing the dispersion of the quantity values being 

attributed to a measurand, based on the information used

NOTE 1 Measurement uncertainty includes components arising from systematic effects, 

such as components associated with corrections and the assigned quantity values of 

measurement standards, as well as the definitional uncertainty. Sometimes estimated 
21

systematic effects are not corrected for but, instead, associated measurement uncertainty 

components are incorporated.

NOTE 2 The parameter may be, for example, a standard deviation called standard 

measurement uncertainty (or a specified multiple of it), or the half-width of an interval, 

having a stated coverage probability.

NOTE 3 Measurement uncertainty comprises, in general, many components. Some of 

these may be evaluated by Type A evaluation of measurement uncertainty from the 

statistical distribution of the quantity values from series of measurements and can be 

characterized by standard deviations. The other components, which may be evaluated by 

Type B evaluation of measurement uncertainty, can also be characterized by standard 

deviations, evaluated from probability density functions based on experience or other 

information.

NOTE 4 In general, for a given set of information, it is understood that the measurement 

uncertainty is associated with a stated quantity value attributed to the measurand. A 

modification of this value results in a modification of the associated uncertainty.

In the context of cloud services, it is critical that the consumer of a measured resource be 

confident about the measurements operated on that resource. These measurements will feed 

metrics that could be used against thresholds to determine the range the acceptable results and 

trigger possible consequences.

The current CSM model starts addressing this aspect of metrology with an attribute “uncertainty”

that is contained in the CSM Observation model.

8 Other	Considerations

8.1 Metrics used for Property Composition

A key aspect of the CSM model is its extensibility, which permits metric definitions to be 

composed from other metric definitions. This is an effort to limit the duplication of information 

without too much of an increase on complexity. CSM allows qualitative or quantitative metrics to 

be defined and composed. This can affect the estimation – measurement results – of a particular 

property in several ways like its uncertainties.

8.2 Calibration & Measurement Standard (Etalon)

Once new metrics and unit of measurements have been defined for cloud service properties that 

can be reusable and comparable, the next step could be the calibration of the measurement 

systems used for measurement of cloud service properties against established measurement 

standards. This would enable a better alignment of the understanding and comparison of the 

properties that compose different cloud service offerings.
22

9 Conclusion

Metrics are a critical aspect of the selection, operation and use of cloud services. Metrics allow 

stakeholders to gain a better understanding of cloud service properties through consistent, 

reproducible and repeatable observations. Metrics can be used for a wide range of objectives from 

decision making to operation. For instance key performance indicator metrics can be used to 

measure specific achievements whereas benchmark metrics can be used as reference to compare 

features against one another.

Metrics need to be well defined and understood so that the different stakeholders involved in 

cloud computing (i.e. cloud service customer and cloud service provider) can rely on them with 

confidence. The Cloud Service Metric (CSM) model proposed in this document is one approach 

to addressing this challenge. The CSM defines a small set of concepts and links them together to 

define what a metric is, what it is composed of and what constrains its expression. The model can 

be logically broken down into two parts. The first part addresses the definition of abstract metrics. 

It specifies what the abstract metric is about, if it is composed of underlying abstract metrics, if it 

is expressed with additional parameters and if there are core rules that constrain it. The second 

part addresses the definition of concrete metrics. It specifies what primary abstract metric a

metric is based on, what values for parameters and rules should be applied to the abstract metric 

parameters and rules definition.

The CSM model can be extended and integrated into other models that address other aspects of 

the metric ecosystem like the context of a metric, the observation and measurement results based 

on a metric or the scenarios that make use of metrics. These other aspects will be explored in 

future work.
23

Annex	A - Definitions	Survey

Table 1 presents a sample collection of measurement terms and definitions coming from different 

domains including, information technology, software, software engineering and physical. The 

terms that were sampled are measure, metric, key performance indicator, benchmark, 

measurement and measurement unit. As result, the table shows that across and among domains 

there are many different definitions for the same term. Most of these definitions tend to have the 

same concepts in their descriptions however a few mix the terms and definitions. For instance the 

OMG SIMM document defines measure as “a method assigning comparable numerical or 

symbolic values to entities in order to characterize an attribute of the entities” and measurement 

as “a numerical or symbolic value assigned to an entity by a measure” and other documents used 

the same definitions but inverted the terms so in the case of the ISO/IEC 15939 document 

measurement is defined as “Set of operations having the object of determining a value of a 

measure” and measure as “variable to which a value is assigned as the result of measurement”.

PHDVXUH

EDVHPHDVXUH ,62,(&

6\VWHPVDQGVRIWZDUH

HQJLQHHULQJ

0HDVXUHPHQWSURFHVV ,62,(&

VRIWZDUH

HQJLQHHULQJ

PHDVXUHGHILQHGLQWHUPVRIDQDWWULEXWHDQGWKH

PHWKRGIRUTXDQWLI\LQJLW

PHDVXUH ,62,(&

6\VWHPVDQGVRIWZDUH

HQJLQHHULQJ

0HDVXUHPHQWSURFHVV ,62,(&

VRIWZDUH

HQJLQHHULQJ

YDULDEOHWRZKLFKDYDOXHLVDVVLJQHGDVWKHUHVXOWRI

PHDVXUHPHQW

PHDVXUH ,62,(&

6\VWHPVDQGVRIWZDUH

HQJLQHHULQJ

0HDVXUHPHQWSURFHVV ,62,(&

VRIWZDUH

HQJLQHHULQJ PDNHDPHDVXUHPHQW

PHDVXUH 1,6763 6RIWZDUH(UURU$QDO\VLV 1,67

VRIWZDUH

HQJLQHHULQJ

7KHQXPHULFDOYDOXHREWDLQHGE\HLWKHUGLUHFWRU

LQGLUHFWPHDVXUHPHQWPD\DOVREHWKHLQSXWRXWSXWRU

YDOXHRIDPHWULF

PHDVXUH ,(((

,(((6WDQGDUGIRUD

6RIWZDUH4XDOLW\0HWULFV

0HWKRGRORJ\ ,(((

VRIWZDUH

HQJLQHHULQJ

$DZD\WRDVFHUWDLQRUDSSUDLVHYDOXHE\FRPSDULQJ

LWWRQRUP%WRDSSO\DPHWULF

PHDVXUH 6$0$7( 6$0$7(3URMHFW 1,67

VRIWZDUH

DVVXUDQFH

ZHXVHPHDVXUHIRUPRUHFRQFUHWHRUREMHFWLYH

DWWULEXWHV

PHDVXUH 600

6WUXFWXUHG0HWULFV

0HWDPRGHO600 20*

VRIWZDUH

HQJLQHHULQJ

DPHWKRGDVVLJQLQJFRPSDUDEOHQXPHULFDORUV\PEROLF

YDOXHVWRHQWLWLHVLQRUGHUWRFKDUDFWHUL]HDQDWWULEXWHRI

WKHHQWLWLHV

TXDQWLW\ -&*0

,QWHUQDWLRQDOYRFDEXODU\

RIPHWURORJ\%DVLFDQG

JHQHUDOFRQFHSWVDQG

DVVRFLDWHGWHUPV %,30 PHWURORJ\

SURSHUW\RIDSKHQRPHQRQERG\RUVXEVWDQFHZKHUH

WKHSURSHUW\KDVDPDJQLWXGHWKDWFDQEHH[SUHVVHGDV

DQXPEHUDQGDUHIHUHQFH

PHWULF

EDVHPHWULF '63B %DVH0HWULF3URILOH '07)

DPHWULFSURYLGHGGLUHFWO\ZLWKRXWDGHSHQGHQF\RQ

RWKHUPHWULFYDOXHV

GLUHFWPHWULF ,(((

,(((6WDQGDUGIRUD

6RIWZDUH4XDOLW\0HWULFV

0HWKRGRORJ\ ,(((

VRIWZDUH

HQJLQHHULQJ

DPHWULFWKDWGRHVQRWGHSHQGXSRQDPHDVXUHRIDQ\

RWKHUDWWULEXWH

LQWHUYDOPHWULF '63B %DVH0HWULF3URILOH '07)

PHWULFWKDWDSSO\WRDWLPHLQWHUYDO$QH[DPSOHRIDQ

LQWHUYDOPHWULFLVWKHDYHUDJH&38XWLOL]DWLRQRID

VHUYHURYHUWKHSDVWKRXU

PHWULF *)'53

8VDJH5HFRUG)RUPDW

5HFRPPHQGDWLRQ 2*) ZHEVHUYLFH

7KLVDWWULEXWHLGHQWLILHVWKHPHWKRGRIPHDVXUHPHQW

XVHGIRUTXDQWLI\LQJWKHDVVRFLDWHGUHVRXUFH

FRQVXPSWLRQLIWKHUHDUHPXOWLSOHPHWKRGVE\ZKLFKWR

PHDVXUHUHVRXUFHXVDJH

PHWULF

,(((6WG



,(((6WDQGDUG*ORVVDU\

RI6RIWZDUH(QJLQHHULQJ

7HUPLQRORJ\ ,(((

VRIWZDUH

HQJLQHHULQJ

DTXDQWLWDWLYHPHDVXUHRIWKHGHJUHHWRZKLFKD

V\VWHPFRPSRQHQWRUSURFHVVSRVVHVVHVDJLYHQ

YDULDEOH

PHWULF 1,6763 6RIWZDUH(UURU$QDO\VLV 1,67

VRIWZDUH

HQJLQHHULQJ

7KHGHILQLWLRQDOJRULWKPRUPDWKHPDWLFDOIXQFWLRQXVHG

WRPDNHDTXDQWLWDWLYHDVVHVVPHQWRISURGXFWRU

SURFHVV

PHWULF

ZZZSURVFLFRP

PHWULFVKWP   EXVLQHVV

$PHWULFLVQRWKLQJPRUHWKDQDVWDQGDUGPHDVXUHWR

DVVHVV\RXUSHUIRUPDQFHLQDSDUWLFXODUDUHD

7HUP 6RXUFH 7LWOH 2UJDQL]DWLRQ &DWHJRU\ 'HVFULSWLRQ

Table 1 - A sample of measurement related terms and definition in the IT space
24

EHQFKPDUNLQJ :LNLSHGLD :LNLSHGLD EXVLQHVV

WKHSURFHVVRIFRPSDULQJRQH
VEXVLQHVVSURFHVVHV

DQGSHUIRUPDQFHPHWULFVWRLQGXVWU\EHVWVRUEHVW

SUDFWLFHVIURPRWKHULQGXVWULHV

PHDVXUHPHQW

HVWLPDWRU *%UHY 6/$+DQGERRN 70)RUXP

PHWKRGWRREWDLQRUFRPSXWHDPHDVXUHGYDOXHRID

PHWULFDOVRWKHYDOXHLWVHOI

PHDVXUHPHQW

-RXUQDORIUHVHDUFK

1%69RO1R

0D\-XQH )RXQGDWLRQRI0HWURORJ\ 1%6 PHWURORJ\

$PHDVXUHPHQWLVDVHULHVRIPDQLSXODWLRQVRISK\VLFDO

REMHFWVRUV\VWHPVDFFRUGLQJWRDGHILQHGSURWRFRO

ZKLFKUHVXOWVLQDQXPEHU7KHQXPEHULVSURSRUWHGWR

XQLTXHO\UHSUHVHQWWKHPDJQLWXGHRULQWHQVLW\RIVRPH

TXDQWLW\HPERGLHGLQWKHWHVWREMHFW7KLVQXPEHULV

DFTXLUHGWRIRUPWKHEDVLVRIDGHFLVLRQHIIHFWLQJVRPH

KXPDQJRDORUVDWLVI\LQJVRPHKXPDQQHHGWR

VDWLVIDFWLRQRIZKLFKGHSHQGVRQWKHSURSHUWLHVRIWHVW

REMHFW

PHDVXUHPHQW -&*0

,QWHUQDWLRQDOYRFDEXODU\

RIPHWURORJ\%DVLFDQG

JHQHUDOFRQFHSWVDQG

DVVRFLDWHGWHUPV %,30 PHWURORJ\

SURFHVVRIH[SHULPHQWDO\REWDLQLQJRQURUPRUH

TXDQWLW\YDOXHVWKDWFDQUHDVRQDEO\EHDWWULEXWHGWRD

TXDQWLW\

PHDVXUHPHQW ,62,(&

6\VWHPVDQG6RIWZDUH

(QJLQHHULQJ

0HDVXUHPHQW3URFHVV ,62,(&

VRIWZDUH

HQJLQHHULQJ

6HWRIRSHUDWLRQVKDYLQJWKHREMHFWRIGHWHUPLQLQJD

YDOXHRIDPHDVXUH

PHDVXUHPHQW 600

6WUXFWXUHG0HWULFV

0HWDPRGHO600 20*

VRIWZDUH

HQJLQHHULQJ

DQXPHULFDORUV\PEROLFYDOXHDVVLJQHGWRDQHQWLW\E\

DPHDVXUH

PHDVXUHPHQWXQLW

XQLWRIPHDVXUH 600

6WUXFWXUHG0HWULFV

0HWDPRGHO600 20*

VRIWZDUH

HQJLQHHULQJ

DTXDQWLW\LQWHUPVRIZKLFKWKHPDJQLWXGHVRIRWKHU

TXDQWLWLHVZLWKLQWKHVDPHWRWDORUGHUFDQEHVWDWHG

PHDVXUHPHQWXQLW -&*0

,QWHUQDWLRQDOYRFDEXODU\

RIPHWURORJ\%DVLFDQG

JHQHUDOFRQFHSWVDQG

DVVRFLDWHGWHUPV %,30 PHWURORJ\

UHDOVFDODUTXDQWLW\GHILQHGDQGDGRSWHGE\

FRQYHQWLRQZLWKZKLFKDQ\RWKHUTXDQWLW\RIWKHVDPH

NLQGFDQEHFRPSDUHGWRH[SUHVVWKHUDWLRRIWKHWZR

TXDQWLWLHVDVDQXPEHU

7HUP 6RXUFH 7LWOH 2UJDQL]DWLRQ &DWHJRU\ 'HVFULSWLRQ

PHWULF ,(((

,(((6WDQGDUGIRUD

6RIWZDUH4XDOLW\0HWULFV

0HWKRGRORJ\ ,(((

VRIWZDUH

HQJLQHHULQJ VHHVRIWZDUHTXDOLW\PHWULF

PHWULF 6$0$7( 6$0$7(3URMHFW 1,67

VRIWZDUH

DVVXUDQFH

ZHXVHPHWULFIRUPRUHDEVWUDFWKLJKHUOHYHORU

VRPHZKDWVXEMHFWLYHDWWULEXWHV

PHWULF *%UHY 6/$+DQGERRN 70)RUXP DFRPPRQO\LGHQWLILHGDQGPHDVXUDEOHFRQFHSW

PHWULF ,7,/JORVVDU\

JORVVDU\DQG

DEEUHYLDWLRQV ,7,/

,7VHUYLFH

PDQDJHPHQW

6RPHWKLQJWKDWLVPHDVXUHGDQGUHSRUWHGWR

KHOSPDQDJHDSURFHVV,7VHUYLFHRUDFWLYLW\

PHWULF

0HUULDP:HEVWHU

'LFWLRQDU\

0HUULDP:HEVWHU

'LFWLRQDU\

VRIWZDUHPHWULF :LNLSHGLD 6RIWZDUH0HWULF :LNLSHGLD VRIWZDUH

DVRIWZDUHLVDPHDVXUHRIVRPHSURSHUW\RIDSLHFHRI

VRIWZDUHRULWVVSHFLILFDWLRQV

VRIWZDUHTXDOLW\

PHWULF ,(((

,(((6WDQGDUGIRUD

6RIWZDUH4XDOLW\0HWULFV

0HWKRGRORJ\ ,(((

VRIWZDUH

HQJLQHHULQJ

DIXQFWLRQZKRVHLQSXWVDUHVRIWZDUHGDWDDQGZKRVH

RXWSXWLVDVLQJOHQXPHULFDOYDOXHWKDWFDQEH

LQWHUSUHWHGDVWKHGHJUHHWRZKLFKVRIWZDUHSRVVHVVHV

DJLYHQDWWULEXWHWKDWDIIHFWVLWVTXDOLW\

VXPPDWLRQPHWULF '63B %DVH0HWULF3URILOH '07)

DW\SHRIFRXQWHUPHWULFWKDWUHIOHFWVWKHDFFXPXODWLRQ

RIDYDOXH

ZDWHUPDUNPHWULF '63B %DVH0HWULF3URILOH '07)

DW\SHRIDJJUHJDWLRQPHWULFXVHGWRFDSWXUHWKH

PLQLPXPRUPD[LPXPYDOXHUHFRUGHUIRUDPRQLWRUHG

YDOXH

NH\SHUIRUPDQFHLQGLFDWRU

NH\SHUIRUPDQFH

LQGLFDWRU ,7,/JORVVDU\

JORVVDU\DQG

DEEUHYLDWLRQV ,7,/

,7VHUYLFH

PDQDJHPHQW

$PHWULFWKDWLVXVHGWRKHOS

PDQDJHDQ,7VHUYLFHSURFHVVSODQSURMHFW

RURWKHUDFWLYLW\.H\SHUIRUPDQFHLQGLFDWRUV

DUHXVHGWRPHDVXUHWKHDFKLHYHPHQWRI

FULWLFDOVXFFHVVIDFWRUV0DQ\PHWULFVPD\EH

PHDVXUHGEXWRQO\WKHPRVWLPSRUWDQWRI

WKHVHDUHGHILQHGDVNH\SHUIRUPDQFH

LQGLFDWRUVDQGXVHGWRDFWLYHO\PDQDJHDQG

UHSRUWRQWKHSURFHVV,7VHUYLFHRUDFWLYLW\

7KH\VKRXOGEHVHOHFWHGWRHQVXUHWKDW

HIILFLHQF\HIIHFWLYHQHVVDQGFRVW

HIIHFWLYHQHVVDUHDOOPDQDJHG

NH\SHUIRUPDQFH

LQGLFDWRU *%UHY 6/$+DQGERRN 70)RUXP

LQDWHOHFRPFRQFHSWPHWULFFORVHWRWHOHFRP

WHFKQRORJLHVDQGGHYLFHV

EHQFKPDUN

EHQFKPDUN 2[IRUG'LFWLRQDULHV 2[IRUG'LFWLRQDULHV

DVWDQGDUGRUSRLQWRIUHIHUHQFHDJDLQVWZKLFKWKLQJV

PD\EHFRPSDUHGRUDVVHVVHG

EHQFKPDUN :LNLSHGLD :LNLSHGLD FRPSXWHU

WKHDFWRIUXQQLQJDFRPSXWHUSURJUDPDVHWRI

SURJUDPVRURWKHURSHUDWLRQVLQRUGHUWRDVVHVVWKH

UHODWLYHSHUIRUPDQFHRIDQREMHFWQRUPDOO\E\UXQQLQJ

DQXPEHURIVWDQGDUGWHVWVDQGWULDOVDJDLQVWLW

EHQFKPDUNLQJ 2[IRUG'LFWLRQDULHV 2[IRUG'LFWLRQDULHV

HYDOXDWHRUFKHFNVRPHWKLQJE\FRPSDULVRQZLWKD

VWDQGDUG

7HUP 6RXUFH 7LWOH 2UJDQL]DWLRQ &DWHJRU\ 'HVFULSWLRQ
25

Annex	B	- References

1. Gray, M., “Applicability of Metrology to Information Technology,” J. Res. Natl. Stand. 

Technol., Vol. 104,No. 6, pp. 567-578, 1999.

2. Mell, P., Grance, T., “The NIST Definition of Cloud Computing,” NIST Special Publication 

800-145, 2011.

3. ISO/IEC 20926:2009, Software and systems engineering - Software measurement - IFPUG 

functional size measurement method.

4. ISO/IEC 19761:2011, Software engineering - COSMIC: A functional size measurement 

method.

5. ISO/IEC 29881:2010, Information technology - Software and systems engineering - FiSMA 

1.1 functional size measurement method.

6. ISO/IEC 15939:2007, Systems and software engineering - Measurement Process.

7. JCGM 200:2012, International vocabulary of metrology - Basic and general concepts and 

associated terms (VIM) 3rd edition, 2012.

8. Siegel, J., Perdue, J., “Cloud Services Measures for Global Use: The Service Measurement 

Index (SMI),” Service Research & Innovation Institute (SRII) Global Conference, Published 

by IEEE, San Jose, CA, July 24-27, 2012.

9. NIST Cloud Computing Reference Architecture Contract and SLA (Draft), 

http://collaborate.nist.gov/twiki-cloudcomputing/pub/CloudComputing/RATax_Jan20_2012/NIST_CC_WG_ContractSLA_Deliverable_Draf

t_v1_7.pdf.

10. CSCC, Practical Guide to Service Level Agreements Version 1.0, 

http://www.cloudstandardscustomercouncil.org/2012_Practical_Guide_to_Cloud_SLAs.pdf.

11. DMTF DSP0263, Cloud infrastructure Management Interface (CIMI) Model and RESTful 

HTTP-based Protocol, 2012.

12. Mach, R., et al., “Usage Record - Format Recommendation,” GFD. 98, 2006.

13. Unified Modeling Language version 2.0, http://www.omg.org/spec/UML/2.0/. Last accessed, 

November 2, 2012.

14. Stevens, S., “On the Theory of Scales of Measurement,” Science, Vol. 103, No. 2684, pp. 

677-680, 1946.