NISTIR 8099 

Methods and Tools for 

Performance Assurance of Smart 

Manufacturing Systems 

Deogratias Kibira 

KC Morris 

Senthilkumaran Kumaraguru 

This publication is available free of charge from: 

http://dx.doi.org/10.6028/NIST.IR.8099
NISTIR 8099 

Methods and Tools for 

Performance Assurance of Smart 

Manufacturing Systems 

Deogratias Kibira 

KC Morris 

Senthilkumaran Kumaraguru 

Systems Integration Division 

Engineering Laboratory 

This publication is available free of charge from: 

http://dx.doi.org/10.6028/NIST.IR.8099

December 2015 

U.S. Department of Commerce 

Penny Pritzker, Secretary 

National Institute of Standards and Technology 

Willie May, Under Secretary of Commerce for Standards and Technology and Director 
i 

METHODS AND TOOLS FOR PERFORMANCE ASSURANCE OF SMART 

MANUFACTURING SYSTEMS 

Deogratias Kibira, K C Morris, and Senthilkumaran Kumaraguru 

ABSTRACT 

The emerging concept of smart manufacturing systems is defined in part by the introduction of

new technologies that are promoting rapid and widespread information flow within the 

manufacturing system and surrounding its control. These systems can deliver unprecedented 

awareness, agility, productivity, and resilience within the production process by exploiting the 

ever-increasing availability of real-time manufacturing data. Optimized collection and analysis 

of this voluminous data and subsequent distribution of extracted information to guide decisionmaking throughout the enterprise necessitate, however, a complex and dynamic process. To 

establish and maintain confidence that smart manufacturing systems function as intended, 

performance assurance measures will be vital. 

The activities for performance assurance span manufacturing system design, operation, 

performance assessment, evaluation, analysis, decision making, and control. Changes may be 

needed for traditional approaches in these activities to address smart manufacturing systems. 

This paper reviews the current methods and tools used for establishing and maintaining required 

system performance. This paper then identifies trends in data and information systems, 

integration, performance measurement, analysis, and performance improvement that will be vital 

for assured performance of smart manufacturing systems. Finally, we analyze how those trends 

apply to the methods studied and propose future research for assessing and improving 

manufacturing performance in the uncertain, multi-objective operating environment. 
ii

1 Introduction.......... ................................................................................................................... 1

2 Background .............................................................................................................................. 2

2.1 Understanding performance assurance for smart manufacturing systems ....................... 2

2.2 Performance assurance as quality assurance .................................................................... 5

2.3 Importance of measurement ............................................................................................. 6

2.4 Measuring to achieve strategic goals ............................................................................... 7

2.5 Existing Sources of Metrics and Indicators...................................................................... 8

3 Methods for assuring performance of manufacturing systems ................................................ 9

3.1 Performance measurement frameworks ........................................................................... 9

3.1.1 Balanced score card ................................................................................................ 10

3.1.2 Performance measurement matrix .......................................................................... 11

3.1.3 Strategic measurement and reporting technique – SMART ................................... 12

3.1.4 Performance prism .................................................................................................. 13

3.1.5 Comment on performance measurement systems ................................................... 13

3.2 Enhancements to performance measurement frameworks ............................................. 14

3.2.1 Integrated performance measurement ..................................................................... 14

3.2.2 Multi-level, dynamic performance measurement ................................................... 14

3.2.3 Integration with new tools ...................................................................................... 15

3.2.4 Research to develop new frameworks .................................................................... 16

3.2.5 Performance measurement and change in operating environment ......................... 16

3.2.6 Change management ............................................................................................... 16

3.3 Performance analysis methods ....................................................................................... 17

3.3.1 Statistical methods and machine learning ............................................................... 17

3.3.2 Operations research methods .................................................................................. 18

3.3.3 Decision theory ....................................................................................................... 19

3.3.4 Computer simulation ............................................................................................... 20

3.4 Emerging methods for SMS performance analysis ........................................................ 22

3.5 Performance improvement methods............................................................................... 23

3.5.1 Derivatives of the PDCA cycle ............................................................................... 23

3.5.2 Lean manufacturing, Six Sigma, and total quality management ............................ 24

3.5.3 Using real-time data analytics for continuous performance improvement ............. 25
iii

3.6 Software tools ................................................................................................................. 26

4 Applying performance management methods to SMS .......................................................... 27

4.1 Targeted performance objective for smart manufacturing – Agility.............................. 28

4.2 Targeted performance objective for smart manufacturing – Sustainability ................... 29

4.2.1 Standards for sustainable manufacturing ................................................................ 29

4.2.2 Reference data ......................................................................................................... 30

4.2.3 Sustainable practices and continuous improvement ............................................... 30

4.3 Targeted performance objective for smart manufacturing – Productivity ..................... 31

4.4 Challenges to overcoming operational exceptions ......................................................... 32

4.5 Challenges with collection and use of data for performance improvement ................... 33

4.6 Methodology steps for performance assurance for SMS ............................................... 34

5 Summary and future research directions ............................................................................... 36

5.1 Summary ........................................................................................................................ 36

5.2 Addressing requirements for performance assurance for SMS ...................................... 36

6 References ............................................................................................................................. 38
1

1 INTRODUCTION 

Maintaining high performance required for today’s manufacturing systems requires employing 

methods and tools to manage performance throughout the system life cycle. During system 

design, for example, methods and tools model, analyze, and test the system so that anticipated 

manufacturing needs and operating environments are investigated before the system is built. In 

the operation of the system, methods and tools are employed to ensure that the system maintains 

planned performance through monitoring, assessing performance, and responding appropriately 

to performance deviations. In this paper, we review three categories of methods and tools used in 

manufacturing systems for the following high-level objectives: 

 Performance measurement

 Performance analysis

 Performance improvement

We discuss how these methods, tools, and systems can be used in, and are impacted by, 

emerging changes to the manufacturing environment. The new era of manufacturing systems, 

known as smart manufacturing systems (SMS), is marked by a wide availability of operational 

performance data that was previously not available for performance management. In the past, 

such data was either not collected at all or, when collected, was used primarily for production 

control on the factory floor. The availability of Information and Communications Technologies 

(ICT), such as ubiquitous sensors and wireless communications, is resulting in an onslaught of 

readily accessible data that could also be used for planning at higher organizational levels, if 

context and meaning could be deduced from the data. Poor interoperability of this data is still a 

barrier, in large measure because the data is digital, and therefore commonly available in 

proprietary, mutually incompatible, formats. Current efforts to solve the data interoperability 

challenge across manufacturing hierarchical levels will blur historical lines between 

organizational planning and operational control. 

Another shift in the manufacturing environment relates to the type of performance being 

measured. Traditionally, productivity and quality were the primary gauges of performance. A 

more productive system is one with higher throughput for a given amount of input resources, 

which positively impacts a business’s bottom line. Now, however, two other motivating factors 

are taking on a more critical role: sustainability and agility. Sustainable manufacturing refers to 

the creation of manufactured products employing processes that are non-polluting, energy- and 

natural-resource conserving, and economically sound to the manufacturing organization and 

consumers. Sustainable manufacturing focuses on three main impact areas: environmental, 

economic, and social. Agility refers to an organization’s ability to anticipate, respond quickly to, 

and even take advantage of changing market demands and changing flow of supplies as well as 

responding to internal disruptions and disturbances. Increased attention to these two new 

performance factors could negatively impact productivity but is crucial for an organization to 

succeed in today’s markets. Techniques for assessing and balancing multiple factors using 

operational data in real time are still needed and are an area of ongoing research in SMS. In a 
2

2010 report, the Smart Manufacturing Leadership Coalition (SMLC) (SMLC 2010), an 

independent, public-private partnership, defined a vision for SMS. That vision states that the 

future holds a “fundamental shift in manufacturing processes toward demand-dynamic 

economics, flexible factories, and demand-driven supply chain service enterprises.” As discussed 

in Davis et al. 2012, the SMLC believes that this vision is multi-dimensional—it must be 

implemented vertically across planning levels and horizontally across the product life cycle. In 

this paper we focus on one part of that vision: the manufacturing plant. 

Fundamental to assuring performance is measuring performance. All of the methods and 

techniques described in this paper rely on measurement. Performance measurement provides the 

state of performance. At higher levels in the factory, the measurements are rolled up into key 

performance indicators (KPIs). An organization can include, within its strategy, the improvement 

of a set of KPIs (e.g., environmental sustainability), which may influence the specification of 

hardware and software, and data collection and measurement policy. For example, SMS are 

accompanied with greater investment in automation and Information and Communications 

Technology (ICT) to pursue KPIs that are defined by today’s performance objectives. 

Performance measurement is followed by performance analysis so that critical factors governing 

performance are identified and decisions are made for improvement. 

Our motivation in this paper is to understand current trends in existing methods and tools in 

support of SMS, and to identify opportunities provided by emerging technologies for 

performance assurance within the manufacturing plant. The rest of the paper is arranged as 

follows. Section 2 is a background to SMS, performance measures, and aggregate performance 

indicators. Section 3 reviews a variety of methods and tools, and previous research in their 

application to manufacturing systems with a focus on operations management. Section 4 

discusses performance assurance issues and the adaptation of current methods and tools for SMS. 

Section 5 concludes the paper with a summary and future research needs. 

2 BACKGROUND 

This section overviews the performance challenges and approaches to performance measurement 

and improvement.

2.1 Understanding performance assurance for smart manufacturing systems 

Performance assurance establishes confidence that the system performs as intended. This 

requires the ability to assess a proposed design and monitor performance of a manufacturing 

system operation. In essence, performance assurance emphasizes the proactive approach, i.e., 

forecasting and monitoring, and preventing problems (King, 2005). Figure 1 illustrates the 

context of the manufacturing performance assurance challenge. The figure shows a system with a 

given input purposed to result in a level of desired performance but subjected to both internal and 

external disruptions and disturbances. By our definition, disruptions are those whose occurrence 
3

results in temporary shutdown of at least one unit of the manufacturing operations, while 

disturbances are events that do not lead to stoppage of production. Internal disruptions and 

disturbances can be both planned and unplanned events on the manufacturing floor.

Traditionally, these events are anticipated and appropriately planned for to achieve and maintain 

required performance.

Figure 2 is a schematic representation of a rigorous approach to manufacturing performance 

assurance during system design and development. The first activity is to specify appropriate

KPIs. The second activity is to establish target system performance envelope according to the 

KPIs. The goal is that the system built will achieve and maintain minimum performance 

requirements, within the set bounds, irrespective of disturbances and disruptions. The third

activity is to determine needed system components to deliver the required performance. The 

integration of such components (physical and virtual) requires developing a system architecture. 

The system component models of hardware, software, information, and tools, are integrated

based on system engineering principles and tools (Kossiakoff et al. 2011). Lastly, predictive 

tools such as simulations are applied to test the system performance in a virtual environment to 

determine if the desired system behavior will be obtained. The results of the test may show that 

the system needs modifying or redesigning with a different set of components and retesting until 

performance, for anticipated range of operational conditions and inputs, is within the 

performance envelope. 

Manufacturing system

Disruptions and disturbances Actual performance Reference performance +

-

Performance gap

System decision making

System inputs

Feedback on actual performance Figure 1: Performance assurance context for a manufacturing system
4

Define key performance indicators Determine performance envelope Identify and integrate system components Investigate system behavior Test and validate the system Manufacturing system performance

System architecture

Modeling methodology Predictive Design modeling revision

Figure 2: Performance assurance process during manufacturing system design

Performance assurance requires methods and tools to track performance, recognize the effects 

and sources of disruptions and disturbances, and guide decision making. Tracking a system over 

time with a view to improving performance through corrective actions can be modeled using 

classical control theory. Applying control theory to a manufacturing system has been a research 

area for some time and still ongoing. As early as 1979, Parnaby (1979) argued that 

manufacturing systems are too complex to apply control theory for continuous real-time overall 

optimization. Instead, he proposed, a theoretically-based heuristic procedure for discontinuous 

control. Considering that today’s manufacturing systems are more automated, monitored, and 

controlled, research that includes continuous control is now sensible. There are different methods 

of control, including supervisory control, discrete-event control, hierarchical control, intelligent 
5

control, optimal control, and adaptive control. We describe one of these types of control:

adaptive control and how it can be applied to performance management.

Adaptive control is a set of techniques that provides a systematic approach for automatic 

adjustment of control settings in real-time to achieve or maintain required performance as system 

operating parameters change (Landau et al. 2011). Knowledge of system characteristics is

obtained while the system is operating and a controller adapts to the changes. In addition, the 

control laws change themselves as the system changes. In SMS, the adaptive control approach 

could conceivably be used to monitor performance as the internal situation (e.g., increase in 

energy usage of a machine) of the manufacturing system changes and determine required inputs 

to achieve or maintain required performance.

Performance assurance for SMS is analogous in purpose and scope to that of software systems 

and architecture and therefore, similar approaches and methods are applicable (Rotithor 2013). 

Performance assurance systems for software ensure that the software features and functionality 

meet requirements and that the code is bug free. By analogy, the “features” of manufacturing 

systems are hardware, software, and management systems while “functions” are behaviors

defined by the purpose for which each system is built. “Bug free” is elimination or minimization 

of performance failures triggered by disruptions and disturbances and the system being properly 

aligned with the intended performance objectives.

Dan Shoemaker (Shoemaker 2013) reviewed the state of the practice for software assurance. 

Two notable activities in this area include the Standard Assurance Environment standards 

emerging from OMG (OMG 2013) and the Common Weakness Enumeration (CWE) repository 

at MITRE (Common Weakness Enumeration 2014). The OMG environment developed a 

rigorous framework and method for the user to specify test cases for systems assurance. The 

CWE is a database of known weaknesses in systems engineering including discussion of the 

problems and solutions. The CWE also includes a risk assessment framework for analyzing and 

prioritizing the potential weaknesses relative to a given project to better understand where to 

focus testing efforts.

2.2 Performance assurance as quality assurance 

The purpose of performance assurance is for a system to perform as intended. A closely related 

activity for a product is quality assurance (Dale et al. 2007). Quality assurance covers all 

activities (from product design to delivery) performed so that a product or service fulfills its 

performance requirements. If a manufacturing system is considered as a product, then the 

principles of product quality assurance can apply to a manufacturing system. 

Table 1 compares the objectives, methods, and scope of product quality assurance with those for 

manufacturing system performance assurance. Table 1 defines the objectives for system 

performance as achieving high performance defined relative to specific objectives and goals. 
6

Examples of these are overall equipment effectiveness (Cesarotti et al. 2013), agility (Putnik et 

al. 2007), and energy consumption (Yang et al. 2013). Whatever goals an organization may have,

the first activity is always taking measurements. Making measurements is discussed next.

Table 1: Comparing performance assurance with quality assurance

Product quality assurance System performance assurance

Objectives Achieving high-quality product 

and preventing design/production 

mistakes/defects

Achieving high performance and 

preventing performance deterioration

Tools and Methods Using statistical methods, Right 

first time

Systems engineering methods and tools 

Diagnostics and prognostics methods

Scope Fit for purpose System performs as desired

Simulation purpose

(Exemplar method)

Failure testing Simulated testing over the performance 

envelope

2.3 Importance of measurement

Measurement provides necessary information to decision makers and plays a role in monitoring 

performance, enhancing communication, and diagnosing and solving management problems 

(Chen 2006). Taticchi et al. (2010) noted that the process of measurement is crucial to improving 

business performance. Keegan et al. (1989) and Ahmed et al. (2002) proposed that performance 

measures should be derived from and be aligned with the business strategy. They synthesized a 

set of common performance objectives for manufacturing, namely, safety and environment, 

flexibility, quality, and dependability. They carried out an extensive review of performance 

measures used at that time, and synthesized those necessary for “world-class manufacturing”,

namely, top management commitment, customer service, price/cost, quality, facility control, 

speed, innovation, and technology.

Literature on performance measurement is extensive, although few performance measurement 

systems are oriented towards manufacturing systems (Hon 2005). In his extensive review of 

performance measures, Hon (2005) identified that in addition to cost, the measurements oriented 

to manufacturing that are in common use are quality, productivity, time, and flexibility. Another 

set of important manufacturing measures identified by Chen et al. (2007) are on-time delivery, 

cost of quality, and quality rate. We discuss how existing sets of indicators and measures can be 

adapted for SMS. But first, we clarify related terminology as used in this paper.

The terms indicators, metrics, and measurements are sometimes used interchangeably.

Technically, they are different. Feng et al. (2010) distinguished indicators from metrics. An 

indicator is a parameter, which points to, provides information about, or describes the state of a 

phenomenon with significance and relevance to performance objectives. ISO 22400-1 (2014) 
7

specifies that a KPI is one that is critical to the current and future success of the organization. 

Indicators should, among other things, be numerically and precisely quantifiable. On the other 

hand, a metric is a measurable quantity for tracking an indicator or indicators. A metric can be

composite, i.e., made up of two or more measurements. Measurements are the magnitude or 

values of actual data gathered from the process based upon a standard or unit of measurement. 

To track the performance of a system with respect to an indicator, it is necessary to perform 

appropriate and accurate measurements. To develop a measurement system for performance of 

manufacturing processes, we can borrow questions proposed by Tsourveloudis et al. (2002): 

what do we measure? How and when do we measure? How do we interpret the results? In this 

paper, we discuss factors that determine what and when to measure. How to measure will be 

covered in future work. 

2.4 Measuring to achieve strategic goals

Digalwar et al. (2005) established that a prerequisite to managing performance is measuring 

performance. The nature of performance is that it is constantly changing and is unique to the 

system and operational situation. Establishing a single set of performance metrics across 

manufacturing enterprises would not only be impractical, but would also fail to address the 

dynamic nature of performance. Competitive priorities identified for performance management 

typically vary from one firm to another and shift with business demands. Measurements and 

metrics for performance management originate from both internal and external sources and are 

also driven by performance priorities of each organization (Hallgren and Olhager 2009). In most 

cases, a manufacturing strategy is formulated to exploit core competencies and to serve market 

segments to achieve competitive advantages, and appropriate measures are determined based on 

the above-mentioned performance priorities and their origins.

Developing measurements that are useful for decision making has been a challenge for some 

time and is commonly discussed in the context of a knowledge or DIKW (Data, Information, 

Knowledge, Wisdom) pyramid. This first pyramid in Figure 3 illustrates how intelligence, or 

wisdom, is built by drawing conclusions from data, turning it first into information when context 

is added, then into knowledge where meaning is drawn from the information, and finally into 

wisdom where it can be used to guide decision making. The image suggests that each level in the 

hierarchy can be derived from the others. We adopt this image to illustrate the importance of 

measurement to performance assurance. 

The second triangle in Figure 3 applies this image in the context of performance management. At 

the top level of the pyramid are the strategic goals. At the base are the measured values. 

Measurements are abstracted by metrics into indicators. Indicators provide a context for the 

measures turning them into information. Indicators in turn can be structured to logical deductions 

leading to KPIs, which are the basis for decision making. The final triangle provides an example 

of measures, indicators, a KPI and a strategic goal. The strategic goal example is sustainability, 
8

for which a major factor can be water consumption during production. The measurements for 

water are water consumed, waste water, and number of parts produced. The percentage of water 

reclaimed is one of the indicators calculated from the measures. Ultimately, interest is in water 

usage per part as a KPI that can be compared with previous periods, a benchmark, a set target, or 

a standard. Other KPIs can be likewise abstracted.

Data Information Knowledge Wisdom Measurements Indicators Key performance indicators

Strategic goals Gallons water consumed, Gallons waste water, number parts produced Percent of reclaimed water Water usage per part produced Sustainability

Figure 3: The DIKW Pyramid (left) applied to performance management (center) and 

manufacturing sustainability due to water consumption (right)

2.5 Existing Sources of Metrics and Indicators

Metrics abound. Every organization and indeed every opportunity for measured improvement 

has its own metrics. The performance measurement frameworks discussed in section 3 produce 

a large number of metrics. Other sources for metrics include industry associations, public 

repositories, and research projects. Metrics developed from manufacturing research often take on 

the focus of a given performance objective, such as productivity, energy consumption, agility, 

and sustainability. We summarize some of these.

In 2014, Technical Committee 184/ Subcommittee 5 (TC 184/SC 5) within the International 

Organization for Standardization (ISO), published the first two, i.e., ISO 22400-1 (ISO 2014) 

and ISO 22400-2 (ISO 2011), of four standards on KPIs for manufacturing operations 

management. Industry associations that are particularly relevant to Smart Manufacturing and

publish material related to performance metrics are the Manufacturing Enterprise Solutions 

Association (MESA) and the Supply Chain Council. NIST has established a public repository of 

metrics for sustainability assessment which consolidates the work of a number of organizations 

in the sustainable manufacturing area (NIST Manufacturing Engineering Laboratory 2009). 

Further, MESA has established a metrics’ working group as a forum for manufacturers to share 

information and best practices about performance measurement and improvement (MESA 

International 2014). The group hosts several activities, including the development and 

maintenance of a metrics guide book. MESA is also working with ISO on plant-level assessment 

standards by defining KPIs at the level of manufacturing operations management to help supply 
9

chain partners by providing commonly accepted performance measures (ISO 2011; 2014). NIST, 

in a joint effort with the MESA metrics group, is developing a method and models for selecting 

KPIs appropriate for any manufacturing process. The standardized method should simplify the 

KPI selection process while maintaining the reliability and validity of measurements (NIST 

Engineering Laboratory 2013). 

The Supply Chain Council published the Supply Chain Operation Reference (SCOR). This 

Guide is a diagnostic tool for improving supply chain management and can also help in 

implementing targeted strategic objectives. SCOR was designed for supporting the supply chain 

of an organization but is useful within the organization as well.

A key requirement of SMS is that they should have minimal impact on the natural environment. 

Indicators for sustainable manufacturing have been developed through research. In 2001, the 

Lowell Center for Sustainable Production, University of Massachusetts, proposed an indicator 

framework for raising companies’ awareness and to move towards sustainable manufacturing 

(Veleva and Ellenbecker 2001). The research proposed twenty-two indicators and a guide to their 

application in manufacturing production. Kibira et al. (2010) specified high-level indicators for 

the environmental, economic, and social dimensions of sustainability. Joung et al. (2013) 

categorized indicators for sustainable manufacturing, while Feng et al. (2010) and Reich-Weiser 

et al. (2008) overviewed metrics for sustainable manufacturing. NIST developed a sustainable 

manufacturing indicator repository for use by manufacturers (NIST Manufacturing Engineering 

Laboratory 2009).

3 METHODS FOR ASSURING PERFORMANCE OF MANUFACTURING 

SYSTEMS

Thomann (1973) defined methods as “rules or procedures that guide someone in accomplishing a 

purpose.” Methods consist of guidelines or rules of thumb. The procedures of evaluation may be 

both qualitative and quantitative. Tools are based on methods and are often supplied as computer 

software. Many definitions or references to tools and methods show that a tool can be composed 

of various methods. In this section, we review performance measurement systems, research 

trends in performance measurement, performance analysis systems, emerging methods for 

performance analysis, performance improvement systems, and software tools.

3.1 Performance measurement frameworks

Performance measurement is the process of collecting, analyzing, and reporting performance 

information. The decision on what to measure stems from the objectives and policies of the 

organization. Measurements from the shop floor are analyzed and results are rolled to higher 

levels to determine system performance. These processes are illustrated in Figure 4. This section 

reviews a sample of the measurement frameworks. Performance assurance involves (i) setting a 
10

performance target level and (ii) comparing prevailing performance against the target. The most 

commonly used performance measurement frameworks are:

• balanced score card (Kaplan et al. 1992),

• performance measurement matrix (Keegan et al. 1989),

• strategic measurement and reporting technique – SMART (Cross and Lynch 1988-1989),

• performance prism (Neely et al. 2002)

Efforts to enhance these frameworks to advance measurement systems to address changing 

industrial needs, and discussed in this paper, have been:

• integration of performance measurement (Bititci et al. 1997),

• dynamic measurement (Ghalayini et al. 1997),

• enhancement of performance measurement systems with various techniques and tools

(Charnes et al. 1995; Saaty 2009),

• change management (AIAG 2009).

3.1.1 Balanced score card

Traditionally, manufacturers have used financial measures to evaluate performance (Gomes et al. 

2004). Such measures typically assess current performance against previous periods. Current 

manufacturing systems are characterized, however, by change, and the effects of this change 

must be tracked both periodically and continuously. Therefore, cost and profit measures alone 

would not be timely in revealing areas within the organization that need to be improved. 

Manufacturing systems need measures for evaluating and guiding future investments, and longterm retention and acquisition of customers, suppliers, and employees. 

The balanced score card (BSC) was developed to provide such features. In addition to financial

measures, the BSC includes “operational measures on customer satisfaction, internal processes, 

and the organization’s innovation and improvement activities–operational measures that are the 

drivers of future financial performance” (Kaplan and Norton 1992). Since the BSC was first 

developed, it has undergone two refinements to make it easier to define measures and to provide 

more relevance (Kaplan and Norton 2001; Lawrie and Cobbold 2004). Table 2 summarizes the

perspectives and measurements for the balanced score card. 
11

Shop Floor Operations Machine level Cell level Factory level Enterprise level Measures Objectives

Figure 4: Flow of performance information from shop floor to enterprise level

Table 2: Balanced score card measurements

Perspective Generic measurements

Financial Return on capital employed, value added, sales, capital flow

Customer Customer satisfaction, retention, acquisition, profitability, market 

share

Internal business process Innovation – how well the company identifies customer needs

Operations – quality, cycle time, costs

After-sales service – repair, service, returns

Learning and growth People – employee retention, training, skills, morale

Systems – measures of availability, real-time information

3.1.2 Performance measurement matrix

The performance measurement matrix was also developed to answer the need for balanced 

measurement. It was proposed by Keegan et al. (1989) after realizing that many organizations 

have far too many (irrelevant) performance measures. It was felt that some of these measures

were obsolete and inconsistent and that they could actually frustrate implementers of

organizational strategy. The premise of the performance measurement matrix is that measures 

have to be classified into financial and non-financial impacts on business performance, each 

depending on both internal and external drivers. Neely et al. (2000) modified the original

performance measurement matrix. The modified measurement matrix assumes two basic types of 

performance measures, those that relate to results (competitiveness, financial performance), and 

those that focus on the determinants of the results (quality, flexibility, resource utilization, and 

innovation). The results are termed “lagging” while the determinants are “leading” indicators.
12

3.1.3 Strategic measurement and reporting technique – SMART 

The SMART performance measurement framework was developed by Wang Laboratories, Inc. 

to improve performance measurement over the traditional measures (Cross and Lynch 1988-

1989). SMART integrates organizational objectives with operational performance. The SMART 

performance criteria and their definitions, which originated with Doran (1981), used in the 

SMART framework have become iconic:

Specific – what are we going to do for whom?

Measurable – is it quantifiable?

Attainable/Achievable – can we get it done within the time frame?

Relevant – will this objective affect the desired goal?

Time bound – when will this objective be accomplished?

The SMART framework uses both external and internal performance measures. There are four

levels in the pyramid of objectives and measures (Figure 5): corporate vision and strategy, 

market and financial performance measures for business units, objectives and measures (e.g., 

customer satisfaction and flexibility, and quality, delivery, process time), and cost for each 

department and work center. SMART’s strength lays in its direct intention of integrating 

corporate objectives with operational performance, as well as its focus on managing strategic 

objectives. However, as Striteska et al. (2012) have pointed out, SMART’s weakness in 

comparison with BSC and the measurement matrix is that it does not provide a mechanism for 

specifying KPIs.

Operations Product quality Delivery Cycle time Waste Customer satisfaction Flexibility Productivity Markets Financial Corporate vision Business units

Business operating systems

Department and work centers

Objectives Measures

Figure 5: The SMART performance measurement framework
13

Figure 6: Performance prism

3.1.4 Performance prism

The focus of the performance prism is to identify all the stakeholders and endeavor to deliver 

appropriate value to them. As the name suggests, its concepts can be illustrated using a triangular 

prism, as illustrated in Figure 6. The base represents stakeholder contribution while its opposing 

side represents stakeholder satisfaction. Three sides represent the strategies, capabilities, and

business processes that deliver products and services. There are five perspectives that the prism 

seeks to integrate through five fundamental questions:

Stakeholder Satisfaction: Who are the stakeholders and what do they want and need? 

Stakeholder Contribution: What is wanted and needed from the stakeholders? 

Strategies: What strategies need to be put in place to satisfy these sets of wants and 

needs? 

Capabilities: What capabilities – people, practices, technology and infrastructure – need 

to be put in place to allow us to operate processes more effectively and efficiently? 

Processes: What processes need to be put in place to satisfy the wants and needs? 

3.1.5 Comment on performance measurement systems 

Performance measurement systems and frameworks need improvements to be more relevant for 

smart manufacturing assurance. The features lacking in existing frameworks are:

• Integration with other management and manufacturing systems and data 

• Multi-level dynamic performance assessment

• Real-time dynamic performance assessment across multiple criteria
14

• Integration into operational control systems, which would require not only evaluation of 

multiple performance indicators but mechanisms for specifying constraints for optimizing 

among multiple criteria

• Methods and framework for collecting and communicating measurements from shop 

floor 

Research efforts to address the first two bullets are reviewed in sub-sections 3.2.1 and 3.3.2

while the remaining three bullets are discussed in the last section of this paper.

3.2 Enhancements to performance measurement frameworks

This sub-section discusses integrated performance, multi-level performance measurements, and 

the use of tools to enhance measurement frameworks. We also address needs for performance 

measurement frameworks to address change in operating environment. 

3.2.1 Integrated performance measurement

Integrated performance measurement systems introduce a proactive closed-loop control system 

in performance management where corporate strategies are employed in all business processes, 

activities, and tasks. Feedback is obtained through actual measurement from the lowest level

activities progressively to higher levels, e.g., from business activities, to business processes and

business units all the way up to corporate level. Integrated performance measurement seeks to 

establish structural relationships between performance measures, and develop a reference model 

for comparing different integration approaches (Bititci et al. 1997; Neely 2002). 

3.2.2 Multi-level, dynamic performance measurement

Research into this area is geared to developing measurement systems in continuous pursuit of 

company objectives. An example is the integrated dynamic performance measurement system

(IDPMS), which provides an approach to support alignment of objectives across managerial and 

operational levels (Ghalayini et al. 1997). The system comprises and integrates three main areas 

that facilitate performance measurement and improvement, i.e., management, process 

improvement team, and shop floor. The system has feedback loop between the shop floor and the 

process improvement team. The three areas are linked through specification and reporting on the 

defined areas of success, performance measures, and performance standards. The feedback on 

performance as the operating environment changes provides the dynamic updates. These

concepts are illustrated in Figure 7.

Although research in this area has been ongoing for almost two decades, fully developed systems

in use are not common in literature. The results from this research, if published, could be used to 

complement performance improvement frameworks such as the Plan-Do-Check-Act cycle

(discussed in Section 3.6).
15 Management

Process improvement teams

Shop floor

Policies Feedback on performance

Figure 7: The integrated dynamic performance measurement system framework 

3.2.3 Integration with new tools

Specific tools to enhance performance measurement include Analytical Network Process (ANP) 

(Saaty 2009) and Data Envelopment Analysis (DEA) (Charnes et al. 1995). Examples of these 

are Ucal and Öztaysi (2009), who used ANP to quantify performance of a manufacturing system 

based on selected performance criteria and sub-criteria of a manufacturing firm, and Benbarka 

(2007), who explained how DEA can be used as a tool to measure organizational performance 

over time. 

Other forms of enhancing measurement systems are expected through application of emerging 

technologies. The prospects for adapting current approaches to performance assurance are

discussed in Section 4. Here, we summarize the following recent technology trends that will 

impact measurement frameworks:

• increasing availability of real-time system information

• improved acquisition and communication of information such as customer orders to the 

design and production system, enhancing the ability for mass customization

• improved integrated performance management systems for addressing multiple objectives 

of manufacturing

• proliferation of continuous improvement programs and standards

• making an integrated performance measurement part of overall planning and operational 

systems.

Some of the main challenges to leveraging the above trends in technology are inconsistent 

information quality and, still, interoperability limitations. SMS applications are composed of 
16

different components and heterogeneous technologies. New standards need to be developed or 

existing standards enhanced to transfer across different platforms for processing, analytics, and 

use. Automated tools still require human expertise to reliably extract and identify relevant data 

for existing analytical tools.

3.2.4 Research to develop new frameworks 

Wibisono (2011) argued that performance measurement systems are becoming obsolete and that 

new dynamic frameworks are needed. In particular, he observed that current systems cannot 

balance short-term vs long-term measures, internal vs external measures, and financial vs 

operational measures. He identified nine steps in formulating an improved performance 

measurement system for manufacturing. These are (i) determine a PMS framework, (ii) identify 

company environment, (iii) formulate a company’s mission statement, (iv) analyze current PMS, 

(v) determine performance variables, (vi) determine cause-effect amongst variables, (vii) 

determine performance standards, (viii) determine improvement priority, and (ix) formulate 

recommendations to address gaps in performance. Wibisono’s ideas were tested by conducting 

interviews and a field survey in ten large manufacturing industries. Each company provided 

experts to validate the framework in terms of their key performance indicators, suitability of the 

indicators, benchmarking of the indicators, and linkage amongst the indicators. The response 

from this exercise indicated that the framework would offer advantages over the existing 

balanced score card and performance prism, which require “further adjustments” before 

implementation.

3.2.5 Performance measurement and change in operating environment 

Manufacturing systems are characterized by constant change. Major changes that routinely affect

manufacturing systems include type of products made and environmental regulations, yet most 

assessment and measurement systems used today tend to be static. Change also affects the ability 

of measurement frameworks to address users’ assessment needs. To address change needs, 

Salloum (2011) developed a framework for manufacturing organizations to develop dynamic 

performance measurement systems. To be effective, performance measurement systems have to 

be integrated into existing management information systems (MIS). In this connection, 

Nudurupati et al. (2011) discussed the role of MIS in implementing a performance measurement 

system from its design through implementation, use, and review. 

The above publications indicate that future performance measurement frameworks for SMS 

might enable users to continuously update measurement systems as technology and strategic 

goals change. 

3.2.6 Change management

Change management is the systematic approach to dealing with transitioning from one state of an 

organization to a different future state. Recognition that purely financially based performance 
17

measures are no longer adequate has led to changes in performance measurement frameworks. 

Because many attempted implementations of new systems fail (Bourne et al. 2003), the process 

of implementing new performance measurement systems requires careful change management. 

Engineering change management is a process for which the Automotive Industry Action Group 

(AIAG) has done extensive work in collaboration with global OEM-supplier VDA-Germany, 

GALIA-France, Odette-Sweden, and JAMA-Japan (AIAG 2009). AIAG has published 

recommendations for change management that could be globally applied.

3.3 Performance analysis methods

Smart manufacturing systems are characterized by machines being outfitted with sensors and 

monitors and then networked together using wireless technologies. These devices and 

technologies are collecting larger amount of data than was previously available. The data being 

collected across the manufacturing operation—and external data from customer orders, product 

field data, and suppliers—is flooding into manufacturing enterprises and calls for enhanced or 

new analysis methods.

The methods being discussed in this section are widely used in analysis and decision making in 

manufacturing systems. Methods commonly employed for analyzing data produced by 

manufacturing systems employ a variety of techniques, each with its own strengths, limitations, 

and success record depending on the situation to which it is applied. Established methods include 

statistical methods and machine learning, decision theory, computer simulation, and operations 

research.

3.3.1 Statistical methods and machine learning

Manufacturing engineers on the shop floor cannot observe or record a system’s entire body of 

data and have traditionally relied on a collected sample to draw conclusions about the system.

The data mining and analytical methods being developed, however, are poised to provide

conclusions from not only large volumes of static data but also in real time from large streaming 

data. These methods are discussed in subsequent sections. Still, conventional statistical 

techniques (as reviewed in the next two paragraphs) will continue to be important since they are 

also part of advanced analytics methods being developed by research organizations and industry. 

Statistical analysis methods include statistical process control, correlation analysis, regression 

analysis, hypothesis testing, categorical modeling, analysis of variance (ANOVA), general linear 

model, computational statistics, and computational Bayesian modeling. Statistical methods are 

particularly useful during operation to measure, predict, and control performance of systems.

One area of application is quality control, where statistical methods are used to monitor by using 

measurements of a quality characteristic of a product thereby enabling control of the underlying 

production process (Montgomery 2009). These methods determine if a system is operating as 

intended. Correlation analysis determines whether there is a relationship between sets of data;
18

and where a relationship exists, it is determined through regression analysis. Hypothesis testing 

tests a claim about a parameter in a population, using measured data. Analysis of variance is used 

to compare three or more data sets.

Statistical methods applications in manufacturing are for process improvement and productivity 

gains, in addition to product quality improvement. Because of the increasing complexity of the 

systems and the larger number of parameters involved, statistical methods, through modeling, 

optimization of manufacturing parameters, and monitoring and control, have greatly increased 

(Delorme 2006). One of the critical areas for statistical methods for performance assurance is in 

establishing and maintaining reliability of manufacturing equipment. Raheja et al. (2006) define 

assurance as processes for ensuring that a product performs well during its expected life. In the 

case of machines on the shop floor, accelerated experiments and other assurance methods are 

used to reduce equipment failures during actual use. Barabady (2007) used the term “production 

assurance” to describe the methods for ensuring that equipment and machines are available to 

meet the requirements for demand deliveries. Lastly, statistical methods are now poised to 

analyze the vast data available to obtain more insight and to better predict future system status

(Halvorsen 2006).

Machine learning is a discipline of constructing algorithms that can recognize and learn from 

patterns in data (Samora et al. 2008). The algorithms can be from a variety of fields including 

statistics and information theory. Machine learning uses data as a learning set for future 

decisions. Statistical methods and machine learning are poised to become leading methods for 

performance assurance because of the large increase in data obtained from manufacturing 

systems in real time. Section 3.6 discusses the role of these methods in continuous performance 

improvements based on data analysis.

3.3.2 Operations research methods

The field of Operations Research (OR) comprises analytical methods that are employed to obtain

optimal or near-optimal solutions to complex decision problems. Examples of the methods are 

linear programming, nonlinear programming, stochastic programming, network flow 

programming, integer programming, queuing theory, inventory theory, and data envelopment 

analysis. OR methods help model the structure of complex situations, and utilize this 

understanding to predict system behavior and improve system performance. OR applications in 

manufacturing include part routing, workflow improvements, elimination of bottlenecks, 

inventory control, business process re-engineering, site selection, facility layout, and general 

operational planning (Heger 2006). 

A major area of OR for manufacturing applications is production scheduling (Mönch et al. 

2011). Solutions to scheduling problems can lead to improved performance through optimal use 

of resources. Engau (2012) proposed OR methods for solving environmental sustainability
19

problems. There are a number of OR techniques and methods from which to select for particular 

applications in specific SMS environments. To optimize multiple performance objectives, OR 

methods can be combined with data analytics and simulation models. For example, they could be 

used in conjunction with multi-criteria decision making methods to optimize manufacturing

performance through several simulated scenarios. In summary, potential areas for OR are

• Optimizing layout and configuration of the manufacturing system

• Deriving a production plan that optimizes a given objective such as minimum cost

• Assessing, comparing, and tracking performance of different units 

This potential notwithstanding, currently OR has had little impact in real life manufacturing 

environments. This is mainly because solutions from these models are often obtained after 

unrealistic assumptions and time consuming (Semini 2011). Moreover, the situation on the shop 

floor changes continuously. Summary of general weaknesses of OR are

• They mainly apply where factors can be quantified. In many cases such as human 

behavior, this many not be possible.

• They involve assumptions that require assigning quantitative values to factors, with a 

result that can be subjective and inaccurate.

• They are less efficient for systems with continuously changing data.

• The techniques and models described in literature are often different from those actually

used in practice (Semini 2011).

• Optimal solutions are often computationally complex (Brucker 1979).

3.3.3 Decision theory

Decision theory is the study of preferences, uncertainties, and other issues related to making 

choices for obtaining the best outcome of a situation. Dieter and Schmidt (2013) describe several 

steps for making decisions. The steps start with determining and prioritizing objectives, and end 

with making and evaluating the future impact of the decisions. The authors also describe three 

situations that make decision making difficult. These are (i) decision under uncertainty—each

state of nature has an assigned probability of occurrence, (ii) decision under risk—each action 

can result in two or more outcomes, but the probabilities of occurrences unknown, and (iii) 

decision under conflict—where decisions to improve any performance objective results in a 

deterioration of another.

A first step in making a decision is to precisely describe the objective(s). The potential presence 

of more than one objective has led to the development of multi-criteria decision-making 

methods. Multiple criteria decision making (MCDM) refers to making decisions in the presence 

of multiple, often conflicting, criteria. Triantaphyllou (2000) determined that there are many 

ways to classify MCDM problems and that one of them is by the type of data they use. Data can 

be deterministic, stochastic, or fuzzy. However, the methods for solving MCDM problems are 
20

largely of two types, and they depend on the problem setting: one type having a finite number of 

alternative solutions and the other an infinite number of solutions. 

During manufacturing system design or operation, the characteristics, parameters, or attributes 

upon which a decision has to be made can take any value within a given range. Therefore, the 

potential alternative solutions can be infinite. The problem is then referred to as a multiple 

objective optimization problem. Selection of a solution can depend on the preferences of the 

stakeholders. Triantaphyllou (2000) reviewed a number of MCDM methods used. They include 

the weighted sum method (WSM), weighted product method (WPM), and analytic hierarchy 

process (AHP), elimination and choice translating reality (ELECTRE), and technique for order 

preference by similarity (TOPSIS). Another common approach is to rank the alternatives from 

the best to the worst, based on the stakeholder’s preference precedence. The Analytical 

Hierarchy Process (AHP), developed by Saaty (1980; 1994) is one of the most commonly used 

ranking methods in industries. It can handle both qualitative and quantitative attributes. 

SMS with multiple objectives related to several indicators would benefit from multi-criteria 

decision making methods. An example of MCDM in manufacturing is the selection of a process 

plan and machine settings for manufacturing a machined product. Performance indicators are 

often conflicting such that improving one (e.g., energy consumption) diminishes the performance 

of another (e.g., cutting tool life). Given that, in a typical job shop, there are alternative resources 

that can generate the same feature on a part, specifying the best set of resources (i.e., the optimal 

selection of process plan) is the main objective problem. The resources and corresponding 

processes can be specified in an operation method matrix. Before determining the process plan 

that has the optimal impact on environmental sustainability and productivity, a range of process 

plan parameters (process settings) has to be assigned. MCDM through multi-objective 

optimization can address this problem.

3.3.4 Computer simulation

Simulation provides the ability to model, analyze, and iteratively test manufacturing systems in a 

virtual environment to optimize plans before implementation (after which failures become

costlier). The main methods for simulation are system dynamics, discrete event simulation, 

virtual reality modeling, agent-based simulation, and Monte Carlo simulation. A simulation 

model can incorporate production schedules, priorities, equipment capabilities, layout, costs, and 

other factors. Different simulation models can be built to analyze a system depending on the 

stage in the system life cycle. Simulation models built for manufacturing system concept 

development and design traditionally differ from models for operational decision-making. Here 

we examine these models in turn; however, in SMS these distinctions may begin to blur as more 

real-time data becomes available.
21

An early application area for computer simulation in manufacturing decision-making was the 

general domain of production system design. The results of running simulation models are useful

to plan for long-term demand and demand variations to determine plant capacity requirements, 

and inventory levels. The main goal identified by Košturiak et al. (1999) is that in a production 

system design, the planned and actual requirements and resources are correlated as much as 

possible. To facilitate this undertaking, Chomiakow (2007) used system dynamics to develop a 

generic pattern for modeling manufacturing plants that includes factors such as production 

capacity and inventory, demand and sales, costs, and sales price. The generic pattern can be

adapted to different systems and operating environments. 

The design and development of a manufacturing system is preceded by the discovery of general 

production requirements through the use of system dynamics models. The output of these models 

can form the input to discrete-event simulation models for decision making regarding process 

design, resources selection, and layout. Negahban and Smith (2014) carried out a survey which 

showed that the major concerns investigated by discrete-event simulation during systems design 

are production rates, general system layout, material handling, cell systems, and automation 

equipment. At the process level, virtual reality modeling facilitates workstation design and 

human task performance analysis. 

Simulation is often criticized for not directly providing optimal solutions. Nonetheless, it is one 

of the most widely applied methods for assuring manufacturing performance (Shipp et al. 2012; 

Bryner 2012). Simulation enables data input at fine levels of detail and limitless configurations, 

sometimes referred to as “what if” scenarios, to be considered. Simulation can be used to 

generate and fill gaps in missing data so that a manufacturing system can be analyzed by other 

methods. It can provide system operation visualization capabilities to allow better understanding 

of component interactions and performance. Simulation applications are projected to have wider 

applications in SMS. For example, Shao et al. (2014) demonstrated how simulation can be used 

to generate data to evaluate manufacturing data analytics applications. For this approach to be 

effective in real application, however, data-generating models require improved verification and 

validation methods. Other potential applications include use of simulation in the loop for 

continuous SMS performance improvement and for filling gaps in system-collected data.

Despite great potential for SMS, in their report on role of simulation for future manufacturing,

McLean and Leong (2000) observed major challenges for wide-spread application of this 

technology. These are (i) cost, which is mainly driven by requirement for modeling expertise and 

cost of software acquisition, and (ii) data interface problems, which are due to limited 

interoperability between manufacturing applications and simulation tools. Other problems are the 

burden of having to produce each new simulation from scratch, the need for a diverse set of 

skills, verification and validation of models, data availability, and interpretations of simulation 

outputs. In addition, simulation models may need high levels of fidelity to be of use in SMS. 
22

Since that publication (McLean and Leong 2000), there have been efforts to address some of the 

above challenges, although the solutions are still limited in scope. For example, simulation tools 

are providing more visualization capabilities, which allow the user to manipulate objects and 

optimize a design in a virtual environment. Such tools enable the creation of a virtual enterprise 

that can be operated side by side with the real system to optimize operation in real time. 

Secondly, simulation tools are becoming more readily available and easier to learn. Regarding 

integration of tools, many simulation applications are still not easy or possible to integrate if 

obtained from different vendors. However, efforts have been initiated for integration through 

developing standards for neutral data representations. Examples are the Core Manufacturing 

Simulation Data (CMSD) and the OAGIS standards. 

The CMSD standard was developed so that data can be shared between simulations and 

manufacturing applications (SISO 2011). CMSD can facilitate exchanging data across different 

simulation models in a supply chain or between departments. A limitation is that it does not 

support integrating simulations along a vertical scale, i.e., across hierarchical levels. Another 

standard for data interoperability is the OAGIS (OAGi 2014), from the Open Applications 

Group. OAGIS approaches the integration problem by establishing integration scenarios for a set 

of applications including enterprise resource planning, production scheduling, manufacturing 

execution system, and capacity analysis. The main emphasis of the OAGIS standard is at the 

enterprise level. There is also the ISA-95 standard, which defines interfaces between enterprise 

activities and shop floor activities (ANSI/ISA 2005). This standard is oriented primarily towards

the operations level. There remain gaps in these standards for modeling the exchange of data 

between and amongst different manufacturing applications, models, and simulation tools. 

3.4 Emerging methods for SMS performance analysis

Systems engineering is a discipline expected to play a big role in SMS. Systems engineering uses 

requirements analysis to develop functional and physical definitions as a basis for systems and 

product designs (Kossiakoff et al. 2011). The Object Management Group standardized SysML as

a language for representing system designs for more widespread deployment of system 

engineering by creating a market for vendor support of SysML-based tools (OMG 2015). SysML 

is a diagramming method based on the Unified Modeling Language (UML) and is specifically 

tailored to the needs of system representation. SysML supports the coupling of system 

requirements with the system design. A large advantage in this approach is that requirements for 

a system can be managed along with the system. Changes to either can be automatically 

processed to identify the implications for the other. SysML also facilitates system deployment in 

automating the development of tests of an implemented system based on the planned 

requirements. Fundamentally, SysML supports a style of design known as Model-based System 

Engineering where the model of a system becomes the master of the system as opposed to a 

more traditional document-centric approach (MBSE 2015).
23

As already observed, manufacturing system operation needs decision making to achieve and 

maintain desired performance. Such decisions have to be made using different types of models 

and large amounts of dynamically collected data. These models, including operations research, 

data analytics, and machine learning, need unification so that the user does not have to generate 

inputs multiple times using different data abstractions. Decision Guidance Management System 

(DGMS) is an example of a platform that provides such unification. DGMS is a productivity 

platform for fast development of applications that require closed-loop data acquisition, learning, 

prediction, and decision optimization (Brodsky 2008). Brodsky and other researchers at George 

Mason University conceived DGMS to support closed-loop data acquisition learning, prediction,

and decision optimization by using a decision-guidance database and a productivity tools’

platform for fast development of decision-guidance databases. Decision guidance management 

systems (DGMS) support what-if analysis, monitoring and control, statistical learning, and 

decision optimization using the Decision Guidance Query Language. Application of DGMS to 

SMS is still in the research stage but its potential has been demonstrated for decision making to 

minimize energy consumption in machining operations (Shao et al. 2011).

3.5 Performance improvement methods

Performance improvement involves analyzing the operations of an organization and setting up 

approaches to improve performance. A widely used performance improvement method is PlanDo-Check-Act (PDCA) (Imai 1986). PDCA is a four-step cycle for affecting positive change and 

has been applied in various areas of performance improvement (Moen et al. 2011). Performance 

improvement methods such as Lean manufacturing, Six Sigma, and Total Quality Management 

that are discussed later in this section were conceived and are implemented on the basis of the 

PDCA cycle. The PDCA cycle emphasizes that even after improvements have been identified 

and implemented, further improvements are sought. PDCA is now the foundation of all ISO 

management standards (ISO 2012). The PDCA approach is to plan, implement, check for 

improvements, and act by revising plans to achieve better results, in a continuous cycle. PDCA 

can be used at any stage of SMS life cycle and can be integrated with other methods. In the 

following, we review derivatives of the PDCA cycle in manufacturing and provide an 

introduction to real-time performance improvement approaches.

3.5.1 Derivatives of the PDCA cycle

Deming (1986) introduced the Plan-Do-Study-Act (PDSA) cycle, which is a modification of the 

PDCA. In PDSA “study” replaces “check”. PDSA also involves asking questions such as “what 

did we learn?” and “what went wrong?” It is considered to be more effective than PDCA. 

Langley, Nolan, and Nolan (1996) modified the PDSA cycle by appending three activities and 

produced a “model for improvement.” The activities added are: defining an acceptable level of 

performance, feeding back information to learn whether the change is an improvement, and 

specifying the changes necessary to result in improved performance. Another derivative is the 

Define-Measure-Analyze-Design-Verify (DMADV), which is sometimes called Design for Six 
24

Sigma (DFSS). DFSS is useful for designing and testing (Cronemyr 2007). There is also the 

Design-Measure-Analyze-Improve-Control (DMAIC) (de Mast et al. 2012), which stresses 

means for identifying root cause of problems and developing solutions. The solutions to the 

problems can be generic or specific to the given problem. Although many of these methods were 

developed for quality improvement, they equally apply to other performance objectives. They are 

also often used hand-in-hand with lean techniques for eliminating waste in manufacturing.

3.5.2 Lean manufacturing, Six Sigma, and total quality management 

Lean manufacturing, Six Sigma, and Total Quality Management are among the common 

performance improvement methods used in manufacturing. The following paragraphs overview 

these methods.

Lean manufacturing is a systematic process for eliminating waste within manufacturing 

processes (Holweg 2007). Waste is any activity that does not add value to the final product. 

Lean manufacturing emerged from the Toyota Production System (TPS), which aims to 

minimize work-in-progress inventories in the manufacturing process. TPS uses Just-in-Time 

tools. The goal of lean manufacturing is broader than TPS as it focusses on systematic 

elimination of all forms of waste from the operations (Yang et al. 2011). One major method to 

implement lean manufacturing is to use the PDCA cycle where waste reduction is performed on 

a continuous, iterative, basis. Tools have been developed to identify waste, measure waste, and 

eliminate waste in manufacturing systems.

Six-Sigma is a philosophy that emphasizes the idea of getting to the root cause of quality 

problems to improve the process (Schroeder et al. 2008). It achieves this by a process of 

continuous improvements such as the PDCA cycle (Andersson et al. 2006). Six-Sigma was 

originally developed by Motorola during the 1980s. Six Sigma techniques and tools aim to have

only 3.4 defective parts per million cases. Improving a process inevitably incurs a cost to 

implement and the decision for improvement of each process in the manufacturing system 

depends on the strategic importance of the process as well as the benefits expected.

Total Quality Management (TQM) focuses on achieving objectives to provide products and 

services with a level of quality that the customer wants (Oakland 2014). The approach to 

achieving the objectives is employee involvement and developing an integrated systematic 

approach for continuous improvement. TQM evolved out of the work done by Edwards Deming 

to improve quality of Japanese-manufactured products. When American manufacturers started 

losing the market to Japanese products, they decided to employ TQM techniques starting in the 

1980s. Ford Motor Company, under Chief executive officer Don Petersen, was one of the first 

major manufacturers to implement TQM in the U.S. and almost instantly its market share 

rebounded and, for a while, overtook General Motors. There are twelve points to implement

TQM: committed leadership, adoption and communication of TQM, closer customer 
25

relationships, closer supplier relationships, benchmarking, increased training, open organization, 

employee empowerment, zero-defects mentality, flexible manufacturing, process improvement, 

and measurement.

3.5.3 Using real-time data analytics for continuous performance improvement

Continuous improvement refers to ongoing efforts to improve products or processes. 

Improvement can occur in a “breakthrough” event all at one time or be in the form of

“incremental innovation” over time (Sadikoglu et al. 2010). Incremental innovation is the action 

of performing a series of small improvements in the product or production process. Incremental 

innovation implies real changes in the product design as required by consumers or changes in the 

manufacturing process due to, for example, advances in technology (Abernathy et al. 1978). The 

concept of continuous improvement has existed in manufacturing since the beginning of the 

industrial revolution. The techniques that have been discussed in this section (Section 3) are part 

of the activities that drive an organization towards “continuous improvement,” in general. An 

example framework for continuous performance improvement of manufacturing systems was 

proposed by Hernandez-Matias et al. (2008). This framework increases the capability of 

modeling tools to model systems by creating a structured database with different levels of detail 

to obtain KPIs. The following paragraphs discuss how data from a manufacturing process can be 

harnessed for real-time continuous improvement of manufacturing systems.

Many engineered systems are designed with clear objectives and are usually operated under 

controlled conditions; therefore, performance improvement activities based on mechanistic 

models and first principles can be reliably applied (Sharma et al. 2014). During operation, 

however, circumstances emerge that were not factored into the design. In such cases, data 

becomes an indispensable asset for improvement. The large amount of constantly streaming data 

seen today has created a need to increase pace in the advancement of data analytics methods. 

Many new approaches are under development to use state-of-the-art real-time data-processing 

technologies recently available from the information technology community (LaValle et al. 

2010). Among these are the real-time analytical systems being looked at to augment performance 

improvement activities of enterprises. Driven by the motivation to increase adaptability and 

responsiveness of SMS, new capabilities are being developed for analytics’ solutions to process 

large amounts of data to develop actionable intelligence and may even be introduced into 

automated controls. 

A McKinsey report defined four types of data analytics important to manufacturing: data 

visualization, correlation analysis, significance testing, and neural networks (Auschitzky et al. 

2014). The report describes the application of these techniques to identify patterns to prioritize 

data collection and analysis, identify core determinants of performance deviations, test the 

significance of variables in root cause analysis of problems, and model complex processes for 

optimization. These types of analysis can be applied in a manufacturing environment to improve 
26

performance at all levels of the business from machine performance issues through higher-level 

enterprise optimizations. For example, at the machine level, Park et al. (2015) described a datadriven method for energy prediction for a milling machine that would help make design 

decisions based on energy demands. At the enterprise level, Song et al. (2014) used data 

analytics to identify system bottlenecks and improve the design of system flow. 

To realize the potential of big data for performance improvement of an organization, 

understanding the relationship between proper analytical techniques and disparate data sources to 

extract usable insight is currently under investigation, but has yet to see widespread application. 

Creating fully deployable analytics solutions is a complex process requiring skills from many 

disparate areas. To successfully build a real-time analytical solution for a manufacturing 

organization requires rethinking and reengineering operational processes and data collection, 

storage, and analysis (Qin, 2014). Advanced tools, software and systems are required to capture, 

store, manage and analyze data sets, all in a timeframe that preserves the intrinsic value of data 

(Huang et al. 2014). Kumaraguru et al. (2014) identified the need for integrating such data 

analytical tools into a continuous performance management cycle for SMS. In view of this, 

focused research efforts are needed to link the foundational elements of SMS with operational 

and enabling elements of such systems (Zhao et al. 2014). 

Continuous performance improvement methods frequently employ data-driven prediction 

models. Bridging the gap between predicted performance and actual performance will need 

diagnostic and problem-resolution capabilities. Continuous monitoring of metrics and indicators 

at different strategic, tactical, and operational levels in real-time will enable drilling-down to root 

causes of performance problems. The next section provides a sample of software tools that are 

currently available for this purpose.

As we close discussion on data analytics, it is pertinent to mention a new technique similar to 

data analytics that has potential benefit and performance improvement. It is called process 

mining. Process mining focusses on analyzing information from event logs produced by 

manufacturing and business processes to plan, analyze, and optimize operations in a dynamic 

environment (Pospišíl et al. 2013). Process mining has already proved feasible and effective in 

identifying areas in processes that need improvement. Yahya (2014) provides existing industrial 

works on process analysis in manufacturing using process mining.

3.6 Software tools

Many commercial platforms, tools, and performance management systems have been developed 

for specific evaluation, analysis, and improvement of manufacturing systems. Table 3 provides a 

sample of methods and software categories. Many vendors also provide solutions with abilities to 

integrate data from different organizational departments and subsystems. While Table 3 samples 

from categories of methods discussed, many tools are marketed for functions such as bill-of-
27

materials processing, planning and scheduling, order promising, demand forecasting, and 

inventory management. MESA has compiled a list of software vendors from among its members

(https://services.mesa.org/member). To analyze these tools provided by various vendors, a 

classification or categorization of the tools, possibly based on their application, is needed. One 

example is the identification of tools needed in each phase of a manufacturing system life cycle 

that includes conceptual design, detailed design, testing, improvement, and operation. 

Alternatively, software tools can be analyzed after classifying their applications with respect to 

performance objectives of SMS. An example of a performance objective is sustainability. 

Sustainability relies heavily on life cycle assessment, which is a technique to assess the 

environmental and potential impacts of a product, process, or service. 

Table 3: Example software tools from each category

Category Method Purpose 

Performance 

measurement

Learning, statistical, 

various analyses

Implementing a nine-step metric assessment, action 

planning, and performance collaboration 

Performance 

analysis

Statistical method Addressing operational readiness, equipment 

monitoring, maintenance, and system effectiveness 

and improvement

Machine learning Inductive inference

Operations research Optimization

Simulation Manufacturing process Simulation 

Data analytics Predictive analytics

Process mining Extracting knowledge from event logs

Performance 

improvement 

Lean Six Sigma Lean process improvement 

Total quality management Automating processes, integrating business 

systems, and fostering collaboration and 

continuous improvement

4 APPLYING PERFORMANCE MANAGEMENT METHODS TO SMS 

This section discusses development of an SMS performance-assurance system based on the 

methods discussed. It first reviews today’s manufacturing performance objectives at a higher 

level as discussed in the introductory chapter. These are agility, which is driven by the need for 

rapid adaptability of manufacturing processes in response to customer needs and internal 

impediments; sustainability, which is driven by an increased global awareness of the impact of 

human activity on the natural environment; and, productivity, which is a business driver for 

maximizing returns on investments. We then discuss sources of and challenges to overcoming 
28

performance exceptions and data collection. This section ends with proposed general steps for 

performance assurance methodology.

4.1 Targeted performance objective for smart manufacturing – Agility 

A central theme for performance of SMS is agility because of the need to respond to 

unpredictable events that are a feature of today’s manufacturing environment. The high-level 

focus to achieve agility is on customer satisfaction, learning, and willingness to change. To make 

manufacturing systems more agile, solutions such as modular organizations, flexiblemanufacturing systems, lean manufacturing, holonic-manufacturing systems, and virtual 

corporations have been proposed. 

Literature review shows that many researchers have investigated agile performance of 

manufacturing systems over the years (Gunasekaran 1999; Ramasesh et al. 2001; Tsourveloudis 

et al. 2002; Sherehiy et al. 2007; Vinodh et al. 2008). Each researcher analyzed particular 

attributes of agility, mainly at two organizational levels: enterprise and manufacturing. In a 

major review of performance measurements, Hon (2005) perceived agility from only the 

perspective of flexibility and responsiveness but did not define agility measures. Sherehiy et al. 

(2007) carried it further and added responsiveness, culture of change, speed, integration and low 

complexity, high quality and customized products, and mobilization of core competencies. 

A reference model that addresses agility is the Supply Chain Operations Reference (SCOR), 

which was developed to describe business activities associated with all phases of satisfying a 

customer's demand (SCC 2012). The SCOR model identifies performance attributes for 

reliability, responsiveness, agility, costs, and asset management efficiency. SCOR defines agility 

as the “ability to respond to market place changes to gain or maintain competitive advantage.”

Although this perspective has traditionally viewed agility as a capability to effectively respond to 

events that are external to the organization, a recent trend has been to include internal events as 

well. SCOR defines non-computable agility metrics for flexibility and adaptability. While the 

model specifies computable metrics for the other performance attributes, i.e., reliability, 

responsiveness, costs, and asset management, it does not do the same for flexibility and 

adaptability, even though these attributes would enable defining the necessary specific 

measurements for agility. However, the SCOR model identifies business processes for which 

agility measures can be specified.

To design an agile manufacturing system, it is necessary to focus on essential characteristics that 

affect agility, e.g., changeover time, product variety, and versatility. Modeling methods and tools 

(e.g., simulation) can be used to select and investigate agile behavior (i.e., agility performance) 

of a particular configuration of the manufacturing system when subjected to disturbances and 

disruptions. A trade-off can then be made between the cost of the required system components 

and the desired system performance. Ramasesh et al. (2001) applied this simulation approach 
29

and developed a modeling methodology for appraising a system’s performance and justifying 

decisions by exploring the financial value of investing in technologies that facilitate defined 

agility characteristics in manufacturing systems.

4.2 Targeted performance objective for smart manufacturing – Sustainability 

There are various ways and levels of assessing the sustainability of manufacturing systems, 

processes, and products. One of the most commonly used methods is Life Cycle Assessment 

(LCA); however, the approach is mainly product-centric in that the assessment treats the 

manufacturing stage of a product as a single transformation event. As such, the majority of 

research efforts for manufacturing sustainability have not particularly focused on process 

assessment. Process level assessment requires methods and tools for modeling the operation of 

the production process to analyze the inputs, transformation process, outputs, and its

environmental impacts such as energy consumption. Such methods use manufacturing data 

associated with material and manufacturing process to quantify the inputs, outputs, and byproducts. This approach enables aggregation of impacts and comparison of alternative processes.

In this section, we overview tools, standards, and life cycle data. We also discuss how 

sustainable manufacturing practices can be integrated into continuous improvement efforts.

4.2.1 Standards for sustainable manufacturing

Over the last few decades, many standards have been developed so that manufacturing 

companies can steer their operations towards sustainable practices. Candidate standards for 

process assessment include ISO 20140-1:2013, Automation Systems and Integration, which 

provides a framework for assessment of environmental influence of manufacturing processes 

from a work unit to the entire factory (ISO 2013). ISO 20140-1:2013 is for automation systems 

and integration and defines the requirements for the environmental-influence data to be captured 

from individual manufacturing equipment.

Other standards include BS 8905, which provides a framework for the concepts, techniques, 

tools, and methodologies that can be used to support decisions regarding the sustainable use of 

materials including use in the manufacturing process, i.e., conversion of materials into products 

(BSI 2011). There are also standards for assessing particular products such as electronics (IEEE 

2013; IEC 2015), plastics (ANSI 2010), or office equipment (ANSI 2007). These product 

standards are used to analyze the environmental impact of a product during its entire life cycle 

based on given impact categories. 

Although best-known LCA standards are in the ISO 14000 series standards, they are not 

specifically suitable for manufacturing processes as they were developed for the entire product 

life cycle.
30

To focus on sustainability in manufacturing, standards are needed that assess material

consumption, energy usage, and waste in associated manufacturing processes and operations. 

ASTM International is in the process of developing standards for sustainable manufacturing 

assessment to complement already-existing standards such as ISO 20140 (ISO 2013), ISO 50001 

(ISO 2011) and ULE 880 (ULE 2010). We cite two of these standards: ASTM WK35702 and 

ASTM WK35705 (ASTM International). ASTM WK35702, Guide for Evaluation of 

Environmental Aspects of Sustainability of Manufacturing Processes, is targeted for gate-to-gate 

sustainability evaluation of the manufacturing process. This standard will provide a procedure 

and a guide for sustainability performance of manufacturing processes. The second is the ASTM 

WK35705, Guide for sustainability characterization of manufacturing processes. These two 

standards will support the calculation of the sustainability performance of a specific process and 

thereby provide a basis for assessing and improving the sustainability of manufacturing systems.

4.2.2 Reference data

Reference data is needed to consistently quantify resource consumption and environmental 

impacts of manufacturing processes, products, and services. This data is needed both during 

design and operation of a manufacturing system. An example of needed data is life cycle 

inventory (LCI) data. LCI is the amount of resources and ways they are used during 

manufacture of a product, as well as residual materials generated during processing, and scrap at 

the end of a product’s life. In an effort to avail this data, the UPLCI1consortium at Wichita State 

University, among others, is developing and compiling life cycle data from other researchers and 

manufacturers (Overcash et al. 2012). There is also the CO2PE!2Project, which has, through 

collaborative effort, developed a methodology for availing high-quality LCI data for assessment 

and in-depth analysis of individual manufacturing unit processes (Kellens et al. 2012).

4.2.3 Sustainable practices and continuous improvement

Sustainable practices may affect traditional performance since optimum operating parameters of 

a system for both performance objectives may not be the same. Walker (2012) proposed the 

enhancement of performance analysis systems to include environmental sustainability. We also

note that some of the objectives of sustainable manufacturing are the same as those of 

performance improvement methods. Although performance improvement methods such as lean 

manufacturing, Six Sigma, and TQM do not directly address sustainability, these strategies can 

improve sustainability performance by reducing waste and lowering resource consumption 

(Herrmann et al. 2008). These strategies can purposely be aligned to become part of the 

organization’s sustainable manufacturing strategy (Wilson 2010). Smart manufacturing planning 

can include these strategies to simultaneously achieve objectives of process efficiency and 

sustainability without modifications to their underlying principles.

 

1 Unit Process Life Cycle Inventory (http://cratel.wichita.edu/uplci/)

2 Cooperative Effort on Process Emission in Manufacturing (http://www.co2pe.org/?Homepage)

.
31

4.3 Targeted performance objective for smart manufacturing – Productivity 

Traditionally, productivity is defined as the measure of output compared with a given input. 

Examples of inputs are labor hours, machine hours, and material. Productivity can be measured 

at different levels of the organizational hierarchy from a single machine to the entire 

organization. Productivity is distinguished from commonly used performance objectives such as 

return-on-investment (ROI), which is a cost-based measure often used at the highest level in the 

organization. Analytical tools such as simulation and statistical analysis play a role in analyzing 

productivity through examination of their output reports. Advanced statistics can also analyze 

historical process data, identify patterns and relationships among discrete process steps and 

inputs, and optimize factors with the greatest effect on yield and productivity (Auschitzky et al.

2014). With SMS, the shift is to use real-time shop-floor data and to develop advanced analytics 

to track productivity performance and improve operations in real time.

With the need for proactive or forward-looking measurements, some researchers believe that 

new metrics are needed for productivity since traditional metrics such as utilization rate are not 

helpful for isolating problem causes and identifying opportunities for improvement (Huang et al. 

2003). Cesarotti et al. (2013) proposed overall equipment effectiveness (OEE) as a quantitative 

measure of productivity of production equipment in industry. OEE has been used to monitor and 

control production and can be used to enhance availability, performance, and quality by using 

smart technologies. Muchiri et al. (2008) reviewed the application of OEE as a measure of 

manufacturing performance. 

In addition to a new outlook towards productivity measurement for smart manufacturing, 

different strategic objectives and operating situations may require different measures. Herron et 

al. (2006) developed a methodology for quantifiable productivity improvement that starts with a 

productivity needs analysis, to give an overview of the current manufacturing situation and

identify key productivity measures. Since productivity is highly linked with equipment 

availability, a manufacturing system can be designed for high productivity by selecting 

manufacturing equipment that is easily maintained so as to minimize any possible downtime 

during operations. This would be complemented by making asset status an essential part of the 

company information system. Many productivity factors can be measured only during operation 

of the system through tracking, monitoring, predictive analytics, and control. In this regard, 

SMS will need a framework to define productivity measures relevant to the industry and 

operating conditions.

Productivity metrics for SMS aside, Davis et al. (2012) pointed out that performance objectives 

focusing on productivity (input/output) should be replaced by those that target improving 

customization, flexibility, responsiveness, energy efficiency, and environmental effectiveness. 

Cesarotti et al. (2013) proposed additional measures including capacity utilization, saturation, 

quality, and availability.
32

4.4 Challenges to overcoming operational exceptions

An important decision concerning performance measurement is when to measure. Traditionally, 

performance measurements are conducted periodically to detect performance deviations, as well 

as spontaneously when an event that can potentially lead to performance deterioration takes 

place. Long-term and short-term performance goals determine the planned period between 

consecutive performance measurements, whereas measurements triggered by operational 

exceptions are rarely predictably scheduled (Bruccoreli et al. 2005). A sample of operational 

exceptions, described as disruptions and disturbances, is shown in Table 4. The consequence of 

disruptions and disturbances is deterioration in system performance. Two of such events that 

occur with high frequency in manufacturing operations are hereby selected for further 

discussion. These are (i) system failures, including failure of equipment, tools, sensors, and 

control systems; and (ii) customer order reprioritizations including rush orders, changes in 

priorities, changes in part mix, and order cancellations. The approaches to reduce the impact of 

these types of events are discussed in the next two paragraphs.

Two strategies to minimize the impact of system failures on performance are periodic 

performance monitoring and preventive maintenance scheduling. Preventive maintenance is 

typically scheduled based on statistical distributions of past data. This preordained scheduling 

strategy may not be optimal. This traditional approach is termed preventive-maintenance 

forecasting. On the other hand, periodic performance monitoring is a maintenance practice based 

on predictive analytics that relies on monitoring of performance metrics and continuous 

evaluation of operations. In predictive analytics, machine learning techniques are employed and 

system failures are predicted based on patterns of data in routinely monitored performance 

metrics. By identifying performance deterioration patterns in real-time, preventive maintenance 

can be scheduled only when the failure is imminent, consequently cutting down unnecessary 

preordained scheduled maintenance as in the case of forecasting. This strategy may offer a more 

optimal solution. Determining what to measure so as to predict maintenance needs is an active 

research area and significant progress is being made as a result of ubiquitous sensors 

(https://www.phmsociety.org; Cheng et al. 2010). 

Customer order reprioritization is a production planning and control problem. Typically, a master 

production schedule is developed from firm customer orders, demand forecasts, and resources

availability. The arrival of a new, higher-priority, order or change in an existing order creates a 

“disturbance” that invalidates the existing schedule by requiring production reprioritization. 

Changes in a schedule often impact cycle times and thus invalidate already-promised due dates. 

In addition, customer orders often arrive with unpredictable frequency. Reprioritizing orders and 

other activities has to rely on information to dynamically integrate manufacturing needs through 

the production system. Appropriate response to changing production requirements and priorities 

requires multi-level integrated production planning, modeling, and monitoring. Research work is 

needed in this area.
33

Table 4: Operating characteristics requiring agile manufacturing systems

Internal External

Disturbances Labor shortage

Deterioration in quality output

Changing labor performance levels

New equipment installation 

New software installation 

Product quality deterioration

Changed product routing

New product introduction

Rush order 

Order cancellation 

Order change

Product returns

Customer order reprioritization 

Raw material unavailability

Changing regulations

Disruptions System failure

Scheduled maintenance

New production line configuration

Changed product line

Natural disasters, e.g., bad weather 

Power or water outages 

Delayed material supply

Poor or wrong material supply

4.5 Challenges with collection and use of data for performance improvement

Kennerley and Neely (2003) classified factors affecting the evolution of performance 

measurement systems under process, people, systems, and culture. Further reflecting upon the 

systems perspective, they suggested that the evolution of performance management systems will 

vitally depend on flexible IT systems to effectively and efficiently acquire, collate, sort, analyze, 

interpret, and disseminate data. Since then, there has been an explosion in the availability of data 

for manufacturing enterprises. The major source of data is the manufacturing system shop floor 

using smart devices.

There are inherent challenges in making sense of all this data at each level of the DIKW pyramid 

(Figure 3). At the data level, there are challenges of collecting and using data, sometimes called 

the 7 Vs of big data—large volume, high velocity, large variety, uncertain veracity, uncertain 

validity, volatility (timeliness), and value (Khan et al. 2014). Structured Query Language 

(ISO/IEC 2008) is one of the means of storing and retrieving the data. In addition to devices, 

automated data collection needs communication interfaces for obtaining data directly from 

machine controllers or a collection device developed for the machine. The data are 

communicated using a neutral format such as XML.

At the information level, challenges are associating data with relevant performance question. In 

addition to collected operational data, there is other data from disparate functional units involved 

in product design, manufacturing planning, enterprise planning, and human-machine interfaces. 

This includes resources data, inventory data, manufacturing process data, life cycle inventory 

data, and product data. External sources such as customer orders and partners in the production 

network also generate data. This type of large-scale data collection and transformation poses data 

compatibility and interoperability issues. Most systems today are not designed to integrate at 

such a scale.
34

At the knowledge level, is the challenge of providing relevant, compelling, views of information 

for different users. The ability to deliver such views pervasively across the organization will 

provide decision guidance. At this level, the challenges lie in the ability to correlate the KPIs to 

operating parameters and to create a plan for making appropriate adjustments to meet 

performance targets. 

4.6 Methodology steps for performance assurance for SMS

This is a summary of an approach for performance assurance of SMS. The steps in this approach 

are supported by the methods and techniques reviewed. The steps (not necessarily in the order 

provided) are:

• Set strategic goals

• Establish key performance indicators (KPIs)

• Define SMART (specific, measurable, actionable, realistic, time-based) metrics that 

relate to the goals

• Establish and set performance benchmarks

• Test the system over a performance envelope

• Measure performance against benchmarks

• Trace the source(s) or causes(s) of problems

• Take actions that will result in desired performance. 

Setting strategic goals: These are the long-term achievement targets of an organization, and 

clearly articulating them is a crucial responsibility of management. These goals, in turn, drive 

definition of the indicators and measures that track performance of the manufacturing system. 

The complex challenges of SMS require an organization to set high-level goals and targets that 

balance system responsiveness and adaptability with productivity and sustainability needs.

Establishing KPIs: This is the process of specifying KPIs that should be emphasized. KPIs that 

define SMS performance are mainly related to responsiveness to changes of demand, 

environmental sustainability, and high return on investments. Other KPIs can be defined

depending on such factors as type of industry, geography, production costs, and regulations.

Choosing SMART metrics and perform measurements: This is the activity to specify computable 

metrics that track performance according to the KPIs. During system development, the data used 

in the analysis is based on original equipment manufacturer specifications, as well as projections 

of system operation and external environments. During system operation, sensors, radiofrequency identification (RFID) of component parts, meters, and other instruments collect data in 

real time. Choosing appropriate measurements to carry out is instrumental to investing in the 

appropriate ICT hardware and software.
35

Testing the system: Testing includes forecasting the operation of a system to determine if 

required performance can be achieved. The process is often carried out in a virtual environment. 

A performance envelope is defined by setting a minimum and maximum range of desired 

performance according to the KPIs and metrics. Because of multiple KPIs, conflicts such as 

between cost and quality may happen. Typically, each KPI has bounds of acceptability. A 

measure of the trade-off between cost and quality can be productivity. Smart manufacturing also 

requires a performance envelope related to sustainability. Thus, if productivity and sustainability 

set the boundaries for performance, agility as defined by ability to respond to disruptions and 

disturbances has to be maintained within that performance envelope. 

Establishing performance benchmarks: Performance benchmarks are the desired performance 

levels of indicators and are often influenced by external drivers. There are different ways of 

setting performance benchmarks such as considering the best organizations or best-in-class 

performance. Other means of setting benchmarks include surveys, interviews, focus group 

meetings, and site visits. They may also be based on a standard or from publications.

Measuring performance: Performance measurement is a report on the state of the KPIs. 

Measurement frameworks have been discussed in previous sections. Performance evaluation 

compares obtained measures with benchmarks so as to look for deviations. Performance 

measurements require data: some is collected other data is stored in databases. Through

improvements in computing technology, data stored across many applications and databases, 

each with a specific functional role can now be better accessed from multiple network-linked 

locations. For example, material data can be accessed by more than one functional unit of the 

enterprise including procurement, product development, testing, manufacturing, and marketing. 

Tracing the source of performance problems: This is the activity that is carried out after noticing 

measurements that are inferior to benchmarks (or set targets) in one or more KPIs. The 

manufacturing processes and measurements relevant to the KPIs are investigated to determine 

the source of poor performance. Data analytics methods and techniques such as Pareto Analysis, 

Cause & effect analysis, and Failure mode effect analysis can be applied to distinguish multiple 

factors contributing to performance problems in a complex system.

Taking action to produce required performance: The actions needed to produce the desired 

performance are obtained from decision models based on performance analysis and improvement 

methods reviewed. Multi-objective optimization models are examples of such models. The 

models can be constructed and employed for decision making based on current state of the 

system, data analysis, and required future performance. The type of models and the input data 

depends on the type of system, performance indicator and operating environment, among other 

factors.
36

5 SUMMARY AND FUTURE RESEARCH DIRECTIONS

In this section, we brief high-level assurance needs for SMS, and propose future research 

directions.

5.1 Summary 

This paper has discussed how existing methods and tools can be enhanced by emerging 

technologies to measure, analyze, and improve the performance of SMS. Performance 

measurement frameworks were developed for enterprise level assessment. The availability of 

workstation, machine, and process data, and the need to assess performance at these levels will

require that these methods are also adapted to lower operational levels. Secondly, interoperability 

and communication will become more crucial if this data and shop floor performance analyses 

are to be used for enterprise level planning. Performance analysis methods such as computer 

simulation project system performance according to a defined set of inputs. The availability of 

large streaming data puts additional requirements on the modeling and analysis of systems. On 

performance improvement, it has been discussed that methods such as DMAIC provide a 

framework for selecting improved system components and configurations during systems design

as defined by performance objectives. 

The methods and tools reviewed can be used to build a system assurance methodology. Each 

method has strengths for specific, defined, purposes leading to better decision making. The 

presence of multiple KPIs for SMS calls for trade-off analysis to determine the best course of 

action. Further research work needed to apply reviewed methods is next discussed.

5.2 Addressing requirements for performance assurance for SMS

While automated equipment such as sensors, embedded systems, and networks collect and 

communicate performance data, they also introduce new forms of possible failure and increase 

the requirements for modeling and analyses. We summarize, at a high level, the challenges and 

needed research directions for measurement, analysis, and improvement of SMS installed with 

these systems. 

Performance measurement

Performance-measurement frameworks were originally developed for use at the enterprise level 

of the organization, using data collected only periodically. Measurement and assessment are now 

needed, however, continuously and in real time, requirements that are further complicated by the 

challenges of operation in a big-data environment. Realization of performance measurement for 

SMS requires research in the following areas:

• Identifying correct data elements: Accurate performance assessment of SMS cannot be 

accomplished without proper data sourcing, collection, and supply. Data analytics is a 

viable method to associate data type to required performance, but research is needed to 
37

improve this method to continuously update an analyzed data set, adding new elements 

while removing no-longer-needed ones.

• Assess in real time: The capability to consistently provide current-status models 

throughout an SMS does not yet exist. Research is needed, for example, to partition 

streaming data into data blocks that are continuously updated. Other needs include 

improved performance measurements that use efficient computational algorithms to 

perform assessments, and highly interoperable interfaces to assure distribution of those 

assessments.

Performance analysis

Performance analysis requires developing a mathematical, logical, abstract, or other type of

model of a system. The models for SMS need high levels of abstraction especially at the process 

level of the hierarchy. The larger number of indicators and metrics for SMS increase 

requirements for analysis, and that analysis is likely to be more complex, costly, and timeconsuming. To enable performance analysis for SMS, research work is needed in the following 

areas:

• Integrated analysis: Methods such as simulation, data analytics, and optimization can be 

combined to obtain better analysis results. To this end, research work is needed to 

develop better integration interfaces between and among analysis models and data

sources such as the shop floor and databases. 

• Dynamic optimization: This is the process of determining the maximum or minimum of 

an objective function over time. Research work will be needed to effectively combine 

real-time assessment methods with optimization.

Performance improvement

The activities for performance measurement and analysis are geared to achieving performance 

improvement. Performance improvement requires feedback from performance assessment so as 

to decide on an appropriate course of action. Major challenges are timely and accurate 

communication of performance information and the decision action that needs to be made. 

Therefore, research in the following areas is needed:

• Enhancement of performance improvement frameworks: Performance improvement 

frameworks such as DMADV or DMAIC rely on planning, assessment, and analysis. 

Research work should be directed to enhancing these frameworks to provide the timely 

information retrieval and sharing required by SMS.

• Adaption of approaches for performance improvement in other domains: Common 

weakness enumeration has been developed for assurance of computer software. This 

method identifies common known weaknesses in software including a discussion of 

solutions. If high-fidelity models can be developed for SMS, research can investigate, 

classify, and create a database of common failures in performance of SMS. While new 

situations emerge that we not factored into system design, those that are encountered as 

the system is operated can be regularly included in the database. This can help direct 
38

system testing efforts during development and determine proper response after

performance failure.

DISCLAIMER

No approval or endorsement of any commercial product by the National Institute of Standards 

and Technology is intended or implied. Certain commercial software systems are identified in 

this paper to facilitate understanding. Such identification does not imply that these software 

systems are necessarily the best available for the purpose.

ACKNOWLEDGMENT

The work described in this paper was partially funded through a Cooperative Agreement No. 

70NANB13H153 between Morgan State University and the National Institute of Standards and 

Technology. The work described was funded by the United States Government and is not subject 

to copyright.

6 REFERENCES

1. Abernathy, W.J., and Utterback, J.M. Patterns of industrial innovation, Technology Review,

80, 40-47.

2. Ademe, T., Ngeru, J., Hargrove, S.K. A discrete event simulation model for the redesign of 

a reconfigurable factory test-bed in manufacturing, MSU Internal Report, 2008.

3. AIAG (Automotive Industry Action Group). Press Release. Available online at: 

https://oldsite.aiag.org/staticcontent/press/releases/GENERAL/PR_ECM_Final.pdf (accessed on 

12/14/2015)

4. Andersson, R., Eriksson, H., and Torstensson, H. Similarities and differences between TQM, six 

sigma and lean. The TQM Magazine, 2006, 18, 282-296.

5. Ahmad, M.M., and Dhafr, N. Establishing and improving manufacturing performance measures, 

Robotics and Computer Integrated Manufacturing, 2002, 18, 171-176. 

6. ANSI 2007. BSR/GEI Electronic office equipment sustainability performance - 200x: A 

standard for the sustainability of electronic office equipment products and applications.

Available online on: 

http://publicaa.ansi.org/sites/apdl/Documents/Standards%20Action/2007%20PDFs/SAV383

4.pdf Accessed on 2/06/13.

7. ANSI 2010. Sustainability assessment for Plastics. Available online on: 

http://publicaa.ansi.org/sites/apdl/Documents/Standards%20Action/2010%20PDFs/SAV410

6.pdf. (Accessed on 01/14/2013).

8. ANSI/ISA 95. ANSI/ISA–95.00.03–2005 - Enterprise-Control System Integration: Part 3: Activity 

Models of Manufacturing Operations Management.

9. ASTM International, New Guide for evaluation of environmental aspects of manufacturing 

processes, Available via: http://www.astm.org/WorkItems/WK35702.htm, (Accessed on 

1/16/2015).
39

10. ASTM International, New Guide for Sustainability characterization of manufacturing 

processes, Available online at: http://www.astm.org/WorkItems/WK35705.htm., (Accessed 

on 1/16/2015).

11. Auschitzky, E., Hammer, M.C. and Rajagopaul, A. How big data can improve 

manufacturing, Insights and Publications, Available via: 

http://www.mckinsey.com/insights/operations/how_big_data_can_improve_manufacturing, 

Accessed on 9/12/2014.

12. Barabady, J. Production Assurance: Concepts, Implementation and Improvement, PhD Thesis, Lulea 

University of technology, Division of Operations and Maintenance Engineering, 2007.

13. Benbarka, A. Assessment of Manufacturing Performance Using Data Envelopment Analysis, 

Master of Science in Inter-disciplinary Studies, University of Texas at Arlington, 2007. 

14. Bessant, J., Caffyn, S., Gilbert, J., Harding, R., and Webb, S. Rediscovering continuous 

improvement, Technovation, 1994, 14: 17-29.

15. Bititci, U.S., Carrie, A.S., and McDevitt, L. Integrated performance measurement systems: a 

development guide, International Journal of Operations & Production Management, 1997, 

17, 522 – 534. 

16. Bourne, M., Neely, A., J. Mills, and K. Platts. 2003. Why some performance measurement initiatives 

fail: Lessons from the change management literature, International Journal of Business Performance 

Management, 5: 2003, 245-269.

17. Brodsky, A., and X. S. Wang. 2008. Decision-guidance management system (DGMS): Seamless 

integration of data acquisition, leaning, prediction, and optimization. In Proceedings of the 41st

Annual Hawaii International Conference on System Sciences (HICSS 2008), 71–81. Hawaii, U.S.A.

18. Bruccoleri, M., Renna, P., and Perrone, G. Reconfiguration: a key to handle exceptions and 

performance deteriorations in manufacturing operations. International Journal of 

Production Research, 43, 2005, 4125-4145.

19. Bryner, M. Smart Manufacturing: The Next Revolution, A CEP Preprint, AICHE. Available 

online at: http://www.aiche.org/sites/default/files/docs/embeddedpdf/1012_SmartManufacturing_Preprint.pdf, (Accessed 8/28/2013)

20. BSI (British Standards Institution). BS 8905 - British standard 2011: Framework for the assessment 

of the sustainable use of materials. Available online at 

http://shop.bsigroup.com/en/ProductDetail/?pid=000000000030216209, (Accessed on 8/28/2015)

21. Burcker, P. NP-Complete operations research problems and approximation algorithms, 

Zeitschrift fur Operations Research, 1979, 23, 73-94.

22. Cesarotti, V., Giuiusa, A., and Introna, V. Using Overall Equipment Effectiveness for Measuring 

Manufacturing Systems Design, Operations Management, 2013, 51-80. 

23. Charnes, A. Cooper, W.W., Lewin, A.Y., and Seiford, L.M. Data Envelopment Analysis: Theory, 

Methodology and Applications, 1995, Kluwer Publishers. Chen, K-Y. Performance measurement of 

implementing manufacturing execution system, Materials Science Forum, 2006, 505-507, 1117-

1122.

24. Chen, C.-C., Cheng, W.-Y. Customer-focused and product-line-based performance measurement, 

International Journal of Adv. Manufacturing Technology, 2007, 34, 1236-1245. 
40

25. Cheng, S., Azarian, M. H., and Pecht, M. G. Sensor Systems for Prognostics and Health 

Management, Sensors, 10, 5774-5797.

26. Chomiakow, D. A Generic Pattern for Modelling Manufacturing Companies. Available online at: 

http://www.systemdynamics.org/conferences/2007/proceed/papers/CHOMI428.pdf, (Accessed on 

8/16/2014).

27. Common Weakness Enumeration (CWE), A Community-Developed Dictionary of Software 

Weakness Types, 2014. Available online at: http://cwe.mitre.org (Accessed on 1/21/2015)

28. Cronemyr, P. DMAIC and DMADV – differences, similarities and synergies, International 

Journal of Six Sigma and Competitive Advantage, 2007, 3, 193-209. InderScience 

Publishers

29. Cross, K. F. and Lynch. R. The “SMART” way to define and sustain success. National 

Productivity Review, 1988-1989, 9, 23-33. 

30. Dale, B. G., van der Wiele, T., van Iwaarden J. Managing Quality, Fifth Edition, Blackwell 

Bublishing, 2007.

31. Davis, J., and Edgar, T. Smart manufacturing, manufacturing intelligence and demanddynamic performance, Conference: Foundation of Computer-Aided Process Operations, 

Available online at: https://smart-process-manufacturing.ucla.edu/presentations-andreports/spm-publications/FOCAPOSMLC2012v10.pdf (Accessed on 1/7/2014).

32. Davis, J., Edgar, T., Porter, J., Bernaden, J. Sarli, M. Smart manufacturing, manufacturing 

intelligence and demand-dynamics performance. Computers & Chemical Engineering, 2012,

47,145-156.

33. De Mast, J., and Lokkerbol,. J. Ana Analysis of the Six Sigma DMAIC Method from the Perspective 

of Problem Solving, International Journal of Production Economics, 2012, 604-614

34. Delorme, A. Statistical methods. Encyclopedia of Medical Device and Instrumentation, 

2006, 6, 240-264. Wiley-InterScience 

35. Deming, W.E. 1986.Out of the Crisis. MIT Press. Cambridge, MA, page 88 Digalwar, A.K., Metri, 

B.A. Performance measurement framework for world class manufacturing, International Journal of 

Applied Management and Technology, 2005, 3, 83-101.

36. Dieter, G. E. and Schmidt, L. C. Engineering Design, Fifth Edition, McGraw-Hill, 2013.

37. Doran, G. T. There's a S.M.A.R.T. way to write management's goals and objectives. Management 

Review, 1981, 70, 35-36.

38. Engau, A. Overview and Opportunities of Operations Research (OR/MS) in Sustainability 

and the Environment, Available online at: 

http://www.math.ucdenver.edu/~aengau/presentations/120207_csis_sustainability.pdf

(Accessed 5/14/2014)

39. FactoryTalk VantagePoint EMI, available on line at: 

http://www.rockwellautomation.com/rockwellsoftware/performance/vantagepoint/overview.page

(accessed on 3/4/2014)

40. Feng, S.C., Joung, C.B. Development overview of sustainable manufacturing metrics, Proceedings 

of the 17th CIRP International Conference on Life Cycle Engineering, 2010, May 19-21,Heifei, 

China 
41

41. Ghalayini, A. M., Noble, J.S., and Crowe, T. J. An integrated dynamic performance 

measurement system for improving manufacturing competitiveness, International Journal of 

Production Economics, 48, 207-225.

42. Gomes, C. F., Yasin, M.M, Lisboa, J. V. A literature review of manufacturing performance measures 

and measurement in an organizational context: a framework and direction for future research, 

Journal of Manufacturing Technology Management, 2004, 15(6), 511-530.

43. Hallgren, M. and Olhager, J. Lean and agile manufacturing: external and internal drivers and 

performance outcomes, International Journal of Operations and Production Management, 2009, 29, 

976-999.

44. Halvorsen, L. A Comprehensive model for manufacturing analytics, Presented at the WBF North 

American Conference, 5-8 March 2006, Atlanta, GA. Available online at: 

http://www.controlglobal.com/assets/Media/MediaManager/wp_06_041_wbf_halvorsen.pdf

(Accessed on 10/20/2014)

45. Heger, D. A. An introduction to operations research – benefits, methods and application, Fortuitous 

Online Database 2006. Available online at: http://www.fortuitous.com/docs/primers/OR-intro.pdf

(accessed on 7/3/2015).

46. Hernandez-Matias, J.C., Vizan, A., Perez-Garcia, J. and Rios, J. An integrated modeling framework 

to support diagnosis for continuous improvement, Robotics and Computer Integrated 

Manufacturing, 2008, 24, 187-199.

47. Herrmann, C., Thiede, S., Stehr, J. and Bergmann, L. An environmental perspective on lean 

production, Manufacturing Systems and Technology for the New Frontier, The 41st CIRP 

Conference, Manufacturing Systems, May 26-38, 2008, Tokyo, Japan, 83-86.

48. Herrmann, C., Bergmann, L., Thiede, S., and Zein, A. Framework for integrated analysis of 

production systems, Advances in life cycle engineering for sustainable manufacturing businesses, 

Proceedings of the 14th CIRP conference, Tokyo, Japan, 2007, 195-200.

49. Herron, C., and Braiden, P.M. A methodology for developing sustainable quantifiable productivity 

improvement in manufacturing companies. International Journal of Production Economics, 104, 

143-153. 

50. Holweg, M. The Genealogy of lean production, Journal of Operations Management, 25, 2, 420-437.

51. Hon, K.K.B. Performance and evaluation of manufacturing systems, CIRP Annals – Manufacturing 

Technology, 2005, 54, 139-154. 

52. Huang, S., Chen, Y., Chen, X., Liu, K., Xu, X., Wang, C., and Halilovic, I. The next 

generation operational data historian for IoT based on informix. In Proceedings of the 2014 

ACM SIGMOD international conference on Management of data, 2014, 169-176. ACM.

53. IEC 2011. TC111 TR 62725 Sustainability Assessment for Electrical and Electronic Products.

Available online on: 

http://www.iec.ch/dyn/www/f?p=103:38:0::::FSP_LANG_ID,FSP_APEX_PAGE,FSP_ORG_ID,FS

P_PROJECT:25,20,1314,IEC/TR%2062725%20Ed.%201.0. Accessed on 02/06/2013. 

54. IEEE 2013. IEE 1680 Environmental assessment of electronic products. Available online 

on: http://grouper.ieee.org/groups/1680/. Accessed on 09/06/2013.

55. Imai, M. Kaizen: The Key to Japan’s Competitive Success, 1986, New York: Random House, pp. 60.

56. ISO 2006, Environmental management – life cycle assessment – principles and framework. 

ISO 14040. ISO Office, Geneva. 

http://www.iso.org/iso/catalogue_detail.htm?csnumber=37456
42

57. ISO 2014, IS) 22400-1: 2014(en) Automation systems and integration – Key performance indicators 

(KPIs) for manufacturing operations management – Part 1: Overview, concepts and terminology. 

Available on: https://www.iso.org/obp/ui/#iso:std:iso:22400:-1:ed-1:v1:en, 2014, (Viewed on 

2/14/2015.)

58. ISO 22400-2: 2011 Automation systems and integration – Key performance indicators (KPIs) for 

manufacturing operations management – Part 2: Definitions and descriptions of KPIs. Available on: 

ftp://ftp.nist.gov/pub/mel/michalos/Software/MTConnectCmdSimAgent/doc/ISO-TC184-

SC5_N1143_ISO_DIS_22400-2.pdf. (Viewed on 7/3/2015.)

59. ISO 2011, ISO: 50001 Assessment for Energy Use and Management. ISO Office, Geneva, 

http://www.iso.org/iso/home/standards/management-standards/iso50001.htm, ISO

60. ISO Standards, Available on: http://www.iso.org/iso/home.html, (accessed on 10/15/2014)

61. ISO 2013 20140-1:2013 Automation Systems and Integration – Evaluating Energy 

Efficiency and Other Factors of Manufacturing Systems that Influence the Environment –

Part 1: Overview and General Principles. Available online at: 

http://www.iso.org/iso/home/store/catalogue_tc/catalogue_detail.htm?csnumber=54809

(Accessed on 8/7/2014).

62. ISO/IEC 9075-1:2008. Information technology – Database Languages – SQL – Part 1: Framework 

(SQL/Framework) http://www.iso.org/iso/catalogue_detail.htm?csnumber=45498

63. Joung, C.B., Carrell, J., Sarkar, P. Feng, S.C. Categorization of indicators for sustainable 

manufacturing, Ecological Indicators, 2013, 24, 148-157. 

64. Judi, H. M., Jenal, R., Genasan, D. Quality control implementation in manufacturing companies: 

Motivating factors and challenges. Computer and Information Science, Numerical Analysis and 

Scientific Computing: Applications and Experiences of Quality Control, edited by Ivanov, O. April 

2011.

65. Kaplan, R. S. and Norton. D.P. Balanced scorecard – measures that drive performance, The 

Magazine, January 1992, Harvard Business Review. 

66. Kaplan, R.S. and Norton, D.P. The strategy-focused organization. Harvard Business School Press: 

Boston, 2001a, USA.

67. Lawrie, G. and Cobbold, I. Third-generation balanced scorecard: evolution of an effective strategic 

control tool. International Journal of Productivity and Performance Management, 2004, 53, 611-

623.

68. Keegan, D. P., Eiler, R. G. and Jones, C. R. Are your performance measures obsolete? 

Management Accounting, 1989, 71, 45-50. 

69. Kellens, K., Dewulf, W., Overcash, M., Hauschild, M. Z., and Duflou, J. R. Methodology 

for Systematic Analysis and Improvement of Manufacturing Unit Process Life Cycle 

Inventory (UPLCI) CO2PE! Initiative (cooperative effort on process emissions in 

manufacturing. Part 2: Case Studies), International Journal of Life Cycle Assessment, 2012, 

17, 242-251.

70. Kennerley, M., and Neely, A., Measuring performance in a changing business 

environment, International Journal of Operations & Production Management, 2003, 23, 

213-229. 
43

71. Khan, M. A., Uddin, M.F., Gupta, N. Seven V’s of Big Data: Understanding big data to extract 

value, Proceedings of 2014 Zone 1 Conference of the American Society for Engineering Education 

(ASEE Zone 1), 3-5 April 2014, Bridgeport, CT, Pages 1-5, 

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6820689 (Accessed on 8/28/2015).

72. Kibira, D., Jain, S., McLean, C. A system dynamics modeling framework for sustainable 

manufacturing, Proceedings of the System Dynamics Society, July 2009, Albuquerque, NM, 

http://www.systemdynamics.org/cgi-bin/sdsweb?P1285+0.

73. King, B., 2005. Performance Assurance for I.T. Systems, Auerbach Publications.

74. Kossiakoff, A., Sweet W. N., Seymour, S. C., and Biemer, S. M. Systems Engineering: 

Principles and Practice. Second Edition, 2011,. John Wiley & Sons, Wiley InterScience

75. Košturiak, J., and McGregor, M. 1999. Simulation in production system life cycle, Computers in 

Industry, 38, pp 159-172. 

76. Kumaraguru, S., Kulvatunyou, B., and Morris, K. C. Integrating Real-Time Analytics and 

Continuous Performance Management in Smart Manufacturing Systems. In Advances in Production 

Management Systems. Innovative and Knowledge-Based Production Management in a Global-Local 

World, 2014, 175-182. Springer Berlin Heidelberg.

77. Landau, I.D., Lozano, R., M’Saad, M., Karimi, A. Adaptive Control, Algorithms, Analysis and 

Applications, 2011, Springer-Verlag London Limited. DOI 10.1007/978-0-85729-664-1_1, 

78. Langley, G., Nolan, K, and Nolan, T. The foundation of improvement, Quality Progress, June 1994, 

page 81. 

79. LaValle, S., Lesser, E., Shockley, R., Hopkins, M.S., and Kruschwitz, N. Big Data, 

analytics and the path from insights to value, MITSloan Management Review, Winter 2011.

http://sloanreview.mit.edu/article/big-data-analytics-and-the-path-from-insights-to-value/ 

(Accessed on 8/28/2015).

80. Martinez, V., Pavlov, A., and Bourne, M. Reviewing performance: an analysis of the 

structure and functions of performance management reviews. Production Planning & 

Control, 21, 70-83.

81. McLean C. and S. Leong. The role of simulation in strategic manufacturing. In Proceedings 

of the 2001 International working Conference and Strategic Manufacturing, 2001.

82. MBSE Works. What is MBSE Works? – What you need to know. Available online at: 

http://mbse.works (Accessed on 6/12/2015).

83. Moen, Ronald; Norman, Clifford. Evolution of the PDCA Cycle. 2011. Available online at: 

http://pkpinc.com/files/NA01MoenNormanFullpaper.pdf (accessed on 8/14/2014).

84. Mönch, L. and Fowler, J.W. A Survey of problems, solution techniques, and future 

challenges in scheduling semiconductor manufacturing operations, J. Scheduling, 2011, 14, 

583-599

85. Montgomery, D.C. Introduction to Statistical Quality Control. Sixth Edition. 2009. John 

Wiley & Sons, Inc.

86. Muchiri, P., and Pintelon, L. Performance measurement using Overall Equipment Effectiveness 

(OEE): Literature review and practical application discussion, International Journal of Production 

Research, 2008, 64, 3517-3535. 
44

87. Neely, A.D. Integrating performance measurement systems, Business Performance Measurement, 

2002, 174-197, Cambridge University Press.

88. Neely, A.D., Adams, C. and Kennerley, M. The performance prism: The scorecard for managing 

business success, 2002. Financial Times/Prentice Hall 

89. Negahban, A., and Smith, J. S. Simulation for manufacturing system design and operation: 

Literature review and analysis, Journal of Manufacturing Systems, 2014, 33, 241-261.

90. Nudurupati, S.S., Bititchi, U.S., Kumar, V., Chan, F.T.S. State of the art literature review on 

Performance Measurement, Computers & Industrial Engineering, 2011, 60, 279-290.

91. OAGi. Open Application Group’s Integration Specification (OAGIS). Available on-line via: 

http://www.oagi.org/dnn2/DownloadsandResources/OAGIS100PublicDownload.aspx. (Accessed on 

11/23/3014)

92. OMG Systems Modeling Language, Available via: http://www.omgsysml.org, (Accessed on 

2/14/2015)

93. OMG. How to Deliver Resilient, Secure, Efficient, and Easily Changed IT Systems in Line with 

CISQ Recommendations, 2013. Available online at: 

http://www.omg.org/CISQ_compliant_IT_Systemsv.4-3.pdf. (Accessed on 7/3/2015).

94. Overcash, M. and Twoney, J. Unit process life cycle inventory (UPLCI) – A structured framework 

to complete life cycle studies, Leveraging Technology for a Sustainable World, 19th CIRP 

International Conference on Lice Cycle Engineering, Berkeley, 2012, 1-4.

95. Park, J., Law, K.H., Bhinge, R., Biswas, N., Srinivasan, A., and Dornfeld, D.A., Helu, M., and 

Rachuri, S. A generalized data-driven energy prediction model with uncertainty for a milling 

machine tool using Gaussian Process, Proceedings of the ASME 2015 International Manufacturing 

Science and Engineering Conference, MSEC2015 July 8-12, 2015, Charlotte, North Carolina, USA, 

MSEC2015-9354 (ahead of print).

96. Parnaby, J. Concept of a manufacturing system, International Journal of Production Research, 

1979, 17, 123-135.

97. Pospišíl, M., Mates, V. , Hruska, T, and Bartick, V. Process mining in a manufacturing company for 

predictions and planning, International Journal on Advances in Software, 2013, 6, 3 & 4, 293-297. 

Available online at: http://www.iariajournals.org/software/

98. Powell, T. C. Total quality management as competitive advantage: A review and empirical study, 

Strategic Management Journal, 1995, 16, 1, 15-37. 

99. Proficy Plant applications, Available online at: http://www.ge-ip.com/products/plantapplicationsperformancemanagement/c491. (Accessed on 3/4/2014)

100. Putnik, G.D., and Sluga, A. Reconfigurability of manufacturing systems for agility implementation, 

Digital Enterprise Technology, 2007, 91-98.

101. Qin, S. J. Process data analytics in the era of big data. AIChE Journal, 2014, 60, 3092-3100.

102. Raheja, D. G. and Allocco, M. Assurance Technologies Principles and Practices: A 

Product, Process, and System Safety Perspective, 2006, Second Edition, WileyInterScience.

103. Ramasesh, R., Kulkarni, S. and Jayakumar, M. Agility in manufacturing systems: an 

exploratory modeling framework and simulation. Integrated Manufacturing Systems, 2011, 

12,534-548.
45

104. Reich-Weiser, C., Vijayaraghavan, A., and Dornfeld, D. Metrics for Sustainable Manufacturing, 

Proceedings of the 2008 International Manufacturing Science and Engineering Conference, Illinois, 

October, 2008.

105. Ritchter, L., Lübkemann, J., and Nyhuis, P. Development of a model for the redesign of plant 

structures, International Journal of Social, Education, Economics and Management Engineering, 

2014, 8, 3295-3298. 

106. Rotithor, H. A taxonomy of performance assurance methodologies and its applications in high 

performance computer architecture, International Journal of Software Engineering & Applications

(IJSEA), 2013, 4, 1-17. Available online at: https://www.researchgate.net/publication/236202859

(Accessed 12/1/2014)

107. Saaty, T. L. The analytic hierarchy process, McGraw-Hill, New York, NY, 1980.

108. Saaty, T. L. Fundamentals of decision making and priority theory with the AHP, RWS Publications, 

Pittsburg, PA, 1994.

109. Saaty, T. L. Theory and applications of the analytic network process: Decision making with benefits, 

opportunities, costs, and risks. 2009. RWS Publications. 

110. Sadikoglu, E., and Zehir, C., Investigating the effects of innovation and employee performance on 

the relationship between total quality management practices and firm performance: An empirical 

study of Turkish firms, International Journal of Production Economics, 2010, 127, 13-26. 

111. Salloum, M. Towards dynamic performance measurement systems: A framework for manufacturing 

organizations, Malardelen University Press Licentiate Thesis. No. 140. 2011.

112. SCC (Supply Chain Council), Supply Chain Operations Reference Model, Revision 11.0, Supply 

Chain Council, 2012.

113. Schroeder, R. G., Linderman, K., Liedtke, C., and Choo, A. S. Six Sigma: definition and underlying 

theory, Journal of Operations Management, 2008, 26, 536-554.

114. Semini, M. Applicability of operations research in manufacturing logistics, Doctoral Thesis, 

Department of Production and Quality Engineering, Norwegian University of Science and 

Technology, 2011.

115. Shao, G., Kibira, D., Brodsky, A., and Egge, N. Decision support for sustainable 

manufacturing using Decision Guidance Query Language, International Journal of 

Sustainable Engineering, 2011, 3, 251-265.

116. Shao, G., Jain, S., and Shin, S. Data analytics using simulation for smart manufacturing, 

Proceedings of the 2014 Winter Simulation Conference, Savannah, Georgia, December 7-

14.

117. Sharma, A. B., Ivančić, F., Niculescu-Mizil, A., Chen, H., and Jiang, G. Modeling and 

analytics for cyber-physical systems in the age of big data. ACM SIGMETRICS 

Performance Evaluation Review, 2014, 41, 74-77.

118. Sherehiy, B., Karwowski, W., and Layer, J.K. A review of enterprise agility: Concepts, 

frameworks, and attributes, International Journal of Industrial Ergonomics, 37, 445-460.

119. Shipp, S.S., Gupta, N., Lal, B., Scott, J.A. , Weber, C. L., Finnin, M. S., Blake, Newsome, S., and 

Thomas, S. Emerging Trends in Advanced Manufacturing, IDA Paper P-4603, Institute for Defense 

Analyses, Approved for public release March 2012. Available online at: 

http://www.wilsoncenter.org/sites/default/files/Emerging_Global_Trends_in_Advanced_Manufactur

ing.pdf. (Accessed on 8/7/2013).
46

120. Shoemaker, D. and Mead, N.R. Software assurance measurement – State of the practice. Technical 

Note, Software Engineering Institute, Carnegie Mellon University, http://www.sei.cmu.edu

Simulation Interoperability Standards Organization (SISO). Core Manufacturing Simulation Data 

(CMSD) Standard. Available on line via: 

http://www.sisostds.org/StandardsActivities/DevelopmentGroups/CMSDPDGCoreManufacturingSi

mulationData.aspx. (Accessed on 12/6/2014)

http://www.sisostds.org/ProductsPublications/Pressreleases/CMSDPressRelease.aspx (Accessed on 

12/6/2014)

121. SMLC (Smart Manufacturing Leadership Collation). U.S. Public-Private Partnership 

Program in Smart Manufacturing: Questions and Answers about a Partnership, 2010. 

Available online at: https://smart-process-manufacturing.ucla.edu/presentations-andreports/spm-publications/SMLCpublicprivatepartnership.pdf/at_download/file. (Accessed 

on 12/7/2013).

122. Smola, S. and Vishwanathan, S. V. N. Introduction to Machine Learning, Cambridge University 

Press, 2008.

123. Son, S., Yahya, B., Song, M., Choi, S., Hyeon, J., Lee, B., Jang, Y., Sung, N. Process mining for 

manufacturing process analysis: A Case study, In Asia Pacific Business Process Management, 

Brisbane, Australia, July 3-4, 2014.

124. Striteska M., Spickova, M. Review and comparison of performance measurement systems, 

Journal of Organizational Management Studies, 2012, 12, Article ID 114900, 13 pages, 

Available on line at: 

http://www.ibimapublishing.com/journals/JOMS/2012/114900/114900.pdf (Retrieved on 

12/26/2014).

125. Sustainable Manufacturing Indicator Depository, NIST Manufacturing Engineering Laboratory 

Programs, Available online at: http://www.mel.nist.gov/msid/SMIR/index.html, 2009 (accessed on 

12/1/2014).

126. Taticchi, P., Tonelli, F., and Cagnazzo, L., Performance measurement and management: a literature 

review and a research agenda, Measuring Business Excellence, 2010, 14, 4-18. 

127. Thomann, J. Meta-Methodology: An Overview of what it is and how it was developed, 

Presented at the 58th American Educational Research Association Annual Meeting, New 

Orleans, Louisiana, February 26-March 1, 1973.

128. Triantaphyllou, E. Multi-criteria decision making methods: A comparative study, Kluwer 

Academic Publishers, Applied optimization. Dordrecht ; Boston, Mass.: Kluwer Academic 

Publishers.

129. Tsourveloudis, N., Valavanis, K. P. On the measurement of enterprise agility, Journal of 

Intelligent and Robotic Systems, 2002, 33, 329-342. 

130. Ucal, I., and Öztayşi, B. ANP in performance measurement and its application in a manufacturing 

system, Proceedings of the International Symposium on the Analytic Hierarchy Process, 2009.

131. Underwriters Laboratories Environment (ULE). 2010, ULE 880 Sustainability assessment standard 

based on governance, environment workplace, customers and suppliers, social and community,

ULE, GreenBiz, http://www.greenbiz.com/blog/2010/08/02/introducing-ule-880-sustainabilitymanufacturing-organizations. (Accessed on 11/20/2014)
47

132. Veleva, V., and Ellenbecker, M. Indicators of sustainable production: framework and 

methodology, Journal of Cleaner Production, 2001, 9, 529-549. 

133. Vinodh, S., Sundararaj, G., Devadasan, S.R., Maharaja, R., Rajanayagam, D., and Goyal 

S.K., Dessac: A decision support system for quantifying and analyzing agility, 

International Journal of Production Research, 2008, 46, 6759-6780, Taylor and Francis.

134. Walker, A. Witness simulation software, Proceedings of the 2012 Winter Simulation 

Conference, C. Laroque, J. Himmelspach, R. Pasupathy, O. Rose, and A.M. Uhrmacher, 

eds., 

135. Wibisono, D. A. Framework of performance measurement system for manufacturing companies. The 

South East Asian Journal of Management, 2011, 5, 107-116. 

136. Wilson, A. White paper on sustainable manufacturing, Strategic Sustainability Consulting.

Available on line at: www.sustainabilityconsulting.com (9/29/2014).

137. Yang, L., Deuse, J., Jiang, P. Multiple-attribute decision-making approach for an energy-efficient 

facility layout design. International Journal of Advanced Manufacturing Technology, 2013, 66, 795-

807.

138. Yang, M. G., Hong, P., and Modi, S. B. Impact of lean manufacturing and environmental 

management on business performance: An empirical study of manufacturing firms, International 

Journal of Production Economics, 2011, 2, 251-261

139. Zhao, J. L., Fan, S., and Hu, D. Business challenges and research directions of management analytics 

in the big data era. Journal of Management Analytics, (ahead-of-print), 1-6.