# An Overview of a Smart Manufacturing System Readiness Assessment

Kiwook Jung, Boonserm Kulvatunyou, Sangsu Choi, Michael P. Brundage

► To cite this version:

Kiwook Jung, Boonserm Kulvatunyou, Sangsu Choi, Michael P. Brundage. An Overview of a Smart Manufacturing System Readiness Assessment. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2016, Iguassu Falls, Brazil. pp.705-712, 10.1007/978-3-319-51133-7_83. hal-01615704

HAL Id: hal-01615704

https://inria.hal.science/hal-01615704

Submitted on 12 Oct 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# An Overview of a Smart Manufacturing System
## Readiness Assessment

Kiwook Jung¹, Boonserm Kulvatunyou¹, Sangsu Choi², and Michael P. Brundage¹

¹National Institute of Standards and Technology (NIST)
Gaithersburg, MD, USA

²IGI, LLC. Clarksburg, MD, USA
{kiwook.jung, serm, mpbl}@nist.gov

**Abstract.** Smart manufacturing, today, is the ability to continuously maintain and improve performance, with intensive use of information, in response to the changing environments. Technologies for creating smart manufacturing systems or factories are becoming increasingly abundant. Consequently, manufacturers, large and small, need to correctly select and prioritize these technologies correctly. In addition, other improvements may be necessary to receive the greatest benefit from the selected technology. This paper proposes a method for assessing a factory for its readiness to implement those technologies. The proposed readiness levels provide users with an indication of their current factory state when compared against a reference model. Knowing this state, users can develop a plan to increase their readiness. Through validation analysis, we show that the assessment has a positive correlation with the operational performance.

**Keywords:** Smart manufacturing readiness · Smart factory · Maturity model

## 1 Introduction

Manufacturers lack a concrete methodology to choose and prioritize emerging technologies that aid in the creation of smart manufacturing systems and factories. On top of this, manufacturers may need to implement organizational and process improvements to realize the full benefits from these technologies. A survey indicate that most manufacturers have trouble making such improvements [1]. While larger companies can bring in consultants to assist with such issues, small and medium size manufacturers typically do not have the funds to do the same.

Existing methods such as the Supply Chain Readiness Level [1] and MESA Manufacturing Transformation Strategy [3] exist, but they largely ignore the use of information and communication technologies (ICT) as a primary foundation for making those improvements. There are existing works, which study the impact of Information Technology (IT) adoption to businesses (also known as business & IT alignment). However, each study typically focuses on evaluating a single technology such as the Enterprise Resource Planning (ERP) system or Manufacturing Execution System (MES). These studies have not taken into account other aspects of the organization that can affect the impact of the respective technology adoption.
This paper describes our initial work to develop a method for assessing a factory's readiness for incorporate emerging ICT technologies to become a smart factory. In our view, a smart factory uses ICT to maintain and improve its operational performance in response to its changing environment. Consequently, the method breaks down the assessment into four maturity components including the IT, the information connectivity, organization, performance management program maturities. Our method then combines these assessments into a single Smart Manufacturing System Readiness Level (SMSRL) index, which can be used for benchmarking individual factories or as criteria for selecting a supplier among several factories. In addition to describing our method, we discuss a correlation analysis that we performed. That analysis shows that the SMSRL index has a positive correlation with the operational performance.

Next we discuss related work in more detail before describing the SMSRL assessment method. Then, the result from the validation study is presented and a conclusion and remarks are given.

## 2 Related Work

Several types of readiness levels are used within the manufacturing sector. **Technology Readiness Level (TRL)** indicates the maturity of a technology for commercial adoption [4]. Similarly, **Manufacturing Readiness Level (MRL)** indicates the maturity of a manufacturing process technology [5]. An organization can use the same scale to indicate the maturity/capability of its respective technology as well. These methods do not evaluate a particular company for its readiness to adopt a particular technology.

**Supply Chain Readiness Level (SCRL)** [1] provides a method to assess the ability of the supply chain to operate and to achieve specific operational performance targets. The readiness levels are associated with characteristics within fifteen (15) categories that discretely provide an improvement roadmap for a supply chain design and the operation. Similar to the TRL and MRL, SCRL does not provide a methodology for a particular organization to assess its readiness to adopt a particular technology, which may correspond to some categories of the characteristics.

The MESA manufacturing transformation strategy (MTS) provides a framework based on the ISA-95 standard [3] to prepare an organization for Manufacturing-Operation Management (MOM) technologies adoption in four (4) business domains including Business Processes, Organization Structure, Personnel Skill Sets, and Manufacturing System Technology.

The SMSRL assessment objective is similar to that of the MESA-MTS albeit with the scope going beyond MOM technologies. Technically, the SMSRL index provides an indication of the current state with respect to a reference model. Both the SMSRL and the MESA-MTS allow the reference model to evolve as new technologies emerge and become available. Because of this, the assessment piece of the method, by design, is kept independent of the reference model.
# 3 Method

**Erro! Fonte de referência não encontrada.** provides an overall architecture of the readiness level assessment. It summarizes the steps, the inputs, and the outputs involved in the assessment followed by improvement plan development. The process is iterated after the plan has been implemented. The primary purpose of the proposed assessment based on the SMSRL index is to 1) help manufacturers determine their current level and 2) develop a customized improvement plan.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Overall assessment framework</figcaption></figure>

## 3.1 Profiling the Current State

The SMSRL proposed in this paper is based on the Factory Design and Improvement (FDI) reference-activity model defined in [6]. That model provides a set of reference activities; information entities for input, output, and constraints on each activity; and relevant software functions using the IDEF0 functional requirement modeling method [7]. On the basis of this model, we developed a questionnaire for profiling [8]. It is to be answered by relevant factory personnel. The questionnaire is organized into four measurement dimensions (C1 to C4) each of which consists of measurement items (process, designated personnel, etc.) as shown in **Erro! Fonte de referência não encontrada.** See the citations in **Erro! Fonte de referência não encontrada.** for the sources of these dimensions. Next we discuss each dimension in more detail.

<figure><img src="image_3.png" /><figcaption>Fig. 2. SMSRL measurements</figcaption></figure>

The Organizational maturity dimension (C1) is conceptually defined as the comprehensiveness of the activities in the reference activity model performed by the manufacturers. It is measured by 1) whether there is a process that formally manages each activity; and 2) whether there is a responsible human resource assigned to the activities.
The IT maturity dimension (C2) is conceptually defined as the degree to which IT resources are available and working. The IT resources refer to computerized tools and methods. For example, a paper-based analysis method for layout design would not be qualified as an IT resource.

The Performance Management maturity dimension (C3) is conceptually defined as the degree to which the performance measures are used and monitored. This dimension also takes into account the connectivity between different operational performance measures where appropriate.

Lastly, the Information connectivity maturity dimension (C4) is conceptually defined as the maturity of the method to exchange the required information and the degree to which the information is shared/exchanged.

Profiling the current state consists of three operations: scope determination, information collection and consolidation. The scope is represented by relevant activities and stakeholders. The FDI model also indicates stakeholders relevant to each activity based on the ISA-88 manufacturing control architecture [9]. Information collection and consolidation are performed collaboratively among the group of stakeholders that are relevant to the scope.

## 3.2 Evaluate Current State

The evaluation of the current state compares the profile to the reference activity model. Computation methods, as shown in **Erro! Fonte de referência não encontrada.**, are applied to each measurement dimension resulting in quantitative measures that can be used for comparison and benchmark.

The counting measure is the ratio between the number of elements employed in the current practice and those suggested in the reference activity model. For example, the number of software functions (per the reference model) available in the factory divided by the number of all the software functions identified in the reference model gives a C2 measure.

<table><caption>**Table 1.** Computation Methods for SMSRL</caption><thead><tr><td>SMSRL Construct</td><td>Computation Method</td></tr></thead><tbody><tr><td>C1: Organizational maturity [10]</td><td>Counting measure, Activity maturity scoring scheme</td></tr><tr><td>C2: IT maturity [10]</td><td>Counting measure</td></tr><tr><td>C3: Performance management maturity [12]</td><td>Counting measure</td></tr><tr><td>C4: Information connectivity maturity [13]</td><td>Incidence matrix-based similarity measure, Incidence scoring scheme</td></tr></tbody></table>

The Activity maturity scoring scheme is based on the Capability Maturity Model Integration (CMMI) [14]. The stakeholder of each activity scores the maturity of the activity based on the scale shown in **Erro! Fonte de referência não encontrada.**

The incidence matrix is commonly used to represent and analyze interactions between entities in a complex system. Here, an incidence matrix is used to represent the information entities connected between activities; and as such can be used to quantify
the information connectivity maturity (C4). It is an n x n matrix where n is the number of activities under evaluation, the row is the activity that provides the information entity (from-activity or sender activity), and the column is the activity taking the information entity as an input (to-activity or receiver activity). The cell, the incidence, indicates the maturity of the information flow from the row to the column. **Erro! Fonte de referência não encontrada.** shows a schematic view of an incidence matrix. The maturity of the information flow is marked by the scoring scheme shown in **Erro! Fonte de referência não encontrada.** with the highest score (1) being connected by a standard data exchange. All reference incidence matrices assume the highest score where there is the information connectivity between the from- and to-activity. A score of 0.7 represents that software capability for data exchange among activities exists, but information is not currently exchanged. When the data is exchanged manually between activities, the score is 0.3. A score of zero indicates that there is no data exchange between the activities.

<table><caption>**Table 2.** Activity maturing scoring scale</caption><thead><tr><td>Linguistic scale</td><td>Task score</td><td>Characteristics</td></tr></thead><tbody><tr><td>Not performed</td><td>0</td><td>-</td></tr><tr><td>Initial</td><td>1</td><td>Processes established, but unpredictable</td></tr><tr><td>Managed</td><td>3</td><td>Processes characterized for projects</td></tr><tr><td>Defined</td><td>5</td><td>Process characterized for the organization</td></tr><tr><td>Qualitative</td><td>7</td><td>Processes measured and controlled</td></tr><tr><td>Optimizing</td><td>9</td><td>Focus process improvement</td></tr></tbody></table>

<table><caption>**Table 3.** Activity incidence matrix</caption><thead><tr><td>From/To</td><td>To Act<sub>1</sub></td><td>...</td><td>To Act<sub>j</sub></td></tr></thead><tbody><tr><td>From Act<sub>1</sub></td><td>inc<sub>11</sub></td><td>...</td><td>inc<sub>1j</sub></td></tr><tr><td>...</td><td>...</td><td>...</td><td>...</td></tr><tr><td>From Act<sub>i</sub></td><td>inc<sub>i1</sub></td><td>...</td><td>inc<sub>ij</sub></td></tr></tbody></table>

<table><caption>**Table 4.** Incidence scoring scheme</caption><thead><tr><td>Score</td><td>Scoring Rule</td><td>Definition</td></tr></thead><tbody><tr><td>1</td><td>if a &isin; (S<sub>j</sub> &cap; B<sub>m</sub>) then, c = 1 &times; Ref</td><td>Standard data formats for activity j (and) compatible data formats for software system m</td></tr><tr><td>0.7</td><td>if a &isin; B<sub>m</sub> then, c = 0.7 &times; Ref</td><td>Compatible data formats for software system m</td></tr><tr><td>0.3</td><td>if a &notin; B<sub>m</sub> then, c = 0.3 &times; Ref</td><td>Manual transformation required from output data a to compatible data formats for software system m</td></tr><tr><td>0</td><td>if Ref = 0 then, c = 0 &times; Ref</td><td>No exchange required</td></tr><tr><td>0</td><td>If i or j = &empty; then, c = 0 &times; Ref</td><td>The current state does not perform the activity i or j. to be performed</td></tr><tr><td>0</td><td>If i = j then, c = 0 &times; Ref</td><td>Recursive</td></tr></tbody></table>
Where *i* is the sender activity; *a* is the output data format of the activity *i*; *S*<sub>j</sub> is a set of standard data formats associated with the receiver activity *j*; *B*<sub>m</sub> is a set of compatible data formats for the receiver software system *m*; and *c* is the incidence score.

The evaluation result can be visualized as shown in **Erro! Fonte de referência não encontrada.** Each indicator can be used individually or combined into a single SMSRL index. For simplicity, a single SMSRL index was computed using an average of C1, C2, C3 and C4. The overall index and/or individual construct can be used to prioritize the factory improvements or to evaluate potential suppliers.

<figure><img src="image_4.png" /><figcaption>Fig. 3. An exemplary assessment result</figcaption></figure>

**3.3 Develop Improvement Plan**

In the last step, the evaluation result is used to develop and prioritize an improvement plan. A classification analysis shown in the next section provides a high-level improvement recommendation. Our future work lies in developing a method to provide a more detailed recommendation.

**4 Validation Study**

This section investigates the validity of the proposed assessment using a similar approach to [13]. First, data about the relationships between the SMSRL and operational performance was collected. Then, hypothesis tests for the statistical significance of the relationships were performed. Lastly, we analyzed patterns of the SMSRL that can guide an improvement plan development. These activities are explained below.

**Data Used for the Validation.** Existing studies in the domain of business and IT alignment were used for the validation. A detailed analysis on the existing studies can be found in [8]. Different alignment constructs (i.e., measurement items) from these studies were mapped to performance categories (e.g., operational, financial) and were statistically correlated using empirical data.
**Validation Method.** To establish the relationship between the SMSRL assessment and the performance categories, the measurement items of the SMSRL assessment are mapped to those considered in the studies (operational, financial, value-based, and overall). A similarity value between the SMSRL assessment and the target study is then calculated using the n-gram measure (intersection divided by union). This gives the basis for the correlation analysis shown in the next subsection.

**Hypothesis Test.** Four hypothesis tests were performed. Statistically significant, positive-correlations with the SMSRL index were found on the operational performance, overall performance, and value-based performance as shown in **Erro! Fonte de referência não encontrada.** The financial performance was not found (hence not shown) to have a statistically significant positive-correlation.

<table><caption>**Table 5.** Hypothesis test results</caption><thead><tr><th>Hypothesis</th><th>p-value</th><th>Sig</th></tr></thead><tbody><tr><td>H1: the higher the similarity value, the higher the operational performance attributable to alignment</td><td>0.713</td><td>Yes (p &lt; 0.05)</td></tr><tr><td>H2: the higher the similarity value, the higher the overall performance attributable to alignment</td><td>0.404</td><td>Yes (p &lt; 0.05)</td></tr><tr><td>H3: the higher the similarity value, the higher the value-based performance attributable to alignment</td><td>0.529</td><td>Yes (p&lt;0.05)</td></tr></tbody></table>

**High-level Recommendation.** A k-means-clustering analysis on the simulated SMSRL results has been performed (k=3). Based on its result shown in **Erro! Fonte de referência não encontrada.**, a high-level recommendation can be made for each SMSRL cluster. The cells with bold-font values show the category of improvement a factory should focus on to have the largest impact on a respective performance category. For example, the first row indicates that improvements in the information connectivity (C4) is likely to have the best impact on the operational performance.

<table><caption>**Table 6.** High-level recommendation</caption><thead><tr><th rowspan="2">SMSRL<br>Centroid<br>(mean score)</th><th rowspan="2">Performance Category</th><th colspan="4">Standardized Coefficient of<br>Independent Variables</th></tr><tr><th>C1</th><th>C2</th><th>C3</th><th>C4</th></tr></thead><tbody><tr><td rowspan="2">Low (0.1957)</td><td>Financial</td><td>-0.0276</td><td><strong>0.1169</strong></td><td>-0.0035</td><td>-0.0048</td></tr><tr><td>Operational</td><td>-0.0996</td><td>0.0511</td><td>-0.0471</td><td><strong>0.0753</strong></td></tr><tr><td rowspan="2">Med (0.4608)</td><td>Financial</td><td><strong>0.0179</strong></td><td>0.004</td><td>-0.0021</td><td>-0.0278</td></tr><tr><td>Operational</td><td>-0.065</td><td>-0.0013</td><td>-0.1052</td><td>-0.0139</td></tr><tr><td rowspan="2">High (0.6453)</td><td>Financial</td><td>-0.0250</td><td><strong>0.0439</strong></td><td>-0.0074</td><td>-0.1083</td></tr><tr><td>Operational</td><td>-0.0270</td><td>0.0327</td><td>-0.0109</td><td><strong>0.0672</strong></td></tr></tbody></table>
# 5 Conclusion and Remark

We introduced a new, smart manufacturing system readiness assessment (SMSRL). SMSRL measures the readiness using maturity scoring of four dimensions: Organizational, IT, Performance management, and Information connectivity maturities. The core of the smart manufacturing concept is the ability to use information effectively. The SMSRL assessment provides a quantitative measure of this ability. Such measure, which is in the form of an index, can be used for benchmarking. The statistical analysis shows that the index has a positive correlation with three types of performance: operational, overall, and value-based.

The SMSRL index provides a real number as its readiness measure. The SCRL, on the other hand, provides discrete readiness levels. Each type of measure has its advantages. Discrete measures lend themselves readily to definitional levels. Real-numbered levels do not; however, they can be used in other quantitative analysis – such as the ones shown in section 4. Discrete measures cannot

In our future work, we will 1) develop a method to provide more detailed improvement recommendations 2) extending and/or experimenting with other models used as a reference for the assessment.

# References

1. Manufacture Innovation, http://nistmep.blogs.govdelivery.com

2. Tucker, B.: SCRL-model for Human Space Flight Operations enterprise supply chain (2010)

3. MESA: Transforming Manufacturing Maturity with ISA-95 Methods (2011)

4. Mankins, J. C.: Technology readiness levels. White Paper, April, 6 (1995)

5. Wheeler, D. J., Ulsh, M.: Manufacturing Readiness Assessment for Fuel Cell Stacks and Systems for The Back-Up Power and Material Handling Equipment Emerging Markets. National Renewable Energy Laboratory (2009)

6. Jung, K., et al.: An Activity Model for Smart Factory Design and Improvement. Production Planning & Control

7. IEEE 1320.1 IEEE Functional Modeling Language – Syntax and Semantics for IDEF0, International Society of Electrical and Electronics Engineers, New York (1998)

8. Jung, K.: Reference Activity Model-based Factory Design and Operations Evaluation Framework, Pohang University of Science and Technology.

9. ANSI/ISA-88.00.01-2010 Batch Control Part 1: Models and Terminology

10. Raymond, L., Paré, G.: Measurement of Information Technology Sophistication in Small Manufacturing Businesses. Information Resources Management Journal, 5(2), 4-16 (1992)

11. Powell, D., et al.: Lean production and ERP systems in small-and medium-sized enterprises: ERP support for pull production. Intl. J. of Production Research, 51(2), 395-409 (2013)

12. Maasouman, M. A., & Demirli, K.: Development of A Lean Maturity Model for Operational Level Planning. Intl. J. of Advanced Manufacturing Technology, 1-18 (2016)

13. Chung, S. H., et al.: The Impact of Information Technology Infrastructure Flexibility On Strategic Alignment and Application Implementations. The Communications of the Association for Information Systems, 11(1), 44 (2003)

14. CMMI Product Team. (2010). CMMI for Development, Version 1.3.