# Environmental KPI Selection Using Criteria Value and Demonstration

Deogratias Kibira, Shaw Feng

► To cite this version:

Deogratias Kibira, Shaw Feng. Environmental KPI Selection Using Criteria Value and Demonstration. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2017, Hamburg, Germany. pp.488-495, 10.1007/978-3-319-66926-7_56 . hal-01707258

HAL Id: hal-01707258

https://inria.hal.science/hal-01707258

Submitted on 12 Feb 2018

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Environmental KPI Selection Using Criteria Value and Demonstration

Deogratias Kibira¹, Shaw Feng²

¹Department of Industrial and System Engineering, Morgan State University, Baltimore, MD, U.S.A
deogratias.kibira@morgan.edu

²Engineering Laboratory, National Institute of Standards and Technology (NIST)
Gaithersburg, MD, U.S.A
shaw.feng@nist.gov

## ABSTRACT

Determining key performance indicators (KPI) is a first step in achieving environmental sustainability of manufacturing operations. KPI selection is a multi-criteria decision making problem, because of various criteria that must be considered. Intuitively, one can rank candidate KPIs by specifying a numerical value indicating the effectiveness of a KPI in satisfying each criterion. However, linking selection criteria to KPI objectives, ranking how well a KPI satisfying a given criterion, and assigning a value to each rank lead to better KPI rankings. Values for each score are crucial. This paper shows steps to capture values to derive a criterion value function that is used to rank candidate KPIs. Selected KPIs can be used for assessing and monitoring sustainability performance, which must be considered together with including traditional (e.g., throughput) measures. A machine shop is used to show how an objective of reducing emissions from energy use in manufacturing can be pursued by monitoring the energy used to produce a unit product.

**Keywords:** KPIs · Sustainability · Selection criteria · Value function

## 1 Introduction

Achieving sustainability of manufacturing processes requires efficient and effective methods for defining, selecting, deploying, and monitoring key performance indicators (KPIs). Selecting KPIs is a multi-criteria decision making problem for which several methods are available. One approach is to let stakeholders assign a score of a KPI in satisfying a selection criterion. This process is repeated for each candidate KPI and final KPI ranks are obtained from aggregation of these scores.

Ezell (2007) and Collins et al. (2016) showed enhancement to this approach by capturing stakeholder “value” for each score point on the Likert scale. Our previous research also used a multi-variate value model where stakeholders score (and provide value of) each candidate KPIs against each selection criterion (Kibira et al. 2017). The score represents the degree to which a KPI satisfies a criterion. Each selection criterion is linked to a
KPI objective. This way, each KPI is evaluated for its contribution to the defined envi-
ronmental objective. The developed procedure has been submitted to the American Socie-
ty for Testing and Materials (ASTM) as a work item to become a standard guide for iden-
tification and selection of environmental KPIs for manufacturing processes (ASTM Inter-
national). This procedure requires an in-depth understanding of developing and using
value functions for selection criteria. This process and the linking of criteria to KPI objec-
tive are discussed in this paper within the context of environmental assessment of manu-
facturing processes. This paper also demonstrates KPI deployment and performance
monitoring in a machine shop.

Typically, value functions capture experts’ assessment of the value of each score of a
KPI against a criterion. To develop a value function, Duarte et al. (2006) first defined the
minimum and maximum possible measures of the score but assumed a linear relationship
between assigned score and value. Keeney and Lilien’s (1987) developed a method of
assigning value at salient points on a common probability distributionThis paper derives
criteria value functions using a combination of above-mentioned approaches.

The rest of the paper is organized as follows. Section 2 presents concepts of the value model and shows steps to develop a value function. Section 3 presents examples of developing and using value functions. Section 4 presents a demonstration of implementing KPIs for performance monitoring. Section 5 is a discussion and conclusion of the paper.

## 2 Criteria value based KPI ranking and deployment process

Figure 1 shows our proposed approach for selecting and implementing KPIs for a manu-
facturing system based on value. This section overviews the steps and the process from
identifying selection criteria to ranking the candidate KPIs.

**Identifying KPI selection criteria:** Selection criteria should be fundamental to the KPIs as opposed to being a means to another criterion (Keeney and Lilien1987). For example, a criterion such as “quantifiable” can be a means to ensure that a KPI is “calculable.” Therefore, these two may not need to appear in the same criteria set. KPI objectives are used to identify criteria and are obtained from sustainability goal(s). For example, if reducing energy use by, say, 20% is the target, it implies that KPI should be measurable and/or computable. To obtain a complete representative list, criteria groups, e.g. financial-oriented or management-oriented criteria. To keep the analysis manageable, a decision can be made to select the best 5-10 criteria to make up the set.

**Candidate KPIs:** Typically, candidate KPIs are proposed by top management and pre-
sented to organizational units responsible for KPI implementation so they can be evaluat-
ed. However, the candidate KPIs can also be identified if there is a gap between KPIs
currently in use and those that are needed to achieve environmental objectives.
<figure><img src="image_2.png" /><figcaption>Figure 1. Process and steps of KPI selection and demonstration</figcaption></figure>

**Value model for KPI ranking:** For a value model, (1) each criterion is weighted for its contribution, and (2) each candidate KPI is measured against each measure criterion. Most previous researchers used the additive model to compute total value of a candidate KPI (Keeney and Lilien 1987; Duarte et al. 2006; Ezell 2007; Collins et al. 2016). Thus:

$$v(x_1, x_2, \dots, x_n) = \sum_{i=1}^{n} w_i v_i(x_i) \qquad (i)$$

Where $v$ is the overall value function, and $v_i(x_i)$ are the individual criteria values at measurement level $x_i$, $w_i$ are the scaling constants (weights), whose total should equal to 1.

$$\sum_{i=1}^{n} w_i = 1, \quad w_i > 0 \qquad (ii)$$

## Value function development

**Horizontal measurement:** This measurement scale is used to indicate the degree to which a KPI satisfies a criterion. After identifying this scale, the minimum and maximum possible values are specified. For example, Table 1 shows measurement scales as well as minimum and maximum values for three of the criteria described in Kibira et al. (2017).

<table><caption>Table 1. Measurement scale of sample KPIs</caption><thead><tr><th>Criterion</th><th>Designation</th><th>Measurement scale</th><th>Minimum</th><th>Maximum</th></tr></thead><tbody><tr><td>Quantifiable</td><td>x<sub>1</sub></td><td># of metrics and data</td><td>0</td><td>Total # of metrics and data</td></tr><tr><td>Cost effectiveness</td><td>x<sub>2</sub></td><td>$ (or max/min)</td><td>$0</td><td>$ max savings (or 1)</td></tr><tr><td>Calculable</td><td>x<sub>3</sub></td><td># of variables and data</td><td>0</td><td>Total # of variables and data</td></tr></tbody></table>

**Vertical measurement:** Values for each level on the horizontal scale are determined by analysts and subject matter experts (or stakeholders). KPI values increase with degree of satisfaction of each criterion by the KPI. Therefore, value functions for KPI selection would in general exhibit an increasing trend. Alarcon et al. (2011) proposed four relationships (i.e., linear, convex, concave, and S-shaped) that a value function can take, as seen in Figure 2
<figure><img src="image_3.png" /><figcaption>**Figure 2.** Common shapes of the value function</figcaption></figure>

To determine the actual values, let the minimum measurement be designated $x_{min}$ and maximum be $x_{max}$. On a 0 – 1 scale for value, $v(x_{min}) = 0$ and $v(x_{max}) = 1$. Keeney and Li-lien (1987) preferred to start with the mid-value (designated $x'$). The subject matter expert or stakeholders determines this value, where $v(x') = 0.5$. Other points between $x_{min}$ and $x'$ and between $x'$ and $x_{max}$ can be determined to yield additional points on the function $v$. If sufficient points can be garnered using experts, a sketch can complete the graph of the function.

**KPI ranking:** Next, for each KPI in the candidate set stakeholders independently assign a score showing agreement that the KPI satisfies the criterion. A value is obtained from the value function for each score. An average is calculated for the values obtained from all stakeholders for each criterion for each KPI.

## 3 Examples of applying steps in criteria value-based KPI selection

This section presents examples to illustrate how the steps may be applied. Some of the processes illustrated in Figure 1 are demonstrated in Kibira et al. (2017). Discussion will be on KPI selection criteria, value function development and KPI demonstration.

**Selection criteria:** These are specified by production managers, supervisors, and shop floor workers. Lower-level KPI objectives for reducing material consumption include reduction in virgin material use and increase in use of recycled materials. The KPI objectives are used to identify criteria that would meet these objectives.
**Value function for “Cost effectiveness” criterion:** This criterion implies the degree of perceived cost benefit of implementing the KPI. Let the measure for this criterion be expressed as the “savings” measured on a scale from 0 to 10, which is the difference between the income (or saved costs) and expenditure of implementing a KPI. Let the minimum savings, Smin, be 0 and the maximum, Smax, be 10.

The next step is to determine the shape of the value function. If savings through monitoring KPIs is a new strategic approach, any efforts in that direction are greatly encouraged. Therefore, initial measures are highly valued. A concave shaped value function, where the increase in value is maximized at the point of minimum measure, is suitable. See Figure 2(b). As you progress towards the maximum, the curve is more horizontal as the decision maker would generally assign less value to additions to high-level savings.

Once the general shape is established, what follows is to determine salient points on the curve. The expert is asked to express “How much savings, say y, such that the value from the minimum to y is equal to the value from y to the maximum?” Let this savings be labelled S' and x' designate the cost effectiveness criterion. On a scale from 0 – 1, v₁(0) = 0, v₁(S') = 0.5, and v₁(Smax) = 1. Proceeding from this point, mid-value assessments are made for additional pairs of cost effectiveness levels to generate other data points.

**Ranking KPIs:** Stakeholders independently assigned a score on the measurement scale for each KPI against each criterion. The value corresponding to this score was obtained from the value function. The final value of a KPI was a sum of values obtained from all stakeholders for all the criteria. The values (obtained from the value function) are scaled to the 0-10 range. All three stakeholders perform the same process and their results averaged. The final ranking in an example used in (Kibira et al. 2017) is summarized in the chart in Figure 3. This chart shows that the “energy per part” KPI has the highest rank. This is used for monitoring energy performance in the demonstrated machine shop.

<figure><img src="image_4.png" /><figcaption>**Figure 3.** Final assessment of individual KPIs (Kibira et al. 2017)</figcaption></figure>
# 4 KPI demonstration

This section shows the monitoring of performance using the highest ranking KPI, i.e., energy per part. Let us assume that the high-level goal was “to reduce global warming potential due to energy consumption in the manufacturing process without compromising throughput.” To evaluate achievement of the above goal, it is necessary to break down energy consumption into lower level objectives and to monitor the energy use.

We use a case study of a small machine shop that manufactures metal products. The shop comprises of a foundry, one milling machine, one lathe machine, a drilling machine, and an ultrasonic inspection center. There are two classes of products: A and B. Figure 4 shows the work flow. Production starts when the parts are loaded onto the shop. After casting, A requires turning operations while B requires milling. Final process is drilling although some of A do not use it. All parts pass through the inspection station.

<figure><img src="image_5.png" /><figcaption>**Figure 4.** Workflow through the shop</figcaption></figure>

**Energy modeling and simulation:** A discrete event simulation model of the shop was constructed using AnyLogic simulation software tool. To attribute energy to a unit part, we use a framework such as that developed by Seow et al. (2011). Two types of energy are distinguished: direct and indirect. Direct energy is the type used in the actual production process, e.g., heat to melt metal. Indirect energy is that used in the ambient working area such as heating, ventilation, and air-conditioning (HVAC), and lighting.

**Direct energy:** The direct energy for the casting process is obtained by combining energy used to bring the metal from room temperature to melting temperature and the fusion energy required. Machining energy is related to machining time. Both these quantities are calculated using empirical expressions for machining of mild steel products (Sonmez et al. 1999; Sardinas et al. 2006). For product inspection, the energy consumed is equal to the energy rating of the ultrasonic tester multiplied by duration of the inspection.

**Indirect energy:** Indirect energy consumed in each section of the shop depends on the type of manufacturing activity. Indirect energy is calculated by considering HVAC and
lighting rating requirements for manufacturing activities carried out in these sections.
Energy per part is obtained by dividing the result by the total of parts produced.

**Simulation output:** Simulation experiments are carried out to investigate the impact of batch size on both energy consumptions per part. Batch size is the variable used because it affects many factors including setup time and setup cost, inventory levels, lead times, safety stock, and order fulfilment. In general, small batch sizes are associated with higher overall set-up time while large batch size, without lot-splitting, can lead to increased idling and thus, reduced throughput. The effect of batch size is investigated for its effect on energy consumption and throughput in the multi-stage production environment. Note that set-up time for casting is not a constant for all batch sizes.

The energy consumption is shown in the graph in Figure 5. For each experiment, Parts A and B are loaded alternately onto the shop in batches of equal size. Batches are varied from an initial size of 5 units. For a batch-size less than ten units, there is significant increase in energy use per part while any batch sizes exceeding 20 units, the decrease in energy use is not significant. On the other hand, the total number of units produced falls almost evenly with increase in batch size. The decision maker can use this graph to balance energy per part and throughput for each situation.

<figure><img src="image_6.png" /><figcaption>**Figure 5.** Variation of energy consumed per part with batch size</figcaption></figure>

# 5 Summary and discussion

This paper has discussed using value of a score for a KPI against a criterion to evaluate a candidate KPIs. This approach was adopted after realizing that the relationship between such scores and benefit from the score is not always linear. Thus, the concept of value of each score, as a basis for decision-making is relevant to the KPI selection process and largely hinges on constructing value functions for each criterion.
Expert knowledge and stakeholder contribution is crucial for deriving value functions used for ranking candidate KPIs. Selection for KPIs is based on the resulting ranks. Selected KPIs can be used for assessing and monitoring environmental measures such as energy consumption. Analysis and tradeoff can be made between different measures. Simulation modeling may be applied to further investigate performance due to decisions made at different control levels as well as possible interactions between different KPIs.

**Acknowledgments:** This effort has been sponsored in part under the cooperative agreement No. 70NANB13H153 between NIST and Morgan State University. This material is declared a work of the U.S. Government and is not subject to copyright protection in the U.S.A.

**Disclaimer:** Certain commercial software products or services may be identified in this paper only for demonstration purposes. This use does not imply approval or endorsement by NIST, nor does it imply that these products are necessarily the best for the purpose.

# 6 References

1. Alarcon, B., Aguado, A., Manga, R., and Josa, A. (2011). A Value Function for Assessing Sustainability: Application to Industrial Buildings, *Sustainability*, 3: 35-50.

2. ASTM International, WK55825, New Guide for Definition, Selection, and Organization of Key Performance Indicators for Environmental Aspects of Manufacturing Processes, *Draft ASTM Standard*.

3. Collins, A., Hester, P. T., Ezell, B., and Horst, J., (2016). An Improvement Selection Methodology for Key Performance Indicators. *Env., Systs. & Decisions*, 36: 196-208.

4. Duarte, B.P.M., and Reis, A. (2006). Developing a Projects Evaluation System Based on Multiple Attribute Value Theory, *Computer & Ops. Res.*, 33: 1488-1504.

5. Ezell, B. C. (2007). Infrastructure Vulnerability Assessment Model (I-VAM), *Risk Analysis*, 27: 571-583.

6. Keeney, R. L. and Lilien G. L. (1987), New Industrial Product Design and Evaluation Using Multi-Attribute Value Analysis, *J. of Product Innovation Mngt.*, 4(3): 185-198.

7. Kibira, D., Brundage, M., Feng, S., and Morris, K.C. (2017). Procedure for Selecting Key Performance Indicators for Sustainable Manufacturing, *ASME-MSEC2017-2877*.

8. Seow, Y., and Rahimifard, S. (2011). A Framework for Modeling Energy Consumption within Manufacturing Systems, *CIRP J. of Mfg. Sc. and Tech*, 4: 258-264.

9. Sonmez, A., Baykasoglu, A., Dereli, T., and Filiz, I. (1999). Dynamic Optimization of Multipass Milling Operations via Geometric Programming, *Int. J. of Machine Tools & Mnf.*, 39: 297-332.

10. Sardinas, R. Q., Reis, P., Davim, P. (2006). Multi-objective Optimization of Cutting Parameters for Drilling Laminate Composite Materials by Using Genetic Algorithms, *Composites Science and Technology*, 66(15): 3083-3088.