# Barriers to Performance Measurement Systems Effectiveness

Rui Manuel M. Sousa, Flávio Cunha, José Dinis-Carvalho

► To cite this version:

Rui Manuel M. Sousa, Flávio Cunha, José Dinis-Carvalho. Barriers to Performance Measurement Systems Effectiveness. 8th European Lean Educator Conference (ELEC), Nov 2022, Galway, Ireland. pp.23-29, 10.1007/978-3-031-25741-4_3. hal-04735678

HAL Id: hal-04735678

https://inria.hal.science/hal-04735678v1

Submitted on 15 Oct 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Barriers to Performance Measurement Systems
## Effectiveness

Flávio Cunha, José Dinis-Carvalho and Rui M. Sousa

Department of Production and Systems, Algoritmi Centre, School of Engineering, University of
Minho, Campus of Azurém, 4800-058 Guimarães, Portugal

id10046@alunos.uminho.pt;

dinis@dps.uminho.pt;

rms@dps.uminho.pt;

**Abstract.** In many organizations, it is common to find performance measurement systems (PMS) whose practical utility turns out to be very limited due to several factors/barriers. The purpose of this paper is to identify and categorize the main barriers to the effectiveness of PMS, and to explore how they can be eliminated or, at least, how their impact can be mitigated. A systematic literature review, using PRISMA methodology, is carried out to identify which factors are most frequently referred to as barriers to the effectiveness of PMS, due to their negative influence in terms of implementation, continued use or maintenance of these systems. Those barriers are grouped and classified into categories according to their similarities. The initial findings point out to 19 types of barriers (e.g. inappropriate indicators, lack of employee involvement and lack of connection to the strategy) divided into six categories. Then, cause-effect relationships between barriers should be explored and investigated, in order to identify interdependencies that can further influence the effectiveness of the PMS. From a perspective of continuous improvement, after identifying the most common barriers, as well as the existing cause-effect relationships, solutions and methodologies should be suggested that could help to mitigate or eliminate the impact on the effectiveness of the organizations' PMS.

**Keywords:** Performance Measurement System; Key Performance Indicators; Continuous Improvement; Operational Excellence; Lean Production

## 1 Introduction

The collection and analysis of data on the current state of a production unit allows the starting point for improvement to be described and represented and enables the analyst to identify where the focus should be placed [1]. Performance data is a particular important type of data that is crucial to drive and manage performance improvement, but using and maintaining a PMS (Performance measurement system) in an effective way can be quite a challenge. PMS is defined by Neely et al. [2] as a group of metrics
used to quantify the efficacy and efficiency of actions. Bititci [3] defines the creation and implementation of a PMS as the process of defining targets, developing a group of performance metrics, collecting, analyzing, reporting, interpreting and acting on performance data. Although there are several models of PMS, their implementation, use and maintenance continues to fail. McCunn [4] states that 70% of the performance measurement initiatives fail. Therefore, it becomes extremely important to identify why PMS fail and which are the barriers to PMS effectiveness, so those failure modes can be eliminated or mitigated in order to improve the chances of success of a PMS.

## 2 Methodology

A systematic literature review (SLR) was selected as methodology to help on identifying the main barriers to PMS effectiveness. The SLR was performed accordingly to the steps described by PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology [5]. Firstly, a search for scientific articles and book chapters was performed on Scopus and Web of Science databases. The research was performed using "performance measurement" as a keyword, in publications cited in five or more articles and from the areas of study of Engineering, Management, Business, Social Sciences and Computer Sciences. From this, 1787 publications were obtained from Scopus and 1728 from Web of Science.

After removing duplicates 2808 publications remained. A filtering was conducted by analyzing the title and the keywords, and 2550 publications were removed in this step. On the 258 publications obtained, the abstract was analyzed and 176 were removed. The remaining publications were read and analyzed and 31 were included in the analysis of the barriers to PMS effectiveness.

The barriers to PMS effectiveness described in the literature were classified into six categories (system, data, technology, indicators, people and culture) and types of barriers. The results of this classification are described in the findings section.

## 3 Findings

After classifying the barriers found on the literature according to the proposed categories and types of barriers, the results obtained are presented in Table 1. This table presents all types of barriers organized by categories, as well as the references where the barriers were identified.
<table><caption>**Table 1.** Summary of categories, subcategories and references</caption><thead><tr><th>Category</th><th>Types of barriers</th><th>References</th></tr></thead><tbody><tr><td rowspan="3">Indicators</td><td>Inappropriate indicators</td><td>[6]–[24]</td></tr><tr><td>Lack of balance of indicators</td><td>[11], [18], [19], [25]</td></tr><tr><td>Too many indicators (complexity)</td><td>[8], [19], [21], [22]</td></tr><tr><td rowspan="6">System</td><td>Communication system</td><td>[9], [19], [23], [25]</td></tr><tr><td>Complexity</td><td>[4]</td></tr><tr><td>Lack of connection to strategy</td><td>[4], [7], [12], [14], [17]–[20], [22], [26]–[28]</td></tr><tr><td>Lack of use for improvement</td><td>[11], [17], [24], [25], [28]–[30]</td></tr><tr><td>Problems in defining objectives/goals</td><td>[11], [21]</td></tr><tr><td>Unclear system</td><td>[4], [9], [12], [16]–[19], [21], [25], [30]–[32]</td></tr><tr><td rowspan="4">People</td><td>Employee involvement</td><td>[8]–[10], [16], [17], [19], [20], [24], [28], [29], [32], [33]</td></tr><tr><td>False expectations</td><td>[34]</td></tr><tr><td>Lack of resources or expertise</td><td>[4], [7], [10], [13], [30], [35]</td></tr><tr><td>Not understanding the indicators</td><td>[9], [16], [19], [20], [22], [25]</td></tr><tr><td rowspan="3">Culture</td><td>Blame culture</td><td>[4], [9], [15], [16], [19], [24]–[27], [30], [31], [34]</td></tr><tr><td>Lack of rewards/compensations</td><td>[14], [19]</td></tr><tr><td>Top management commitment</td><td>[4], [7], [10], [13], [16], [17], [19], [25], [28], [32]</td></tr><tr><td rowspan="2">Technology</td><td>Inadequate IT tools</td><td>[9], [23], [31]</td></tr><tr><td>Time and resources required</td><td>[4], [7], [8], [13], [17], [20], [26]–[28], [32], [34], [35]</td></tr><tr><td>Data</td><td>Difficulty in collecting, analyzing and presenting data</td><td>[6], [7], [9], [13], [18], [19], [22], [26]–[28], [33]</td></tr></tbody></table>

## 4 Discussion

As simple and visual way of presenting the barriers to PMS effectiveness found, an Ishikawa diagram was used. In Figure 1 is represented the Ishikawa diagram where in the main spines are the categories and in the smaller spines of each main spine are the types of barriers identified. In front of each category and type of barrier identified is the frequency which each one is referred in the literature.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Ishikawa diagram with barriers to PMS effectiveness</figcaption></figure>

Of the several types of barriers found in the literature, some are referred more frequently than others. The categories, types of barriers and the frequency with which they are referred are presented in Graph 1.

<figure><img src="image_4.png" /><figcaption>Graph 1. Categories, types of barriers and frequency they are referred</figcaption></figure>

The fact that the indicators used are not appropriate is the most referred type of barrier in the literature. After this type of barrier the most referred ones are: lack of employee involvement; unclear system; difficulty in collecting, analyzing and presenting data; and, lack of connection to the strategy.
There can be a cause-effect relationship between several of the barriers identified making it more complex. For example, the fact that there is a blame culture in the or- ganization can cause a lack of involvement from the employees, resulting in resistance to an effective use of the PMS. The ineffectiveness of a PMS can be caused by a single barrier or by the combination of several.

## 5 Conclusion

As discussed previously the relation between the barriers identified can be complex. The failure of a PMS can result from one or from the combination of several of the barriers identified. Therefore, in order to maximize the odds of achieving an effective PMS, the cause-effect relationships between barriers should be investigated and meth- odologies and techniques that eliminate or mitigate the barriers should be developed and used.

As future investigation, all the recommendations that emerge as way to eliminate or mitigate the barriers to PMS effectiveness, could be transformed into to methodologies that can be grouped in a model, a guide allowing to properly implement, use and main- tain a PMS.

The barriers identified in this research were obtained through a systematic literature review that composes a secondary data source. The barriers found should be validated through the collection and analysis of data from a primary source.

## Acknowledgements

This work has been supported by FCT – Fundação para a Ciência e Tecnologia within the R&D Units Project Scope: UIDB/00319/2020.

## References

[1] M. Imai, *Gemba Kaizen*, 2nd Editio. McGraw-Hill, 1997.

[2] A. Neely, M. Gregory, and K. Platts, “Performance measurement system design: A literature review and research agenda,” *Int. J. Oper. Prod. Manag.*, vol. 15, no. 4, pp. 80–116, 1995, doi: 10.1108/01443579510083622.

[3] U. S. Bititci, *Managing business performance: The science and the art*. John Wiley & Sons, 2015.

[4] P. McCunn, “The balanced scorecard...the eleventh commandment,” *Manag. Account.*, vol. 76, no. 11, pp. 34–36, Dec. 1998, doi: Available: https://www.proquest.com/trade-journals/balanced-scorecard-eleventh-commandment/docview/195676560/se-2?accountid=39260.

[5] D. Moher, A. Liberati, J. Tetzlaff, D. G. Altman, and T. P. Group, “Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement,” *PLoS Med*, vol. 6, no. 7, 2009, doi: 10.1371/journal.pmed.1000097.

[6] M. G. Brown, *Keeping Score: Using the Right Metrics to Drive World-class*
*Performance*. Quality Resources, 1996.

[7] P. Charan, R. Shankar, and R. K. Baisya, “Modelling the barriers of supply chain performance measurement system implementation in the Indian automobile supply chain,” *Int. J. Logist. Syst. Manag.*, vol. 5, no. 6, pp. 614–630, 2009, doi: 10.1504/IJLSM.2009.024794.

[8] S. S. Nudurupati, U. S. Bititci, V. Kumar, and F. T. S. Chan, “State of the art literature review on performance measurement,” *Comput. Ind. Eng.*, vol. 60, no. 2, pp. 279–290, 2011, doi: 10.1016/j.cie.2010.11.010.

[9] S. Okwir, S. S. Nudurupati, M. Ginieis, and J. Angelis, “Performance measurement and management systems: a perspective from complexity theory,” *Int. J. Manag. Rev.*, vol. 20, no. 3, pp. 731–754, 2018, doi: 10.1111/ijmr.12184.

[10] Şerban Radu-Alexandru and H. Mihaela, “Performance Management Systems – Proposing and Testing a Conceptual Model,” *Stud. Bus. Econ.*, vol. 14, no. 1, pp. 231–244, 2019, doi: 10.2478/sbe-2019-0018.

[11] A. M. Schneiderman, “Why balanced scorecards fail,” *J. Strateg. Perform. Meas.*, vol. 2, no. 11, pp. 6–11, 1999.

[12] J. R. Dixon, A. J. Nanni, and T. E. Vollmann, *The new performance challenge*. 1990.

[13] S. D. Sousa, E. M. Aspinwall, and A. Guimarães Rodrigues, “Performance measures in English small and medium enterprises: survey results,” *Benchmarking An Int. J.*, vol. 13, no. 1/2, pp. 120–134, Jan. 2006, doi: 10.1108/14635770610644628.

[14] T. Watts and C. J. McNair-Connolly, “New performance measurement and management control systems,” *J. Appl. Account. Res.*, vol. 13, no. 3, pp. 226–241, 2012, doi: 10.1108/09675421211281308.

[15] M. Wouters and C. Wilderom, “Developing performance-measurement systems as enabling formalization: A longitudinal field study of a logistics department,” *Accounting, Organ. Soc.*, vol. 33, no. 4–5, pp. 488–516, 2008, doi: 10.1016/j.aos.2007.05.002.

[16] M. Zairi, *Measuring performance for business results*. Springer Science & Business Media, 1994.

[17] A. A. de Waal and H. Counet, “Lessons learned from performance management systems implementations,” *Int. J. Product. Perform. Manag.*, vol. 58, no. 4, pp. 367–390, 2009, doi: 10.1108/17410400910951026.

[18] F. Franceschini, M. Galetto, and D. Maisano, *Management by Measurement: Designing key indicators and performance measurement systems*. SpingerScience & Business Media, 2007.

[19] M. Franco and M. Bourne, “Factors that play a role in ‘managing through measures,’” *Manag. Decis.*, vol. 41, no. 8, pp. 698–710, 2003, doi: 10.1108/00251740310496215.

[20] A. M. Ghalayini and J. S. Noble, “The changing basis of performance measurement,” *Int. J. Oper. Prod. Manag.*, vol. 16, no. 8, pp. 63–80, 1996, doi: 10.1108/01443579610125787.

[21] E. Giovannoni and M. P. Maraghini, “The challenges of integrated performance measurement systems: Integrating mechanisms for integrated measures,” *Accounting, Audit. Account. J.*, vol. 26, no. 6, pp. 978–1008, 2013, doi: 10.1108/AAAJ-04-2013-1312.

[22] K. J. Hatten and S. R. Rosenthal, “Why-and How-to Systematize Performance
Measurement," *J. Organ. Excell.*, vol. 20, no. 4, pp. 59-73, 2001, doi: 10.1002/npr.1108.

[23] C. Lohman, L. Fortuin, and M. Wouters, "Designing a performance measurement system: A case study," *Eur. J. Oper. Res.*, vol. 156, no. 2, pp. 267-286, 2004, doi: 10.1016/S0377-2217(02)00918-9.

[24] A. Neely and M. Bourne, "Why Measurement Initiatives Fail," *Meas. Bus. Excell.*, vol. 4, pp. 3-7, 2000, doi: 10.1108/13683040010362283.

[25] A. Meekings, "Unlocking the potential of performance measurement: A practical implementation guide," *Public money Manag.*, vol. 15, no. 4, pp. 5-12, 1995, doi: 10.1080/09540969509387888.

[26] M. Bourne, "Researching performance measurement system implementation: The dynamics of success and failure," *Prod. Plan. Control*, vol. 16, no. 2, pp. 101-113, 2005, doi: 10.1080/0953728051233133011.

[27] M. Bourne, A. Neely, K. Platts, and J. Mills, "The success and failure of performance measurement initiatives: Perceptions of participating managers," *Int. J. Oper. Prod. Manag.*, 2002, doi: 10.1108/01443570210450329.

[28] M. Bourne, *Handbook of Performance Measurement*. GEE Publishing, 2004.

[29] M. Bourne, A. Neely, J. Mills, and K. Platts, "Why some performance measurement initiatives fail: lessons from the change management literature," *Int. J. Bus. Perform. Manag.*, vol. 5, no. 2-3, pp. 245-269, 2003, doi: 10.1504/ijbpm.2003.003250.

[30] M. Kennerley and A. Neely, "A framework of the factors affecting the evolution of performance measurement systems," *Int. J. Oper. Prod. Manag.*, vol. 22, no. 11, pp. 1222-1245, 2002, doi: 10.1108/01443570210450293.

[31] U. Bititci, P. Garengo, V. Dörfler, and S. Nudurupati, "Performance Measurement: Challenges for Tomorrow," *Int. J. Manag. Rev.*, vol. 14, no. 3, pp. 305-327, 2012, doi: 10.1111/j.1468-2370.2011.00318.x.

[32] B. Townley, D. J. Cooper, and L. Oakes, "Performance Measures and the Rationalization of Organizations," *Organ. Stud.*, vol. 24, no. 7, pp. 1045-1071, 2003, doi: 10.1177/01708406030247003.

[33] R. Simons, "Control in an Age of Empowerment," *Harvard Business Review Press*, Harvard Business Review Press, 1995.

[34] G. T. Gabris, "Recognizing Management Technique Dysfunctions: How Management Tools Often Create More Problems than They Solve," *Public Product. Rev.*, vol. 10, no. 2, pp. 3-19, 1986, doi: 10.2307/3380448.

[35] M. L. R. da Costa, E. G. de Souza Giani, and E. V. C. Galdamez, "Vision of the balanced Scorecard in micro, small and medium enterprises," *Sist. Gestão*, vol. 14, no. 1, pp. 131-141, 2019, doi: 10.20985/1980-5160.2019.v14n1.1505.