Journal of Japanese Management Vo.4, No.1, November 2019ISSN 2189-9592

The impact of quality of performance measures on theperformance of public sector organisations: Quantitative analysis
based on questionnaire survey
Takehiro Metoki
Faculty of Economics, Musashi University
Japan Cost Accounting Association
E-mail: t-metoki@cc.musashi.ac.jp
Abstract
Although the introduction of PMS has progressed in many organisations, it has notnecessarily been effectively utilised. Since the introduction and use of PMS has become a global
trend, it has been reported that perfunctory introduced PMS has a negative impact onorganisational performance. The issue of the quality of performance measures, which is shownto have a significant impact on the effectiveness of PMS in public sector organisations, has notbeen fully discussed. Low-quality performance measures can lead to incorrect decisions andactions and can undermine the credibility of PMS. Furthermore, if PMS constructed with a large
amount of tax is not used properly, then it will lead to tax waste. Therefore, it is necessary tomaintain and improve the quality of performance measures when designing and using PMS in
public sector organisations. Previous studies pointed out that the quality of performancemeasures have a significant impact on organizational performance, the relationship between
the two has not been clarified quantitatively. This study presents empirical evidence that thequality of performance measures (validity, legitimacy, and functionality of performancemeasures) increase positive outcomes of performance management systems use in public sector
organisation and suppress negative outcomes of performance management systems such as lossof usefulness and side effect. Data for analysis was collected by mailing questionnaire survey; it
was conducted in FY2016 in 791 Japanese cities (designated cities, core cities, special cities, andother cities) Japan. The number of valid responses was 327 (valid response rate 41.3%).
Keywords
Performance management systems, Quality of performance measures, Validity of performancemeasures, Legitimacy of performance measures, Functionality of performance measures

30

Journal of Japanese Management Vo.4, No.1, November 2019(1) Introduction

ISSN 2189-9592

the organisation (van Helden and Reichard,

The use of performance management2013).

These

studiesareincreasingly

systems (PMS) has become a global trend inclarifying the relationship between PMS

public sector organisations (Pollitt 2006). The

designuse of PMS not only creates incentives toperformance in public sector organisations.

alignHowever,organisational

objectives

with

and
the

use

and

issue oforganisational
the quality of

individual goals but also provides valuable

performance measures, which is shown to

feedback

have a significant impact on the effectiveness

on

the

progressof listed goals (Cavalluzzo and Ittner, 2004;of PMS in public sector organisations, has not

Heinrich,

Schack,been fully discussed (van Dooren et al., 2015).

1996). Owing to these benefits, PMS, also

Low-quality performance measures can leadcalled public sector evaluation ”gyousei-

to incorrect decisions and actions and canhyouka”, has been introduced by many public

undermine

sector organisations in Japan since the late

PMS. Furthermore, if PMS constructed with1990s (Matsuo, 2009; Metoki, 2012).

a large amount of tax is not used properly,

1999;

Kravchukandthecredibilityof

Although the introduction of PMS hasthen it will lead to tax waste. Therefore, it isprogressed in many organisations, it has notnecessary to maintain and improve thenecessarily been effectively utilised. Since thequality of performance measures whenintroduction and use of PMS has become adesigning and using PMS in public sector

global trend, it has been reported thatorganisations.

perfunctory introduced PMS has a negative

In this regard,

Bouckaert(1993)

impact on organisational performance (Bevan

recognises the three aspects of validity,and Hood, 2006; de Bruijn, 2002; Newberry

legitimacy, and functionality as the quality ofand Pallot, 2004).

inperformance measures important for PMS in

public sector organisations not only impacts

public sector organisations. In the case of

positively, but also has the risk of negative

validity, performance measures are measured

consequences; if it is not designed and used

as objective measures without being distorted

appropriately, then it may lead to erroneousby external influences, and legitimacy means

decision-making and actions. (Cuganesan etthat all organizational members support theal., 2014; van Dooren et al., 2015). This leads

design and use of performance measures.

to the question of what considerations should

Finally,

be taken when designing and using PMS.

measured performance measure is realistic

Furthermore,

PMS

functionality

means

that

the

In this regard, research has shown thatand suitable for managerial use. In line withthe effect of PMS on the relationship between

Bouckaert (1993), this research considers the

PMS design and use and results in public

quality of performance measures to be

sector organisations differs depending on how

composed of 1) validity, 2) legitimacy, and 3)

tomanagement

functionality. According to Bouckaert (1993),

information (Metoki and Senoo, 2012) andthe quality of performance measures affects

various contingency factors inside and outside

the

use

of

performance31

performanceofpublicsectorJournal of Japanese Management Vo.4, No.1, November 2019organisations.

Despitetheirimpacts,

It

ISSN 2189-9592

revealed

that

contractibility

which

research only presents the concepts thatmeasured by the three concepts of clarity ofmake up the quality of performance measures,

goals, measurability, and the degree to which

and there is no quantitative analysis of the

managers

impact on the performance of public sectortransformation

organisations. Therefore, in this study, therelationship between the incentive-oriented

relationshipof

use of the performance measurement system

performance measures and the performanceand performance. However, it has beenof public sector organisations is analysed

pointedquantitatively based on the results of aorganisations does

questionnaire survey conducted in 791 cities

positive results; as stated earlier, it canin Japan.

produce negative results if not properly

betweenthequality

know

out

and

process

control

themoderates

thethat PMS innotpublicalwayssectorproduce

The rest of this paper is structured as

designed and used (Cuganesan et al. 2014;

follows. Section 2 summarises related studies

van Helden and Reichard 2013). In thisand presents the analysis framework of thisregard, Bouckaert (1993) pointed out that the

study. Section 3 employs variables, anduse of PMS will facilitate false decision-

Section 4 discusses the results of the analysis.

making and behaviour if the quality of

Finally, Section 5 presents the contributions

performance measures is not maintained atand limitations of this study and draws

an appropriate level. Similarly, Melnyk et al.

conclusions.

(2014) found that performance measures andtarget are not modified according to the(2)Previous

researchandanalysissituation in which the organisation is placed;frameworkin other words, in certain cases, organisations

1. Positive and negative consequences ofcontinue to use these measures without

using PMS

proper modification. (Melnyk et al. 2014).

Metoki and Senoo (2012), using 264

Therefore, it is necessary to clarify theobservations based on a questionnaire survey

relationshipon Japanese government oraganisations,performance measures in the PMS of publicrevealed that the results achieved by using

sectorPMS

performance.

differperformance

dependingonmanagement

the

use

of

betweenorganisations

the
and

qualityoforganisational

information(financial outcomes/improved responses to

2. Building an analysis framework

residents’ needs/psychological empowerment

In this study, we will focus on theof those in charge of public sector reforms).

concepts

Additionally, Speklé and Verbeeten (2014)

functionality, and conduct an exploratory

studied the relationship between PMSs usage

analysis of the relationship between theand organisational performance using 101

quality of performance measures and theobservations

questionnaires

performance of public sector organisations.

targeting Dutch public sector oraganisations.

However, neither Bouckaert (1993) nor van

based

on

32

of

validity,

legitimacy,

andJournal of Japanese Management Vo.4, No.1, November 2019ISSN 2189-9592

Dooren et al. (2015), who indicate thethat measured performance measures areimportance of the quality of performancesuitable for use. Therefore, if the performancemeasures in public sector oraganisations,

measures becomes unrealistic because ofprovides a measure of the quality ofchanging situation, it will be necessary to stop

performance measures. Therefore, we willusing

address this gap in this study. First, validity

measures.

is the first concept that

constitutes

the

or

redesigning

Conversely,

therecentperformancestudieshavequality of performance measures. Bouckaert

pointed out that the effect of PMS in public(1993) considers validity as a performance

sector organisations may not be necessarily

measurethat

objectively,

positive, but also negative (Cuganesan et al.,without

being

external

2014; van Helden and Reichard, 2013).

is

the

Therefore, when analysing the relationshipconsistency of measurement, and hence its

between the quality of performance measuresdesign must ensure that the measurement’s

andresults do not vary due to the influence oforganisations, we consider both positive andexternal factors, other than the quality ofnegative outcomes.

ismeasured
distorted

influences. Additionally,byvalidity

theperformanceofpublicsectorstaff behaviour. Therefore, it is important

Based on the aforementioned studies, athat performance measures are designed andmultiple regression analysis is conductedmeasuredensure thatwith validity, legitimacy, and functionality as

measurement results are not distorted byindependent variables and the performance of

external influences. The second concept thatthe public sector organisation as dependent

constitutes

performancevariables. In the analysis, environmental

measures is legitimacy. As per Bouckaert

uncertainty, operational complexity, and the(1993), legitimacy supports the design andnumber of employees are taken as control

use of performance measures by all members

variables in consideration of the varyingof the organisation. Additionally, it isenvironments

reported that legitimacy is ensured by theorganisations and the diverse businessparticipation

thecontents of the responding organisations. The

organisation in the design and use ofanalysis framework of this study is shown inperformance

Figure 1.

appropriately to

the

of

quality

theofmembers

measures.

This

of

prevents

surrounding

publicsectorunauthorized manipulation and gaming of
performance measures. Therefore, legitimacy(3) Employment of data and variables

is expected to increase by the participation of

1. Dataorganisational members in designing, using,andmodifying

performance

Data for analysis was collected bymeasures.

mailingquestionnaire

survey;itwas

The third concept that constitutes the quality

conducted in FY2016 in 791 Japanese citiesofmeasures is(designated cities, core cities, special cities,

considersand other cities). Prior to sending theperformance

functionality. Bouckaert(1993)

functionality to be a realistic measurement soquestionnaire, we asked two practitioners,
33

Journal of Japanese Management Vo.4, No.1, November 2019ISSN 2189-9592

Independent variables

Dependent variables

Quality of performance measures‣Validity of performance measures‣Legitimacy of performance measures‣Functionality of performance measures

Performance of the public sector organisation
・Positive outcomes
・Negative outcomes

Figure1 The analysis framework

who supervised the related work in the publicoraganisations

sector organisation, and two management

1. Additionally, respondents comprised staff

accounting

at a section manager level or higher in the

researcherswithspecialized

are

shown

in

Table

knowledge in performance management to

responding

check the validity of the wording and question

test was performed on the data collected fromitems. The questionnaire was received by the327 public sector organisations used in the

supervisor of the management department ofanalysis. The results confirmed that the data

the public sector organisation, and theofquestionnaire was sent with a request letter

generally

anddistribution

reply envelope. In

consideration ofthedepartment. A goodness-of-fit

publicsectorcompatibleorganisation

was

with

the

=

7.770, df =

(χ 2

group

incentives for answering the questionnaire, it

3, p = .051). Additionally, we conducted a test

was clarified that a report on the analysisof the difference between the size of the target

results will be sent to the applicant. In orderorganisation and the non-target organisation

to improve the collection rate, a reminder was(public

sent before the collection deadline. The final

employees), but no significant difference wasnumberfound between the two. The results did notofresponding

was 339 (recovery

oraganisations

rate 42.9 %). In

the

of

organisations

inadequate

response

and

that
were

responses

was 327 (valid

rate 41.3%). Details

of

the

of

gave
judged

2. Variable manipulation
2.1 Validity of performance measures

response

Bouckaert (1993) regards the validity

respondingof performance measures as being measured

Table 1 Response distribution
Group classification

numberdata.

inappropriate for analysis. The number ofvalidorganization:

show any significant non-response bias in theanalysis, we used data, excluding the
responses

sectorNumber of sent

Number of valid response (rate)

designated cities

20

14

70.0%

core cities

48

20

41.7%

special cities

36

8

22.2%

other cities

687

285

41.5%

total

791

327

41.3%

34

Journal of Japanese Management Vo.4, No.1, November 2019ISSN 2189-9592

as an objective measure without being

accurately reflect the results of the staff’s

distorted by external influences. In otheractions, or that the staff's work is good. The

words, there is a need to use objective

questionnaire asked whether it is directly

performance measures to produce verifiablereflected in the performance measures or

results and ensure that they are not changed

whether these measures are objective; theby external influences. Therefore, we referred

responses were sought from 5 points, from ‘1

to

(not

Moers

(2006),

which

analysed

the

at

all)

to

5

(exactly)’.

In

the

relationship between performance measures’variabilisation, these average values arecharacteristics and performance (Table 2).

scored (α = .695). Although Cronbach's α isSpecifically, we determined whether theslightly below the general level of .70, it isresults of the performance measures are set

close to .70. Hence, we believe that there is noso that they cannot be affected externally andmajor problem in validity.

whethertheperformancemeasures

Table 2 Validity of performance measures
Range

Mean

S. D.

1–5

3.03

.816

1–5

2.86

.784

1–5

2.98

.746

1–5

3.50

.743

The results of performance measures are appropriately designed
VALID_1 to ensure that they are not affected by external influences, otherthan those emerging from staff’s actions.
VALID_2
VALID_3

Performance measures accurately represent whether the staff
exhibited superior performance.
If staff exhibited superior performance, it was directly reflectedin performance measures

VALID_4 Performance measures are objective and verifiable

2.2 Legitimacy of performance measures
Bouckaertthesystem usage. Based on Bourne et al. (2002),

legitimacy to be that the design and use ofthis study discusses the participation of

performance measures are supported by allorganizational members in the design and use

members of the oraganisationso that theof management systems; moreover, the study

performance measures can be designed andsets and amends performance measures,

used to prevent unauthorized manipulationcollects data, and conducts analysis to achieve

and gaming. Legitimacy is ensured by thenumerical targets for performance measures.

participation of organizational members inFive questions are set regarding the degree ofthe design and use of the PMS. Therefore,

participation of non-managerial staff in

Abernethy

also

processes such as the evaluation of theempirically clarified that the degree ofachievement status of the performance

employee participation in the management

measures (Table 3). The above questions were

information system design has a positive

responded on the basis of 5 points, from ‘1 (notimpactparticipate at all) to 5 (Actively participating)’.

on

and

the

(1993)

considers

performance of the management information

Bouwenssatisfaction(2005)

levelsand35

Journal of Japanese Management Vo.4, No.1, November 2019ISSN 2189-9592

Table 3 Legitimacy of performance measures
Range

Mean

S. D.

LEGIT_1 Setting of performance measures

1–5

3.72

.801

LEGIT_2 Selecting the data for setting of performance measures

1–5

3.63

.829

LEGIT_3 Amending performance measures

1–5

3.60

.822

LEGIT_4 Actions to achieve performance targets

1–5

3.88

.713

LEGIT_5 Analysis to achieve numerical targets for performance measures

1–5

3.65

.821

While constructing the variables, thesefailing which they can have a negative impact

average values were scored (α = .905).

on organisational performance (Melnyk et al.,2014). Therefore, we set four questions to

2.3 Functionality of performance measures
Bouckaert (1993) sees functionalityaflexibly in response to environmental changes

realistic measurement so that measured(Table 4). These responses were given on theperformanceforbasis of 5 points, from ‘1 (not at all) to 5

use. Therefore, when there is a difference(exactly)’. When constructing the variables,

between the initial performance measurethese average values were scored (α = .713)1.

measuresareasdetermine whether the target can be changed

suitablesetting and the actual situation, it isimportant to adjust the settings in accordance

2.4

with the actual situation; it is also importantorganisations

to determine whether the monitoring cancontributetowardfacilitating

Performanceofpublicsector

The use of PMS brings various benefits

thisto an organisation. Non-financial as well as

adjustment. The measures and targets infinancialpublic sector organisations are often inflexible,

performance assessment, as shown in many

which may not be conducive to adjustments

quantitative studies (e.g. Davis and Albright,

made

and

2004; Ittner and Larcker, 2001). However,environment. Smith (1995) refers to this type

public sector organisations cannot measureof

performance

performance using financial results such as

evaluation. de Brujin (2002) also argues thatsales and profit measures. Therefore, many

once performance measures are set, thepublic sector studies have relied on the items

continuous reproduction of existing services isof van de Ven and Ferry (1980), which areencouraged and the demand for change can beconsidered suitable for measuring public

ignored. If there is a change in the external or

sector performance, and determined theinternal environments of organisations, thenaccuracy of productivity and operations.

the measures must be modified appropriately,

Measurements were made using the following

In order to change the functionality of
performance measures, we used fourquestions. However, when the question item
‘Change of target during the period is notallowed (reversal scale)’ was included,
Cronbach's α was .596, which made the scaleless reliable. Therefore, variablisation was
performed, which excluded this item.

according

risk

to

the

as ossification ofsituation1

36

results

have

been

used

for

Journal of Japanese Management Vo.4, No.1, November 2019ISSN 2189-9592

Table 4 Functionality of performance measures
Range

Mean

S. D.

1–5

2.88

1.175

1–5

3.15

.982

1–5

3.14

1.010

FNCT_1 Targets are updated regularly according to certain criteria
When setting targets, adjustments are made throughout the year
FNCT_2 for situations that could not have been foreseen when setting
targets unforeseen events.
Ongoing monitoring was conducted within the organization to
FNCT_3 assess whether targets are realistic or whether they should bechanged.

seven items: productivity, accuracy of work,

efficiency, and improvement of residents’

an increase in the number of innovative newsatisfaction (α = .826). The second factor was

projects, satisfaction from and reputation ofnamed ‘Behavioural performance’ 2 because

the work, the achievement of targets, work

items related to the impact on staff behaviour,

efficiency,

staff

such as cost awareness and motivation,

motivation towards work. In this study, we

showed a high factor loading (α = .806). The

added six items, such as budget adjustment

third factor was termed ‘Loss of usefulness’and cost awareness, which were measured bybecause items related to the loss of usefulness

major PMS studies such as Hall (2008).

of PMS—such as the fact that performance

Additionally, it has been pointed out thatmeasuresare

PMS yields both positive and negative

indicateorganisational

outcomes. Therefore, we also measured theresidents and parliament—indicated a highseven negative outcomes emerging from use

factor loads (α = .707). The fourth factor wasof PMS, as described by Cuganesan et al.

named ‘Side effects’ because items restricting(2014). We conducted an exploratory factor

communication within the oraganisation,

analysis (using the main factor method andsuch as the inhibition of organisational

promax rotation) on the performance of public

flexibility

sector organisations, based on these 20

oraganisation, showed a high factor loading (αquestions. We identified the elements that= .704). Additionally, in the variabilisation ofcontribute to the performance of public sector

each item, the average value of the question

organisations. As a result, as shown in Table

items that showed a factor load of .35 or more

5, it became clear that the performance ofwas scored.

and

improvementin

underutilised

and

and

poorlyperformance

wrinkleswithintothepublic sector organisations is composed offour factors. The first factor was named ‘Task

2.5 Control variable

performance’ because it comprised the result

In order to control the analysis results,of work activities, such as work quality,

the number of employees was set as a control

Items that make up the second factor
include items that may not be directly linked
to behavioural performance, such as an‘increase in the number of innovative new

project. However, these items are also
considered to be indirectly generated bybehavioural changes, and hence the second
factor is named ‘behavioural performance’.

237

Journal of Japanese Management Vo.4, No.1, November 2019ISSN 2189-9592

Table 5 Factor analysis on the performance of public sector organisations

EFT_2 Quality or accuracy of work
Improvement in the achievement
EFT_5 of business and departmental
targets
Improvement in operational
EFT_4
efficiency
Increase in business volume
EFT_1 (production volume) and service
provision
Improvementin
informationcollection and recording methods
EFT_12formeasuring
outputandoutcomes
Increase in the satisfaction and
EFT_6
reputation of residents and users
Coordination of Budgets and plan
EFT_8
with other departments
Information exchange with staff in
EFT_13other departments
Reduction in the time incurred for
EFT_11 making decision associated withtargets, policies, and work schedule
Improvement in cost reduction
EFT_9
awareness
Improvement staff motivation
EFT_7
towards work
Increase in the number of
EFT_3
innovative new business projects
Performance measures and related
EFT_15
information are underutilised
Performance measures are not tied
EFT_16to organisational targets
Performance measures tend to be
EFT_14 poor measures of performance toresidents and parliament.
Performance measures are not
EFT_18 designed to reflect business results
or workload
Measurement and evaluation of
performance measures has led to
EFT_17 the creation of traps and excessivecompetition
withintheorganisation
Measurement and evaluation of
performance measures has led to
EFT_19
reduced organisational flexibility
and disrupted change
Eigenvalue
Factor correlation

Range

Mean

1–5

3.41

Factor1
Factor2
Factor3 Factor4
Task
Behavioural Loss of
Side
S.D.
performanceperformance usefulness effects(α= .826） (α= .806） (α= .707） (α= .704）
0.683
0.861
-0.1490.036-0.008

1–5

3.65

0.694

0.755

-0.005-0.008

0.008

1–5

3.47

0.749

0.611

0.203

0.105

0.017

1–5

3.18

0.685

0.537

0.063

0.063

-0.034

1–5

3.23

0.732

0.421

0.222

-0.039-0.015

1–5

3.21

0.751

0.398

0.174

-0.116

0.065

1–5

3.25

0.831

-0.116

0.805

0.028

-0.067

1–5

3.17

0.778

-0.0330.679-0.021

0.068

1–5

3.01

0.750

0.004

0.574

-0.053-0.040

1–5

3.38

0.764

0.160

0.570

0.014

0.015

1–5

3.18

0.713

0.213

0.432

-0.147

0.011

1–5

2.88

0.727

0.228

0.428

0.085

-0.011

1–5

2.91

0.891

0.082

-0.0480.816-0.070

1–5

2.69

0.889

-0.027-0.0160.753-0.025

1–5

2.45

0.887

-0.275

0.165

0.485

0.117

1–5

2.80

0.862

0.154

-0.050

0.400

0.127

1–5

1.99

0.698

-0.0040.039-0.021

0.789

1–5

2.29

0.752

-0.010-0.082

0.089

0.662

6.068
―
0.670
-0.562-0.240

1.858

1.260

1.008

Factor1
Factor2
Factor3
Factor4―-0.545-0.111

―
0.427

―

38

Journal of Japanese Management Vo.4, No.1, November 2019ISSN 2189-9592

variable; this is because items such asloading in the item indicating whether

environmental

uncertainty,

business procedures and processes werecomplexity,

organisation

expectedandtoaffectthebusiness
scale

are

clarified by manuals and regulations. The

performancesecondfactorwas

named‘Measurable

management of public sector organisations.

outcome’ (α = .714) because it showed a highRegarding environmental uncertainties andfactor loading on the items related to theoperational complexity, we referred to Speklé

measurable outcome of the department (lower

and Verbeeten (2014) and Oura and Matsuo

part of Table 6). Additionally, in the(2017)

conduct

variabilisation of each item, the average value

exploratory factor analysis (main factorof the question items that showed a factor

method and promax rotation)3. Additionally,load of .35 or more was scored. Finally, inwhen conducting factor analysis, items fororder to control the scale of the organization,

which the ceiling effect was confirmed andthe value obtained by logarithm conversion of

items that showed only factor loadings

the number of employees is used in the

below .35 for all factors were deleted. As aanalysis.

result,

to

the

set

questionsfirstfactoranddenoting

theuncertainty of the work environment had a(4) Analysis resultshigh factor loading in the item indicating
uncertainties

relatedtothe

Based on the above variables, wefutureconducted a multiple regression analysis;

predictability of work, and therefore thisvalidity, legitimacy, and functionality offactor was termed ‘Task predictability’ (αperformance= .666). The second factor was named

independent variables, and the performance‘Business design difficulty’ (α = .776) becauseof public sector organisations was taken as ait showed a high factor loading in the items

dependent

indicating difficulty in predicting needs anduncertainty, operational complexity, andeffects related to business design (upper partscale (number of employees) were set asof Table 6). Although Cronbach's α ofcontrol variables. The correlation coefficient‘business predictability’, which is the first

between variables (Table 6) and the results offactor, is below the general standard of .70,

multiple regression analysis (Table 7) are as

there is no major problem in validity because

follows. As a result of the analysis, it became

it is close to .70. It was judged. The first factor

clear that any variable that makes up theof business complexity is ‘Business process

quality of performance measures affects theclarity’ (α = .790) because it had a high factor

performance of public sector organisations.

Speklé and Verbeeten (2014) do not perform
factor analysis for each question item on both
uncertainty of work environment and work
complexity, but they are variabilised by thesimple average method. However, as a resultof performing the same operation in thisstudy, the value of Cronbach's α became low,

and hence factor analysis was performed forenhancing the accuracy of the analysis.
4 The variance of inflation of the input
independent variable was confirmed, but the
maximum value was 1.325, and hence it isconcludedthattheproblemofmulticollinearity does not exist.

339

measuresvariables

4

were

.

taken

as

Environmental

Journal of Japanese Management Vo.4, No.1, November 2019ISSN 2189-9592

First, it improves operational results such as

performance measures: B = .217, β = .228, t =thevalidity

legitimacyfunctionality

of
of
of

performancemeasures,4.504; legitimacy of performance measures: B

performancemeasures,= .203, β = .262, t = 5.405; and performance

performancemeasures,measures functionality: B = .150, β = .242; tproductivity, and work efficiency (validity of=4.857).

Table 6 Factor analysis on the control variables
Uncertainty of the task environment

There is a high possibility thatENVDYN_1 unpredictable events will occur in daily
operations
It is difficult to predict when a new project
ENVDYN_2
will start
It is difficult to estimate the required
ENVDYN_4
budget for the following year accurately
Achievement of targets depends heavily
ENVDYN_3
on the external environment
When designing a business, it is difficult
ENVDYN_6
to predict needs
When designing a business, it is difficult
ENVDYN_7
to predict the effect
Eigenvalue
Factor correlation

Business complexity
There is an appropriate and efficient
ENVHOST_2
process for conducting business
There is a standard procedure that should
ENVHOST_1be referenced for addressing differences inbusiness execution
Procedures for conducting business are
ENVHOST_4
clearly shared among various departments
Procedures for conducting business are
ENVHOST_3stipulated through laws, regulations, andrules, among others
Quantitatively understand what the
ENVHOST_6
department should achieve
A single quantitative measure can grasp
ENVHOST_5
the results of each businessCurrent performance measures accurately
ENVHOST_7represent what the department should
achieve
Eigenvalue
Factor correlation

Factor2
Business
designdifficulty(α=.776)

Range

Mean

S.D.

Factor1Task
predictability(α=.666)

1–5

3.11

0.888

0.767

-0.130

1–5

3.11

0.965

0.702

-0.001

1–5

2.76

0.945

0.410

0.238

1–5

3.28

0.779

0.382

0.108

1–5

2.72

0.739

-0.058

0.797

1–5

2.84

0.804

0.069

0.794

2.376
―
0.311

1.343

Factor1
Factor2―

Factor1
Factor2
Business
Measurable
process clarity
outcome(α=.790)(α=.714)

Range

Mean

S.D.

1–5

3.02

.930

.794

.010

1–5

2.93

1.067

.757

-.019

1–5

3.26

.902

.673

.071

1–5

3.21

.982

.534

-.001

1–5

2.88

.937

-.060

.977

1–5

2.53

.956

.027

.656

1–5

3.23

.781

.189

.362

3.104
1.281
Factor1―
Factor2.447―
ENVDYN_1 to ENVDYN_7 asked about uncertainties in the work environment and measured them on a five-point
scale (‘“1 not at all’” – ‘“5 exactly’”). ENVDYN_5 was removed for factor analysis because both factors showed factor
loadings less than .35. ENVHOST_1 to ENVHOST_10 asked about the complexity of the business and measured it
on a 5-point scale (‘“1 not at all’” – ‘“5 exactly’”). Since ENVHOST_9 confirmed the ceiling effect, ENVHOST_8 and
ENVHOST_10 showed factor loadings less than .35 for both factors, and hence were excluded from the factor analysis.

40Journal of Japanese Management Vo.4, No.1, November 2019ISSN 2189-9592

Table 7 Correlation coefficient between variables
Variables(1)TSK_PERF

1

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

(10)

(2)BEH_PERF.661** 1(3)RER_NPERF-.446** -.446** 1(4)SIDE_ NPERF-.218* -.157** .393** 1(5)VALID.419** .408** -.342** -.125* 1(6)LEGIT.419*** .342** -.289** -.282** .263** 1(7)FNCT

.415*.377** -.397** -.075.304** .284*

1(8)PREDICT

-.084

-.029

.112*

.023

-.165** -.081-.112* 1(9)DIFFICULT-.126* -.144** .118*.127*-.166** -.118* -.120* .292** 1(10)CLARITY.179** .257** -.216** -.035.283** .155** .293** -.282** -.191** 1(11)MEASURABLE.350** .309** -.250** -.069

.333*

(11)

.199** .244** -.237** -.177** .400** 1

1）Pearson's correlation coefficient
2）TSK_PERF: Task Performance，BEH_PERF: Behavioural performance ，RER_NPERF: Loss of usefulness,
SIDE_ NPERF: Side effects，VALID: Validity of performance measures，LEGIT: Legitimacy of performancemeasures，FNCT: Functionality of performance measures，PREDICT: Task predictability，DIFFICULT: Businessdesign difficulty，CLARITY: Business process clarity，MEASURABLE: Measurable outcome3）**p<0.01, *p<0.05 (two-sided)

Table 8 The results of multiple regression analysis
Behavioural

Task PerformanceΒ

t-valueβperformanceΒβ3.007*** .729

Loss of usefulness

t-valueΒ2.110**

5.299

βt-value

Side effectsΒ

β12.476*** 3.328

t-valueConstant term

.960

7.253***

VALID

.217

.228

4.504*** .243

.246

4.617*** -.209-.178-3.235*** -.045

-.040

-.658

LEGIT

.203

.262

5.405*** .159

.193

3.792*** -.123-.130-2.464** -.239-.257-4.444***

FNCT

.150

.242

4.857***

.127

.201

3.822***

-.197

-.260

-4.809*** .015

.020

.340

PREDICT

.032

.038

.775

.088

.103

2.009** .016

.015

.289

-.036

-.037

-.626

DIFFICULT

-.016

-.021

-.432

-.041

-.053

-1.055

.017

.018

.347

.095

.105

1.835*

CLARITY

-.045

-.065

-1.253

.049

.069

1.259

-.019

-.022

-.393

.031

.038

.605

.096

.129

2.343**

-.088

-.098

-1.738*

-.013

-.015

-.238

.016

.024

.494

-.103

-.130

-2.607* -.063-.081-1.476

MEASURABLE

.143

.195

3.749***

Number of staff

.030

.047

1.010

R2

.368

.306

.261

.097

Adj. R2

.352

.288

.241

.074

F-value

22.500***16.881***

13.531***4.154***

1）Estimation based on ordinary least squares (OLS) method2）B: Non-standard regression coefficient ，β: Standard regression coefficient ，R2: Coefficient of determination ，Adj. R2:
Modified coefficient of determination3）*** p < 0.01, ** p < 0.05, * p < 0.1（two-sided）

Additionally,animprovementin

β = .193, t = 3.792; and performance measurespositive outcomes was observed, such asof functionality: B = .127, β = .201; t = 3.822).

improvement of staff motivation and the

As pointed out by Bouckaert (1993) andactive exchange of information with other

van Dooren et al. (2015), it is important tolegitimacy of performance measures: B = .159,

improve the quality of performance measureswhen designing and using PMS in public
41

Journal of Japanese Management Vo.4, No.1, November 2019ISSN 2189-9592

sector organisations. This suggests that-.178, t = -3.235; legitimacy of performance

improving the validity, legitimacy, andmeasures: B =-. 123, β = -.130, t = -2.464;

functionality of measures improves theand functionality of performance measures: B

performance of public sector organisations. In=-. 197, β = -.260; t = -4.809). Additionally,

other

and

legitimacy of performance measures, that is,using PMS for public sector organisations,staff participation in setting and using

the performance measures are designed in a

performance measures is also supressed sidemanner that they are not distorted byeffectsexternal influences. It is essential to build acompetition

mechanism that will allow managers andflexibility of organisation (B = -.239, β =non-managerial staff to participate in the-.257; t =design and use of performance measures; thisal. (2014) mentions the possibility that themechanism

will

contributetowardsuse of PMS in public sector organisations not

improving

task

and

behavioural

only produces positive outcomes but also

performance. Groen et al. (2017) clarified thatnegative outcomes, it has not been revealed

the involvement of operational employees inthat how to suppress these negative outcomesthewords, when

designdesigningandof PMS use, such
in

the

body

-4.444). Althoughasexcessive

andreducedCuganesan etimplementationofof PMS. In this respect, an improvement inmeasures improves thejob

the quality of performance measures, such as

performance of employees. It can be said thatvalidity, legitimacy, and functionality of

thisperformance measures may suppress theperformanceisconsistent

withprevious

studies. Additionally, the measurability ofnegative outcomes.

outcomes affects the relationship between

In public sector oraganisations, once

each variable related to the quality of

performance measures are set, they are often

performancenot reviewed for reasons such as year-on-

measures

and

task

and

behavioural performance. In other words, it isyear comparisonssuggested that it is necessary to giveother organisations. Ensuring comparability

sufficient consideration to the quality ofis,performance measures when the business or

accountability of the government. But leaving

project allows the quantitative measurement

improvements of quality of performanceof the results of each department or

measures can also lead to poor performance of

project. These variables related to the quality

public sector organisations. Therefore, it isof performance measures not only increase

necessary

positive outcomes but also show the potential

performance measures periodically to sustain

to suppress negative outcomes. In otherthe quality of the performance measures.

ofcourse,toandcomparisons

importantverify

and

in

with

fulfilling

reviewthewords, the validity, legitimacy, functionalityof performance measures suppresses negative(5) Contribution of this research and future

outcomesissuesofperformance

management

systems such as loss of usefulness (validity of

Although previous studies pointed out

209, β =that the quality of performance measures has

performancemeasures:

B

=-.

42

Journal of Japanese Management Vo.4, No.1, November 2019asignificant

impacton

ISSN 2189-9592

organizationalquestion items by relying on the literature on

performance (van Dooren et al., 2015), thenegative outcomes of PMS use, such as therelationship between the two has not been

study by Cuganesan et al. (2014). There is aclarified quantitatively. In this regard, thispossibility that the negative outcomes may

study presents empirical evidence that the

not be fully understood. The points comprise

validity, legitimacy, and functionality of

the limitations of this research, which must

performancebe evaluated in future research.

measuresincreasepositiveoutcomes. This is the first contribution of thisstudy.

Additionally,

althoughnegative

Acknowledgements

outcomes from the use of PMS have beenThis paper’ original version (Metoki, 2018)

pointed out (Cuganesan et al., 2014), thewon the Best Paper Award at the Japan Costmeans to suppress them have not been

Accounting Association (JCAA) and waspresented. This study addresses this gap by

publisheddemonstrating that negative outcomes can berecommendation of the president. The author

suppressed by improving the quality ofwould like to thank the president of JCAA,

performance measures, that is, the validity,

and two anonymous reviewers of Journal of

legitimacy, and functionality of performance

Japanese Management.

in

this

journalwiththemeasures. However, while making the abovementioned contributions, it must be pointed

References

out that this research has its limitations. This

Abernethy, Margaret A. and Jan Bouwens.

study analyses the quality of performance(2005).

measures by focusing on the

Innovation Implementation. Abacus, 41

validity,

legitimacy, and functionality of performance

DeterminantsofAccounting(3) , pp.217-240.

measures presented by Bouckaert (1993).

Bevan, Gwyn and Christopher Hood. (2006).

Additionally, it attempts to grasp the quality

What's Measured is What Matters:of the performance measures by considering

Targets and Gaming in the English

the validity of the construct, as much as

Public Health Care System. Public

possible,

Administration, 84 (3), pp.517-538.

relyingonprevious

research.

However, since the concept of quality of

Bouckaert, Geert. (1993). Measurement andMeaningful

elements, this study does not capture allmeasures’

Productivity and Management Review,
17, pp.31-43.

measurement,

Bourne, Mike. John J. Mills, Mark Wilcox,

importance, measurement cost, sensitivity,

Andy Neely, and Ken Platts. (2002).

and noise discussed in Kawai (2016) are

Designing, Implementing and Updating

considered to be factors related to the quality

Performanceof performance measures in public sector
organisations. Additionally, when measuring

International Journal of Operations andProduction Management, 20 (7), pp.754-

negative outcomes, the study formulated

771.

aspects.

For

characteristics

example,
such

as

the

43

Management.

Publicperformance measures is composed of various

Measurement

Systems.

Journal of Japanese Management Vo.4, No.1, November 2019

deBruijn,Hans.

(2002).

Management Accounting Research, 36,
pp.51-66.

Performance

Measurement in the Public Sector:
Strategies to Cope with the Risks of
Performance

ISSN 2189-9592

Hall,

Measurement.

Matthew.

(2008).

The

Comprehensive

Effectof

Performance

International Journal of Public Sector
Management, 15 (7), pp.578-594.

Measurement Systems on Role Clarity,

Cavalluzzo, Ken S. and Christopher D. Ittner.

Managerial Performance. Accounting,(2004).

Implementing

Measurement

PsychologicalandOrganizations and Society, 33 (2/3),

Performance

Innovations:

Empowerment

Evidence

pp.141-163.

Accounting,Heinrich, Carolyn J. (1999). Do Government

Organizations and Society, 29 (2), pp.243267.

Bureaucrats Make Effective Use of

Cuganesan, Suresh. James Guthrie andJournal of Public Administration
Research and Theory ,9 (3), pp.363-393.

fromGovernment.

Performance Management Information?

Vedran Vranic. (2014). The Riskiness of
PublicSectorPerformancevan Helden, G. Jan and Christoph Reichard.

Measurement: A Review and Research(2013). A Metareview of Public Sector

Agenda. Financial Accountability and

Performance

Management, 30 (3), pp.279-302.

Tékhne-Review of Applied Management

Management

Research.

Investigation of the Effect of Balanced

Studies, 11, pp.10-20.
Ittner, Christopher. D. and David F. Larcker.

Scorecard Implementation on Financial(2001). Assessing Empirical Research in

Performance. Management Accounting

Management Accounting: A Value-based

Research, 15 (2), pp.135-153.
van Dooren, Wouter. Geert Bouckaert andPerspective. Journal of Accounting and

Performance
Public Sector.

Kawai, Takaharu. (2016). Gyouseki Sokutei

London and New York: Routledge and(Performance Measuring Objects and Its

Taylor and Francis Group.

Properties). Journal of Cost Accounting

Davis, Stan and Tom Albright. (2004). An

John

Halligan.

Management

Economics, 32, pp.349-410.

(2015).

in theno Sokutei Taishou to Shihyou Tokuseiin Japan, 40(2), pp.87-99.
Kravchuk, Robert and Ronald Schack. (1996).

Grafton, Jennifer. Anne M. Lillis and Sally K.
Widener. (2010). The Role of Performance
Measurement

andEvaluationin

Designing

Effective

Building Organizational Capabilities andMeasurement

Performance. Accounting, Organizations

Government Performance and Results

and Society, 35(7), pp.689-706.

Act 1993. Public Administration Review,

Groen, Bianca. A. C. Marc. J. F. Wouters andSystems

Performance
under

the

56 (4), pp.348-358.

Celeste P. M. Wilderom. (2017). Employee

Matsuo, Takami. (2009). Jichitai no Gyouseki
Kanri

Job Performance: A Survey Study Basedmanagementongovernment). Chuokeizai-sha, Inc. Japan.

Self-determination

Theory.
44

Shisutemu(Performance

Participation, Performance Metrics andsystemsinlocal

Journal of Japanese Management Vo.4, No.1, November 2019

Departments. Management Accounting

Melnyk, Steven. A., Umit Bititci, Ken Platts,

Research, 15, pp.247-266.

Jutta Tobias, and Bjørn Andersen. (2014).
Is

Performance

Management

Measurement

Fit

for

the

and

Oura, Keisuke and Takami Matsuo. (2017).

Future?

ChihouManagement Accounting Research, 25(2), pp.173-186.

Management

Information

Effectiveness

in

and

Japanese

Its
Local

Government). Journal of Cost Accounting

in Japan, 41(1), pp.103-115.
Pollitt, Christopher. (2006). Performance
Management in Practice: A Comparative

performance management systems in

Study of Executive Agencies. Journal of

local government). The Journal of Cost

Public Administration Research andAccounting Research, 42(9), pp.97-109.
Metoki, Takehiro. (2012). Gyouseki Kanri

Theory,16 (1), pp.25-44.
Smith, Peter. (1995). On the Unintended

Shisutemu to shite no Gyousei Hyouka (AConsequences of Publishing Performance

Survey of Performance Management in

Data in the Public Sector. International

The
Journal of Musashi University, 60 (2),Journal of Public Administration,18 (2/3),
pp.277-310.

Local

qualityniokeruonJapanesemeasures

DanntaiYuukousei (The Use of Performance

Takehiro.

performance

Koukyou

Gyouseki Kanrij Youhou no Riyou to(2018). Gyouseki
Shihyou no Shitsu ga Gyousei Soshiki
no Pafoumansu ni ataeru Eikyou:
Sitsumonhyou Cyousa ni Motozuku
Teiryou Bunseki (The effects ofMetoki,ISSN 2189-9592

Government).

pp.45-72.

Speklé, Roland F. and Frank H. M. Verbeeten.

Metoki, Takehiro and Takeyoshi Senoo.

(2014).

The

Use

of

Performance(2012). Examining Managerial Uses of

Measurement Systems in the Public

Accounting

andSector:Measurement

Information

Performanceinin Japan, 36 (2), pp.115-129.
Moers, Frank. (2006). Performance Measure

Performance.

Van de Ven, Andrew. H. and Diane L. Ferry.
(1980).

The
Accounting Review, 81 (4), pp.897-924.
andonManagement Accounting Research, 25(2), pp.131-146.

Local

Government. Journal of Cost Accounting

Properties

Effects

MeasuringandAssessingOrganizations. Wiley, New York.

Delegation.

Newberry, Susan and June Pallot. (2004).
Freedom or Coercion? NPM Incentives in
New

ZealandCentralGovernment(Received: July 31, 2019)(Accepted: November 1, 2019)45