# KPI-Supported PDCA Model for Innovation Policy Management in Local Government

Antonio Candiello, Agostino Cortesi

► To cite this version:

Antonio Candiello, Agostino Cortesi. KPI-Supported PDCA Model for Innovation Policy Management in Local Government. 10th Electronic Government (EGOV), Aug 2011, Delft, Netherlands. pp.320-331, 10.1007/978-3-642-22878-0_27. hal-01589101

HAL Id: hal-01589101

https://inria.hal.science/hal-01589101

Submitted on 18 Sep 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# KPI-Supported PDCA Model for Innovation Policy Management in Local Government*

Antonio Candiello, Agostino Cortesi

Dipartimento di Informatica, Università Ca' Foscari,
Venice, Italy
{candiello,cortesi}@unive.it

**Abstract.** A comprehensive model for the management, monitoring and assessment of the innovation projects implemented by the local government is presented. The model is based on the classic Deming PDCA quality-oriented process. It is defined in collaboration with our local government partner in order to measure the effective impact of the innovation policies developed by the public administrations. An "eGovernment Intelligence" framework has been designed and is currently being developed and tested. The main features are: (a) the qualification of the policies/projects and the definition of innovation targets, (b) a systematic and staggered measurement of the relevant innovation, economic and social indicators at the needed scale, (c) a detailed, geo-referentiated analysis of the territorial evolution pattern of the indicators, (d) the reassessment of policies and projects against the results obtained.

**Key words:** eGovernment, Business Intelligence, Quality Management.

## 1 Introduction

A key issue for local governments aiming at promoting digital skills and supporting social and economic developments, is the combination of eGovernment policies and ICT policies. The effects of correlated innovation projects should be measurable via focused analysis of specific statistical indicators [1, 2]. These can be either direct eGovernment or ICT indicators (as is the case, for instance, of portal/online services access, wide band internet coverage of population, availability of PCs, eCommerce transactions, etc) or indirect/impact socio-economical indicators (as, for instance, average income, local GDP growth, availability of qualified software engineers and so on).

The impact analysis for general ICT availability and specifically for publicly funded ICT initiatives is a difficult task. Dedicated studies have been conducted in order to analyze the impact of ICT in education [3], local area [4] long-term

<footnote>* Work partially supported by Regione Veneto – Direzione Sistema Informatico.</footnote>
economy [5], social capital / quality of life [6] and democracy [7]. Government
monitoring [8] has also a direct impact on democracy and transparency, offering
citizens a direct control of local administration's projects/actions.

ICT innovation policies implemented by local government and correlated
projects should be evaluated [9] in an adequately wide time-span, covering in
some cases several years, and the related impact measurements should follow
this pattern, by registering at definite temporal intervals the state of the mon-
itored indicators. A comprehensive strategy for medium-term impact measure-
ment, combined with a thoughtful choice of both direct ICT indicators and
more general socio-economical indirect impact indicators could help to out-
line the searched correlations between indicator changes and the effect of the
innovation projects [10]. The scale of local government here considered is re-
gional/subregional, and the smallest administrative units are the municipalities.

Statistical key performance indicators (KPI) for sub-regional and sub-provincial
local territories can be obtained via three main sources:

1. by retrieving online indicators available from official Institutions like the International Eurostat, Worldbank and others, the corresponding National Institutions and even sub-national entities like the local Chambers of Commerce,

2. by harvesting the (raw) web with webbots, data scrapers, crawlers, spiders, searching for digital skill evidence trails left by the citizens and the business to build new classes of indicators,

3. by addressing citizens/enterprises specific surveys aimed at a better understanding of the actual use of the eGovernment services and translating them in indicators.

Each strategy has its pros and cons. The highest quality data is offered via
the first approach, which represents the basic reference thanks to its officiality
– and comparability. For instance, the *Nomenclature of Territorial Units for
Statistics* (NUTS), classification used by Eurostat – a three-level hierarchical
system for dividing up the economic territory of the Europe Community from
National to regional scale – allows the comparison of the socio-economical in-
dicators among EU sub-administrative units. However, these indicators are not
always provided in finer territorial detail than the regional one, corresponding
to NUTS-3 in Eurostat language; it is sometimes possible to scale down to the
provincial level, and sometimes to the municipal level, by accessing National
statistical data or, for business-related indicators, Chambers of Commerce pe-
riodic reports. The main problem with this approach is that the data officially
monitored are mainly socio-economical data. It is difficult to find systematic
measurements of ICT innovation indicators; these are, often, the result of occa-
sional specialized research surveys managed by interested Institutions.

The “raw” web webbot-based strategy is gaining relevance, due to the rapid
growth of the quantity and the quality of the information that is available on
the web. We expect an increasing trustworthiness of the measurements that
will overcome the critical point of this strategy, i.e. the reliability of web-related
indicators data. It is also interesting from a technical point of view, as it requires
to address the issue of extracting data from web sources that are dynamically
changing. The *webbots*, said also *data scrapers* when extracting data from a
single web source, or *crawlers*, *spiders* when moving between links finding for
the needed information [11], need to be updated when the source web sites
change their content or even their appearance. Indicators connected to social
community-related sources, like blogs, forums, social networks, extracted by web
raw data have the advantage to be continuously up-to-date.

Government-supported surveys offer the opportunity to select specific ICT
innovation questions, generally not inspected by official national/international
statistics – at a cost, however, of a consistent effort needed for the preparation,
communication and submission of the survey campaigns. Survey campaigns are
generally limited in time (campaigns rarely last for more than a year) and in
space (they are conducted frequently at a provincial or municipal level). Online
tools for survey submissions and a regional-level eGovernment infrastructure [12]
could help in maintaining some efficiency trade-off in the use of this information
channel.

The goal of our research is to explicit a comprehensive model for (a) the
management of eGovernment policies and ICT innovation policies, (b) their sys-
tematic monitoring and (c) the impact analysis measurement and to (d) support
the model with an integrated eGovernment Intelligence information system ca-
pable of registering and monitoring such policies, monitor the relative innovation
projects against their goals, systematically evaluating their impact and finally
reviewing the policies themselves on the basis of the resulting analysis.

We conducted our research in collaboration with our local government partner, Regione Veneto (in northern-east Italy).

The information system developed by Regione Veneto provides public ad-
ministrators capability to continuously improve their services via an objective
evaluation of the resulting impact; the local government stakeholders, citizens
and enterprises, on the other hand, could be better informed and up-to-date
regarding goals set in advance for the policies and the success rate of the local
government funded ICT initiatives carried out for the public benefit.

The paper is organized as follows. In Section 2 a comprehensive model for
policy management is introduced, and in Section 3 the supporting eGovernment
Intelligence framework, its modules and its interaction with the model are pre-
sented. Then, in Section 4, some conclusions are drawn.

## 2 A Model for the Management of ICT Innovation Policies and Projects in Local Government

A comprehensive [13] model for monitoring the ICT innovation projects, validat-
ing the related policies and evaluating the effective direct and indirect impact
on the areas affected could improve the success rate of the local government
initiatives [14, 15]. Policies and related projects, which we consider relating to
eGovernment [2] as well as related to the wider context of ICT infrastructures
[16] should be assessed also by the citizens themselves [17]. We adapted the classic Deming plan-do-check-act (PDCA) cycle to the Local Government requirements for ICT innovation policies management. Each policy management PDCA phase is identified by an organizational process and is supported by specific subsystems of the eGovernment Intelligence framework. The complete model is shown in Fig. 1.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** The comprehensive model for Policy Management, supported by the framework and tools of eGovernment Intelligence.</figcaption></figure>

The goals of this model are:

- finding an *objective validation* for the effectiveness of eGovernment and ICT innovation projects,

- *qualifying and quantifying* the effectiveness through appropriate impact statistical territorial indicators,

- *gathering the relevant indicators* via automatic (webbots/scrapers) and semi-automatic (extractors/wrappers), completing the data when needed with focused survey campaigns,

- *representing and mapping the indicators* showing the explicit relation with the affecting innovation projects and the areas involved.

We classify the indicators in two categories: (a) *direct* ICT indicators enabling the Information Society, connected to the technology – examples of indicators in this category are population reached by the internet wide band, eGovernment services offered, eGovernment use of services, ratio of PCs/users, ICT education knowledgeability; (b) *indirect* socio-economical indicators related to the resultant impact of the ICT main enablers over the local communities, participation, sociality, economy and culture.

Direct, ICT indicators are easier to manage, as they are strictly related to the ICT innovation projects. For instance, internet wide band penetration ratio is related to infrastructure funding by the local governments, while the growth of
the number of accesses to eGovernment Portals depends on quality of the offered online services. These indicators require however the setup of specific measurement processes, as ICT innovation evolution is not systematically monitored by the National or Regional Institutions dedicated to statistical data analysis.

Indirect, socio-economical indicators are more easily found in the periodic data reporting produced by National and International statistical or economical Institutions, but these are second-level correlated to ICT innovation projects, i.e. there is the need to estimate and then evaluate their effective correlation with the intermediate ICT indicators which can then be put in direct correlation to the monitored innovation projects. For instance, an investment for wide-area internet wide band could in the long-term sustain new business developments, widen social communities opportunities and in general improve the quality of life. The management of indirect socio-economical indicators requires however carefully staggered gathering of statistical data, and the estimation of the correlations between them and the “raw” ICT indicators.

In the current phase of the research, we concentrate our efforts in extracting direct ICT innovation and eGovernment indicators and in selecting simple cases for the socio-economical impact indicators without modeling the effective correlations between the two classes of indicators – we are leaving this task for subsequent research phases.

# 3 The eGovernment Intelligence Technical Framework

We developed the eGovernment Intelligence framework around the following elements:

- the reference myPortal/myIntranet regional Liferay-based eGovernment architecture supported by our local government partner,

- the policy manager, a GUI panel for the management of policies, projects, the selection of impact indicators and the setting of targets,

- the events scheduler for the reliable planning of monitoring events with a minimal temporal unit of a day,

- the impact monitor for the management of webbots/data scrapers, wrapers/adapters and for the launch of email/online survey campaigns,

- the geo-analyzer, a geo-referentiated visualization engine built around the SpagoBI open source platform.

Let us discuss them in detail.

## 3.1 An Extensible and Service Oriented Architecture for Local eGovernment

Our reference architecture is based on a dual model with an external interface towards citizens and a complementary internal interface dedicated to local (municipal) government staff, both interconnected and supporting a constellation of web services-based tools and applications.
The external, front-side of eGovernment is the *government-to-citizen* (G2C) domain, where web publishing is used to give information to citizens, to report news regarding tax procedures, laws as well as local informations about events; citizens browse the web searching for specific information. This is provided by the *myPortal* platform which unifies at the moment a hundred local public administrations.

The internal, back-side of eGovernment is the *government-to-government* (G2G) domain. The *myIntranet* project addresses this issue by selecting the appropriate technology in a service oriented architecture to better support internal collaborations. The myPortal/myIntranet framework has been consolidated in a WS-oriented architecture, open to specific dual-sided tools and applications.<fnref n="1" />

After the initial adoption stage, the project was extended from its initial community to all other local (municipal) communities willing to take advantage of it. This led to the definition of a de facto standard for eGovernment portals and intranets for small and medium administrations in the whole region. The current technological environment for both myPortal and myIntranet is based on the open source JSR286-compliant portlet container Liferay.

## 3.2 The Policy Manager

Innovation policies are defined and funded by local governments, and then developed into explicit actions/projects. Such projects have definite dates of deployment, they can be geographically localized and they have associated milestones. A policy manager has been designed to provide the system all the available data regarding policies, actions, projects. The policy manager is accessible via the myIntranet G2G backend (see Fig. 2). The policy manager is the tool that supports local government transparency by offering citizens a preventive declaration of intents for each funded policy, and allowing them to verify the effective achievement of the established goals.

## 3.3 The Events Scheduler

The scheduler is the core of the eGovernment Intelligence framework. It manages the execution of the daemons that scan the web, access the online repositories or launch the survey campaigns via the myPortal local municipality portals. The scheduler uses as the smallest temporal unit the day, and offers the possibility to activate programmed monitoring for timed events ranging from daily intervals, passing through weekly, monthly, quarterly and other sized periodic intervals. The reliability of the scheduler is a central requirement for the eGovernment Intelligence framework, as the analysis phases need complete data sets in order to produce trustworthy and significative statistical elaborations.

<footnote n="1">¹ See: myportal.regione.veneto.it</footnote>
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** PLAN: planning the ICT innovation projects and definition of targets.</figcaption></figure>

## 3.4 The Impact Monitor

A class of webbots (for online sources) and wrappers/adapters (for directly accessible sources) and relative subsystems have been designed in order to extract the validated, high quality indicators from official sources that expose structured data from International, National or subnational Institutions. As online data source, Eurostat offers the widest option choices for the gathering of structured data, from CSV to XML to several web interfaces and API to manage data; other official data sources are more limited in the choices they offer, sometimes even limited to a fixed-format PDF file sent via email. We developed specific extractors for the data reported by the National Statistical Institute collected along with other socio-economical data by the regional Statistical Office managed by our local government partner. Specifically, income, number of inhabitants/families, age distribution data is available year by year at the required territorial resolution of a single municipality. Several local Chambers of Commerce, which are distributed at a provincial level, entered in the research collaboration offering the availability of their periodic reporting regarding the evolution of enterprises, like the number of new firms, sales income, etc.

The indicators extracted via this first channel – extractors for official / institutional data retrieval – mainly of socio-economical nature, have the highest reliability. They also offer the advantage of being completely available at the municipality level. This is a low frequency channel, as the updates are typically released with an yearly periodicity. The scheduler has to be instructed to wake up the relative daemons at the right date when the updated data are available.
Webbots and data scrapers can also be useful tools to complete the highly reliability data of the first channel with information found on the general web. We experimented simple keyword-based searches via major search engines like Google and Yahoo! (by searching for ICT-related words combined to locality names) and with the same technique we extracted ICT production/consume indicators by inspecting Youtube, Flickr and other popular digital repositories. eGovernment indicators are also inspected (indirectly) via Yahoo! Sites, that counts the referring links to the selected web sites. We are currently experimenting the extraction of information from blogs- and forum-related web sites. These indicators are mainly used to estimate the general population ICT knowledgeability via the analysis of the content produced.

The indicators extracted via this channel – webbots and data scrapers for web data harvesting – have a weaker reliability, due to the nature of raw information they are derived from. On the other way, there is the advantage that this data can be tuned both with respect to the time (as even daily updates can be considered), constituting a *high frequency* channel, and in space (a municipality level is reachable).

Webbots/wrappers for online official data and webbots/scrapers for raw web data constitute the core of the impact monitor (see Fig. 3).

<figure><img src="image_4.png" /><figcaption>Fig. 3. DO: execution of innovation projects and KPI monitoring.</figcaption></figure>

As a third, complementary, input channel, we integrated in the monitor the eGif engine [18, 12]. The eGif tool already fits neatly in the myPortal/myIntranet dual model. The eGif G2G/myIntranet interface exposes a complete *survey editor* that allows designers to build arbitrary complex survey structures, including multiple choices, indented questions and different choices for statistical variables.
The indicators obtained via this channel – citizen feedback regarding eGovernment or ICT use – are costly for the effort required in the survey campaign management, but can be useful to complete the statistical analysis for specific themes/areas.

## 3.5 The geo-visualizer

The impact analysis of gathered data is managed with the support of a visualization engine (see Fig. 4). We selected the open source SpagoBI engine (see [19] for a comparing review with other open source BI platforms). The visualizer exposes the indicators data over the local territories it allows to localize the impact of the developed innovation projects, both for eGovernment [2] and for ICT infrastructures in general (see Fig. 5). We are currently experimenting extensions of the SpagoBI platform in order to be able to use also multi-dimensional geo-referentiated data patterns, as the *travelling time distance grid* research case that we tested for mountain-area municipalities.

<figure><img src="image_5.png" /><figcaption>**Fig. 4.** CHECK: verification of goal achievements via impact monitor analysis.</figcaption></figure>

The policy makers are then ready to pass to the *act* phase by reviewing and improving the ICT innovation policies (see Fig. 6). The cycle is then completed and can be restarted.

## 4 Conclusions

In this paper a comprehensive policy management model and its supporting eGovernment Intelligence framework has been presented; the model, drawn from
<figure><img src="image_6.png" /><figcaption>Fig. 5. ICT business presence in Regione Veneto, year 2010.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 6. ACT: review of ICT innovation policies and related projects.</figcaption></figure>
quality management methodologies, offers the capability to measure the local im-
pact of eGovernment and ICT infrastructure policies. The policy management
model and the coupled eGovernment Intelligence framework should help public
administrators in reviewing and improving the projects by inspecting the result-
ing impact in detail. Both are currently being tested in collaboration with our
local government partner.

A first set of twenty core indicators (ranging from socio-economical data like population, income, business presence, to ICT-related data related to education, eGovernment usage, user-produced content, wide band infrastructures), was also extracted from official Institutions and raw web sources. A complete data set of the indicators has been created for all of the 581 municipalities of Regione Veneto for the last four years; the results, reported on the SpagoBI-powered maps, are currently discussed with regional government staff and the relations with the local ICT innovation initiatives were analyzed. We are jointly beginning promotions and experimentations of the complete model cycle involving a limited number of local municipalities and selecting definite sets of policies and related KPIs – this test in the field should be completed roughly in a year. We are also working on extensions of this model to include *what-if* simulations that should allow policy makers the possibly to forecast the effects of the planned interventions.

References

1. Janssen, M.: Measuring and benchmarking the back-end of e-government: A participative self-assessment approach. In Wimmer, M.A., Chappelet, J.L., Janssen, M., Scholl, H.J., eds.: EGOV 2010. Volume 6228 of Lecture Notes in Computer Science., Springer, Heidelberg (2010) 156–167

2. Neuroni, A., Rascon, A., Spichiger, A., Riedl, R.: Assessing and evaluating value and cost effectiveness of e-government initiatives: Focusing the step of the financial evaluation. In Chun, S.A., Sandoval, R., Philpot, A., eds.: dg.o 2010. ACM Digital Library, Digital Government Society (2010)

3. Balanskat, A., Blamire, R., Kefala, S.: The ICT Impact Report – A review of ICT impact on schools in Europe. Technical report, European Schoolnet in the framework of European Commission ICT Cluster (2006)

4. Ciarli, T., Rabelotti, R.: ICTs in industrial districts: an empirical analysis on adoption, use and impact in the Biella textile district. Technical report, Contribution to the EAEPE Conference Maastricht, November 7-9, 2003 (2003)

5. Venturini, F.: The long-run impact of ICT. Empirical Economics **37-3** (2009) 497–515

6. M., G., Anderson, B.: The impact of local ICT initiatives on social capital and quality of life. Technical report, Chimera Working Paper 2006-6, Colchester: University of Essex (2006)

7. Wiklund, H.: A Habermasian analysis of the deliberative democratic potential of ICT-enabled services in Swedish municipalities. New Media Society **7** **701** (2005)

8. Sousa, A.A., Agante, P., Gouveia, L.B.: Governmeter: Monitoring government performance. a web based application proposal. In Andersen, K.N., Francesconi, E., Grönlund, A., van Engers, T.M., eds.: EGOVIS 2010. Volume 6227 of Lecture Notes in Computer Science., Springer, Heidelberg (2010) 158–165
9. Bernroider, E.W., Koch, S., Stix, V.: Elements of comprehensive assessments of it infrastructure projects in the austrian ministry of finance. In Andersen, K.N., Francesconi, E., Grönlund, A., van Engers, T.M., eds.: EGOVIS 2010. Volume 6227 of Lecture Notes in Computer Science., Springer, Heidelberg (2010) 84–91

10. Misuraca, G., Ferro, E., Caroleo, B.: Assessing emerging ict-enabled governance models in european cities: Results from a mapping survey. In Wimmer, M.A., Chappelet, J.L., Janssen, M., Scholl, H.J., eds.: EGOV 2010. Volume 6228 of Lecture Notes in Computer Science., Springer, Heidelberg (2010) 168–179

11. Schrenk, M.: Webbots, Spiders, and Screen Scrapers: A Guide to Developing Internet Agents with PHP/CURL. No Starch Press (2007)

12. Candiello, A., Albarelli, A., Cortesi, A.: An ontology-based inquiry framework. In Gangemi, A., Keizer, J., Presutti, V., Stoermer, H., eds.: DEGAS 2009. Volume 426 of CEUR Workshop Proceedings. (2008)

13. Ojo, A., Janowski, T.: A whole-of-government approach to information technology strategy management. In Chun, S.A., Sandoval, R., Philpot, A., eds.: dg.o 2010. ACM Digital Library, Digital Government Society (2010)

14. De', R., Sarkar, S.: Rituals in e-government implementation: An analysis of failure. In Wimmer, M.A., Chappelet, J.L., Janssen, M., Scholl, H.J., eds.: EGOV 2010. Volume 6228 of Lecture Notes in Computer Science., Springer, Heidelberg (2010) 226–237

15. Janssen, M., Klievink, B.: Ict-project failure in public administration: The need to include risk management in enterprise architectures. In Chun, S.A., Sandoval, R., Philpot, A., eds.: dg.o 2010. ACM Digital Library, Digital Government Society (2010)

16. Lampathaki, F., Charalabidis, Y., Passas, S., Osimo, D., Bicking, M., Wimmer, M.A., Askounis, D.: Defining a taxonomy for research areas on ict for governance and policy modelling. In Wimmer, M.A., Chappelet, J.L., Janssen, M., Scholl, H.J., eds.: EGOV 2010. Volume 6228 of Lecture Notes in Computer Science., Springer, Heidelberg (2010) 61–72

17. Tomkins, A.J., PytlikZillig, L.M., Herian, M.N., Abdel-Monem, T., Hamm, J.A.: Public input for municipal policymaking: Engagement methods and their impact on trust and confidence. In Chun, S.A., Sandoval, R., Philpot, A., eds.: dg.o 2010. ACM Digital Library, Digital Government Society (2010)

18. Candiello, A., Albarelli, A., Cortesi, A.: Three-layered qos for egovernment web services. In Chun, S.A., Sandoval, R., Philpot, A., eds.: dg.o 2010. ACM Digital Library, Digital Government Society (2010)

19. Golfarelli, M.: Open source bi platforms: A functional and architectural comparison. In Pedersen, T.B., Mohania, M.K., Tjoa, A.M., eds.: DaWaK 2009. Volume 5691 of Lecture Notes in Computer Science., Springer, Heidelberg (2009) 287–297