# Analysis of the Methodologies for Evaluation of E-Government Policies

Dalibor Stanimirovic, Tina Jukic, Janja Nograsek, Mirko Vintar

► To cite this version:

Dalibor Stanimirovic, Tina Jukic, Janja Nograsek, Mirko Vintar. Analysis of the Methodologies for Evaluation of E-Government Policies. 11th International Conference on Electronic Government (EGOV), Sep 2012, Kristiansand, Norway. pp.234-245, 10.1007/978-3-642-33489-4_20. hal-01543590

HAL Id: hal-01543590

https://inria.hal.science/hal-01543590

Submitted on 21 Jun 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Analysis of the methodologies for evaluation of e-government policies

Dalibor Stanimirovic, Tina Jukic, Janja Nograsek, and Mirko Vintar

University of Ljubljana, Faculty of Administration, Ljubljana, Slovenia
{dalibor.stanimirovic, tina.jukic, janja.nograsek, mirko.vintar}@fu.uni-lj.si

**Abstract.** Methodologies for evaluation of e-government policies do not provide enough valuable information to policy makers in conducting quality planning of e-government initiatives. Consequently, user acceptance of e-government services is below government anticipations, while the expected effects in terms of reducing costs and increasing the effectiveness of public administration are still in early stages. Paper presents an overview of existing methodologies for evaluation of e-government policies, identifies characteristics of recent evaluations and conceptualizes a theoretical framework for their comparative analysis. Analysis of more than 50 evaluation methodologies offers an insight into the current evaluation practice, enables detection of its deficiencies as well as their mitigation and could facilitate a significant contribution to more evidence-based evaluation of e-government policies.

**Keywords:** e-government policy, evaluation methodology, evaluation and development level, comparative analysis.

## 1 Introduction

Despite extensive research in the recent years [1-3] and considerable investments in the field; EU countries are investing approximately 2.2% of GDP in public sector ICT [4-6], the phenomenon of e-government remains ambiguous and still lacks a unified definition. OECD studies indicate that further e-government development is one of the most important factors of public sector rationalization, as well as faster countries' development [7-9]. E-government development so far has been marked by a large gap between supply and demand of public e-services in most countries, which can be prevailingly attributed to "politically driven" development rather than evidence-based evaluation and selection of e-government policies [10-12]. Some countries (e.g. Estonia) [13-15] have been accomplishing much better results in evaluation and implementation of e-government policies compared to several other countries with much higher investments. Past experience in the field and public finance trends evidently
require the development of methodologies<fnref n="1" /> for evaluation of e-government policies which could enable e-government decision-makers to conduct more qualified and quantified preparation, execution and evaluation of e-government policies – be it before or after their implementation (ex-ante or ex-post).

Despite the increasing number of evaluation methodologies, the numerous aspects of their study and comparison have largely been disregarded. They are basically too diverse and lack a unified and clear theoretical framework [12], [16-17], which would allow a comparison of differences between them. The latter arise from various reasons: different (EU, UN, Brown University, EIU etc.) and heterogeneous promoters (international, national, consulting, research institutions etc.) [12], diverse environments [18-19], various rationales and contextual background as well as the number and selection of indicators [18-19]. Significant differences between evaluation methodologies are reflected within their main evaluation focus and evolving stage as well.

The paper is trying to overcome these limitations and establish the rudiments for theoretical framework which could facilitate a comparative analysis of existing methodologies in the field. Deriving from the aforementioned research objectives the paper is focusing primarily on the following interrelated research questions:

1. Overview of the existing methodologies for evaluation of e-government policies.

2. Identification and characterization of the key evaluation levels within e-government policies.

3. Analysis of existing methodologies for evaluation of e-government policies according to identified evaluation levels and development levels.

The research is based on the study of abundant literature, relatively scarce research reports available from the field, and an in-depth analysis of the methodologies which have been already implemented in practice. Paper essentially represents a review and comparative analysis of the methodologies for evaluation of e-government policies. The research was conducted within the research project aiming to determine which methodologies could be applied for evaluation of e-government policies in Slovenia.

## 2 Methodologies for evaluation of e-government policies – state of the art

According to the subject of evaluation, methodologies could be classified in typical groups presented below.

### 2.1 Front-office maturity and readiness

The best-known benchmark measurements in EU have been conducted by Capgemini [4-6], while the most renowned benchmarkings on the global scale have been carried

<footnote n="1">¹ The collective term “methodologies” will be used hereinafter, denoting approaches, indicator models, measurement frameworks and similar undertakings for evaluation of e-government policies.</footnote>
out by the UN [14], [20], Accenture [21] and Brown University [22]. While focusing primarily on web site analysis (front office), all these methodologies used completely different indicators, hardly ensuring comprehensive evaluation of e-government policies on the national level [12], [16], [23]. While other important benchmark measurements converging on e-readiness and information society in general are: The Global Information Technology Report [24], Digital economy rankings [25] and United Nations e-Government Survey [14], [20].

## 2.2 Effects and impacts of e-government policies

Ex-ante and ex-post evaluations of e-government policies are subject of numerous methodologies, among which we could highlight: MAREVA [26], eGEP [19], WiBe 4.0 [27] and Australian AGIMO [18]. MAREVA and WiBe 4.0 are dealing with ex-ante and ex-post evaluations of e-government policies on the basis of parameters such as profitability, risks, benefits to external users and civil servants, services and project necessity. eGEP and AGIMO similarly analyze costs, related risks, provision and maintenance of e-services, as well as evaluate their performance and impacts.

Implementation of e-government policies requires revision of the sourcing issues [28], careful scrutiny of the complex outsourcing implications [29], [30] and provision of indicators for objective evaluation of outsourcing process [31].

Given the complex effects of e-government policies on public sector organizations, research is engaged in analysis of joined-up e-government model [32], organizational changes in the direction of network government [33], management and external factors which affect e-government development [34], business process change, information management capacity, organizational capabilities and culture [28], [35], [36].

## 2.3 National-level development

National-level development is partially discussed in United Nations e-Government Survey [14], [20] through indicators such as e-participation, e-inclusion and e-consultation. Martin and Byrne [37] focused on critical factors of information society development providing a set of indicators for evaluation of e-government such as accessibility, digital divide, human rights, social inclusion, economic sustainability and life-long learning. Economic activities on national level could significantly affect e-government development in individual country. Scarce research [11], [38], [39] is specifically emphasizing correlation between national economic indicators (GDP per capita, competitiveness, use of ICT in the private sector, innovation index and internet access) and e-government development on the national level and on the EU level [38].

## 2.4 Evaluation of e-government policies – issues and barriers

Evaluation of e-government policies is generally difficult [5], [9], [16], [23], given the frequent lack of clarity of objectives owing to the different and often competing views held by different stakeholders. Effective evaluation requires good metrics, regular
monitoring and reporting, disciplined use of robust evaluation frameworks and long-
term evaluation practice largely depending on overall evaluation culture [40], [41].

3 Key evaluation levels within e-government policies

Overview of evaluation methodologies revealed they are focused predominantly on service level, while there are only a few methodologies, which could be actually applied for evaluation of e-government policies and decision-making at higher levels. Methodologies are generally partial and mostly focused on evaluating changes that occur in the “front-office” operation, while “back-office” changes caused by ICT have largely failed to gain significant attention. Existing research facilitates extraction and synthesis of the key evaluation levels (Fig. 1) which are described below. Pyramid structure of the model indicates the direction of policy-making process and assumes hierarchical relationships between individual levels.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Five-level model for evaluation of e-government policies</figcaption></figure>

**3.1 Infrastructural level**

Infrastructural level primarily refers to maturity or environmental readiness for e-
government and e-commerce. Research in this area is focused either on the internal or
external aspect of e-government. Internal aspect research is primarily engaged in [42-
43]: development strategies, policies and action plans, legal frameworks, the existence
and use of appropriate information infrastructure, training of human resources,
knowledge management, financial issues, motives and obstacles for the development
of e-government. Research on the external aspect of the environment maturity is par-
ticularly concerned with [42-43]: ownership, user interest and degree of e-government
service usage and issues related to the general development of e-government.
## 3.2 Project level

Research at project level is primarily engaged in: 1) ex-ante evaluations of projects aiming to establish priorities for further development, 2) ex-post evaluations of projects aiming to evaluate the effects of projects and 3) decisions on the insourcing and/or outsourcing of projects. Regarding the first two points, a review shows that methodologies of this type often underestimate public benefits (public value) and hidden costs, such as costs of organizational change. Research implies significant advances in outsourcing of ICT projects. Studies [31], [44-45] often reveal the hidden costs, vendor-lock in and loss of competencies as the most problematic issues, rarely dealing with the other potential negative consequences of outsourcing [30], [46-47].

## 3.3 Organizational level

Studies dealing with changes in the organizational structure are focusing on the reduction of hierarchical levels, decentralization, standardization, coordination and transformation of the existing organizational relations [32], [48-49]. Research dealing with business process reengineering is analyzing horizontal integration of functions and services, vertical integration of organizations, information exchange, changes in time and place of operation [35], [50-51]. Research exploring the changes in organizational culture is primarily dealing with: changes in the organizational philosophy and strengthening the sense of affiliation and confidence [34], [52]. Changes in human resources refer to the new skills, knowledge and specific managerial abilities [36].

## 3.4 Political-sociological level

Proliferation of ICT and development of e-government have changed the social structure and political-sociological paradigm of the social community [14], [53]. Complex political-sociological effects of ICT and e-government have a significant impact on the social environment; they are affecting old and creating new forms of work and changing perception of the world and social relations [54-57]. Accordingly, existing methodologies are converging on the following aspects of e-government evaluation: accessibility [7], [20], [52], citizens' trust and confidence [21], [58-59], digital divide [7], [40], [24], [53], social stratification and cohesion, human rights and democratic participation [8], [15], [37], openness, transparency and corruption [6], [14], [20].

## 3.5 National level

Research reveals that economic activities on the national level significantly affect e-government development, exposing GDP per capita as the most influential economic indicator [38-39]. Sing et al. [39] assume that GDP plays a crucial role in the development of e-government via three influential factors (technological infrastructure, human capital and management index). Other prospective indicators occasionally overlapping with political-sociological indicators are [60-61]: competitiveness, use of ICT in the private sector, innovation index [38], education and urbanization [11].
## 4 Analysis of existing methodologies for evaluation of e-government policies

The review of existing methodologies was conducted in the second half of 2011. During that time the research team scanned journals and conference proceedings, books, reports of international organizations and other institutions, policy papers, development strategies and other related documents containing e-government related research. Focusing particularly on measurement, assessment and evaluation of e-government policies and their effects we identified more than 50 relevant references. The frequency of references is becoming much higher in the second half of the last decade, proving the field is evolving rapidly and attracting more interest. Taking into account development level of evaluation methodologies, we have identified basically three types of references: 1) purely theoretical papers aiming to develop some kind of conceptual framework for evaluation of e-government policies, 2) research efforts developed up to the degree of pilot application and 3) methodologies developed in the practice for the practice (practical application). Further on, evaluation methodologies have been classified according to the identified evaluation levels, using the serial number of methodologies from the list of references (Fig. 2).

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Classification of methodologies according to the evaluation levels and development levels</figcaption></figure>

Analysing the diverse variety of evaluation methodologies identified in this area, certain general characteristics were identified and summarised below:
* Majority of the identified methodologies for evaluation of e-government policies are presented in scholarly papers and books.

* Small number of methodologies is appearing in the form of specific handbooks, some of which include a tool for evaluation of e-government policies, for example WiBe 4.0 or VAST (software packages, Excel spreadsheets etc.).

* Certain methodologies are rather abstract containing speculatively selected indicators often encompassing non evidence-based theoretical platforms, while their utilization does not facilitate the acquirement of quantifiable evaluation results.

* Methodologies are to a large extent narrowly focused assessing predominantly one of the evaluation levels presented in the five-level model.

* Mature methodologies are consisted of a large number of indicators, normally aligned for evaluation of e-government policies in the originating countries.

* Methodologies generally do not provide a comprehensive evaluation of complex e-government policies impacts and their potential long-term public benefits.

* Various groups of indicators evaluating the itemized evaluation levels are appearing in dozens of different methodologies, including a large number of overlapping. Definitions of indicators vary widely, while evaluations are based on completely different methodological platforms, their results are very difficult to compare.

Particular features of methodologies evaluating individual levels are outlined herein-after:

* Methodologies evaluating infrastructural level are mainly focused on ICT infrastructure and interoperability, human resources, legal framework and standards, policies and strategies, horizontal building blocks and other, often technical aspects. While generally focusing on only some of the itemized aspects (evaluation of particular technical aspects is very complex, e.g. interoperability) and allowing only a narrow insight into the context of e-government, they fail to provide a credible picture of the overall state of e-government.

* Methodologies evaluating project level are generally very exhaustive in terms of the large number of indicators; however they rarely address the concept of public benefits comprehensively, while the vast amount of data needed for applied indicators considerably complicates their utilization and transfer to other environments.

* Methodologies evaluating organizational level often address various organizational dimensions at least indirectly; failing to provide a full insight into the matter, consensus and clear rationalization of e-government induced organizational changes.

* Methodologies evaluating political-sociological level are mostly partial, focusing usually on policy aspect, accessibility and digital divide. Other methodologies in the area addressing particularly social aspect contain general and intangible indicators, since the concepts such as trust, confidence, social cohesion, social relations etc. are difficult to define unequivocally, while their understanding differs according to the cultural and institutional environment.

* Methodologies evaluating national level mainly explore the national-economic categories and their relations with the various aspects of development and implementation of e-government. They hardly formulate a clear research framework,
while interdependence, direction and way of influence between economic indica-
tors and e-government are not sufficiently explored and adequately elaborated.

After general systemization of identified methodologies (Fig. 2), we focused more closely on methodologies which have already achieved practical implementation. Based on these criteria we analysed 13 methodologies [14], [18], [19], [22], [24], [25], [26], [27], [64], [65], [67], [75], [85] which have been enumerated in category “Practical application” (Fig. 2). Analysis revealed substantial limitations and deficiencies. Although they have achieved a high level of maturity, and are used for evaluation of e-government policies in practice, they fail to address the evaluation of e-government policies in an all-encompassing manner. Most of the outlined methodologies are focused on only one level within the presented five-level model, preventing the comprehensive and quality evaluation of e-government policies.

Development of a comprehensive and practically applicable methodology for eval-
uation of e-government policies is obviously a difficult task. This is demonstrated in
Fig. 2, confirming that methodologies which have tried to cover several evaluation
levels are developed only up to conceptual framework or maximum pilot application.
The latter shows that covering larger number of evaluation levels usually means a
lower development level and consequently reduces the potential of methodologies for
their practical application. This is not unexpected, since the focus on several evalua-
tion levels means more complex methodology structure and a larger number of indi-
cators, which exacerbates the transparency and complicates the use of methodology.

Research results indicate that achievement of the highest development level and practical application of methodologies for evaluation of e-government policies is largely dependent on the number of evaluation levels the methodology is focused on, and vice versa, meaning that the comprehensiveness of evaluation methodologies is to a large extent conversely related to their development level.

5 Conclusion and future work

Growing number of evaluation methodologies and their substantial diversity regard-
ing the evaluation focus and level of maturity significantly complicate the establish-
ment of a theoretical framework that would allow a wide-ranging comparison and
analysis of the differences between methodologies. Numerous difficulties were en-
countered trying to delineate the evaluation levels covered by particular methodology,
since the contained indicators are not clearly defined, enabling their speculative use
on different evaluation levels. Various dilemmas emerged in determining which eval-
uation methodology achieved higher development level, as well. Although, the devel-
opment level of methodologies was defined primarily on the criterion of their use in
practice, objective definition of development level raises some very important ques-
tions of principle. These issues should be properly resolved in further research and
succeeding experiments trying to establish a balanced theoretical framework for com-
parative analysis of evaluation methodologies.

Despite aforementioned limitations, conducted analysis provides a valuable insight
into the current e-government evaluation practice and facilitates exposure of inade-
quately evaluated areas in the domain of e-government policies. The analysis results represent an advance in research of evaluation metrics and may eventually provide a solid platform for establishment of comprehensive methodology for evaluation of e-government policies and consequently initiate more user oriented, cost effective and performance-based development of e-government. Evidently, the problems in the development of e-government are strongly interrelated with the low quality and underdeveloped methodologies for evaluation of e-government policies and their effects. Extensive research and existing methodologies reveal that the past development of e-government, and particularly e-services was based primarily on political preferences and only exceptionally on professionally verifiable and measurable impacts of these services. Addressed shortcomings will have to be resolved, in order to ensure quality evaluation and implementation of e-government policies and ultimately accelerate the development of appropriate e-services with added value for all stakeholders.

## References

1. Silcock, R.: What is E-government? Parliamentary Affairs 54, 88–101 (2001)

2. OECD. Organization for Economic Co-operation and Development: E-government: Analysis Framework and Methodology, Paris (2001)

3. Yildiz, M.: Examining the Motivations for E-Government from an Institutional Theory Perspective: Evidence from Turkey. The National Conference on Digital Government Research, Boston (2003)

4. Capgemini: Smarter, Faster, Better eGovernment, 8th Benchmark Measurement. European Commission Directorate General for Information Society and Media (2009)

5. Capgemini, Rand, DTi, IDC, Sogeti: Digitizing Public Services in Europe: Putting ambition into action. Report of the 9th Measurement. European Commission (2010)

6. Capgemini, IDC, Rand Europe, Sogeti and DTi: 2011 eGovernment Benchmark Pilot on Open Government and Transparency. European Commission (2011)

7. OECD. Organization for Economic Co-operation and Development: Focus on Citizens: Public Engagement for Better Policy and Services, Paris (2009)

8. OECD. Organization for Economic Co-operation and Development: Rethinking e-Government Services: User-centred approaches, Paris (2009)

9. OECD. Organization for Economic Co-operation and Development: E-Government Indicators: Proposal for a new Framework and Methodology, Paris (2010)

10. Kunstelj, M., Jukic, T., Vintar, M.: Analysing the demand side of e-government: what can we learn from Slovenian users? In: Wimmer, M., Scholl, J., Grönlund, Å. (eds.) EGOV 2007. LNCS, vol. 4656, pp. 305–317. Springer, Regensburg (2007)

11. Kim, C.K.: A Cross-national Analysis of Global E-government. Public Organization Review 7, 317–329 (2007)

12. Vintar, M., Nograsek, J.: How much can we trust different e-government surveys? The case of Slovenia. Information polity 15, 199–213 (2010)

13. Dutta, S.: Estonia: A Sustainable Success in Networked Readiness? The Global Information Technology Report 2006–2007. Palgrave Macmillan, Hampshire (2007)

14. UN. United Nations: United Nations E-Government Survey, Leveraging e-government at a time of financial and economic crisis. Department of Economic and Social Affairs (2010)

15. WEF. World Economic Forum: The Global Information Technology Report 2010–2011. Transformations 2.0 (2011)
16. Bannister, F.: The curse of the benchmark: an assessment of the validity and value of e-government comparisons. Int. Review of Administrative Sciences 73, 171–188 (2007)

17. Janssen, D., Rotthier, S., Snijkers, K.: If you measure it, they will score: An assessment of international eGovernment benchmarking. Information Polity 9, 121–130 (2004)

18. AGIMO. Australian Government Information Office: Demand and Value Assessment Methodology - DAM & VAM. Commonwealth of Australia (2004)

19. RSO SpA & LUISS Management: eGEP - eGovernment Economics Project: Expenditure Study Final Version. DG Information Society and Media, Milano (2006)

20. UN. United Nations: United Nations E-Government Survey, From E-Government to Connected Governance. Department of Economic and Social Affairs, Division for Public Administration and Development Management, New York (2008)

21. Accenture: From e-Government to e-Governance: Using new technologies to strengthen relationships with citizens. Institute for Health and Public Service Value, Dublin (2009)

22. West, D.M.: Improving Technology Utilization in Electronic Government around the World. Center for Public Policy, Brown University, Rhode Island (2008)

23. Dadayan, L.: Measuring Return on Government IT Investments. Proceedings of the 13th European Conference on Information Technology Evaluation. Genoa (2006)

24. WEF, World Economic Forum: The Global Information Technology Report 2009-2010, ICT for Sustainability. Soumitra Dutta & Irene Mia (eds.) (2010)

25. EIU. Economist Intelligence Unit: Digital economy rankings 2010 (2010)

26. ADAE: MAREVA methodology guide: Analysis of the value of ADELE projects. Fourth High Level Seminar on Measuring and Evaluating E-Government, Dubai (2007)

27. Rothig, P.: WiBe 4.0 Methodology. Economic Efficiency Assessments in Particular with Regard to the Use of Information & Communication Technology (2010)

28. Scholl, H.J.: Electronic Government: Information Management Capacity, Organizational Capabilities, and the Sourcing Mix. Government Information Quarterly 23, 73–96 (2006)

29. Greaver, M.F.: Strategic Outsourcing: A Structured Approach to Outsourcing Decisions and Initiatives. AMACOM, New York (1999)

30. Kern, T., Willcocks, L.P., Van Heck, E.: The winner's curse in IT outsourcing: Strategies for avoiding relational trauma. California Management Review 44, 47–69 (2002)

31. Lacity, M.C., Willcocks, L.P.: Information Systems and Outsourcing. Studies in Theory and Practice. Palgrave Macmillan (2009)

32. Klievink, B., Janssen, M.: Realizing joined-up government – Dynamic capabilities and stage models for transformation. Government Information Quarterly 6, 275–284 (2009)

33. Fleur Van Veenstra, A., Janssen, M., Tan, Y.H.: Towards an Understanding of E-Government Induced Change - Drawing on Organization and Structuration Theories. In: Wimmer, M.A. et al. (eds.) EGOV 2010, LNCS, vol. 6228, pp. 1–12. Berlin (2010)

34. Schedler, K., Schmidt, B.: Managing the e-government organization. International Public Management Review 5, 1–20 (2004)

35. Scholl, H.J.: E-government: A Special Case of ICT-enabled Business Process Change. In: 36th Hawaii International Conference on System Sciences (2003)

36. Leitner, C., Kreuzeder, M.: Organisational Changes, Skills and the Role of Leadership Required by eGovernment. In: Wimmer, M. A. et al. (eds.) EGOV 2005, LNCS, vol. 3591, pp. 210–217. Springer-Verlag, Berlin (2005)

37. Martin, B., Byrne, J.: Implementing e-Government: widening the lens. RMIT University, – Australia, Melbourne (2003)

38. Bavec, C., Vintar, V.: What Matters in the Development of the E-Government in the EU? In: Wimmer, M.A., Scholl, H. J., Gronlund, A. (eds.) EGOV 2007, LNCS, vol. 4656, pp. 424–435. Springer-Verlag, Berlin (2007)
39. Singh, H., Das, A., Joseph, D.: Country-Level Determinants of E-Government Maturity. Communications of the Association for Information Systems 20, 632–648 (2007)

40. OECD. Organization for Economic Co-operation and Development: E-government and Administrative Simplification: Good Governance for Development in Arab Countries, Dubai (2007)

41. Hamilton, B.A.: Building a Methodology for Measuring the Value of E-Services. USA Social Security Administration (2002)

42. Kunstelj, M., Dečman, M.: Metrika na področju spremljanja razvoja e-uprave, In: Vintar, M., Grad, J. (eds.) E-uprava: Izbrane razvojne perspective, pp. 37–60. Ljubljana (2004)

43. Kunstelj, M., Vintar, M.: Evaluating the progress of e-government development: A critical analysis. Information Polity 9, 131–148 (2004)

44. Taylor, H.: Critical risks in outsourced IT projects: the intractable and the unforeseen. Communications of the ACM 49, 75–79 (2006)

45. Jensen, P.H.: Public Sector Outsourcing Contracts - The Impact of Uncertainty, Incentives and Transaction Costs on Contractual Relationships. VDM Verlag, Saarbrücken (2007)

46. Linder, J.C.: Outsourcing for Radical Change: A Bold Approach to Enterprise Transformation. AMACOM, New York (2004)

47. Power, M.J., Desouza, K., Bonifazi, C.: The Outsourcing Handbook: How to Implement a Successful Outsourcing Process. Kogan Page, Chicago (2006)

48. O'Donnell, O., Boyle, R., Timonen, V.: Transformational aspects of E-government in Ireland: Issues to be addressed, Electronic Journal of e-Government 1, 23–32 (2003)

49. Griffin, D., Foster, A., Halpin, E.: Joined-up E-government: an exploratory study of UK local government progress. Journal of Information Science and Technology 1, 58–83 (2004)

50. Layne, K., Lee, J.: Developing fully functional E-government: A four stage model. Government Information Quarterly 18, 122–136 (2001)

51. Elnaghi, M., Alshawi, S., Missi, F.: A Leadership Model for e-Government Transformation. In: European and Mediterranean Conference on Information Systems (2007)

52. Ho, A.T.K.: Reinventing Local Governments and the E-Government Initiative. Public Administration Review 62, 434–444 (2002)

53. Norris, P.: Digital Divide: Civic Engagement, Information Poverty and the Internet. Cambridge University Press, Cambridge (2001)

54. Slevin, J.: The Internet and Society. Polity Press, Cambridge (2000)

55. Anttiroiko, A.V.: Toward the European Information Society. Communications of the ACM 44, 31–36 (2001)

56. Thurlow, C., Lengel, L., Tomic, A.: Computer mediated communication: Social interaction and the internet. Sage Publications, Thousand Oaks (2004)

57. Shim, D.C., Eom, T.H.: E-Government and Anti-Corruption: Empirical Analysis of International Data. International Journal of Public Administration 31, 298–316 (2008)

58. Tolbert, C.J., Mossberger, K.: The Effects of E-Government on Trust and Confidence in Government. Public Administration Review 66, 354–70 (2006)

59. Parent, M., Vandebeek, C.A., Gemino, A.C.: Building Citizen Trust through e-Government. Government Information Quarterly 22, 720–736 (2005)

60. Bolgherini, S.: The Technology Trap and the Role of Political and Cultural Variables: A Critical Analysis of the E-Government Policies. Review of Policy Research 24, 259–275 (2007)

61. Lor, P.J., Britz, J.J.: Is a knowledge society possible without freedom of access to information? Journal of Information Science 33, 387–397 (2007)
62. Mahalik, D.K.: Outsourcing in e-Governance: A Multi Criteria Decision Making Approach. Journal of Administration & Governance 5, 24–35 (2010)

63. CIO Council: Value Measuring Methodology: How-To-Guide. Washington (2002)

64. OECD: Guide to measuring the information society. Directorate for science, technology and industry, Working party on indicators for the information society. France, Paris (2005)

65. Rama Rao, T.P., Venkata Rao, V., Bhatnagar, S.C., Satyanarayana, S.J.: E-Governance Assessment Frameworks (EAF Version 2.0). Government of India (2004)

66. UNESCO: E-government Tool Kit for Developing Countries. Asia Pacific Regional Bureau for Communication and Information, New Delhi (2005)

67. Baudu, S., Dzhumalieva, S.: Value Assessment Tool - VAST. Licensed by the European Commission under EUPL. Directorate General for Informatics (DIGIT) (2010)

68. Republic of Korea: e-Government in Korea. Ministry of Public Administration and Security. OECD e-Leaders Meeting 2010, Belgium, Brussels (2010)

69. Devadoss, P.R., Pan, S.L., Huang, J.C.: Structurational analysis of e-government initiatives: a case study of SCO. Decision Support Systems 34, 253–269 (2002)

70. Moon, M.J.: The Evolution of E-Government among Municipalities: Rhetoric or Reality? Public Administration Review 62, 424–433 (2002)

71. Chen, Y.C., Perry, J.: Outsourcing for E-Government: Managing for Success. Public Performance & Management Review 26, 404–421 (2003)

72. Štemberger, M.I., Jaklič, J.: Towards E-government by business process change – A methodology for public sector. International Journal of Information Management 27, 221–232 (2007)

73. Pollitt, C.: Technological Change: a central yet neglected feature of public administration. In: NISPA10, Slovenia, Ljubljana (2010)

74. Deloitte & Ipsos Belgium: Benchmarking deployment of eHealth services III (2011)

75. European Commission, Capgemini, & IDC: eGovernment Benchmarking in 2011 (2011)

76. Wauters, P., Nijskens, M., Tiebout, J. (Capgemini): The User Challenge, Benchmarking The Supply Of Online Public Services (2007)

77. Meyer, I., Hüsing, T., Didero, M., Korte, W.B. (Empirica): eHealth Benchmarking (2009)

78. Weehuizen, R., van Oranje, C.: Innovative and adaptive pan-European services for citizens in 2010 and beyond. DG Information Society & Media (2007)

79. Boyer-Wright, K.M., Kottemann, J.E.: An Empirical Assessment of Common Fundamentals in National E-Readiness Frameworks. Journal of Global Information Technology Management 12, 55–74 (2009)

80. Picci, L.: The quantitative evaluation of the Economic Impact of E-Government: A structural Modelling Approach. Information Economics and Policy 18, 107–123 (2006)

81. Rhee, D.Y.: The Study on Determinants of E-Government Evolution. Journal of Korean Association for Regional Information Society 12, 183–201 (2009)

82. Schwester, R.W.: Socio-Demographic Determinants of E-government Adoption: An Examination of Major U.S. Cities. Journal of Public Management and Social Policy 16, 21–32 (2010)

83. Serrano-Cinca, C., Rueda-Tomás, M., Portillo-Tarragona, P.: Determinants of E-Government Extension. Online Information Review 33, 476–498 (2009)

84. Gallego-Álvarez, I., Rodríguez-Domínguez, L., García-Sánchez, I.M.: Are Determining Factors of Municipal E-Government Common to a Worldwide Municipal View? An Intra-Country Comparison. Government Information Quarterly 27, 423–430 (2010)

85. Holzer, M., Kim, S.T.: Digital Governance in Municipalities Worldwide, A Longitudinal Assessment of Municipal Websites Throughout the World. New Jersey (2007)