# Evaluation of Government Information Systems Effectiveness: The Case of e-Participation Portals in Russia

Dmitrii Trutnev, Lyudmila Vidiasova

► To cite this version:

Dmitrii Trutnev, Lyudmila Vidiasova. Evaluation of Government Information Systems Effectiveness: The Case of e-Participation Portals in Russia. 10th International Conference on Electronic Participation (ePart), Sep 2018, Krems, Austria. pp.26-38, 10.1007/978-3-319-98578-7_3. hal-01985608

HAL Id: hal-01985608

https://inria.hal.science/hal-01985608

Submitted on 18 Jan 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Evaluation of government information systems effectiveness: the case of e-participation portals in Russia

Dmitrii Trutnev[0000-0001-6230-0752] and Lyudmila Vidiasova[0000-0002-8006-7066]

ITMO University, 49 Kronverksky pr., Saint Petersburg, 197101, Russia
trutnev@egov-center.ru, bershadskaya.lyudmila@gmail.com

**Abstract.** This paper is devoted to the study of methods for assessing the effec-
tiveness of public information systems that can identify the relationship between
the characteristics of individual factors of their creation and assess the impact of
these factors on their overall effectiveness and ability to create public values.
After reviewing, selecting and adapting an evaluation methodology, it was tested
with several e-participation portals. This choice was due to both the growing in-
terest in increasing the effectiveness of these portals, and the relative availability
of initial information for analysis. During the research, a number of hypotheses
were formulated and tested. The results obtained and presented in this paper con-
firm the suitability of the chosen methodology to solve the stated tasks of effec-
tiveness assessment of government information systems and the influencing fac-
tors.

**Keywords:** Government Information Systems, Effectiveness, Public Values, e-Participation.

# 1 Introduction

Information and Communication Technologies (ICT) penetrated steadily into all spheres of life in past decades. In social communication, the given process has occurred quite spontaneously [1]. However, in the field of communication between a government, citizens and business the management component of this ensemble plays a huge role [2].

In scientific literature at first much attention has been paid to e-services delivery
[3,4] and the development of “good governance” [5] with ICT. Gradually, scientists
came to the conclusion about the impact of digital transformation influence on public
values formation of various types [6,7,8]. Electronic interaction includes not only the
interface interaction with users of public services portals but also rather complex con-
nections of the participating information systems. Moreover, researchers already pro-
posed a list of public values components counting: efficiency, effectiveness, intrinsic
enhancements, transparency, participation, collaboration [8]. Viale Pereira [9] revealed
value generating the mechanisms pointing out a significant potential of open data initi-
atives in creating values.
Undoubtedly, the created values themselves are significant indicators of the quality of information systems, but their effectiveness, often understood as the ability to realize demanded public values by optimally using resources and balancing the associated risks [10] is also crucial. Particularly, precise measurement and management of the information systems effectiveness becomes important in the face of limited resources and the need to early obtain the necessary benefits.

At the same time, considering examples of countries with a highly centralised management system (like Russia), it is necessary to consider the existing peculiarities of government information systems management.

This paper describes an attempt to assess the effectiveness of e-participation portals in Russia from the perspectives of such stakeholders' as Federal Government, Regional Authorities and Citizens.

The paper has the following structure: Literature review provides a review of appropriate government information systems effectiveness assessment approaches and indicators. Methodology section demonstrates the author's framework for information system evaluation. Section “Findings” illustrates the results of Russian projects in the field of e-participation based on the methodology proposed. Section “Conclusion and discussion” concentrates on the key research results and the future steps.

## 2 Literature review

We started our research from the analysis of government information systems effectiveness assessment approaches in the studied area.

In one of the early works devoted to the goal we studied it was said that Effectiveness is determined by comparing performance to a goal and the way to assess system effectiveness is first to determine the task objectives of the system, or of the organizational units utilizing the system, and then to develop criterion measures to assess how well the objectives are being achieved [11]. This point of view remains relevant up to our time. The following main problems mentioned in Scott's work remain important so far: 1) Objectives and measures of accomplishments are often not defined adequately at the outset of an IS implementation effort [12]; 2) Efficiency-oriented and easily quantified objectives and measures are typically employed while effectiveness-oriented and qualitative objectives and measures are ignored [13]; 3) Objectives and measures used to evaluate the system are not the same as those defined initially [14]. In more recent works, discussion of these problems is rare, although our experience shows that they still exist and have a significant impact on the achievement of the necessary effectiveness of the information systems being created and on the ability to evaluate effectiveness by comparing goals and outcomes.

Bozeman and Moore are the founders of methods for assessing the effectiveness of public services (and the information systems providing them) using the concept of the public value created by them. It was they who proposed to measure “context-specific preferences of individuals concerning, on the one hand, the rights, obligations, and benefits to which citizens are entitled, on the other hand, obligations expected of citizens and their designated representatives” [15,16]. Research into the further development
and application of this approach has shown its usefulness for improving the quality of
government decisions in the field of application of information technologies, including
the area of improving communication between the government and citizens [17,18,19].
Indeed, a number of studies has proven that the focus on measurement and achievement
of public values leads to an increase in the effectiveness of government agencies and
their information systems: Effectiveness of public organisations itself creates public
values [15]. Citizens expect efficiency, openness, and responsiveness from public or-
ganisations [20,21]. E-government can be used for improving the efficiency of public
organisations by cutting processing costs, and making strategic connections between
and among government agencies [22] through developing better ICT infrastructures,
re-designing public functions [23], sharing public information and empowering public
staff [24]. Since public organisations run on taxpayers' money, citizens value the im-
proved efficiency of public organisations through e-government [25].

The studies surveyed revealed that more than 100 indicators of public value and ef-
fectiveness were proposed and used, among them: ensuring environmental sustainabil-
ity, transforming citizens' interests into political decisions, openness, internal effi-
ciency, etc. Unlike the traditional model of public administration, the concept of social
value emphasises that the list of social values is formulated not by those who provide
public services, but by the citizens who can express their preferences through various
instruments of interaction [26]. It also argues that public perceptions of values such as
trust and democracy should play a leading role and be complemented by other values
such as cost-efficiency and effectiveness [27].

Judging by the large number of publications on research on the effectiveness of state
information systems, one of the most popular is the PVIT (Public Value of Information
Technology) technique [28]. Perhaps, one of the most extensive and carefully imple-
mented studies on the effectiveness of e-government using this method was conducted
in 2012 by Professor K. Karunasena in Sri Lanka [29]. Many subsequent studies, for
example, in 2016 in South Africa [30], in 2017 in Turkey [31] and in Taiwan [32] were
based on the modification of Karunasena's methodology.

An early analysis of the declared objectives of IT projects and their relationship to
content and target indicators is critical, since the expected benefits of implementing
state-owned IT projects can be realized only if they are clearly identified already in the
design process and are reflected in the key documentation of the project being launched
[33]. In response to this need for government authorities, the Department of Public Ad-
ministration of the University of Albany (USA) in 2012 developed and proposed for
free use a methodology and tool for the evaluation of the PVAT (Public Value Assess-
ment Tool) project portfolio [8], which is further logical development of the PVIT
methodology. The PVAT methodology was applied to the formation of IT project port-
folios in several US government organizations and showed good results that are of prac-
tical importance for increasing the effectiveness of their outcomes [34].

Returning to the definition of the effectiveness of information systems given in the
CobiT 5 standard [10] and temporarily refusing the impact of risks, we can calculate
the overall effectiveness of an information system using indicators characterising the
goals set, the goals achieved, and the costs incurred. However, considering the need to
take these indicators into account in the set of organisational subsystems and in a variety
of dimensions, such simple calculations become complicated. A practical model for
solving the task of calculating the complex efficiency was justified and proposed in the
form of the Global Organizational Effectiveness Index Subsystem (GOEIS) meta-
model [35].

3 Research Methodology

This research aims to investigate the effectiveness of government information systems, understood as their ability to realise demanded public values by using resources opti- mally. To adequately accomplish this aim, a theoretical framework is required for providing the foundation for the implementation of both the quantitative and qualitative studies.

The theoretical framework is developed based on such theoretical concepts as the
theory of public value, the sources of public value creation, inventories of public value
and information systems effectiveness measurement described in Literature review sec-
tion. In summary, the effectiveness of public value creation greatly depends on public
bodies and their information systems organisations, various stakeholders, and their in-
teractions [21, 19].

Based on these theoretical perspectives and the indicators derived from various e-
government performance evaluation methodologies discussed in Literature review, a
theoretical framework for evaluating the effectiveness of government information sys-
tems in Russia is hypothesised and shown in Figure 1.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** The theoretical framework for evaluating the effectiveness of government information systems</figcaption></figure>

The relationships stemming from the stakeholder's conscious demands to the effective-
ness of government information systems, that are created to meet these demands, be-
came the basis for the formulation of a series of reflective research hypotheses as shown
in Figure 1 and summarised in Table 1.
<table><caption>**Table 1.** An overview of the hypotheses</caption><thead><tr><td>Hypothesis</td><td>Description</td></tr></thead><tbody><tr><td>H1</td><td>All stakeholders are known, and effective tools for identifying their demands are available</td></tr><tr><td>H2</td><td>The interests of all stakeholders are properly prioritised and consolidated into a consistent system of goals and programs/projects for their achieving</td></tr><tr><td>H3</td><td>The government IS development programs/projects are aimed at achieving the set goals and is provided with the necessary resources</td></tr><tr><td>H4</td><td>The created government information systems generate the required public values</td></tr><tr><td>H5</td><td>The identified stakeholders' needs are used in assessing the effectiveness of created government information systems in the form of target indicators</td></tr><tr><td>H6</td><td>The costs of continuous stakeholders' needs identification and prioritising efforts are considered</td></tr><tr><td>H7</td><td>The costs of government information systems creation and maintenance are considered</td></tr><tr><td>H8</td><td>The target and realized public values are measured by the same indicators</td></tr></tbody></table>

This research is going to be both confirmatory and exploratory. The confirmatory nature of the research is reflected by its objective to test a hypothesised theoretical framework for evaluating the public value of government information systems in Russia. The exploratory nature of the research is characterised in its pursuit of investigating the sources of effectiveness, how government information systems create public value for stakeholders in Russia, and how the existing practices in implementing IT initiatives can be improved for delivering better public value to its stakeholders [36].

A mixture of qualitative and quantitative approaches was chosen to implement the research, since it allows the researcher to test a theory by understanding the various factors in the theory and establishing relationships between the factors, and also to explore the reasons behind the relationships [37].

To test and validate the theoretical framework, the data were collected from open sources (regional state information systems development plans, Public Procurement Portal, descriptions of the public IS introduction results submitted to the all-Russia competition Prof-IT, state statistics portal, etc.), some of which provide data in digitalised forms, but some in the form of analytical and descriptive texts that are to be pre-processed for further analysis.

To calculate the effectiveness of the studied government information systems, taking into account their number of dimensions and sub-systems, the GOEIS meta-model has been chosen. This meta-model is not limited and gives the opportunity to add or remove elements or vectors, depending on the system characteristics to be evaluated. Table 2 shows the meta-model and its components.

To narrow the scope of research, government information systems supporting e-participation in Russia were selected as the object for pilot research. Taking into account the specifics of a rigid vertical state management system in Russia, three major stakeholder groups were selected for the study: the parent authorities that dictate the development goals of state information systems development are mandatory for execution at
# 4 Results

The results of expert decisions on the numerical values of vectors for the analysed elements are given in Table 3.

<table><caption>**Table 3.** Initial data obtained as a result of e-participation sites expert evaluation</caption><thead><tr><th rowspan="2">e-participation Portal</th><th colspan="6">Hypothesis - Effecting Vector's Values (%)</th></tr><tr><th>H1 H1 H1<br>V1 V2 V3</th><th>H2 H2 H2<br>V1 V2 V3</th><th>H3 H3 H3<br>V1 V2 V3</th><th>H4 H4 H4<br>V1 V2 V3</th><th>H5 H5 H5<br>V1 V2 V3</th></tr></thead><tbody><tr><td><strong>State-owned portals</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>www.roi.ru</td><td>90 80 45</td><td>95 85 50</td><td>50 30 25</td><td>30 30 25</td><td>20 25 20</td></tr><tr><td>priemnaya.parlia-ment.gov.ru</td><td>85 65 30</td><td>80 55 45</td><td>40 20 20</td><td>20 20 20</td><td>20 25 20</td></tr><tr><td>openregion.gov-mur-man.ru</td><td>75 55 50</td><td>70 65 55</td><td>45 35 20</td><td>35 35 20</td><td>25 25 20</td></tr><tr><td>idea.kemoblast.ru</td><td>80 65 45</td><td>70 65 55</td><td>30 30 15</td><td>30 30 15</td><td>25 20 15</td></tr><tr><td>open.krasnodar.ru</td><td>70 55 40</td><td>80 60 50</td><td>45 40 20</td><td>25 40 20</td><td>20 25 20</td></tr><tr><td>open.tatarstan.ru</td><td>90 70 60</td><td>80 75 50</td><td>35 30 25</td><td>30 30 25</td><td>30 30 20</td></tr><tr><td>golos.openrepublic.ru</td><td>70 35 55</td><td>65 50 55</td><td>50 40 30</td><td>25 40 30</td><td>25 30 20</td></tr><tr><td>narodportal.ru</td><td>60 40 35</td><td>60 50 40</td><td>55 45 30</td><td>25 45 30</td><td>30 35 25</td></tr><tr><td>ag.mos.ru</td><td>85 65 50</td><td>77 70 55</td><td>35 30 30</td><td>30 30 30</td><td>30 30 20</td></tr><tr><td>gorod.gov.spb.ru</td><td>80 70 60</td><td>80 50 40</td><td>40 45 40</td><td>25 45 40</td><td>25 30 25</td></tr><tr><td>open-penza.ru</td><td>85 60 50</td><td>70 50 50</td><td>50 35 45</td><td>30 35 45</td><td>25 30 25</td></tr><tr><td>vmestekirov.ru</td><td>75 65 45</td><td>65 60 55</td><td>40 30 35</td><td>20 30 35</td><td>20 25 20</td></tr><tr><td><strong>Average Effecting Vector's Value</strong></td><td><strong>79 60 47</strong></td><td><strong>74 61 50</strong></td><td><strong>43 34 28</strong></td><td><strong>27 34 28</strong></td><td><strong>25 28 21</strong></td></tr><tr><td><strong>Independent portals</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>narexpert.ru</td><td>5 40 60</td><td>10 30 80</td><td>20 30 60</td><td>50 30 60</td><td>15 35 65</td></tr><tr><td>www.angrycitizen.ru</td><td>10 55 85</td><td>20 40 85</td><td>15 30 55</td><td>60 30 55</td><td>30 35 60</td></tr><tr><td>onlinePetition.ru</td><td>25 45 80</td><td>40 35 75</td><td>25 35 50</td><td>55 35 50</td><td>25 40 60</td></tr><tr><td>change.org</td><td>40 50 75</td><td>50 40 80</td><td>30 25 60</td><td>60 25 60</td><td>25 30 70</td></tr><tr><td>beautiful_petersburg.rf</td><td>35 55 80</td><td>40 50 80</td><td>25 30 55</td><td>55 30 55</td><td>30 35 65</td></tr><tr><td></td><td>23 49 76</td><td>32 39 80</td><td>23 30 56</td><td>56 30 56</td><td>25 35 64</td></tr><tr><td><strong>Average Effecting Vector's Value</strong></td><td><strong>5 40 60</strong></td><td><strong>10 30 80</strong></td><td><strong>20 30 60</strong></td><td><strong>50 30 60</strong></td><td><strong>15 35 65</strong></td></tr></tbody></table>

For each of the analysed portals, calculations were made of the effectiveness, both global and in sections of hypotheses and individual vectors using the GOEIS technique. The results of calculations for all portals are summarised in Table 4
<table><caption>**Table 4.** Consolidated settlement results for all analysed e-participation portals</caption><thead><tr><th rowspan="2">e-participation Portal</th><th colspan="5">Portals' Effectiveness by Hypothesis and Vectors (%)</th></tr><tr><th>H1</th><th>H2</th><th>H3</th><th>H4</th><th>H5</th></tr></thead><tbody><tr><td><strong>State-owned portals</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>www.roi.ru</td><td>32</td><td>40</td><td>4</td><td>2</td><td>1</td></tr><tr><td>priemnaya.parliament.gov.ru</td><td>17</td><td>20</td><td>2</td><td>1</td><td>1</td></tr><tr><td>openregion.gov-murman.ru</td><td>21</td><td>25</td><td>3</td><td>2</td><td>1</td></tr><tr><td>idea.kemoblast.ru</td><td>23</td><td>25</td><td>1</td><td>1</td><td>1</td></tr><tr><td>open.krasnodar.ru</td><td>15</td><td>24</td><td>4</td><td>2</td><td>1</td></tr><tr><td>open.tatarstan.ru</td><td>38</td><td>30</td><td>3</td><td>2</td><td>2</td></tr><tr><td>golos.openrepublic.ru</td><td>13</td><td>18</td><td>6</td><td>3</td><td>2</td></tr><tr><td>narodportal.ru</td><td>8</td><td>12</td><td>7</td><td>3</td><td>3</td></tr><tr><td>ag.mos.ru</td><td>28</td><td>30</td><td>3</td><td>3</td><td>2</td></tr><tr><td>gorod.gov.spb.ru</td><td>34</td><td>16</td><td>7</td><td>5</td><td>2</td></tr><tr><td>open-penza.ru</td><td>26</td><td>18</td><td>8</td><td>5</td><td>2</td></tr><tr><td><strong>Average Values:</strong></td><td><strong>23</strong></td><td><strong>23</strong></td><td><strong>4</strong></td><td><strong>3</strong></td><td><strong>2</strong></td></tr><tr><td><strong>Independent portals</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>narexpert.ru</td><td>1</td><td>2</td><td>4</td><td>9</td><td>3</td></tr><tr><td>www.angrycitizen.ru</td><td>5</td><td>7</td><td>2</td><td>10</td><td>6</td></tr><tr><td>onlinePetition.ru</td><td>9</td><td>11</td><td>4</td><td>10</td><td>6</td></tr><tr><td>change.org</td><td>15</td><td>16</td><td>5</td><td>9</td><td>5</td></tr><tr><td>beautiful_petersburg.rf</td><td>15</td><td>16</td><td>4</td><td>9</td><td>7</td></tr><tr><td><strong>Average Values:</strong></td><td><strong>9</strong></td><td><strong>10</strong></td><td><strong>4</strong></td><td><strong>9</strong></td><td><strong>6</strong></td></tr></tbody></table><table><thead><tr><th colspan="3">Global Effectiveness</th><th colspan="3">Average</th></tr><tr><th>V1</th><th>V2</th><th>V3</th><th>V1</th><th>V2</th><th>V3</th></tr></thead><tbody><tr><td>16</td><td></td><td></td><td>57</td><td>50</td><td>33</td></tr><tr><td>8</td><td></td><td></td><td>49</td><td>37</td><td>27</td></tr><tr><td>11</td><td></td><td></td><td>50</td><td>43</td><td>33</td></tr><tr><td>10</td><td></td><td></td><td>47</td><td>42</td><td>29</td></tr><tr><td>9</td><td></td><td></td><td>48</td><td>44</td><td>30</td></tr><tr><td>15</td><td></td><td></td><td>53</td><td>47</td><td>36</td></tr><tr><td>8</td><td></td><td></td><td>47</td><td>39</td><td>38</td></tr><tr><td>7</td><td></td><td></td><td>45</td><td>43</td><td>32</td></tr><tr><td>13</td><td></td><td></td><td>51</td><td>45</td><td>37</td></tr><tr><td>13</td><td></td><td></td><td>50</td><td>48</td><td>41</td></tr><tr><td>11</td><td></td><td></td><td>52</td><td>42</td><td>43</td></tr><tr><td><strong>11</strong></td><td></td><td></td><td><strong>50</strong></td><td><strong>44</strong></td><td><strong>35</strong></td></tr><tr><td>4</td><td></td><td></td><td>20</td><td>33</td><td>65</td></tr><tr><td>6</td><td></td><td></td><td>27</td><td>38</td><td>68</td></tr><tr><td>8</td><td></td><td></td><td>34</td><td>38</td><td>63</td></tr><tr><td>10</td><td></td><td></td><td>41</td><td>34</td><td>69</td></tr><tr><td>10</td><td></td><td></td><td>37</td><td>40</td><td>67</td></tr><tr><td><strong>8</strong></td><td></td><td></td><td><strong>32</strong></td><td><strong>37</strong></td><td><strong>66</strong></td></tr></tbody></table>

Based on the results of the surveyed e-participation portals effectiveness calculations, supporting organisational structures descriptions and related information systems, it became possible to draw preliminary conclusions on the hypotheses formulated at the beginning of the research. Since the obtained results indicate an unsatisfactory situation, the conclusions given in this paper are concentrated on the problems found. These conclusions are summarised in Table 5

<table><caption>**Table 5.** Main conclusions on hypotheses</caption><thead><tr><th>Hypotheses</th><th>Conclusions</th></tr></thead><tbody><tr><td><strong>H1</strong> All stakeholders are known, and effective tools for identifying their demands are available</td><td>The demands of key stakeholders have been identified with varying degrees of completeness often using improper tools and approaches.</td></tr><tr><td><strong>H2</strong> The interests of all stakeholders are properly prioritised and consolidated into a consistent system of goals and programs/projects for their achieving</td><td>The key stakeholders' needs are systematised and presented in the form of the official objectives design of the planned projects does not allow to achieve these goals entirely. The indicators used rarely reflect the achievement of the set goals.</td></tr></tbody></table>
<table><caption>**Table 5.** Main conclusions on hypotheses (Continued)</caption><thead><tr><td>Hypotheses</td><td>Conclusions</td></tr></thead><tbody><tr><td>H3 The government IS develop-ment programs/projects are aimed at achiev-ing the set goals and is provided with the necessary resources</td><td>The implementation of programs/projects is aimed at attaining target indicators, not goals, since Target in-dicators do not reflect the achievement of official goals (see Conclusions of H2).</td></tr><tr><td>H4 The created government infor-mation systems generate the required public values (social, political, eco-nomic)</td><td>Created e-participation portals generate the required public values to some extent. However, the composi-tion of these values does not coincide in composition and size with the required.</td></tr><tr><td>H5 The identified stakeholders' needs are used in assessing the effectiveness of created government information systems in the form of target indicators</td><td>The practice of assessing the created e-participation portals effectiveness is observed in the initial state only and the use of identified stakeholders' needs for this purpose is rare.</td></tr><tr><td>H6 The costs of continuous stakehold-ers' needs identification and prioritis-ing efforts are considered</td><td>In the studied open information sources, no infor-mation was found on the cost characteristics of ef forts to identify and prioritise the stakeholders' needs.</td></tr><tr><td>H7 The costs of government infor-mation systems creation and mainte-nance are considered</td><td>For all surveyed e-participation portals, this hypoth-esis was entirely valid</td></tr><tr><td>H8 The target and realized public val-ues are measured by the same indica-tors</td><td>This hypothesis is true. However, the adopted target indicators are not able to assess the degree of achievement of the objectives, as was said earlier.</td></tr></tbody></table>

Some differences between government and independent e-participation portals are of interest. Government portals are more successful in identifying the needs of stake-holders and in formulating the goals of development programs than independent portals which are more successful in these goals implementation. This finding is illustrated by the data presented in Figure 2.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** The degree of the hypotheses fairness for state and independent portals</figcaption></figure>

However, it is possible to observe significant differences between state and inde-pendent portals in their striving to satisfy various stakeholder groups. This is evident from Figure 3.
<figure><img src="image_4.png" /><figcaption>Fig. 3. Different focus of efforts to meet the needs of stakeholders</figcaption></figure>

In general, the results of this study showed the suitability of the method chosen to assess the overall effectiveness of public information systems and to assess the contribution of some organisational factors to overall effectiveness, which may be useful for more effective management of these systems.

# 5 Discussion

The study contributes to understanding on management factors in IT projects development and implementation. The framework proposed could be of interest for GCIOs and IT- managers facing some issues of IT effectiveness and its measurement indicators. The model proposed involves the assessment of external environment, upper-level management system, goal setting, planning and organisation of government information systems operation, operational management, outcome parameters, and collection of data on the status and results, as well as integrated effectiveness.

The author's approach to combine a management cycle and public values types (social, political, economic) was first applied on e-participation cases since these portals provide new ways to solve citizens' needs and demands not just in an operational way (as e-services) but in some part of qualitative decision-making.

The limitations of the study are connected with the limited number of assessed projects as well as the orientation on very specific Russian projects context. The study counted data from the contest, at the same time some interviews with portals developers could shed light on internal management operations.

The following studies will be focused on further improving the described model of effectiveness evaluation. Among our immediate tasks we can mention the following: searching for and testing source data more objective than those used in this study, application this model to other IT project types, the comparison of calculated effectiveness levels of different managerial systems.

# Acknowledgements

This work was conducted with support of RFBR grant №. 16-36-60035 "The research of social efficiency of e-participation portals in Russia".
References

1. Faik, I., Walsham, G.: Modernization through ICTs: towards a network ontology of technological change. Information Systems Journal 23 (4), 351-370 (2013).

2. Bannister, F., Connolly, R.: ICT, public values and transformative government: a framework and programme for research. Government Information Quarterly 31, 119-128 (2014).

3. Helbig, N., Gil-Garcia, J.R., Ferro, E.: Understanding the complexity of electronic government: implications from the digital divide literature. Government Information Quarterly 26 (1), 89-97 (2009).

4. Janssen, M., Chun, A., Gil-Garcia, J.R.: Building the next generation of digital government infrastructures. Government Information Quarterly 26 (2), 233-237 (2009).

5. Gisselquist, Rachel M., What Does Good Governance Mean? (January 1, 2012). WIDER Angle, January 2012.. Available at SSRN: https://ssrn.com/abstract=2466430.

6. Janowski, T.: Digital government evolution: from transformation to contextualization. Government Information Quarterly 32 (3), 221-236 (2015).

7. Luna- Reyes, L.F., Gil-Garcia, J.R., Romero, G.: Towards a multidimensional model for evaluating electronic government: proposing a more comprehensive and integrative perspective. Government Information Quarterly 29 (3), 324-334 (2012).

8. Harrison, T., Guerro, S., Burke, G.B., Cook, M., Cresswell, A., Helbig, N., Pardo, T.: Open government and e-government: Democratic challenges from a public value perspective. Information Polity 17, 83-97 (2012).

9. Viale Pereira, G., Macadar, M.A., Luciano, E.M., Testa, M.G.: Delivering public value through open government data initiatives in a Smart City context. Inf Syst Front 19, 213-229 (2017).

Hamilton, S., Chervany, N.L.: Evaluating information systems effectiveness - part i: comparing evaluation approaches. MIS Q 5 (3), 55-69 (1981).
DOI=http://dx.doi.org/10.2307/249291.

10. ISACA. Cobit 5 - A Business Framework for the Governance and Management of Enterprise IT. Rolling Meadows, IL (USA), ISACA, 2012. http://www.isaca.org/.

11. Scott Morton M. S. (1971) Management Decision Systems: Computer-based Support for Decision Making. — Boston: Harvard University.

12. Keen, P.G.W., Scott Morton, M.S.: Decision Support Systems: An Organizational Perspective. Addison Wesley, Reading, Massachusetts (1978).

13. Ginzberg, M.J.: Improving MIS Project Selection. OMEGA 7 (6), 527-537 (1979).

14. Leintz, B.P., Chen, M. (1981): Assessing the impact of new technology in information systems. Long Range Planning Volume 14, Issue 6, December 1981, Pages 44-50

15. Moore, M. H.: Creating public value: Strategic management in government. London: Har-
vard University Press (1995).

16. Bozeman, B.: Public value and public interest: Counterbalancing economic individualism. Washington: Georgetown University Press (2007).

17. Kelly, G., Muers, S.: Creating public value: An analytical framework for public service reform. London: Cabinet Office Strategy Unit (2002). http://webarchive.nation-
alarchives.gov.uk/20100407164622/http://www.cabinetoffice.gov.uk/strategy/seminars/pub-
lic_value.aspx, last accessed 2018/03/16.

18. Alford, J., O'Flynn, J.: Making sense of public value: Concepts, critiques and emergent meanings. International Journal of Public Administration 32, 171-191(2009).

19. Benington, J.: Creating the public in order to create public value? International Journal of Public Administration 32, 232-249 (2009).
20. Kernaghan, K.: Integrating values into public service: The values statement as centrepiece. Public Administration Review 63(6), 711-719 (2003).

21. Jorgensen, T. B., Bozeman, B.: Public values an inventory. Administration & Society 39(3), 354-381 (2007).

22. Heeks, R.: Success and Failure Rates of eGovernment in Developing/Transitional Countries: Overview. (2008). Retrieved March 15, 2018, from http://www.egov4dev.org/success/sfrates.shtml, last accessed 2018/03/16.

23. Al-Omari, A., Al-Omari, H.: e-Government readiness assessment model. Journal of Computer Science 2(11), 841-845 (2006).

24. Falivene, G. M., Silva, G. M.: Reflections and proposals on public officials training and promotion of e-government. International Journal of Electronic Government Research 4(2), 43-58 (2008).

25. Gauld, R., Goldfinch, S., Horsburgh, S.: (2010). Do they want it? Do they use it? The demand-side of e-government in Australia and New Zealand. Government Information Quarterly 27, 177-186 (2010).

26. Alford, J., Hughes, O.: Public value pragmatism as the next phase of public management. American Review of Public Administration 38 (2), 130-148 (2008).

27. Bryson, J., Crosby, B., Bloomberg, L.: Public Value Governance: Moving Beyond Traditional Public Administration and the New Public Management. Public Administration Review 74 (4), 445-456 (2014).

28. Di Maio, A.: Value for Money Is Not Enough in Public Sector IT- projects. Stamford, C.T.: Gartner Research (2003).

29. Karunasena, K.: An Investigation of the Public Value of e-Government in Sri Lanka. PhD Dissertation (2012). https://researchbank.rmit.edu.au/eserv/rmit:160100/Thanthri_Waththage.pdf, last accessed 2018/03/16.

30. Sigwejo, A. & Pather, S. (2016) A citizen-centric framework for assessing e-government effectiveness, The Electronic Journal of Information Systems in Developing Countries, 74(8), 1-27.

31. Hiziroglu, A.: Measuring The Public Value of E-Government: Evidence From Turkey. In: Proceedings of The IRES International Conference, Rio de Janeiro, Brazil, 27th -28th January 2017, http://www.worldresearchlibrary.org/up_proc/pdf/684-14879327641-8.pdf, last accessed 2018/03/16.

32. Chu, P.Y. Tseng, H.L., Lee, C.P., Huang, W.L., Huang, T.Y., Hung Y.T.: A Longitudinal Research of Public Value and Electronic Governance Development in Taiwan. In: 10th International Conference on Theory and Practice of Electronic Governance (ICEGOV’17), Proceedings pp.459-464 (2017).

33. Benington, J., Moor, M.: Public Value Theory and Practice. Basinstoke: Palgrave Macmillan (2011).

34. Bryson, J., Crosby, B.: Public Value and Public Administration. Bloomberg Georgetown University Press (2015).

35. Páscoa, C., Moreira, V., & Tribolet, J. (2012). Effectiveness index as an organizational cockpit instrument, Procedia Technology Volume 5, 2012, Pages 244-254, 4th Conference of ENTERprise Information Systems – aligning technology, organizations and people (CENTERIS 2012). Elsevier.

36. Creswell, J. W. Research design: Qualitative, quantitative and mixed methods approaches (3rd ed.). California: Sage Publication Inc. (2009).

37. Woolley, C. M. (2009). Meeting the mixed methods challenge of integration in a socio-logical study of structure and agency. Journal of Mixed Methods Research, 3(1), 7-25.