# From Routine to Revolt: Improving Routine Health Data Quality and Relevance by Making Them Public

Thanh Ngoc Nguyen, Petter Nielsen

► To cite this version:

Thanh Ngoc Nguyen, Petter Nielsen. From Routine to Revolt: Improving Routine Health Data Quality and Relevance by Making Them Public. 14th International Conference on Social Implications of Computers in Developing Countries (ICT4D), May 2017, Yogyakarta, Indonesia. pp.548-558, 10.1007/978-3-319-59111-7_45. hal-01650062

HAL Id: hal-01650062

https://inria.hal.science/hal-01650062

Submitted on 28 Nov 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# From Routine to Revolt: Improving Routine Health Data Quality and Relevance by Making them Public

Thanh Ngoc Nguyen¹

Petter Nielsen²

¹,² Department of Informatics, University of Oslo, Norway

¹thanhng@ifi.uio.no

²pnielsen@ifi.uio.no

**Abstract.** Health Information Systems in developing countries struggle with vicious cycles of lack of information use. Substantial investment has been spent to improve the situation but results are still very limited. Adding to the body of research on strategies and solutions to break out of such cycles, this paper focuses on the effects of making routine data public through mass media and using data to fuel debates on critical health issues. Based on an action research project building a reporting system for accidents and emergencies during the Tet holiday in Vietnam, this paper discuss how making data public can have direct impact on the use and quality of health data in the health system. We discuss and draw implications related to tactics to improve the demand and use of routine health data.

**Keywords:** Information use, data quality, rapid scaling

## 1 Introduction

Health Information Systems (HISs) have long been recognized as an important component of public health systems because they provide vital data for effective planning and sound decision making [1]. Although HISs play an important role in improving healthcare service delivery, they often fail, especially in developing countries [2]. Many HISs end their lifecycle in the pilot stage because they do not scale and provide useful data for managers [3]. Together with the scaling challenge, developing countries are also challenged by health managers’ limited use of data in decision making. Nutley, Gnassou [4] argue that turning data into action is critical for improving the health services delivery and outcomes. There are many reasons attributed to limited data use, including culture and capacity [5] as well as the information needs of decision makers [6]. Another complication is the poor quality of data. Braa, Hanseth [3] use the notion of a vicious cycle to refer to situations in which “national health data are used little because they are of poor quality, and their relative lack of use, in turn, makes their quality remain poor” (p. 379). This is a hard to break cycle. Several attempts have been made to break out of the vicious data use cycle (see
for example Nutley and Reynolds [1], Braa, Heywood [2], Rhoads and Ferrara [3]).
These attempts can broadly be classified into two streams. The first deals with prob-
lems related to the data supply side such as data quality, availability, and access. The
second stream focuses on the demand side by encouraging data use based on changing
social institutions such as information culture and human capacity.

With a focus on understanding data use and approaches to generate data demand from decision makers, we conducted an action research which involved the design, customization, and implementation of an information system that supports the collection and analysis of various health indicators during the Lunar New Year in Vietnam between 2014 and 2016. Data reported from more than one thousand hospitals and health facilities (accounting for 95% hospitals in Vietnam) revealed a harsh reality of violence and traffic accidents during Lunar New Year holidays. And the general public was perplexed when figures of traffic accidents announced by MoH and Ministry of Public Security (MPS) contradicted each other significantly. It also sparked a big public debate on data quality, data collection and related processes.

In developing countries, routine health data and indicators are not usually high in
demand. Those collecting it commonly collect it only for the purpose of upward re-
porting and receive little if any feedback. And those receiving it commonly do not
trust the data nor use it for decision making. This paper presents a contrasting case to
this where the mundane and neglected routine health data suddenly was in the spot-
light.

The rest of this paper is organized as follows. In the next section, we review related research and debates on the problem of data use and offer our perspective. Research method and the case description are provided in section 3 and 4 respectively. We discuss the findings and implications for theory and practice in section 5 before conclude the paper in section 6.

2 Related Research

The ultimate goal of any HISs intervention is to ensure quality data that informs deci-
sion making. Informed decisions will lead to better use of limited and often scarce
resources where they are needed. If data are not used, HISs and efforts to collect,
aggregate and distribute health data are meaningless. Unfortunately, despite invest-
ments in and improvements of HISs, lack of data use is common (Braa et al., 2012,
Wyber et al., 2015). This section presents recent studies that discuss efforts to tackle
the problem of little data use.

Lomas [7] in a study from the healthcare sector in Canada emphasizes the role of
intermediaries in disseminating research results, thus facilitating the use of health
data. He argues that there is a poor understanding of context where the research result
is generated and realities facing policy makers. A solution to this problem is to im-
prove the communication between the two sides. He discusses the approach that has
been applied in Canada for many years, which involves the institutionalization of the
knowledge brokering roles in disseminating the data. In particular, research results
should be communicated effectively and succinctly through Mythbusters and Evi-
dence Boost, which are a form of research summary, to other groups of users such as
legislative, administrative, and industrial decision-makers.

Bowen, Erickson [8] identify 8 different barriers that constrain the use of data for decision making in a study that involves health staff from 11 Regional Health Authorities in Manitoba province of Canada in three years. They argue that efforts to improve data accessibility and data use competency are important but might not increase the level of data use if the key barriers to the issue are not properly lifted. Their findings include an interesting discovery of “politics trumps evidence” which refers to a phenomenon where data are sought to back a political decision rather than inform decision making process. Sometimes, decision makers could not make an evidence-informed decision because they lack of means and supporting structure to implement a decision. Nutley and Reynolds [9] synthesize previous works related to the issue of health data use including the World Health Organization’s Health Metric Network tool [10], the Performance of Routine Information System Management (PRISM) framework [11, 12], Lomas [7], and Patton [13] to propose a comprehensively logical model that encompasses a set of eight processes and activities to strengthen data use.

In a developing country context, Braa, Heywood [14] conduct an action research with the aim to break the “vicious cycle” of data quality and data use. The underlying assumption of their intervention is that piecemeal increased data use can gradually improve the data quality and build up data use capacity. Quarterly workshops on data use are organized at both national and district levels. During the workshops, district staff will show data from their districts and present their own interpretation while other district staff will discuss and criticize. In three years between 2005 and 2008, there are noticeable changes in data use as a result of the intervention. Similarly, based on action research in Kenya with the aim to establish mutually agreed activities among key stakeholders to improve data use for action, Manya et al. (2015) argue that data quality audits must be organized at the same time with monthly data review meetings and training in data management in order to address the problem of little use of data. Moyo, Kaasbøll [15] also discusses the process of development and introduction of a League Table to compare the performance of different health districts in Malawi. They argue that the module is a useful tool to remedy the problem of information transparency. Also according to them, transparency of information should be treated as a totality of all aspects including disclosure, clarity, and accuracy rather than separately.

In the extant literature on the use of health data, the attention is central on how to
provide health managers high quality data to support the decision making process.
There is little discussion on how health data mean to the public and how the public
can participate in the routine health system design and use. Presumably, most data
from the routine health management information systems (HMIS) are supposed to
serve managerial purposes. The public who is subject of all health policies is often
ignored from such systems and they do not have access to their data. They neither
know about how such systems operate, what data are collected nor raise their voice on
how such systems should be designed to benefit them. The primary aim of this re-
search is to contribute to that discussion and more specifically focus on finding ap-
propriate approaches that could make boring routine health data become attractive to
the public.

3 Method

This research follows an action research approach [16]. Action research is a method that aims to solve practical problems and at the same time generate theoretical knowledge [17]. Action researchers have dual responsibilities: to improve the situation and to report findings of their study. Action research is different from other methods in its ability to develop knowledge for both theory and practice. In studying technology in social context, action research is a strong candidate [18]. While there are many forms of action research, canonical action research is commonly used as it ensures the rigor of the research [19].

Data for this study mainly come from the daily interactions, participation and ob-
servations of the first author in the daily activities of the project. A research diary was
used to capture important events, incidents, or other important discussions related to
the system development and implementation. To improve the reliability, data from
this source were regularly shared with the second author to independently verify.
Other modes of data collection include interviews. A total of 15 interviews with dif-
ferent kinds of informants: MoH officials (2), the technical and support teams (6),
provincial health administrators (2), and data clerks from hospitals (5) was conducted
in early 2014 through face to face and phone. Each interview lasted between 30 and
45 minutes. Notes were taken during the interviews for subsequent use in the analysis
step. Informants who were hospital users were selected from those who called the
supporting team to get support. The purpose of these interviews was thus to under-
stand better users' difficulties in using the system. Examples of interview questions
for this type of informants were "How do you find the design of that functionality?
How should it be improved?". Archival records were another means utilized for col-
lecting data. To understand the view of the public on various social issues such as
traffic accidents and violence during Tet, mass media articles were also considered,
mostly in the electronic form (e-newspapers) including statements of officials from
the Government and Assembly.

After being collected, data was grouped into broad themes such as technical infra-
structure and social infrastructure, the rapid process of development and implementa-
tion of the system, the inconsistency of reported data in comparison with other availa-
ble sources, and the public's attitudes and the perspectives of different governmental
agencies. Several concepts from the literature information transparency, development, and data use were used to guide the analytical process. Part of the analysis process is the narrative of the case, which is now presented.

# 4 The Case

In early 2014, the Department of Medical Services Administration (VAMS) of the Ministry of Health decided to build an online system to support collecting data related to accident and emergency during the Lunar New Year (Tet) in Vietnam. A medical doctor from VAMS, who is responsible for Information Technology (IT) and health data, consulted a technical team (here after called the DHIS2 Team) that had worked in previous health information systems projects at MoH. His aim was to explore the possibilities of using an open source platform especially developed for the health sector called DHIS2 (dhis2.org) to build an online system for Tet reporting. As the decision by VAMS was made very late, they only had one week to complete the system. After some considerations, the DHIS2 Team confirmed that they were able to build the system and immediately embarked on the mission. DHIS2 is a software platform developed by University of Oslo, Norway. It provides a flexible mechanism in handling data elements and forms, making it easy to define new datasets. However, the default data entry form in DHIS2 only allowed data entry for one period at a time. This was considered to be confusing for the users. The team thus decided to design a custom data-entry form that shows multiple periods in a single screen.

After building the data entry forms, the DHIS2 Team deployed the system on one of MoH servers. A few months before Tet, MoH had implemented a hospital quality and inventory system which was also based on the same DHIS2 software. User accounts for this system were generated using a script and forwarded in Excel files to each health province which subsequently forwarded to their subordinate hospitals. The DHIS2 Team decided to leverage this existing user base. Since they already had access and knew the system, this would shorten the implementation process and eliminate the need of training. An official letter requesting hospitals to use the Tet reporting system was distributed by the MoH.

As it was the first year using the Tet reporting system, many hospitals still viewed the system as a "pilot". As a consequence, lack of rigor in data entry was observed. Some users entered garbage numbers into important fields like injured cases by firecrackers, as if such figures would not be subsequently aggregated and reported to the government and visible to the public via mass media. It should be also noted that according to the law in Vietnam, the head of province will receive disciplinary action if there is illegal use of firecrackers in his/her province. To avoid such data entry incidents, the IT specialist from MoH who was assigned to work on this system had to frequently verify if there were any suspicious numbers entered. He also called the person who was responsible for reporting on that day in the hospital to confirm the reported numbers.
During the Tet 2014, data from more than one thousand hospitals were daily aggregated by MoH and subsequently reported to the Government Office. Many journalists approached MoH to acquire data to write articles because they believe accident and emergency during the Tet are matters of public concern. After articles were published, the public was shocked to know that nearly 7,000 people were hospitalized, out of which 15 people died, due to violence during the 9 days of the Tet holiday. The Deputy Chair of Social Committee, Vietnam Assembly said: "This figure was an alarm of the increasing violence in our society. It was very unusual because the Tet holidays were the time dedicated for joy and relax. The Committee welcomes MoH for its first time publication of the data which we did not have previously." [20]. The Prime Minister insisted that such figures were very serious and directed all concerned agencies to propose and implement effective measures to mitigate violence [21].

In addition, many scholars, educators, and psychologists proactively joined the debate, trying to locate the root causes of the issue. An economist from the Center of Economic Research in the South stated: "People solve their conflicts by using violence because they have lost their trust on justice and government" [22]. And a researcher from Research Center of Sino-Nom said: "It is a really crisis of the crowd, once its psychology is compressed throughout the year and it bursts on the Tet. It accumulates all conflicts with the root cause of unstable and insecure society. Culture, morality, and education in Vietnam have never been degraded like they are now" [20, 22]. There was a broad agreement that in combination with stricter punishment for violent crimes, schools should focus more on ethical and behavioral education for youngsters.

Regarding the reported figures of traffic accidents, the public was at the same time very puzzled by the gap [23] between the figures announced by Ministry of Public Security (MPS) and Ministry of Health (MoH). While the figures published by MPS showed a significant decrease in the number of injuries (-25%) and death (-5%), MoH announced an increase of traffic accidents during the Tet holiday. Quantitatively, MPS reported only 408 traffic accidents whereas MoH reported more than 40,000 hospitalized cases related traffic accidents. This triggered debates related to the trustworthiness of these two different information sources. For instance, the Head of Road Transportation Department (MPS) argued that the police could only collect data from traffic accidents that were reported to them while, in reality, most victims were transferred directly to hospitals. This argument was supported by a statement made by a representative of National Committee of Transportation Safety (NCTS): "NCTS will make a proposal to the Prime Minister to get permission to use data from MoH as an official source for traffic accident reporting, instead of exclusively using data from the police" [24].
# 5 Analysis and Discussion

The design and implementation of the Tet reporting system can be considered as a successful story because it rapidly achieved national coverage and generated data that triggered public debates on several social issues. We now provide analysis on how the system attracted the public's attention to what is commonly treated as boring and mundane routine data.

## 5.1 Collecting data that highly concern the public and commodifying health data

Hospital users often see reporting duty as a burden rather than something useful for them or the healthcare sector (Krickeberg, 2007, Kuhn and Giuse, 2001, Littlejohns et al., 2003). In the case of the Tet reporting system, the number of data elements was minimal and only important data elements were included. The small dataset approach also helped reduce complexities of the system both at technical and social levels. The flat structure of dataset made the design and use of the system become easy and helped shrink “the time to market” (Smith, 2004). This small dataset approach has another implication. Data entry forms of HISs in developing countries tend to be overly complex (Sahay et al., 2010) and merely serve the need of health managers. The public, however, should have the right to know about data that directly concern their life. In other words, HISs should also be designed in a way that it collects kinds of data that are highly related to the lives of majority of citizens. This is not the case in Vietnam where figures on traffic accidents and violence are completely absent on the routine report.

Routine health data are often too complex to understand and use even for health professionals [15]. In order to attract attention of the public, the data need to be simplified and commodified. For example, the number of deaths related to traffic accidents and number of injuries related to violence are simple and easy enough for anyone to understand. The use of raw numbers to some extent gives better impression about the situation compared to the use of indicators. Regardless the size of the population, thousands of people injured by violence is something hard to believe and unacceptable.

Commodifying data also relates to the two aspects of information transparency which are disclosure and clarity [15, 25]. While disclosure only focuses on the availability and accessibility of data to interested parties which are very often governmental agencies and not-for-profits organizations, commodifying also involves making sense of data to the public. In the attempt to engage the public in the debate of critical health issues, clarity must be more focused to the level that it is understandable for the public who do not know anything about the healthcare domain.
## 5.2 Engaging mass media and fueling public debates

Routine health data are often moved upward, i.e. from health facilities to higher level. Little if any feedback from higher level health managers are provided to health facilities after data are reported [26]. To attract the public's attention on critical health issues, novel channels of rapid and large-scaled data propagation must be sought. One such channel is mass media, i.e. newspapers and magazines. With large numbers of readers and hourly news posting, newspapers (both paper-based and online) can draw attentions from the public in a quick and effective way. Also, thanks to the rhetoric of journalists, titles of articles are often written in an impressive and attractive way which no doubt makes the boring and mundane data become lively and attract the attention of the public.

As a general practice, mass media often engage important and renowned figures to comment on emergent social issues. They can be professors from universities, experts in various domains, and incumbent officials or politicians. This approach is to bring the readers multiple perspectives on a particular issue. In the case of violence, the debate of data accuracy was sparked by MPS officials who expressed suspicion on the big number of hospitalized cases because of violence. There was a firm response from the Head of Medical Service Administration (VAMS, MoH) about that. The response included the unpacking of procedures and systems through which the number was aggregated. This was an important step to keep the dialog continue and remain constructive. Later, the IT specialist who was the key contact point of the project agreed to give an exclusive interview to an online newspaper on the history and backstage story of how the system was built and implemented.

## 5.3 Creating feedbacks and triangulating data

Also through newspapers the public knew more about the work of hospitals during the holiday. This gave users the feeling of participation. Through the circulation process, data are extensively validated and flexibly interpreted by a large collection of human actors. We argue that data circulation processes are crucial for sustainable HISs as it blurs the boundaries between data collection and data use, creating the congeniality [27] between the ICT initiative and the environment. The findings of our case are slightly distinct from literature discussing the role of new social media such as Facebook, Twitter etc. on changing awareness of the public and creating new channel of two-way communication between public health experts and the public. Recent works on social media emphasize the significance and effectiveness of using social media to engage the public in public health issues [28-31]. However, we argue that depending on the situation, traditional media are still very powerful means in amplifying and visualizing health issues to the public which in turn would help to increase accountability and transparency.

We believe one factor that played an important role in the success of the Tet system was the feedback mechanism. The extant literature often reports cases of HISs in
which data reported by data entry clerks do not receive adequate feedbacks to im-
prove the quality of data [32]. In our case, there exist two types of feedback mecha-
nisms: feedback from MoH to hospitals and from the public to hospitals. As the re-
porting process took place on a daily basis, hospitals received feedback from MoH
within almost the same day. This instant feedback played a crucial role in not only
improving data quality but also increasing the completeness of data. In order to have
timely data to report to government and share with the public, the technical team had
to play the role of designer and user at the same time. For example, they had to check
data on an hourly basis to early discover any mistake in reported data. They had to
call hospitals to verify the figures and sometimes directly modified them. In another
flow of feedback, when a data entry clerk reported a figure through the system, the
figure was aggregated at the national and subsequently shared with the public. As a
member of the public, the clerk also received that aggregated figure and had her own
interpretation thus she became a user of the system.

Many studies report the problem of fragmentation in HISs in developing countries (see for example Chilundo and Aanestad [33]). Fragmentation refers to the situation where multiple, duplicate and overlapping systems coexist but cannot provide data necessary for decision making. While we agree that overlap in data collection should be minimized to the best extent, in some special occasions, it is a chance for restructuring and improvement. For example, in the case of traffic accident reporting, MPS and MoH are the two agencies that have routine data collection systems. The contradiction in data reported by the two systems made the public suspicious about the veracity of the data. The number of traffic accidents reported by MPS system was unreasonably too small compared to what reported by MoH system. As a result of this triangulation, the Head of National Traffic Safety Committee agreed that data from MoH system would be incorporated into the report next year.

5.4 Getting full-coverage data ready and publishing data

One of the issue that face HISs practitioners and researchers is the dilemma of “all
or nothing” (see Braa, Hanseth [3]). Incomplete data for a geographical area have
little value in use. Complete and Full-coverage data is only attainable when a full
scale implementation of HISs is achieved. Scaling, however, is a big challenge as it is
not about mechanically replicating the same technical artefact to other contexts but
also multiple sociotechnical rearrangements [34].

Research emphasizes the role of installed base in terms of existing technology and established use practices in shaping (and being shaped) the outcome of ICT interventions [35]. Taking advantage of what already exists to kick-start, or bootstrap [36], is an advisable strategy. The Tet reporting system was built on the existing DHIS2 software system already implemented in the hospitals. By leveraging the existing software, the team cut down the expensive and lengthy processes of full life cycle of software development. Since users at the hospital were knowledgeable of the DHIS2 system, expensive and time consuming training was not necessary. DHIS2 has a data
entry module that is highly configurable to adapt to a wide range of requirements. However, this module is sometimes too complicated for beginners. The technical team was inspired by the design of social media tools such as Facebook where anyone can create an account and start to use Facebook without attending any training classes. Therefore, they designed the system in a way that minimal or no training was needed. For instance, the data entry form was customized to enable data entry for multiple days in a single screen. Also, a report printing button was integrated into the data entry form, making it easier for users to view and export data to other formats such as Microsoft Excel for further processing. This approach was also more responsive to changing requirements. For instance, the Tet reporting system required an additional dataset. This could be easily be made through the dataset management module of DHIS2. No line of code was required. As a result, the system became ready in a short time.

Many researchers emphasize the role of good data in improving the quality of health service delivery [37, 38]. Substantial efforts have been made in building reliable and sustainable HISs. However, having data does not automatically lead to data use. Noir and Walsham [39] discuss the ceremonial and mythical use of HISs in the public health sector in India where health centers entered data into local computers, exported them to flash drives and carried the flash drives to higher level to submit data. The authors challenge the simplistic view that ICTs adoption will directly generate efficiency and gains. In our case study, the collected data were processed by the MoH team and subsequently shared with other stakeholders and news agencies to trigger data use process. The decision of MoH to publish the data to mass media created a feedback loop that reinforced the data reporting activities at hospitals, motivating hospitals to collect and report accurate data.

Publishing data goes in hand with accountability and transparency because MoH must be responsible for data it published. Prior to the implementation of this system, MoH only used the data for internal management. It was not obliged to make the data public. This decision was thus an important step toward public transparency. And public transparency led to public accountability.

# 6 Conclusion

Making life better is a goal of any ICTs project. This research presents an empirical case where data collected by a software system were used to trigger public debates that could contribute to shape policy and the public's attitudes. The approach discussed in the paper emphasizes the significance of selecting matters of public concern (few critical data elements) and rapidly scaling the implementation to get full data coverage. Collected data are disseminated and amplified by mass media to attract public's attention and trigger debates. Through the mass media, public express their concerns on critical health issues. As a collective effect, voices of people are better heard by the government. Lessons learnt from this research are hoped to provide pub-
lic health administrators and IT practitioners design ideas and principles that help
design sustainable HISs that empower people especially those who are marginalized.
Time constraint did not allow us to venture deeply into the interpretations of collected
data, i.e. geographical analysis of violent cases or traffic accidents. We believe the
quantitative analysis in this area will be useful for the public and policy makers in
preventing and reducing the cases related to traffic and violence. Our findings also
contribute to the debates of how to improve the efficiency of HISs through the use of
data for decision making.

References

1. AbouZahr, C. and T. Boerma, Health information systems: the foundations of public health. Bulletin of the World Health Organization, 2005. 83(8): p. 578-583.

2. Heeks, R., Information systems and developing countries: Failure, success, and local im-
provisations. The information society, 2002. 18(2): p. 101-112.

3. Braa, J., et al., Developing health information systems in developing countries: the flexible standards strategy. Management Information Systems Quarterly, 2007. 31(2): p. 381-402.

4. Nutley, T., et al., Moving data off the shelf and into action: an intervention to improve data-informed decision making in Cote d'Ivoire. Global health action, 2014. 7.

5. Braa, J. and S. Sahay, Integrated Health Information Architecture: Power to the Users: Design, Development and Use. 2012.

6. Davies, P., et al., Conceptualising the information needs of senior decision makers in health. Health Inform Syst Knowl Hub, 2011. 18: p. 1-20.

7. Lomas, J., The in-between world of knowledge brokering. Bmj, 2007. 334(7585): p. 129-132.

8. Bowen, S., et al., More than “using research”: the real challenges in promoting evidence-informed decision-making. Healthcare Policy, 2009. 4(3): p. 87.

9. Nutley, T. and H.W. Reynolds, Improving the use of health data for health system strengthening. Global health action, 2013. 6.

10. Network, H.M. and W.H. Organization, Assessing the national health information system: an assessment tool. 2008: World Health Organization.

11. Aqil, A., PRISM case studies: strengthening and evaluating RHIS. MEASURE Evaluation, USAID, 2008.

12. Aqil, A., T. Lippeveld, and D. Hozumi, PRISM framework: a paradigm shift for designing, strengthening and evaluating routine health information systems. Health Policy and Planning, 2009. 24(3): p. 217-228.

13. Patton, M.Q., Utilization-focused evaluation. 2008: Sage publications.

14. Braa, J., A. Heywood, and S. Sahay, Improving quality and use of data through data-use workshops: Zanzibar, United Republic of Tanzania. Bulletin of the World Health Organization, 2012. 90(5): p. 379-384.

15. Moyo, C., et al., The Information Transparency Effects of Introducing League Tables in the Health System in Malawi. The Electronic Journal of Information Systems in Developing Countries, 2016. 75.

16. Avison, D.E., et al., Action research. Communications of the ACM, 1999. 42(1): p. 94-97.

17. Winter, R., Learning from experience: Principles and practice in action-research. 1989: Falmer Pr.

18. Baskerville, R.L. and A.T. Wood-Harper, A critical perspective on action research as a method for information systems research, in Enacting Research Methods in Information Systems: Volume 2. 2016, Springer. p. 169-190.
19. Davison, R., M.G. Martinsons, and N. Kock, Principles of canonical action research. Information systems journal, 2004. 14(1): p. 65-86.

20. MoH_Portal. 6.200 cases hospitalized due to violence: usual or unusual? 2015; Available from: http://moh.gov.vn:8086/news/Pages/TinKhacV2.aspx?ItemID=569.

21. Life&Law. Prime Minister: Violence during Tet is not minor. Life and Law 2015; Available from: http://www.doisongphapluat.com/xa-hoi/thu-tuong-khong-the-xem-thuong-chuyen-danh-nhau-dip-tet-a84907.html.

22. Vu, A., Why aggressive behavior is on the throne. RFA, 2015.

23. Tuong, T., Every institution reports a different number: the public can not trust. 2016.

24. Ha, M. Death by traffic accident: contradictory figures. 2016; Available from: http://thahhnien.vn/thoi-su/chet-vi-tai-nan-giao-thong-moi-co-quan-bao-cao-mot-kieu-667572.html.

25. Schnackenberg, A.K. and E.C. Tomlinson, Organizational transparency a new perspective on managing trust in organization-stakeholder relationships. Journal of Management, 2014: p. 0149206314525202.

26. Charles, N. and W. Geoff, The great legitimizer: ICT as myth and ceremony in the Indian healthcare sector. Information Technology & People, 2007. 20(4): p. 313-333.

27. Sanner, T.A., T.D. Manda, and P. Nielsen, Grafting: Balancing Control and Cultivation in Information Infrastructure Innovation. Journal of the Association for Information Systems, 2014. 15(4): p. 220-243.

28. Abroms, L.C., R. Schiavo, and C. Lefebvre, New media cases in Cases in Public Health Communication & Marketing: the promise and potential. Cases in Public Health Communication & Marketing, 2008. 2: p. 3-9.

29. Heldman, A.B., J. Schindelar, and J.B. Weaver Iii, Social media engagement and public health communication: implications for public health organizations being truly "social". Public Health Reviews, 2013. 35(1): p. 1.

30. Newbold, K.B. and S. Campos, Media and social media in public health messages: a systematic review. Hamilton, ON: McMaster Institute of Environment and Health, 2011.

31. Willoughby, J.F. and H. Smith, Communication Strategies and New Media Platforms Exploring the Synergistic Potential of Health and Environmental Communication. Science Communication, 2016: p. 1075547016648151.

32. Braa, J. and C. Hedberg, The struggle for district-based health information systems in South Africa. The information society, 2002. 18(2): p. 113-127.

33. Chilundo, B. and M. Aanestad, Negotiating multiple rationalities in the process of integrating the information systems of disease specific health programmes. EJISDC: The Electronic Journal on Information Systems in Developing Countries, 2005(20): p. 2.

34. Sahay, S. and G. Walsham, Scaling of health information systems in India: Challenges and approaches. Information Technology for Development, 2006. 12(3): p. 185-200.

35. Grisot, M., O. Hanseth, and A.A. Thorseng, Innovation of, in, on infrastructures: articulating the role of architecture in information infrastructure evolution. Journal of the Association for Information Systems, 2014. 15(4): p. 197-219.

36. Hanseth, O. and M. Aanestad, Design as bootstrapping. On the evolution of ICT networks in health care. Methods of information in medicine, 2003. 42(4): p. 384-391.

37. Donaldson, M.S. and K.N. Lohr, Health Data in the Information Age:: Use, Disclosure, and Privacy. 1994: National Academies Press.

38. Rhoads, J. and L. Ferrara, Transforming healthcare through better use of data. Electron Healthc, 2012. 11(1): p. 25-31.

39. Noir, C. and G. Walsham, The great legitimizer: ICT as myth and ceremony in the Indian healthcare sector. Information Technology & People, 2007. 20(4): p. 313-333.