# Strategies for Standardizing Health Information Analysis

Olav Poppe, Johan Ivar Sæbø, Jørn Braa

► To cite this version:

Olav Poppe, Johan Ivar Sæbø, Jørn Braa. Strategies for Standardizing Health Information Analysis. 15th International Conference on Social Implications of Computers in Developing Countries (ICT4D), May 2019, Dar es Salaam, Tanzania. pp.260-271, 10.1007/978-3-030-18400-1_21 . hal-02285278

HAL Id: hal-02285278

https://inria.hal.science/hal-02285278

Submitted on 12 Sep 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Strategies for Standardizing Health Information Analysis.

## Flexible Standards Revisited.

Olav Poppe[1], Johan Ivar Sæbø[2] and Jørn Braa[3]

[1,2,3] Department of Informatics, University of Oslo, 0316 Oslo, Norway
olavpo@ifi.uio.no

**Abstract.** *Purpose:* This paper analyses an initiative led by WHO within the health information domain to standardise analysis of health information through the use of analytical dashboards, using the concept of flexible standards. We focus on the implementation of these standards within existing, working information systems, analysing the implementation strategies used, and how these are enabled by the flexibility of the standards.

*Design/methodology/approach:* The study follows an action research approach, where the authors have been involved in the development and implementation of the initiative being discussed. *Findings:* By analyzing the approaches taken by several countries to implement these standards we show how these different approaches are enabled by the flexibility of the standards. *Practical implications:* This paper demonstrates the potential importance of flexibility in standardisation initiatives around health information, with particular relevance to voluntary standardisation efforts involving independent actors, in this case Ministries of Health.

*Originality/value:* The flexible standards concept is employed to study a multi-country initiative involving WHO and several national governments. We contribute to the literature on flexible standards by showing that beyond flexibility in the standards, flexibility in the software platform in which the standards are implemented, and the variation allowed in the standardisation process at an organisational level, are important factors that facilitate standards implementations.

**Keywords:** Health Information Systems, Standards, Information use

## 1 Introduction

In a widely cited 2007 MISQ paper, Braa et al [1] proposed the concept of flexible standards, arguing that use of such adaptable standards can be a strategy for development of integrated health information systems (HIS) in developing countries. Empirically, they use examples from three countries of bottom-up processes that lead to the emergence of health information standards. In this paper we use the *flexible standards* concept to understand a top-down initiative led by WHO within the same domain.

The underlying problem that WHO seeks to address with this initiative is the lack of agreed data standards, overlapping and siloed data systems and poor data quality and data use in many developing countries. One of the key roles of WHO as a
normative organisation is to develop standards and guidance for its member states,
including in the area of health information. However, a pervasive challenge has been
the limited penetration and *use* of these standards by countries. Thus, while great
strides have been made towards improving the capacity of HIS to *collect* data, the
challenge remains that the relevant data is not always collected, and the quality of the
data is often poor.

As an effort to promote information use and adoption of its guidance, WHO decided to develop a set of standardized packages centred around *dashboards*. A dashboard for health management is typically a collection of appropriately defined visualizations like charts, maps, and simple tables, focusing on key *indicators* used to monitor the provision and quality of health services. The dashboards encompass standards at several layers: which indicators to display, how these indicators are defined, and how they should be presented to communicate key information in the most appropriate manner to support decision making. To strengthen the inherent normative values related to the use of the dashboards, WHO also developed a related public health curriculum. In summary the packages and curriculum include different types of standards, related to design, terminology, performance, and procedure [2].

The standard dashboards have been configured for the open source DHIS2 software platform, which is used for health information management in over 70 countries. Countries using this software platform can import a configuration file to install the WHO standard packages in their national systems.

The process of developing the standard configuration packages (hereafter referred to as just standard packages) described above is elaborated in Poppe et al [3]. The focus of this paper is on the implementation of these standard packages within existing, working systems in countries. Key problems associated with implementing standards in countries are related to lack of a clearly defined and authoritative procedure by WHO to facilitate their implementation, acceptance by user organisations, perceived use value in relation to cost of implementation and a certain momentum of users and other stakeholders implementing or being willing to implement the standard [2].

Different strategies have been followed to implement these international standards in national HIS. In this paper, we will identify and describe these strategies, and seek to identify what characteristics of the standards and the standardisation approach have enable these strategies. The rest of the paper is organised as follows: in the next section, we review the literature related to the *flexible standards* concept. In section three, we present our methodology, and in section four experiences from several countries that have implemented the standard packages. We then discuss the role of the flexibility of standards to our case, before concluding.

## 2 Related literature

Braa *et al* [1] argue that standards should be designed so that “they emerge as a complex adaptive system that can adapt to a changing environment and thereby contribute to the sustainability of the HIS”, and that this “can only be achieved if the
standards themselves are flexible” (pp. 396-397). Drawing on Hanseth *et al* [4], they argue that standards can have two forms of flexibility: *Use flexibility*, which refers to how a standard can be applied for different purposes or in different environments, and *change flexibility*, meaning how easy it is to change a standard. Change flexibility can be achieved by vertical and horizontal modularization, that results in a system of simple standards rather than one large and complicated standard.

Van der Ende *et al* [5] use a similar concept of “standard flexibility”, referring to “the number and degree of changes to a standard over time”. They argue that not enough attention has been given by researchers on standard’s characteristics and the effect of those characteristics on the content and survival of standards. They argue that more flexible standards are easier to adopt and have a better likelihood of succeeding.

## 2.1 Standardisation strategies

The main topic of the 2007 Braa *et al* paper is not *flexible standards* as such, but to propose a *strategy* for developing flexible information systems standards [1]. The strategy presented has two parts. First, to create an *attractor* “that emerges as a new standard and which evolves into a system of standards” (p. 396). Second, that “individual standards must be created in a manner which allows the whole complex system of standards to be adaptive to the local context” (p. 396). The strategy allows “radical change through small steps” (p. 399).

Hanseth and Bygstad [6] identify three different strategies for information system standardisation: *Anticipatory standardisation* is the traditional, formalized standardisation model; *integrated solutions* are, like anticipatory standardisation, a formalized approach, but focused on supporting user requirements rather than message specifications; finally, *flexible generification* is different in that it has “more focus on users’ practices and needs, a stronger focus on developing working solutions and a correspondingly lower focus on standardization as such” [6] (p. 656). This latter strategy is thus similar to the one proposed by Braa *et al*. [1], who suggest creating working solutions that become attractors and *emerge* as new standards. Nguyen *et al* [7] suggest that *meta-standardisation* should be added as a fourth strategy, which they define as developing new (meta) standards by connecting and mapping existing standards.

## 2.2 The Role of Technology

Several researchers also point to the role of technology in information systems standardisation, as well as for integration of HIS, which, we argue, is strongly related in that integration implies a level of standardisation. Effah and Abousi [8] study a national standardisation effort around a proprietary software system. While the top-down standardisation of the software itself succeeded, it failed to meet the information needs at sub-national level, and to support integration and interoperability with other systems.
Sæbø et al [9] studied strategies for integrating previously vertical information systems into a national data warehouse in four countries. In South Africa, a new minimal data set standard was developed in parallel with existing information systems. Zanzibar followed a more traditional standardisation process, where different stakeholders agreed in advance to a new data set standard, which was then implemented. The approaches pursued in Sierra Leone and Botswana were similar in the sense that no immediate harmonization or standardisation was agreed at the organizational level. However, in Sierra Leone, a form of semantic standardisation was handled in the data warehouse, solving the standardization “backstage” without involving the various actors. This was not done in the case of Botswana, and this lack of semantic standardisation created a complicated and difficult to use system. This highlights the role played by the HIS software platform as an enabler in integration and standardisation processes.

Braa et al [1] also highlight the importance of a flexible software solution that supports the development of flexible standards. Similarly, Nyella and Kimaro argue that the ability of the software platform being implemented to address needs of diverse actors made it an important tool in coordinating the process of developing of an integrated HIS in Tanzania [10].

## 3 Methodology

The research described here has been conducted as part of the Health Information Systems Programme (HISP). HISP is a large, long-term action research project doing research related to the development and implementation of sustainable HIS in low- and middle-income countries. HISP has evolved into a heterogeneous network of Ministries of Health, universities, individuals and organisations, described further in Braa et al [11].

The activities related to the standardisation initiative discussed here have been led by Ministries of Health, WHO and other global agencies. The authors have to varying degrees participated in the *development* of the standards from the start of the process in 2014, including discussing standard formats and potential implementation strategies, and creating the computer-based versions of the standards. This work is still ongoing. One of the authors was seconded to WHO for a 2-year period between 2014 and 2016, working primarily on this initiative.

We have been directly involved, to varying degrees, in the *implementation* of these standards in Sierra Leone, India, Guinea, Laos and Ghana. From these countries, we have collected data in the form of notes, documents and electronic communication. Furthermore, we have received additional data from colleagues in the HISP network who have been involved in the implementation of these standards both in some of the above-mentioned countries, as well as Uganda, Bangladesh, Togo and Mozambique.

Work in countries have generally involved, first, to assess the reporting structure of the national Health Management Information System (HMIS), e.g. existing reporting tools, indicator definitions, analytics dashboards and so on, and secondly, based on the assessment, country needs, and decision-making processes in-country, pursuing
different strategies to implement relevant WHO standard packages. This has in most cases been done for one or a few health areas at the time. The various strategies used is the topic of this paper.

## 4 Case

The project described here is part of a standardisation effort started by WHO in 2014, and later incorporated into the Health Data Collaborative (HDC) initiative. HDC is a “partnership of international agencies, governments, philanthropies, donors and academics, with the common aim of improving health data” [12]. The standardisation effort is centred around the development of best practice dashboards for analysis of health data within different health programme areas, which together with a training curriculum is meant to help countries improve their health information systems and make better use of the data that is collected routinely from health facilities.

A dashboard can be defined as “a graphical summary of various pieces of important information”<fnref n="1" /> (see figure 1). The graphics in this case are charts, maps and tables, which visualise, in different ways, key health indicators. Indicators are, within public health, used to denote information used to measure the extent to which health targets are met, e.g. “Immunisation coverage”. In addition to the dashboards and indicators, data elements denoting variables collected at facility level (e.g. “BCG doses given” or “Confirmed malaria cases”) are included, together with additional disaggregations (e.g. “0-11 months”, “1+ years” or “Male”, “Female”).

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Example of a dashboard, with visualisation of TB indicators.</figcaption></figure>

The specifications for the WHO dashboards, as well as the accompanying public health curriculum, can in theory be applied to any suitable technology, be it paper or various software applications. However, these dashboards, indicators, data elements and disaggregations have so far been configured only in the DHIS2 software. The configurations have then been exported as JSON files in the “DHIS2 Metadata Exchange Format” so that they can be imported into any DHIS2 database to produce the standard WHO configuration for one particular disease programme. Two versions

<footnote n="1">¹ https://en.oxforddictionaries.com/definition/dashboard</footnote>
of the packages have been created for each programme, in order to facilitate their adoption in different countries: one complete package including all content for data collection and data analysis (which we refer to as the *complete* package), and one that includes only the indicators and dashboards (which we refer to as the *dashboard* package). These are the *standard packages* discussed here.

DHIS2 itself is an open source, web-based software for collection, management and analysis of health information. The fact that DHIS2 has become a *de-facto* standard in Sub-Saharan Africa and parts of South-East Asia, which includes countries with a high burden of priority diseases such as Malaria, HIV and Tuberculosis, was the motivation for WHO to develop the standard packages.

In the rest of this section, we will look at the approaches taken in some of the countries that have adopted or adapted one or more of the standard packages. We focus of Sierra Leone and Laos, but present briefly some experiences from India, Uganda and Guinea as well.

## 4.1 Sierra Leone

In Sierra Leone, work on the WHO packages have been done both for TB and Malaria, following different approaches. For TB, a subset of the recommended data was already being collected in DHIS2 through the HMIS reporting tools, while the national TB programme collected data through a separate system. The national TB programme had been using data collection tools according to an older WHO standard and were in a process of updating these, and it was thus decided to make a coordinated shift both to using DHIS2 for TB data, and at the same time to start using the most recent WHO standard. The TB package was installed in the national DHIS2 database, and modified with the addition of some variables needed by the national TB programme.

The malaria programme already used DHIS2 as their main data collection and reporting tool, but only a limited set of analytical outputs had been configured. The WHO malaria package was not imported, but a consultant worked with staff from the programme and discussed their needs and the applicability of the indicators and dashboard items as suggested by WHO. The result was an “WHO inspired” dashboard, that built on and used the data and metadata already present in Sierra Leone.

## 4.2 Laos

DHIS2 has been used in Laos since 2014 and gradually most major health programs have been included in the national DHIS2 platform. In 2017 the Ministry of Health (MoH) decided that all routine data should be reported through the DHIS2. However, several health programs continued parallel reporting through their own systems, resulting in discrepancies in numbers between the two sources of data.

The EPI program was among the programs that were included in the national DHIS2-based system, but continued to collect more or less the same data in their own excel based system as well. When the standard package for immunisation became
available, it was decided to install the dashboard version of this and link it to the existing immunisation data collection tools in DHIS2. This demonstrated that the MoH DHIS2 system included the data required by the EPI program, according to the WHO recommendations.

The use of the TB, Malaria, and HIV packages follow a similar trajectory. The dashboards were imported in full, whereupon they would mostly display no data as little was in fact collected through DHIS2. However, the case of Laos illustrates how the implementation of the disease dashboards set off a range of other activities, which are worth noting. The various dashboards only work well if there is reasonably complete data, and if the right disaggregations are available. By importing the dashboards and using them to make visible the data quality issues, the various health programs would see both what would be possible with an integrated DHIS2, and what was missing to make it work properly. The process of setting up the dashboards for TB led to a revamp of the whole reporting structure for TB, which was previously managed with an Access based system. The same was done with Malaria and HIV; the former has implemented the standard package as part of a full revamp of their own system, including implementation of individual case-based management in DHIS2. The latter program implemented anonymous registration of all STI cases, which could be used to calculate any of the indicators in the dashboard

## 4.3 Other country experiences and summary

Table 1 summarises the experiences from Sierra Leone and Laos, as well as from implementations in India, Guinea and Uganda. These provide an illustration of the various approaches taken; however, the standard packages have also been implemented elsewhere. For example, a workshop was organised in January 2019 where the TB package was installed in the national HMIS of Benin, Burkina Faso, Mali, Ivory Coast, Liberia and Cameroon.

<table><caption>**Table 1.** Overview of implementation approach in countries</caption><thead><tr><td>Country and programme</td><td>Prior situation</td><td>Implementation approach</td></tr></thead><tbody><tr><td>Sierra Leone - TB</td><td>Subset of data in HMIS, parallel reporting by programme.</td><td>Complete TB package implemented in DHIS2 and customised with additional variables.</td></tr><tr><td>Sierra Leone - Malaria</td><td>Data collection in HMIS but limited analytical outputs.</td><td>Existing indicators and dashboards updated with WHO standard as reference/inspiration.</td></tr><tr><td>Laos - EPI</td><td>In HMIS, programme used parallel system</td><td>Dashboard package installed and mapped to existing data elements.</td></tr><tr><td>Laos - TB, HIV, malaria</td><td>Subset of data in national HMIS</td><td>Dashboard package installed and mapped to existing data elements, highlighting gaps in current data collection</td></tr><tr><td>India – malaria<sup>2</sup></td><td>DHIS2-based system being established. Malaria data</td><td>Malaria dashboard installed and modified according to available data. Standard</td></tr></tbody></table>

<footnote n="2">² National Vector-Borne Disease Control Programme</footnote>
<table><tr><td><strong>Guinea - TB</strong></td><td>Data collection configured in national HMIS, not yet used and without dashboards configured.</td><td>package used as reference to identify gaps in revision of data collection tools. Complete TB package installed, replacing the existing data set.</td></tr><tr><td><strong>Uganda - TB, HIV, malaria</strong></td><td>Data collection integrated in HMIS.</td><td>Dashboard packages modified to re-use existing indicators, before being installed and mapped to existing data elements. Modifications made according to national programmes' needs.</td></tr></table>

# 5 Discussion

Above, we presented some experiences from countries that have implemented WHO standard packages in their national HIS. While the standard packages cover different categories of standards as defined by Timmermans and Berg [2], the procedural standards, namely to manage various health programs by pre-defined steps of analysis, is the most important for WHO to achieve. How each country arrives there is to a large degree flexible and dependent on the existing system. We will in this section look at first, the different strategies pursued in these countries to implement the packages. Then, we discuss where the flexibility that enables countries to follow these different standardisation strategies lie: in the design of the standards; the software platform in which they are implemented; and at the organisational level. Finally, we identify some lessons learned and look at the broader implications of this initiative.

## 5.1 Adoption strategies

Looking across the strategies or approaches to implementing the standard packages, we can identify two dimensions. The first dimension is *how* the package was implemented: installed and used as-is; installed with modifications done before or after the installation; if it was manually replicated in the software platform without installing the standard package; or if it was used as an inspiration to make modifications to the existing system. Where metadata was installed or imported, a second dimension is the *type* of packages used for a particular health programme, i.e. the complete or dashboard version. This is illustrated in figure 2.

## 5.2 Levels of flexibility

Countries have been able to follow a number of different approaches in implementing the WHO standards. This, we argue, is a result of flexibility at several levels: in the standards packages themselves; of the software platform in which the packages are implemented; and at the organisational level, where some variations and modification to the standards being implemented is “permitted”.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Overview of implementation strategy taken by countries for different standard packages.</figcaption></figure>

Braa *et al* [1] argue that for standards to be adaptable to different settings, they should be developed as a simple system of standards, rather than one complex standard. Seen as a whole, the standard packages developed based on requirements from different WHO departments and health programmes can be seen as exactly such a system of standards. As described above, countries have adopted different packages, sometimes gradually with some time between each. The decision to adopt each of these packages can be politically sensitive, because it can have broader implications than the adoption of the standard itself, and this decision is typically made by the national health programmes. WHO *could* have developed one large, complex package covering all the relevant health programmes, that would have had to be implemented as a whole. This would have required simultaneous agreement and support from all national health programmes and would have had less chance of being adopted.

For the individual standards that make up the system of standards, Hanseth the *et al* define two types of flexibility: *use flexibility* and *change flexibility* [4]. We argue that each of the standard packages have both types of flexibility. The many ways in which the standards can be implemented, either as something that is installed in a software, used as a template which is replicated, or as inspiration, show that they have a high degree of *use flexibility*. *Change flexibility*, according to Braa *et al* [1] is achieved through modularization. The standard packages discussed here are modularized vertically by health programme, and layered horizontally, broadly speaking into a dashboard layer and a data collection layer. The importance of the vertical modularisation of the standard packages into health programmes was discussed above. It is also clear from our empirical material that the horizontal modularization has been important, enabling strategies of, broadly speaking, implementing complete packages or dashboard-only packages where existing data elements and data collection forms are used.

This horizontal modularization was possible because of the software in which the standards were being implemented. The DHIS2 data model has a layered structure, where all the key elements like visualizations, indicators, data elements and disaggregations into e.g. age and sex categories are independent units that can be
combined and configured independently of each other. This enables, for example, that the standard WHO *indicators* can be imported in the software, and then easily be configured and mapped to existing data elements that are not part of the imported configuration.

The flexibility of the software also allows for quite different implementation processes, technically, leading to the same result in terms of dashboards and indicators. One fundamental aspect of this is the flexible and layered data model, as explained above. However, another key element of the flexibility of the software platform is that it allows system administrators (as opposed to software developers) to import, modify or replicate the standard packages, or modify existing content with the standards as reference, largely through a graphical user interface.

While the flexibility built into the standards and in the software platform in which they are being implemented are important, flexibility in the approach to this process by WHO has also been important. We call this flexibility at the organisational level. While there is some variability in the view of different health programmes in WHO in how much countries should ideally change the standard configurations provided by WHO; the overall sentiment is that countries can, and in some cases should, adjust and adapt the standards to their context. It is also acknowledged that changes to data collection, which might be necessary in order to fully produce the outputs of the recommended dashboards, can be a multi-year process that involves development and printing of register and paper forms for all health facilities, training of health workers and so on. In those cases, it would be better to immediately configure a dashboard that is close to but not identical to the reference standard, rather than waiting for something “perfect”. The standards should be seen as enablers for information use, not constraints. Had the purpose of the standardisation effort been primarily to enable countries to provide reports to WHO, requiring semantic standardisation with identical identifiers etc, such a flexible approach would not be feasible.

Table 2 summarises the different dimensions of flexibility that have enabled the implementation strategies discussed in section 5.1.

<table><caption>**Table 2.** Dimensions of flexibility enabling diverse implementation strategies.</caption><thead><tr><td>Flexibility</td><td>Enabled by</td><td>Level</td></tr></thead><tbody><tr><td>Choice of standard package - dashboard only or dashboard and data collection</td><td>Design of standard, with two versions of each module<br>Loosely coupled data model in DHIS2 software, allowing configuration of indicators to existing data collection instruments.</td><td>Design<br>Software</td></tr><tr><td>Technical implementation - installation of package as-is, installation with modifications, or manual replication.</td><td>Configurability of the software platform by administrators (as opposed to software developers).<br>Focus on procedural standardisation, i.e. data analysis, rather than semantics (e.g. indicator identifiers).</td><td>Software<br>Organisational</td></tr><tr><td>Adaptations and modification to the implemented packaged.</td><td>Focus on procedural standardisation, i.e. improved data analysis, rather than semantics.</td><td>Organisational</td></tr></tbody></table>
## 5.3 Outcomes of the standardisation processes

All cases show some relation between the standardization processes and wider health information system design and development. In Guinea and Uganda, the health programs in question were already collecting or in the process of starting to collect their data through DHIS2, and only minor changes were done either to the imported material or the native configuration. In Sierra Leone, the decision to integrate TB into the national DHIS2 was seen as a non-controversial and logical concurrent project to setting up the WHO dashboard and reporting tools.

However, the events followed a different trajectory in Laos, where the introduction of the WHO standard packages worked as a key attractor for change both at the overall MoH level as well as within each of the programs. MoH had already decided that all health programmes should report through the national DHIS2 system, and the standard packages from WHO have been important in mobilising stakeholders behind this position, both through the authority of WHO as a standard setter and through the momentum building up by seeing other programs joining the approach. In particular in the EPI program this has been the important convincing factor for their acceptance.

The standardisation process in Laos provided an impetus for alignment of different health programmes into one integrated HIS. Similarly, it contributed to the integration of TB reporting in the national HMIS in Sierra Leone and in Guinea, though in the latter case the integration process had already started prior to the standardisation initiative. We see a similar tendency in other countries not presented here. This points to an interesting paradox of integrated independence, that is also revealed in particular in the case of Laos; while each of the health programs are strengthening their information system, the overall MoH DHIS2 platform framework is also strengthened and allowing data to be correlated and analysed across programs and through integrated dashboards.

One question that surfaces from the above discussion is to what extent this can be called standardisation, when there is so much flexibility in the standard itself and the way it is implemented? As stated above, while the standard packages cover different categories of standards [2], procedural standardisation, namely to manage various health programs by pre-defined steps of analysis, is the most important for WHO. Consequently, what needs to be achieved for the standardisation effort to be successful is standardisation related to practices of information use. Standardisation of the underlying components, i.e. the data elements, data collection forms, disaggregations and so on is of secondary importance. This is somewhat similar to the flexible generification strategy, with emphasises working solutions over standardisation as such [6]. The implication of this is that potential international reporting or exchange of data etc in the future is not necessarily facilitated by this standardisation effort.

# 6 Conclusion

This paper draws on experiences from a handful of countries that have adopted one or more WHO standards for data analysis. Ministries of Health have followed different
strategies in order to adopt these international standards in their national HIS, and in
this paper we have sought to identify the characteristics of the standards and the
standardisation approaches that have enabled these strategies.

Drawing on the concept of *flexible standards*, we have shown how implementation
of the standards by countries have been facilitated by flexibility built into the
standards, flexibility of the software platform in which the standards are implemented,
and flexibility at the organisational level. This distinction, in particular the importance
of flexibility in the software in which the standards are implemented, adds to the
existing literature around flexible standards. Furthermore, this paper shows how the
flexible standards concept can be applied also within an international standardisation
process, involving independent national Ministries of Health.

Our empirical data also shows that this standardisation process, nominally focused
on standardising and encouraging information use, has wider implications. For
example, by highlighting data management and quality issues, and as a driver for
integration of vertical information systems. How the WHO standards can be attractors
for change is an area for further research.

7 References

1. Braa, J., O. Hanseth, A. Heywood, W. Mohammed, V. Shaw. 2007. Developing health information systems in developing countries: the flexible standards strategy. *Management Information Systems Quarterly*, 31(Special Issue).

2. Timmermans, S. and M. Berg. 2003. *The Gold Standard*. Philadelphia: Temple University Press.

3. Poppe, O., J. I. Sæbø, P. Nielsen and T. A. Sanner. 2017. Standardising Through Software. *Selected Papers of the IRIS*, 3(8).

4. Hanseth, O., Monteiro, E. & Hatling, M., 1996. Developing information infrastructure: The tension between standardization and flexibility. *Science Technology Human Values*, (21).

5. van den Ende, J., G. van de Kaa, S. den Uijl and H. J. de Vries, 2012. The Paradox of Standard Flexibility: The Effects of Co-evolution between Standard and Interorganizational Network. *Organization Studies*, 33(5-6).

6. Hanseth, Ole and B. Bygstad. 2015. Flexible generification: ICT standardization strategies and service innovation in health care, European Journal of Information Systems, 24(6).

7. Nguyen, T., T. Chi Pham, E. Santos, H. T. Son. 2018. Meta-standardization as a remedy for tensions and complexities: The case of medical service lists in Vietnam. *The Electronic Journal of Information Systems in Developing Countries, 84*(2).

8. Effah, J. and A. Abuosi. 2013. Standardizing a developing country health information system through proprietary software: Ghana's experience. *Journal of Health Informatics in Developing Countries, 7*(2).

9. Sæbø, J.I, E. K. Kossi, O. H. Titlestad, T. R. Rolland and J. Braa. 2011. Comparing strategies to integrate health information systems following a data warehouse approach in four countries. *Information Technology for Development, 17*(1).

10. Nyella, Edwin E. and H. Kimaro. 2015. HIS Standardization in Developing Countries: Use of Boundary Objects to Enable Multiple Translations, 8(1).
11. Braa, J., E. Monteiro, and S. Sahay. 2004. Networks of action: sustainable health information systems across developing countries. *Management Information Systems Quarterly*, 28(3).

12. Health Data Collaborative. 2018. Who We Are. https://www.healthdatacollaborative.org/who-we-are/. Accessed 10.10.2018.