# Sustainability Qualifiers of Health Management Information Systems Implementation: Case Study of DHIS2 in India

Jyotsna Sahay, Sundeep Sahay, Arunima Sehgal Mukherjee

► To cite this version:

Jyotsna Sahay, Sundeep Sahay, Arunima Sehgal Mukherjee. Sustainability Qualifiers of Health Management Information Systems Implementation: Case Study of DHIS2 in India. IFIP Joint Working Conference on the Future of Digital Work: The Challenge of Inequality (IFIPJWC), Dec 2020, Hyderabad, India. pp.194-206, 10.1007/978-3-030-64697-4_15. hal-03450692

HAL Id: hal-03450692

https://inria.hal.science/hal-03450692

Submitted on 26 Nov 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Sustainability qualifiers of Health Management Information Systems implementation: case study of DHIS2 in India

Jyotsna Sahay<fnref n="1" />, Sundeep Sahay<fnref n="2" />, and Arunima Sehgal Mukherjee<fnref n="1" />,<fnref n="2" />

**Abstract.** This paper reports on a rare story of sustainability success of a HMIS implementation in a low and middle income country context of India. The narrative is set in the Indian state of Odisha, where the DHIS2 is being implemented for the state Health Management Information System since 2008. The authors of this paper have been engaged both in research and practice supporting the implementation since the start, and draw upon this rich and longitudinal data source to analyze from the perspective of the state government "the challenges they faced with respect to sustainability, and how have they overcome them" Three sets of sustainability qualifiers were identified -benefits/continued benefits, institutionalisation / routinisation and development - which have been key in enabling sustainability. However, going forward, the authors identify the need for additional qualifiers to strengthen the aspect of data use, which till date remains weak as compared to data quality management.

**Keywords:** Sustainability, HIS, LMICs

## 1 Introduction: Importance of HMIS to strengthen health systems performance

There has been a growing awareness that improving population health is significant for development of a society, particularly for low and middle-income countries (LMICs), who are typically under resourced and fragile. The World Health Organisation (WHO) identifies Health Information Systems (HIS) as one of the core building blocks to strengthen health systems. India, like many other LMICs have made concerted efforts to reform and integrate its systems, with the routine Health Management Information System (HMIS) being the springboard for reform. Rationalisation of the existing data management system and related practices and transition from legacy paper-based system to ICT enabled ones are at the core of the reform.

<footnote n="1">¹ HISP India, India</footnote>

<footnote n="2">² Department of Informatics, Universitetet i Oslo</footnote>
Our paper focuses on the sustainability of the HMIS, which provides the foundation of the national health information system (HIS), spanning from the lowest level of the community through the district and state levels to the national and global. The HMIS provides data for both strengthening local level care processes and also for state and national level programme management and policy making processes. By the fact that the HMIS contains data from all public health facilities in the country, it becomes the foundation for providing effective health services to the whole population, particularly the rural and marginalized populations who cannot afford privately delivered care services.

Given that the HMIS spans the country, it is inherently complex and houses multiple competing interests, and thus challenging to successfully implement. IT. Efforts globally to strengthen HMIS through computerisation have resulted in large-scale failures [1] and the potential of technology remains largely unrealised. A key enduring challenge thus is of sustainability, implying the capability of the HMIS to endure over time and space, with-out external support. The dual challenge of achieving sustainability is of how systems can be deeply institutionalised within governmental systems so best serves the needs of the state, while yet being flexible and adaptable to evolve with the evolving informational needs and priorities of the government. A key challenge in the analysis of sustainability is the dominant focus on the “supply side” (provision of computers, mobile phones, internet etc), which ignores an understanding of the “demand side” dynamics (such as user needs and their capacities). This paper seeks to address this bias, by understanding the HMIS sustainability challenge, as seen from the perspective of a state government in India. The following research question is addressed:

What challenges of HMIS sustainability does a state government in a LMIC context experience, and how can they best overcome these challenges?

We approach this research question through an analysis of a ten-year experience of a HMIS based on the free and open source digital platform (DHIS2- District Health Information System – see dhis2.org) implementation in the Indian state of Odisha. This analysis conducted within a temporal perspective, provides rich insights into how a state government could strive to address the various socio-technical sustainability challenges encountered, with the aim of improving their health system performance.

The paper is organised as follows. In section 2, we discuss conceptually the challenge of sustainability. Section 3 details the methods followed in the case study narrated in section 4. Our case study analysis, with a focus on sustainability is presented in section 5, which is followed by discussions and conclusions in section 6.

## 2 The enduring challenge of building sustainability

This section comprises of three main parts. In the first, we discuss some key sustainability challenges to HMIS as reported in literature. In the next, we discuss some conceptual approaches to study sustainability, and their respective strengths and weaknesses. Finally, we present our conceptual framework to guide the analysis.
## 2.1 Challenges to HMIS sustainability

Traditionally computerization of HMIS in LMICs have followed technical trajectories representing a “supply side push.” Sustainability tends to get equated with establishing required digital infrastructure and mitigating risks threatening the long-term viability of IT [2]. This approach ignores the socio-technical nature of the challenges, such as how to deal with legacy systems and their institutionally embedded practices. Computerization efforts cannot start from scratch, as history is important with both enabling and constraining influences on new initiatives [3][4].

Another implementation challenge concerns the centralized nature of initiatives, which marginalize supporting care processes responsive to community needs [5]. This strong predisposition of national health programmes to be vertically structured, often supported by different donors for specific programme needs, have led to a proliferation of fragmented and compartmentalized systems, that are not able to sustain since they are not nurtured within a state unified framework. Reform efforts thus often miss out on an organized and unified central data resource and requisite infrastructure required to build integrated health information architecture [6].

Another impediment to sustainability concerns the weak governance and stewardship of national HMIS, which limit the use of health data standards and clearly defines the roles and responsibilities of different stakeholders towards the HMIS. While many LMICs, including India, have come out with policy announcements to support free software and open standards, in practice this does not happen in an effective manner, and governmental procurement systems continue to support proprietary systems [7].

The raison d'etre of a HMIS is to support "data use for action". Research abounds on stories of data not being used for action but primarily only to support upward bureaucratic reporting. Various reasons contribute to this including capacity, infrastructure, high existing workloads and weak culture of using data for action. Weak data use implies poor demand for data, and increasingly less attention being paid to providing quality data. This leads to a vicious cycle of data not being used leading to poorer quality data and more non-use. The end outcome of this vicious cycle is that the HMIS does not sustain over time [8].

In summary, multiple conditions impede the realization of sustainable HMIS. We next discuss how these can be conceptually understood.

## 2.2 Conceptually understanding sustainability

The Information Systems research perspective offers different perspectives to understand sustainability. We discuss some of them, along with their strengths and weaknesses.

The *diffusionist perspective* inspired by Roger's model [9] traces the trajectory of an innovation and its adoption over time conceptualized as an “S” curve. This diffusion approach focuses primarily on the supply side of an innovation, assuming that it takes birth at the “centre” and then gradually is adopted at the “peripheral” levels. Attributes such as donor funding and system usability and perceived usefulness qualify sustainability concerns. We believe this approach is limited in its understanding of sustainability as it does not actively consider the demand side of user practices which emerge
within an embedded social and institutional context. It primarily represents a top-down approach focusing on preserving the technical fidelity of the interventions [10], ignoring local user needs [2][11][12] and ignoring that local processes are dynamic, non-linear, full of unexpected events, and rife with probability of 'sustainability failure' [13]. Nhampossa [14] has argued against the efficacy of the diffusionist perspective in the context of Mozambique. He writes that such an approach is limited as it assumes that knowledge and innovation will only emanate from the centre, and the periphery is incapable of building anything new. He advocates instead to adopt a "translation" approach, where the movement of technology is not seen as one giant leap from point A to B, but a series of small steps or translations, where at each steps new forms of socio-technical networks emerge, which shape the process of evolution and also the contents of the technology.

To address challenges inherent in techno-centric and top-down approaches, the socio-technical philosophy has been drawn upon by IS researchers. This perspective conceptualizes technology as a “socio-technical network” [12] wherein boundaries between the social and technical are blurred and organisational actors enact distinct technology-in-practices based on their cognitive schemas and the social contexts they are embedded in [15]. Participatory design based approaches become important in shaping the technology based practices. Understanding the interplay of contextual dynamics and actors’ rationality, stakeholders and their interests, and the logic of their negotiations [6] are central to these approaches [16] and to identify the design-reality gaps and build sustainability [17].

Braa et al. [17] advocate the networks of action approach to address the HMIS sustainability challenge in LMICs. This action research approach seeks to enable learning in collectives rather than in isolated instances, to enable sharing of experiences and resources, and avoid “reinventing the wheel of mistakes”. Learning amongst peers fundamentally challenges the top-down diffusionist thinking, and the action in this approach is to enable the sharing of resources, ideas and experiences across the different actors in the action network. This approach has been key in the sustainability of HISP (Health Information Systems Programme) network over 80 countries through two decades. Enduring the test of time and scale, is an active proof of sustainability.

The practice based approaches operate at a more micro and pragmatic level, on the assumption that systems become sustainable manner only when it is routinely used institutionalized in the everyday work of the organization. This includes stakeholders engaging in joint activities, insightful discussions, and experience & knowledge sharing, as a key means of addressing recurring problems [18][19]. The practice based approaches has been combined with the institutional work perspective to understand how practices are shaped by and also shape institutional influences [20]. While helpful in grasping the key requisite of sustainability – institutionalisation, these approaches leave unaddressed the challenge of how to translate the requisite into practice, and how systems respond are to change. To address this gap, Fleiszer and others [21] discuss the “qualifiers of sustainability”, which we now discuss.
**Analytical perspective based on “sustainability qualifiers”**

Fleiszer et al [11] enumerate the following qualifiers of sustainability:

a) *Benefits, continued benefits.* Sustainability is typically equated with the persistence of the innovation-related benefits. [22][23][24]. Effective innovations are sustainable if they continue to provide benefits [25] with respect to the objectives of the innovation for clients [23] [26] and system providers [27]. This involves the continued maintenance of resolution to problems and their enhancement [23][28]. While achievement of objectives is central to sustainability, equally important is how benefits are perceived, beyond what is documented by formal evaluation [28].

b) *Routinisation/institutionalisation.* This refers to the embedding of structures and processes around an innovation into habitual practices of individuals, organisations and systems [13][22][29][30]. This involves a process of ‘mutual adjustment’ between an innovation and its context, such that the innovation eventually ‘loses its separate identity’[22 p. 94][28] and becomes standard ‘business as usual’[31 p. 261]. While routinisation implies cycles of repeated action in a social structure, institutionalisation implies the concretisation of organisational infrastructure (e.g. established committees, dedicated budgets, embedded data management technologies) around the routines [32].

c) *Development.* Fleiszer et al [21] identify two inter-connected perspectives around development. One, one which addresses the evolution of the innovation and another that draws attention to the emerging changes in stakeholder needs and how the innovation adapts to them. Development then represents additional or ongoing innovation [30][33]. While a management-focused approach focuses on performance improvements, a stakeholder centric approach focuses on ‘ongoing development’ of an innovation [34] in response to evolving circumstances [33] [35]. From this perspective, sustainability represents the continual enhancement of users’ abilities and resources to maintain an innovation and associated changes [22][24][26][28].

In summary, our analytical perspective is built around the identification of sustainability qualifiers, focusing on user-based practices and their shaping by the institutional context. Such a perspective helps place the focus primarily on the demand side of the innovation, as contrasted to the typical supply side bias that underpins many sustainability analysis.

# 3 Empirical approach and methods

## 3.1 Research setting

The empirical component of our analysis is the eastern Indian state of Odisha with a population size of 47 million. Odisha has a high degree of geographic inaccessibility of health services, a significant tribal population, a heavy reliance on informal health providers, and affected periodically by natural disasters such as floods and typhoons. Despite these odds, the state has made remarkable progress on strengthening and sustaining their HMIS, which makes it an interesting case for us to analyze on how this has been achieved from the perspective of the state.

Our empirical work involves two forms of engagement of practice and situated research. In terms of practice, the authors of this paper have been engaged with national
and state level HMIS reform efforts through their design and implementation since 2008 till today. This engagement has emerged through a local NGO called Health Information Systems Programme, India (HISP India) which has supported Odisha in their HMIS strengthening since 2008. This engagement has provided rich insights into the context, including various centre-state tensions. In terms of research, we have conducted in 2019 a detailed empirical analysis of selected districts in the state, to understand how the HMIS has evolved over a 10-year period. Our focus has been on understanding the implementation trajectory of the HMIS, and what are the sustainability qualifiers and how they have been achieved.

## Data collection

**Mode of practical engagement:** In 2008, the National Rural Health Mission (NRHM), Ministry of Health, started a process of health systems reform for strength-ening public systems. The aim was to bring architectural corrections, including in the HMIS by making it more decentralized, standardized and evidence based. The authors were integrally engaged in this reform process, and its subsequent implementation in states, including Odisha. As a part of this process, we visited Odisha 3-4 times, and had discussions and presentations with the state on how they wanted to adopt the reform, including the new formats and software system. As a part of the reform process, the national level designated a centralised web-portal built on a proprietary platform, for all states to report their HMIS data. Odisha adopted the DHIS2 open source platform, as the state portal, and to also comply with their national reporting requirements.

This national level process continued till 2012, after which Odisha made a bilateral arrangement with a NGO, HISP India, to support their DHIS2 based HMIS, which continues till today. This long term engagement of HISP India, of which the authors are members off, has yielded immense data in terms of technical reports, presentations, correspondence with the state and contracts. This data has been systematically compiled, key events mapped, and the implementation trajectory identified.

**Mode of situated empirical assessment:** A second source of data was through in person interaction with DHIS2 users and related stakeholders at the state and sub-state (district and block) levels. The district constitutes the middle layer of linking the state to the block which is responsible for service delivery. The fieldwork was carried out in two phases over a period of four months. (March – July 2019). Six of the thirty districts in the state were covered, and in each, three blocks were visited. Some sub-block level interactions were also carried out.

**Phase 1.** This was over a week and carried out by a team of four members with multi-disciplinary expertise. The team focused on identifying key practices around data collection, reporting, analysis and use practices, including how the DHIS2 supported or not these practices. At the state, the researchers met the IT and M&E team responsible for the HMIS upkeep and use. The district and sub-district visits then followed, to meet the data and programme managers.

**Phase 2.** Here, interactions at the district level M&E team formed the starting point, followed by block and sub block level visits. The focus was on primarily understanding the field-level practices around the collection, reporting and use of data for local action.
In addition to speaking to users, various paper-based data records and reports, both formal and informal, were studied to understand how DHIS2 was being used.

## Data analysis

The focus was to interpretively understand the unfolding of the implementation trajectory, comprising of the interplay of contextual contingencies, stakeholders’ interests and the mediation of technology. At the end of each day, each researcher made their field visit notes, which summarized the discussions, key observations and learnings. This was followed by joint discussions as sense-making sessions of the implementation trajectory and the underlying sustainability qualifiers. Prior to moving to the next district, the notes were revisited, and comparative experiences were also discussed. Slowly, an overall picture of the implementation trajectory was pieced together and mapped against the data collected through the mode of practical engagement.

# 4 Case study

DHIS2 was introduced in Odisha as a state portal to support facility level reporting and analysis while enabling export of required data to the national portal. Being open source and designed with user-friendliness and flexibility in mind, the State saw DHIS2 as an effective tool to support their decentralisation agenda. The implementation trajectory is outlined over three phases.

**Phase I.** Initiation (2008-2010): A situation analysis conducted as a part of the national reform process, also included Odisha state. In addition to the national man-dated data standards, HISP India was part of a state specific process to identify the local data needs, which necessarily did not need to be reported to the national level. Through the flexibility offered by DHIS2, these state requirements were incorporated into the database, and its deployment was supported by a series of workshops, orientation and training sessions. The software and technical support was provided free to the state, supported by a national budget, a process, which continued till 2012.

**Phase II.** Transition to DHIS2 based systems (2010 – 2012): This phase involved complete transition from the paper based HMIS to DHIS2. The platform adaptation, capacity building and handholding support were major activities carried out in this period, working closely with the state and district level teams. Customizations were primarily about incorporating state specific requirements not addressed by the national process. In addition, trouble shooting and bug fixing issues were addressed on a continuous basis, particularly focused on strengthening work practices around data quality management and data use.

**Phase III.** (2012 – ongoing) In 2012, national level support to states was withdrawn, and Odisha was advised to develop bilateral relationships with HISP India to continue support for the DHIS2 implementation process, if they so chose. By this time, the state HMIS team had gained adequate understanding of DHIS2 and were seeing its value in practice of it being able to support their local needs. They found that DHIS2 could rapidly support their evolving needs at a low cost, without having to build systems from
scratch. They thus decided to continue the HISP India relationship, something which
continues till today.

**4.1 Understanding the outcomes of the implementation process**

Three key outcomes of the implementation process were identified: i) significant im-
provements in data coverage; ii) significant improvements in data quality; and, iii) a
steady progression towards an “integrated state data warehouse” based on DHIS2.
These three value-adding processes are now discussed.

**Improved processes of data coverage.** These benefits arose through the transition from a paper-based to a web-based free and open source digital platform – DHIS2. This transition address historical challenges related to redundant and inconsistent data formats, inconsistencies in data recording and reporting practices, errors resulting from manual data processing and lack of timeliness in reporting. While the earlier system was geared only towards upward reporting, the DHIS2 allowed for the first time for local levels to see their own data, and address problems of inflated reporting. Now even the block level users could directly do data entry into the web-based system, and slowly the state achieved 100% data coverage. A district user commented on the value of increased visibility of data:

*Increasing numbers of data elements means more data to be presented in monthly meetings. It becomes possible to focus on poor performers. Increasing data quality and reporting rate has also been seen in the district. Urban areas with many private facilities are currently the focus as they have lower reporting rates and poorer data quality*

**Data quality management.** Well-defined institutional protocols for data quality management were defined around data reporting and evolved through regular use. These protocols took into account the existing practices related to data collection review, confirmation and submission, and built upon aim. The aim was not to eliminate the legacy practices but to build upon it, make them more stringent and visible through digitization. To support digitization, explicit guidelines, responsibilities and resources were agreed, such as analysis of data validation errors generated by the system. Through data rationalization, the amount of data to be collected was significantly reduced, enabling improvements in data quality. The DHIS2 allowed for easy identification of facilities that did not report data, enabled correctness checks at the very point of data entry, could lock the data once confirmed to prevent late changes in data, run data validation checks, and implement functionalities such as role based user authorizations.

These technical improvements were only made possible through the establishment
of institutional processes. Validation committees at each level of the reporting hierar-
chy (state, district and block) were setup to routinely review data quality and to provide
support for making corrections. The validation committees did not seek to change the
authority structure but to enhance the visibility and accountability of the information
function. Conversations around data of the committees and data providers helped in-
creasing awareness about the value of data and to build a sense of pride around its
upkeep. With increasing maturity in data quality, there was a shift from data use for
performance evaluation and control to corrective action to strengthen service delivery.
With this, increasingly more stakeholders got involved in the conversations around data. A block level functionary noted:

> "... 2006 there was a big booklet of around 25+ pages that he had to fill. Each data element was disaggregated by Schedule Caste, Schedule Tribe and Others, further broken by male and female. Data was entered in Excel to be aggregated, which generated a lot of mistakes. District level only had block level data and could not drill down... Now, we are getting directly contact with the lowest level, we can look at their data and call them if something is wrong. Even state level is in contact with the lower levels now as they can see their data. State level is also contacting SCs"... In 2006, only one programme was included in HMIS. Now they have integrated multiple programme to make reporting easier ..."

**Gradual progress towards an integrated state data warehouse.** Driven by the initial agenda of the implementation, the early focus was on strengthening data coverage and quality, primarily for the national level programmes. Over time, requirements emerged for addition of both national and state specific programmes, which were incorporated in the DHIS2. As many as ten programmes have now been integrated, and the DHIS2 is evolving into a state data warehouse, as a repository of all health data. However not all the integrations have been well embedded into routine use, and uptake has been variable. The focus now is on building the use of the data, and creating a more integrated perspective in terms of use and policy making.

# 5 Case analysis: identifying sustainability qualifiers

Our analytical framework identified three key sustainability qualifiers: i) benefits, continued benefits; ii) routinization and institutionalization; and iii) development. We discuss here how these qualifiers were achieved in the empirical case, from the perspectives of the data providers, the M&E and IT teams.

**Benefits, continued benefits:** For the data providers, data rationalization reduced their workload, and DHIS2 enabled a transition from the manual and time taking work to something more easy, accessible and more efficient. Achieving 100% coverage, meant data providers could avoid the threat of reprimand for non or late reporting. For the M&E team, all data in one database, enhanced visibility of the health status, and their ability to drill down to the lowest level to identify events and their causes. This greatly strengthened their analytical abilities to conduct monitoring and evaluation. For the IT team, there was firstly the opportunity to work with a state-of-the-art digital platform, and in collaboration with the responsive HISP India technical support team, also build their capacities and strengthen local ownership. This was self-motivating.

**Routinization, institutionalization:** The process of routinization and institutionalization is well illustrated through the data management function. An explicitly defined protocol for data quality management, was established by the state, imparting it the necessary legitimacy and became routinely used across the state. The protocol took was based on legacy data management practices, rather than creating something radically new, and were thus not perceived as a threat. Mechanisms such as data review meetings and Validation Committees provided the structure to ensure compliance of the designed practices and processes, thus leading to a deep institutionalization of practices.
**Development:** Key to the sustainability process was the continuous development taking place both technically and institutionally. For example, the guidelines for data quality management was continuously improved with due discussions and clarifications with relevant stakeholders and disseminated at regular intervals. Various technical improvements were introduced, such as multi-level data quality checks, self-validation of data, supervisory approvals prior to report submission, and integrated reports at block level which were used for review meetings. Supporting these development processes was a stable and visionary governance structure, who realized the need to have a strong and low-cost technical partner in HISP India. Whenever new features were required or modifications and troubleshooting, HISP India would quickly respond leading to an ongoing evolution of systems. These developments were of course enabled through the flexibility and customizability offered by DHIS2, and that it was not locked through proprietary licenses. Some areas of development have proved to be tricky, such as the scaling of a malaria surveillance system, because of national level priorities being different. In some development exercises, the legacy practices were deeply embedded and thus challenging to deinstitutionalize.

In the following summary table, we highlight the key sustainability qualifiers.

<table><caption>**Table 1:** Sustainability qualifiers shaping Odisha HMIS implementation trajectory</caption><thead><tr><td>Sustainability qualifiers</td><td>Supporting practices</td><td>Institutional shaping of practices</td></tr></thead><tbody><tr><td>Benefits, continued benefits</td><td><strong>Data providers:</strong> reduced data load; easier access to entry; improved quality<br/><strong>M&E team:</strong> Increased visibility of data; improved visualization and analytics<br/><strong>IT team:</strong> Working on state of art technology with responsive HISP India support team</td><td>Establishing <strong>validation committees</strong> to have monthly conversations around data<br/><strong>Monthly review meetings</strong> based on HMIS data<br/>Ongoing <strong>support agreements</strong> with HISP India continued over 10 years</td></tr><tr><td>Routinization, institutionalization</td><td>Routines of review meetings, data quality checks, data analysis and use were clearly established, and imparted necessary legitimacy to be institutionalized, such as through budgets and human resources</td><td>District and block level structures created for data quality review and use; at state level, dedicated IT team and data center identified; state M&E team enhanced the demand for quality data, putting pressure on the supply side</td></tr><tr><td>Development</td><td>Demand for new functionalities and modules rapidly incorporated; regular capacity building to build awareness and use</td><td>State promoted the progressive policy of building an integrated data warehouse based on DHIS2</td></tr></tbody></table>
# 6 Discussion and conclusions

The paper presents a rare example of a HMIS in a LMIC that has endured and continues to thrive even after 12 years. This has been despite various pressures such as varying national priorities on software and many other competing systems. In an environment where there are frequent changes in state government administration, the Odisha governance structure has been consistent in supporting DHIS2 and the technical team – HISP India – behind it. The state approach and actions highlight their acknowledgement that system implementation is a matter of years and not months and the importance of supporting both continuity and innovation.

The paper contributes to understand sustainability and how it can be achieved from the perspective of the state government. The notion of sustainability qualifiers seen as user practices and their institutional shaping, provides a framework to understand key determinants of sustainability. All these qualifiers are seen within a temporal and not a snapshot perspective – how benefits are continued, how routines are institutionalized over time, and how developments continue with ongoing innovation and support.

While the state can indeed be proud of their continued sustainability of their HMIS, some qualifiers are identified for enabling continued evolution. While great progress has been made in establishing systems and processes for higher quality and coverage of data, the same progress cannot be noted on data use. Good quality data can be seen as a necessary but not sufficient condition for enhancing data use. This requires additional qualifiers such as institutional incentives, dissemination of successful case studies and stories, and the foregrounding of data use champions. Further, the process could benefit by expanding the network of stakeholders who depend on the data beyond the health sector, such as the local politicians and district administrators. This expansion will help to strengthen the demand side, and create champions on data use, which can serve as self-reinforcing mechanisms to attract other users and made innovations.

While our empirical case is focused on Odisha and the DHIS2 platform, there are some general principles which we believe can be relevant to other contexts and systems. One, to focus on sustainability from the perspective of the state and the owning institution. Two, to understand sustainability from a temporal perspective. The three qualifiers have helped to identify key determinants of sustainability. Four, qualifiers are best understood through the lens of user practices and how they are institutionally shaped.

# References

1. Heeks, R.: Information systems and developing countries: failure, success, and local improvisation. The Information Society 18(2), 101-112. (2002). doi:10.1080/01972240290075039.

2. Korpela, M., Soriyan, H., Olufokunbi, K., Mursu, A.: Blueprint for an African systems development methodology: An action research project in the health sector. In: Avgerou C (ed) Implementation and Evaluation of Information Systems in Developing Countries. Proceedings of the Fifth International Working Conference of IFIP WG 9.4, Bangkok, Thailand.273-285. (1998).
Aanestad, M.: Cultivating Networks: Implementing Surgical Telemedicine. Department of Informatics, University of Oslo. https://www.researchgate.net/publication/242383259_IM-PLEMENTING_SURGICAL_TELEMEDICINE. (2002).

3. Callon, M.: The sociology of an actor-network: The case of the electric vehicle. In: Callon M, Law J, Rip A (eds) Mapping the dynamics of science and technology Macmillan Press, London, pp 19-34. (1986). doi: 10.1007/978-1-349-07408-2.
Mukherjee, A.: Empowerment: The invisible element in ICT4D projects? The case of public health information systems in India and Kenya. Department of Informatics, University of Oslo, Norway. (2017). https://www.mn.uio.no/ifi/english/research/networks/hisp/research-library/thesis/mukherjee2017empowerment.pdf.

4. Braa, J., Sahay, S.: Integrated health information architecture: power for the users: design development and use. Matrix Publishing, New Delhi. (2012).

5. Sahay, S.: Free and Open Source Software as Global Public Goods? What are the distortions and how do we address them? Electronic Journal of Information Systems in Developing Countries. 85:e12080. (2019). doi: 10.1002/isd2.12080.

6. Misund, G., Høiberg, J.: Sustainable Information Technology for Global Sustainability. Digital Earth - Information Resources for Global Sustainability Symposium 21-259 (2003) Brno, Czech Republic, accessed: from http://wwwia.hiof.no/~gunnarmi/omd/dig_earth_03/

7. Rogers, E.: Diffusion of Innovations. 5th edn. Simon & Schuster, New York (2003).

8. Carroll, C., Patterson, M. Wood, S., Booth, A., Rick, J., Balain, S.: Conceptual framework for implementation fidelity. Implementation Science 2(40). (2007). doi: 10.1186/1748-5908-2-40.

9. Venkatesh, V., Morris, M., Davis, G., Davis, F.: User acceptance of information technology: toward a unified view. MIS Quarterly 27(3), 425-78. (2003). doi: 10.2307/30036540.

10. Kling, R., McKim, G., King, A.: A bit more to it: scholarly communication forums as socio-technical interaction networks. Journal of the American Society for Information Science and Technology 54(1), 47-67. (2003). doi:10.1002/asi.10154.

11. Best, M., Kumar, R.: Sustainability failures of rural telecenters: challenges from the sustainable access in rural India (SARI) project. Information Technologies and International Development 4(4),31-45. (2008). doi:10.1162/itid.2008.00025.

12. Nhampossa, J.: Re-thinking technology transfer as technology translation: a case study of health information systems in Mozambique. Department of Informatics, University of Oslo, Norway. (2005). https://www.mn.uio.no/ifi/english/research/networks/hisp/research-library/thesis/nhampossa2006.pdf.

13. Orlikowski, W.: Using technology and constituting structures: A practice lens for studying technology in organisations. Organisation Science 11(4), 404-428. (2000). doi: 10.1287/orsc.11.4.404.14600.

14. Walsham, G., Symons, V., Waema, T.: Information systems as social systems: implications for developing countries. Information Technology for Development 3(3), 189-204. (1988). doi: 10.1080/02681102.1988.9627126.

15. Braa, J., Monteiro, E., Sahay, S.: Networks of action: sustainable health information systems across developing countries. MIS Quarterly 28(3), 337-362. (2004). doi:10.2307/2514864.

16. Yahya, H., Braa, K.: Mobilising local networks of implementers to address health information systems sustainability. The Electronic Journal of Information Systems in Developing Countries 48(6), 1-21. (2011). doi: 10.1002/j.1681-4835. 2011.tb00342.x.

17. Li, L., Grimshaw, J., Nielsen, C., Judd, M., Coyte, P., Graham, I.: Evolution of Wenger's concept of community of practice. Implementation Science 4(11). (2009). doi:10.1186/1748-5908-4-11
18. Orlikowski, W., Barley, S.: Technology and institutions: what can research on information technology and research on organisations learn from each other? MIS Quarterly 25(2), 145-165. (2001) doi:10.2307/3250927.

19. Fleiszer, A., Semenic, S., Ritchie, J., Richer, M., Denis, J.: The sustainability of healthcare innovations: a concept analysis. Journal of Advanced Nursing 71(7), 1484–1498. (2015). doi: 10.1111/jan12633.

20. Shediac-Rizkallah, M., Bone, L.: Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Education Research 13(1), 87–108. (1998). doi:10.1093/her/13.1.87.

21. Scheirer, M., Hartling, G., Hagerman, D.: Defining sustainability outcomes of health programs: Illustrations from an on-line survey. Evaluation and Program Planning 31(4), 335-346. (2008). doi: 10.1016/j.evalprogplan.2008.08.004.

22. Greenhalgh, T., Robert, G., Bate, P., Kyriakidou, O., Macfarlane, F., Peacock, R.: How to spread good ideas: a systematic review of the literature on diffusion, dissemination and sustainability of innovations. In health Technical report: National Co-ordinating Centre for NHS Service Delivery and Organisation R & D (NCCSDO). (2004).

23. Racine, D.: Reliable effectiveness: a theory on sustaining and replicating worthwhile innovations. Administration and Policy in Mental Health 33(3), 356–387. (2006). doi: 10.1007/s10488-006-0047-1.

24. Scheirer, M., Dearing, J.: An agenda for research on the sustainability of public health programs. American Journal of Public Health 101(11), 2059–2067. (2011). doi: 10.2105/AJPH.2011.300193.

25. Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., Charns, M.: The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implementation Science 7, 17. (2012). doi: 10.1186/1748-5908-7-17.

26. Scheirer, M.: Is sustainability possible? A review and commentary on empirical studies of program sustainability. American Journal of Evaluation 26(3), 320–347. (2005). doi: 10.1177/1098214005278752.

27. Becker, M., Lazaric, N., Nelson, R., Winter, S.: Applying organisational routines in understanding organisational change. Industrial and Corporate Change 14(5), 775–791. (2005). doi:10.1093/icc/dth071.

28. Gruen, R., Elliott, J., Nolan, M., Lawton, P., Parkhill, A., McLaren, C., Lavis, J.: Sustainability science: an integrated approach for health programme planning. Lancet 372(9649), 1579–1589. (2008). doi:10.1016/S0140-6736(08)61659-1.

29. Slaghuis, S., Starting, M., Bal, R., Nieboer, A.: A framework and a measurement instrument for sustainability of work practices in long-term care. BMC Health Services Research 1(1), 314. (2011). doi:10.1186/1472-6963-11-314.

30. Trottier, L., Denis, J., Villeneuve, M.: Anchoring and Sustaining Change in Healthcare Organisations. CHSRF/CIHR GETOS Chair & CIHR/AnEIS Program, Montreal, QC. (2007).

31. Davies, B., Edwards, N.: Sustaining knowledge use. In: Knowledge Translation in Health Care: Moving from Evidence to Practice. Straus SE, Tetroe J, Graham ID (eds) John Wiley & Sons, Sussex, UK, pp 237-248. (2013).

32. Buchanan, D., Fitzgerald, L., Ketley, D.: The sustainability and spread of organisational change: modernising healthcare. Routledge, London. (2007).

33. Fixsen, D., Naoom, S., Blase, K., Friedman, R., Wallace, F.: Implementation Research: A Synthesis of the Literature. University of South Florida, Tampa, FL. (2005).