# Usability Problems and Obstacles to Addressing Them in Health Information Software Implementations

Magnus Li

► To cite this version:

Magnus Li. Usability Problems and Obstacles to Addressing Them in Health Information Software Implementations. 15th International Conference on Social Implications of Computers in Developing Countries (ICT4D), May 2019, Dar es Salaam, Tanzania. pp.241-252, 10.1007/978-3-030-19115-3_20 hal-02281298

HAL Id: hal-02281298

https://inria.hal.science/hal-02281298

Submitted on 9 Sep 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Usability Problems and Obstacles to Addressing them in Health Information Software Implementations

Magnus Li

University of Oslo, Norway

magl@ifi.uio.no

**Abstract.** Usability is widely recognized as a desired attribute of technology, referring to how usable it is to the intended users. As health information systems in developing countries are increasingly digitalized, interaction with familiar analog technologies is replaced by digital user interfaces for many health workers. An array of literature documents usability problems in such initiatives, arguing for their adverse effects on the users, and the system as a whole. What makes it challenging to achieve usability in this context has, however, not been explored extensively. This empirical paper attempts to bring focus to this gap, by defining the concept of *usability design obstacles*, which refer to aspects that complicate the usability design process. The relevance of the concept is illustrated through three empirical vignettes based on two action research projects in Uganda and India. Three concrete obstacles are outlined. These are 1) constraints introduced by software, 2) constraints introduced by legacy design, and 3) scale and heterogeneity of user groups. It is argued that to address the usability problems commonly outlined and discussed by existing literature, more focus on how to overcome such usability design obstacles is crucial. The obstacles identified in the empirical case of this paper also represent avenues for further research, which are discussed.

**Keywords:** Usability, Design, Generic Software, User Interfaces, Developing Countries.

## 1 Introduction

Health information systems (HIS) play a key role in the strengthening of healthcare by supporting operational and strategic decisions with evidence-based information. Yet, many developing countries are struggling with poor use of such information, bad data quality, and low reporting rates [1]. With increased access to computers, smartphones, and the internet, these countries are taking advantage of digital software packages to support the collection, flow, analysis, and presentation of such health-related information [2]. To the end-users at the lower levels, this means that established paper-based reporting routines are replaced by interactions with computers and smartphones. The digital literacy of these users may at times be limited, while the tasks to be completed may require advanced operations. Further, digital tools often come in addition to, not instead of, paper-based systems and are thus creating additional workload [3]. Hence, how well the software is designed to support the end-users in their work is potentially an important determinant of the success of these implementation initiatives, possibly affecting user acceptance, data quality, and timeliness [4].
Usability problems in health information software and their impact have been discussed in a variety of research literature, both based on experiences from developed countries [e.g., 5, 6], and in developing country contexts [e.g., 4, 7]. What this paper terms *usability design obstacles* have received less attention. That is, contextual characteristics and aspects that make the process of solving such problems difficult. Based on experiences from implementations of a health information software in India and Uganda this paper argues that the achievement of usability in such projects is associated with some interesting obstacles that will require more focus from researchers and practitioners in order to reach the aim of more usable health information software. Considering the documented negative impacts of insufficient usability, it is timely to go beyond identifying specific usability problems and their effects to explore the underlying reasons for why they exist and how they can be solved. Part of this will be to understand the obstacles to solving them. This research paper explores the question: *What are major obstacles to strengthening usability in health information software implementations?*

The aim here is not to provide an exhaustive list, but rather to illustrate the relevance of the concept of usability design obstacles, by highlighting and discussing some seen as particularly challenging and in need of further research. Before addressing this question, a background and brief overview of related literature are provided. This includes defining usability design obstacles, as a phenomenon distinct from usability problems. An empirical case is then presented through three vignettes that highlight the challenges experienced during different software implementation projects in Uganda and India. Finally, these are categorized into three high-level obstacles, which are argued to be appropriate for future research that seeks to contribute to theory and practice on usability in health information software in developing countries.

## 2 Usability in Health Information Software

Digitalization of health information systems routines is driven by the variety of benefits that digital systems can yield, such as storing, aggregating, combining, analyzing, and presenting large amounts of data efficiently [2]. The process of digitalization might also impose a variety of unintended, undesirable consequences, for instance, in the replacement of well-established paper forms with digital interfaces. Ash et al. [8] illustrate how the implementation of a computerized physicians order entry system introduced adverse effects such as more time used for data entry, decreased end-user satisfaction, and unwanted changes in workflows and interpersonal relations. Ghosh et al. [9] document beneficial properties of paper that the end-users value, and which are lost in the process of digitalization of a mobile banking system in Ghana. The authors note that “recovering or preserving these affordances in a digital solution requires additional consideration to design and implementation.” p7. Pal et al. [4] reports from a project in India where data reporting forms used by a group of community health workers (CHWs) were digitized. After implementation, data entry on mobile screens were slower, and the CHWs reported that they preferred the old paper-
based system to the new digital system. The challenges were attributed to the com-
plexities of the existing data entry process, and to the limitations of the digital media.
For example, paper margins were commonly used to explain errors, an affordance that
the computer system did not accommodate. Moreover, none of the participants fully
understood fundamental concepts in the system, such as how to go back to start, re-
name, save or send the report. The authors noted that the implementation of digital
systems "involves altering existing practices and, in many ways, fundamentally
changing the CHW workflow and relationships with communities" [4]. Based on such
challenges, a variety of literature calls for an extended focus on usability in health in-
formation systems software development and implementation [5-7, 10]. Usability is a
term that describes the quality of the interaction that takes place between a user and a
product. A system with high usability should enable the intended users to achieve spe-
cific goals with effectiveness, efficiency, and satisfaction [11]. This means that us-
ability is not a result of the layout and structure of a system alone but how well the
system works with a set of users in a specific context of use. Typical measures of us-
ability include learnability, efficiency, memorability, errors, and satisfaction [12], all
of which may be of relevance to health information software implementations in de-
veloping countries with resource constraints and users with varying digital literacy,
suffering from high workloads and understaffing. For instance, high learnability and
memorability may reduce the need for extensive training and ensuring that users can
easily identify the right actions for the intended task [13]. Efficiency is an obvious ad-
vantage in a situation where the use of information software is only but one of several
tasks during the workday [1]. Errors during data entry is a frequent problem experi-
enced in digital health information reporting regimes, documented to range between 2
and 14 percent for novice users pending on the amount of training received [13]. A
common argument is thus that systems should be designed to minimize the risk of en-
try errors [6]. Furthermore, user satisfaction is both of intrinsic value by improving
the lives of the health worker and may further affect user acceptance which may have
a positive impact on the adoption and correct use of the system [7].

## 2.1 Usability problems and obstacles to solving them

To move beyond outlining and describing specific usability problems, this paper
makes a distinction between problems experienced by users of a system, and the diffi-
culties software implementers face when trying to avoid or solve them. Usability
problems are the standard term used to refer to specific aspects of a system that makes
use difficult to end-users, and thus reduce usability [e.g., 12]. A standard term for as-
pects that might challenge the ability to solve such problems does however not exist.
'Obstacle'<fnref n="1" /> is a good candidate and can be found used for this purpose in Seffah et al.
[14] and Bak et al. [15]. Other literature uses the term obstacle either to describe
something end-users has to overcome due to usability problems [16], or present us-
ability itself as an obstacle to reach a greater goal of a system [7]. This paper suggests
using the term usability design obstacle to refer to contextual aspects that might con-

<footnote n="1">¹ Defined by the Oxford English Dictionary as "something that stands in the way or that ob-
structs progress"</footnote>
strain or obstruct the ability to ensure usability in a system. Table 1 provides a summary with definitions of usability problems, usability as an obstacle, and usability design obstacles

<table><caption>**Table 1.** Usability problems, usability as an obstacle, and usability design obstacles.</caption><thead><tr><td>Term</td><td>Meaning</td><td>Example of use</td></tr></thead><tbody><tr><td>Usability problems</td><td>Specific aspects of a system that makes use difficult to end-users, and thus reduce usability.</td><td>The software has several usability problems affecting user satisfaction and efficiency.</td></tr><tr><td>Usability as an obstacle</td><td>The limited usability of a system represents a constraining factor in reaching a greater goal.</td><td>Insufficient usability in data collection software is an obstacle to ensuring data quality in the health information system.</td></tr><tr><td>Usability design obstacles</td><td>Contextual aspects that might constrain or obstruct the ability to ensure usability in a system.</td><td>Ensuring usability in the software is difficult due to several obstacles.</td></tr></tbody></table>

## 2.2 Usability problems

Usability problems with software in health-care settings have been widely documented in both developed and developing contexts. For instance, Koppel et al. [6] report how problems that can be subscribed to the usability of the UI of an order entry system facilitated frequent fatal medication errors. Atashi et al. [7] present results from a usability evaluation of a nation-wide health information system in Iran, outlining a variety of severe usability problems that may affect data quality, efficiency, and user satisfaction. Based on this, the authors argue that the design should take the end-users mental models and prior knowledge more extensively into account. Khajouei et al. [5] present a review of the literature reporting usability problems in computerized physician medicine ordering systems, where several frequently occurring issues are outlined. The authors argue that usability can be improved by ensuring that systems follow pre-established work-routines. Medhi et al. [13] report usability problems experienced by novice users of mobile interfaces in India, Kenya, South-Africa, and the Philippines. These relate to typical digital design concepts such as hierarchical navigations, vertical scroll-bars, and soft-key mappings. Nabovati et al. [10] identify several severe usability problems in a widely used laboratory and radiology information system, many in which they classify as major or even catastrophic.

## 2.3 Usability design obstacles

As the set of literature presented in the previous section, most research on the usability of health information software in developed and developing countries contributes by identifying and categorizing usability problems. Often, they further argue for de-
signs that are more sensitive to the intended users. Aspects which is here termed us-
ability design obstacles have been outlined and discussed by general usability litera-
ture. For instance, Bak et al. [15] outline six obstacles to usability testing in organiza-
tions based on surveys and a literature review. Major obstacles identified include the
mindset of the developers, resource demands, finding and motivating customers to
participate, conducting the tests, and institutionalizing routines within the organiza-
tion. Edwards et al. [17] briefly discuss how usability problems are handled to affect
the design in such projects. They comment that a common aspect of health informa-
tion systems design projects is the use of commercial software products. Even if the
software is customizable, a substantial part of the design is beyond the power of the
implementers and will require changes in the core application. This aspect may con-
strain the ability to respond to usability problems, thus acting as a usability design ob-
stacle. Martin et al. [18] reports similar experiences, where usability problems that
cannot be solved through technical redesign tend to end up solved through end-user
training.

Usability design obstacles typical to health software implementations in develop-
ing country contexts are however under-researched. An exception is Ghosh et al. [9]
which reflects on how routines and understandings around existing paper forms play
an important role and might hinder the ability to achieve usability in digital interfaces.

In sum, there is a broad consensus that usability problems within health software
are frequent and may result in fatal consequences. To cope with these challenges, sys-
tems should be designed according to the context of use, and the end user's practices
and mental models. Investigations and explicit discussions on usability design obsta-
cles are however limited. After outlining the research methods of this research and
three empirical vignettes, three such prominent obstacles will be described and dis-
cussed.

3 Methods

The research providing the foundation for the empirical case of this paper is a part of the Health Information Systems Program (HISP), an Action Research (AR) project following health information systems strengthening and software implementations in a variety of developing countries during the last two decades [19]. The vignettes presented here are resulting from a two-year AR project in Uganda, and an ongoing AR project in Uttar Pradesh in India.

AR is a methodology that aims at identifying challenges within systems and intro-
ducing, evaluating, and learning from interventions. It is often presented as a five-
stage cyclic process with 1) a diagnosis of the system of focus, 2) action planning di-
rected by theory and existing research, 3) action by introducing an intervention, 4)
evaluating the outcomes, and 5) specify the learnings [20]. In Uganda, several cycles
of AR was completed, and the author of this paper had a central role throughout the
process by being involved directly in all stages of the project. A variety of discus-
sions, interviews, observation, and participatory design activities were conducted and
documented as data. This was analyzed using thematic analysis [21]. Parts of the data
has been re-analyzed for the sake of this paper. In India, the AR project is in an initial
phase, and the data presented here are from the diagnostic stage, where the author has
played a central role in identifying usability problems in a software already being
used across the state. As part of this, interviews and discussions were conducted with
four software implementers, two key actors at the organization of implementation,
and six end-users of the implemented software. Data were analyzed through thematic
analysis and compared with data from the Uganda-project.

4 Case: Implementing DHIS2

This section will present empirical experiences from Uganda and India that illustrate
how some usability problems are difficult to solve due to usability design obstacles. A
commonality between the projects is the replacement of existing paper-based regimes,
and the use of the DHIS2 software as a central component in the new digital solution.
First, a brief description of this software will be provided, before moving on to three
vignettes as examples of implementation.

4.1 DHIS2 and three implementations

DHIS2 is an open source software package that was originally developed in close col-
laboration with end-users to support routine reporting in the South-African health sys-
tem in the nineties [22]. Proven to be a success, the software has over time been im-
plemented in an array of countries, to support collection, transfer, analysis, and pre-
sentation of health-related data in several domains [e.g., see 2]. This implies that the
system is increasingly being taken to settings that deviate from the logic originally in-
scribed in the software. To respond to this, the global software development team are
aiming at making the system generic rather than specific to implementations, to allow
for variety. When the DHIS2 is installed to be used in a new setting, local imple-
menters will start by configuring the system to correspond to the specific needs of the
use-case. Organizational hierarchies, health programs, and data elements are defined,
and dashboards, reports, graphs, maps, and data entry forms are designed to allow for
the collection and presentation of data. To facilitate this, the software comes with a
variety of customizable tools. For instance, data entry forms can be created using a
standard setup, or by using HTML, CSS, and JavaScript to achieve custom layouts.
To support more extensive customization, the DHIS2 software allows for the develop-
ment of third-party applications or ‘apps’. An application programming interface
(API) enables developers to interact with the software’s core resources (e.g., organi-
zational structures, user rights, health programs, data elements, forms, etc.). The final
app can be installed into the software so that it appears as an integrated component to
the users. By being mediators between the implementing organization and the soft-
ware, the implementers of DHIS2 could act as customizers, developers, and designers.
We now turn to three vignettes providing insight into the process of implementing the
DHIS2 software.
**Vignette 1: Health commodity ordering in Uganda**

In 2012, DHIS2 was implemented as a national health commodity ordering system in Uganda, used by health workers at public clinics and hospitals to report on bi-monthly consumption (see [23]). The user interface was customized to suit the domain-specific concepts of logistics management and various contextual particularities. However, due to lack of flexibility for customization of aspects such as how to display forms and deadlines, the interface required the end-users to change their work practices and adopt terminology and logical concepts that were alien to them. Moreover, when presented on a digital surface, the form was hard to navigate which caused frequent entry errors. To respond to these usability problems, a participatory design process was initiated with end-users of the system. However, the level of customization needed was beyond minor configurations of the form and would require the development of a third-party app. Although this provided the implementers with extended design-flexibility, development required more time and competence.

Moreover, the users of the system were widely heterogeneous regarding frequency of use and technical competence. For the less digitally fluent, the paper-based forms were central to their understanding of the reporting process, and when design prototypes deviated too much from the paper forms, these users were often confused about the relation between the interface and the familiar entry process. Thus, the possibility to design radically new layouts that could make it easier to present the extensive form on desktop screens was limited by the layout of the paper form, and tradeoffs had to be found. In sum, the process was characterized by obstacles related to finding a balance between design-flexibility and use of resources, paper, and digital layouts, and the diversity of needs and preferences within the user-group.

**Vignette 2: Routine reporting in Uttar Pradesh, India**

In the state of Uttar Pradesh in India, DHIS2 has been implemented as a standardized system for routine health reporting from thousands of health facilities to the state level. Around 45 different reporting forms have been implemented in the system to be used on desktop screens by health workers at facility and district level. The forms are designed based on the existing paper-based layouts by using the built-in configuration tools in the software. The layout is argued to be beneficial as it provides a familiar design to the users. However, the presentation of these large forms on a digital surface has introduced usability problems, making the data entry process cumbersome and more time-consuming. Also, more technically fluent users indicate the desire for more advanced functionality, and less paper-inspired layouts to make their work more efficient. A process of redesign is now being initiated to strengthen usability. As in Uganda, how to balance the varying needs and how to reflect these in DHIS2 represents obstacles that imply a challenging process forward.

**Vignette 3: Doctors diary reporting system in Uttar Pradesh, India**

Another part of the project in Uttar Pradesh is the implementation of DHIS2 to support around 20.000 doctors to report their daily workload and challenges experienced
regarding infrastructure and equipment. Here, another generic part of the software is
used to set up data storage functionality, and the user interface. In the DHIS2 soft-
ware, this generic component is designed for other types of data collection processes
in mind, and the customization capabilities are limited. This has caused a layout non-
intuitive to the users, making fundamental tasks both time-consuming and prone to
failure. The implementers' ability to ensure usability has thus been constrained by the
flexibility of the software. Plans for improvements involves user involvement and de-
velopment of a third-party app. Although promising regarding design-flexibility, the
process is complicated by obstacles such as the geographic distribution and hetero-
geneity of the users, and the required resources and competence to develop third-party
apps.

5 **Analysis and Discussion**

The three vignettes give a brief insight into obstacles implementers face when at-
tempting to achieve usability during the implementation of DHIS2. While much liter-
ature has pointed out a variety of specific usability problems in health software, these
examples illustrate that responding to such matters is not straightforward. These diffi-
culties can be categorized into three high-level usability design obstacles: 1) con-
straints introduced by the software, 2) constraints introduced by legacy design, and 3)
scale and heterogeneity of user groups.

***Obstacle 1: Constraints introduced by the software***

Much of the software used in health information systems today are commercial, off-
the-shelf, and *generic* [17], and not built specifically to the use-case of implementa-
tion. DHIS2 is one example of these types of software, designed to support variety,
and implemented and made locally relevant through a process of customization. De-
sign affecting usability is as such happening at two levels and with contrasting aims
and rationales. The global designers are culturally and geographically distant from the
final end-users, and aim at providing usability in basic, widely used aspects of the
software, while empowering local implementers with customization capabilities [24].
In line with Edwards et al. [17], the ability for the local designers to mold the soft-
ware according to the user's mental models and established practices, and to address
usability issues that are discovered during and after implementation may be greatly
constrained by these options of customization [25]. When constrained by the lack of
such options, Edwards et al. [17] promote the importance of close relations and fre-
quent communication between implementation projects and the global software devel-
opment team. Optimizations beyond the ability of the local implementers could be
communicated to core developers to be improved in the generic software. However,
as generic software aims at supporting variety, and usability is achieved by being sen-
sitive to specifics within a domain, organization, and group of users, universal usabil-
ity across all use-cases would be impossible. The ability to create third-party apps is
one means of dealing with this issue in DHIS2. Albeit flexible, this introduces new
tensions on resources and competence. The same tension appears in the planned
process in Uttar Pradesh, where the flexibility of DHIS2 to support variety is pushed
to the limit. This represents an obstacle that leaves implementers with decisions re-
lated to the balance between flexibility and costs, which will ultimately affect the
ability to ensure usability for the end-users.

**Obstacle 2: Constraints introduced by legacy design**

In the two first vignettes, well-established paper-based systems have been replaced by
the DHIS2 software. By designing the digital form based on the paper layout, the de-
signers aim to provide a familiar design to the users, making the transition from ana-
log to digital easier. However, as illustrated by Pal et al. [4] & Ghosh et al. [9], a vari-
ety of useful characteristics of the original paper forms are lost in this process, and the
layout may not be appropriate for presentation and navigation on digital surfaces [23].
This situation poses a major obstacle to implementers as the use of familiar concepts
and elements must be balanced with new ways of structuring the data entry when
moving from paper to computer surfaces. In the process, designers must attempt to
understand which elements and properties of the existing design that provide useful
affordances to the users, and how to bring these into or offer substitutes in the digital
surface [9]. For global generic software designers, finding a one-size fits all solution
may be an impossible task as the right concepts and elements to bring into the digital
design may vary significantly between different domains, use-cases, and cultural set-
tings. Thus, the local implementers are faced with tough decisions, opportunities, and
constraints introduced by both the technical and analog artifacts.

**Obstacle 3: Scale and heterogeneity of user groups**

Customization aims at making the generic more specific, and usability is reliant on
designing systems according to the intended users and their context [11]. A common
way to ensure this fit is by involving the relevant users in the development process
[26]. The ‘specific’ in our three vignettes is however strikingly diverse with an enor-
mous scale in the number of use-sites and heterogeneity of end-users. In Uttar
Pradesh, the routine reporting forms are used thousands of health facilities, and the
doctors reporting system is aimed at supporting 20 000 doctors. Similarly, in Uganda,
the commodity ordering system was used by health workers in a high number of
health facilities. Within each project, all sites may have different practices and rou-
tines, and users with different mental models and experiences with technology. De-
signs that could make use easier for a novice rural health worker may be perceived as
inefficient by a statistician at the district health office. The statistician might wish for
advanced functionality based on prior experience with statistical software, but this
might confuse other users and reduce usability. Reaching for ideal solutions and us-
ability for all might be impossible, and questions such as who to design for and how
and who to involve in the design process represents an obstacle. One option would be
to allow end-users to customize the software themselves according to their needs [25,
27]. This might, however, complicate training activities, support, and further updates
to the software. Also, imagining that health workers in rural areas, overloaded with
clinical work will spend the time to customize advanced software packages may be
overly optimistic. It may be especially problematic when working with software that
is to be implemented in a developing country context where differences in computer-
literacy between users can be significant. How to align such diversity within the "spe-
cific" represents a significant obstacle with no easy solutions and is yet to be handled
in implementation projects such as those in Uttar Pradesh.

# 6 Conclusion and Future Research

This empirical paper has attempted to move beyond outlining usability problems in health information software by putting emphasis on usability design obstacles to ensuring or improving usability in such implementation projects. The relevance of the concept of usability design obstacles has been illustrated through three vignettes reporting from two case-studies. Three high-level obstacles have been identified and outlined; 1) constraints introduced by software, 2) constraints introduced by legacy design, and 3) scale and heterogeneity of user groups. All representing aspects that might constrain or obstruct the ability to ensure usability during implementation of health information software. Based on these, three avenues appropriate for further investigation in future research are suggested in the following section.

## 6.1 Moving forward

The experiences presented in the vignettes and the three main usability design obstacles outlined give rise to three questions that call for a further investigation by researchers, and that are relevant to practitioners attempting to ensure usability when implementing or trying to improve health information software.

*How can generic software cater for local usability?*

For global software designers, reducing this obstacle could, for instance, involve reflecting on technical features that can empower local implementers with the means to achieve local usability through customization. For implementers, this is not just a question of flexibility, but also how to utilize the design-space provided with limited resources at hand. Theories, concepts, and models from research on meta-design [27], and software platform architectures [28] discuss such technical and institutional arrangements and could be of relevance to further investigations on this topic.

*How to balance between the skeuomorph and designs adapted for contemporary digital interfaces?*

Rather than merely copying paper layouts to a digital surface or designing brand-new
layouts which provide no familiarity to the user, compromises should be found. Going
into detail on the design elements of existing systems and "shed light on specific af-
fordances of paper that low-income, low-literate users in the developing world may
especially find beneficial" [9], could be a critical step towards digital interfaces that
are more usable to end-users such as health workers. Here, the concept of skeuomorph design, that is, bringing and adapting design-elements from the physical world into digital interfaces might be suitable and find its renaissance.

*How to facilitate participation and balance between needs and requirements in large, distributed, and heterogeneous user groups?*

The “specific” in which achieving usability during development or customization im-
plies, is often rather diverse in health software implementation initiatives in develop-
ing countries. Making health software more usable to health workers will require re-
search beyond the ‘generic global’ versus the ‘local specific’ and discuss how distri-
bution and heterogeneity can be handled and balanced at the level of implementation.

**References**

1. Lippeveld, T. Routine health information systems: the glue of a unified health system. in *Keynote address at the Workshop on Issues and Innovation in Routine Health Information in Developing Countries*, Potomac, March. 2001.

2. Nielsen, P. and J.I. Sæbø, Three strategies for functional architecting: cases from the health systems of developing countries. Information Technology for Development, 2016. 22(1): p. 134-151.

3. Sahay, S., et al., Grand challenges of public health: How can health information systems support facing them? Health policy and technology, 2018. 7(1): p. 81-87.

4. Pal, J., et al. Changing data practices for community health workers: Introducing digital data collection in West Bengal, India. in *Proceedings of the Ninth International Conference on Information and Communication Technologies and Development*. 2017. ACM.

5. Khajouei, R. and M. Jaspers, The impact of CPOE medication systems’ design aspects on usability, workflow and medication orders. Methods of information in medicine, 2010. 49(01): p. 03-19.

6. Koppel, R., et al., Role of computerized physician order entry systems in facilitating medication errors. Jama, 2005. 293(10): p. 1197-1203.

7. Atashi, A., et al., User Interface problems of a nationwide inpatient information system: a heuristic evaluation. Applied clinical informatics, 2016. 7(1): p. 89.

8. Ash, J.S., et al., Categorizing the unintended sociotechnical consequences of computerized provider order entry. International journal of medical informatics, 2007. 76: p. S21-S27.

9. Ghosh, I., et al. The persistence of paper: a case study in microfinance from Ghana. in *Proceedings of the Seventh International Conference on Information and Communication Technologies and Development*. 2015. ACM.

10. Nabovati, E., et al., Usability evaluation of Laboratory and Radiology Information Systems integrated into a hospital information system. Journal of medical systems, 2014. 38(4): p. 35.
11. ISO, ISO 9241-11:2018, in *Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts*. 2018, International Organization for Standardization, Geneva, Switzerland.

12. Nielsen, J., Usability engineering. 1994: Elsevier.

13. Medhi, I., et al., Designing mobile interfaces for novice and low-literacy users. ACM Transactions on Computer-Human Interaction (TOCHI), 2011. **18**(1): p. 2.

14. Seffah, A. and E.J.C.o.t.A. Metzker, The obstacles and myths of usability and software engineering. 2004. **47**(12): p. 71-76.

15. Bak, J.O., et al. Obstacles to usability evaluation in practice: a survey of software development organizations. in *Proceedings of the 5th Nordic conference on Human-computer interaction: building bridges*. 2008. ACM.

16. Barnicle, K. Usability testing with screen reading technology in a Windows environment. in *Proceedings on the 2000 conference on Universal Usability*. 2000. ACM.

17. Edwards, P.J., et al., Evaluating usability of a commercial electronic health record: A case study. International Journal of human-Computer studies, 2008. **66**(10): p. 718-728.

18. Martin, D., et al., Implementing an HIS project: everyday features and practicalities of NHS project work. Health Informatics Journal, 2004. **10**(4): p. 303-313.

19. Braa, J., et al., Networks of action: sustainable health information systems across developing countries. MIS quarterly, 2004: p. 337-362.

20. Baskerville, R.L. and A.T. Wood-Harper, A critical perspective on action research as a method for information systems research, in *Enacting Research Methods in Information Systems: Volume 2*. 2016, Springer. p. 169-190.

21. Braun, V. and V. Clarke, Using thematic analysis in psychology. Qualitative research in psychology, 2006. **3**(2): p. 77-101.

22. Braa, J. and S. Sahay, Participatory Design within the HISP network. Routledge International Handbook of Participatory Design, 2012: p. 235.

23. Li, M., Utilizing the Space for User Participation: Experiences from a Public Health Project in Uganda, in *Information Systems Research Seminar in Scandinavia*. 2017.

24. Dittrich, Y., Software engineering beyond the project-Sustaining software ecosystems. Information and Software Technology, 2014. **56**(11): p. 1436-1456.

25. Kimaro, H.C. and O.H. Titlestad, Challenges of user participation in the design of a computer based system: The possibility of participatory customisation in low income countries. Journal of Health Informatics in Developing Countries, 2008. **2**(1).

26. Kujala, S., User involvement: a review of the benefits and challenges. *Behaviour & information technology*, 2003. **22**(1): p. 1-16.

27. Fischer, G., Rethinking software design in participation cultures. Automated Software Engineering, 2008. **15**(3): p. 365-377.

28. Roland, L.K., et al., P for Platform: Architectures of large-scale participatory design. *Scandinavian Journal of Information Systems*, 2017. **29**(2).