# Challenges for Health Indicators in Developing Countries: Misconceptions and Lack of Population Data

Flora Nah Asah, Petter Nielsen, Johan Ivar Sæbø

► To cite this version:

Flora Nah Asah, Petter Nielsen, Johan Ivar Sæbø. Challenges for Health Indicators in Developing Countries: Misconceptions and Lack of Population Data. 14th International Conference on Social Implications of Computers in Developing Countries (ICT4D), May 2017, Yogyakarta, Indonesia. pp.593-604, 10.1007/978-3-319-59111-7_48. hal-01650061

HAL Id: hal-01650061

https://inria.hal.science/hal-01650061

Submitted on 28 Nov 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Challenges for Health Indicators in Developing Countries: Misconceptions and Lack of Population Data

Flora Nah Asah¹, Petter Nielsen¹ & Johan Ivar Sæbø¹

University of Oslo

florana@ulrik.uio.no

**Abstract.** Indicators are foundational for planning, monitoring and evaluating of health services in developing countries. Most health indicators use population-based data, to enable comparison across geographical areas and over time. This paper is based on an interpretative case study on health indicators and how they are calculated and used at health facilities in Cameroon. We found that health managers at different levels of health systems do not share the same understanding of health indicators and we observed a wide-spread absence of population data. We further observed that health managers derive alternative ways of calculating indicators in the absence of population data. This paper contributes by discussing the implications of a lack of a common understanding of health indicators and the absence of population data to calculate health coverage indicators. Though this study was limited to data and program managers at district and regional levels, the findings raise issues that have wider applicability in the implementation of electronic health information system as well as how indicators such as UHC goals are calculated.

**Keywords:** health indicators, population-based data, health management information system, Cameroon

## 1 Introduction

A key goal of a health system is to provide the necessary healthcare services to all those who are in need; thereby improving the health status of the population. International organisations such as WHO and United Nations, in partnership with governments of countries, have developed global initiatives to monitor performances of the health system. One concrete example is the United Nations' Sustainable Development Goals (SDGs) which target major health problems alongside other related issues, constitutes a coordinating framework for these efforts (WHO 2010). Universal Health Coverage (UHC) is one of the goals of this initiative. UHC has been defined as the ability of all people who need health services to receive them without incurring financial hardship (WHO 2010, Kieny & Evans 2013). To monitor countries' health progress and performances towards supporting UHC, a range of health determinants' indicators have been developed. These indicators are measurable and time-bound. Examples of these health indicators include; *children under 5 sleeping under insecticide Treated Nets (ITNs)* and *births delivered in a birth facility*.
Health indicators are powerful tools used at all levels of the health system, for monitoring and evaluating and communicating information about the population's health (Mant 2001; Klazinga et al 2001, Maniz 2003). They could be used to track how well (quality) and how far (quantity) countries' health system are performing (Klazinga, et al 2001, Maniz 2003).

An indicator has two parts; numerator and denominator that go into the formula for calculating them (Klazinga, et al 2001, Maniz 2003). Numerators are the things we count, i.e. infants immunized or new cases of TB. Denominators are the group with which the things we count are compared, i.e. total population or all births in a year. In the health system, indicators could also be used to measure a variety of dimensions concerning the health situation of the population; mortality, morbidity, health status.

There exist different types of indicators which could be used to monitor health progress, namely; health status, health systems, risk factors and health service coverage. Of important to this paper are health service coverage indicators.

These indicators reflect extent to which people in need actually receive the health service they need (Tanahashi 1978, WHO 2009). This group of indicators measure the effectiveness of health programs. They help service providers to understand how effective an intervention is, and whether one target group is reached more effectively than another. They also help to identify underserved areas or regions which need more attention (Boerma, et al 2014). Most UHC indicators of UHC fall in this group.

Indicators in this group use population data as denominator, as it facilitates comparison of health status or health service provision over time and space. Coverage indicators are equally specific and valid for small and large geographic entities, and for any given time interval (WHO 2009). Examples of such indicators are; immunization coverage, HIV testing coverage, and delivery in facility rate. A common challenge is that there is lack of reliable population data to calculate them.

Population data is usually not accessible for smaller areas such as communities or districts (WHO 2009, Linard, et al 2012). Often, available population data is either outdated, not available for current administrative entities, not available for certain target populations (such as women of child-bearing age), or they may be duplicated sources of population data that do not necessarily provide the same figure (Linard et al, 2012, Leegwater, et al 2015, Bharti, et al 2016). The purpose of this paper is two-fold. First, we are empirically exploring the understanding and use of health indicators by health program managers for decision-making. Second, we draw implications from this situation related to local practices and global initiatives such as UHC.

This study is derived from an ongoing implementation of an electronic system as a measure to strengthen Health Management Information system (HMIS) in Cameroon. This system is the main tool used for the management of health information nationwide. In this system, indicators are core element of data analysis, used to measure healthcare services.
## 2 Related Literature

In this section, we present relevant literature on health indicators to establish an understanding of what they are, what makes a good or poor indicator, and how they are used in health management.

Indicator(s) is developed by international organizations, reference groups and interagency groups, countries, academics, advocacy groups and others. From a performance management perspective, Flowers et al. (2005) describe indicator as a measure used to express the behavior of a system. In the public health context, indicators are specific tools for programme management, including the analysis and diagnosis of problems, and for taking correctives actions (Sahay, et al 2009).

Indicators are key statistical measure used to describe a situation, track progress and performance over time and to compare entities doing similar work. They could be used as a guide to decision making and set priorities (Donabedian 1966). They help to inform policy and policy-makers and can be used to improve quality of care and promote accountability (Donabedian 1966, Mainz 2003).

The literature explains that the process of selecting indicators should be systematic and based on facts rather than on feelings (Mant 2001). The process should involve all stakeholders and be based on “who wants the indicator”, “how it is used” and “by whom” (Mant 2001; Klazinga, et al 2001). Heywood et al (2001) explains that the process should employ understanding, discipline, teamwork and negotiation. Therefore, an ideal indicator should: (i) be based on agreed definitions; (ii) measure what it is intended to measure; (iii) give the same results if used by different people in different places; (iv) be simple to calculate using readily available data; (v) fit local needs, capacity and culture and the decisions to be made; (vi) be highly sensitive to changes in the situation concerned; (vii) permit useful comparison; (viii) be evidence-based (Heywood, et la 2001, Mainz 2003, Larson, et al 2004).

Indicators can be classified as: input, process, output, outcome and impact. Input indicators are the resources needed to implement work. Input indicators measure resources, both human and financial, allocated to a particular program. Process indicators measure whether planned activities did take place. For example, in antenatal care, process indicators can be *antenatal 1st visit before 20 weeks rate*, *antenatal client re-test rate*, and *post-natal visit with 2 weeks rate*. Output indicators measure first level results associated to an intervention. They are defined as what we produce. Some examples include; *Infant 1st PCR test around 6 week uptake* or *delivery by caesarean section rate*. Outcome indicators measure what we have achieved and should be linked to concrete goals. Examples include; *delivery at facility for women under 18 years* and *live birth under 250 grams in facility rate*. Finally, impact indicators are the cumulative effect of the overall program (Mainz 2003, Heywood, et al 2001, WHO 2016).

Indicators can be calculated in different ways; ratios, proportions, or rate (Worning, et al 1992, Mainz 2003). Ratios are numbers expressed in relation to another by dividing one number by the other. Here, the numerator is not part of the denominator. Proportion measures a part or amount that is part of a whole. The numerator is part of the denominator. Rate represents the frequency of an event in a specified period. In
calculating a rate indicator, the numerator is the number of occurrences of an event during a period of time. The denominator is the number of person exposed to that event in the time period. These indicators are illustrated in table 1 below.

<table><caption>**Table 1:** Different types of indicators</caption><thead><tr><td>Indicator type</td><td>Description</td><td>Example</td></tr></thead><tbody><tr><td>Ratio</td><td>Numerator is not included in the denominator</td><td>Ratio of male TB deaths to female TB deaths</td></tr><tr><td>Proportion</td><td>Numerator is contained in the denominator</td><td>Proportion of children one year old immunized against measles</td></tr><tr><td>Rate</td><td>Frequency of the event during a specific time in a given population</td><td>Deaths of children less than one year of age per 1000 live births</td></tr></tbody></table>

Population data is required as denominator to calculate these types of indicators. Since these are integral for health management, the lack of population data poses a challenge.

## 3 Research Context

The empirical setting within which the study was conducted is Cameroon. It is a low-income country, situated in the sub-Saharan Africa (SSA) region. It has an estimated population of 20.6 million (Chen, et al 2004). In Cameroon, basic public and social amenities for the vulnerable are either absent or inadequate. Nationally, 29.7 percent of the population does not have access to safe drinking water and 66.9 percent lack adequate sanitation, resulting in regular outbreaks of cholera and other water-related diseases (UNICEF 2015). The burden of healthcare financing is born largely by households through out-of-pocket payments (OOP). The government of Cameroon spends an average of USD 61 as per capita per person on health. Out of this amount, only USD 17 paid by the state, USD 8 comes from international donors, and USD 36 is OOP (Cameroon Economic Update 2013). Over the past two decades, health indicators have remained poor, and in some cases even worsened. Cameroon is struggling with high mortality and morbidity especially in rural communities. Mortality rate for children under 5 is 148 per 1000 live births, ranking Cameroon as 18th amongst 20 countries in the world with the highest mortality rate. Only 13 percent of children under the age of five sleep under insecticide-treated nets, in a country where malaria accounts for more than 40 percent of all deaths in this age group. Maternal mortality rate is alarmingly high, 670 per 100,000 births as compared to 546 per 100,000 live births in SSA. In addition, many women and girls have limited access to, and utilization of, prevention-of-mother-to-child transmission (PMTCT) services, resulting in HIV infection transfer to children (UNICEF 2015).

The healthcare system adheres to the district health approach, organised in three levels: the operational level, corresponding to district health care; hub of all health interventions; the intermediate level which is responsible for technical support, while
the central level deals with the development of health policies. Different programmes operate at all three levels, engaged in the provision of specialised services such as maternal and child health, malaria, HIV/AIDS, TB, and are supported by different donor agencies. The health system suffers from qualitative and quantitative shortage of human resources, and lack of technical and managerial expertise (Cameroon Economic Update 2013).

The health management information system (HMIS) in Cameroon is fragmented and characterised by vertical and fragmented information system and non-standardized data collection methods. DHIS2, the electronic tool used is housed at "Cellular National d'Informations Sanitaires" (CNIS), the department responsible for the management of health information in the country.

## 4 Research Design and Methods

The study is drawn from an interpretative strand (Walsham, 1995). Interpretive research in Information Systems (IS) is useful as it helps researchers understand the problem in the contextual nature (Klein and Myers 1999).

Data was collected by the first author using qualitative methods. It included interviews, group discussion, and a document review. A total of 2 focus discussion and 7 interviews were conducted from January and July of 2016, with Maternal and child health Programme Managers at district and national levels; data managers at district healthcare facilities, Matron in-charge of data management in wards, and Sister-in-charge of health facilities. 22 health and program managers participated in the study, of which 13 were females. Ten had no access to a computer and only 6 had Internet access. The document review included annual reports, strategy documents, and program reports.

Purposive sampling technique was used to select interviewees (Creswell 2007). This technique is used to achieve a homogeneous sample; that is sample of cases who share the same characteristics e.g. background or occupation. In this study, the interviewees shared a similar occupation; i.e. involved in data management at their respective facilities. An interview guide with broad themes around data management was used, focusing on understanding and use of health indicators. Permission to conduct the study was obtained from the Office of Regional Health delegate of each region and signed informed consent was obtained from each interviewee. The principle of data saturation was applied; i.e. interviews were ended when further probing were not adding new information. Permission to audio record interviews was obtained from interviewees at the start of every discussion and interviews were transcribed.

Data analysis was driven by the interpretive process and a descriptive approach with content analysis was used to analyse the data (Elo, et al 2008). The interviews were transcribed verbatim. The interviews were read through several times to obtain a sense of the whole. The text was then divided into condensed meaning units. Open codes were used to group headings into categories to formulate a general description of the research questions. The interviews revealed themes which are interrelated. Dur-
ing this process, the data, themes and topics was discussed among the authors and
based on these discussions the data was revisited.

5 Analysis

This study is based on how nurse managers at district and regional health facilities, program managers at district and regional levels, and district managers’ understand and use health indicators.

A general observation in Cameroon is that population data is either outdated or not
available. In the absence of reliable census data, the National Bureau of Statistics
issues population figures per province and district to the Ministry of Public Health
annually. However, for areas below the districts, it provides percentage per population
group and annual projected growth rate per age group that has to be calculated by
those who want to use population data. Usually, healthcare managers lack the neces-
sary numeracy skills to perform such calculations. Also, these figures are not publicly
available.

Data is collected at health facilities, and reported upward through health districts to the national level. At health facilities, daily registers are used to record activities from various units of the facility; outpatient, antenatal unit, labour and delivery, immunization and in-patient. At the end of each month, data from these registers are collated manually in to the Monthly Reporting Activity <fnref n="1" />(MRA) and also on Program Template forms. Thereafter, these forms are forwarded to the district office where data is captured electronically and their respective databases. MRA is captured into DHIS2 while program-specific data is captured on pre-designed Excel templates, hence forwarded to the regional level. At the regional level, data is also aggregated and synchronized to get the profile of the entire region before it is forwarded to the national level where analyses is done. We observed that the MRA is not comprehensive, as it does not contain all the data elements of health programs, as explained by one district manager:

"Previously, programs manage have their own reporting system but since the new CNIS director was appointed, he is trying to put some order in the system; starting by standardizing data collection tools. The data elements in MRA were selected based on the 100 indicator datasets as stipulated by WHO."

**5.1 Data Quality – Data Review and giving feedback**

It is essential to perform data quality checks. Exploring issues of data quality, we
observed that data reviews and feedbacks are not standardized processes of data man-
agement. They are often done haphazardly and not frequently conducted at most facil-

<footnote n="1">¹ The MRA does not have certain data elements which programs have to report on. Consequently, individual program have created their own data collection tools to collect program specific data</footnote>
ities. District managers explain that due to lack of feedback, data clerks have cultivat-
ed the habit of falsifying data:

"When data clerks submit data and receive no feedback, they believe nobody checks the data submitted. Thus, they have cultivated the habit of cooking data and submitting."

Despite being aware that neither data quality reviews are done nor feedbacks pro-
vided, district managers said they trust the quality of data: "I only type and submit I
do not use data. I cannot say anything about the quality of data. But I trust the data
submitted by the matron in-charge"

## 5.2 Understanding indicators: Nurse Managers at district and regional health facilities

We observed that health managers at operational levels seldom discuss about indica-
tors and do not share the same level of understanding of indicators. In particular, fa-
cility level managers interviewed were not aware that there is a difference between an
indicator and data element. They were neither able to understand what makes up an
indicator nor how an indicator is calculated. They were also not able to differentiate a
data element from an indicator. During the discussions, they would refer to "indica-
tors" but when probing it became apparent that what they meant were "data ele-
ments". When asked to present a list of monthly priority indicators, a matron in-
charge showed a list of monthly summary data element forms, noting: "these are all
our indicators that we report monthly. I write everything here on these forms and
submit to them." A similar situation was observed among district program managers.
When asked to give example of some of the SDGs indicators in the program she man-
ages she noted:

"Last week I went to the hospital for a visit. As I was walking out of the hospital, I caught sight of this beautiful picture with the following caption "breastfeeding is a sustainable development goal as will reduce infant mortality". That was the very first time I heard about sustainable development goals. I was so happy."

Despite giving some examples of SDGs indicators, she was still unable to name
any indicators they are reporting against.

While attending a monthly district meeting, Program managers were presenting
monthly reports on facility visits conducted in their sub-districts. In the course of
presenting these reports, they use phrases such as "low ANC coverage, low immun-
ization coverage" as outcome of their visits. It was not clear from the presentations
how they arrived at the decisions.

5.3 Used of indicators: Nurse Managers at healthcare facilities

We also observed that health managers’ use of indicators was minimal. A nurse man-
ager explained that she uses indicators when preparing her performance-based busi-
ness plan to help allocate targets (excerpt of plan presented on table below). A general
complaint was the difficulties to do the calculations as illustrated in table 2 below. For
example, column 2 depicts how population data are presented to managers at health facilities.

<table><caption>**Table 2:** Level of achievement of the objectives of the last quarter business plan</caption><thead><tr><td>Indicators</td><td>Monthly Target Calculation</td><td>Objective for the previous quarter (A)</td><td>Objective planned for the quarter (B)</td></tr></thead><tbody><tr><td>Out Patient Consultations (new cases): Nurse</td><td>Total pop. of catchment area /12 x 80% x 90%</td><td>4025</td><td>4026 + 660</td></tr><tr><td>Referral received in the hospital</td><td>Pop x 1/12 x 1%</td><td>55</td><td>55 + 18</td></tr><tr><td>Cases of STIs treated</td><td>Total pop. of catchment area x 3% /12 x 80%</td><td>134</td><td>134 + 30</td></tr><tr><td>Children completely vaccinated</td><td>Total pop. of catchment area /12 x 4%</td><td>223</td><td>223 + 70</td></tr><tr><td>Normal Assisted Delivery</td><td>Total pop. of catchment area x 4% /12 x 80% x 70% x 90%</td><td>112</td><td>112 + 50</td></tr></tbody></table>

A matron in-charge explains how she uses indicators:

"At the beginning of the year we have priority activities and also set which are reviewed annually. For example I know in this hospital, our consultation at the end of last year was 40,000. For this year, we should move up to about 45,000. If that is not achieved, we will know that we have not performed well and need to do something so that we can achieve more consultations. Two years ago, our consultations moved from 20,000 to 30,000. As a result of this increase, the director saw that the consultation area has become small. He expanded the waiting area, bought more chairs so that patients should sit down while waiting to be seen by a doctor."

In the explanation above, the manager talked of "indicators" but what she is referring to are "data elements". She added that to calculate indicators, they would use the performance of the previous year or quarter to evaluate their facility's performance against set targets. To set indicator targets, it was observed that there are no clearly defined criteria to use. With reference to the table 2 above, Columns 3 and 4, the nurse manager from another facility shared her experience:

"As we prepare the business plan for PBF, we have working sessions with the PBF team. They will say for example, on the indicator: "Outpatient consultations (new cases): nurse", since we achieved 4025 in the last quarter, what do you think if we shift your objective for next quarter to 660 that is 4025 +660= 4685. What do we do, we just have to accept? They do not take into consideration any other factor. It is our responsibility to ensure that the number cases are achieved at the end of the quarter. If we do not achieve it, we shall not benefit a single franc on this particular indicator."

Another facility manager added:
"To set a target, we merely take the performance for the previous quarter, and add at least 5 percent. Five percentages is just an amount we decide to use. The idea is to encourage them to perform better than the previous quarter. However, there is a challenge because most often those targets sets are not achieved."

## 5.4 Use of Indicators at District level

District managers use indicators to compile quarterly reports as explained by one of them:

"Programs have specific indicator targets. Take for example, the indicator "immunization coverage." I am a statistician so when data arrives my office, after capturing, I do my calculation base on the targets and population projection I have. Hence, I compare the results (figures) with what was submitted for the previous months to know when I have achieved the target or not."

Another Program manager shared her experiences:

"For example, as a HIV/AIDS program manager, this program is a priority program. I have monthly targets that were set and sent to me from Yaoundé [the capital in Cameroon] at the beginning of the year. Consider, for example, the indicator, "Treatment coverage among HIV/AIDS positive pregnant women". Based on the data submitted, at the end of each month, I calculate the indicator and send the report to the national office in Yaoundé."

Furthermore, the district program manager explains:

"If for examples the targets are not met, I will have to wait for instructions from Yaoundé before going out or planning for an intervention because Yaoundé makes all the decisions."

## 5.5 At the national level

A director at the national explained that all decisions are made at the central level, the reason why data analyses are done there; at the national level. He added that indicators are used to evaluate program, but lamented on the quality of data. For example, he mentioned that most of the indicators routinely collected for his program are irrelevant. He also added that district managers are not involved in decision-making as this is the responsibility of those at the central level:

"Districts are not involved in decision-making. Except for the HIV/AIDS program, where program managers at district level have targets. Reasons being it is a priority program, with lots of international funding from PEPFAR. However, about which SDGs my program is reporting against, SDGs have not yet been made published. They are still waiting for the indicators and targets."

Further, he added, indicators are mostly discussed only when preparing annual reports. This is the responsibility of the national program manager for monitoring and evaluation.
# 6 Discussion

The study provides tangible examples of health managers’ understanding and use of indicators in healthcare settings. So how transferable are the findings? We found that these behavior were similar to other countries reported by Silvia, et al 2013; Thaizy, et al 2015.

The world's political agenda for the next 15 years (2016 - 2030) on health systems' strengthening is to reduce poverty through SDGs in general and UHC goals in particular. These are global initiative to monitor countries health performance. UHC are time-bound and measurable indicators to be monitored and reported against, annually by each country. Though internationally developed, most of the indicators are similar to those in a country's operational plans. Thus, decision and interventions made by policy makers and healthcare providers should be based on these indicators.

At the operational level, data to generate these indicators come from various health interventions done at district and communities levels. Therefore, it is important that healthcare professionals understand indicators and also know how they are calculated, as it might help them to monitor and evaluate their performances, and in the event of an outbreak, they intervene immediately.

While managers are should understand health indicators, it is important the government provides reliable population-based data. In Cameroon, population-data is available for the regions, and for administrative units below the district, projected population estimates are provided instead, and it is the responsibility of the district or facility managers to calculate these estimates, who in most cases lack the necessary numeracy skills.

A reliable source of population data is through census, but census is not frequently conducted. Data available is either outdated or does not exist in administrative unit below the district. However, there are other reliable sources of population-data. Citing an example from the developed countries, in Norway, the civil registration and vital statistics (CRVS) systems are reliable sources of population data. The government has introduced a method whereby these systems are maintained through incentives given to bother public and private entities and citizens to engage in proper and timely registration of vital events (Nielsen et al 2015). Thus, the saying goes “population-data is the true denominator for development” (Purcell 2016).

Population-based provide an important piece of mosaic of evidence for decision-makers as well as healthcare providers. It can be used to assess the magnitude of health problems of which population are most vulnerable, such as to track and evaluate the effectiveness of health intervention for UHC.

Furthermore, health care providers dealing with indicators need to understand their meanings and what goes into calculating it (Sahay, et al 2009). In addition, indicators should be standardized as it improves data quality (Maniz 2003, Flower, et al 2005).
# 7 Conclusion

The study found that healthcare managers have difficulties understanding data elements and indicators. In the absence of population data, they use alternative method to calculate indicators. Although managers have priority indicators they report on monthly, coverage indicators are not calculated using population as denominator. This study extends research on how health information is used by healthcare providers based on a case study in Cameroon. Though this study was limited to data and program managers at district and regional levels of the health systems, the findings raise issues that have wider applicability in the implementation of electronic health information system as well as how health indicators such as UHC goals are calculated.

The primary concerns of case study research centers around validity and reliability (Merriam 1985). To address these concerns; for validity, we employed the process of triangulation, i.e. the use of a variety of data sources (interviews, focus group discussions and documents review) as opposed to relying solely upon one source. We have also included verbatim quotations (Johnson 1997) in the analysis section, consulted with senior directors at the ministry of health. To ensure reliability, the following measures were employed; interviews were recorded and transcribed, and during data analysis themes identified were discussed and agreed among the authors, before it was included in the paper (Roberts, et al 2006).

# 8 References

Bharti, N. Djibo, A. Tatem, A. Grenfell, B. Ferrari, M. Measuring populations to improve vaccination coverage. *Scientific Reports*, 6:3454 (2016)

Cameroon Economic Update. Towards greater equity. The World Bank. July 2013 issue 6. Available at http://www-wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2013/08/27/000333037_20130827110226/Rendered/PDF/806710WP0ENGLI0Box0379812B00PUBLIC0.pdf [Accessed April 2016]

Chen, S., Ravallion, M. How have the World's poorest fared since the early 1980s? Washington, D.C., World Bank, Development Research Group, Poverty Team, (2004)

Creswell, J. Research Design: Qualitative, Quantitative, and mixed methods approaches. Thousand Oaks, CA: Sage Publications. (2007)

Donabedian, A. Evaluating the quality of medical care. *Milbank Mem Fund Q* 1966; 44:166-206

Elo, S., Kyngas, H. The Qualitative content analysis process. *JAN*, vol. 62, Issue 1, April 2008, 107 – 115.

Flowers, J., Hall, P., Pencheon D. Mini-symposium – Public health indicators. *Journal of the Royal Institute of Public Health* (2005) 119, 239–245

Heywood, A. Rohde, J. Using information for action: A manual for health workers at facility level. South Africa: EQUITY Project (2001).

Johnson, B. Examining the validity structure of qualitative research Education; Winter 1997; 118, 2; *Research Library* pg. 282 (1997)

Klein, K., Myers, D. A Set of principles for conducting and evaluating interpretive field studies in Information systems. *MIS Quarterly* 23, 67 – 94: (1999).
Kieny, M-P. & Evans, D. Universal Health coverage, *East Mediterr. Health J.* 19:4, 2013. UNICEF: Trends in Maternal Mortality 1990-2015. Estimates by WHO, UNICEF, UNPFA, World Bank Group and the United Nations Population Division. (2015)

Klazinga, N., Stronks, K., Delnoij, D., Verhoeff, A. Indicators without a cause. Reflections on the development and use of indicators in health care from a public health perspective. *Int. Journal of Quality in Health care* 2001; volume 13, Number 6, pp 433-438

Larson, C., Mercer, A. Global health indicators: an overview. CMAJ, Nov.9, 2004; 171 (10)

Leegwater A, Wong W, Avila C. A concise, health service coverage index for monitoring progress towards universal health coverage. *BMC Health Services Research*. 2015;15:230.

Linard, C., Gilbert, M., Snow,R., Noor, A., Tatem, A. Population Distribution, Settlement Patterns and Accessibility across Africa in 2010, *PLoS One*. 2012; 7(2): e31743. Published online 2012 Feb 21

Mainz, J. Developing Clinical indicators. *Int. Journal Qual Health Care* 2003; 15 (suppl.1): i5-i11. (2003)

Mant, J. Process versus outcome indicators in the assessment of quality of healthcare. *Int. J. Qual Health Care*, 2001 Dec; 13(6):475-80. (2001)

Merriam, S. The case study in educational research: a review of selected literature, *J Educ Thought*, 1985, vol. 19 3(pg. 204 -17)

Nielsen, P., Sahay, S., Latifov, M. Civil Registration and Vital Statistics: A Neglected Area in Information Systems Research. Proceedings of the 13th International Conference of Social Implications of Computers in Developing Countries, Negombo, Sri Lanka, May 2015. (2015)

Purcell, W, Hall, I, Bernstein, L, Gift, L, McCray, E, Mermin, J. The Importance of Population Denominators for High-Impact Public Health for Marginalized Populations *JMIR Public Health Surveill* 2016;2(1):e26

Roberts, P., Priest, H., Traynor, M. Reliability and validity in research. *Nursing standard* (through 2013); Jul 12-Jul 18, 2006; 20, 44; Proquest.

Sahay, S., Latifov, M. The data indicator (Mis)Match: Experiences from trying to strengthen this link in the Health Information System in Tajikistan. Proceedings of the 10th International Conference on Social Implications of Computers in Developing countries, Dubai, UAE, May 2009. Dubai School of Government. (2009)

Silvia, G., Ramos, D., Boldrini, A., Silveira, K., Bernardes, A., Rocha, F. Usage of quality indicators in hospital nursing in services in Brazil. *Journal of Hospital Administration*, 2013, Vol. 2, No. 4 (2013)

Tanahashi, T. Health service coverage and its evaluation. *Bulletin of the World Health Organization*, 56 (2): 295-303. (1978)

Thaizy, S., Pedro, P., Andréia, G., Marilane de Oliveira, F. The importance of using quality indicators in nursing care. *Rev. Gaúcha Enferm*. vol.36 no.2 Porto Alegre Apr./June 2015

Walsham, G. Interpretive Case Studies in IS Research: Nature and Method, *European Journal of Information Systems*, Vol 4. No 2, pp.74-81 (1995)

Worning, A., Mainz, J., Klanzinga, N., Gotrik, JK, Johansen, S. Policy on quality development for the medical profession in Danish. *Ugesks Laeger*, 1992; 154: 3523-3533 (1992)

WHO 2009. World Health Statistics. Available at
http://www.who.int/whosis/whostat/EN_WHS09_Table4.pdf [Accessed April 2016] (2009)