Human health has only ever improved because of advances in technology. From the development of modern sanitation to the advent of penicillin, anesthesia, vaccines and magnetic resonance imaging, science, research and technology have always been key drivers of better health.
It's no different today. Advances in technology are continuing to push back the boundaries of disease. Digital technologies enable us to test for diabetes, HIV and malaria on the spot, instead of sending samples off to a laboratory. 3-D printing is revolutionizing the manufacture of medical devices, orthotics and prosthetics. Telemedicine, remote care and mobile health are helping us transform health by delivering care in people's homes and strengthening care in health facilities. Artificial intelligence is being used to give paraplegic patients improved mobility, to manage road traffic and to develop new medicines. Machine learning is helping us to predict outbreaks and optimize health services.
Propelled by the global ubiquity of mobile phones, digital technologies have also changed the way we manage our own health. Today we have more health information -and misinformation -at our fingertips than any generation in history. Before we ever sit down in a doctor's office, most of us have Googled our symptoms and diagnosed ourselves -perhaps inaccurately. Similarly, digital technologies are being used to improve the training and performance of health workers, and to address a diversity of persistent weaknesses in health systems.
Harnessing the power of digital technologies is essential for achieving the Sustainable Development Goals, including universal health coverage and the other "triple billion" targets in WHO's 13th General Programme of Work. Such technologies are no longer a luxury; they are a necessity.
A key challenge is to ensure that all people enjoy the benefits of digital technologies for everyone. We must make sure that innovation and technology helps to reduce the inequities in our world, instead of becoming another reason people are left behind. Countries must be guided by evidence to establish sustainable harmonized digital systems, not seduced by every new gadget.
That's what this guideline is all about.
At the Seventy-First World Health Assembly, WHO's Member States asked us to develop a global strategy on digital health. This first WHO guideline establishes recommendations on digital interventions for health system strengthening and synthesizes the evidence for the most important and effective digital technologies.
The nature of digital technologies is that they are evolving rapidly; so will this guideline. As new technologies emerge, new evidence will be used to refine and expand on these recommendations. WHO is significantly enhancing its work in digital health to ensure we provide our Member States with the most up-to-date evidence and advice to enable countries to make the smartest investments and achieve the biggest gains in health. Ultimately, digital technologies are not ends in themselves; they are vital tools to promote health, keep the world safe, and serve the vulnerable.

# Dr Tedros Adhanom Ghebreyesus
Director-General, World Health Organization WHO extends its gratitude to the following members of the external review group for their peer review of this guideline: Patricia Garcia, Teng Liaw, Steve Ollis, Xenophon Santas and Maxine Whittaker.
WHO acknowledges the following observers, who represented various organizations: David Heard, Carl Leitner, Ingvil Von Mehren Saeterdal, Merrick Schaefer, Adele Waugaman and William Weiss. Special thanks to Susan Norris and the broader WHO Guidelines Review Committee for their guidance in this process.

# Executive summary
Digital health, or the use of digital technologies for health, has become a salient field of practice for employing routine and innovative forms of information and communications technology (ICT) to address health needs. The term digital health is rooted in eHealth, which is defined as "the use of information and communications technology in support of health and health-related fields". Mobile health (mHealth) is a subset of eHealth and is defined as "the use of mobile wireless technologies for health". More recently, the term digital health was introduced as "a broad umbrella term encompassing eHealth (which includes mHealth), as well as emerging areas, such as the use of advanced computing sciences in 'big data' genomics and artificial intelligence".

# The World Health Assembly Resolution on Digital Health unanimously approved by WHO Member
States in May 2018 demonstrated a collective recognition of the value of digital technologies to contribute to advancing universal health coverage (UHC) and other health aims of the Sustainable Development Goals (SDGs). This resolution urged ministries of health "to assess their use of digital technologies for health  and to prioritize, as appropriate, the development, evaluation, implementation, scale-up and greater use of digital technologies..." Furthermore, it tasked WHO with providing normative guidance in digital health, including through the promotion of evidencebased digital health interventions. Amid the heightened interest, digital health has also been characterized by implementations rolled out in the absence of a careful examination of the evidence base on benefits and harms. The enthusiasm for digital health has also driven a proliferation of short-lived implementations and an overwhelming diversity of digital tools, with a limited understanding of their impact on health systems and people's well-being. This concern was highlighted most notably in the consensus statement of the WHO Bellagio eHealth Evaluation Group, which opened by stating: "To improve health and reduce health inequalities, rigorous evaluation of eHealth is necessary to generate evidence and promote the appropriate integration and use of technologies." While recognizing the innovative role that digital technologies can play in strengthening the health system, there is an equally important need to evaluate their contributing effects and ensure that such investments do not inappropriately divert resources from alternative, non-digital approaches.

# Role of digital health in health system strengthening and Universal Health Coverage
The goal of UHC is to ensure the quality, accessibility and affordability of health services. However, shortfalls remain in ensuring access to all who need health services and in ensuring that they are delivered with the intended quality without causing financial hardship to the people accessing them. The Tanahashi framework published by WHO in 1978 provides a time-tested model for understanding health system performance gaps and how they prevent the intended coverage, quality and affordability of health services. This cascading model illustrates how health systems lose performance because of challenges at successive levels, each dependent on the previous level. Health system challenges -such as geographical inaccessibility, low demand for services, delayed provision of care, low adherence to clinical protocols and costs to individuals/patients -contribute to accumulated losses in health system performance. These shortfalls limit the ability to close the gaps in coverage, quality and affordability, and undermine the potential to achieve UHC.

# Figure 1 Layers of UHC achievement affected by health system performance
This adapted Tanahashi model illustrates that each health system performance layer builds on the components below it but also falls short (dotted lines) of the optimal, desired level (Figure 1). Digital health interventions could contribute to efforts to address challenges that limit achievement of that health system goal.
Source: adapted from Tanahashi, 1978. Digital technologies provide concrete opportunities to tackle health system challenges, and thereby offer the potential to enhance the coverage and quality of health practices and services. Digital health interventions may be used, for example, to facilitate targeted communications to individuals in order to generate demand and broaden contact coverage. Digital health interventions may also be targeted to health workers to give them more immediate access to clinical protocols through, for example, decision-support mechanisms or telemedicine consultations with other health workers. The range of ways digital technologies can be used to support the needs of health systems is wide, and these technologies continue to evolve due to the inherently dynamic nature of the field. A starting point for categorizing the different ways that digital technologies are being used to overcome defined health system challenges is provided by WHO's Classification of digital health interventions v1.0.
A digital health intervention is defined here as a discrete functionality of digital technology that is applied to achieve health objectives and is implemented within digital health applications and ICT systems, including communication channels such as text messages.

# Objectives of the guideline
The key aim of this guideline is to present recommendations based on a critical evaluation of the evidence on emerging digital health interventions that are contributing to health system improvements, based on an assessment of the benefits, harms, acceptability, feasibility, resource use and equity considerations. For the purposes of this version of the guideline, the recommendations examine the extent to which digital health interventions, primarily available via mobile devices, are able to address health system challenges along the pathway to UHC. By reviewing the evidence of different digital interventions against comparative options, as well as assessing the risks, this guideline aims to equip health policy-makers and other stakeholders with recommendations and implementation considerations for making informed investments into digital health interventions. This guideline urges readers to recognize that digital health interventions are not a substitute for functioning health systems, and that there are significant limitations to what digital health is able to address. Digital health interventions should complement and enhance health system functions through mechanisms such as accelerated exchange of information, but will not replace the fundamental components needed by health systems such as the health workforce, financing, leadership and governance, and access to essential medicines. An understanding of which health system challenges can realistically be addressed by digital technologies, along with an assessment of the ecosystem's ability to absorb such digital interventions, is thus needed to inform investments in digital health. Additionally, the adoption of the recommendations in this guideline should not exclude or jeopardize the provision of quality non-digital services in places where there is no access to the digital technologies or they are not acceptable or affordable for target communities.
The recommendations in this guideline represent a subset of prioritized digital health interventions accessible at a minimum via mobile devices, and this guideline will gradually include a broader set of emerging digital health interventions over subsequent versions. This includes recommendations on the following digital health interventions, accessible at a minimum via mobile devices

# Target audience
The primary target audiences for this guideline are decision-makers in ministries of health, public health practitioners and other stakeholders who will benefit from an understanding of which digital health interventions have an evidence base to address health system needs. This guideline may also prove beneficial to organizations that invest resources into digital health as implementation and development partners. This document aims to strengthen evidence-based decision-making on digital approaches by governments and partner institutions, encouraging the mainstreaming and institutionalization of effective digital interventions.

# Foundational Layer: ICT and Enabling Environment

# LEADERSHIP & GOVERNANCE

# STRATEGY & INVESTMENT

# SERVICES & APPLICATIONS LEGISLATION, POLICY& COMPLIANCE

# STANDARDS & INTEROPERABILITY

# Health Content

# Information that is aligned with recommended health practices or validated health content

# Digital Health Interventions

# A discrete function of digital technology to achieve health sector objectives

# Digital Applications

# ICT systems and communication channels that facilitate delivery of the digital interventions and health content

# Implementation context
Digital health has the potential to help address problems such as distance and access, but still shares many of the underlying challenges faced by health system interventions in general, including poor management, insufficient training, infrastructural limitations, and poor access to equipment and supplies. These considerations need to be addressed in addition to the specific implementation requirements introduced by digital health.
Digital health interventions are applied within a country context and a health system, and their implementation is made possible by a number of factors including: (i) the health domain area and associated content; (ii) the digital intervention or functionality provided; (iii) the software and communication channels for delivering the digital health intervention; and mediated by (iv) a foundational layer of the ICT and the enabling environment (see Figure 2). Furthermore, these components need to be made appropriate to the local context and ensure effective implementation through reflection on the behaviour and organizational changes that would also be required. Lastly, digital health interventions are intended to fit into an overall digital health architecture. While the unit of analysis for this guideline focuses on the value of specific digital interventions, there is an equally important need to support a cohesive approach to implementation, in which different digital interventions can leverage one another, as opposed to operating as isolated initiatives.

# Figure 2 Components contributing to digital health implementations
As the context may drive the eventual impact of the digital health interventions, the broader health system and enabling environment become especially critical. There is considerable value in assessing the ecosystem in a given context or country, in reviewing health system needs and tempering expectations based on the ICT and enabling environment available within a setting. In the absence of a robust enabling environment, there is the risk of a proliferation of unconnected systems and a severe impact on the effectiveness and sustainability of the health intervention.
The development of this guideline followed the methods described in the second edition of the WHO handbook for guideline development. This institution-wide process at WHO entailed the identification of critical questions and outcomes, retrieval of the evidence, assessment and synthesis of that evidence, the formulation of recommendations, and planning for the implementation, dissemination, impact evaluation and updating of the guideline.
The guideline development process also included two rounds of online surveys and three in-person consultations. These consultations included (i) an advisory meeting in February 2016 to establish the goal of the guideline in light of other WHO resources and to determine underlying framework;
(ii) a scoping meeting in September 2016 to prioritize and draft the critical questions and outcomes; and (iii) a final meeting in June 2018 to review the synthesized evidence and formulate recommendations. Online surveys were used before and after the September scoping meeting to inform the refinement and prioritization of the questions.

# Scope of interventions and outcomes
The scoping process resulted in priority questions across the following digital health interventions prioritized for evidence review within the guideline (included in Annex 2). The definitions of the interventions included in this guideline are provided in Table 1.

# Table 1 Definitions of included digital health interventions

# Digital health intervention Definition Synonyms and other descriptors
Birth notification Digital approaches to support the notification of births, to trigger the subsequent steps of birth registration and certification, and to compile vital statistics  Birth event alerts  Enabling health workers and community to transmit alerts/ notifications when a birth has occurred

# Death notification
Digital approaches to support the notification of deaths, to trigger the subsequent steps of death registration and certification, and to compile vital statistics, including cause-of-death information  Death surveillance  Death event alert  Enabling health workers and communities to transmit alerts/ notifications when a death has occurred

# Stock notification and commodity management
Digital approaches for monitoring and reporting stock levels, and consumption and distribution of medical commodities. This can include the use of communication systems (e.g. SMS) and data dashboards to manage and report on supply levels of medical commodities  Stock-out prevention and monitoring  Alerts and notifications of stock levels  Restocking coordination  Logistics management and coordination

# Client-toprovider telemedicine
Provision of health services at a distance; delivery of health services where clients/patients and health workers are separated by distance  Consultations between remote client/ patient and health worker  Clients/patients transmit medical data (e.g. images, notes and videos) to health worker

# Providerto-provider telemedicine
Provision of health-services at a distance; delivery of health services where two or more health workers are separated by distance  Consultations for case management between health workers  Consulting with other health workers, particularly specialists, for patient case management and second opinion Transmission of customized health information for different audience segments (often based on health status or demographic categories). Targeted client communication may include: i. transmission of health-event alerts to a specified population group; ii. transmission of health information based on health status or demographics; iii. alerts and reminders to clients; iv. transmission of diagnostic results (or of the availability of results).
 Notifications and reminders for appointments, medication adherence, or follow-up services  Health education, behaviour change communication, health promotion communication based on a known client's health status or clinical history  Alerts for preventive services and wellness  Notification of health events to specific populations based on demographic characteristics

# Health worker decision support
Digitized job aids that combine an individual's health information with the health worker's knowledge and clinical protocols to assist health workers in making diagnosis and treatment decisions  Clinical decision support systems (CDSS)  Job aid and assessment tools to support service delivery, may or may not be linked to a digital health record  Algorithms to support service delivery according to care plans and protocol Digital tracking of patients' / clients' health status and services within a health record (digital tracking)
Digitized record used by health workers to capture and store health information on clients/patients in order to followup on their health status and services received. This may include digital service records, digital forms of paperbased registers for longitudinal health programmes and case management logs within specific target populations, including migrant populations.
 Digital versions of paper-based registers for specific health domains  Digitized registers for longitudinal health programmes, including tracking of migrant populations' benefits and health status  Case management logs within specific target populations, including migrant population The interventions included in this guideline are those prioritized through the process described above from the wider range of digital interventions available. Figure 3 depicts which interventions were reviewed in this guideline, as well as interventions that were excluded at the scoping stage.
Report counterfeit or substandard drugs by clients

# Notification of public health events from point of diagnosis

# Health System Managers

# Scoping considerations regarding health domains and delivery channels
Considering the diversity of the uses of ICT in health, the guideline process established that it was also necessary to define the scope of the prioritized questions in relation to (i) health domains; (ii) types of digital device (i.e. mobile devices); and (iii) delivery channels for the interventions (e.g. SMS text messaging, multimedia applications, voice calls, interactive voice response).

# Health domains
During the scoping consultations described above, the domains to be covered by the guideline were determined, and they are presented in Table 2.

# Table 2 Health domains covered by the guideline

# Digital health intervention
Health domains included in systematic review

# Birth notification
All -no restrictions

# Death notification
All -no restrictions

# Stock notification and commodity management
All -no restrictions

# Client-to-provider telemedicine

# All -no restrictions
Mobile devices are now used widely in almost all settings, and this has been the primary driver for research and investment in digital health efforts across low-and middle-income countries.
The mobile nature of these devices also offers unique opportunities for service delivery. Given the current and growing importance of mobile devices for delivering digital health interventions, particularly in low-and middle-income countries, it was decided that this guideline would focus on digital health interventions that were accessible via mobile devices. This decision was also based on the need to define clear parameters for the systematic reviews.

# Presentation of the guideline
For each recommendation, a summary of the evidence is given in Chapter 3 on the positive and negative effects of the intervention, its acceptability and feasibility, the equity, gender and human rights impacts, resource use, and on any other considerations reviewed at the GDG meeting.
The language that was used to interpret the evidence on effects is consistent with the approach recommended by the Cochrane EPOC Group. Where the WHO team identified any existing WHO recommendations relevant to this guideline, these were integrated into the text, and in all instances transcribed exactly as published in the respective source guidelines. Where needed, additional remarks are included to contextualize these recommendations, and citations for the source documents are given for more details.

# Summary of recommendations
The recommended interventions should be accessible via mobile devices at a minimum. However, these interventions can also be delivered through other non-mobile digital devices, including desktops and laptops.

# Contribution to universal health coverage (UHC)
Digital health intervention (accessible at a minimum via mobile devices)

# Accountability coverage
Birth notification Recommendation 1: WHO recommends the use of digital birth notification under these conditions:  in settings where the notifications provide individual-level data to the health system and/or a civil registration and vital statistics (CRVS) system, and  the health system and/or CRVS system has the capacity to respond to the notifications.

# (Recommended only in specific contexts or conditions)
Responses by the health system including the capacity to accept the notifications and trigger appropriate health and social services, such as initiating of postnatal services.
Responses by the CRVS system include the capacity to accept the notifications and to validate the information, in order to trigger the subsequent process of birth registration and certification.

# Contribution to universal health coverage (UHC)
Digital health intervention (accessible at a minimum via mobile devices)

# Accountability coverage

# Death notification
Recommendation 2: WHO recommends the use of digital death notification under these conditions:  in the context of rigorous research, and  in settings where the notifications provide individual-level data to the health system and/or a CRVS system, and  the health system and/or CRVS system has the capacity to respond to the notifications.

# (Recommended only in the context of rigorous research and in specific contexts or conditions)
Responses by the health system include the capacity to accept the notifications and trigger appropriate health and social services.
Responses by the CRVS system include the capacity to accept the notifications and to validate the information, in order to trigger the subsequent process of death registration and certification.

# Availability of commodities and equipment

# Stock notification and commodity management
Recommendation 3: WHO recommends the use of digital stock notification and commodity management in settings where supply chain management systems have the capacity to respond in a timely and appropriate manner to the stock notifications.
(Recommended only in specific contexts or conditions)

# Availability of human resources for health
Client-to-provider telemedicine Recommendation 4: WHO recommends the use of client-toprovider telemedicine to complement, rather than replace, the delivery of health services and in settings where patient safety, privacy, traceability, accountability and security can be monitored.

# (Recommended only in specific contexts or conditions)
In this context, monitoring includes the establishment of standard operating procedures that describe protocols for ensuring patient consent, data protection and storage, and verifying provider licensing and credentials.

# Availability of human resources for health

# Effective coverage
Provider-to-provider telemedicine Recommendation 5: WHO recommends the use of provider-toprovider telemedicine in settings where patient safety, privacy, traceability, accountability and security can be monitored.

# (Recommended only in specific contexts or conditions)
In this context, monitoring includes the establishment of standard operating procedures that describe protocols for ensuring patient consent, data protection and storage, and verifying provider licensing and credentials. health status and services, combined with decision support under these conditions:  in settings where the health system can support the implementation of these intervention components in an integrated manner; and  for tasks that are already defined as within the scope of practice for the health worker.  where the health system can support the implementation of these intervention components in an integrated manner;  for tasks that are already defined as within the scope of practice for the health worker; and  where potential concerns about data privacy and transmitting sensitive content to clients can be addressed. While the recommendations included in this guideline are based on distinct digital interventions, they all contribute to the health systems' needs in different but interlinked ways. For health system managers, the recommendation on digital stock notification aims to drive availability of commodities at the point of services. From the clients' and patients' perspectives, this would include ability to access health information and services more immediately, such as through client to provider telemedicine and targeted client communication. Likewise, health workers need to be accessible and adhere to practices for delivering high-quality care, through interventions such as decision support and mLearning. Figure 4 illustrates the linkages across the different recommendations and the interlinked ways that these digital interventions can cohesively address health system needs. 1. Introduction
Digital health, or the use of digital technologies for health, has become a salient field of practice for employing routine and innovative forms of information and communications technology (ICT) to address health needs. The term digital health is rooted in eHealth, which is defined as "the use of information and communications technology in support of health and health-related fields" (1). Mobile health (mHealth) is a subset of eHealth and is defined as "the use of mobile wireless technologies for public health" (2,3). More recently, the term digital health was introduced as "...a term encompassing eHealth (which includes mHealth), as well as emerging areas, such as the use of advanced computing sciences in 'big data' genomics and artificial intelligence" (3,4). Furthermore, it tasked WHO with providing normative guidance in digital health, including "through the promotion of evidence-based digital health interventions" (4).
Amid all the heightened interest, digital health has also been characterized, however, by implementations being widely rolled out in the absence of careful examination of the evidence base on benefits and harms (7). The enthusiasm for digital health has also driven a proliferation of short-lived implementations and an overwhelming diversity of digital tools, with a limited understanding of their impact on health systems and people's well-being. This concern was highlighted most notably in the consensus statement of the WHO Bellagio eHealth Evaluation Group, which opened by stating: "To improve health and reduce health inequalities, rigorous evaluation of eHealth is necessary to generate evidence and promote the appropriate integration and use of technologies" (8). While recognizing the innovative role that digital technologies can play in strengthening the health system, there is an equally important need to evaluate their contributing effect to ensure that such investments do not inappropriately divert resources from alternative, non-digital approaches.
- 2 Role of digital health in health system strengthening and universal health coverage UHC aims to ensure the quality, accessibility and affordability of health services. However, shortfalls remain in ensuring access to all who need health services and in ensuring that they are delivered with the intended quality without causing financial hardship to the people accessing them (9). The Tanahashi framework published by WHO in 1978 provides a time-tested model of understanding health system performance gaps and how they prevent the intended coverage, quality and affordability of health services to individuals (10). This cascading model illustrates how health systems lose performance because of challenges at successive levels, each dependent on the previous level. Health system challenges -such as geographical inaccessibility, low demand for services, delayed provision of care, low adherence to clinical protocols and costs to individuals/ patients -contribute to incremental losses in health system performance that cumulatively impact on the health of individuals. These shortfalls limit the ability to close the gaps in coverage, quality and affordability, and undermine the potential to achieve UHC (Figure 1.1).

# Figure 1.1 Layers of UHC achievement affected by health system performance
This adapted Tanahashi (10) model illustrates that each health system performance layer builds on the components below it but also falls short (dotted lines) of the optimal, desired level. Digital health interventions could contribute to efforts to address challenges that limit achievement of that health system goal (11).
To deliver effective and affordable coverage of health services to all, this guideline extends the conceptual foundation of the Tanahashi framework, as follows (11).
 Accountability -Accountability coverage represents the proportion of people in the target population (registered a subset of the total population) in the health system (for example, through civil registration and vital statistics mechanisms, population censuses, the issuance of national or health identifiers), which importantly establishes the different population denominators of health care provision.
 Supply comprises the availability of commodities and equipment, of human resources and of health facilities, and facilitates access to appropriate services with qualified health workers in geographically accessible health facilities, where and when patients need them. Even where health services are available, there may be barriers to accessing them for target populations.
 Demand -driving demand and increasing access can ensure that gaps in contact coverage (i.e. the gap between the total availability of services and the actual contact that individuals have with facilities, health workers and services) do not further undermine health system performance. Individuals often need multiple interactions and follow-up with the health system for health interventions to be effective, and continuous coverage defines the extent to which the full course of the interventions is achieved.
 Quality is related to effective coverage and can be undermined by gaps that that result when health interventions are delivered suboptimally, such as when health workers do not abide by treatment protocols.
 Affordability -direct and indirect costs to the patient can have catastrophic financial effects. Efforts made to ensure individuals are protected from impoverishment due to health interventions are reflected in the affordability layer as improved financial coverage.
Digital technologies introduce novel opportunities to address health system challenges, and thereby offer the potential to enhance the coverage and quality of health practices and services (Figure 1.2) (11,12). Digital health interventions may be used, for example, to facilitate targeted communications to individuals through reminders and health promotion messaging in order to stimulate demand for services and broaden access to health information. Digital health interventions may also be targeted to health workers to give them more immediate access to clinical protocols through, for example, decision-support mechanisms or telemedicine consultations with other health workers.

# Figure 1.2 Digital health interventions to overcome health system challenges
A digital health intervention is defined here as a discrete functionality of digital technology that is applied to achieve health objectives (13).

# Target audience
The primary target audience for this guideline is decision-makers in ministries of health and public health practitioners, to aid them to develop a better understanding of which digital health interventions have an evidence base to address health system needs. This guideline may also prove beneficial to organizations that invest resources into digital health systems as implementation and development partners. This document aims to strengthen evidence-based decision-making on digital approaches by governments and partner institutions, encouraging the mainstreaming and institutionalization of effective digital interventions within supportive digital systems.

# Context and the enabling environment
The maturity of the ecosystem, comprising the enabling and ICT environments, has a critical influence on the relevance and impact of the recommended digital health interventions. The enabling environment is defined as the attitudes, actions, policies and practices that support the effective and efficient functioning of organizations and programmes. For digital health, this includes factors such as the leadership, governance mechanisms, regulatory and policy frameworks, strategy and financial investment, workforce capacity, standards and interoperability, and sociocultural considerations -as articulated within the pillars of the WHO/ITU eHealth Strategy Toolkit (18). The ICT environment consists of the infrastructure and the mechanisms for executing the digital health intervention, such as the hardware and digital applications.
There is considerable value in assessing the ecosystem in a given context or country, reviewing health system needs and tempering expectations and plans for adoption of different interventions based on the ICT and enabling environments available within a setting. In the absence of a robust enabling environment, there is the risk of a proliferation of unconnected systems and a severe impact on the effectiveness and sustainability of digital tools. To help assess ecosystem readiness and the maturity of the ecosystem, several resources exist, including the WHO Score assessment tool (21), MEASURE Evaluation's Health Information Systems Interoperability Maturity Toolkit (22), the Partnership for Maternal, Newborn and Child Health's ICT planning workbook (23) and the Global Digital Health Index (24).
As with any introduction of innovations and new approaches, digital health interventions require changes in behaviour and transitions to new practices. One example is moving away from entrenched paper-based systems to digital approaches. Implementations will succeed only if the digital health intervention is taken up by users, adds value, and facilitates the desired change or action. As such, implementers must be aware of the motivations, barriers and resistance to the disruption of the status quo that may affect the fidelity of deployment and understand that this will temper the possible benefit of the digital health interventions.
The adoption of the recommendations in this guideline should not exclude or jeopardize the provision of quality health services in places where there is no access to the digital interventions, or because they are not acceptable or affordable for target communities. Additionally, in contexts where the ecosystem may not be mature enough to accommodate specific digital health interventions, there should be a focus on strengthening the health system and addressing gaps in the enabling environment to facilitate the implementation of these recommendations in the future.

# Linkages to the broader digital health architecture
Digital health interventions are intended to integrate with and fit into an overall digital health architecture. The digital health architecture provides an overview or blueprint to describe how different digital applications (software and ICT systems) and related functionalities would interact with each other within a given context (25). While the unit of analysis for this guideline focuses on the value of specific digital interventions, there is an equally important need to support a cohesive approach to implementation, in which different digital interventions can operate together, rather than as duplicative and isolated implementations. Stakeholders will benefit from a thorough review of guidance found in the following complementary sources.

# Living guidelines approach
This guideline includes recommendations on a list of prioritized digital health interventions accessible via mobile devices, representing a subset of a much larger set of digital interventions. This guideline aims to incorporate a broader set of emerging digital health interventions gradually in subsequent versions. The WHO Classification of digital health interventions v1.0 (13), provides a starting point to tackle the evolving nature of digital health and to identify interventions for future inclusion in updated guidelines. This version applies WHO Guidelines Review Committee procedures (34) to a priority list of emerging digital innovations, while also acknowledging that future guideline versions will need to incorporate the evidence for additional digital health interventions. This approach to updating WHO guidelines is known as "living guidelines".
The living guidelines approach also facilitates the updating of existing recommendations as new evidence becomes available and the inclusion of additional health domains that might not have been reflected in this initial release. For example, the evidence and recommendations for the digital health intervention of targeted client communication (TCC) was restricted to specific health areas and a subsequent version of the guideline will expand on this area to include the use of TCC for noncommunicable diseases. Chapter 6 (Disseminating and updating the guideline) also details the living guidelines approach for updating and broadening the set of digital health interventions falling under a WHO guideline development process.
The development of this guideline followed the methods described in the second edition of the WHO handbook for guideline development (35). This institution-wide process at WHO entailed the identification of critical questions and outcomes, retrieval of the evidence, assessment and synthesis of that evidence, the formulation of recommendations, and planning for the implementation, dissemination, impact evaluation and updating of this guideline.
The guideline development process also included two rounds of online surveys and three in-person consultations. These consultations included (i) an advisory meeting in February 2016 to establish the goal of the guideline in light of other WHO resources and to determine underlying frameworks; (ii) a scoping meeting in September 2016 to prioritize and draft the critical questions and outcomes; and (iii) a final meeting in June 2018 to review the synthesized evidence and formulate recommendations. Online surveys were used before and after the September scoping meeting to inform the refinement and prioritization of the questions.

# Identification of priority questions

# Process for defining the scope of interventions and outcomes
The initial advisory meeting in February 2016 was used to explore the strategic direction of this guideline, including defining the objectives and the framing of digital health interventions. Since there were no preceding WHO guidelines with defined terminologies for specific digital health interventions, this meeting examined frameworks and standardized classifications that could be leveraged for the formulation of the priority questions. These included the WHO Classification of digital health interventions v1.0 (13), which would serve as the source for prioritizing the interventions (see below and Figure 2.1). The health system challenges (see Annex 1) informed the development of outcomes.
Following the advisory meeting, the responsible officer's team at WHO compiled a set of questions using the standard PICO (population, intervention, comparator, outcomes) format. This initial set of questions was reviewed during a virtual consultation in June 2016 with participants from the February advisory meeting, as well as by technical focal points across WHO, to ensure the appropriateness of the outcomes. The draft questions then underwent further revisions during the scoping meeting in September 2016, conducted in person with global technical experts.

# Prioritization of interventions and outcomes
To supplement the scoping meeting, WHO circulated two rounds of virtual surveys across global and regional networks, including the Asia eHealth Information Network (AeHIN) (30), the Global Digital Health Network (31), Health Information For All (36) and the Implementing Best Practices (IBP) Initiative (37). The first survey was conducted in August 2016 to obtain a general sense of priority interventions and outcomes prior to the scoping meeting in September 2016. During the scoping meeting, the panel of technical experts further refined and prioritized the questions.
Following the in-person scoping meeting, WHO distributed a second survey to prioritize the revised questions. This survey asked respondents to rank outcomes and interventions along a nine-point scale based on how critical the questions were for decision-making, where a rating of 1 indicated that the outcome was not important and a rating of 9 indicated that the outcome was critical (6). Over 300 respondents from all WHO regions participated across the two surveys.
Findings from this second survey helped to narrow down the final list of priority questions.

# Scoping of interventions and outcomes
The scoping process focused on the following digital health interventions that were prioritized for evidence review (see Annex 2 for the questions in the PICO format):

# Digital health intervention

# Definition Synonyms and other descriptors

# Birth notification
The capture and onward transmission of minimum essential information on the fact that a birth has occurred, with that transmission of information being sufficient to support eventual registration and certification of the vital event.
Digital approaches to support the notification of births, to trigger the subsequent steps of birth registration and certification, and to compile vital statistics (13,38)
 Birth event alerts  Enabling health workers and community to transmit alerts/ notifications when a birth has occurred

# Death notification
The capture and onward transmission of minimum essential information on the fact that a death has occurred, with that transmission of information being sufficient to support eventual registration and certification of the vital event.
Digital approaches to support the notification of deaths, to trigger the subsequent steps of death registration and certification, and to compile vital statistics, including cause-of-death information (13,38)

# Health worker decision support
Digitized job aids that combine an individual's health information with the health worker's knowledge and clinical protocols to assist health workers in making diagnosis and treatment decisions (13,18)
 Clinical decision support systems (CDSS)  Job aid and assessment tools to support service delivery, may or may not be linked to a digital health record  Algorithms to support service delivery according to care plans and guidelines

# Tracking of patients' /clients' health status and services (digital tracking)
Digitized record used by health workers to capture and store health information on clients/patients in order to followup on their health status and services received (13,18). This may include digital service records, digital forms of paper-based registers for longitudinal health programmes (40), and case management logs within specific target populations, including migrant populations.
 Digital versions of paper-based registers for specific health domains  Digitized registers for longitudinal health programmes including tracking of migrant populations' benefits and health status  Case management logs within specific target populations, including migrant population

# Provision of educational and training content to health workers
The management and provision of education and training content in digital form for health professionals (13,18). In contrast to decision support, mLearning does not need to be used at the point of care. The interventions included in this guideline are those prioritized through the process described above from the wider range of digital interventions available (13) (Figure 2.1). The excluded digital health interventions can be readily identified for subsequent updates to this guideline (see section 6.3).
Digital health interventions excluded during the scoping process for this version of the guideline are:

# Scoping health domains and delivery channels
In addition to delineating the specific digital health intervention, the guideline development process established that it was also necessary to define the scope of the prioritized questions in relation to (i) health domains; (ii) types of digital device (e.g. mobile devices); and (iii) delivery channels for the interventions (e.g. digital applications, SMS text messaging, voice calls, interactive voice response, etc.).

# Health domains
During the scoping consultations described above, the following decisions were made on the domains (Table 2.2) to be covered by the guideline.

#  Digital health interventions targeting health workers, health system managers and health systems more broadly:
The guideline questions regarding these interventions were not restricted to a specific health condition and were aimed to be inclusive of all health domains and services provided at the primary care level. This decision was made because these interventions, such as notification of stock levels, or decision support, were recognized as having functions that cut across multiple health domains and were often implemented across a whole health system. The systematic reviews commissioned for these interventions extracted information on the health domains covered in order to conduct subgroup analyses where appropriate, and to highlight any potential differences across health domains.

#  Digital health interventions primarily targeting clients/individuals: This guideline includes one intervention -targeted client communication (TCC)
-that is typically linked with or directed to health behaviours associated with specific health topics, such as completing treatment for sexually transmitted infections or returning for family planning appointments. Consequently, it was decided that the scope of the guideline question for this specific intervention focusing on clients' use of services needed to specify the range of health topics.
In this first version of the guideline, the population focus for the intervention of TCC was sexual, reproductive, maternal, newborn, child and adolescent health (SRMNCAH) since this was the entry point for initiating the guideline development process. A planned update to this guideline will include TCC for additional health domains, including noncommunicable diseases.

# Table 2.2 Scope of health domains included in systematic reviews on digital health interventions

# Digital health intervention accessible at a minimum via mobile devices

# Health domains included in systematic review

# Birth notification
All -no restrictions

# Death notification
All -no restrictions

# Stock notification and commodity management
All -no restrictions

# Client-to-provider telemedicine
All -no restrictions

# Provider-to-provider telemedicine
All -no restrictions

# Targeted client communication (targeted communication to individuals)
Sexual, reproductive, maternal, newborn, child and adolescent health Targeted client communication for noncommunicable diseases was not included in this version but has been prioritized for the next update of this guideline

# Tracking of patients' /clients' health status and services
All -no restrictions

# Health worker decision support
All -no restrictions

# Provision of educational and training content to health workers (mLearning)
All -no restrictions
Mobile devices are now used widely in almost all settings (40), and this has been the primary driver for research and investment in digital health efforts across low-and middle-income countries. The mobile nature of these devices also offers unique opportunities for service delivery at the point of care. Given the current and growing importance of mobile devices for delivering digital health interventions, particularly in low-and middle-income countries, it was decided that this guideline would primarily focus on digital health interventions that were accessible via mobile devices. This decision was also based on the practical consideration to define clear parameters for the systematic reviews.
The phrase "accessible via mobile devices" was chosen explicitly to indicate that interventions may be used across a variety of digital devices, but that there should at least be a way to engage with the digital intervention through a mobile interface. These devices range from different types of mobile phones and smartphones, to tablets and other point-of-care handheld devices. The searches for the systematic reviews required that interventions have, at a minimum, a mobile component, and could also have additional ways of engaging with the information, including through desktop computers. While this guideline focuses on interventions accessible via mobile devices as the inclusion criteria for primary studies, the evidence retrieval included information on linkages to non-mobile digital systems. For example, the review on stock management included information on the linkages to web-based data dashboards accessed on desktop computers for visualizing the data at district and national levels.
Lastly, although the systematic reviews included accessibility via mobile devices to ensure that these digital interventions are applicable in low resource settings where extensive computerized systems may not be available, it does not preclude the recommendations from being used on non-mobile digital devices, such as desktop computers.

# Delivery channels
The guideline development process did not place any restrictions on the delivery channels for the included digital health interventions -whether the interventions would be delivered via voice messaging, text messaging or interactive voice response, for example.

# Evidence retrieval
Two main types of evidence were considered for this guideline:
 evidence on the effectiveness digital health interventions based on randomized controlled trials (RCTs), non-randomized studies (NRS), controlled before-and-after studies (CBAs) and interrupted time series studies (ITSs); and  evidence on factors affecting the acceptability, feasibility and implementation of digital health interventions based on qualitative studies.
The evidence on resource use and cost-effectiveness was confined to that found in the studies included in the reviews of effectiveness, including RCTs and NRS. No further searches for costeffectiveness evidence were undertaken. Additional information about resource requirements was gathered through an assessment of programme documents and discussions with implementers. This assessment provided detailed information from the health system perspective on the major cost drivers for implementing each intervention, to inform the guideline development group's (GDG) discussions regarding the resources required. The compiled set of evidence was presented in the evidence-to-decision frameworks (see Web Supplement 1).
The Web Supplements are available at www.who.int/reproductivehealth/publications/ digital-interventions-health-system-strengthening/en/).

# Evidence on the effectiveness of digital health interventions
Cochrane systematic reviews were used as the primary source of evidence on the effectiveness of digital health interventions. Using the priority questions agreed on during the scoping process, the WHO steering group commissioned new Cochrane reviews or identified existing or ongoing Cochrane reviews. When ongoing Cochrane reviews were identified, the authors were invited to collaborate with the technical team (see Annex 3 of this guideline document) to ensure that the reviews would be as relevant as possible for the guideline.
The search strategies to identify relevant studies, and the specific criteria for study inclusion and exclusion, are described within the individual systematic reviews (see Web Supplements 2G-2L).
Most of the included reviews were based on the methods recommended by the Cochrane Effective Practice and Organisation of Care (EPOC) (42) and Consumers and Communication (43) groups.

# Evidence on factors affecting the acceptability, feasibility and implementation of digital health interventions
Systematic reviews of qualitative studies were the primary source of evidence on factors affecting the acceptability, feasibility and implementation of digital health interventions. Using the priority questions agreed on during the scoping process, the WHO steering group commissioned one new Cochrane review of qualitative studies and identified two ongoing reviews. When ongoing reviews were identified, the authors were invited to collaborate with the GDG (see Annex 3 to ensure that the reviews were as relevant as possible for the guideline. These three systematic reviews of qualitative studies covered the following topics:
 health workers' perceptions and experiences of digital health interventions in primary care (Web Supplement 2A)  health workers' and students' perceptions and experiences of mLearning (Web Supplement 2B)  clients' perceptions and experiences of targeted client communication (Web Supplement 2C).
In addition, two of the Cochrane reviews of effectiveness commissioned for the WHO guideline development process also included a secondary objective focused on identifying factors influencing the implementation of the interventions in question. For this objective, the reviews included studies of any design that reported quantitative, qualitative or descriptive data. The two systematic reviews identified covered the following topics:
 tracking health commodity inventory and notifying stock levels accessible via mobile devices (Web Supplement 2D)  birth and death notification accessible via mobile devices (Web Supplement 2E).
Descriptions of the search strategies to identify the qualitative studies, the specific criteria for inclusion and exclusion of qualitative studies, and the databases searched are included in each of the individual systematic reviews. Similar information is available in each of the individual systematic reviews that included a secondary objective on identifying the factors influencing the implementation of the interventions in question.
Finally, an overview of systematic reviews was commissioned to explore the factors influencing the acceptability, feasibility and implementation of telemedicine interventions. This overview included reviews that fulfilled the PRISMA Group's definition of a systematic review (44) and that included qualitative studies, surveys or mixed-method studies. The details of the methods used are available in the overview report (Web Supplement 2F).

# Cross-cutting factors affecting the acceptability, feasibility and implementation of digital health interventions
To identify common factors affecting acceptability, feasibility and implementation that cut across the digital health interventions included in this guideline, an overarching analysis of findings was undertaken using the findings from the systematic reviews of qualitative studies, the overview of systematic reviews and the mixed-methods analyses done alongside the reviews of effectiveness.

# Assessment, synthesis and grading of the evidence
Assessment of risk of bias/methodological limitations of primary studies included in the reviews
For the effectiveness studies included in the systematic reviews of the effects of interventions, the risk of bias was assessed using the explicit criteria outlined in the Cochrane handbook for systematic reviews of interventions (45), and the guidance from the Cochrane EPOC group (42). Each included study was assessed and rated by the review authors as being at low, high or unclear risk of bias for each risk-of-bias domain. These assessments provided an overall risk of bias for each included study and each outcome, where appropriate.
For the qualitative studies included in the qualitative evidence syntheses, the methodological limitations were assessed by applying a quality appraisal framework to each study. The adaptation of the Critical Appraisal Skills Programme's quality assessment tool was used for qualitative studies (46).
Two of the Cochrane reviews included a secondary objective focused on identifying factors influencing the implementation of the interventions in question, and included studies of any design that reported quantitative, qualitative or descriptive data. For these additionally included studies, the methodological limitations were assessed using the "ways of evaluating important and relevant data" (WEIRD) tool for the critical appraisal of programme descriptions, implementation descriptions and other mainly descriptive types of evidence (47,48).
For the effectiveness studies, qualitative studies and other studies included in the assessment of implementation factors for two of the reviews, no studies were excluded based on an assessment of the risk of bias or of the methodological limitations, but instead this information was used to assess the certainty of the review findings, as part of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) or Confidence in the Evidence from Reviews of Qualitative Research (GRADE-CERQual) approaches (49-52) (see the last subsection in section 2.4). An adapted version of the 'Enhancing transparency in reporting the synthesis of qualitative research' (ENTREQ) statement was used for the criteria to judge the methodological limitations of the included systematic reviews (53).

# Synthesis of evidence
For systematic reviews of the effects of interventions, meta-analyses were conducted to estimate an overall effect for outcomes if the intervention characteristics and outcome measures were sufficiently similar across the included studies -that is, if the interventions, participants and the underlying question were similar enough for statistical pooling to be feasible. Where interventions and outcomes were not sufficiently similar to allow meta-analysis, results were reported using a structured narrative summary. Subgroup analyses were planned to focus on factors such as study setting, health care setting, provider type and intervention characteristics, but there were generally insufficient data to allow them to be conducted. Summary tables were created for the main comparisons and included the most important outcomes, the findings for each and the assessment of the certainty of this evidence.
For the syntheses of the qualitative evidence, the data were analysed to identify themes. Findings were then compiled for each theme. Details of the analytical approaches used for each synthesis are described in Web Supplement 2. Summary tables were created to include each synthesis finding and an assessment of the confidence in the evidence for it.
For the overview of systematic reviews to explore the factors influencing the acceptability, feasibility and implementation of telemedicine interventions, three authors analysed the data using a thematic approach. The details of this approach are found in Web Supplement 2. Summary tables were developed to include each overview finding and an assessment of the confidence in the evidence for it.
Where possible, evidence from the reviews on effectiveness, qualitative evidence syntheses and systematic review of systematic reviews was used to highlight the impacts of the interventions on gender, equity and human rights.

# Assessment of the certainty of review evidence
The GRADE approach was used to assess the certainty of the evidence on the effectiveness of the interventions for all the outcomes identified in the PICO questions, and a GRADE evidence profile was prepared for each outcome for each review comparison (52). Based on this approach, the certainty of evidence for each outcome was rated as high, moderate, low or very low. Within the GRADE approach, RCTs were considered to provide high-certainty evidence, while NRS and observational studies were considered to provide low-certainty evidence. The evidence for each outcome was then downgraded when justified by the assessments of risk of bias, inconsistency, imprecision, indirectness and publication bias. This grading was undertaken by the review authors in collaboration with the technical team. The final assessment was based on a consensus among the review authors.
The GRADE-CERQual approach was used to assess the confidence that should be placed in each review finding from (i) the qualitative evidence syntheses; (ii) the secondary analysis of factors influencing the implementation of the interventions included in two of the Cochrane reviews; and (iii) the telemedicine systematic review of systematic reviews. In the GRADE-CERQual approach, confidence in the evidence is based on the following four components: the methodological limitations of included studies; the coherence of the review finding; the adequacy of the data contributing to a review finding; and the relevance of the included studies to the review question (47,49). After assessing each of the four components, a judgement was made about the overall confidence in the evidence supporting each review finding. All findings started as high confidence and were then downgraded if there were important concerns about any of the CERQual components. The overall confidence was judged as high, moderate, low or very low. This grading was undertaken by the review teams in collaboration with the technical team. The final assessment was based on consensus among the review authors.

# Roles and responsibilities of contributors
The guideline development process was guided by the WHO steering group, the technical team, the GDG, the external review group, and external partners and observers (see Annex 3 for the list of contributors in these main groups). An advisory group representing global experts also contributed to the guideline development process prior to establishing the formal GDG.

# WHO steering group
The WHO steering group comprised WHO staff members and consultants representing WHO regional offices, and WHO departments, including the following (in alphabetical order): The steering group, whose members are listed in Annex 3, contributed to the scoping of the guideline, drafting of the questions in PICO format, and interpretation of the findings from the systematic reviews. The steering group will also oversee the dissemination of the guideline (Chapter 6).

# Technical team
The technical team, whose members are listed in Annex 3, comprised guideline methodologists from the Norwegian Institute of Public Health, the systematic review team, and systematic reviewers from Cochrane Response, an evidence consultancy unit operated by Cochrane. The technical team provided guidance on formulating the priority guideline questions so as to ensure that these questions could then be addressed by systematic reviews. The technical team also collaborated with the WHO steering group in developing the systematic review protocols; in undertaking and managing the systematic reviews; in appraising the evidence from systematic reviews using the GRADE methodology (for reviews of intervention effectiveness) and the GRADE-CERQual methodology (for qualitative evidence syntheses) (49-52); in populating the evidenceto-decision frameworks; in supporting meeting processes for the GDG; and in preparing the final guideline document.
Additional support for undertaking the systematic reviews was provided by the Cochrane EPOC Group and the Cochrane Consumers and Communication Group, including in relation to scoping the priority guideline questions and ensuring that the systematic reviews followed standard Cochrane methods and processes.

# Guideline development group
The GDG comprised 28 external (non-WHO) international stakeholders with expertise in research and implementation for digital health interventions, including health programme managers, government representatives, researchers and implementers. The members of the group, who are listed in Annex 3 were identified in a way that ensured geographical representation and gender balance. Their short biographies were published at the WHO website for public review and comment prior to the first GDG meeting.
Selected members of the group participated in a scoping meeting held in September 2016 (see the beginning of section 2.1) and provided input into the final version of the priority guideline questions and outcomes that guided the evidence review. The GDG examined and interpreted the evidence and formulated the final evidence-based recommendations at a face-to-face meeting in June 2018. The group also reviewed and approved the final guideline document.

# External review group
An external review group of six additional expert stakeholders (listed in Annex 3) peer-reviewed the final guideline document to identify any factual errors, and commented on the clarity of the language, contextual issues and implications for implementation. It was not within the remit of this group to change any recommendations formulated by the GDG.

# Declarations of interest by external contributors
In accordance with the second edition of the WHO handbook for guideline development (35), all GDG, technical team and external review group members were required to complete and submit a WHO declaration-of-interests form before engaging in the guideline process. The standard WHO form for declaration of interests was completed and signed by each expert and sent electronically to the responsible technical officer. The WHO steering group assessed the declarations and determined whether any identified conflict warranted one of several actions: exclusion from the GDG, exclusion from deliberations and voting in one or more of the topic areas, inclusion in all of evidence review sessions but exclusion from final voting on recommendations or no action required. In addition, all experts were instructed to notify the responsible technical officer of any change in relevant interests during the course of the guideline development process, for a review of conflicts of interest accordingly. See Annex 4 for a summary of the declaration-of-interest statements and how any conflicts were managed. Additionally, the responsible officer's team also posted on the WHO website the names and brief biographies of GDG members.

# Consolidation of evidence
The technical team supervised and finalized the preparation of the evidence profiles and evidence summaries. These were then consolidated into an evidence-to-decision framework for each guideline question. The evidence-to-decision frameworks (see Web Supplement 1) provided explicit and systematic presentations of the evidence for each question on the following criteria.
 Effectiveness -the evidence on the critical outcomes was summarized to answer the questions: "What are the desirable and undesirable effects of the intervention/option?" and "What is the certainty of the evidence on effects?"  Acceptability -this criterion addressed the question: "Is the intervention/option acceptable to clients and health workers?"
 Feasibility -factors such as the resources, infrastructure and training requirements determine the feasibility of implementing an intervention. The question addressed was: "Is it feasible for the relevant stakeholders to implement the intervention/option?"  Resource use -this criterion addressed the questions: "What are the resources associated with the intervention/option?" and "Is the intervention/option cost-effective?"  Gender, equity and rights -this criterion encompassed evidence or considerations on whether or not an intervention would reduce health inequities. The question addressed was: "What is the anticipated impact of the intervention/option on equity?"
For each guideline question, judgements were made on the impact of the intervention under these criteria, to guide the GDG's recommendation decisions.

# Decision-making and formulation of recommendations
The WHO steering group provided the evidence-to-decision frameworks, including evidence summaries, GRADE evidence profiles, and other documents related to each guideline question, to the GDG in advance of the final in-person GDG meeting. The purpose of the final GDG meeting was to reach a majority decision on each recommendation, including its direction and conditions, based on the evidence and implementation experiences presented. During this face-to-face meeting in June 2018, and under the leadership of the GDG co-chairs (Annex 3), GDG members collectively reviewed the frameworks and contributed to the drafting of the recommendations.
The GDG meetings were guided by the following process: (i) presentation of the evidence-todecision frameworks for the specific interventions by the relevant systematic review teams; (ii) discussion followed by indicative voting on the different components of the evidence-todecision frameworks (effectiveness, acceptability, feasibility, resource use, gender, equity and rights); (iii) discussion followed by voting to determine the category of recommendation (see the recommendation categories below); and (iv) a discussion on any conditions. The views of the GDG were gauged based on online voting before moving towards a decision on a recommendation for each guideline question.
Based on the discussions and voting process, the responsible officer's team at WHO drafted the recommendations during the meeting and presented these to the GDG for its remarks on the research priorities and implementation considerations. GDG members were invited to a subsequent webinar in October 2018 for any clarifications needed ahead of reviewing the draft guideline document.
Finally, the technical team had also drafted implementation considerations for each intervention, based on the findings of the evidence syntheses and the gaps identified in the evidence base. The GDG and the WHO steering group added to these implementation considerations during the GDG meeting and subsequent review of this document.

# Recommendation categories
In line with other published WHO guidelines (54)(55)(56), GDG members voted to classify each recommendation into one of the following categories:
 recommended -the intervention or option should be implemented;
 not recommended -the intervention or option should not be implemented;
 recommended only in specific contexts or conditions -the intervention or option is applicable only to the condition, setting or population specified in the recommendation, and should be implemented only in these contexts; or  recommended only in the context of rigorous research -there are important uncertainties about the intervention or option; in such instances, implementation can still be undertaken on a large scale, provided that it takes the form of research that is able to address unanswered questions and uncertainties related to effectiveness of the intervention and its acceptability and feasibility.

# What do we mean by a recommendation "only in the context of rigorous research"?
The recommendation category "Recommended only in the context of rigorous research" is used in this guideline when the evidence reviewed for a guideline question demonstrated important uncertainties or left unanswered questions about the intervention.
Where uncertainties relate to the effectiveness of an intervention, future research should ideally compare people who are exposed to the option with people who are not, and include a baseline assessment. These comparison groups should be as similar as possible to ensure that the effect of an intervention is assessed rather than the effect of other factors. Programmes evaluated without a comparison group or baseline assessment are generally at a higher risk of bias and so may not measure the true effect of an intervention. RCTs are the most robust way to assess the effectiveness of an intervention. Randomization may not be feasible though for some kinds of intervention (for example, interventions that can be implemented only across a whole jurisdiction) -in these cases, other study designs should be considered, such as interrupted time series analyses or controlled before-and-after studies.
Where unanswered questions or uncertainties are linked to the acceptability or feasibility of the intervention, future research should include well-conducted qualitative studies, and quantitative designs such as surveys, to explore these issues.

# Voting process
Voting on the recommendations was conducted electronically while the GDG meeting was in session, such that GDG members were blinded to the reactions of their peers. The GDG cochairs announced the voting results while the recommendation was being discussed. Majority decision was defined as the agreement of two thirds or more of the GDG, provided that those who disagreed did not feel strongly about their position. Strong disagreements would have been recorded in this guideline; no such disagreements occurred in the GDG meeting. The GDG determined any contexts for the recommendations by the same process of majority decision, based on discussions about the balance of evidence on the effects (benefits and harms) of the interventions across different contexts.
The WHO steering group, systematic review team and observers were not eligible to vote. If the issue to be voted on involved primary research or systematic reviews conducted by any of the participants who had declared a conflict of interest, those individuals were allowed to participate in the discussion but were not allowed to vote on the issue in question.

# Document preparation and peer review
Following the final GDG meeting, the responsible technical officer from the WHO steering group prepared a draft of the full guideline document that reflected as accurately as possible the deliberations and decisions of the GDG. Other members of the steering group and the technical team provided comments on the draft document before it was sent electronically to the GDG members for further comments and to the external review group for peer review. The technical team reviewed the feedback provided by the GDG and the external review group and revised the draft guideline as needed. After the GDG meetings and external peer review, further modifications to the document by the steering group and technical team were limited to corrections of factual error and improvements in language to address any lack of clarity. The revised final version was returned electronically to the GDG members for their approval.

# Presentation of the guideline
The recommendations are presented in the executive summary of this guideline. For each recommendation, a summary of the evidence is given in Chapter 3 on the positive and negative effects of the intervention, its acceptability and feasibility, the equity, gender and human rights impacts, resource use, and on any other considerations reviewed at the GDG meeting. The language that was used to interpret the evidence on effects is consistent with the approach recommended by the Cochrane EPOC Group (42). Where the WHO steering group identified any existing WHO recommendations relevant to this guideline, these were integrated into the text, and in all instances transcribed exactly as published in the respective source guidelines. Where needed, additional remarks are included to contextualize these recommendations, and citations of the source documents are given for more details.

# Evidence and recommendations
This guideline provides ten evidence-based recommendations on the digital health interventions that were prioritized during the scoping process (see sections 2.1 and 2.2). These recommendations are made with the expectation that their implementation is grounded in an understanding of the ecosystem readiness and maturity, as outlined in Chapter 4. For each of the digital health interventions reviewed in this guideline, this chapter elaborates on the following components:
 background information on the specific digital health intervention  an overview of the specific evidence  the recommendation along with a justification and remarks  specific implementation considerations.
Overall gaps in the evidence are described in Chapter 5; specific gaps and research questions for each of the interventions is detailed in Annex 5. In addition, Web Supplement 1 contains the evidence-to-decision frameworks and elaborates on the specific findings for each intervention as it relates to its effectiveness, acceptability, feasibility, resource use, and gender, equity and human rights concerns. The Web Supplements cited here are available at www.who.int/reproductivehealth/publications/digital-interventions-health-system-strengthening/en/

# Cross-cutting acceptability and feasibility findings
Most of the digital health interventions in this guideline are targeted at or expected to be used by health workers. The following findings point to factors that influence the acceptability and feasibility of digital interventions used by health workers. These findings are based on qualitative Although the systematic reviews included accessibility via mobile devices to ensure that these digital interventions are applicable in low resource settings where extensive computerized systems may not be available, it does not preclude the recommended interventions from being used on non-mobile digital devices, such as desktop computers.
evidence syntheses and overviews of digital health interventions for health workers in primary care (Web Supplement 2A); mLearning (Web Supplement 2B) stock notification and tracking commodities (Web Supplement 2D), and birth and death notification (Web Supplement 2E).

# Acceptability for health workers
Factors that may increase acceptability Digital health interventions allow health workers to expand their range of tasks as well as take on tasks previously assigned to higher-level workers. This can be experienced as satisfying and fulfilling, both for those to whom tasks are shifted, as well as to those from whom tasks are shifted (moderate confidence, Web Supplement 2A). Health workers working in rural and remote contexts particularly appreciate the efficiency of digital health technologies as these allow them to offer services through the device (moderate confidence, Web Supplement 2A). Health workers are likely to perceive digital health technologies to be more efficient because of the increased speed with which they allow them to work (moderate confidence, Web Supplement 2A). These technologies are also likely to save travelling time for health workers in both urban and rural settings, allowing them to spend more time with their clients 1 in urban areas or to provide services remotely to clients in rural areas (moderate confidence, Web Supplement 2A). Health workers may appreciate the portability of digital health technologies because this allows them to be flexible, to work when convenient, and not have to be office-bound to access information (low confidence, Web Supplement 2A). Health workers, particularly lay health workers in low-and middle-income settings, also perceive digital health technologies as allowing them to better coordinate the delivery of care through connecting them to other people and sectors in the health system and to clients and communities (moderate confidence, Web Supplement 2A).
Some health workers also report that digital health technologies raise their social status and increase the trust and respect they receive in communities. This is in part due to the device itself but is also because they use these devices to access health workers at higher levels of care. Community health workers, feel that the devices increase the respect they receive from health professionals and from the community (moderate confidence, Web Supplements 2A and 2E). Similar findings are seen among health workers in training, although there is also some concern that clients/patients and colleagues might regard their use of mobile devices as unprofessional because of their association with recreation (low confidence, Web Supplement 2B. to current work is not understood and appreciated by supervisors, or when they themselves perceive the intervention as peripheral to their work. While some health workers do not object to the additional work, others expect to be remunerated for it (low confidence, Web Supplements 2A and 2E).

# Factors that may decrease acceptability
Health workers may also be concerned about loss, damage and theft and may complain about having to carry both a personal and a work phone (low confidence, Web Supplements 2A and 2B).
In some settings, health workers use their personal mobile phones and Internet access for work purposes, although this use is not necessarily formalised and health worker expenses are not always covered (low confidence, Web Supplements 2A and 2E). This can include expenses for air time or for charging their phone. Health workers may see these personal costs as a burden. However, they may feel a moral imperative to assist their clients by using their own phones despite the personal costs this incurs (low confidence, Web Supplement 2A).
Health workers' perceptions and experiences of digital health interventions are likely to be shaped by their pre-existing digital literacy. Health workers who manage well have positive views about the use of mobile devices. However, health workers who struggle to use these technologies have negative perceptions about its usefulness, may not understand the information generated by these technologies, and are also anxious about making errors. In some instances, poor digital literacy threatens job security (high confidence, Web Supplement 2A). However, even technologically more competent users are reported as needing support and repeat training in the use of the programmes and devices (low confidence, Web Supplement 2B).

# Feasibility for health worker
Many health workers, particularly in rural and remote areas, experience logistical challenges when using digital health technologies, including poor network connectivity and access to electricity to charge their mobile phones (high confidence, Web Supplements 2A, 2B, 2D, 2E and 2F). In some instances, poor connectivity also results in client dissatisfaction because it creates delays in receiving health services (high confidence, Web Supplement 2A).
Health workers want easy-to-use, reliable equipment and ongoing technical support (high confidence, Web Supplements 2A, 2D and 2F). They also feel that the use of these technologies can be expanded to a wider range of settings, services, and illnesses (high confidence, Web Supplement 2A). However, health workers often report usability issues, and poor integration with other digital systems (high confidence, Web Supplements 2C and 2F). Although the introduction of digital health interventions into existing healthcare systems may be important, this requires many changes and may be difficult to achieve (low confidence, Web Supplement 2F). For instance, institutional support and local champions may be considered important for ensuring integration into existing systems, but staff reorganization and the breakdown of existing partnerships may undermine this support (low confidence, Web Supplement 2F).
Health workers may experience a number of problems with the design of the programmes or of the device itself, including programmes in languages they are not proficient in, inaccurate rendering of the local language font, small screens, devices being ill-suited for note-taking, and SMS character limitations (low confidence, Web Supplement 2A and 2B). Although the involvement of staff and clients in the planning, design and implementation of the digital system is considered important by health workers (moderate confidence, Web Supplements 2A and 2D), this is not always done (moderate confidence, Web Supplement 2F). Health workers may be dissatisfied with digital health when technology changes are too rapidly introduced, or when their expectations of the technologies are not met (low confidence, Web Supplement 2A).
Some stakeholders are also concerned about the confidentiality of medical information and data security (moderate confidence, Web Supplement 2F). Health workers may try to protect clients' confidential information when using digital health devices, in particular when the information concerns stigmatised conditions such as HIV/AIDS (low confidence, Web Supplement 2A). Achieving informed consent for sharing records and images can also be challenging, particularly in settings with low levels of basic literacy or digital literacy (moderate confidence, Web Supplement 2F).
Training is important for staff acceptance and system use (high confidence, Web Supplements 2A, 2B, 2D, 2E and 2F). While some health workers experience difficulties in understanding and using digital health technologies, health workers and trainers feel that training and familiarity with these technologies can help overcome these difficulties. Some health workers feel hampered in learning to use mobile health technologies if it is not also used by their clinical mentors (moderate confidence, Web Supplement 2A). This may be particularly important as health workers requiring technical support may receive this support from higher level staff or from peers (low confidence, Web Supplement 2A). Supportive supervision is also considered important for staff acceptance and system use (moderate confidence, Web Supplement 2D).
Digital systems can make it possible to track and monitor health workers' activities. Health workers may feel that this changes how they work and may make their work more visible. Some health workers may perceive this as positive, but it may leave other health workers with the sense of "big brother watching". Supervisors may feel that this allows them to be more aware of the work of lower level health workers and to address problems (low confidence, Web Supplements 2A and 2D).
Even where challenges tied to the design and usability of digital systems and devices are addressed, these systems may not be able mitigate a number of broader health systems challenges, for example, an underlying lack of medical commodities (low confidence, Web Supplement 2D).

# Acceptability and feasibility for clients/individuals
The following findings point to factors that are likely to influence the acceptability and feasibility of digital health interventions targeted at or expected to be used by clients/patients. These findings are summarized based on overviews and qualitative evidence syntheses related targeted client communication (Web Supplement 2C) and telemedicine (Web Supplement 2F). More detailed descriptions on the acceptability and feasibility findings are available within the sections focused on the specific interventions.
Some individuals describe targeted communication and telemedicine services in positive terms.
For instance, some clients appreciate the fact that someone is taking the time to send them messages as this can make them feel like someone is interested in their situation and invested in their well-being. These clients describe the messages as providing support, guidance and information, and giving a sense of direction, reassurance and motivation (moderate confidence, Web Supplement 2C). Similarly, some clients using telemedicine services see these as offering reassurance and a sense of safety and appreciate the increased access and the consistency and continuity of care that it can offer (low confidence, Web Supplement 2F). Some clients also feel that telemedicine services have increased their independence and self-care (low confidence, Web Supplement 2F).
However, individuals who are dealing with health conditions that are often stigmatised or very personal (e.g. HIV, family planning and abortion care) worry that their confidential health information will be disclosed or their identity traced due to their participation in targeted communication programmes (high confidence, Web Supplement 2C). Some individuals using telemedicine services prefer face-to-face contact (low confidence, Web Supplement 2F).
Additionally, individuals believe there should be little or no charge tied to digital health programmes, such as joining the programme, downloading apps, or charges related to sending and receiving SMS/phone calls (high confidence, Web Supplement 2C).
Targeted communication and telemedicine services can potentially increase access for some groups of individuals. For instance, telemedicine services can give individuals who speak minority languages access to health workers who speak this language (high confidence, Web Supplement 2F); and may save money and reduce the burden of travel for clients with caring or work responsibilities, living far from health care facilities or with few funds (low confidence, Web Supplements 2C and 2F).
However, access to and use of these services can be particularly difficult for some individuals. These include individuals with poor access to network services, electricity (high confidence, Web Supplement 2C) or mobile devices (moderate confidence, Web Supplements 2A and 2C); clients who speak minority languages, have low literacy or digital literacy skills (moderate confidence, Web Supplement 2C) or hearing impairments (high confidence, Web Supplement 2A). Clients with stigmatized health conditions may also be particularly concerned about the privacy of their information (high confidence, Web Supplement 2C).

# Accountability coverage
The proportion of those in the target population registered into the health system

# Accountability coverage: birth and death notification Background
A global scale-up plan for strengthening civil registration and vital statistics (CRVS) systems has been developed by the World Bank and WHO with the goal of achieving "universal civil registration of births, deaths and other vital events, including reporting cause of death, and access to legal proof of registration for all individuals by 2030" (57). A key component of this plan is to prioritize and strengthen the linkages between CRVS systems and health (57)(58)(59). This includes the use of digital information systems to strengthen CRVS systems and expanding the coverage of registration services among underserved populations, such as people residing in rural areas (57)(58)(59)(60).
In these respects, the global proliferation of mobile phones and cellular network connectivity (41) is increasingly being leveraged, especially in resource-limited settings, to drive the development and use of digital civil registration systems (11,12,(60)(61)(62)(63).
Notification is the capture and onward transmission of minimum essential information on the fact of birth or death has occurred, and represents the first step in the process leading to eventual registration and certification of the vital event. Increasing the efficiency of birth and death notification as well as promoting linkages between the health and civil registry sectors (many births are first known in the health sector) can strengthen civil registration processes and the use of health services (61,62). Digital mechanisms to facilitate notifications may enhance these linkages as well as catalysing civil registration. Furthermore, added to their ability to conduct notifications, the increased access to mobile devices among community-based individuals such as vaccination programme workers, community health workers and village elders can potentially expand the coverage of civil registration systems to underserved rural and remote regions (60)(61)(62)(63).
For birth notifications, other information related to the birth may be transmitted via mobile phones in the form of phone calls, inputs to an interactive voice response or unstructured supplementary service data (USSD) system, SMS text messages, messages from mobile devicebased applications (apps) or calls or messages to publicly known short codes or access numbers. The content of the birth notification may vary by country or implementation, but may include the name of the child born, the name and address of the parents, the place and date of birth, and details of birth outcomes.

# Lack of population denominator
Lack of access to information or data Delayed reporting of events

# Illustrative health system challenges
Similarly, for death notifications, information related to the death may be transmitted via mobile phone calls, inputs to an interactive voice response or USSD system, SMS text messages, messages from apps, or calls or messages to publicly known short codes or access numbers. The content of the death notification may vary by country or implementation, but may include the name of the deceased, the name and address of a relative, the place and date of death, and details of the cause of death.
This guideline question reviewed the added value of the notification of birth and death events via mobile devices as an additional channel for supporting the establishment of a CRVS system and strengthening linkages to it.

# Overview of the evidence
The following is a summary of the evidence on birth and death notification accessible via mobile devices. Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.
 Births: There is limited evidence on the effectiveness of using mobile devices for birth notification as the certainty of this evidence was assessed as very low.
 Deaths: No evidence on effectiveness was identified for death notification via mobile devices.
The qualitative evidence suggests the intervention is probably acceptable to health workers and enables them to be more proactive in identifying pregnancies and coordinating emergency services. They report earning more trust and respect from their communities due to the ability to communicate with and coordinate emergency services. Conversely, acceptability for clients of birth notification may be reduced by sociocultural norms, such as the extent to which stillbirths, births to unmarried mothers or maternal deaths are acknowledged in communities. The evidence also points to the potential costs of notification as a barrier and to the need to demonstrate the advantages of birth or death notification to communities.
The qualitative evidence highlights several feasibility issues including, the need for adequate local staffing and for strong underlying health and civil registration system infrastructure, resources and processes. Health workers' competing priorities and lack of adequate incentives may affect the successful adoption of these strategies. Inadequate attention is sometimes given to legal frameworks governing civil registration, and governments may need to modify these frameworks to allow new types of health care cadre and other key informants to notify births and deaths. Strong underlying health and civil registration system infrastructure, resources and processes are necessary to achieve the impact of using mobile devices for birth and death notification.
No evidence on resource use was identified. Resource use considerations are listed within the evidence-to-decision framework in Web Supplement 1.

# Gender, equity and human rights
The qualitative evidence indicates that while birth and death notification via mobile devices can help to reach under-registered populations, there may be inequities in the implementation of this intervention that are related to the availability of supportive infrastructure (network connectivity, for example), literacy in the use of information and communications technologies (ICT), and available funding resources.

# Recommendation and justification/remarks

# Birth notification
(Recommended only in specific contexts or conditions) recommendation 1

# WHO recommends the use of digital birth notification under these conditions:
 in settings where the notifications provide individual-level data to the health system and/or a civil registration and vital statistics (CRVS) system, and  the health system and/or CRVS system has the capacity to respond to the notifications.
Responses by the health system include the capacity to accept the notifications and trigger appropriate health and social services, such as initiating of postnatal services.
Responses by the CRVS system include the capacity to accept the notifications and to validate the information, in order to trigger the subsequent process of birth registration and certification.

# Death notification (Recommended only in the context of rigorous research and in specific contexts or conditions)

# WHO recommends the use of digital death notification under these conditions:
 in the context of rigorous research, and  in settings where the notifications provide individual-level data to the health system and/or a CRVS system, and  the health system and/or CRVS system has the capacity to respond to the notifications.
Responses by the health system include the capacity to accept the notifications and trigger appropriate health and social services.
Responses by the CRVS system include the capacity to accept the notifications and to validate the information, in order to trigger the subsequent process of death registration and certification.
Birth notification  The guideline development group (GDG) acknowledged the limited evidence but emphasized that birth notification represents a vital first step in a care cascade that can ultimately lead to increased and timely access to health services and other social services. The GDG also believed that the use of mobile devices to perform this task was likely to provide a more expedient means of effecting the notification and subsequent health services.
 GDG members noted that while birth notification should not be viewed as a substitute for legal birth registration, it could provide an opportunity to accelerate the registration by linking birth notifications to national civil registration systems. The GDG also recognized that digital notification of births could facilitate providing newborns with legal identity and future access to health and other social services.

# Death notification
 The GDG remarked that a lack of information on deaths, especially deaths outside of facilities, exacerbates data gaps in understanding the rates and causes of mortality.
 The GDG therefore decided, while noting the limited evidence, to recommend death notification via mobile devices in the context of rigorous research and where notifications can be linked to health and/or CRVS systems.
 The GDG noted that while data on deaths and causes of death are very useful for health planning, they expressed concerns about adding the responsibility of CRVS-related functions to already poorly resourced, understaffed and overburdened primary care health systems.
 The GDG also recognized the sociocultural sensitives of communities notifying about deaths through digital devices and recommended that further research be taken to understand these considerations.

# Remarks that apply to both birth and death notification
 It should also be noted that increases in the notification of births and deaths would also require that civil registration services have, in turn, the capacity to manage a higher demand for registration and certification services.
 The ability for the health system and/or CRVS system to respond and act appropriately on the birth and death notification was seen as a critical component for successful implementation. If such linkages are not in place, the notification of birth and death events would not add any value and would incur an additional cost to the system.
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.

# Legislation, policy and compliance
 The implementation of birth and death notifications needs to be in the context of national policies, laws and guidelines. This may require modifications of legal frameworks to include mobile notification in established practice and to enable cadres of informants such as community health workers and community leaders to conduct the notifications if current policies do not already provide for this.
 Consider whether changes to legal frameworks will be needed to allow birth and death notification to occur via mobile device or be carried out by new groups of health workers or other cadres, as mentioned above, and how this would be linked to the issuance of birth/death certificates. For example, consider whether there will need to be any modifications to existing processes to accommodate signatures and approvals currently conducted on paper-based forms. This review and modification should take place in the context of a broader legal review of CRVS-related laws and regulations and would require collaboration among the institutions that cover the health sector, civil registration sector and the local governments.
 Consider the specific data storage, privacy and confidentiality issues. Implementers should understand, for example, the implications and necessary regulations if the database of notified births and deaths is also being held by mobile network operators, and the potential for commercial uses of the data. Additionally, a relevant authority needs to ensure the right to data protection by monitoring and enforcing a set of data protection laws.

# Services and applications
 Consider establishing mechanisms to prevent duplicate notifications. Unique identification can be used to address this (for example, by issuing national identities; possibly identification of the parents). Where national IDs are not available, consider an interim measure of IDs being provided by health facilities, drawing from codes in master facility lists. Implementers may also want to consider local de-duplication processes, such as using routine coordination meetings across health workers to de-duplicate birth/death notifications before they are transmitted to the civil registrar.
Workforce  When developing birth and death notification systems, consider mechanisms to ensure the completeness of the data, and whether demand-generation activities are needed to incentivize reporting by explaining its benefits. Implementers should be aware, however, of any reporting targets placed on health workers, and ensure birth and death data are validated before being released to the civil registration system.
 Consider how best to ensure the quality and timeliness of birth and death data, for instance by checking on low performers identified through digital performance data or spot checks.
Other ways to help improve data quality include standardizing the definitions associated with reporting birth and death events, such as for stillbirths, and making these definitions accessible to those inputting the data.
 Implementers should note that increases in notification would in turn require that the health system and civil registration services were prepared to absorb higher demand for registration. This is a potential bottleneck in the registration and validation process and could deter populations from continuing notifications.
 Consider how to improve accessibility and shorten the connection between the health workers or communities providing the notifications and the CRVS sector undertaking the registration. Consider, for instance, increasing the number and proximity of registration service points, and look at the use of digital systems to speed up the registration process at these points.

# Considerations for equity, gender and human rights
 Explore sociocultural barriers associated with communicating about births/deaths and address the way these dynamics will influence notifications via digital devices.
 Consider linking birth notification to health services that have high coverage, such as immunization services or health facilities that offer very high rates of institutional delivery. It is important, however, to consider whether an increase in notifications can be absorbed by the civil registration system.

# Availability of commodities: stock notification and commodity Background
The availability of health commodities at point of services is critical to strengthening the quality of care and supporting the pillars of universal health coverage (UHC) (64). Health commodities include health products, and health and medical supplies that may be needed for the provision of health services, including medicines, vaccines, medical supplies such as contraceptives dressings, needles and syringes, and laboratory/diagnostic consumables (65,66). Various high-level initiatives, including the UN Commission on Life-Saving Commodities for Women's and Children's Health, have advocated equitable access to life-saving medicines and other health commodities (67,68). Stock-outs of critical medical commodities remain an issue, however, particularly in rural settings, where infrastructural limitations and geographical barriers can obstruct access to commodities at the point of care.
The rapid global expansion of mobile devices has emerged as providing a potential opportunity for mitigating the challenges of commodity distribution and stock-outs. Approaches can include the use of communication systems such as text messaging (SMS) and data dashboards to manage and report on supply levels. Specific examples by which mobile tools may be used to improve supplychain management include to track inventories of health commodities, notify their stock levels, forecast demand for commodities, monitor cold chain-sensitive commodities, and manage the distribution of health commodities (13).
Although broader initiatives to strengthen logistics management information systems are ongoing (69), this guideline question reviewed the added value of extending the systems via mobile devices to address commodity management at primary health care levels.

# Availability of commodities and equipment

# Ensuring availability of commodities and equipment
Insufficient supply of commodities at point of service

# Lack of transparency in commodity transactions
Lack of access to information or data

# Illustrative health system challenges

# Overview of the evidence
The following is a summary of the evidence on stock notification and commodity management accessible via mobile devices. Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.
There is limited available evidence on the effectiveness of and resources required as the certainty of the evidence was assessed as very low.
The qualitative evidence suggests that access to digital data on stock availability at all levels of the health system may be useful by health system managers as it allows them to respond to anticipated stock shortages and ensure ongoing supply of needed health commodities. Staff at the subnational levels may be concerned, however, about the data at their level becoming available simultaneously with those at the national level since this would take away their opportunity to contextualize the data or to explain shortcomings in stock availability.
Barriers to optimal implementation of stock notification and commodity management via mobile devices include an underlying lack of stock at national or district level and a mismatch between national ordering routines and local needs. The qualitative evidence on the feasibility of digital health interventions, more broadly, also highlights challenges including those of network connectivity, access to electricity, usability of the device, sustaining training and support to health workers using the digital tools, and system integration.

# Resource use
No evidence on resource use was identified. Resource use considerations are listed within the evidence-to-decision framework in Web Supplement 1.

# Gender, equity and human rights
The qualitative evidence on gender, equity and human rights concerning digital health interventions suggests health workers based in peripheral facilities and rural communities may find these interventions helpful in overcoming geographical barriers and linking to the broader health system, including when communicating about stock levels. Health workers in these settings may be more likely to experience poor network coverage and access to electricity, though, and may have lower levels of training and literacy in the use of technologies and fewer resources, including limited access to the mobile devices that may be needed.

# Recommendation and justification/remarks stock notification and commodity management
(Recommended only in specific contexts or conditions)

# Recommendation 3
WHO recommends the use of digital stock notification and commodity management in settings where supply chain management systems have the capacity to respond in a timely and appropriate manner to the notifications.
 Despite the limited evidence on effectiveness and the identified feasibility barriers, the guideline development group (GDG) felt that the use of mobile devices was likely to provide a more expedient means of effecting stock notifications and ensuring the subsequent availability of commodities at the point of services. This, in turn, may increase the ability of health services to manage health issues in a timely and appropriate way.
 The GDG also assessed stock notification via mobile devices to be a relatively low-risk intervention with potentially high impact, including the potential to save resources through an improved allocation of commodities and reduced wastage. The GDG further believed that the availability of timely stock data would increase transparency and promote accountability.
 Addressing the identified barriers to implementation as well as ensuring responsiveness to the stock notifications were seen as critical ways to build trust and drive the effective use of the digital intervention. If there are no mechanisms for health managers to respond to the incoming data, or a lack of infrastructure or financial resources to purchase new commodities, the gathering of stock data and issuance of notifications would not add any value and would incur an additional cost to the system.
 Although the condition within this recommendation requires that the health system be responsive to the stock notifications, the GDG also remarked the importance of building the capacity of weaker health systems so that this intervention may be used effectively.

# Linkage with other WHO recommendations
This discussion aligns with recommendation 15 of the WHO guideline on health policy and system support to optimize community health worker programmes, which recommends the use of mobile health technology to support supply chain functions, including adequate reporting, to enhance the availability of health commodities (17).

# Implementation considerations
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.

# Legislation, policy and compliance
 Ensure there is no harm or reprisal to health workers for reporting stock-outs or wastage; instead, the emphasis should be on explaining the benefits of reporting stock-outs so that they can be addressed. To motivate continued reporting, ensure that some action is possible when stock-outs are reported.

# Standards and interoperability
 Prioritize integrating notifications with existing data reporting systems, including national or subnational information management systems where available, such as supply chain, logistics and warehouse management information systems. Consider integrating the stock notification system with a data dashboard that displays the notification, receipt of commodity at the station and action taken among other data for ensuring transparency.
 Consider the need for training at all levels of the health care system, including the training of health workers to send stock reports, of support staff such as cold-chain technicians to manage stock and of facility workers to assess stock levels. Training should be reinforced by the basic processes of inventory management and stock distribution. Since the management staff at national and subnational levels make decisions on whether or not, according to the data, to supply health facilities and health workers with stock replenishments, the introduction of the digital system should also be accompanied with refresher training on the basic processes of supply chain management. Training should include the use of the technology, such as the use of text messages for the notification and the use of data dashboards.

# Services and applications
 When designing digital systems for stock notification, consider how the system can be made easy to use, with effective display of the data through fact sheets and simple graphical and tabular illustrations.
 Ensure that the digital systems and ordering routines are flexible enough to respond to local needs. For instance, where systems deal with quarterly stock orders, ensure they can also accommodate unexpected or seasonal needs.

# Availability of human resources
Ensuring availability of human resources

# Accessibility of health facilities
Ensuring access to health facilities

# Accessibility of health facilities and human resources for health: client-to-provider telemedicine Background
Despite progress in addressing health workforce shortages, challenges in the equitable access to health workers serves as a major hindrance to achieving the full requirements of effective coverage of human resources for health (70). Geographical inaccessibility and the preference of health workers for working in urban environments are among some of the well-documented reasons for imbalances in the distribution of health workers (71). While there is a wide range of ongoing efforts to reduce inequities in access to health workers, including incentives and alternative approaches to training, digital approaches such as telemedicine have also been explored as a mechanism of making health services available to underserved communities (71).
Within the WHO/ITU National eHealth strategy toolkit, telemedicine is defined as supporting "the provision of health care services at a distance" (18). Although other definitions elaborate on telemedicine as the use of ICT for medical diagnostic, monitoring and therapeutic purposes at a distance (72-75), the driving principle is centered on the provision of remote clinical support as a means of overcoming geographical barriers (72). Telemedicine can function between clients and health workers who are separated by distance, as well as among health workers based in different locations. The type of exchange between these actors varies and may include remote consultations, remote monitoring of vital signs or diagnostic data, and the transmission of medical files such as images for review, commonly referred to as "store and forward" (72-75).
In 2010, WHO reported extensively on the global status of telemedicine, including factors affecting its uptake in low-and middle-income settings (72). In more recent years, the emergence of mobile technologies has shifted the landscape, triggering new considerations for connecting clients and health workers (3). This guideline question builds on this preceding resource from WHO and examined the evolved use of telemedicine via mobile devices between clients and health workers.

# Geographic inaccessibility
Insufficient supply of qualified health workers Delayed provision of care

# Illustrative health system challenges

# Overview of the evidence
The following is a summary of the evidence on client-to-provider telemedicine. Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.
The evidence on effectiveness suggests that this intervention may improve some outcomes, such as fewer unnecessary clinical visits, reduced mortality among individuals with heartrelated conditions, exclusive breastfeeding, and increase health-related quality of life assessed 1-6 months after the intervention. However, it may make little or no difference on other outcomes, such as hospital admissions for heart-related conditions or older individuals receiving home-based care.
The qualitative evidence suggests that health workers appreciate the ability to offer immediate care, to follow up on missing clients and offer informed care, advice and emotional support to clients, even when physical contact is not possible. However, health workers feel that some cases still warrant face-to-face contact and are also concerned that loss of face-to-face communication will change the health worker-client relationship and lead to poorer quality care. Health workers may also be concerned about having to work beyond their clinical capacity and about potential issues of clinical liability.
From the client's perspective, the qualitative evidence suggests these individuals may appreciate being able to communicate with health workers from their homes and see telemedicine services as offering reassurance and increased access and the consistency and continuity of care that it can offer. Some clients may also feel that telemedicine services have increased their independence and self-care, although some health workers may be concerned about clients' ability to manage their own conditions.
The qualitative evidence on the feasibility of digital health interventions, in general, highlighted challenges related to network connectivity, access to electricity, usability of the device, sustaining training and support to health workers using the digital tools, concerns about data privacy and obtaining informed consent.

# Resource use
The evidence on resource use was assessed as having very low certainty. Resource use considerations are listed within the evidence-to-decision framework in Web Supplement 1.

# Gender, equity and human rights
This intervention may positively impact on equity by facilitating access to health services, particularly for individuals who speak minority languages. It also may reduce the burden of travel, particularly for people with caring or work responsibilities and those living far from health facilities. However, access to telemedicine services may be difficult for other groups, though, including people with hearing impairments or poor digital literacy.

# Recommendation and justification/remarks
Client-to-provider telemedicine (Recommended only in specific contexts or conditions)

# WHO recommends client-to-provider telemedicine:
 under the condition that it complements, rather than replaces, face-to-face delivery of health services; and  in settings where patient safety, privacy, traceability, accountability and security can be monitored.
In this context, monitoring includes the establishment of standard operating procedures that describe protocols for ensuring patient consent, data protection and storage, and verifying health worker licenses and credentials.
The guideline development group (GDG) felt that despite the mixed available evidence on effectiveness spanning a wide range of health conditions, client-to-provider telemedicine has the potential to expand access to health services. It may also potentially reduce the burden of travel and decrease inequities for populations that have difficulties in accessing health services through conventional approaches.
 This recommendation recognizes that while telemedicine may enhance access to health services, it should not be used to replace or detract from efforts to strengthen the health workforce.
 The establishment of standard operating procedures and mechanisms to ensure patient safety, privacy, traceability and accountability of services was deemed to be a necessary condition to mitigate the potential risks and harms of implementing this recommendation.

# Implementation considerations
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.  Explore whether changes in regulations are necessary to support any changes needed to health workers' scopes of practice. Develop policies and protocols to clarify the liability issues of health workers using telemedicine systems.

# Legislation, policy and compliance
 Explore reimbursement models and mechanisms of integrating client-to-provider telemedicine within existing service delivery models.
 Ensure that there are mechanisms for documenting and tracing past exchanges and decisions made during consultations.
 Ensure that use of the technology does not impact negatively on the relationship between client and health worker, particularly when users are learning about the technology and how to operate the devices. Extensive training on the technology and operating the device should be done before introducing the system for use directly with clients.
 Ensure that health workers remain able to use their own skills, judgement and knowledge within the changed context.
 Develop guidelines in collaboration with health workers that protect them from clients contacting them outside of normal working hours, such as in the context of emergencies or other considerations. If this contact is encouraged or expected, how can it best be managed to avoid overwhelming the health worker? Will health workers be compensated for this type of client support?
 Involve the relevant professional bodies as well as the health workers and clients in the planning, design and implementation of the telemedicine programme to ensure that their needs and concerns are met, such as to educate health workers on the legal frameworks governing telemedical exchanges.

# Considerations for equity, gender and human rights
 Pay special attention to the needs, preferences and circumstances of particularly disadvantaged or hard-to-reach groups, including people with low literacy or few digital literacy skills, people with limited control over or access to mobile devices, people speaking minority languages, migrant populations in new settings, and people with disabilities such as sight or hearing impairment.
 Consider how services can be made available to people with disabilities such as sight or hearing impairments, with poor access to electricity or poor network coverage, who cannot afford mobile devices or charges to use them, and people who have limited autonomy, for example because their access to devices is controlled by another person. Strategies to increase access to telemedicine in these cases may be provided through public kiosks or outreach through community health workers, as examples.
 Consider using telemedicine to link clients who speak minority languages to health workers who also speak the language.

# Availability of human resources

# Ensuring availability of human resources

# Accessibility of health facilities
Ensuring access to health facilities

# Accessibility of health facilities and human resources for health:
Access to qualified health workers with the appropriate competencies, skills and behaviours is an even greater obstacle to improving health outcomes than the availability of health workers (70,71). Geographical inaccessibility and the unequal distribution of health workers also contribute to limitations in the effective coverage of human resources for health (62). Digital approaches, most notably telemedicine between different types of health workers, have emerged as a potential way to overcome the barriers of long distances to qualified health workers and shortages in their numbers.

# Geographic inaccessibility
Insufficient supply/coverage of qualified health workers Insufficient health worker competence

# Illustrative health system challenges
Provider-to-provider telemedicine, as with client-to-provider telemedicine, facilitates the provision of health services at a distance and is primarily used to link less skilled health workers with more specialist ones (72). The communication between health workers may be made for a variety of reasons, including to get assistance with diagnoses, to remotely monitor clients' health status through vital signs and to conduct case-management consultations. This communication between health workers may occur asynchronously through the exchange of video and image files to be reviewed later (also referred to as store-and-forward exchanges) or synchronously in realtime exchanges (13,18,(72)(73)(74)(75).
Although telemedicine is one of the more established forms of ICT-enabled health service delivery (72), this guideline question expands on the existing evidence base, particularly in light of the advances in facilitating health workers' exchanges via mobile devices.

# Overview of the evidence
The following is a summary of the evidence on provider-to-provider telemedicine. Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.
The evidence suggests that provider-to-provider telemedicine may improve health worker performance, reduce the time for clients to receive appropriate care or follow-up, and decrease length of stay among individuals visiting the emergency department. However, the intervention may make little or no difference to other health status and well-being outcomes such as clinical improvements in individuals.
The qualitative evidence suggests that health workers appreciate the opportunity to communicate with each other and reduce their professional isolation. In particular, lower-level health workers noted how telemedicine services allowed them to access advice from higher-level health workers, which they saw as enabling better quality of care and client satisfaction. While some health workers may perceive provider-to-provider telemedicine as supportive, others may note challenges in collaboration, and concerns about liability and loss of control during the provision of care.
The qualitative evidence on the feasibility of digital health interventions, in general, highlights challenges related to network connectivity, access to electricity, usability of the device, sustaining training and support to health workers using the digital tools, concerns about data privacy and obtaining informed consent.
The evidence on resource use was assessed as having very low certainty. Resource use considerations are listed within the evidence-to-decision framework in Web Supplement 1.

# Gender, equity and human rights
The qualitative evidence on provider-to-provider telemedicine suggests that this intervention may improve equity by enabling health workers to facilitate access to higher-level care on behalf of their clients. Yet poor access to the digital technology, or the personal expenses associated with its use, may exclude some health workers, and thereby their clients, from these services.

# Recommendation and justification/remarks
Provider-to-provider telemedicine (Recommended only in specific contexts or conditions)
WHO recommends provider-to-provider telemedicine in settings where patient safety, privacy, traceability, accountability and security can be monitored.
In this context, monitoring includes the establishment of standard operating procedures that describe protocols for ensuring patient consent, data protection and storage, and verifying health worker licenses and credentials.
 The guideline development group (GDG) noted that provider-to-provider telemedicine has the potential to improve access to quality care and to reduce the isolation of health workers working in remote settings.
 Although the cost of the telemedicine system may vary depending on the modality used (exchange of image files, voice calls, remote monitoring), the GDG felt that provider-toprovider telemedicine could support care delivery by peripheral health workers.
 Due to concerns about liability issues, the GDG suggested that standard operating procedures/protocols be established to ensure patient safety, privacy, traceability and accountability of services and to mitigate the potential harms of implementing provider-to-provider telemedicine.
 It was also noted that the nature of telemedicine is changing and that a wide range of delivery channels are being used across health workers to facilitate communication exchanges.

# Implementation considerations
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.

# Legislation, policy and compliance
 Explore whether changes to licensing and legislation are necessary to support any changes in health workers' scopes of practice. Clarify liability issues for health workers using telemedicine systems and determine what can and cannot be done during remote consultations; the approach should not be a substitute for the adequate training of health workers.
 Ensure a clear legal framework for the implementation of telemedicine, including the licensing and regulation of care health workers using it. Additional clarifications are also required in cases of cross-border telemedicine, in which consultations are occurring across different jurisdictions.
 Ensure that there are mechanisms for documenting and tracing past exchanges and decisions made during consultations.

# Interoperability and standards
 The use of telemedicine requires that the health worker can access the patient's relevant clinical history. Integration with digital health record systems that can be accessed by the health worker and in which the patient's identity can be verified may be considered as a way to facilitate continuity of care.
Workforce  Ensure that the distribution of roles and responsibilities between different health workers is clear, including through regulations and job descriptions.
 Explore whether changes to health worker salaries or incentives are needed to reflect any changes in scopes of practice.
 Build trust between professionals considering establishing links between facilities across institutions, for example through twinning programmes.

# Continuous coverage
The extent to which clients receive the full course of intervention required to be effective
Targeted client communication 2 -defined as the transmission of health content or information to a specific audience based on their health status or demographic profile (13) -represents an approach for engaging with individuals to increase their knowledge about health and healthseeking behaviours, about where to find or how to access services, or for helping to retain them within health services when follow-up is needed. This includes the transmission of health information to individuals about health promotion, for spreading awareness of services and behaviours, transmission of reminders about services or treatments to encourage adherence to recommended practice, and transmission of notifications about diagnostic results (13). Using registered phone numbers or other contact information, the delivery of health content to individuals can be via a range of digital channels, including text messaging, voice, interactive voice response, multimedia applications and games (apps on mobile devices), and social media.
Several WHO guidelines have explored the use of targeted client communication via mobile devices as a potential tool to improve medication adherence. Most notably, the 2016 Consolidated guidelines on the use of antiretroviral drugs include a recommendation on the use of text messaging as part of a package of interventions to support adherence to antiretroviral therapy (15). Similarly, the 2017 Guidelines for treatment of drug-susceptible tuberculosis and patient care also recommend the use of text messages and voice calls to support health education and treatment adherence (16).
Building on this previous work, this guideline question reviews the use of targeted client communication via mobile devices across a broader range of health topics and populations of interest for sexual, reproductive, maternal, newborn, child and adolescent health (SRMNCAH).
Note that the use of targeted client communication in the prevention and management of noncommunicable diseases will be examined in a subsequent version of this guideline.
2 Although WHO's Classification of digital health interventions v1.0 uses the term "client" (13), the terms "individual" and "patient" may be used interchangeably, where appropriate.

# Low demand for services

# Low adherence to treatments
Loss to follow up

# Lack of access to information
Illustrative health system challenges

# Overview of the evidence
The following is a summary of the evidence on targeted client communication via mobile devices.
Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.
The evidence on effectiveness suggests targeted client communication may have positive impacts on some behaviours and health outcomes, such as: oral contraception use by adolescents, modern contraception use by adults, adherence to antiretroviral medications, attendance of antenatal care appointments, taking iron and folate tablets during pregnancy, skilled birth attendance, receipt of childhood vaccinations, and attendance of HIV appointments among exposed children.
However, the evidence also indicates that targeted client communication may make little or no difference to other outcomes, such as: health status as assessed by CD4 count and adherence to prenatal antiretroviral medication.
The evidence on targeted client communication also suggests the intervention has some unintended negative consequences, such as women experiencing physical violence in the context of receiving targeted communications for sexual and reproductive health (SRH) services.
The certainty of the evidence was assessed as very low for some outcomes such as: adherence to antiretroviral medication and attendance for STI/HIV testing among adolescents, breast and cervical cancer screening; and women's attendance for neonatal appointments.
The qualitative evidence suggests that targeted client communication is generally acceptable to individuals, but that some population subgroups have concerns about the confidentiality of health information, particularly for sensitive health issues such as HIV infection and other aspects of SRH.
Some clients describe digital targeted client communication programmes as providing them with support and connectedness. The fact that someone is taking the time to send them messages can make clients feel like someone is interested in their situation, invested in their well-being and cares about them. Some clients describe this as leading to feelings of encouragement, increased self-confidence and self-worth, and describe the messages as providing support, guidance and information, giving a sense of direction, reassurance and motivation. Some clients also feel that the sense of caring and support that they receive from health workers through these types of programmes has a positive influence on their relationship with their health worker.
However, clients who are dealing with health conditions that are often stigmatised or personal (e.g. HIV, family planning and abortion care) worry that their confidential health information will be disclosed, or their identity traced due to their participation in these types of programmes. This was noted particularly for vulnerable populations, including adolescents and pregnant women living with HIV, in which the transmission of sensitive health information could disclose their health status or compromise their privacy when seeking health information and services.
Clients' perceptions and experiences of digital targeted client communication are influenced by characteristics of the content; the format; and the delivery mechanisms. The evidence also indicates that access to and use of targeted client communication may be particularly difficult for certain groups of individuals, such as people with low literacy or with limited or controlled access to mobile devices.
The qualitative evidence on the feasibility highlights a number of constraints. These include reliable network connectivity, access to electricity and mobile devices, and the availability of mechanisms to obtain informed consent when enrolling clients into the service. Health systems may experience challenges when attempting to communicate with clients who regularly change their phone numbers without informing the health worker or clients who have poor access to a mobile device.

# Resource use
The evidence suggests targeted client communication via mobile devices may use fewer resources than non-digital interventions.

# Gender, equity and human rights
The qualitative evidence suggests targeted client communication may be particularly difficult for certain population groups, including individuals with poor access to network services or electricity; with limited or controlled access to mobile devices, particularly women and adolescents; individuals who speak minority languages or have low literacy skills or low digital literacy skills; or individuals with conditions that cause them to be particularly concerned about the confidentiality of information exchanged via digital devices. Examples of ways to address sensitive content and data privacy include ensuring that individuals are actively made aware of how to opt out of receiving the targeted client communication.

# Recommendation and justification/remarks
Justification/remarks  The guideline development group (GDG) considered this intervention to offer the potential to improve health behaviours and reduce inequities among individuals with access to mobile devices. The GDG, however, highlighted that measures should be taken to address inequities in access to mobile devices so that further inequity is not perpetuated in accessing health information and services, including mechanisms to ensure individuals who do not have access to mobile devices can still receive appropriate services.
 The GDG also raised the need to address potential concerns about sensitive content and data privacy, including potential negative unintended consequences. This could be done, for example, through mechanisms that actively allow individuals to opt out of services.

# Linkages with other WHO recommendations
The GDG noted that WHO has previously made recommendations related to targeted client communication for improving HIV and tuberculosis medication adherence, which contributed to the considerations for this recommendation. These previous recommendations are listed below.
In the Consolidated guidelines on the use of antiretroviral drugs for treating and preventing HIV infection (15), the following interventions demonstrated benefit (all with moderate-quality evidence) in improving adherence and viral suppression:  Tracers refer to communications with the patient, including via home visits, SMS text messages or voice telephone calls.

# Implementation considerations
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.

# Legislation, policy and compliance
 Ensure that clients are actively made aware of how to opt out of receiving the targeted client communication. Attention needs to be paid to ensure that consenting procedures clearly communicate to the clients the intended uses of their data, including to the intentions to continue contacting them, over what period of time, and their right "to be forgotten", or opt out. Procedures need to be in place to ensure that participants are not unduly pressured to provide personal information.

# Services and applications
 Ensure that individuals know the messages are coming from a trusted sender such as a government or health institution, health worker or other familiar entities worthy of their attention.
 Ensure that any sensitive content or personal data transmitted and stored are held on a secure server with protocols in place for destroying the data when appropriate.
 Effective digital communication relies on behaviour change to achieve the intended impact. Such communication should be conducted in the context of a comprehensive communications strategy so that messages received through mobile devices are reinforced by other mechanisms. For example, digital messages should be consistent with the information communicated by health workers, print media and other sources. Further considerations to review when developing content for digital communication include the following.
 Consider the languages used for the content to reach the target audiences, including whether they are in active spoken or written use.
 Ensure that messages are clear and simple. Avoid jargon, technical terms and shortened forms of text. Consider testing to ensure that messages are understood as intended and that any necessary colloquial translations are used.
 Consider the tone of the messages and whether clients are likely to perceive them as friendly and motivational as opposed to shaming or frightening.
 Consider how the content can be tailored to the client, for instance by using their name, local information or personalized reminders.
 Consider whether to include two-way communication with clients to enable their interaction and response to the health system.
 Ensure that the content of the communication reflects the reality of the available commodities and services. For example, encouraging women to seek family planning at their nearest health facility is appropriate if a full range of contraception and advice is available there, including the relevant commodities.
 Ensure the mode of content delivery is appropriate for the setting's network connectivity. For example, in contexts with low connectivity coverage, not all populations may be reached through digital channels making use of multimedia or mobile app-based communications.
Consider offering messages in a variety of formats (text, audio and video) depending on the setting and infrastructural limitations.

# Equity and sociocultural considerations
 Pay attention to the circumstances of people who have poor access to electricity or poor network coverage, people who cannot afford a mobile device or voice and data charges and people who have limited autonomy, for example because their access to phones is controlled by another person. For the latter case, the GDG felt that the programme should target content accordingly and ensure that users were not put at increased risk.
 Develop concurrent initiatives where such inequity exists so that individuals who do not have access to mobile devices can still receive appropriate services.
 Pay particular attention to the needs, preferences and circumstances of especially disadvantaged or hard-to-reach groups, including people with low literacy or few digital literacy skills, people speaking minority languages, migrant populations in new settings, people affected by emergency situations and people with disabilities such as sight or hearing impairment. Also consider any demographic characteristics, sexual identity or preferences that could put a targeted population at greater risk and ensure that the way the information is provided and accessed is sensitive to this.
 Ensure there are little, or no charges tied to the programme, for instance those associated with downloading apps or sending or receiving the content. Implementers may need to negotiate with mobile network operators and other partners to determine options for subsidizing communication costs or employing voucher systems.

# Effective coverage
The proportion of individuals receiving satisfactory health services among the target population

# Effective coverage:
Health worker decision support
Quality of care, defined as the "degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge", is a foundational component of universal health coverage (76). Quality of care has consistently been documented as suboptimal, particularly across low-and middle-income countries. Commonly cited reasons for poor quality of care have included health workers' inaccurate diagnosis, inappropriate or unnecessary treatment, inadequate or unsafe clinical practices, along with a range of other systemic issues such as insufficient commodities and infrastructural limitations (76).
Although low quality of care stems from numerous deeply rooted health system challenges, decision support tools that offer guidance to health workers have been leveraged as a mechanism to augment adherence to recommended clinical practices (77-80). In their digital form, decision support systems for health workers are defined as electronic systems designed to aid directly in decision-making, in which characteristics of individual patients are used to generate patient-specific assessments or recommendations that are then presented to clinicians for consideration (13,18). Digital decision support for health workers (13), also referred to as clinical decision support systems (CDSS), may be used for a wide range of clinical interactions, including diagnosis and treatment, to facilitate appropriate referrals, minimize errors in medication prescription, and ensure the provision of thorough and accurate care (79). Functionally, decision support tools may be designed to guide health workers through algorithms and rules based on clinical protocols, provide the health worker with checklists for case management and referrals, screen clients by risk or other health status and to assist in health worker activity planning and scheduling (13).
Poor adherence to guidelines

# Lack of or inappropriate referrals
Insufficient health worker competence

# Illustrative health system challenges
The use of decision-support tools has been well established and is supported by some emerging evidence (80). However, over the last decade, health worker decision support has transitioned from being operated on fixed computerized systems to mobile devices, which provide unique opportunities for point-of-care assessment, diagnosis and management. Furthermore, most health care systems in low-and middle-income countries, especially in rural areas, do not have the required infrastructure for desktop computer-based decision support systems and are increasingly investing in making these tools accessible via mobile devices.
This guideline question will explore the added value of digital decision support tools available at primary health care levels and accessible to health workers via mobile devices. Furthermore, as the function of this digital health intervention is broadly applicable across programmatic areas, the guideline question will explore the use of such digital job aids across health conditions within primary care settings.

# Overview of the evidence
The following is a summary of the evidence for decision support for health workers via mobile devices. Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.
There is limited evidence on the effectiveness of health worker decision support via mobile devices directed to clinical health workers. For the intervention directed to community health workers, the evidence suggests that this may have positive effects on individuals taking prescribed medication but may make little or no difference to the individuals' overall health status. When directed to community health workers, decision support may make little or no difference to clients' satisfaction with the information they receive.
The qualitative evidence suggests health workers find the intervention useful and reassuring for guiding the delivery of care. However, some health workers perceive algorithms as too prescriptive, and are concerned that they may lose their clinical competencies by blindly following treatment algorithms. The evidence also suggests that clients find the intervention acceptable and enables health workers to be more thorough when providing care.
The qualitative evidence on the feasibility of digital health interventions, in general, highlights challenges related to network connectivity, access to electricity, usability of the device, sustaining training and support to health workers using the digital tools.

# Resource use
No evidence on resource use was identified.

# Gender, equity and human rights
The evidence on gender, equity and human rights on digital health interventions broadly suggests health workers based in peripheral facilities and rural communities may find these interventions helpful in overcoming geographical barriers and linking to the broader health system, including to access clinical guidance. Health workers in these settings may, though, be more likely to experience poor network coverage and access to electricity, may have lower levels of training and literacy with digital technology, and may have fewer resources, including having limited access to mobile devices.

# Recommendation and justification/remarks
Health worker decision support (Recommended only in specific contexts or conditions)

# Recommendation 7
WHO recommends the use of digital health worker decision support in the context of tasks that are already defined as within the scope of practice for these health workers.
 The GDG expressed that the use of health worker decision support tools when used on mobile devices may improve provision of services point of care. The GDG noted, however, that decision support should not be used for tasks that are beyond the current scope of practices as this may introduce the risk of health workers delivering services for which they have not received adequate training, or of overwhelming the health workers with an expanded set of tasks.
 The GDG highlighted the importance of ensuring the validity of the underlying information, such as the algorithms and decision-logics.
 The GDG also acknowledged additional literature that was not assessed as part of this guideline, on decision support systems via fixed/stationary digital devices. The GDG felt that this evidence suggested the potential of such tools in improving patient/client outcomes could be extrapolated to mobile use, which may offer additional opportunities for settings where the infrastructure for fixed devices is weak.
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.
 Health workers may find it helpful in increasing the acceptability to clients/patients of using digital decision support if they explain that they will be using a digital device and seek clients' permission before using them. Clients should also be made aware that the information from the counselling may be saved and used at future visits to improve quality and continuity. Any concerns with acceptability may be mitigated by, for example, health workers showing the client the inputs and results or listening to the messages or videos together with them so that the device does not become a barrier in the consultation.
 Before using the decision support system, implementers should assess health workers' skills and knowledge to ensure that they have adequate capacity to obtain accurate data before input, to avoid erroneous outputs.
 Referral linkages might need to be strengthened to support possible increases in the number of patients seeking care for previously undetected needs now being revealed by the decision support system.

# Services and applications
 Check the relevance and quality of the decision support content (such as algorithms) and that it aligns with evidence-based clinical guidance, such as WHO or national guidance. Engaging expert groups on the clinical/health topic area may also be necessary as existing guidance may not have sufficient clarity.
 Ensure adequate time for testing all paths of the algorithm with any changes to the software. This type of validation can be done through mechanisms such as an independent review and using mock cases to test the intended output from the algorithms. Also consider built-in mechanisms to update content remotely as algorithms evolve.
 Both health workers and clients should understand that the support provided is based on existing guidelines and policy. While health workers may deviate from the recommendations, they should be clear about their rationale for doing so. Where possible, enable cases to be documented in which health workers feel they need to deviate from the guidance proposed by the decision support system.
 Ensure that use of the device does not impact negatively on the relationship between patient and health worker, particularly when the provider is learning to use the device. As above, this may be helped, for example, by health workers showing patients the inputs and results or listening to the messages or videos together with them so that the device does not put up a barrier. Finally, pay attention to user experience so that correct use of the system is easy for health workers and does not demand more time compared with alternative approaches without it.

# Standards and interoperability
 For the ease of viewing the patient's health history, decision support tools are often integrated with digital health records. See section 3.8 for the evidence and discussion surrounding the combination of decision support with digital tracking of clients' health status and services.

# Multiple points of coverage: digital tracking of clients' health status and services combined with decision-support and targeted client communication
The use of paper-based systems in the delivery of health services introduces a clerical burden on health workers. Additionally, the ability for health workers to keep track of clients effectively, and follow up on services, whether within the facility or in the community, is essential to the continuity of care (12).
Digital tracking is the use of a digitized record to capture and store health information on clients in order to follow-up on their health status and services received (13, 40,81). This may include digital forms of paper-based registers and case management logs within specific target populations, as well as electronic patient records linked to uniquely identified individuals. Digital tracking makes possible the registration and follow-up of services and may be done through an electronic medical record (EMR) or other digital forms of health records. Digital tracking aims to reduce lapses in continuity of care by stimulating timely follow-up visits and may incorporate decision support tools to guide health workers at the point of care in executing clinical protocols, delivering appropriate care, scheduling upcoming services and following checklists for appropriate case management.
Digital tracking and decision support systems may also be linked with demand-side interventions to engage clients/patients, such as through targeted client communication via mobile devices. Targeted client communication in this context is defined as the transmission of targeted health content or reminders to a specified population or to individuals within a predefined health or demographic group (13).

# Effective coverage
The proportion of individuals receiving satisfactory health services among the target population

# Continuous coverage
The extent to which clients receive the full course of intervention required to be effective

# Accountability coverage
The proportion of those in the target population registered into the health system

# Insufficient continuity of care
Lack of quality/ reliable data

# Delayed provision of care
Poor adherence to guidelines

# Illustrative health system challenges
This guideline has sought to understand the benefit of an integrated package consisting of three different digital health interventions, to support health worker practices as well as to stimulate client-side demand for health services and stimulate behaviour change.
This guideline reviewed the following intervention combinations:

# Overview of the evidence
The following is a summary of the evidence on the digital tracking of clients' health status and services (shortened to digital tracking), in combination with health worker decision support and targeted client communication. Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.

# Effectiveness (a) Digital tracking and decision support:
The evidence on the effectiveness of digital tracking combined with decision support suggests it may improve health service use and health outcomes, such as: attendance of antenatal care appointments, taking iron tablets during pregnancy, immediate breastfeeding, receipt of the third dose of polio vaccine, and use of postpartum contraception six months after birth.
However, digital tracking combined with decision support probably makes little to no difference on other outcomes, such as: the proportion of children under five who are vaccinated, proportion of women who give birth in a facility, women breastfeeding exclusively for six months, or on the proportion of women using contraception within six months of birth.
There was limited evidence on the effect of digital tracking combined with decision support on the use of emergency visits for children under five and on the timeliness of receiving services, as the certainty of this evidence was assessed as very low.

# (b) Digital tracking with targeted client communication:
No evidence was identified for this intervention combination.

# (c) Digital tracking with decision support and targeted client communication:
There was limited evidence in demonstrating the effectiveness of combining digital tracking with both decision support and targeted client communication, as the certainty of this evidence was assessed as very low.
The qualitative evidence suggests that most health workers see advantages to digital technologies compared with paper-based systems. These include quicker recording of required client data and services delivered, easier access to client data, easy identification of mistakes, and not having to carry paper registers. Health workers are often reluctant, however, to use digital tracking when they have to maintain both digital and paper-based systems, since this increases their work burden.
There was limited evidence documenting the feasibility of these integrated interventions specifically. Challenges have been highlighted, however, by the qualitative evidence on the feasibility of digital health interventions in general, including those of network connectivity, access to electricity, usability of the device, sustaining training and support to the health workers using the digital tools, and system integration.

# Resource use
No evidence on resource use was identified. Resource use considerations are listed within the evidence-to-decision framework in Web Supplement 1.

# Gender, equity and human rights
The qualitative evidence on these digital health interventions suggests health workers based in peripheral facilities and rural communities may find the interventions useful in overcoming geographical barriers and linking to the broader health system. Health workers in these settings may also, however, be more likely to experience poor network coverage and poor access to electricity, may have lower levels of training and literacy with technology, and may have fewer resources, including having poorer access to mobile devices. Justification/remarks  The guideline development group (GDG) recognized that this intervention package may pose challenges, particularly in settings in which the health system may not be able to manage the infrastructural and technical complexity of such a multifaceted intervention.

# Recommendation and justification/remarks
The GDG also felt that the intervention may require substantial upfront resource use but believed that the intervention may reduce costs in the long term by transitioning away from inflexible paper-based systems.
 Despite the risk of increasing complexity by implementing a system with multiple digital components, the GDG believed that implementing these interventions in an integrated manner offered opportunities to (i) reduce health workers' time spent on redundant activities such as reporting; (ii) increase the timeliness and responsiveness of health workers by linking data from client health tracking systems to the actions recommended from decision support tools; and (iii) provide a more holistic view of the client and their interactions with the health system.
 While there is value in a multi-pronged digital intervention that simultaneously targets supply side factors (i.e. decision support to health workers), and demand-side factors (i.e. targeted client communication), the technical and human resource requirements for such an intervention should be considered. The GDG suggests the three components be implemented in a gradual manner, particularly in settings where the enabling environment and infrastructure may not be sufficiently mature to support such a multifaceted intervention.
 In line with the separate recommendation on targeted client communication via mobile devices (see section 3.6 for more detail), the GDG's recommendation to combine it into digital tracking is conditional on measures being taken to address inequities in access to mobile devices and address concerns about sensitive content. Similarly, the inclusion of the decision support component will require alignment to the tasks and scope of practice for health workers to avoid potential harms and added burden (see section 3.7 for more detail).

# Linkage with other WHO recommendations
These findings align with recommendation 11 of the WHO guideline on health policy and system support to optimize community health worker programmes, which suggests that practising community health workers "document the services they are providing and that they collect, collate and use health data on routine activities, including through relevant mobile health solutions" (17).

# Implementation considerations
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.
Legislation, policy and compliance  Accurate client/patient identification to facilitate the digital tracking of health services across different facilities and health workers requires adequate policy and legal processes and protections. This can include the use of a card-based or biometric-based identifier, as an example, and having telecommunications infrastructure that is available consistently across facilities and programmes.
 Consider whether the digital tracking would have adequate infrastructural support to be maintained over time. The start-up costs and infrastructural requirements of a digital tracking system tend to be higher than for paper-based interventions. When used appropriately and effectively, the costs of digital interventions are amortized, and cost-savings may materialize in the long run. However, in contexts where basic health infrastructure is limited, including in human resources, digital tracking systems may be very resource-intensive to set up and maintain.

# Standards and interoperability
 The digital tracking should be linked to a system that provides a unique identity for each individual. Such unique IDs help health workers search for clients, reduce the potential for duplicate registration of clients in community and facility systems and ensure continuity of care. This unique ID could, in turn, be linked to a local or national ID system to provide a foundational digital identity that can facilitate longitudinal follow-up and linkages across different levels of the health system and digital health interventions.
Workforce  Consider phasing implementations to avoid overburdening health workers. For example, consider introducing integrated packages only once health workers have already been implementing at least one of the interventions and are familiar with digital technologies.
 Focus on introductory and ongoing training of health workers in using these tools, including support for technical troubleshooting during the provision of care. Health workers may have challenges in using technology during the provision of services, which can negatively impact the quality of care, or result in the technology not being used. Use metrics to assess health workers' use of the digital system and identify opportunities to reinforce training.

# Equity and sociocultural considerations
 Inequities may be reduced for populations included within the digital tracking system because it helps to ensure that they receive services. Inequities may arise, however, for those outside of the digital tracking system whose service provision might not be accounted for. Such inequity needs to be monitored during implementation. The problem can be addressed by first enumerating the target population and so increasing the accuracy of the denominator by which populations are eligible for services.
 The digital tracking of individuals' health status may be controversial in some circumstances, for example among migrants or other groups who lack firm legal status in particular settings.
The extent to which such groups may trust tracking depends on who is doing the tracking and how the information is likely to be used. It is important to take these concerns, and local policies on digital identities, into account when designing a programme to ensure it does no harm.

# Effective coverage
The proportion of individuals receiving satisfactory health services among the target population

# Effective coverage: digital provision of training and educational content to health workers Background
Broadly defined as the management and provision of educational and training content in digital form for health professionals, electronic learning (eLearning) has emerged as one approach to increasing health workers' access to training and educational resources (18). More recently, the widespread reach of mobile devices has prompted the use of such technologies to deliver training content to health workers, also known as mobile learning (mLearning). Such training content may be exchanged using channels such as SMS text messaging, the multimedia messaging service, applications ("apps"), games, and other forms of digital modality (82). In particular, low-and middle-income countries and remote areas with limited ICT infrastructure and geographical barriers may seek to leverage mobile devices to maximize access to educational content and continuing medical education (82).
Although the use of digital tools for strengthening the health workforce is referenced in several WHO resources (15,70,71,83), these do not examine the specific considerations on digital provision of training and educational content. This guideline question assesses the potential contributions and implications of providing digital training and educational content via mobile devices, as part of complementary efforts to support workforce needs for in-service training and continued education.
Poor adherence to guidelines

# Lack of access to information
Insufficient health worker competence

# Illustrative health system challenges

# Overview of the evidence
The following is a summary of the evidence on the provision of digital training and educational content for health workers accessible via mobile devices. Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.
The evidence suggests that this intervention may increase health workers' knowledge. However, the effects of this intervention on other outcomes, including health workers' performance, skills and attitudes, is uncertain because there is no direct evidence, or the evidence is of very low certainty.

# Acceptability 3
The qualitative evidence from medical and nursing students suggests that these users see a number of advantages to mLearning tools, including the ease and portability of accessing materials and ability to personalize content to their own needs. They may have some concerns, however, for instance about the validity and accuracy of the information, as well as potential negative effects when used during patient interactions.
The qualitative evidence on the feasibility of digital health interventions highlights challenges related to network connectivity, access to electricity, usability of the device, sustaining training and support to health workers using the digital tools.

# Resource use
No evidence on resource use was identified. Resource use considerations are listed within the evidence-to-decision framework in Web Supplement 1.

# Gender, equity and human rights
The qualitative evidence on digital health interventions broadly suggests health workers based in peripheral facilities and rural communities may find these interventions helpful in overcoming geographical barriers and linking to the broader health system. However, health workers in these settings may also be more likely to experience poor network coverage and access to electricity, may have lower levels of training and literacy with digital technology, and may have fewer resources, including poorer access to the mobile devices that may be needed for some programmes. Justification/remarks  Despite the availability of evidence primarily focused on improving health worker knowledge, the guideline development group (GDG) felt that the potential benefits of the intervention outweighed the potential harms.

# Recommendation and justification/remarks
 The GDG also noted that mLearning offered an additional delivery channel for continuing health education, and thereby expanding access to in-service training resources and professional development opportunities to a broader set of health workers.
 The GDG also considered the potential for cost savings for continued health education, when compared with the costs of expanding face-to-face in-service training.
 It should be noted that this intervention only applies to post-certification health workers and used in the context of in-service training and continued health eduction.

# Linkage with other WHO recommendations
The WHO guideline on health policy and system support to optimize community health worker programmes suggests an emphasis on face-to-face learning for pre-service community health workers, to be supplemented by eLearning on aspects where it is relevant (17).
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.
 Consider network capacity and coverage especially if mLearning materials may be videos which can my time consuming to download in certain settings.

# Legislation, policy and compliance
 Consider if health workers can earn credits for continuing education using these materials, as a way of increasing their uptake.
Workforce  To increase the acceptability of mLearning devices, it may be important to improve awareness among staff and supervisors about the value of portable devices and to develop ground rules or codes of conduct for when and how devices should be used.
 Similarly, it may be helpful to give patients explanations of device use, and to ask patients' permission before using a device. Ensure also that use of devices does not impact negatively on  the relationship between health workers and clients, particularly if being used in the context of service delivery, and especially when health workers are learning to use the devices.
 Involve the relevant professional bodies, including national certification or institutional boards, to ensure that the content of the mLearning programmes aligns with the current scopes of practice and national training curriculums for health workers.

# Services and applications
 Ensure that the information is from a source that is considered trustworthy and credible by health workers in your setting. For example, the information loaded on the mLearning system should be based on validated content or should align with national or WHO clinical guidance.
 Consider which types of training content are best delivered via mLearning channels and which through other or mixed channels, including through in-person training.
 Where available, mLearning materials should be curated and accredited as formal training courses.
 Ensure that the programme is user-tested among health workers, both those in practice and those in training, to ensure that their needs and concerns are met.
 Ensure that health workers can easily store content for future reference.
 Consider how health workers can tailor the content to suit their specific needs. For instance, develop content in a modular format so that health workers can select information for particular review.

# Implementation considerations
Digital health has the potential to help address problems such as distance and access, but still shares many of the underlying challenges faced by health system interventions in general, including poor governance, insufficient training, infrastructural limitations, and poor access to equipment and supplies. These considerations need to be addressed in addition to specific requirements introduced by digital health. As the context will moderate the eventual impact of digital health interventions, the broader health system and enabling environment become especially critical.

# Birth notification
Recommended in specific conditions ACCOUNTABILITY Recommendation 2

# Death notification
Recommended in the context of rigorous research and specific conditions Recommendation 6

# Targeted client communication
Recommended in specific conditions DEMAND

# Client-to-provider telemedicine
Recommended in specific conditions SUPPLY Recommendation 4

# Stock notification & commodity management
Recommended in specific conditions SUPPLY Recommendation 3

# Provider-to-provider telemedicine

# Recommended in specific conditions

# Health worker decision support

# Recommended in specific conditions digital tracking + decision support
Recommended in specific conditions QUALITY Gaps within these different implementation components can jeopardize the quality and impact of the implementation. For example, the delivery of inaccurate health information poses a risk on the health outcomes that may result from this implementation. Likewise, the inappropriate selection of hardware, software and communication channels may present challenges to the usability and reach of the implementation. Additionally, limitations in the maturity of the ICT and enabling environment can prevent the uptake of the intervention and potentially strain the health system by diverting resources and inducing fragmentation of services. Furthermore, these implementation components should be designed appropriate to the local context based on intended user needs, in reflection of the absorptive capacity of the health system, and the behavioural and organizational changes that would be required to adapt to these digital interventions.

# Overarching implementation considerations
Implementations need to be made appropriate to the local needs, intended users, and overall ecosystem comprised of the ICT and enabling environment. The National eHealth strategy toolkit produced jointly by WHO and ITU (18) provides useful considerations for assessing the ICT and enabling environment and can be used to help countries in determining their readiness to adopt the digital health interventions.

# Components of ICT and enabling environment Description

# Leadership and governance
This includes coordination mechanisms at the national level, alignment with health goals and political support, and awareness and engagement from stakeholders

# Strategy and investment
This includes aligning financing with health priorities and ensuring funding to achieve the objectives of the strategy

# Legislation, policy and compliance
This includes a legal, policy and enforcement policy environment to establish trust and protection for individuals and industry

# Services and applications
This includes the systems and functionalities that need to be in place to enable stakeholders to access, use and share health information
This includes the physical infrastructure, core services and hardware (such as networks) that underpin a national digital health environment. An example is identification authentication services

# Standards and interoperability
This includes the standards that enable consistent and accurate collection and exchange of health information across health systems and services
This includes the available education and training programmes for health workforce capacity-building in digital health Source: Adapted from WHO/International Telecommunication Union National eHealth strategy toolkit (18) In addition to considerations surrounding the ICT and enabling environment, the following crosscutting implementation issues were identified from systematic reviews of the global evidence. These considerations have been mapped to the different components in Table 4.1.
Note that the following section is not intended to be an exhaustive list of implementation considerations, but rather aims to highlight the issues commonly cited during the evidence syntheses conducted for the guideline or identified by the guideline development group. Implementers should seek more comprehensive implementation resources before designing and implementing recommended digital health interventions.

# Leadership and governance
 Involve health workers, facility staff and other users in the design, user testing and implementation of the programme, and include them in decisions about changes to the programme. Ensure stakeholder consultation and engagement throughout the process.

# Strategy and investment
 Assess how the programme will be integrated into existing health care systems, including how it might change workflows and the delivery of services. For example, how will the daily routines of health workers need to change to include digital technologies? Will there be tasks or activities, such as manual tabulation of data, that will no longer be required?
 As with introducing new interventions, develop policies for change management to optimize acceptability, feasibility, and overall uptake. This requires an understanding of the users of the digital intervention and others targeted by it, their perceptions and interactions with the intervention, and the context in which the intervention is implemented.
Legislation, policy and compliance  Put systems in place to ensure data privacy, ownership, access, integrity and protection of patient information. Ensure that these systems meet national legal standards. Also ensure that these systems meet the concerns of clients 4 and that health workers, clients and other stakeholders are aware of and able to use these systems. This is particularly important in contexts where individual health information has financial value and may be particularly vulnerable (information used for reimbursement in a health insurance scheme, for example), where more rigorous enforcement is needed. Security is needed to address not only risks to patient confidentiality, but also risks to data integrity such as unauthorized alteration of data.
 Develop systems for ensuring informed consent among all populations, including those with limited literacy.
 Establish a plan or processes to replace manual/paper-based systems -to reduce the burden of operating a dual system.

# Standards and interoperability

# Future research
This chapter on future research highlights crosscutting evidence gaps observed across a range of interventions in relation to effectiveness, resource use and cost-effectiveness, and gender, equity and rights. In addition, specific research questions are provided for each of the interventions, based on the gaps identified through the evidence-to-decision framework and GDG.

# Overarching research gaps
The following sections describe the overarching research priorities identified through this guideline process. These reflect the general areas in which the available evidence was found to be of low or very low certainty or confidence, or where no direct evidence was identified.
Where studies were available, in some cases the certainty or confidence of the evidence was affected by poor reporting of outcomes, studies with small numbers of participants, and limited representation across different settings.
Annex 5 maps the state of evidence and its gaps based on the findings from the effectiveness reviews for the included digital health interventions.
For many of the interventions, the available evidence on effectiveness was sparse. Future research should measure health system process improvements that may immediately result from the digital intervention, such as health workers' adherence to recommended practice, as well as related distal health outcomes. Researchers should be realistic about the extent to which digital health interventions can impact on distal health outcomes, which are often affected by a variety of factors beyond the interaction with the digital intervention. Additionally, effectiveness studies need to include ways of concurrently monitoring technological performance (for example, do messages reach intended individuals?) and behavioural performance or user engagement (e.g. do individuals who get messages listen to or read them, and subsequently act on them?).

# Resource use and cost-effectiveness
The studies included in the systematic reviews of the effectiveness of the digital interventions considered by the guideline identified limited evidence on the resources used to implement these interventions. Costing studies should assess costs over a longer period, with appropriate accounting of amortization and maintenance of equipment and the continuous user support required. Future research should explore the cost-effectiveness, and potential for cost savings of the identified intervention and additional savings achieved through combining interventions.
Gender, equity and rights

# Considerations for the design of future evaluations
The GDG also identified several issues related to the design of future evaluations of digital health interventions, including the following:
 Health system focused digital interventions, such as stock management and birth and death notification, are often complex in the number of components, behaviours targeted, and organizational levels involved (89). These factors may make designs such as randomized controlled trials for evaluating the effectiveness of these interventions difficult to apply. Other designs may therefore need to be considered, such as controlled before-and-after studies, stepped-wedge randomized controlled trials and interrupted time series studies.
 While there is value in evaluating changes in client/patient health outcomes, intermediate outcomes are also critical for the evaluation of digital health interventions. For example, the effect of decision support on client/patient health outcomes are influenced not only by the information delivered through the digital system, but also by a host of other factors, including access to medicines, their cost, family support, and biomedical factors such as whether the individual responds appropriately to recommended treatments or has comorbidities. A logical framework of how the digital intervention functions may be helpful in understanding the pathways through which the intervention influences a targeted behaviour or health system challenge and in selecting appropriate outcomes along these pathways.
 Digital technologies provide new opportunities to capture research data for measuring the effectiveness of implementations in real time, thus facilitating the ability to conduct evaluations more rapidly. Incorporating the research data collection needs for primary and secondary outcomes of interest at the design stage can ensure that the data needed to measure these outcomes is captured alongside the implementation.
 Rapid changes in digital technologies and the iterative approaches often used for software development may force digital health interventions to evolve during evaluation periods, which may pose challenges for the evaluation process. Detailed process evaluations running alongside impact evaluations may be helpful in understanding the effects of incremental changes in the digital interventions over time.
 Future research efforts should establish common metrics and tools for assessing the effectiveness and cost-effectiveness of digital health interventions 6. Disseminating and updating the guideline

# Dissemination and implementation of the guideline
This guideline will be available in its full version, as well as a condensed form that includes the executive summary, implementation considerations, and future research priorities. WHO will disseminate this guideline as much as possible through regional offices and networks and existing large-scale, global convenors, including the Asia eHealth Information Network (AeHIN) (30), the Global Digital Health Network (31), the Health Data Collaborative's Digital Health and Interoperability Working Group (90), and the IBP Initiative (37) among other peer-learning groups and communities of practice. WHO will also convene regional consultations at policy-maker gatherings and with digital health working groups. Additionally, structured webinars will be used to share the recommendations and maximize the reach of these evidence-based findings on digital health interventions.
It is equally important that this guideline is shared with public health practitioners who have a limited experience with digital health. Where possible, WHO will identify opportunities for presentation panels at conferences for clinicians and public health practitioners across different domains, including health systems strengthening, and digital innovations and UHC, with an emphasis on conferences that focus on low-and middle-income countries.
WHO has also developed a complementary implementation guide, the Planning and costing guide for digital interventions for health programmes, to help implementers and health planners in Ministries of Health select, plan for, cost and implement the recommended digital health interventions in accordance with identified local health needs, the enabling environment and available technologies. This implementation guide will provide a stepwise process to ensuring that the implementation of the recommended digital health interventions fits in meeting the identified needs, and within appropriate contexts.

# Updates and living guidelines approach
This guideline will be subject to a phased approach that treats them as living guidelines, supporting the review of new evidence for specific questions on digital health interventions. This will ensure that new evidence is brought to the guideline development group (GDG) for review. The first planned update to the guideline will be to include the use of targeted client 5 communication for noncommunicable diseases. A virtual GDG will be convened for formulating recommendations based on the evidence tables prepared for this additional priority question. Associated recommendations will be included in version 1.1 of the guideline.
This guideline document recognizes the need to monitor the rapidly evolving nature of digital health, systematically through a continuous scanning and review of the literature and innovation pipelines. The first major update to the guideline is likely to be needed within 18 to 24 months of this initial dissemination, to accommodate new evidence for the existing recommendations and any emerging evidence related to other innovations in the WHO classifications.
WHO's Classification of digital health interventions v1.0 (13) provides the grounding for this livingguidelines approach, to help determine which additional interventions will need the deliberations of the GDG, and to help establish the questions for systematic review and the subsequent synthesis of evidence and development of recommendations. Scans of the evolving evidence base and collaboration with WHO's Innovations Hub (92) will also assist WHO in its vigilance to identify any emerging digital innovations that may warrant review by the GDG but were not reflected in the original classification scheme.
This guideline recognizes that the innovative approach of a living guideline is critical for ensuring Member States stay informed in the rapidly evolving field of digital health. WHO will continue to work closely with the Secretariat of the WHO Guidelines Review Committee (34) to ensure that this process adds value and is tested and refined.
Client An individual who is a potential or current user of health services; may also be referred to as patient or non-patient who uses health information and services. (Although WHO's Classification of digital health interventions v1.0 uses the term "client" (13), the terms "individual" and "patient" may be used interchangeably, where appropriate.)
eHealth The use of information and communications technology (ICT) in support of health and healthrelated fields, including health care services, health surveillance, health literature, and health education, knowledge and research. mHealth is a component of eHealth (1).

# Enabling environment
Attitudes, actions, policies and practices that stimulate and support the effective and efficient functioning of organizations, individuals and programmes or projects. The enabling environment includes legal, regulatory and policy frameworks, and political, sociocultural, institutional and economic factors.
Digital health An overarching term that comprises eHealth (which includes mHealth), and emerging areas, such as the use of computing sciences in the fields of artificial intelligence, big data and genomics (3,4).

# Digital health architecture
An overview or blueprint used to design and describe how different digital applications (software and ICT systems) and other core functionalities will interact with each other within a given context (25).

# Digital health application
The software, ICT systems, and communication channels used in the health sector, such as a software being used for health management information systems (HMIS) or an interactive messaging application ("app") (25).

# Digital health intervention
A discrete function of a digital technology to achieve health sector objectives. The WHO Classification of digital health interventions v1.0 provides an overview of the range of digital health interventions identified in the literature and implementation practices (13). Table 2.1 lists definitions of the specific digital health interventions included in this guideline.

# Digital health ecosystem
The combined set of digital health components representing the enabling environment, foundational architecture and ICT capabilities available in a given context or country.

# Evidenceto-decision framework
A framework to assist people making and using evidence-informed recommendations and decisions. Their main purpose is to help decision-makers use evidence in a systematic and transparent way. When used in a WHO guidelines context, evidence-to-decision frameworks inform guideline development group (GDG) members about the comparative pros and cons of the interventions being considered, ensure that GDG members consider all the important criteria for making a decision, provide GDG members with a concise summary of the best available evidence about each criterion to inform their judgments, help help the GDG members to structure and document their discussions and to identify any reasons for disagreement, making the process and the basis for their decisions transparent.
The ability of multiple ICT systems and software applications to communicate with one another, exchange data and use the information that has been exchanged. mHealth The use of mobile and wireless technologies to support health objectives (2,3).  A5.4 outlines the specific research gaps, with illustrative research questions, identified for each of the interventions included in the guideline. These research questions should be addressed using rigorous methods.  What strategies can be used to ensure equal access to and use of targeted client communication services for all groups, including people with poor access to mobile devices and/or poor network coverage, people who speak minority languages and people with low literacy or poor technological literacy and skills?  Future research assessing the effectiveness of targeted client communication using mobile devices should make efforts to ensure that disadvantaged populations are included. Trials should avoid excluding, wherever possible, participants on the basis of mobile device ownership, literacy levels, language or participation in formal health care programmes.
Other  Where possible, research should take an integrated approach that includes outcomes across the continuum of care in pregnancy, childbirth and child health, as well as across sexual and reproductive health in general.