The mention of specific companies or of certain manufacturers' products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters.
All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use.
Editing: Green Ink, United Kingdom Design and layout: RRD Design LLC

# Evidence and recommendations
This guideline provides ten evidence-based recommendations on the digital health interventions that were prioritized during the scoping process (see sections 2.1 and 2.2). These recommendations are made with the expectation that their implementation is grounded in an understanding of the ecosystem readiness and maturity, as outlined in Chapter 4. For each of the digital health interventions reviewed in this guideline, this chapter elaborates on the following components:  background information on the specific digital health intervention  an overview of the specific evidence  the recommendation along with a justification and remarks  specific implementation considerations.
Overall gaps in the evidence are described in Chapter 5; specific gaps and research questions for each of the interventions is detailed in Annex 5. In addition, Web Supplement 1 contains the evidence-to-decision frameworks and elaborates on the specific findings for each intervention as it relates to its effectiveness, acceptability, feasibility, resource use, and gender, equity and human rights concerns. The Web Supplements cited here are available at www.who.int/reproductivehealth/publications/digital-interventions-health-system-strengthening/en/

# Cross-cutting acceptability and feasibility findings
Most of the digital health interventions in this guideline are targeted at or expected to be used by health workers. The following findings point to factors that influence the acceptability and feasibility of digital interventions used by health workers. These findings are based on qualitative Although the systematic reviews included accessibility via mobile devices to ensure that these digital interventions are applicable in low resource settings where extensive computerized systems may not be available, it does not preclude the recommended interventions from being used on non-mobile digital devices, such as desktop computers.
evidence syntheses and overviews of digital health interventions for health workers in primary care (Web Supplement 2A); mLearning (Web Supplement 2B) stock notification and tracking commodities (Web Supplement 2D), and birth and death notification (Web Supplement 2E).

# Acceptability for health workers
Factors that may increase acceptability Digital health interventions allow health workers to expand their range of tasks as well as take on tasks previously assigned to higher-level workers. This can be experienced as satisfying and fulfilling, both for those to whom tasks are shifted, as well as to those from whom tasks are shifted (moderate confidence, Web Supplement 2A). Health workers working in rural and remote contexts particularly appreciate the efficiency of digital health technologies as these allow them to offer services through the device (moderate confidence, Web Supplement 2A). Health workers are likely to perceive digital health technologies to be more efficient because of the increased speed with which they allow them to work (moderate confidence, Web Supplement 2A). These technologies are also likely to save travelling time for health workers in both urban and rural settings, allowing them to spend more time with their clients 1 in urban areas or to provide services remotely to clients in rural areas (moderate confidence, Web Supplement 2A). Health workers may appreciate the portability of digital health technologies because this allows them to be flexible, to work when convenient, and not have to be office-bound to access information (low confidence, Web Supplement 2A). Health workers, particularly lay health workers in low-and middle-income settings, also perceive digital health technologies as allowing them to better coordinate the delivery of care through connecting them to other people and sectors in the health system and to clients and communities (moderate confidence, Web Supplement 2A).
Some health workers also report that digital health technologies raise their social status and increase the trust and respect they receive in communities. This is in part due to the device itself but is also because they use these devices to access health workers at higher levels of care. Community health workers, feel that the devices increase the respect they receive from health professionals and from the community (moderate confidence, Web Supplements 2A and 2E). Similar findings are seen among health workers in training, although there is also some concern that clients/patients and colleagues might regard their use of mobile devices as unprofessional because of their association with recreation (low confidence, Web Supplement 2B.

# Factors that may decrease acceptability
Some health workers do not experience digital health interventions as efficient as these interventions do not reduce their workload and in some cases increase their workload (moderate confidence, Web Supplement 2A), making them less likely to accept these interventions (moderate confidence, Web Supplement 2F). Health workers may perceive digital health interventions as increasing their workload when it means maintaining two systems (i.e. digital and paper-based), when there are staff shortages, when the addition of the digital health intervention 1 Although WHO's Classification of digital health interventions v1.0 uses the term "client" (13), the terms "individual" and "patient" may be used interchangeably, where appropriate.
to current work is not understood and appreciated by supervisors, or when they themselves perceive the intervention as peripheral to their work. While some health workers do not object to the additional work, others expect to be remunerated for it (low confidence, Web Supplements 2A and 2E).
Health workers may also be concerned about loss, damage and theft and may complain about having to carry both a personal and a work phone (low confidence, Web Supplements 2A and 2B).
In some settings, health workers use their personal mobile phones and Internet access for work purposes, although this use is not necessarily formalised and health worker expenses are not always covered (low confidence, Web Supplements 2A and 2E). This can include expenses for air time or for charging their phone. Health workers may see these personal costs as a burden. However, they may feel a moral imperative to assist their clients by using their own phones despite the personal costs this incurs (low confidence, Web Supplement 2A).
Health workers' perceptions and experiences of digital health interventions are likely to be shaped by their pre-existing digital literacy. Health workers who manage well have positive views about the use of mobile devices. However, health workers who struggle to use these technologies have negative perceptions about its usefulness, may not understand the information generated by these technologies, and are also anxious about making errors. In some instances, poor digital literacy threatens job security (high confidence, Web Supplement 2A). However, even technologically more competent users are reported as needing support and repeat training in the use of the programmes and devices (low confidence, Web Supplement 2B).

# Feasibility for health worker
Many health workers, particularly in rural and remote areas, experience logistical challenges when using digital health technologies, including poor network connectivity and access to electricity to charge their mobile phones (high confidence, Web Supplements 2A, 2B, 2D, 2E and 2F). In some instances, poor connectivity also results in client dissatisfaction because it creates delays in receiving health services (high confidence, Web Supplement 2A).
Health workers want easy-to-use, reliable equipment and ongoing technical support (high confidence, Web Supplements 2A, 2D and 2F). They also feel that the use of these technologies can be expanded to a wider range of settings, services, and illnesses (high confidence, Web Supplement 2A). However, health workers often report usability issues, and poor integration with other digital systems (high confidence, Web Supplements 2C and 2F). Although the introduction of digital health interventions into existing healthcare systems may be important, this requires many changes and may be difficult to achieve (low confidence, Web Supplement 2F). For instance, institutional support and local champions may be considered important for ensuring integration into existing systems, but staff reorganization and the breakdown of existing partnerships may undermine this support (low confidence, Web Supplement 2F).
Health workers may experience a number of problems with the design of the programmes or of the device itself, including programmes in languages they are not proficient in, inaccurate rendering of the local language font, small screens, devices being ill-suited for note-taking, and SMS character limitations (low confidence, Web Supplement 2A and 2B). Although the involvement of staff and clients in the planning, design and implementation of the digital system is considered important by health workers (moderate confidence, Web Supplements 2A and 2D), this is not always done (moderate confidence, Web Supplement 2F). Health workers may be dissatisfied with digital health when technology changes are too rapidly introduced, or when their expectations of the technologies are not met (low confidence, Web Supplement 2A).
Some stakeholders are also concerned about the confidentiality of medical information and data security (moderate confidence, Web Supplement 2F). Health workers may try to protect clients' confidential information when using digital health devices, in particular when the information concerns stigmatised conditions such as HIV/AIDS (low confidence, Web Supplement 2A). Achieving informed consent for sharing records and images can also be challenging, particularly in settings with low levels of basic literacy or digital literacy (moderate confidence, Web Supplement 2F).
Training is important for staff acceptance and system use (high confidence, Web Supplements 2A, 2B, 2D, 2E and 2F). While some health workers experience difficulties in understanding and using digital health technologies, health workers and trainers feel that training and familiarity with these technologies can help overcome these difficulties. Some health workers feel hampered in learning to use mobile health technologies if it is not also used by their clinical mentors (moderate confidence, Web Supplement 2A). This may be particularly important as health workers requiring technical support may receive this support from higher level staff or from peers (low confidence, Web Supplement 2A). Supportive supervision is also considered important for staff acceptance and system use (moderate confidence, Web Supplement 2D).
Digital systems can make it possible to track and monitor health workers' activities. Health workers may feel that this changes how they work and may make their work more visible. Some health workers may perceive this as positive, but it may leave other health workers with the sense of "big brother watching". Supervisors may feel that this allows them to be more aware of the work of lower level health workers and to address problems (low confidence, Web Supplements 2A and 2D).
Even where challenges tied to the design and usability of digital systems and devices are addressed, these systems may not be able mitigate a number of broader health systems challenges, for example, an underlying lack of medical commodities (low confidence, Web Supplement 2D).

# Acceptability and feasibility for clients/individuals
The following findings point to factors that are likely to influence the acceptability and feasibility of digital health interventions targeted at or expected to be used by clients/patients. These findings are summarized based on overviews and qualitative evidence syntheses related targeted client communication (Web Supplement 2C) and telemedicine (Web Supplement 2F). More detailed descriptions on the acceptability and feasibility findings are available within the sections focused on the specific interventions.
Some individuals describe targeted communication and telemedicine services in positive terms.
For instance, some clients appreciate the fact that someone is taking the time to send them messages as this can make them feel like someone is interested in their situation and invested in their well-being. These clients describe the messages as providing support, guidance and information, and giving a sense of direction, reassurance and motivation (moderate confidence, Web Supplement 2C). Similarly, some clients using telemedicine services see these as offering reassurance and a sense of safety and appreciate the increased access and the consistency and continuity of care that it can offer (low confidence, Web Supplement 2F). Some clients also feel that telemedicine services have increased their independence and self-care (low confidence, Web Supplement 2F).
However, individuals who are dealing with health conditions that are often stigmatised or very personal (e.g. HIV, family planning and abortion care) worry that their confidential health information will be disclosed or their identity traced due to their participation in targeted communication programmes (high confidence, Web Supplement 2C). Some individuals using telemedicine services prefer face-to-face contact (low confidence, Web Supplement 2F).
Additionally, individuals believe there should be little or no charge tied to digital health programmes, such as joining the programme, downloading apps, or charges related to sending and receiving SMS/phone calls (high confidence, Web Supplement 2C).
Targeted communication and telemedicine services can potentially increase access for some groups of individuals. For instance, telemedicine services can give individuals who speak minority languages access to health workers who speak this language (high confidence, Web Supplement 2F); and may save money and reduce the burden of travel for clients with caring or work responsibilities, living far from health care facilities or with few funds (low confidence, Web Supplements 2C and 2F).
However, access to and use of these services can be particularly difficult for some individuals. These include individuals with poor access to network services, electricity (high confidence, Web Supplement 2C) or mobile devices (moderate confidence, Web Supplements 2A and 2C); clients who speak minority languages, have low literacy or digital literacy skills (moderate confidence, Web Supplement 2C) or hearing impairments (high confidence, Web Supplement 2A). Clients with stigmatized health conditions may also be particularly concerned about the privacy of their information (high confidence, Web Supplement 2C).

# Accountability coverage
The proportion of those in the target population registered into the health system

# Accountability coverage: birth and death notification Background
A global scale-up plan for strengthening civil registration and vital statistics (CRVS) systems has been developed by the World Bank and WHO with the goal of achieving "universal civil registration of births, deaths and other vital events, including reporting cause of death, and access to legal proof of registration for all individuals by 2030" (57). A key component of this plan is to prioritize and strengthen the linkages between CRVS systems and health (57)(58)(59). This includes the use of digital information systems to strengthen CRVS systems and expanding the coverage of registration services among underserved populations, such as people residing in rural areas (57)(58)(59)(60).
In these respects, the global proliferation of mobile phones and cellular network connectivity (41) is increasingly being leveraged, especially in resource-limited settings, to drive the development and use of digital civil registration systems (11,12,(60)(61)(62)(63).
Notification is the capture and onward transmission of minimum essential information on the fact of birth or death has occurred, and represents the first step in the process leading to eventual registration and certification of the vital event. Increasing the efficiency of birth and death notification as well as promoting linkages between the health and civil registry sectors (many births are first known in the health sector) can strengthen civil registration processes and the use of health services (61,62). Digital mechanisms to facilitate notifications may enhance these linkages as well as catalysing civil registration. Furthermore, added to their ability to conduct notifications, the increased access to mobile devices among community-based individuals such as vaccination programme workers, community health workers and village elders can potentially expand the coverage of civil registration systems to underserved rural and remote regions (60-63).
For birth notifications, other information related to the birth may be transmitted via mobile phones in the form of phone calls, inputs to an interactive voice response or unstructured supplementary service data (USSD) system, SMS text messages, messages from mobile devicebased applications (apps) or calls or messages to publicly known short codes or access numbers. The content of the birth notification may vary by country or implementation, but may include the name of the child born, the name and address of the parents, the place and date of birth, and details of birth outcomes.

# Lack of population denominator
Lack of access to information or data Delayed reporting of events

# Illustrative health system challenges
Similarly, for death notifications, information related to the death may be transmitted via mobile phone calls, inputs to an interactive voice response or USSD system, SMS text messages, messages from apps, or calls or messages to publicly known short codes or access numbers. The content of the death notification may vary by country or implementation, but may include the name of the deceased, the name and address of a relative, the place and date of death, and details of the cause of death.
This guideline question reviewed the added value of the notification of birth and death events via mobile devices as an additional channel for supporting the establishment of a CRVS system and strengthening linkages to it.

# Overview of the evidence
The following is a summary of the evidence on birth and death notification accessible via mobile devices. Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.
 Births: There is limited evidence on the effectiveness of using mobile devices for birth notification as the certainty of this evidence was assessed as very low.
 Deaths: No evidence on effectiveness was identified for death notification via mobile devices.
The qualitative evidence suggests the intervention is probably acceptable to health workers and enables them to be more proactive in identifying pregnancies and coordinating emergency services. They report earning more trust and respect from their communities due to the ability to communicate with and coordinate emergency services. Conversely, acceptability for clients of birth notification may be reduced by sociocultural norms, such as the extent to which stillbirths, births to unmarried mothers or maternal deaths are acknowledged in communities. The evidence also points to the potential costs of notification as a barrier and to the need to demonstrate the advantages of birth or death notification to communities.
The qualitative evidence highlights several feasibility issues including, the need for adequate local staffing and for strong underlying health and civil registration system infrastructure, resources and processes. Health workers' competing priorities and lack of adequate incentives may affect the successful adoption of these strategies. Inadequate attention is sometimes given to legal frameworks governing civil registration, and governments may need to modify these frameworks to allow new types of health care cadre and other key informants to notify births and deaths. Strong underlying health and civil registration system infrastructure, resources and processes are necessary to achieve the impact of using mobile devices for birth and death notification.

# Resource use
No evidence on resource use was identified. Resource use considerations are listed within the evidence-to-decision framework in Web Supplement 1.

# Gender, equity and human rights
The qualitative evidence indicates that while birth and death notification via mobile devices can help to reach under-registered populations, there may be inequities in the implementation of this intervention that are related to the availability of supportive infrastructure (network connectivity, for example), literacy in the use of information and communications technologies (ICT), and available funding resources.

# Recommendation and justification/remarks

# Birth notification
(Recommended only in specific contexts or conditions) recommendation 1
WHO recommends the use of digital birth notification under these conditions:
 in settings where the notifications provide individual-level data to the health system and/or a civil registration and vital statistics (CRVS) system, and  the health system and/or CRVS system has the capacity to respond to the notifications.
Responses by the health system include the capacity to accept the notifications and trigger appropriate health and social services, such as initiating of postnatal services.
Responses by the CRVS system include the capacity to accept the notifications and to validate the information, in order to trigger the subsequent process of birth registration and certification.

# Death notification
(Recommended only in the context of rigorous research and in specific contexts or conditions) recommendation 2

# WHO recommends the use of digital death notification under these conditions:
 in the context of rigorous research, and  in settings where the notifications provide individual-level data to the health system and/or a CRVS system, and  the health system and/or CRVS system has the capacity to respond to the notifications.
Responses by the health system include the capacity to accept the notifications and trigger appropriate health and social services.
Responses by the CRVS system include the capacity to accept the notifications and to validate the information, in order to trigger the subsequent process of death registration and certification.
Birth notification  The guideline development group (GDG) acknowledged the limited evidence but emphasized that birth notification represents a vital first step in a care cascade that can ultimately lead to increased and timely access to health services and other social services. The GDG also believed that the use of mobile devices to perform this task was likely to provide a more expedient means of effecting the notification and subsequent health services.
 GDG members noted that while birth notification should not be viewed as a substitute for legal birth registration, it could provide an opportunity to accelerate the registration by linking birth notifications to national civil registration systems. The GDG also recognized that digital notification of births could facilitate providing newborns with legal identity and future access to health and other social services.

# Death notification
 The GDG remarked that a lack of information on deaths, especially deaths outside of facilities, exacerbates data gaps in understanding the rates and causes of mortality.
 The GDG therefore decided, while noting the limited evidence, to recommend death notification via mobile devices in the context of rigorous research and where notifications can be linked to health and/or CRVS systems.
 The GDG noted that while data on deaths and causes of death are very useful for health planning, they expressed concerns about adding the responsibility of CRVS-related functions to already poorly resourced, understaffed and overburdened primary care health systems.
 The GDG also recognized the sociocultural sensitives of communities notifying about deaths through digital devices and recommended that further research be taken to understand these considerations.

# Remarks that apply to both birth and death notification
 It should also be noted that increases in the notification of births and deaths would also require that civil registration services have, in turn, the capacity to manage a higher demand for registration and certification services.
 The ability for the health system and/or CRVS system to respond and act appropriately on the birth and death notification was seen as a critical component for successful implementation. If such linkages are not in place, the notification of birth and death events would not add any value and would incur an additional cost to the system.

# Implementation considerations
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.

# Legislation, policy and compliance
 The implementation of birth and death notifications needs to be in the context of national policies, laws and guidelines. This may require modifications of legal frameworks to include mobile notification in established practice and to enable cadres of informants such as community health workers and community leaders to conduct the notifications if current policies do not already provide for this.
 Consider whether changes to legal frameworks will be needed to allow birth and death notification to occur via mobile device or be carried out by new groups of health workers or other cadres, as mentioned above, and how this would be linked to the issuance of birth/death certificates. For example, consider whether there will need to be any modifications to existing processes to accommodate signatures and approvals currently conducted on paper-based forms. This review and modification should take place in the context of a broader legal review of CRVS-related laws and regulations and would require collaboration among the institutions that cover the health sector, civil registration sector and the local governments.
 Consider the specific data storage, privacy and confidentiality issues. Implementers should understand, for example, the implications and necessary regulations if the database of notified births and deaths is also being held by mobile network operators, and the potential for commercial uses of the data. Additionally, a relevant authority needs to ensure the right to data protection by monitoring and enforcing a set of data protection laws.

# Services and applications
 Consider establishing mechanisms to prevent duplicate notifications. Unique identification can be used to address this (for example, by issuing national identities; possibly identification of the parents). Where national IDs are not available, consider an interim measure of IDs being provided by health facilities, drawing from codes in master facility lists. Implementers may also want to consider local de-duplication processes, such as using routine coordination meetings across health workers to de-duplicate birth/death notifications before they are transmitted to the civil registrar.
Workforce  When developing birth and death notification systems, consider mechanisms to ensure the completeness of the data, and whether demand-generation activities are needed to incentivize reporting by explaining its benefits. Implementers should be aware, however, of any reporting targets placed on health workers, and ensure birth and death data are validated before being released to the civil registration system.
 Consider how best to ensure the quality and timeliness of birth and death data, for instance by checking on low performers identified through digital performance data or spot checks.
Other ways to help improve data quality include standardizing the definitions associated with reporting birth and death events, such as for stillbirths, and making these definitions accessible to those inputting the data.
 Implementers should note that increases in notification would in turn require that the health system and civil registration services were prepared to absorb higher demand for registration. This is a potential bottleneck in the registration and validation process and could deter populations from continuing notifications.
 Consider how to improve accessibility and shorten the connection between the health workers or communities providing the notifications and the CRVS sector undertaking the registration. Consider, for instance, increasing the number and proximity of registration service points, and look at the use of digital systems to speed up the registration process at these points.

# Considerations for equity, gender and human rights
 Explore sociocultural barriers associated with communicating about births/deaths and address the way these dynamics will influence notifications via digital devices.
 Consider linking birth notification to health services that have high coverage, such as immunization services or health facilities that offer very high rates of institutional delivery. It is important, however, to consider whether an increase in notifications can be absorbed by the civil registration system.

# Availability of commodities: stock notification and commodity Background
The availability of health commodities at point of services is critical to strengthening the quality of care and supporting the pillars of universal health coverage (UHC) (64). Health commodities include health products, and health and medical supplies that may be needed for the provision of health services, including medicines, vaccines, medical supplies such as contraceptives dressings, needles and syringes, and laboratory/diagnostic consumables (65,66). Various high-level initiatives, including the UN Commission on Life-Saving Commodities for Women's and Children's Health, have advocated equitable access to life-saving medicines and other health commodities (67,68). Stock-outs of critical medical commodities remain an issue, however, particularly in rural settings, where infrastructural limitations and geographical barriers can obstruct access to commodities at the point of care.
The rapid global expansion of mobile devices has emerged as providing a potential opportunity for mitigating the challenges of commodity distribution and stock-outs. Approaches can include the use of communication systems such as text messaging (SMS) and data dashboards to manage and report on supply levels. Specific examples by which mobile tools may be used to improve supplychain management include to track inventories of health commodities, notify their stock levels, forecast demand for commodities, monitor cold chain-sensitive commodities, and manage the distribution of health commodities (13).
Although broader initiatives to strengthen logistics management information systems are ongoing (69), this guideline question reviewed the added value of extending the systems via mobile devices to address commodity management at primary health care levels.

# Availability of commodities and equipment

# Ensuring availability of commodities and equipment
Insufficient supply of commodities at point of service

# Lack of transparency in commodity transactions
Lack of access to information or data

# Illustrative health system challenges

# Overview of the evidence
The following is a summary of the evidence on stock notification and commodity management accessible via mobile devices. Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.
There is limited available evidence on the effectiveness of and resources required as the certainty of the evidence was assessed as very low.
The qualitative evidence suggests that access to digital data on stock availability at all levels of the health system may be useful by health system managers as it allows them to respond to anticipated stock shortages and ensure ongoing supply of needed health commodities. Staff at the subnational levels may be concerned, however, about the data at their level becoming available simultaneously with those at the national level since this would take away their opportunity to contextualize the data or to explain shortcomings in stock availability.
Barriers to optimal implementation of stock notification and commodity management via mobile devices include an underlying lack of stock at national or district level and a mismatch between national ordering routines and local needs. The qualitative evidence on the feasibility of digital health interventions, more broadly, also highlights challenges including those of network connectivity, access to electricity, usability of the device, sustaining training and support to health workers using the digital tools, and system integration.

# Resource use
No evidence on resource use was identified. Resource use considerations are listed within the evidence-to-decision framework in Web Supplement 1.

# Gender, equity and human rights
The qualitative evidence on gender, equity and human rights concerning digital health interventions suggests health workers based in peripheral facilities and rural communities may find these interventions helpful in overcoming geographical barriers and linking to the broader health system, including when communicating about stock levels. Health workers in these settings may be more likely to experience poor network coverage and access to electricity, though, and may have lower levels of training and literacy in the use of technologies and fewer resources, including limited access to the mobile devices that may be needed.

# Recommendation and justification/remarks stock notification and commodity management
(Recommended only in specific contexts or conditions)
WHO recommends the use of digital stock notification and commodity management in settings where supply chain management systems have the capacity to respond in a timely and appropriate manner to the notifications.
 Despite the limited evidence on effectiveness and the identified feasibility barriers, the guideline development group (GDG) felt that the use of mobile devices was likely to provide a more expedient means of effecting stock notifications and ensuring the subsequent availability of commodities at the point of services. This, in turn, may increase the ability of health services to manage health issues in a timely and appropriate way.
 The GDG also assessed stock notification via mobile devices to be a relatively low-risk intervention with potentially high impact, including the potential to save resources through an improved allocation of commodities and reduced wastage. The GDG further believed that the availability of timely stock data would increase transparency and promote accountability.
 Addressing the identified barriers to implementation as well as ensuring responsiveness to the stock notifications were seen as critical ways to build trust and drive the effective use of the digital intervention. If there are no mechanisms for health managers to respond to the incoming data, or a lack of infrastructure or financial resources to purchase new commodities, the gathering of stock data and issuance of notifications would not add any value and would incur an additional cost to the system.
 Although the condition within this recommendation requires that the health system be responsive to the stock notifications, the GDG also remarked the importance of building the capacity of weaker health systems so that this intervention may be used effectively.

# Linkage with other WHO recommendations
This discussion aligns with recommendation 15 of the WHO guideline on health policy and system support to optimize community health worker programmes, which recommends the use of mobile health technology to support supply chain functions, including adequate reporting, to enhance the availability of health commodities (17).

# Implementation considerations
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.

# Legislation, policy and compliance
 Ensure there is no harm or reprisal to health workers for reporting stock-outs or wastage; instead, the emphasis should be on explaining the benefits of reporting stock-outs so that they can be addressed. To motivate continued reporting, ensure that some action is possible when stock-outs are reported.

# Standards and interoperability
 Prioritize integrating notifications with existing data reporting systems, including national or subnational information management systems where available, such as supply chain, logistics and warehouse management information systems. Consider integrating the stock notification system with a data dashboard that displays the notification, receipt of commodity at the station and action taken among other data for ensuring transparency.
 Consider the need for training at all levels of the health care system, including the training of health workers to send stock reports, of support staff such as cold-chain technicians to manage stock and of facility workers to assess stock levels. Training should be reinforced by the basic processes of inventory management and stock distribution. Since the management staff at national and subnational levels make decisions on whether or not, according to the data, to supply health facilities and health workers with stock replenishments, the introduction of the digital system should also be accompanied with refresher training on the basic processes of supply chain management. Training should include the use of the technology, such as the use of text messages for the notification and the use of data dashboards.

# Services and applications
 When designing digital systems for stock notification, consider how the system can be made easy to use, with effective display of the data through fact sheets and simple graphical and tabular illustrations.
 Ensure that the digital systems and ordering routines are flexible enough to respond to local needs. For instance, where systems deal with quarterly stock orders, ensure they can also accommodate unexpected or seasonal needs.

# Availability of human resources
Ensuring availability of human resources

# Accessibility of health facilities
Ensuring access to health facilities 1.4 Accessibility of health facilities and human resources for health:

# client-to-provider telemedicine Background
Despite progress in addressing health workforce shortages, challenges in the equitable access to health workers serves as a major hindrance to achieving the full requirements of effective coverage of human resources for health (70). Geographical inaccessibility and the preference of health workers for working in urban environments are among some of the well-documented reasons for imbalances in the distribution of health workers (71). While there is a wide range of ongoing efforts to reduce inequities in access to health workers, including incentives and alternative approaches to training, digital approaches such as telemedicine have also been explored as a mechanism of making health services available to underserved communities (71).
Within the WHO/ITU National eHealth strategy toolkit, telemedicine is defined as supporting "the provision of health care services at a distance" (18). Although other definitions elaborate on telemedicine as the use of ICT for medical diagnostic, monitoring and therapeutic purposes at a distance (72-75), the driving principle is centered on the provision of remote clinical support as a means of overcoming geographical barriers (72). Telemedicine can function between clients and health workers who are separated by distance, as well as among health workers based in different locations. The type of exchange between these actors varies and may include remote consultations, remote monitoring of vital signs or diagnostic data, and the transmission of medical files such as images for review, commonly referred to as "store and forward" (72-75).
In 2010, WHO reported extensively on the global status of telemedicine, including factors affecting its uptake in low-and middle-income settings (72). In more recent years, the emergence of mobile technologies has shifted the landscape, triggering new considerations for connecting clients and health workers (3). This guideline question builds on this preceding resource from WHO and examined the evolved use of telemedicine via mobile devices between clients and health workers.

# Geographic inaccessibility
Insufficient supply of qualified health workers Delayed provision of care

# Illustrative health system challenges

# Overview of the evidence
The following is a summary of the evidence on client-to-provider telemedicine. Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.
The evidence on effectiveness suggests that this intervention may improve some outcomes, such as fewer unnecessary clinical visits, reduced mortality among individuals with heartrelated conditions, exclusive breastfeeding, and increase health-related quality of life assessed 1-6 months after the intervention. However, it may make little or no difference on other outcomes, such as hospital admissions for heart-related conditions or older individuals receiving home-based care.
The qualitative evidence suggests that health workers appreciate the ability to offer immediate care, to follow up on missing clients and offer informed care, advice and emotional support to clients, even when physical contact is not possible. However, health workers feel that some cases still warrant face-to-face contact and are also concerned that loss of face-to-face communication will change the health worker-client relationship and lead to poorer quality care. Health workers may also be concerned about having to work beyond their clinical capacity and about potential issues of clinical liability.
From the client's perspective, the qualitative evidence suggests these individuals may appreciate being able to communicate with health workers from their homes and see telemedicine services as offering reassurance and increased access and the consistency and continuity of care that it can offer. Some clients may also feel that telemedicine services have increased their independence and self-care, although some health workers may be concerned about clients' ability to manage their own conditions.
The qualitative evidence on the feasibility of digital health interventions, in general, highlighted challenges related to network connectivity, access to electricity, usability of the device, sustaining training and support to health workers using the digital tools, concerns about data privacy and obtaining informed consent.

# Resource use
The evidence on resource use was assessed as having very low certainty. Resource use considerations are listed within the evidence-to-decision framework in Web Supplement 1.

# Gender, equity and human rights
This intervention may positively impact on equity by facilitating access to health services, particularly for individuals who speak minority languages. It also may reduce the burden of travel, particularly for people with caring or work responsibilities and those living far from health facilities. However, access to telemedicine services may be difficult for other groups, though, including people with hearing impairments or poor digital literacy.

# Recommendation and justification/remarks
Client-to-provider telemedicine (Recommended only in specific contexts or conditions)

# WHO recommends client-to-provider telemedicine:
 under the condition that it complements, rather than replaces, face-to-face delivery of health services; and  in settings where patient safety, privacy, traceability, accountability and security can be monitored.
In this context, monitoring includes the establishment of standard operating procedures that describe protocols for ensuring patient consent, data protection and storage, and verifying health worker licenses and credentials.
The guideline development group (GDG) felt that despite the mixed available evidence on effectiveness spanning a wide range of health conditions, client-to-provider telemedicine has the potential to expand access to health services. It may also potentially reduce the burden of travel and decrease inequities for populations that have difficulties in accessing health services through conventional approaches.
 This recommendation recognizes that while telemedicine may enhance access to health services, it should not be used to replace or detract from efforts to strengthen the health workforce.
 The establishment of standard operating procedures and mechanisms to ensure patient safety, privacy, traceability and accountability of services was deemed to be a necessary condition to mitigate the potential risks and harms of implementing this recommendation.

# Implementation considerations
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.

# Legislation, policy and compliance
 Clarify the legal framework for the implementation of telemedicine, including relating to the licensing and regulation of telemedicine health workers. The legal framework for remote consultation should also consider cross-border consultations in which the health worker is based in another country or jurisdiction.
 Clarify clinical protocols to explain what can and cannot be done in the remote consultation.
For example, determine what type of cases still warrant face-to-face contact. Consider whether it is possible or desirable for clients to meet health workers in person before connections are made over digital services.
 Explore whether changes in regulations are necessary to support any changes needed to health workers' scopes of practice. Develop policies and protocols to clarify the liability issues of health workers using telemedicine systems.
 Explore reimbursement models and mechanisms of integrating client-to-provider telemedicine within existing service delivery models.
 Ensure that there are mechanisms for documenting and tracing past exchanges and decisions made during consultations.
 Ensure that use of the technology does not impact negatively on the relationship between client and health worker, particularly when users are learning about the technology and how to operate the devices. Extensive training on the technology and operating the device should be done before introducing the system for use directly with clients.
 Ensure that health workers remain able to use their own skills, judgement and knowledge within the changed context.
 Develop guidelines in collaboration with health workers that protect them from clients contacting them outside of normal working hours, such as in the context of emergencies or other considerations. If this contact is encouraged or expected, how can it best be managed to avoid overwhelming the health worker? Will health workers be compensated for this type of client support?
 Involve the relevant professional bodies as well as the health workers and clients in the planning, design and implementation of the telemedicine programme to ensure that their needs and concerns are met, such as to educate health workers on the legal frameworks governing telemedical exchanges.

# Considerations for equity, gender and human rights
 Pay special attention to the needs, preferences and circumstances of particularly disadvantaged or hard-to-reach groups, including people with low literacy or few digital literacy skills, people with limited control over or access to mobile devices, people speaking minority languages, migrant populations in new settings, and people with disabilities such as sight or hearing impairment.
 Consider how services can be made available to people with disabilities such as sight or hearing impairments, with poor access to electricity or poor network coverage, who cannot afford mobile devices or charges to use them, and people who have limited autonomy, for example because their access to devices is controlled by another person. Strategies to increase access to telemedicine in these cases may be provided through public kiosks or outreach through community health workers, as examples.
 Consider using telemedicine to link clients who speak minority languages to health workers who also speak the language.

# Availability of human resources

# Ensuring availability of human resources

# Accessibility of health facilities
Ensuring access to health facilities 1.6 Accessibility of health facilities and human resources for health:

# provider-to-provider telemedicine Background
Access to qualified health workers with the appropriate competencies, skills and behaviours is an even greater obstacle to improving health outcomes than the availability of health workers (70,71). Geographical inaccessibility and the unequal distribution of health workers also contribute to limitations in the effective coverage of human resources for health (62). Digital approaches, most notably telemedicine between different types of health workers, have emerged as a potential way to overcome the barriers of long distances to qualified health workers and shortages in their numbers.

# Geographic inaccessibility
Insufficient supply/coverage of qualified health workers Insufficient health worker competence

# Illustrative health system challenges
Provider-to-provider telemedicine, as with client-to-provider telemedicine, facilitates the provision of health services at a distance and is primarily used to link less skilled health workers with more specialist ones (72). The communication between health workers may be made for a variety of reasons, including to get assistance with diagnoses, to remotely monitor clients' health status through vital signs and to conduct case-management consultations. This communication between health workers may occur asynchronously through the exchange of video and image files to be reviewed later (also referred to as store-and-forward exchanges) or synchronously in realtime exchanges (13,18,(72)(73)(74)(75).
Although telemedicine is one of the more established forms of ICT-enabled health service delivery (72), this guideline question expands on the existing evidence base, particularly in light of the advances in facilitating health workers' exchanges via mobile devices.

# Overview of the evidence
The following is a summary of the evidence on provider-to-provider telemedicine. Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.
The evidence suggests that provider-to-provider telemedicine may improve health worker performance, reduce the time for clients to receive appropriate care or follow-up, and decrease length of stay among individuals visiting the emergency department. However, the intervention may make little or no difference to other health status and well-being outcomes such as clinical improvements in individuals.
The qualitative evidence suggests that health workers appreciate the opportunity to communicate with each other and reduce their professional isolation. In particular, lower-level health workers noted how telemedicine services allowed them to access advice from higher-level health workers, which they saw as enabling better quality of care and client satisfaction. While some health workers may perceive provider-to-provider telemedicine as supportive, others may note challenges in collaboration, and concerns about liability and loss of control during the provision of care.
The qualitative evidence on the feasibility of digital health interventions, in general, highlights challenges related to network connectivity, access to electricity, usability of the device, sustaining training and support to health workers using the digital tools, concerns about data privacy and obtaining informed consent.

# Resource use
The evidence on resource use was assessed as having very low certainty. Resource use considerations are listed within the evidence-to-decision framework in Web Supplement 1.

# Gender, equity and human rights
The qualitative evidence on provider-to-provider telemedicine suggests that this intervention may improve equity by enabling health workers to facilitate access to higher-level care on behalf of their clients. Yet poor access to the digital technology, or the personal expenses associated with its use, may exclude some health workers, and thereby their clients, from these services.

# Recommendation and justification/remarks
Provider-to-provider telemedicine (Recommended only in specific contexts or conditions)
WHO recommends provider-to-provider telemedicine in settings where patient safety, privacy, traceability, accountability and security can be monitored.
In this context, monitoring includes the establishment of standard operating procedures that describe protocols for ensuring patient consent, data protection and storage, and verifying health worker licenses and credentials.
 The guideline development group (GDG) noted that provider-to-provider telemedicine has the potential to improve access to quality care and to reduce the isolation of health workers working in remote settings.
 Although the cost of the telemedicine system may vary depending on the modality used (exchange of image files, voice calls, remote monitoring), the GDG felt that provider-toprovider telemedicine could support care delivery by peripheral health workers.
 Due to concerns about liability issues, the GDG suggested that standard operating procedures/protocols be established to ensure patient safety, privacy, traceability and accountability of services and to mitigate the potential harms of implementing provider-to-provider telemedicine.
 It was also noted that the nature of telemedicine is changing and that a wide range of delivery channels are being used across health workers to facilitate communication exchanges.

# Implementation considerations
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.

# Legislation, policy and compliance
 Explore whether changes to licensing and legislation are necessary to support any changes in health workers' scopes of practice. Clarify liability issues for health workers using telemedicine systems and determine what can and cannot be done during remote consultations; the approach should not be a substitute for the adequate training of health workers.
 Ensure a clear legal framework for the implementation of telemedicine, including the licensing and regulation of care health workers using it. Additional clarifications are also required in cases of cross-border telemedicine, in which consultations are occurring across different jurisdictions.
 Ensure that there are mechanisms for documenting and tracing past exchanges and decisions made during consultations.

# Interoperability and standards
 The use of telemedicine requires that the health worker can access the patient's relevant clinical history. Integration with digital health record systems that can be accessed by the health worker and in which the patient's identity can be verified may be considered as a way to facilitate continuity of care.
Workforce  Ensure that the distribution of roles and responsibilities between different health workers is clear, including through regulations and job descriptions.
 Explore whether changes to health worker salaries or incentives are needed to reflect any changes in scopes of practice.
 Build trust between professionals considering establishing links between facilities across institutions, for example through twinning programmes.

# Continuous coverage
The extent to which clients receive the full course of intervention required to be effective
Targeted client communication 2 -defined as the transmission of health content or information to a specific audience based on their health status or demographic profile (13) -represents an approach for engaging with individuals to increase their knowledge about health and healthseeking behaviours, about where to find or how to access services, or for helping to retain them within health services when follow-up is needed. This includes the transmission of health information to individuals about health promotion, for spreading awareness of services and behaviours, transmission of reminders about services or treatments to encourage adherence to recommended practice, and transmission of notifications about diagnostic results (13). Using registered phone numbers or other contact information, the delivery of health content to individuals can be via a range of digital channels, including text messaging, voice, interactive voice response, multimedia applications and games (apps on mobile devices), and social media.
Several WHO guidelines have explored the use of targeted client communication via mobile devices as a potential tool to improve medication adherence. Most notably, the 2016 Consolidated guidelines on the use of antiretroviral drugs include a recommendation on the use of text messaging as part of a package of interventions to support adherence to antiretroviral therapy (15). Similarly, the 2017 Guidelines for treatment of drug-susceptible tuberculosis and patient care also recommend the use of text messages and voice calls to support health education and treatment adherence (16).
Building on this previous work, this guideline question reviews the use of targeted client communication via mobile devices across a broader range of health topics and populations of interest for sexual, reproductive, maternal, newborn, child and adolescent health (SRMNCAH).
Note that the use of targeted client communication in the prevention and management of noncommunicable diseases will be examined in a subsequent version of this guideline.
2 Although WHO's Classification of digital health interventions v1.0 uses the term "client" (13), the terms "individual" and "patient" may be used interchangeably, where appropriate.

# Low demand for services

# Low adherence to treatments
Loss to follow up

# Lack of access to information
Illustrative health system challenges

# Overview of the evidence
The following is a summary of the evidence on targeted client communication via mobile devices.
Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.
The evidence on effectiveness suggests targeted client communication may have positive impacts on some behaviours and health outcomes, such as: oral contraception use by adolescents, modern contraception use by adults, adherence to antiretroviral medications, attendance of antenatal care appointments, taking iron and folate tablets during pregnancy, skilled birth attendance, receipt of childhood vaccinations, and attendance of HIV appointments among exposed children.
However, the evidence also indicates that targeted client communication may make little or no difference to other outcomes, such as: health status as assessed by CD4 count and adherence to prenatal antiretroviral medication.
The evidence on targeted client communication also suggests the intervention has some unintended negative consequences, such as women experiencing physical violence in the context of receiving targeted communications for sexual and reproductive health (SRH) services.
The certainty of the evidence was assessed as very low for some outcomes such as: adherence to antiretroviral medication and attendance for STI/HIV testing among adolescents, breast and cervical cancer screening; and women's attendance for neonatal appointments.
The qualitative evidence suggests that targeted client communication is generally acceptable to individuals, but that some population subgroups have concerns about the confidentiality of health information, particularly for sensitive health issues such as HIV infection and other aspects of SRH.
Some clients describe digital targeted client communication programmes as providing them with support and connectedness. The fact that someone is taking the time to send them messages can make clients feel like someone is interested in their situation, invested in their well-being and cares about them. Some clients describe this as leading to feelings of encouragement, increased self-confidence and self-worth, and describe the messages as providing support, guidance and information, giving a sense of direction, reassurance and motivation. Some clients also feel that the sense of caring and support that they receive from health workers through these types of programmes has a positive influence on their relationship with their health worker.
However, clients who are dealing with health conditions that are often stigmatised or personal (e.g. HIV, family planning and abortion care) worry that their confidential health information will be disclosed, or their identity traced due to their participation in these types of programmes. This was noted particularly for vulnerable populations, including adolescents and pregnant women living with HIV, in which the transmission of sensitive health information could disclose their health status or compromise their privacy when seeking health information and services.
Clients' perceptions and experiences of digital targeted client communication are influenced by characteristics of the content; the format; and the delivery mechanisms. The evidence also indicates that access to and use of targeted client communication may be particularly difficult for certain groups of individuals, such as people with low literacy or with limited or controlled access to mobile devices.
The qualitative evidence on the feasibility highlights a number of constraints. These include reliable network connectivity, access to electricity and mobile devices, and the availability of mechanisms to obtain informed consent when enrolling clients into the service. Health systems may experience challenges when attempting to communicate with clients who regularly change their phone numbers without informing the health worker or clients who have poor access to a mobile device.

# Resource use
The evidence suggests targeted client communication via mobile devices may use fewer resources than non-digital interventions.

# Gender, equity and human rights
The qualitative evidence suggests targeted client communication may be particularly difficult for certain population groups, including individuals with poor access to network services or electricity; with limited or controlled access to mobile devices, particularly women and adolescents; individuals who speak minority languages or have low literacy skills or low digital literacy skills; or individuals with conditions that cause them to be particularly concerned about the confidentiality of information exchanged via digital devices.

# Recommendation and justification/remarks

# Targeted client communication
(Recommended only in specific contexts or conditions)
WHO recommends digital targeted client communication for behaviour change regarding sexual, reproductive, maternal, newborn and child health, under the condition that concerns about sensitive content and data privacy are adequately addressed.
Examples of ways to address sensitive content and data privacy include ensuring that individuals are actively made aware of how to opt out of receiving the targeted client communication.
Justification/remarks  The guideline development group (GDG) considered this intervention to offer the potential to improve health behaviours and reduce inequities among individuals with access to mobile devices. The GDG, however, highlighted that measures should be taken to address inequities in access to mobile devices so that further inequity is not perpetuated in accessing health information and services, including mechanisms to ensure individuals who do not have access to mobile devices can still receive appropriate services.
 The GDG also raised the need to address potential concerns about sensitive content and data privacy, including potential negative unintended consequences. This could be done, for example, through mechanisms that actively allow individuals to opt out of services.

# Linkages with other WHO recommendations
The GDG noted that WHO has previously made recommendations related to targeted client communication for improving HIV and tuberculosis medication adherence, which contributed to the considerations for this recommendation. These previous recommendations are listed below.
In the Consolidated guidelines on the use of antiretroviral drugs for treating and preventing HIV infection ( 15), the following interventions demonstrated benefit (all with moderate-quality evidence) in improving adherence and viral suppression:  Tracers refer to communications with the patient, including via home visits, SMS text messages or voice telephone calls.

# Implementation considerations
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.

# Legislation, policy and compliance
 Ensure that clients are actively made aware of how to opt out of receiving the targeted client communication. Attention needs to be paid to ensure that consenting procedures clearly communicate to the clients the intended uses of their data, including to the intentions to continue contacting them, over what period of time, and their right "to be forgotten", or opt out. Procedures need to be in place to ensure that participants are not unduly pressured to provide personal information.

# Services and applications
 Ensure that individuals know the messages are coming from a trusted sender such as a government or health institution, health worker or other familiar entities worthy of their attention.
 Ensure that any sensitive content or personal data transmitted and stored are held on a secure server with protocols in place for destroying the data when appropriate.
 Effective digital communication relies on behaviour change to achieve the intended impact. Such communication should be conducted in the context of a comprehensive communications strategy so that messages received through mobile devices are reinforced by other mechanisms. For example, digital messages should be consistent with the information communicated by health workers, print media and other sources. Further considerations to review when developing content for digital communication include the following.
 Consider the languages used for the content to reach the target audiences, including whether they are in active spoken or written use.
 Ensure that messages are clear and simple. Avoid jargon, technical terms and shortened forms of text. Consider testing to ensure that messages are understood as intended and that any necessary colloquial translations are used.
 Consider the tone of the messages and whether clients are likely to perceive them as friendly and motivational as opposed to shaming or frightening.
 Consider how the content can be tailored to the client, for instance by using their name, local information or personalized reminders.
 Consider whether to include two-way communication with clients to enable their interaction and response to the health system.
 Ensure that the content of the communication reflects the reality of the available commodities and services. For example, encouraging women to seek family planning at their nearest health facility is appropriate if a full range of contraception and advice is available there, including the relevant commodities.
 Ensure the mode of content delivery is appropriate for the setting's network connectivity.
For example, in contexts with low connectivity coverage, not all populations may be reached through digital channels making use of multimedia or mobile app-based communications.
Consider offering messages in a variety of formats (text, audio and video) depending on the setting and infrastructural limitations.

# Equity and sociocultural considerations
 Pay attention to the circumstances of people who have poor access to electricity or poor network coverage, people who cannot afford a mobile device or voice and data charges and people who have limited autonomy, for example because their access to phones is controlled by another person. For the latter case, the GDG felt that the programme should target content accordingly and ensure that users were not put at increased risk.
 Develop concurrent initiatives where such inequity exists so that individuals who do not have access to mobile devices can still receive appropriate services.
 Pay particular attention to the needs, preferences and circumstances of especially disadvantaged or hard-to-reach groups, including people with low literacy or few digital literacy skills, people speaking minority languages, migrant populations in new settings, people affected by emergency situations and people with disabilities such as sight or hearing impairment. Also consider any demographic characteristics, sexual identity or preferences that could put a targeted population at greater risk and ensure that the way the information is provided and accessed is sensitive to this.
 Ensure there are little, or no charges tied to the programme, for instance those associated with downloading apps or sending or receiving the content. Implementers may need to negotiate with mobile network operators and other partners to determine options for subsidizing communication costs or employing voucher systems.

# Effective coverage
The proportion of individuals receiving satisfactory health services among the target population

# Effective coverage:
Health worker decision support
Quality of care, defined as the "degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge", is a foundational component of universal health coverage (76). Quality of care has consistently been documented as suboptimal, particularly across low-and middle-income countries. Commonly cited reasons for poor quality of care have included health workers' inaccurate diagnosis, inappropriate or unnecessary treatment, inadequate or unsafe clinical practices, along with a range of other systemic issues such as insufficient commodities and infrastructural limitations (76).
Although low quality of care stems from numerous deeply rooted health system challenges, decision support tools that offer guidance to health workers have been leveraged as a mechanism to augment adherence to recommended clinical practices (77-80). In their digital form, decision support systems for health workers are defined as electronic systems designed to aid directly in decision-making, in which characteristics of individual patients are used to generate patient-specific assessments or recommendations that are then presented to clinicians for consideration (13,18). Digital decision support for health workers (13), also referred to as clinical decision support systems (CDSS), may be used for a wide range of clinical interactions, including diagnosis and treatment, to facilitate appropriate referrals, minimize errors in medication prescription, and ensure the provision of thorough and accurate care (79). Functionally, decision support tools may be designed to guide health workers through algorithms and rules based on clinical protocols, provide the health worker with checklists for case management and referrals, screen clients by risk or other health status and to assist in health worker activity planning and scheduling (13).
Poor adherence to guidelines

# Lack of or inappropriate referrals
Insufficient health worker competence

# Illustrative health system challenges
The use of decision-support tools has been well established and is supported by some emerging evidence (80). However, over the last decade, health worker decision support has transitioned from being operated on fixed computerized systems to mobile devices, which provide unique opportunities for point-of-care assessment, diagnosis and management. Furthermore, most health care systems in low-and middle-income countries, especially in rural areas, do not have the required infrastructure for desktop computer-based decision support systems and are increasingly investing in making these tools accessible via mobile devices.
This guideline question will explore the added value of digital decision support tools available at primary health care levels and accessible to health workers via mobile devices. Furthermore, as the function of this digital health intervention is broadly applicable across programmatic areas, the guideline question will explore the use of such digital job aids across health conditions within primary care settings.

# Overview of the evidence
The following is a summary of the evidence for decision support for health workers via mobile devices. Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.
There is limited evidence on the effectiveness of health worker decision support via mobile devices directed to clinical health workers. For the intervention directed to community health workers, the evidence suggests that this may have positive effects on individuals taking prescribed medication but may make little or no difference to the individuals' overall health status. When directed to community health workers, decision support may make little or no difference to clients' satisfaction with the information they receive.
The qualitative evidence suggests health workers find the intervention useful and reassuring for guiding the delivery of care. However, some health workers perceive algorithms as too prescriptive, and are concerned that they may lose their clinical competencies by blindly following treatment algorithms. The evidence also suggests that clients find the intervention acceptable and enables health workers to be more thorough when providing care.
The qualitative evidence on the feasibility of digital health interventions, in general, highlights challenges related to network connectivity, access to electricity, usability of the device, sustaining training and support to health workers using the digital tools.

# Resource use
No evidence on resource use was identified.

# Gender, equity and human rights
The evidence on gender, equity and human rights on digital health interventions broadly suggests health workers based in peripheral facilities and rural communities may find these interventions helpful in overcoming geographical barriers and linking to the broader health system, including to access clinical guidance. Health workers in these settings may, though, be more likely to experience poor network coverage and access to electricity, may have lower levels of training and literacy with digital technology, and may have fewer resources, including having limited access to mobile devices.

# Recommendation and justification/remarks
Health worker decision support (Recommended only in specific contexts or conditions)
WHO recommends the use of digital health worker decision support in the context of tasks that are already defined as within the scope of practice for these health workers.
 The GDG expressed that the use of health worker decision support tools when used on mobile devices may improve provision of services point of care. The GDG noted, however, that decision support should not be used for tasks that are beyond the current scope of practices as this may introduce the risk of health workers delivering services for which they have not received adequate training, or of overwhelming the health workers with an expanded set of tasks.
 The GDG highlighted the importance of ensuring the validity of the underlying information, such as the algorithms and decision-logics.
 The GDG also acknowledged additional literature that was not assessed as part of this guideline, on decision support systems via fixed/stationary digital devices. The GDG felt that this evidence suggested the potential of such tools in improving patient/client outcomes could be extrapolated to mobile use, which may offer additional opportunities for settings where the infrastructure for fixed devices is weak.

# Implementation considerations
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.
 Health workers may find it helpful in increasing the acceptability to clients/patients of using digital decision support if they explain that they will be using a digital device and seek clients' permission before using them. Clients should also be made aware that the information from the counselling may be saved and used at future visits to improve quality and continuity. Any concerns with acceptability may be mitigated by, for example, health workers showing the client the inputs and results or listening to the messages or videos together with them so that the device does not become a barrier in the consultation.
 Before using the decision support system, implementers should assess health workers' skills and knowledge to ensure that they have adequate capacity to obtain accurate data before input, to avoid erroneous outputs.
 Referral linkages might need to be strengthened to support possible increases in the number of patients seeking care for previously undetected needs now being revealed by the decision support system.

# Services and applications
 Check the relevance and quality of the decision support content (such as algorithms) and that it aligns with evidence-based clinical guidance, such as WHO or national guidance. Engaging expert groups on the clinical/health topic area may also be necessary as existing guidance may not have sufficient clarity.
 Ensure adequate time for testing all paths of the algorithm with any changes to the software. This type of validation can be done through mechanisms such as an independent review and using mock cases to test the intended output from the algorithms. Also consider built-in mechanisms to update content remotely as algorithms evolve.
 Both health workers and clients should understand that the support provided is based on existing guidelines and policy. While health workers may deviate from the recommendations, they should be clear about their rationale for doing so. Where possible, enable cases to be documented in which health workers feel they need to deviate from the guidance proposed by the decision support system.
 Ensure that use of the device does not impact negatively on the relationship between patient and health worker, particularly when the provider is learning to use the device. As above, this may be helped, for example, by health workers showing patients the inputs and results or listening to the messages or videos together with them so that the device does not put up a barrier. Finally, pay attention to user experience so that correct use of the system is easy for health workers and does not demand more time compared with alternative approaches without it.

# Standards and interoperability
 For the ease of viewing the patient's health history, decision support tools are often integrated with digital health records. See section 3.8 for the evidence and discussion surrounding the combination of decision support with digital tracking of clients' health status and services.

# Multiple points of coverage: digital tracking of clients' health status and services combined with decision-support and targeted client communication
The use of paper-based systems in the delivery of health services introduces a clerical burden on health workers. Additionally, the ability for health workers to keep track of clients effectively, and follow up on services, whether within the facility or in the community, is essential to the continuity of care (12).
Digital tracking is the use of a digitized record to capture and store health information on clients in order to follow-up on their health status and services received (13,40,81). This may include digital forms of paper-based registers and case management logs within specific target populations, as well as electronic patient records linked to uniquely identified individuals. Digital tracking makes possible the registration and follow-up of services and may be done through an electronic medical record (EMR) or other digital forms of health records. Digital tracking aims to reduce lapses in continuity of care by stimulating timely follow-up visits and may incorporate decision support tools to guide health workers at the point of care in executing clinical protocols, delivering appropriate care, scheduling upcoming services and following checklists for appropriate case management.
Digital tracking and decision support systems may also be linked with demand-side interventions to engage clients/patients, such as through targeted client communication via mobile devices. Targeted client communication in this context is defined as the transmission of targeted health content or reminders to a specified population or to individuals within a predefined health or demographic group (13).

# Effective coverage
The proportion of individuals receiving satisfactory health services among the target population

# Continuous coverage
The extent to which clients receive the full course of intervention required to be effective

# Accountability coverage
The proportion of those in the target population registered into the health system

# Insufficient continuity of care
Lack of quality/ reliable data

# Delayed provision of care
Poor adherence to guidelines

# Illustrative health system challenges
This guideline has sought to understand the benefit of an integrated package consisting of three different digital health interventions, to support health worker practices as well as to stimulate client-side demand for health services and stimulate behaviour change.
This guideline reviewed the following intervention combinations:

# Overview of the evidence
The following is a summary of the evidence on the digital tracking of clients' health status and services (shortened to digital tracking), in combination with health worker decision support and targeted client communication. Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.

# Effectiveness (a) Digital tracking and decision support:
The evidence on the effectiveness of digital tracking combined with decision support suggests it may improve health service use and health outcomes, such as: attendance of antenatal care appointments, taking iron tablets during pregnancy, immediate breastfeeding, receipt of the third dose of polio vaccine, and use of postpartum contraception six months after birth.
However, digital tracking combined with decision support probably makes little to no difference on other outcomes, such as: the proportion of children under five who are vaccinated, proportion of women who give birth in a facility, women breastfeeding exclusively for six months, or on the proportion of women using contraception within six months of birth.
There was limited evidence on the effect of digital tracking combined with decision support on the use of emergency visits for children under five and on the timeliness of receiving services, as the certainty of this evidence was assessed as very low.

# Linkage with other WHO recommendations
These findings align with recommendation 11 of the WHO guideline on health policy and system support to optimize community health worker programmes, which suggests that practising community health workers "document the services they are providing and that they collect, collate and use health data on routine activities, including through relevant mobile health solutions" (17).

# Implementation considerations
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.
Legislation, policy and compliance  Accurate client/patient identification to facilitate the digital tracking of health services across different facilities and health workers requires adequate policy and legal processes and protections. This can include the use of a card-based or biometric-based identifier, as an example, and having telecommunications infrastructure that is available consistently across facilities and programmes.
 Consider whether the digital tracking would have adequate infrastructural support to be maintained over time. The start-up costs and infrastructural requirements of a digital tracking system tend to be higher than for paper-based interventions. When used appropriately and effectively, the costs of digital interventions are amortized, and cost-savings may materialize in the long run. However, in contexts where basic health infrastructure is limited, including in human resources, digital tracking systems may be very resource-intensive to set up and maintain.

# Standards and interoperability
 The digital tracking should be linked to a system that provides a unique identity for each individual. Such unique IDs help health workers search for clients, reduce the potential for duplicate registration of clients in community and facility systems and ensure continuity of care. This unique ID could, in turn, be linked to a local or national ID system to provide a foundational digital identity that can facilitate longitudinal follow-up and linkages across different levels of the health system and digital health interventions.

# Effective coverage
The proportion of individuals receiving satisfactory health services among the target population

# Effective coverage: digital provision of training and educational content to health workers Background
Broadly defined as the management and provision of educational and training content in digital form for health professionals, electronic learning (eLearning) has emerged as one approach to increasing health workers' access to training and educational resources (18). More recently, the widespread reach of mobile devices has prompted the use of such technologies to deliver training content to health workers, also known as mobile learning (mLearning). Such training content may be exchanged using channels such as SMS text messaging, the multimedia messaging service, applications ("apps"), games, and other forms of digital modality (82). In particular, low-and middle-income countries and remote areas with limited ICT infrastructure and geographical barriers may seek to leverage mobile devices to maximize access to educational content and continuing medical education (82).
Although the use of digital tools for strengthening the health workforce is referenced in several WHO resources (15,70,71,83), these do not examine the specific considerations on digital provision of training and educational content. This guideline question assesses the potential contributions and implications of providing digital training and educational content via mobile devices, as part of complementary efforts to support workforce needs for in-service training and continued education.
Poor adherence to guidelines

# Lack of access to information
Insufficient health worker competence

# Illustrative health system challenges

# Overview of the evidence
The following is a summary of the evidence on the provision of digital training and educational content for health workers accessible via mobile devices. Web Supplement 1 provides the full evidence-to-decision framework for this intervention, detailing the available evidence on effectiveness, acceptability, feasibility, resource use and implications for equity, gender and rights.
The evidence suggests that this intervention may increase health workers' knowledge. However, the effects of this intervention on other outcomes, including health workers' performance, skills and attitudes, is uncertain because there is no direct evidence, or the evidence is of very low certainty.

# Acceptability 3
The qualitative evidence from medical and nursing students suggests that these users see a number of advantages to mLearning tools, including the ease and portability of accessing materials and ability to personalize content to their own needs. They may have some concerns, however, for instance about the validity and accuracy of the information, as well as potential negative effects when used during patient interactions.
The qualitative evidence on the feasibility of digital health interventions highlights challenges related to network connectivity, access to electricity, usability of the device, sustaining training and support to health workers using the digital tools.

# Resource use
No evidence on resource use was identified. Resource use considerations are listed within the evidence-to-decision framework in Web Supplement 1.

# Gender, equity and human rights
The qualitative evidence on digital health interventions broadly suggests health workers based in peripheral facilities and rural communities may find these interventions helpful in overcoming geographical barriers and linking to the broader health system. However, health workers in these settings may also be more likely to experience poor network coverage and access to electricity, may have lower levels of training and literacy with digital technology, and may have fewer resources, including poorer access to the mobile devices that may be needed for some programmes.
The systematic review of mLearning specifically explored factors influencing implementation of mLearning among both pre-and post-qualified health workers. However, this review only included studies on nursing and medical students. The technical team extrapolated findings from this review that would be relevant for health workers.
The qualitative evidence suggests that most health workers see advantages to digital technologies compared with paper-based systems. These include quicker recording of required client data and services delivered, easier access to client data, easy identification of mistakes, and not having to carry paper registers. Health workers are often reluctant, however, to use digital tracking when they have to maintain both digital and paper-based systems, since this increases their work burden.
There was limited evidence documenting the feasibility of these integrated interventions specifically. Challenges have been highlighted, however, by the qualitative evidence on the feasibility of digital health interventions in general, including those of network connectivity, access to electricity, usability of the device, sustaining training and support to the health workers using the digital tools, and system integration.

# Resource use
No evidence on resource use was identified. Resource use considerations are listed within the evidence-to-decision framework in Web Supplement 1.

# Gender, equity and human rights
The qualitative evidence on these digital health interventions suggests health workers based in peripheral facilities and rural communities may find the interventions useful in overcoming geographical barriers and linking to the broader health system. Health workers in these settings may also, however, be more likely to experience poor network coverage and poor access to electricity, may have lower levels of training and literacy with technology, and may have fewer resources, including having poorer access to mobile devices.  in settings where the health system can support the implementation of these intervention components in an integrated manner; and  for tasks that are already defined as within the scope of practice for the health worker; and

# Recommendation and justification/remarks
 where potential concerns about data privacy and transmitting sensitive content to clients can be addressed.
 The guideline development group (GDG) recognized that this intervention package may pose challenges, particularly in settings in which the health system may not be able to manage the infrastructural and technical complexity of such a multifaceted intervention.
The GDG also felt that the intervention may require substantial upfront resource use but believed that the intervention may reduce costs in the long term by transitioning away from inflexible paper-based systems.
 Despite the risk of increasing complexity by implementing a system with multiple digital components, the GDG believed that implementing these interventions in an integrated manner offered opportunities to (i) reduce health workers' time spent on redundant activities such as reporting; (ii) increase the timeliness and responsiveness of health workers by linking data from client health tracking systems to the actions recommended from decision support tools; and (iii) provide a more holistic view of the client and their interactions with the health system.
 While there is value in a multi-pronged digital intervention that simultaneously targets supply side factors (i.e. decision support to health workers), and demand-side factors (i.e. targeted client communication), the technical and human resource requirements for such an intervention should be considered. The GDG suggests the three components be implemented in a gradual manner, particularly in settings where the enabling environment and infrastructure may not be sufficiently mature to support such a multifaceted intervention.
 In line with the separate recommendation on targeted client communication via mobile devices (see section 3.6 for more detail), the GDG's recommendation to combine it into digital tracking is conditional on measures being taken to address inequities in access to mobile devices and address concerns about sensitive content. Similarly, the inclusion of the decision support component will require alignment to the tasks and scope of practice for health workers to avoid potential harms and added burden (see section 3.7 for more detail).
Workforce  Consider phasing implementations to avoid overburdening health workers. For example, consider introducing integrated packages only once health workers have already been implementing at least one of the interventions and are familiar with digital technologies.
 Focus on introductory and ongoing training of health workers in using these tools, including support for technical troubleshooting during the provision of care. Health workers may have challenges in using technology during the provision of services, which can negatively impact the quality of care, or result in the technology not being used. Use metrics to assess health workers' use of the digital system and identify opportunities to reinforce training.

# Equity and sociocultural considerations
 Inequities may be reduced for populations included within the digital tracking system because it helps to ensure that they receive services. Inequities may arise, however, for those outside of the digital tracking system whose service provision might not be accounted for. Such inequity needs to be monitored during implementation. The problem can be addressed by first enumerating the target population and so increasing the accuracy of the denominator by which populations are eligible for services.
 The digital tracking of individuals' health status may be controversial in some circumstances, for example among migrants or other groups who lack firm legal status in particular settings.
The extent to which such groups may trust tracking depends on who is doing the tracking and how the information is likely to be used. It is important to take these concerns, and local policies on digital identities, into account when designing a programme to ensure it does no harm. Justification/remarks  Despite the availability of evidence primarily focused on improving health worker knowledge, the guideline development group (GDG) felt that the potential benefits of the intervention outweighed the potential harms.

# Recommendation and justification/remarks
 The GDG also noted that mLearning offered an additional delivery channel for continuing health education, and thereby expanding access to in-service training resources and professional development opportunities to a broader set of health workers.
 The GDG also considered the potential for cost savings for continued health education, when compared with the costs of expanding face-to-face in-service training.
 It should be noted that this intervention only applies to post-certification health workers and used in the context of in-service training and continued health eduction.

# Linkage with other WHO recommendations
The WHO guideline on health policy and system support to optimize community health worker programmes suggests an emphasis on face-to-face learning for pre-service community health workers, to be supplemented by eLearning on aspects where it is relevant (17).

# Implementation considerations
The specific implementation considerations that emerged from the literature and the GDG's deliberations for this intervention are listed below, organized where appropriate against the framework outlined in the WHO/ITU National eHealth strategy toolkit (18). This is not an exhaustive list of considerations; additional implementation resources and policy documents should be consulted before implementing the recommendations.
 Consider network capacity and coverage especially if mLearning materials may be videos which can my time consuming to download in certain settings.

# Legislation, policy and compliance
 Consider if health workers can earn credits for continuing education using these materials, as a way of increasing their uptake.
Workforce  To increase the acceptability of mLearning devices, it may be important to improve awareness among staff and supervisors about the value of portable devices and to develop ground rules or codes of conduct for when and how devices should be used.
 Similarly, it may be helpful to give patients explanations of device use, and to ask patients' permission before using a device. Ensure also that use of devices does not impact negatively on  the relationship between health workers and clients, particularly if being used in the context of service delivery, and especially when health workers are learning to use the devices.
 Involve the relevant professional bodies, including national certification or institutional boards, to ensure that the content of the mLearning programmes aligns with the current scopes of practice and national training curriculums for health workers.

# Services and applications
 Ensure that the information is from a source that is considered trustworthy and credible by health workers in your setting. For example, the information loaded on the mLearning system should be based on validated content or should align with national or WHO clinical guidance.
 Consider which types of training content are best delivered via mLearning channels and which through other or mixed channels, including through in-person training.
 Where available, mLearning materials should be curated and accredited as formal training courses.
 Ensure that the programme is user-tested among health workers, both those in practice and those in training, to ensure that their needs and concerns are met.
 Ensure that health workers can easily store content for future reference.
 Consider how health workers can tailor the content to suit their specific needs. For instance, develop content in a modular format so that health workers can select information for particular review.