An Evaluation of the Effectiveness of e-IMCI Implementation in Primary Health Care Clinics in South Africa.
Condition: Child Health; Clinical Practice Guidelines; Primary Health Care
Study type: Interventional

Summary: 
      The research hypothesis is that sick children attending primary health care (PHC) clinics who
      are managed by IMCI-trained health workers (HWs) using electronic Integrated Management of
      Childhood Illness guidelines (e-IMCI) receive better quality of care compared to children
      managed by HWs using conventional paper-based IMCI (pIMCI). The aim of the study is to
      evaluate the effectiveness of e-IMCI to improve care for sick children under five years
      attending PHC clinics in one district in KwaZulu-Natal, South Africa.

      Objectives:

        1. To assess feasibility and acceptability of eIMCI implementation in PHC clinics

        2. To compare clinic-based management of sick children using e-IMCI with a gold standard
           IMCI assessment, and those managed using p-IMCI to a gold standard IMCI assessment

        3. To determine the cost effectiveness of e-IMCI compared to p-IMCI implementation in PHC
           clinics

      Primary outcomes:

        -  Proportion of sick children receiving all medications indicated among children managed
           by HWs using eIMCI and HWs using pIMCI.

        -  Proportion of sick children with risk/high risk of Tuberculosis, HIV or HIV exposed,
           and/or malnutrition correctly identified among children assessed using eIMCI and
           children assessed using pIMCI, compared to a gold standard IMCI assessment.

        -  Incremental cost-effectiveness of eIMCI implementation vs standard of care (pIMCI).

      The study will employ a prospective two-arm cluster randomized controlled trial. Sample size:
      a total of 30 clinics in one district will be randomly selected to participate and allocated
      to the intervention (eIMCI) group (n=15) and control (pIMCI) group (n=15). One IMCI trained
      HW will be randomly selected from each clinic to participate. Six observations will be
      conducted with each participating health worker Intervention HWs will receive an IMCI update
      and computer training based on eIMCI. Control HWs will receive a similar update using pIMCI.
      Both groups will receive support visits and intervention HWs will receive additional
      computer/IT support. Health worker knowledge will be assessed pre and post training using a
      self-administered questionnaire.

      Quality of care will be assessed in both groups using exit interviews with mothers and review
      of child health records. In addition, gold standard IMCI assessments will be conducted by an
      IMCI expert to determine correct findings. Assessment and management of the child by the IMCI
      expert will be compared to that of the participating HW to determine quality of care
      provided.
    

Detailed description: 
      Research Hypothesis The research hypothesis is that sick children attending primary health
      care (PHC) clinics who are managed by IMCI-trained health workers (HWs) using electronic
      Integrated Management of Childhood Illness guidelines (e-IMCI) receive improved quality of
      care compared to children managed by HWs using conventional paper-based IMCI (pIMCI).

      Aim To evaluate the effectiveness of e-IMCI to improve care for sick children under five
      years attending PHC clinics in one district in KwaZulu-Natal (KZN), South Africa.

      Objectives

        1. To assess the feasibility and acceptability of implementation of e-IMCI in primary
           health care clinics

        2. To compare the clinic-based management of sick children using e-IMCI with a gold
           standard IMCI assessment, and those managed using p-IMCI to a gold standard IMCI
           assessment

        3. To determine the cost effectiveness of e-IMCI compared to p-IMCI implementation in
           primary health care clinics

           Study oversight Study activities will be overseen by a eIMCI steering group. This will
           include representatives from KZN Department of Health, UKZN, Ilembe district and ELMA
           foundation. The steering committee will meet three times per year for the duration of
           the project to review progress of project activities and provide oversight and advice
           where appropriate.

           Research methodology The study will employ a prospective two-arm cluster randomized
           controlled trial (RCT) to test the hypothesis that implementation of eIMCI will
           effectively improve quality of care for sick children under five years attending PHC
           clinics. A total of 30 clinics in one district will be randomly selected to participate,
           with 15 clinics each randomly allocated to the intervention (eIMCI) and control (pIMCI)
           groups. One IMCI trained HW will be randomly selected from among all IMCI trained HWs in
           each clinic to participate in the study.

           Study site The study will be undertaken in the Ilembe District in KwaZulu-Natal (KZN),
           South Africa. This district was selected in partnership with the KZN Department of
           Health. There are four hospitals, including one regional hospital, two community health
           centres, and 31 primary health care (PHC) clinics. All PHC clinics are reported to have
           at least one IMCI trained health worker.

           Ilembe health district is situated in KZN, and contains one small urban area and a large
           deep rural area, and serves a population of approximately 663 000 people living in
           around 198 000 households. Ilembe district comprises 4 sub-districts namely KwaDukuza,
           Ndwedwe, Mandeni and Maphumulo Ilembe is characterised by high rates of poverty and
           unemployment, and high prevalence of Tuberculosis and HIV.

           Study population

           There are two study populations:

             -  IMCI trained HWs

             -  Mothers of sick children aged < five years attending the clinic. Sampling &
                recruitment PHC clinics: Simple random sampling will be used to select 30 PHC
                facilities to participate in the study from all 31 clinics in Ilembe district.
                Simple randomization will be used to assign clinics in the intervention or control
                groups (n=15 per group). Clinics with no IMCI trained nurses will not be eligible
                to participate and will be replaced.

           IMCI trained nurses: Simple random sampling will be used to select one IMCI trained
           nurse in each clinic to participate in the study from all eligible IMCI trained nurses
           working in the clinic. However, two IMCI trained nurses in selected clinics will
           participate in the intervention, with the intention that e-IMCI will be implemented in
           the entire clinic, rather than just selecting a single individual to participate.

           Mothers of children under five-years: Systematic sampling will be used to select mothers
           attending the clinic with children who are aged under five years to participate in the
           study. All mothers with sick children waiting in the queue each day will be identified.
           In order to reduce observer bias, sampling of children will be undertaken in a
           systematic way that 1) spaces out the number of children selected, and 2) is
           unpredictable to reduce the possibility that the observed HW can identify which child
           will be reviewed. We will finalise this process depending on the size of the clinic, and
           number of children seen daily, with the aim of observing two- three sick children each
           day of observation. Observations in each clinic will continue for one week (4-5 days).

           Sample size Sample size calculation is based on the primary outcome measure, which is
           the proportion of children who receive all treatments indicated and is powered to detect
           a difference of 20% in the proportion of children receiving all treatments in the
           intervention and control groups.

           Sample sizes of 90 in group one and 90 in group two, which were obtained by sampling 15
           clusters (PHC clinics) with 6 subjects (observed children) each in the intervention
           group, and 15 clusters with 6 subjects each in the control group to achieve 81% power to
           detect a difference between the group proportions of 0.2000. The group two proportion is
           0.7000. The group one proportion is assumed to be 0.7000 under the null hypothesis and
           0.9000 under the alternative hypothesis. The test statistic used is the two-sided Z test
           (unpooled). The significance level of the test was 0.0500.

           Description of the intervention:

           The intervention will be undertaken in Ilembe District in KwaZulu-Natal. Training and
           support visits will be overseen and conducted by implementation staff in partnership
           with staff from the DoH.

           Development of an eIMCI/pIMCI training course:

           An eIMCI training course will be developed with two components 1) basic computer skills
           training 2) case studies to develop eIMCI skills Structured piloting of an eIMCI
           training course will be undertaken to ensure that participants receive enough training
           to reach competency.

           the pIMCI training will be developed with the same case studies using a paper-based
           approach to ensure that study participants receive exactly the same IMCI training in
           intervention and control clinics, so that the groups differ only in respect of how IMCI
           is used (computer based vs paper-based).

           Deployment of computers:

           Ahead of implementation of the eIMCI intervention computers and printers will be
           purchased and deployed in the child health/IMCI consulting room in all intervention
           clinics. This computer will be available for use of any IMCI trained health worker
           consulting with sick children during the intervention period.

           Facilitators for district based training:

           Training will be conducted by implementation staff assisted by local DoH trainers in
           Ilembe district. These trainers will be trained ahead of the district training using the
           course developed in the pilot. Although district DoH staff may assist with training we
           plan to involve as few staff members as possible to mimimise bias and contamination
           given that many DoH staff are routinely visiting both intervention and control clinics.

           Training of participating health workers (eIMCI and pIMCI):

           Two IMCI trained health workers will receive training in each clinic (N= 30 HWs in each
           arm). This will be the selected participant plus another IMCI trained health worker in
           the clinic (if there is one available). Participants will receive either a paper-based
           update of pIMCI or eIMCI training.

           Both computer skills and IMCI knowledge will be assessed at beginning and end of the
           training.

           Support visits (IMCI and IT) Following the training, there will be two IMCI support
           visits for all IMCI trained health workers (eIMCI and pIMCI). This will assist eIMCI
           participants to transfer their new skills to the workplace and will address any initial
           problems of participants. This will be a structured visit using case studies and will be
           mirrored for pIMCI participants using paper to ensure participants in both groups
           receive similar support.

           A cellphone based group (WhatsApp group) will be set up to provide support for IT
           problems to eIMCI participants if they experience any problems with the computer or the
           programme. If IT problem cannot be resolved over the phone an IT support specialist will
           visit the clinic to assist the participant. The IT support role will be conducted by
           external IT experts .

           Any study personal who are involved in the evaluation of the study will be separate from
           the training / intervention component.

           Data collection (quantitative) Preparation: Before the start of the RCT the study team
           will work with the Maternal Child and Womens' health directorate and with the Ilembe
           District Health Management Team to prepare the district for implementation of the
           research study. All participating clinics will be visited before the start to inform the
           operational manager about the study, identify IMCI trained health workers who are
           eligible to participate, and thereby create a sampling frame from which IMCI trained HWs
           will be selected. Background information will be collected about provision of child
           health services at each clinic and resources available at the clinic, for example
           equipment, staff, drugs etc including any history of recent drug stock outs at the
           clinic. The appropriate site for deployment of the computer will be identified. A
           further visit may be required for the purpose of selecting participants and placing the
           computer in the clinical area.

           Tracking of implementation of eIMCI: Data collection will commence six weeks after the
           start of eIMCI/pIMCI implementation in the clinic. This implementation period will allow
           participants to gain confidence in using eIMCI and to receive support visits. The
           training and data collection will be staggered to ensure a similar duration of the
           implementation period among all participants. The following data will be tracked in
           eIMCI clinics over the six-week period of implementation and for a total of six months
           post implementation: proportion of children seen by eIMCI trained health workers where
           the eIMCI system is used; number and nature of IT problems experienced by eIMCI nurses;
           how IT problems were addressed; number of additional visits for IT support; any stock
           outs of medication.

           Field worker selection and training: Data will be collected by a team of data
           collectors, comprising one expert IMCI practitioner and one field worker, in each of the
           intervention and control clinics. Data collection teams will collect data either in
           intervention or control clinics, to minimize bias there will be no cross over between
           the two groups. The data collection teams will receive training for two weeks on data
           collection processes, which will include piloting of the data collection techniques in a
           clinic in another district not implementing eIMCI. In addition, expert IMCI
           practitioners will receive any updates or refreshers in IMCI assessment that are deemed
           appropriate to ensure they are able to provide a competent gold standard assessment.

           Data collection activities: The data collection period for each participating health
           worker will be arranged to ensure that the selected HW is on-duty and seeing sick
           children. Data collection teams will be deployed in each participating clinic for
           one-week (four to five working days) to collect data. During the data collection period
           all sick children in the queue to be seen by the observed health worker will be
           identified, and each day a systematic sampling process will be used so that care is
           reviewed for 2-3 children . A minimum of six case reviews will be done for each
           participating health worker. The total number of sick children seen by the observed HW,
           and the time each child spends in the consulting room will be recorded during the period
           of data collection.

           Data will be collected using a combination of exit interviews with mothers, review of
           the patient held records and review of medication received or prescribed. In addition,
           each child will be reassessed to provide a gold-standard IMCI assessment against which
           the HWs management can be compared.

           Data sources:

           Health worker knowledge and skills assessment pre and post training in both groups
           Electronic monitoring of eIMCI use from the software Tracking tool for assessing time
           spent in the consulting room Exit interviews Reviews of patient held records Gold
           standard IMCI assessment

           Data Collection (qualitative) A series of qualitative data collection activities will be
           conducted to assess feasibility and acceptability of eIMCI implementation before, during
           and after eIMCI implementation in participating intervention clinics.

           In-depth interviews/reference groups will be conducted in IsiZulu for mothers and in
           English for HWs by experienced researchers. All in-depth interviews will be audio
           recorded. All the interviews will be transcribed and translated into English in
           preparation for data analysis.

           Reference groups:

           Before the implementation of eIMCI a series of four reference groups will be conducted
           with IMCI trained professional nurses in clinics in another district not implementing
           eIMCI. A group of IMCI trained nurses who routinely provide services to sick children in
           a given clinic will be brought together to participate in a focus group discussion
           (FGD). The aim of the reference groups is to explore the experiences of providing care
           to children in PHC clinics, how child health services are provided, and what the current
           challenges are with providing services to sick children and with IMCI implementation. We
           will then introduce and demonstrate the eIMCI programme, and nurses will be asked to
           comment on potential implementation of this approach.

           Using this approach we will be able to use experiences and responses from nurses to
           identify and anticipate challenges of feasibility and acceptability of implementing
           eIMCI. In addition we will be able to explore overall challenges related to IMCI
           implementation which may be equally relevant to eIMCI. If appropriate, implementation of
           the e-IMCI roll out may be adapted to address these concerns.

           In-depth interviews with mothers:

           In-depth interviews will be conducted with mothers managed by a health worker using
           eIMCI to explore their experience of having the health worker use eIMCI during the
           consultation. Mothers will be eligible to participate if she reports that the HW
           consulted the computer during the consultation. Topics to be explored will include
           experiences during the eIMCI consultation, and how the use of the computer was perceived
           by mothers.

           One mother will be selected for an in-depth interview (IDI) from eight randomly selected
           clinics during the period of data collection (n=8 participants). If necessary, in-depth
           interviews will be continued until no further themes emerge.

           In depth interviews with IMCI trained HWs implementing eIMCI:

           Health workers using eIMCI will be selected for in-depth interviews in six randomly
           selected clinics on completion of data collection. We will purposively select three
           eIMCI participants who used eIMCI for a high proportion of their assessments (>80%
           consultations with sick children) and four participants who implemented eIMCI poorly
           (<50% consultations) to identify barriers and enablers for IMCI implementation. If
           necessary, in-depth interviews will be continued until no new themes emerge.

           In addition, three further IDIs will be conducted with good implementers 6 months after
           completion of data collection to explore issues of sustainability and enablers and
           barriers to implementation in the long term.

           Consultation observations:

           In order to explore how eIMCI implementation affects the nurse- mother relationship
           during the consultation, a series of observations will be undertaken by an expert IMCI
           practitioner on the study team who is also a trained qualitative researcher.
           Consultations will observed among nurses using pIMCI and eIMCI (10 observations per
           group) and a checklist will be used to record information about nurse- mother
           interactions during the consultation. These observations will be conducted after the
           quantitative data collection is complete.

           Ethical considerations General All participants, both IMCI trained health workers and
           mothers, will provide written informed consent. Consent forms will be prepared in both
           the local language (isiZulu) and English, they will be read aloud to each potential
           participant by a trained researcher and any questions will be addressed. All
           participants will be informed that participation is voluntary for both community health
           workers (CHW) and mothers; and that mothers have a right to withdraw from the study at
           any time.

           For any mother who is unable to write, a thumbprint will be obtained. Consent forms will
           not be linked with the study information, and these will be kept separate from the data.
           All signed consent forms will be kept in a locked cupboard for a period of five years
           after the study is completed.

           During data collection If during the interview and assessment process of the child, the
           IMCI fieldworkers observes that the mother did not receive all the treatment the child
           needs, the fieldworker will ensure that the mother gets all the treatment needed for the
           baby according to the reassessment done by the IMCI trained fieldworker. This will be
           arranged in such a way to minimize the bias that could result if the HW was aware of
           correction being made, and to avoid undermining the HW in the eyes of the mother.

           Mothers of sick children who are referred to hospital will be interviewed if it does not
           interfere or disrupt treatment given to children; and the mother agrees to be
           interviewed.

           On completion of data collection the intervention team will work with staff in the
           district to continue to roll out eIMCI and ensure that control clinics will also have
           the opportunity to benefit from eIMCI. Computers used in the evaluation will be donated
           to the clinics for ongoing eIMCI implementation.

           To maintain anonymity and confidentiality in qualitative data, participants' identity
           will be protected through the use of pseudonyms or codes. Participants will be made
           aware of the fact that data will be used for report writing, conference presentation and
           thesis writing.

           Data analysis The quantitative data will be analysed using STATA. Qualitative data will
           be analysed using NVIVO software, framework analysis with pre-determined themes will be
           used in the process of analysing data.

           Cost and cost effectiveness analysis A cost and cost effectiveness analysis will be
           undertaken. The cost analysis will take a provider perspective and provide an estimate
           of incremental financial costs associated with delivering eIMCI. The provider
           perspective is favoured ahead of the societal perspective as the costs incurred by
           mothers across the two approaches to delivery are unlikely to differ significantly. The
           incremental cost of the eIMCI, rather than the full cost, is favoured as it will provide
           an estimate of the required increase in funding associated with a roll-out of eIMCI,
           compared to continuing only with the current approach to IMCI. A financial perspective
           is opted for ahead of an economic costing, as this will provide relevant information for
           budgeting. Start-up costs, such as computers and training, will be included, but all
           research costs will be excluded and only the costs associated with implementation
           considered.

           The costing for the incremental cost effectiveness analysis will also be from the
           provider perspective, but will differ from that used in the simple cost analysis in a
           number of ways. For the cost effectiveness analysis, financial and broader economic
           costs will be included. For example, the inclusion of resources not paid for as part of
           the intervention, but which are diverted to support the intervention. Moreover, the
           incremental costs will be measured relative to the supported pIMCI provided at the
           control clinics, as opposed to the standard of care.

           Using the data on the primary outcome measure, and the cost analysis conducted for the
           purpose, the incremental cost effectiveness of eIMCI will be estimated: the cost per
           additional sick child receiving all medications indicated. This calculation provides an
           estimate of the returns associated with the additional investment in eIMCI.
    

Intervention name(s): electronic IMCI; paper IMCI
Intervention type(s): Behavioral; Behavioral
Intervention description(s): Participants will be trained to use an electronic version of the Integrated Management of Childhood Illness (IMCI) guidelines; Participants will be trained to use routine paper version of Integrated Management of Childhood Illness (IMCI) guidelines
Allocation: Randomized
Intervention model: Parallel Assignment
Intervention model description: A total of 30 clinics in one district will be randomly selected to participate, with 15 clinics each randomly allocated to the intervention (eIMCI) and control (pIMCI) groups. One IMCI trained HW will be randomly selected from among all IMCI trained HWs in the clinic to participate in the study.
Masking: None (Open Label)
Primary purpose: Treatment
Keywords: Child health; Clinical guidelines; South Africa; Africa; electronic health intervention