WHO guideline on school health services. Web Annex D. Systematic reviews of the effectiveness and acceptability of comprehensive school health services: methodology# Contents

# SHS school health services
A glossary of terms used throught the guidance and its web annexes is provided in the guidance document.

# Glossary vi

# Systematic reviews of the effectiveness and acceptability of comprehensive school health services: methodology Web Annex D
Web Annex D summarizes the methodology from the systematic reviews of the effectiveness and acceptability of comprehensive school health services (SHS). A PRISMA-compliant (Preferred Reporting Items for Systematic reviews and Meta-Analyses) systematic review was conducted. Effectiveness and acceptability were assessed both quantitatively and qualitatively.
In addition, the economic cost and benefit of SHS were reviewed.

# D.1 Systematic reviews of effectiveness and acceptability research questions
Two systematic reviews of effectiveness and acceptability (SREA) of comprehensive SHS questions were posed.  - delivered by or supervised by a full-or part-time health worker; a within the context of SHS, a health worker was defined as a health or allied professional who provides dedicated services to school-going learners (services that are not available to other population groups), irrespective of the site where services are provided Comparator Any comparator (versus, for example, no school health service, different type of school health service, or a similar programme delivered through a community-based general or child/adolescent clinic) or no comparator a The service must have been delivered or supervised within the SHS by a health worker. If it was only delivered by trained teachers, it was excluded.

# D.3.2 Effectiveness outcomes
Initially, effectiveness outcomes were listed in broad categories for consideration, as shown in Box D.1.
Box D.1. Broad categories of outcomes considered for the systematic review of the effectiveness of SHS
- Coverage: reach or effective coverage (proportion of the target population that participated in the intervention, for instance)
- Policy existence and uptake (such as percentage of schools implementing a policy)
- Responsiveness to unmet need (including: equity issues; meeting the needs of underserved populations)
- Health and development outcomes (including knowledge, attitudes, behaviours, self-efficacy or skills) related to the following:
-positive development, such as health-seeking behaviour, menstrual hygiene management, self-esteem -unintentional injury -violence (including physical assault)
-sexual and reproductive health, including HIV and other SHS -other communicable diseases, including vaccination -noncommunicable diseases (such as asthma, diabetes, allergies)
-nutrition and physical activity (including diet, obesity)
-chronic physical disabling conditions (like vision or hearing loss, scoliosis)
-oral health -mental health (including mental disability, bullying)
-self-harm, including suicide-related outcomes (ideation, plans, gestures and attempts)
-substance use -morbidity that would otherwise require acute major health service expenditure (hospitalization or emergency department visits)
-overall health complaints
- Other well-being outcomes -quality of life -educational/academic outcomes/positive development (including literacy and numeracy, participation in education (number of days absent/present), intention to remain in school, health literacy)
- Other outcomes -participants' perceptions of the effectiveness of the intervention
- Cost-effectiveness (cost per death or disability-adjusted life year averted)
After consideration of these broad effectiveness outcomes, critical and important outcomes were agreed between the review team and the Steering Group and were endorsed by the Guideline Development Group (GDG), following the definitions given by the WHO Guideline Review Committee (GRC), in which critical outcomes are those that are critical when formulating recommendations and important outcomes are those that should also be taken into consideration but they are not critical for decision-making (1).

# D.3.3 Acceptability outcomes
Initially, acceptability outcomes were also listed in broad categories for consideration, as shown in Box D.2.
Box D.2. Broad categories of outcomes considered for the systematic review of the acceptability of SHS
- Access and use: were the students able to access SHS? Did they use SHS when they had health needs?
- Satisfaction: did the students like SHS? Would they recommend them?
- Preference: of children/adolescents/parents for intervention to be provided at school versus elsewhere
- Quality: what was good or bad about SHS? For instance: barriers to access for the students; confidentiality concerns; students felt respected; amount and clarity of information; stigma
- Relevance: were the SHS relevant to the students' health issues?
- Fidelity: were the implemented SHS close to those outlined in programming guidance documents?
- Appropriate: were the SHS culturally sensitive and inclusive (for students with special needs, for example)?
- Willingness to pay
From the provider perspective After consideration of these broad acceptability outcomes, the specific critical and important outcomes that would be considered in the review were agreed between the review team and the Steering Group and were endorsed by the GDG. These are shown in Table D.3.

# D.3.4 Study design, languages and publication date
To minimize bias across all study designs, in consultation with WHO, high-quality evidence designs were prioritized. These were a priori defined as systematic reviews, randomized controlled trials (RCTs) or non-randomized controlled studies (NCSs).
NCSs were defined for these reviews as intervention studies with a contemporaneous control group where the allocation had not been done randomly or controlled before-and-after studies, "natural experiments" and interrupted time-series studies.
Other potentially eligible study designs were observational (cross-sectional) studies with either quantitative or qualitative data. These were to be considered as supplementary data if the highquality evidence was deemed insufficient to answer the research questions, considering high-income and low-or middle-income countries separately.
All research published in English or other European languages was eligible for inclusion.
Any publication date was eligible.

# D.4 SREA search, screening, selection and full-text review
The raw search results were split between two reviewers and their titles and abstracts were screened for potential eligibility. Based on this screening, the full text of all articles that were deemed potentially eligible by either reviewer were subsequently reviewed for eligibility. A random 1 in 20 sample of studies was assessed by both reviewers. There were few disagreements, but when there were, the studies were discussed with a third, senior reviewer to reach agreement.  Sixteen systematic reviews were found to be partially relevant to the review questions (11 systematic reviews for the effectiveness review, three for the acceptability review and two for both reviews). None was assessed as high-quality according to the AMSTAR (A MeaSurement Tool to Assess systematic Reviews) criteria. These therefore were used as sources to identify relevant primary studies.
Three additional eligible papers were identified in the systematic reviews that had not already been identified in the search.
For the effectiveness review, 11 studies with highquality research designs were identified. There were no RCTs; these were all NCSs. In consultation with the WHO Secretariat and the GDG methodologist, it was decided that there were sufficient NCSs to restrict the searches for studies in high-income countries (HIC) to systematic reviews, RCTs and NCSs. However, since this was not true for LMIC, the original proposal to include observational studies was followed for studies in those contexts. This identified two additional studies (one quantitative and one qualitative) that were included in the review as supplementary data.
Three of the above non-randomized controlled effectiveness studies also included economic outcomes. Five additional non-randomized controlled economic studies were also included.
Thus the searches identified eight economic studies in total, all from HIC.
For the acceptability review, 4 NCSs (all in HIC) were identified, and this review additionally included 18 observational (cross-sectional) studies in LMIC as supplementary data.

# D.5 SREA data extraction
A data-extraction spreadsheet which was prespecified and pilot-tested prior to the main data extraction phase was developed. Descriptive information was extracted from studies, including the year of publication, funding, country and date of study, research objectives, methodology, details of the population, intervention, comparator, outcomes, analysis and sample characteristics (such as sample size, age, gender, race, socioeconomic characteristics and urban/rural setting) from all included studies for which the data were available. One researcher extracted information from all included studies and the data-extraction spreadsheet was checked by the other two researchers. Results were extracted according to the pre-specified outcomes from the protocol.

# D.6 SREA quality appraisal of individual studies
Due to the inherent challenges of studying SHS, it was essential that all studies were rigorously quality-appraised according to best-practice guidelines. This was carried out for each type of study as follows.
- Systematic review: it was pre-specified that the quality of any eligible systematic reviews would be appraised using AMSTAR 2 (2). However, no systematic reviews were found that directly addressed the review questions. They were therefore not included in the reviews but were used only to check that the searches had identified all eligible primary studies and to provide background/supplementary information.
- RCT: it was pre-specified that randomized studies would be assessed using the Cochrane risk of bias (ROB 2.0) tool (3). However, no eligible RCTs were identified.
- Non-randomized study: for non-randomized and observational (cross-sectional) studies, the risk of bias was appraised using the Risk Of Bias In Nonrandomised Studies of Interventions (ROBINS-I) tool (4).
- Economic evaluation: the Critical Skills Appraisal Programme checklist was used to evaluate the quality of economic evaluations (5).
- Qualitative study: qualitative studies were assessed using Cochrane criteria of credibility, transferability, dependability and confirmability (6).

# D.7 SREA data analysis
Because of substantial variation in the sample in terms of the different types of interventions covered and in line with WHO guidance (1), the findings of the included studies were synthesized narratively (7) and organized according to the outcomes of interest as defined in the protocol. Where data allowed and studies were considered methodologically similar, data were pooled using forest plots to create estimates of effect, including odd ratios for dichotomous data and mean differences for continuous data. Data transformation was not undertaken. The method used to produce data synthesis was pre-specified to include all relevant data from each included study relating to the pre-specified outcomes in the protocol and this was followed rigorously to avoid introducing bias. The results of each individual study are presented in tables so the same elements of information are included in the same order for each study. The findings were then thematically analysed according to the outcomes of interests, as defined above. Qualitative data were assessed narratively using the same outcomes as for the quantitative data (such as feasibility of implementation) as themes.
The amount of high-quality (systematic review, RCT or NCS) data was assessed for HIC and LMIC separately.

# D.8 SREA GRADE assessment for overall certainty of evidence
In line with WHO guidelines (1), the confidence in the findings of the studies was assessed for each outcome according to GRADE guidance, which entails assessing the potential for risk of bias, imprecision, inconsistency, indirectness and publication bias for each finding (9). For the nonrandomised studies, the ROBINS-I assessments were incorporated to inform GRADE ratings, in line with recent guidelines (10). The rating for NCSs starts as high if ROBINS-I is used to assess risk of bias (4), and then downgraded. For example, risk of bias was downgraded once for a response rate below 80% and twice for a response rate under 60%; imprecision was downgraded once for wide confidence intervals or for a sample size below 300 and twice for a sample size below 100 participants; inconsistency was downgraded once for unexplained heterogeneity between studies indicated by I2 > 50%. The rating for the observational (cross-sectional) studies started as low and was eligible to be increased for factors strengthening the evidence base, such as: consistency over large numbers of studies and participants; dose-response relationship; or if the effect was present despite confounders tending in the opposite direction (11). For the qualitative studies, the GRADE Confidence in the Evidence from Reviews of Qualitative research (GRADE-CERQual) approach was used (12,13). The GRADE assessment was carried out by one researcher and was reviewed by a senior researcher, in consultation with the WHO Secretariat and the GDG methodologist.