# eHealth literacy measurement tools: a systematic review protocol

Carole Délétroz, Marina Canepa Allen, Maxime Sasseville, Alexandra

Rouquette, Patrick Bodenmann, Marie Pierre Gagnon

<figure><img src="image_1.png" /></figure>

## To cite this version:

Carole Délétroz, Marina Canepa Allen, Maxime Sasseville, Alexandra Rouquette, Patrick Bodenmann, et al.. eHealth literacy measurement tools: a systematic review protocol. Systematic Reviews, 2022, 11 (1), pp.205. 10.1186/s13643-022-02076-2. hal-03888894

HAL Id: hal-03888894

https://hal.science/hal-03888894

Submitted on 9 Dec 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

# eHealth literacy measurement tools: a systematic review protocol

Carole Délétroz<sup>1,2*</sup> ID, Marina Canepa Allen<sup>2</sup>, Maxime Sasseville<sup>1</sup>, Alexandra Rouquette<sup>3,4</sup>, Patrick Bodenmann<sup>5,6</sup> and Marie-Pierre Gagnon<sup>1</sup>

**Abstract**

**Background:** Improving eHealth literacy (eHL) is one of the biggest challenges currently facing the global health-care community. Indeed the use of digital services has the potential to engage patients in care as well as improve the effectiveness of chronic disease self-management, it remains highly dependent on a patient's specific skills and experiences in the health care systems. Although eHealth literacy has gained momentum in the past decade, it remains an underresearched area, particularly eHealth literacy measurement. The aim of the review is to identify patient-reported outcome measures (PROMs) of eHealth literacy for adult populations and to summarize the evidence on their psychometric properties.

**Methods:** We will conduct a systematic literature review of the tools used to measure eHealth literacy for adult population. The search strategy aims to find published studies. A three-step search strategy will be used in this review. Published studies will be searched in CINAHL, PubMed, PsycINFO, and Web of Science from inception until end. Grey literature will be searched to find theses. Database search strategies will be formulated and tested with the assistance of an expert Health Sciences Librarian. The selection of studies will be done by two independent reviewers. Disagreements will be resolved through consensus, and a third reviewer will solve discrepancies. Furthermore, two reviewers will independently evaluate the methodological rigor of the instruments development and testing and assign a grade using the standardized Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) checklist. Disagreements will be discussed with a third reviewer, expert in psychometrics.

Extracted data will be aggregated and analyzed to produce a set of synthesized findings that will be used to develop evidence-informed recommendations in regard of eHL instruments. We will present a synthesis of all instruments, their psychometric properties, and make recommendations for eHL instrument selection in practice. Reporting will be informed by the Preferred Reporting Items for Systematic Reviews and Meta-Analysis and a PRISMA flow diagram.

**Discussion:** This systematic review will summarize the evidence on the psychometric properties of PROMs instruments used to measure eHL and will help clinicians, managers, and policy-makers to select an appropriate instrument.

**Systematic review registration:** PROSPERO CRD42021232765

**Keywords:** eHealth literacy, Health literacy, Psychometrics, Systematic review

**Background**

The generalization of eHealth (electronic health) to numerous activities in the health system is leading to changes in individual and collective health uses and practices. Improving eHealth literacy (eHL), which is a type

<footnote>*Correspondence: carole.deletroz@hesav.ch</footnote>

<footnote n="2"><sup>2</sup> School of Health Sciences (HESAV), University of Applied Sciences and Arts Western Switzerland, Avenue de Beaumont 21, 1011 Lausanne, Switzerland
Full list of author information is available at the end of the article</footnote>
of health literacy, continues to be a challenge worldwide [1, 2].

A growing number of articles in the literature point out the potential benefits of eHealth interventions to improve health behaviors [3], clinical outcomes [4], and patient empowerment in the management of chronic health conditions [5–7]. Therefore, it is essential for clinicians to evaluate eHL among their patients in order to provide care and services that suit their actual needs and capabilities. eHL was found to be associated with more effective contact with physician, enhance use of medical insurance, improve self-management of health needs, and raise perceived understanding of a disease/condition [8]. Current researchers on eHL emphasize the need to use specific measurement to determine whether eHealth initiatives are improving patient care [5, 9, 10].

For patients, it is becoming critical to have access and skills to understand information through electronic services including health portals or electronic patient record. eHealth referred to “the organization and delivery of health services and information using the Internet and related technologies” [11]. As a result, it requires specific skills to use and interact in the digital environment [8]. Norman and Skinner, taken up by the WHO and the European Commission, described eHL as “the ability to seek, find, understand, and appraise health information from electronic sources and apply the knowledge gained to addressing or solving health problem” ([1], p. 3). The effectiveness of eHealth is highly dependent on patient eHL’s skills, complementary to health literacy, that are sometimes new to many patients and unevenly distributed across the population. Populations at risk for limited health literacy are similarly vulnerable to eHL challenges, such as the elderly, those with lower education and patients suffering from chronic conditions [12]. Therefore, a certain part of the population will not be able to effectively make use of eHealth services, which could affect health outcomes and the ability to make an optimal choice of health information [13, 14].

## Rationale

eHL is a multidimensional concept with varying definitions [15, 16]. This paper uses the definition of Norman and Skinner, who state that research, evaluation, and patient use of online health information are the three main domains of eHL. eHL is assessed most often by the self-report measure eHEALS [17], which focuses on finding information on the Internet and assessing it. Norman and Skinner [17] found that the measure consists of one factor in an exploratory factor analysis, but recent work [18] uncovered 2 factors: seeking and appraising. Despite the increasing use of the eHEALS, limited evidence exists regarding whether increasing patient's eHL can improve

health outcomes [19]. Measurement of eHL is difficult, because no single measure can capture the different constructs of outcomes of eHealth interventions [9, 20]. Furthermore, eHL should consider not only individual knowledge and skills but also people's perception of the use of system providing health care as relevant to the management of their condition [21]. Therefore, attempts to assessing eHL by self-reported measure remain challenging in nursing and health research [22, 23].

Several sources of difficulty in measuring eHL can be identified: (1) there is confusion over the term “eHealth literacy” behind the measure [24]. (2) What constitutes eHL within existing instruments is not always defined from the patient’s perspective (validity) [25]. (3) Instruments need to produce consistent and reproducible results (reliability) and, most importantly, instruments need to be suitable for clinical practice [26].

The challenge is to measure eHL as a set of cognitive skills necessary for the effective “use” of eHealth interventions. The literature describes eHL as a metaliteracy that comprises six different subtypes of literacies focusing on the use of online information [15]. The relevance of eHL is demonstrated in recent studies, showing that people’s self-perceived skills to use online information affect health behaviors and self-management of health conditions, and that a lack of such skills may lead to adverse outcomes. Neter and Brainin [8, 27] found relationships between eHL and the presence of chronic illness, perceived self-management skills, and better self-perceived understanding of health status, symptoms, and optional treatments. Hsu and coll [28]. found that eHL skills are associated with various types of positive health behavior, including healthy eating, exercise, and sleep behavior. Furthermore, studies have shown that certain groups do not have the skills and knowledge to use eHealth tools efficiently for their own benefit and might thereby even experiencing difficulty in accessing health care services, such as elderly people, individuals with low health literacy, or presenting with cognitive problems, and those with limited access to technology for various reasons (e.g., limited capacity to use it, inaccessibility to the Internet, chronic degenerative diseases) [29–31]. Due to the possible personal, social, and health impact of eHL, we need to distinguish between instruments measuring patient experiences focusing only on the use of online information and those considering skills needed in using eHealth.

A self-reported measure of a patient’s eHL needs to be based on sound theoretical and empirical evidence and those instruments should be representative of patients’ experience of eHL in daily life (content validity). To date, there is only one self-report measure used in more than one study: the eHealth Literacy Scale (eHEALS) [17].
This measure focuses on finding information on the Internet and assessing it. A broader self-report scale that also addresses generating information was just recently developed [32]; however, only few eHL's studies were performed [10, 33]. There is a risk that self-reported instruments have been developed based on literature only, or from experts' opinions about important aspects of eHL, without considering patients experiencing eHealth in their personal background. A valid measurement instrument of eHL is essential to examine the effects of eHealth, both on an individual level and on a population level. On an individual level—for example, in daily clinical practice—a measurement instrument could support decisions about the extent to which a patient is able to benefit from particular eHealth tools and interventions. On a population level, a proper measurement instrument could provide insight into vulnerable subgroups that face additional challenges in using health care, due to its digitization. It is imperative to determine the validity of existing eHL's self-reported measure. Besides, instruments measuring the patients' eHL must be acceptable and interpretable to both patients and clinicians.

The standardized Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) risk of bias checklists [34, 35] provides a framework for the quality assessment of each included study and the determination of a grade of evidence of health measurement instruments. The quality assessment of each included study is assessed using the “scoring quality form COSMIN boxes” [36]. The overall rating (grade evidence) is based on the quality criteria for good measurement properties proposed by the COSMIN group [37].

The measures of eHL need to produce consistent and reproducible results if they are to be trusted in practice (reliability). These measures are PROMs, which are defined as standardized, validated questionnaires (which are also called instruments) completed by patients [36]. Data arising from PROMs may be used to identify skills for adequate use of eHealth tools and interventions [38]. Therefore, the psychometrics of each measure of eHL need to be evaluated and described to facilitate their use in the relevant context.

Currently, there is one systematic review [19] that identified PROMs of eHL and describes the conceptual models they are based on. The authors identified eight instruments, with only three of them being based upon a conceptual model of eHealth literacy: eHEALS, eHLS, and PRE-HIT. The remaining five instruments were dual tools, i.e., comprised of individual measures for health literacy and E-literacy [19]. This review mentions lacks assessment of the quality (i.e., risk of bias) of published study on reliability or measurement error of each instrument. There is a necessity to consider theoretical and

empirical issues to be able to select the right instrument for the right purpose.

The current systematic review seeks to examine the quality of PROMs (used for patients: adults > 18 years old) and to identify the best instrument for field use. These findings could assist in informing evidence-based practice for the use of PROMs of eHL. We expect that the findings will assist researchers and clinicians in identifying and selecting the most appropriate instrument when measuring eHL.

## Study aim and research question

The aim of this review is to systematically identify, synthesize, and evaluate the methodological quality of all relevant studies on the development and validation of eHL PROMs for adult populations. The research question for this review is: which PROMs have been developed to assess eHL and what are their psychometric qualities?

## Method

### Study method

Following the methods for conducting systematic reviews outlined in the Cochrane Handbook [39], we developed a review protocol in PROSPERO (registration number CRD42021232765). The Preferred Reporting Items For Systematic Reviews and Meta-Analysis (PRISMA-P) [40] and a PRISMA flow diagram [41] will be used to ensure the design and reporting quality of the review. The evaluation of the methodological rigor of the testing of the instruments and determination of the Grade will be done with the standardized Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) checklist [36].

### Search strategy

The search strategy will aim to find published studies describing the development and the validation of eHL self-reported instrument from their inception until July 2021. A three-step search strategy will be used in this review [39]. An initial limited search of PubMed and CINAHL has been undertaken with a screening of the words contained in the title and abstract, looking for terms, keywords, and index term used in relevant articles, and of the index terms used to describe each article. This informed the development of a search strategy which will be tailored for each information source. A second search will be done across four databases: CINAHL, PubMed, PsycINFO, and Web of Science. Database search strategies will be formulated and tested with the assistance of an expert Health Sciences Librarian from Laval University (Laferrière M-C). ProQuest dissertations and Thesis, Opengrey, DART, and BASE will be consulted in order to find theses in the grey literature. Thirdly, the reference
list of all identified reports and articles will be screened for additional studies. A sample search strategy from MEDLINE is presented below (Table 1).

Finally, a last search on PubMed with a filter specifically created for the search of PROMs will be done [42] and MeSh will be updated, if necessary, by M-C L. For each measure of eHL identified, we will do a specific search and if needed, we will contact the authors to identify any relevant studies. Following the search, all identified citations will be collated and uploaded into EndNote X9 [43]. Duplicates will be removed manually.

## Inclusion criteria

This review will consider all studies conducted on the development and/or validation (including psychometric quality) of measures of eHL in adults (> 18 years old). Studies relating to human health and published between 2000 and 2021 in English, French, Spanish, or Italian will be considered for inclusion. The adoption of e-health by health institutions start in the early 2000's with the rapid emergence of technology use during that time [44]. Therefore, only studies published from 2000 onwards will be considered for inclusion in this review. eHL of children and teenagers is conceptualized in a contradictory way in the context of digital health skills: firstly, the “digital natives” are imagined as particularly competent

and user of the digital world, secondly, they are seen as a particular risk group in the context of internet addiction, thirdly, empirical studies show that adolescents in particular consider themselves to be more competent in health than adults in self-reporting [45]. Therefore, studies measuring eHL among children (< 18 years old) will not be included. Table 2 presents the outcomes of the review according to COSMIN's methodology.

## Study selection (selection process)

Titles and abstracts will be screened against the inclusion criteria for the review. Studies that meet the inclusion criteria will be retrieved in full (including full text) and their citation details imported into Covidence [46]. The full text of selected citations will be assessed in detail against the inclusion criteria by two independent reviewers. Reasons for exclusion of full-text studies will be recorded and reported in the final systematic review. Any disagreements that arise between the reviewers at each stage of the study selection process will be resolved through discussion with a third reviewer. The results of the search will be reported in full in the final report and presented in a Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) flow diagram [40, 47].

<table><caption>**Table 1** Search strategy PubMed</caption><thead><tr><td>#</td><td>Question</td><td>Results</td></tr></thead><tbody><tr><td>16</td><td>((#6 AND #9) OR #3) AND (#13 OR #14)</td><td>1,912</td></tr><tr><td>15</td><td>#13 OR #14</td><td>6,048,301</td></tr><tr><td>14</td><td>"Reproducibility of Results"[Mesh] OR "Psychometrics"[Mesh] OR "Surveys and Questionnaires"[Mesh] OR "Interviews as Topic"[Mesh]</td><td>1,444,624</td></tr><tr><td>13</td><td>Psychometr*[Title/Abstract] OR validity[Title/Abstract] OR measure*[Title/Abstract] OR "quantitative evaluation"*[Title/Abstract] OR scale*[Title/Abstract] OR scales[Title/Abstract] OR instrument*[Title/Abstract] OR tool[Title/Abstract] OR tools[Title/Abstract] OR toolkit*[Title/Abstract] OR questionnaire*[Title/Abstract] OR survey*[Title/Abstract] OR interview*[Title/Abstract]</td><td>5,456,755</td></tr><tr><td>12</td><td>(#6 AND #9) OR #3</td><td>2,736</td></tr><tr><td>11</td><td>#6 AND #9</td><td>2,493</td></tr><tr><td>9</td><td>#8 OR #7</td><td>2,361,309</td></tr><tr><td>8</td><td>"Internet"[Mesh] OR "Digital Divide"[Mesh] OR "Computers"[Mesh] OR "Computer Literacy"[Mesh] OR "Telemedicine"[Mesh:NoExp] OR "Mobile Applications"[Mesh] OR "Online Social Networking"[Mesh] OR "Patient Portals"[Mesh]</td><td>177,289</td></tr><tr><td>7</td><td>digital[Title/Abstract] OR computer*[Title/Abstract] OR informatic*[Title/Abstract] OR electronic*[Title/Abstract] OR internet[Title/Abstract] OR app[Title/Abstract] OR apps[Title/Abstract] OR application*[Title/Abstract] OR mobile[Title/Abstract] OR Telehealth[Title/Abstract] OR telemedicine[Title/Abstract] OR technolog*[Title/Abstract] OR "social media"[Title/Abstract] OR "social computing"[Title/Abstract] OR "social network"*[Title/Abstract] OR "patient portal"[Title/Abstract] OR smartphone[Title/Abstract]</td><td>2,287,365</td></tr><tr><td>6</td><td>(("health literac"*[Title/Abstract])) OR ("health literacy"[MeSH Terms])</td><td>10,604</td></tr><tr><td>5</td><td>"health literacy"[MeSH Terms]</td><td>5,79</td></tr><tr><td>4</td><td>("health literac"*[Title/Abstract])</td><td>8,546</td></tr><tr><td>3</td><td>(eHealth[Title/Abstract] OR e-Health[Title/Abstract] OR mhealth[Title/Abstract] OR m-health[Title/Abstract]) AND (literac*[Title/Abstract])</td><td>608</td></tr><tr><td>2</td><td>literac*[Title/Abstract]</td><td>19,618</td></tr><tr><td>1</td><td>eHealth[Title/Abstract] OR e-Health[Title/Abstract] OR mhealth[Title/Abstract] OR m-health[Title/Abstract]</td><td>11,076</td></tr></tbody></table>
<table><caption>**Table 2** Outcomes-measurement quality criteria according to Terwee</caption><tr><td>1</td><td>*Internal consistency: Cronbach's alpha (Cronbach (α)), Kunder-Richardson's coefficient (KR20 et KR21), χ<sup>2</sup></td></tr><tr><td>2</td><td>*Reliability, relative measures (including test-retest reliability, inter-rater reliability and intra-rater reliability): intraclass correlation coefficient (ICC), Kappa coefficient (K), the weighting scheme description—e.g., linear quadratic</td></tr><tr><td>3</td><td>*Measurement error, absolute measures: standard error of measurement (SEM), smallest detectable change (SDC), or limits of agreement (LoA)</td></tr><tr><td>4</td><td>*Content validity, including face validity: qualitative outcomes about concept elicitation, relevance and comprehensiveness by asking patients and/or professionals, and about comprehensibility and comprehensiveness of the PROMs.</td></tr><tr><td>5</td><td>*Construct validity: Structural validity: results of exploratory (EFA) or confirmatory factor analysis (CFA); results of item response theory (IRT) test for determining the (uni-) dimensionality of the scale.<br>Hypotheses testing, design and statistical methods adequate for the hypotheses being tested<br>Cross-cultural validity: confirmatory factor analysis (CFA); differential item function (DIF) between language groups assessed</td></tr><tr><td>6</td><td>*Responsiveness (no gold standard DHL measure available): design and statistical methods adequate for the hypotheses to be tested</td></tr></table>

## Patient and public involvement

This research will be done without patient involvement. As this review aims at synthesizing methodological information about existing eHL instruments, patient and public experience is not directly mobilized. However, patient and public will be involved at a later stage in this project.

## Data extraction

Data will be extracted from the selected studies using the data extraction form (Table 3) proposed by the Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) and the Risk of bias checklist [34] that will be integrated in Covidence.

Two researchers will independently extract the data for all included studies and agree, through consensus, about the accuracy and completeness of the data. Where consensus is difficult to achieve, a third reviewer will be used to reach agreement verification (including psychometrics) to minimize bias and potential errors in data extraction. The data extracted will include general information about the article, and specific details about the instrument as well as the psychometrics. Findings will be extracted and assigned a level of credibility (Table 3).

## Assessment of methodological quality of included studies

We will apply the COSMIN checklist to evaluate the methodological rigor and results of the instruments [34]. The checklist consists in modules that enables specific criteria to be applied to certain tests. The flexibility of the checklist ensures that the same level of scrutiny is applied to evaluate various studies of instruments, even if they have conducted different psychometrics tests. Selected studies will be critically appraised by two independent reviewers for methodological quality using the COSMIN Scoring quality form boxes [48]. Any disagreements that arise between reviewers will be resolved through the

<table><caption>**Table 3** Data extraction form</caption><tr><td>General information</td><td>Author<br>Year<br>Country of origin<br>Papers</td></tr><tr><td>Instrument details</td><td>eHL instrument name<br>HL domain and component (item number)<br>Target population<br>Response category<br>Scoring system (range of score)<br>Burden (minutes for administration)<br>Original language</td></tr><tr><td>Characteristics of PROM</td><td>PROM development<br>Content validity<br>Structural validity<br>Internal consistency<br>Cross-cultural validity<br>Measurement invariance<br>Reliability<br>Measurement error<br>Criterion validity<br>Hypotheses testing for construct validity<br>Responsiveness</td></tr></table>

evaluation by the third reviewer, who is a psychometrics expert (MS). The results of critical appraisal will be summarized in a table.

Following COSMIN guidelines, only the studies that have a “good” methodological quality in content validity will undergo data extraction analysis and synthesis. The quality of included studies will be considered in the
analysis and will therefore be reflected in the findings and a grade will be given in conclusion of the systematic review [34]. If we have multiple studies for one PROM, all results per measurement property of a PROM will be synthesized. Then the results will be evaluated against the criteria for good measure proprieties to have an overall rating for the measurement property (Table 4) in order to give a grade.

## Evaluation of measurement proprieties for included instruments

The quality of each measurement property of an instrument will be evaluated using the quality criteria proposed by Terwee et al. [49] who participated in the group that developed the COSMIN checklist (see Table 4: Rating the measurement properties of the PROM).

Each measurement property will be given a rating result (+'positive', '-negative', '? indeterminate, and 'na' no information available). There is currently no empirical method to pool together results of measurement properties; therefore, synthesis is recommended [50].

## Data synthesis

Research findings will be synthesize using the COSMIN "Scoring quality form boxes" tool [48]. These categories will then be subjected to a synthesis, in order to produce a single comprehensive set of synthesized findings that can be used as a basis for evidence-based practice. When pooling will not be possible, the findings will be presented in narrative form [51]. A narrative synthesis of instrument purpose, rigor, and findings will enable level of evidence recommendations for the selection of PROM.

## Best evidence synthesis: levels of evidence

As recommended by the COSMIN Group 'a best evidence synthesis' will be used to synthesize all the evidence on measurement properties of different instruments. The procedure used will follow the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) framework [34]. This procedure is a transparent approach to rate the quality of evidence that is often used in reviews of clinical trials. Under this procedure, the possible overall rating for a measurement property is 'positive', 'negative', 'conflicting' or 'unknown', accompanied by levels of evidence ('strong', 'moderate' or 'limited') (Table 5).

Three steps will be taken to obtain the overall rating for a measurement property. First, the methodological quality of a study on each measurement property will be done using the COSMIN checklist. According to COSMIN, measurement properties from inadequate quality studies may be included to contribute to 'the best evidence synthesis'. Second, the quality of each measurement property of an instrument will be evaluated using Terwee's quality criteria [37, 50]. Third, the rating results of measurement properties in different studies on the same instrument will be examined whether consistent or not. This best evidence synthesis will be performed by one author and then checked by a second author.

## Meta-bias(es)

Following COSMIN guidelines, the results from inadequate quality studies may be included when synthesizing or summarizing the results from all studies, or if the results of inadequate studies are consistent with the results of better quality [52].

<table><caption>**Table 4** Rating the measurement properties of the PROM [35]</caption><thead><tr><th rowspan="2">PROM</th><th colspan="3">Study 1</th><th colspan="3">Study 2</th></tr><tr><th>Rating</th><th>Rating</th><th>Rating</th><th>Rating</th><th>Rating</th><th>Rating</th></tr></thead><tbody><tr><td></td><td>+/-?</td><td>+/-?</td><td>+/-?</td><td>+/-?</td><td>+/-?</td><td>+/-?</td></tr><tr><td></td><td>Rater 1</td><td>Rater 2</td><td>Consensus</td><td>Rater 1</td><td>Rater 2</td><td>Consensus</td></tr><tr><td>Structural validity</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Internal consistency</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Cross-cultural validity</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Measurement invariance</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Reliability</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Measurement error</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Criterion validity</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Construct validity</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Responsiveness</td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>
<table><caption>**Table 5** Quality of evidence for measurement proprieties of the PROMs COSMIN scoring box tool ([25, 35], p. 62)</caption><thead><tr><th>Overall rating</th><th>Quality of evidence</th><th>Overall rating</th><th>Quality of evidence</th><th>Overall rating</th><th>Quality of evidence</th></tr></thead><tbody><tr><td>+/-?</td><td>High, moderate, low, very low</td><td>+/-?</td><td>High, moderate, low, very low</td><td>+/-?</td><td>High, moderate, low, very low</td></tr></tbody></table>

## Discussion

This review will present a synthesis of eHL's PROMs, their psychometric properties, and make recommendations for selection in practice and research. This review will provide some insight regarding the challenges of using self-report instruments in assessing eHL. We will systematically discuss the main issues in the testing of a subjective health measurement instrument in regard to existing systematic reviews in the eHL study domain (including the specification of the instrument characteristics: the chosen measurement paradigm, the context of use, the structure of the instrument and measurement proprieties). We will discuss operational issues related to the conduct of this systematic review of PROMs. The approach which consists of using eHealth resources has led to improvements in clinical outcomes and health service efficiency and might potentially improve access to care and reduce health-care costs. For clinical practice, it is important to have a good instrument to appraise patients' eHL level in order to correctly use eHealth resources and maximize benefits for the patient's disease self-management. It is also essential to provide patients and providers with a rigorous evaluation of e-HL competencies that facilitate the use of eHealth in the relevant context.

## Abbreviations

eHL: eHealth literacy; eHealth: Electronic health; PROM: Patient-reported outcome measures; COSMIN: COnsensus-based Standards for the selection of health Measurement Instruments PRISMA.: Preferred reporting items for systematic reviews and meta-analysis.

## Acknowledgements

We would like to acknowledge the contribution of Marie-Claude Laferrière, Health Sciences Librarian at Laval University, who shared her understanding of health databases and helped to shape the search strategies.

## Authors' contributions

CD conceived and designed the study, devised search strategies, drafted the inclusion selection form, piloted data extraction and quality assessment with COSMIN methodology, and drafted the manuscript. MCA participated in study design, piloting of inclusion selection form, data extraction and quality assessment with COSMIN methodology and revision of manuscript. MS participated at the study design, as well as third reviewer for inclusion selection form, data extraction, and quality assessment and revision of the manuscript. AR provided statistical and psychometrics results advice and revision of the manuscript and MPG and PB provided direction for the study idea and design, provided statistical advice, and helped revise the manuscript. All authors have read and approved the final manuscript.

## Funding

A doctoral fellowship VITAM has been allocated for the study for the literature search. The HESAV is funding the positions of the first and second reviewer.

## Declarations

### Competing interests

The authors declare that they have no competing interests.

### Author details

¹Faculty of Nursing Sciences, Laval University, 2325 Rue de l'Université, Québec, QC G1V 0A6, Canada. ²School of Health Sciences (HESAV), University of Applied Sciences and Arts Western Switzerland, Avenue de Beaumont 21, 1011 Lausanne, Switzerland. ³Paris-Saclay University, Inserm, UVSQ, CESP, DevPsy, Paris, France. ⁴Public Health and Epidemiology Department, AP-HP Paris-Saclay, Bicêtre Hospital, Le Kremlin-Bicêtre, France. ⁵Department of Vulnerabilities and Social Medicine, Unisanté, Lausanne, Switzerland. ⁶Faculty of Biology and Medicine, Vice-Dean Teaching and Diversity, University of Lausanne, Rue du Bugnon 44, 1011 Lausanne, Switzerland.

Received: 8 November 2021 Accepted: 18 September 2022

Published online: 24 September 2022
## References

1. European Commission. Flash Eurobarometer 404 "European citizens' digital health literacy"; 2014. https://doi.org/10.2759/88726.

2. Novillo OD. Digital health literacy. First Meeting of the WHO GCM/NCD Working Group on Health Literacy for NCDs Digital Health Literacy for NCDs. Geneva; 2017. https://www.who.int/global-coordination-mechanism/working-groups/digital_hl.pdf

3. Cushing CC, Steele RG. A meta-analytic review of ehealth interventions for pediatric health promoting and maintaining behaviors. J Pediatr Psychol. 2010;35(9):937-49. https://doi.org/10.1093/jpepsy/jsq023.

4. Marcolino MS, Oliveira J, D'Agostino M, Ribeiro AL, Alkmim M, Novillo-Ortiz D. The impact of mHealth interventions: systematic review of systematic reviews. JMIR Mhealth Uhealth. 2018;6(1):e23. https://doi.org/10.2196/mhealth.8873.

5. Gee PM, Paterniti DA, Ward D, Miller LMS. e-Patients perceptions of using personal health records for self-management support of chronic illness. Comput Inform Nurs. 2015;33(6):229-37. https://doi.org/10.1097/CIN.0000000000000151.

6. Heinrich C, Karner K. Ways to optimize understanding health related information: the patients' perspective. Geriatr Nurs. 2011;32(1):29-38. https://doi.org/10.1016/j.gerinurse.2010.09.001.

7. Manafo E, Wong S. Promoting eHealth literacy in older adults: key informant perspectives. Can J Diet Pract Res. 2013;74(1):37-41. https://doi.org/10.3148/74.1.2013.37.

8. Neter E, Brainin E. eHealth literacy: extending the digital divide to the realm of health information. J Med Internet Res. 2012;14(1):e19. https://doi.org/10.2196/jmir.1619.

9. Griebel L, Enwald H, Gilstad H, Pohl AL, Moreland J, Sedlmayr M. eHealth literacy research-Quo vadis? Inform Health Soc Care. 2018;43(4):427-42. https://doi.org/10.1080/17538157.2017.1364247.

10. Neter E, Brainin E. Perceived and performed ehealth literacy: survey and simulated performance test. JMIR Hum Factors. 2017;4(1):e2. https://doi.org/10.2196/humanfactors.6523.

11. Boogerd EA, Arts T, Engelen LJ, van de Belt TH. "What Is eHealth": Time for An Update? JMIR Res Protoc. 2015;4(1):e29. https://doi.org/10.2196/resprot.4065.

12. Smith B, Magnani JW. New technologies, new disparities: The intersection of electronic health and digital health literacy. Int J Cardiol. 2019;292:280-2. https://doi.org/10.1016/j.ijcard.2019.05.066.

13. Diviani N, van den Putte B, Giani S, van Weert JC. Low health literacy and evaluation of online health information: a systematic review of the literature. J Med Internet Res. 2015;17(5):e112. https://doi.org/10.2196/jmir.4018.

14. Kickbusch I, Pelikan JM, Apfel F, Tsouros AD. Health literacy: The solid facts. Copenhagen: WHO; 2013.

15. Norman CD, Skinner HA. eHealth literacy: essential skills for consumer health in a networked world. J Med Internet Res. 2006;8(2):e9. https://doi.org/10.2196/jmir.8.2.e9.

16. Norgaard O, Furstrand D, Klokker L, Karnoe A, Batterham R, Kayser L, et al. The e-health literacy framework: a conceptual framework for characterizing e-health users and their interaction with e-health systems. Knowl Manage E-Learn. 2015;7(4):522-40.

17. Norman CD, Skinner HA. eHEALS: the eHealth literacy scale. J Med Internet Res. 2006;8(4):e27. https://doi.org/10.2196/jmir.8.4.e27.

18. Soellner R, Huber S, Reder M. The concept of eHealth literacy and its measurement. J Media Psychol. 2014;26:29-38. https://doi.org/10.1027/1864-1105/a000104.

19. Karnoe A, Kayser L. How is eHealth literacy measured and what do the measurements tell us? A systematic review. Knowl Manage E-Learn. 2015;7(4):576-600. https://doi.org/10.34105/j.kmel.2015.07.038.

20. Buyl R, Beogo I, Fobelets M, Deletraz C, Van Landuyt P, Dequanter S, et al. e-Health interventions for healthy aging: a systematic review. Syst Rev. 2020;9(1):128. https://doi.org/10.1186/s13643-020-01385-8.

21. Kayser L, Karnoe A, Furstrand D, Batterham R, Christensen KB, Elsworth G, et al. A Multidimensional tool based on the eHealth literacy framework: development and initial validity testing of the eHealth Literacy Questionnaire (eHLQ). J Med Internet Res. 2018;20(2):e36. https://doi.org/10.2196/jmir.8371.

22. Falissard B. Mesurer la subjectivité en santé: perspective méthodologique et statistique. Paris: Elsevier Masson; 2008.

23. Waltz CF, Strickland OL, Lenz ER. Measurement in nursing and health research. New York: Springer Publishing Company; 2017.

24. Neter E, Brainin E, Baron-Epel O. The dimensionality of health literacy and eHealth literacy. Eur Health Psychol. 2015;17(6):275-80.

25. van der Vaart R, Drossaert CH, de Heus M, Taal E, van de Laar MA. Measuring actual eHealth literacy among patients with rheumatic diseases: a qualitative analysis of problems encountered using Health 1.0 and Health 2.0 applications. J Med Internet Res. 2013;15(2):e27. https://doi.org/10.2196/jmir.2428.

26. Bannigan K, Watson R. Reliability and validity in a nutshell. J Clin Nurs. 2009;18(23):3237-43. https://doi.org/10.1111/j.1365-2702.2009.02939.x.

27. Neter E, Brainin E. Association between health literacy, eHealth literacy, and health outcomes among patients with long-term conditions. Eur Psychol. 2019;24(1):68-81. https://doi.org/10.1027/1016-9040/a000350.

28. Hsu W, Chiang C, Yang S. The effect of individual factors on health behaviors among college students: the mediating effects of eHealth literacy. J Med Internet Res. 2014;16(12):e287. https://doi.org/10.2196/jmir.3542.

29. Clement S, Ibrahim S, Crichton N, Wolf M, Rowlands G. Complex interventions to improve the health of people with limited literacy: a systematic review. Patient Educ Couns. 2009;75(3):340-51. https://doi.org/10.1016/j.pec.2009.01.008.

30. Jensen JD, King AJ, Davis LA, Guntzviller LM. Utilization of internet technology by low-income adults: the role of health literacy, health numeracy, and computer assistance. J Aging Health. 2010;22(6):804-26. https://doi.org/10.1177/0898264310366161.

31. Watkins I, Xie B. eHealth literacy interventions for older adults: a systematic review of the literature. J Med Internet Res. 2014;16(11):e225. https://doi.org/10.2196/jmir.3318.

32. van der Vaart R, Drossaert C. Development of the Digital Health Literacy Instrument: Measuring a Broad Spectrum of Health 1.0 and Health 2.0 Skills. J Med Internet Res. 2017;19(1):e27. https://doi.org/10.2196/jmir.6709.

33. van der Vaart R, van Deursen AJ, Drossaert CH, Taal E, van Dijk JA, van de Laar MA. Does the eHealth Literacy Scale (eHEALS) measure what it intends to measure? Validation of a Dutch version of the eHEALS in two adult populations. J Med Internet Res. 2011;13(4):e86. https://doi.org/10.2196/jmir.1840.

34. Mokkink LB, Prinsen C, Patrick DL, Alonso J, Bouter LM, De Vet H, et al. COSMIN methodology for systematic reviews of patient-reported outcome measures (PROMs): User manual. Amsterdam: Amsterdam Public Health research institute; 2018.

35. Mokkink LB, Terwee CB, Prinsen CA, de Vet HC, COSMIN Steering Committee. Taxonomy of measurement properties: A commentary on Polit (2015). Int J Nurs Stud. 2016;53:399-400. https://doi.org/10.1016/j.ijnurstu.2015.08.010.

36. Mokkink LB, de Vet HCW, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, et al. COSMIN risk of bias checklist for systematic reviews of patient-reported outcome measures. Qual Life Res. 2018;27(5):1171-9. https://doi.org/10.1007/s11136-017-1765-4.

37. Terwee CB, Prinsen CAC, Chiarotto A, Westerman MJ, Patrick DL, Alonso J, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res. 2018;27(5):1159-70. https://doi.org/10.1007/s11136-018-1829-0.

38. Kim H, Xie B. Health literacy in the eHealth era: a systematic review of the literature. Patient Educ Couns. 2017;100(6):1073-82. https://doi.org/10.1016/j.pec.2017.01.015.

39. Higgins JP, Green S. Cochrane handbook for systematic reviews of interventions, vol. 4. Chichester: Wiley; 2011.

40. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1-9. https://doi.org/10.1186/2046-4053-4-1.

41. Cochrane Canada: David Moher on crystal clear reporting of systematic reviews and EQUATOR Network. 2010 https://www.youtube.com/watch?v=TVFYenon1Jo.

42. Terwee CB, Jansma EP, Riphagen II, de Vet HC. Development of a methodological PubMed search filter for finding studies on measurement properties of measurement 434 instruments. Qual Life Res. 2009;18(8):1115-23. https://doi.org/10.1007/s11136-009-9528-5.

43. Endnote. https://endnote.com/. Accessed 30 June 2021.
44. Kim J, Jung H, Bates DW. History and trends of "personal health record" research in PubMed. Healthc Inform Res. 2011;17(1):3-17. https://doi.org/10.4258/hir.2011.17.1.3.

45. Bittlingmayer UH, Dadaczynski K, Sahrai D, van den Broucke S, Okan O. Digitale Gesundheitskompetenz-Konzeptionelle Verortung, Erfassung und Förderung mit Fokus auf Kinder und Jugendliche. Bundesgesundheitsbl Gesundheitsforsch Gesundheitsschutz. 2020;63(2):176-84. https://doi.org/10.1007/s00103-019-03087-6.

46. Covidence. https://www.covidence.org/. Accessed 30 June 2021.

47. Moher D, Liberati A, Tetzlaff J, Altman DG. The Prisma Group. Preferred reporting items for systematic reviews and metaanalyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. https://doi.org/10.1371/journal.pmed.1000097.

48. COSMIN. https://www.cosmin.nl/tools/guideline-conducting-systematic-review-outcome-measures/5525. Accessed 30 June 2021.

49. Terwee CB, Prinsen C, Chiarotto A, De Vet H, Bouter LM, Alonso J, et al. COSMIN methodology for assessing the content validity of PROMs: User manual. Amsterdam: VU University Medical Center; 2018.

50. Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60(1):34-42. https://doi.org/10.1016/j.jclinepi.2006.03.012.

51. Rodgers M, Sowden A, Petticrew M, et al. Testing methodological guidance on the conduct of narrative synthesis in systematic reviews: effectiveness of interventions to promote smoke alarm ownership and function. Evaluation. 2009;15(1):49-73. https://doi.org/10.1177/1356389008097871.

52. Prinsen CAC, Mokkink LB, Bouter LM, Alonso J, Patrick DL, de Vet HCW, et al. COSMIN guideline for systematic reviews of patient-reported outcome measures. Qual Life Res. 2018;27(5):1147-57. https://doi.org/10.1007/s11136-018-1798-3.

**Publisher's Note**

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Ready to submit your research? Choose BMC and benefit from:

* fast, convenient online submission

* thorough peer review by experienced researchers in your field

* rapid publication on acceptance

* support for research data, including large and complex data types

* gold Open Access which fosters wider collaboration and increased citations

* maximum visibility for your research: over 100M website views per year

At BMC, research is always in progress.

Learn more biomedcentral.com/submissions

<figure><img src="image_4.png" /></figure>