# Measuring digital health literacy and its associations with determinants and health outcomes in 13 countries

Diane Levin-Zamir, Stephan van den Broucke, Éva Bíró, Henrik Bøggild, Lucy Bruton, Saskia Maria de Gani, Hanne Søberg Finbråten, Sarah Gibney, Robert Griebler, Lennert Griese, et al.

<figure><img src="image_1.png" /></figure>

## To cite this version:

Diane Levin-Zamir, Stephan van den Broucke, Éva Bíró, Henrik Bøggild, Lucy Bruton, et al. Measuring digital health literacy and its associations with determinants and health outcomes in 13 countries. *Frontiers in Public Health*, 2025, 13, pp.1472706. 10.3389/fpubh.2025.1472706. inserm-05441718

HAL Id: inserm-05441718

https://inserm.hal.science/inserm-05441718v1

Submitted on 5 Jan 2026

**HAL** is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

Check for updates

## OPEN ACCESS

EDITED BY
Ann Borda,
The University of Melbourne, Australia

REVIEWED BY
María F. Rivadeneira,
Pontificia Universidad Católica del Ecuador,
Ecuador
Rojan Gümüş,
Dicle University, Türkiye

*CORRESPONDENCE
Diane Levin-Zamir
dlevin-za@univ.haifa.ac.il

RECEIVED 29 July 2024
ACCEPTED 07 January 2025
PUBLISHED 20 March 2025
CORRECTED 06 August 2025

**CITATION**
Levin-Zamir D, Van den Broucke S, Biró É, Bøggild H, Bruton L, De Gani SM, Søberg Finbråten H, Gibney S, Griebler R, Griese L, Guttersrud Ø, Klocháňová Z, Kucera Z, Le C, Link T, Mancini J, Miksova D, Schaeffer D, Ribeiro da Silva C, Sørensen K, Straßmayr C, Telo de Arriaga M, Vrdelja M and Pelikan J, on behalf of the HLS19 Consortium of the WHO Action Network M-POHL (2025) Measuring digital health literacy and its associations with determinants and health outcomes in 13 countries.
*Front. Public Health* **13**:1472706.
doi: **10.3389/fpubh.2025.1472706**

**COPYRIGHT**
© 2025 Levin-Zamir, Van den Broucke, Biró, Bøggild, Bruton, De Gani, Søberg Finbråten, Gibney, Griebler, Griese, Guttersrud, Klocháňová, Kucera, Le, Link, Mancini, Miksova, Schaeffer, Ribeiro da Silva, Sørensen, Straßmayr, Telo de Arriaga, Vrdelja and Pelikan, on behalf of the HLS19 Consortium of the WHO Action Network M-POHL. This is an open-access article distributed under the terms of the *Creative Commons Attribution License (CC BY)*. The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Measuring digital health literacy and its associations with determinants and health outcomes in 13 countries

Diane Levin-Zamir<sup>1,2*</sup>, Stephan Van den Broucke<sup>3</sup>, Éva Bíró<sup>4</sup>, Henrik Bøggild<sup>5</sup>, Lucy Bruton<sup>6</sup>, Saskia Maria De Gani<sup>7,8</sup>, Hanne Søberg Finbråten<sup>9</sup>, Sarah Gibney<sup>6</sup>, Robert Griebler<sup>10</sup>, Lennert Griese<sup>11</sup>, Øystein Guttersrud<sup>12,13</sup>, Zuzana Klocháňová<sup>14</sup>, Zdenek Kucera<sup>15</sup>, Christopher Le<sup>9,16</sup>, Thomas Link<sup>17</sup>, Julien Mancini<sup>18</sup>, Dominika Miksova<sup>17</sup>, Doris Schaeffer<sup>11</sup>, Carlota Ribeiro da Silva<sup>19</sup>, Kristine Sørensen<sup>20</sup>, Christa Straßmayr<sup>10</sup>, Miguel Telo de Arriaga<sup>19,21</sup>, Mitja Vrdelja<sup>22</sup> and Jürgen Pelikan<sup>10</sup>, on behalf of the HLS<sub>19</sub> Consortium of the WHO Action Network M-POHL

<sup>1</sup>School of Public Health, University of Haifa, Haifa, Israel, <sup>2</sup>Department of Health Education and Promotion, Clalit Health Services, Tel Aviv, Israel, <sup>3</sup>Psychological Sciences Research Institute, Université Catholique de Louvain, Louvain-la-Neuve, Belgium, <sup>4</sup>Department of Public Health and Epidemiology, Faculty of Medicine, University of Debrecen, Debrecen, Hungary, <sup>5</sup>Public Health and Epidemiology, Department of Health Science and Technology, Aalborg University, Aalborg, Denmark, <sup>6</sup>Department of Health, Dublin, Ireland, <sup>7</sup>Careum Foundation, Careum Center for Health Literacy, Zurich, Switzerland, <sup>8</sup>Careum School of Health, Kalaidos University of Applied Sciences, Zurich, Switzerland, <sup>9</sup>Department of Health and Nursing Sciences, Faculty of Social and Health Sciences, University of Inland Norway, Elverum, Norway, <sup>10</sup>Competence Centre Health Promotion and Healthcare, Austrian National Public Health Institute, Vienna, Austria, <sup>11</sup>School of Public Health, Bielefeld University, Bielefeld, Germany, <sup>12</sup>Department of Nursing and Health Promotion, Faculty of Health Sciences, Oslo Metropolitan University, Oslo, Norway, <sup>13</sup>The Norwegian Centre for Science Education Department, The Faculty of Mathematics and Natural Sciences, University of Oslo, Oslo, Norway, <sup>14</sup>Department of Public Health, Faculty of Health Care and Social Work, Trnava University Trnava, Trnava, Czechia, <sup>15</sup>Czech Health Literacy Institute, Prague, Czechia, <sup>16</sup>Department of Community Health, The Norwegian Directorate of Health, Oslo, Norway, <sup>17</sup>Department of Quality Measurement and Patient Survey, Austrian National Public Health Institute, Vienna, Austria, <sup>18</sup>Aix Marseille University APHM INSERM, IRD, ISSPAM, SESSTIM, Cancer, Biomedicine & Society Group, Marseille, France, <sup>19</sup>Direção-Geral da Saúde, Lisbon, Portugal, <sup>20</sup>Global Health Literacy Academy, Risskov, Denmark, <sup>21</sup>Católica Research Centre for Psychological, Family and Social Well-Being, Universidade Católica Portuguesa, Lisbon, Portugal, <sup>22</sup>Communication Unit, National Institute of Public Health, Ljubljana, Slovenia

**Introduction:** Digital health information sources are playing an increasingly prominent role in health promotion, public health and in healthcare systems. Consequently, digital health literacy skills are likewise becoming increasingly important.

**Methods:** Using a concept validation approach, the aim of the study was to validate the digital health literacy measure HLS<sub>19</sub>-DIGI, applied in the European Health Literacy Survey (2019–2021) of the WHO M-POHL network, analyzing data from 28,057 respondents from 13 countries. The instrument is a modified and extended version of the Digital Health Literacy Instrument (DHLI).

**Results:** The scale displayed high internal consistency. Confirmatory factor analysis (CFA) strengthened the hypothesized one-factor structure. In most countries, the data displayed acceptable fit to the unidimensional Rasch partial credit model (PCM).
Pearson correlation with a measure of general health literacy showed sufficient discriminant validity, and a social gradient was found. Testing for predictive validity showed that the scale score predicts health-related outcomes.

**Discussion:** The study shows that considerable proportions of the general adult populations across countries in Europe have limited DHL skills. The level of DHL has direct potential consequences for some forms of health service utilization, in some countries. Implications of the study include recommendations for improving digital health literacy, promoting organizational health literacy and quality assurance for digital health information and resources.

KEYWORDS

digital health literacy, eHealth literacy, HLS<sub>19</sub>, digital health literacy measurement, measurement scale validation, health information technology literacy, M-POHL

# 1 Introduction

The increasing availability and use of health-related digital resources such as electronic health records, telehealth initiatives, digital health applications, and interactive communication options with public health agencies and health care providers (e.g., for making appointments or reporting medical/test results), places increasing demands on peoples' abilities to use these applications and resources. Health care organizations and governments often promote or initiate and develop digital resources that require users to access, understand, appraise, and apply health information, as well as specific skills such as navigating digital health information. Simultaneously, the process of digitalization has enhanced the amount of online health information, as well as the number and variety of channels that are used for disseminating this information. More recently, artificial intelligence (AI) has entered the digital arena with tools such as ChatGPT and others. Due to the ubiquitous nature of digital communication, commercial companies and individuals are also seeking the public's attention through digital channels, including social media. As a result, more interest-driven, manipulative, or simply false information is disseminated to the public, requiring particularly critical and analytical skills from the public and individual users (1).

It is therefore incumbent upon researchers, practitioners, and policy makers to recognize and realize the importance of understanding and improving people's proficiency in using digital resources for managing their health conditions and for promoting their health. Concerns have been raised about whether increased digitalization of health services will help make health resources available for more people, or whether it will increase health inequalities by creating a digital divide (2). To answer this question, national population surveys should include validated measures of digital health literacy.

As a relatively new concept, digital health literacy (DHL) is distinct from digital literacy (3) in that it refers to a person's specific ability to process health-related information and re-sources obtained from digital sources. It is also distinct from eHealth literacy. While eHealth and digital health are often used interchangeably and are closely related, eHealth focuses mainly on healthcare, while DHL is broader and also refers to mHealth (mobile), artificial intelligence (AI) and other emerging areas of innovation and information technology (4).

## 1.1 Existing research and measures of digital health literacy

The need to assess people's use of electronic sources to access and process health-related information was first recognized by Norman and Skinner (5), who proposed the term eHealth literacy to refer to "the ability to seek, find, understand, appraise health information from electronic sources and apply the knowledge gained to addressing or solving a health problem." In accordance with this definition, Norman and Skinner developed the eHealth Literacy Scale (eHEALS), a self-report measure assessing a person's skills to find and evaluate health information on the Internet, which at the time was the primary online source for health information. Subsequently, eHealth literacy has been commonly assessed in health-related studies (6), and, together with media health literacy, has been conceptualized as the interplay of personal, situational, and contextual factors in the processing of health information from online sources and their interactions (2). A plethora of studies have examined media health literacy and eHealth literacy throughout the lifespan, from childhood and adolescence (7-9) through adulthood and among the older adult (10, 11). Additionally, it has been examined with respect to cultural transition (12), to specific health conditions (13), and to specific health behaviors (14), although studies on performance-based eHealth literacy are still scarce (15, 16). Empirically, general health literacy and eHealth literacy were found to be distinct but related concepts (17).

Yet, the rapid development of digital health sources beyond the Internet warrants new thinking about the definition and measurement of DHL regarding the use and application of digital health resources at the population level. While eHEALS has been used extensively to assess the self-reported consultation of online health information, the past decade has seen a proliferation of digital health information. This has broadened the potential for applying digital tools for health, but also the scope of digital skills needed to use these tools. These skills are at the core of the DHL concept. A recent study refers to DHL as the individual, social, technical, critical, and analytic skills that are required for searching, finding, understanding, evaluating, and applying digitally available health information (18). According to Bittlingmayer et al. (19), DHL refers not only to navigation in the digital space and to the use of digital information sources and their evaluation (e.g., health apps, social media, information sites on the Internet), but also to the individual "option spaces" that arise with
increasing digitalization and the related availability of
health information.

To capture these skills, a more comprehensive self-report scale (Digital Health Literacy Instrument- DHLI), which also addresses the interactive task of adding content, was developed, and validated in the Netherlands (16). While this scale captures various aspects of DHL, it has not been applied or field-tested in large samples, except for its use in an international cross-sectional survey among university students in the specific context of the COVID-19 pandemic (20).

## 1.2 Rational for a further developed digital health literacy measure

Valid and reliable knowledge about people's DHL can help health authorities to discern the experiences and difficulties of patients and of the wider population, including the extent to which they perceive health information provided through digital sources as apparent, accessible, understandable, correct, and applicable. Yet, while DHL has been assessed in some countries in the European Region (21, 22), these assessments mainly used the eHEALS or similar instruments. As such, they have not looked at DHL in a broader societal context, nor provided relevant associations with other aspects of health literacy or examined DHL in a broader international context. Apart from national studies conducted as part of the M-POHL Health Literacy Survey 2019–2021 (HLS₁₉) (23–25), no national health literacy survey to date has assessed and compared DHL to general health literacy on an international level. Moreover, there are no studies that have examined factors and covariates associated with DHL as determinants or its association with health promoting behavior, healthy lifestyles, early detection of diseases, use of health services, or self-care in the case of long-term illness.

The European Health Literacy Survey (HLS₁₉) under the auspices of the WHO Action Network on Measuring Population and Organizational Health Literacy (M-POHL) (26) offered an opportunity to further develop and validate a further developed measure for DHL, to report on DHL at the general population level, and to study its association with general health literacy, with sociodemographic, socioeconomic, and other variables and with possible health-related consequences and outcomes.

This article is part of a series of papers, introducing health literacy tools that have been developed, applied and tested through the HLS₁₉ study (27–30). The aim of this series is to use the data collected in the HLS₁₉ to examine the psychometric properties of the newly or further developed health literacy tools and different aspects of their validity. In order to derive overarching and comparable conclusions about the HLS₁₉ tools, the articles in this series address similar research questions and use the same analytical procedures.

# 2 Materials and methods

## 2.1 The HLS₁₉-DIGI instrument

The DHL instrument further developed for the HLS₁₉ survey, named HLS₁₉-DIGI, is based on the DHLI measure (16) but aligned more strongly with the concept and model of general health literacy

proposed by the HLS-EU study (27), and promoted by M-POHL. This model defines health literacy as the ability to access, understand, appraise, evaluate, and apply health information. Compared to the DHLI (16), the HLS₁₉-DIGI adds the dimension of understanding digitally accessed health information and eliminates the redundancy on the topic of applying health information. In addition, the scope and diversity of digital health resources was broadened to include use and interaction with social media, health apps, wearables, personal health records, and interaction with healthcare providers. Thus, the HLS₁₉ concept of DHL for promoting health includes the ability to search for, access, understand, appraise, validate, and apply online health information, and the ability to formulate and express questions, opinions, thoughts, or feelings when using digital devices.

HLS₁₉-DIGI consists of 10 items measuring two aspects of DHL: (1) the ability to search for, access, understand, appraise, and apply online health information measured by 8 items, namely HLS₁₉-DIGI-HI e.g.: "How easy or difficult is it for you to use the right search terms and queries to find the health information you are looking for?", or "...to judge whether the information you find online is reliable?" (Figure 1); and (2) the ability to clearly formulate questions, opinions, thoughts, or feelings when interacting by typing or posting information on a digital device (measured by 2 items, namely HLS₁₉-DIGI-INT: "How easy or difficult is it for you to clearly formulate your written message when communicating with a health provider," and "... to express your opinion, thoughts or feelings, ask a question in posting on social media including online forums"). In alignment with the format of other scales of the HLS₁₉ survey, respondents are asked to rate each item on a four-point rating scale ranging from 4 = very easy to 1 = very difficult where a higher score indicates higher DHL.

Six additional items assess the frequency with which one uses various digital sources of health information, including websites, social media, digital devices related to health or health care, or digital interaction with the health system (such as online video consultations, digital personal health records and prescriptions, health related apps on mobile phone, etc.). For each of these sources, respondents are asked to indicate how many days, in a typical week, they use them for obtaining health-related information, with response categories ranging from “less than once per week,” “1–3 days per week,” “4–6 days per week,” “once a day,” to “more than once per day.” A mean score ranging from 1 (“less than once per week” or not relevant) to 5 (“more than once per day”) is calculated as a relative measure for the frequency of use of health-related digital resources. We have adapted, shortened and extended the content of the DHLI tool (16), translated it into several languages and tested it, and this paper reports on the psychometric testing. The questionnaire was developed in English, and subsequently translated into 12 languages (Arabic, Czech, Danish, Dutch, French, German, Hebrew, Hungarian, Norwegian, Portuguese, Russian and Slovakian) by the HLS₁₉ national teams and data collection agencies. Two forward translations were performed by 10 countries [Austria, Belgium (Dutch translation), Switzerland (German translation), Germany, Denmark, Hungary, Italy, Norway, and Slovakia], and one forward translation into Czech and Portuguese. Back translation was performed for five languages (Czech, Hebrew, Arabic, Russian and Norwegian). Countries with common languages collaborated in the translation process. For the German version, one forward translation was conducted by the national researchers of the Austrian, German and Swiss team, and one by the German national data collection agency, after which both
<figure><img src="image_4.png" /><figcaption>FIGURE 1</figcaption></figure>

Percentage of respondents who responded “very difficult” or “difficult” to the items measuring the ability to access, understand, appraise and apply health information from digital sources (ranked by the overall percentage), for each country.

versions were compared, and consensus was reached. For the French version, a translation into French was made by the language service of the Swiss Federal Office of Public Health (FOPH), reviewed by different experts, and consent reached between the Swiss, French and Belgian teams.

## 2.2 Data collection

Data on DHL were collected as national surveys for HLS₁₉ in Austria (AT), Belgium (BE), Czech Republic (CZ), Denmark (DK), France (FR), Germany (DE), Hungary (HU), Ireland (IE), Israel (IL), Norway (NO), Portugal (PT), Slovakia (SK) and Switzerland (CH), resulting in a total of 28,057 respondents aged 18 and over, based on multi-stage random sampling or quota sampling procedures. Country specific sample sizes ranged from 1,000 to 4,487. Data collection involved a variety of methods and sampling procedures, including computer-assisted personal interviewing (CAPI), pen-and-paper personal interviewing (PAPI), computer-assisted telephone interviewing (CATI), and computer-assisted web interviewing (CAWI) (Table 1). Some countries applied different data collection methods for different subpopulations. Participation in a CAWI interview requires some familiarity with digital media and was used exclusively in BE, DK, and FR. In CH, CZ, and IL data were collected through CAWI for part of the sample, complemented by other methods.

The HLS₁₉ study protocol required national samples to be weighted by gender, age groups, population density, and geographical areas/ units, based on national census data to increase representativeness.

In addition to the 10 items of the HLS₁₉-DIGI and the six additional items on the use of digital health information sources, data on the following sociodemographic and socioeconomic variables were collected: gender, age (in years), self-perceived social status (from 1 = “lowest self-perceived social status” to 10 = “highest self-perceived social status”) (27), the highest level of completed education (lower secondary or less: ISCED 0–2; higher secondary: ISCED 3; above secondary: ISCED 4–8), employment status (employed and

unemployed or retired), and financial deprivation (“difficulty to pay all bills at the end of the month,” scored on a scale from 4 = “very easy” to 1 = “very difficult”) and whether or not the respondent has training in a health profession. To test for differential item functioning (DIF), we dichotomized variables on education level (ISCED 0–3 and 4–8) and social status (levels 1–4 and levels 5–10), and age categories were computed. More detailed information can be seen in Supplementary Table 1. Respondents were also asked to rate their self-reported general health on a scale ranging from very good to very bad, and to report limitations due to health problems (“severely limited”, “limited but not severely”, “not limited at all”), the existence of a long-term/chronic illness (lasting for 6 months or more: more than one, one, no) and their use of healthcare services (number of visits to GP/family doctor and number of specialist consultations in the past year) (13). General health literacy (GEN-HL) was measured, using the HLS₁₉-Q12 self-assessment scale (26), which is a 12-item revised short version of the HLS-EU-Q47 (27), providing for a comprehensive operationalization of health literacy by measuring the perceived difficulty to access, understand, appraise and apply health information in three health domains (healthcare, prevention and health promotion). A four-point rating scale (4 = “very easy,” 3 = “easy,” 2 = “difficult,” and 1 = “very difficult”) was offered for the HLS₁₉-Q12.

## 2.3 Data analysis

In a first step, perceived difficulties to access, understand, appraise, and apply online health information were analyzed for all countries by calculating the percentage of respondents who answered “very difficult” or “difficult” for each item. The scores of HL measures were calculated based on dichotomized as well as polytomous answer categories. To compensate for the potential resulting loss of information, item sets of categorical variables are also described by the average percentage response patterns (APRP), or the average percentage of how often a response category was selected within an item set. Given a data matrix consisting of m categorical items with k identical response categories for
<table><caption>TABLE 1 Data collection characteristics for digital health literacy by country.</caption><thead><tr><td>Country</td><td>Languages</td><td>Type of data collection</td><td>Sampling procedure</td><td>Period of data collection</td><td>Valid responses</td></tr></thead><tbody><tr><td>Austria</td><td>German</td><td>CATI</td><td>Multi-stage random sampling</td><td>16.03.2020-26.05.2020</td><td>2,967</td></tr><tr><td>Belgium</td><td>Dutch, French</td><td>CAWI</td><td>Quota sampling</td><td>30.01.2020-28.02.2020 and 01.10.2020-26.10.2020</td><td>1,000</td></tr><tr><td>Czech Republic</td><td>Czech</td><td>CATI, CAWI</td><td>Random digital procedure and random quota sampling</td><td>10.11.2020-24.11.2020</td><td>1,599</td></tr><tr><td>Denmark</td><td>Danish</td><td>CAWI</td><td>Multi-stage random sampling</td><td>11.12.2020-05.02.2021</td><td>3,602</td></tr><tr><td>France</td><td>French</td><td>CAWI</td><td>Quota sampling</td><td>08.01.2021-18.01.2021</td><td>1,000</td></tr><tr><td>Germany</td><td>German</td><td>PAPI</td><td>Multi-stage random and quota sampling</td><td>13.12.2019-27.01.2020</td><td>2,143</td></tr><tr><td>Hungary</td><td>Hungarian</td><td>CATI</td><td>Multi-stage random sampling</td><td>02.12.2020-20.12.2020</td><td>1,195</td></tr><tr><td>Ireland</td><td>English</td><td>CATI</td><td>Random digit dialing approach</td><td>24.07.2020-07.12.2020</td><td>4,487</td></tr><tr><td>Israel</td><td>Hebrew, Arab, Russian</td><td>CATI, CAWI</td><td>Multi-stage random sampling</td><td>15.12.2020-10.1.2021</td><td>1,315</td></tr><tr><td>Norway</td><td>Norwegian</td><td>CATI</td><td>Random sampling procedure within each stratum</td><td>04.04.2020-13.05.2020</td><td>2,855</td></tr><tr><td>Portugal</td><td>Portuguese</td><td>CATI</td><td>Random stratified sampling</td><td>10.12.2020-13.01.2021</td><td>1,247</td></tr><tr><td>Slovakia</td><td>Slovak</td><td>CAPI</td><td>Multi-stage random sampling</td><td>22.06.2020-14.09.2020</td><td>2,145</td></tr><tr><td>Switzerland</td><td>French, German, Italian</td><td>CAWI, CAWI</td><td>Multi-stage random sampling</td><td>05.03.2020-29.04.2020</td><td>2,502</td></tr></tbody></table>

n respondents, an APRP is a compositional measure consisting of k percentages that describe for each of the k response categories how often the n respondents selected the respective category on average when answering the m categorical items (26). APRP were calculated separately for the eight items measuring the ability to access, understand, appraise, and apply online health information, and for the two items measuring the ability to interact with digital health resources.

Post-stratification weights were applied to the univariate, bivariate, and regression analyses described below. The base weights were calculated by the national teams and differ depending on the survey procedure. For country-by-country analyses, the effective weights were scaled so that their sum equals the number of valid cases in a country dataset. For analyses across all participating countries, the weights are rescaled so that the sum of weights by country equals 1,000, i.e., the countries have equal weight.

Next, scale scores measuring each of the two dimensions of the HLS<sub>19</sub>-DIGI were calculated by combining the respondents' replies to the eight items related to accessing, understanding, appraising, and applying health information from digital sources HLS<sub>19</sub>-DIGI-HI, and separately the two items measuring the difficulty to interact with digital sources HLS<sub>19</sub>-DIGI-INT. Following Pelikan et al. (27), two types of scores are calculated and compared with respect to their performance. First, the

“d-type score,” which is based on a count of the dichotomized items obtained by combining the “easy” and “very easy” and “difficult” and “very difficult” categories. The advantage of dichotomization is that it reduces the potential inequalities due to extreme responses in subpopulation which could benefit for example comparisons within countries or international comparisons, besides being easier to communicate the meaning of the score. Yet the disadvantage of dichotomization may be the loss of information. In order to address this issue, “p-type” scores were also calculated, as the mean of the numeric values ranging from 1 to 4. Both scores are scaled to the range of 0 to 100. The higher the score, the higher the level of DHL. Scores were computed only for respondents who had answered at least 80% of the items.

Spearman correlations were calculated among the eight items measuring the first DHL dimension (HLS<sub>19</sub>-DIGI-HI = i.e., the ability to search for, access, understand, appraise, and apply online health information), for the dichotomized items for all countries. Internal consistencies of the scales were tested for all countries using the Cronbach alpha coefficient, and the unidimensionality of the scales was tested using confirmative factor analysis (CFA) and Rasch analyses, based only on the polytomous items, respectively. For the latter, the data per country were tested against the unidimensional Rasch Partial Credit Model (PCM) (28). For each country, overall data-model fit, single item
fit, the ordering of response categories, response dependency, one-dimensionality, and differential item functioning (DIF) were evaluated using an amended sample size of *n* = 480, corresponding to 20 respondents for each of 24 thresholds or *n* = 240 (10 respondents per threshold) for certain countries (28). These analyses were only performed for the first dimension (eight-item) of the tool (HLS₁₉-DIGI-HI), as it would not be appropriate to perform them on the two-item scale measuring the difficulty of interacting with digital sources (HLS₁₉-DIGI-INT). A detailed description of the methodological approach can be found in Griese et al. (29) and Finbraten et al. (30).

Finally, to investigate the determinants and health outcomes of DHL, the association of the HLS₁₉-DIGI-HI subscale with sociodemographic and socioeconomic variables, general health literacy as measured by the HLS₁₉-Q12, perceived health status and health care use was assessed by calculating Spearman's ρ correlations and multiple linear regression analysis. Variables included in the regression models were: age, education (ISCED 0–8), self-perceived social status (1–10), financial deprivation, self-reported general health status, GP visits and specialist consultations.

# 3 Results

## 3.1 Use of digital information sources for health

Table 2 shows the proportion of responses, for all countries combined, regarding the use of different digital sources to obtain health information. Among those using digital resources for health, on average, websites are the most frequently consulted digital source on health information, followed by social media, digital interaction with the health system, a digital device related to health or health care, a health app on a mobile phone, and other.

## 3.2 Perceived difficulty to process digital health information

Figure 1 shows, for each item, the percentage of respondents who replied “very difficult” or “difficult” to the eight items measuring the ability to access, understand, appraise, and apply health information from digital sources, for each country (weighted equally). Percentage

distributions for all four categories, for each item, are available in the Supplementary material of the International Report of the HLS₁₉ Consortium (26). Although there is a considerable variation in perceived difficulty across countries, the ranking of the combined two difficulty response categories shows similar patterns across countries, with few exceptions. The percentages of combined “difficult” and “very difficult” responses range from 21.8% for item 1 (“to use the proper words or search query to find the information you are looking for”) to 54.1% for item 4 (“to judge whether the information is reliable”). For both items, the percentage of respondents finding these tasks (very) difficult is the lowest in Norway (9.5 and 30.6%) and highest in Germany (38.5 and 82.6%, respectively).

Figures 2, 3 show the APRP for the two DHL dimensions of the HLS₁₉-DIGI (i.e., the ability to access, understand, appraise, and apply online health information, and to interact with digital devices related to health, respectively) for each country. On average, nearly 40% of the respondents across all countries reported difficulties in processing digital health information (i.e., they responded “difficult” or “very difficult”), with Norway having the lowest proportion (22%) and Germany the highest (58%). Figure 3 shows that, on average, more than a quarter of respondents (29%) reported difficulties interacting with digital devices for health, but again, the numbers vary by country. Respondents from Portugal reported the lowest difficulties (11%) and respondents from Slovakia the highest (48%). In contrast, 51% of respondents experience interacting with digital devices for health as easy, with scores varying between 35% in Ireland and 78% in Portugal, and 20% as very easy, with scores varying from 10% in Slovakia to 38% in Norway.

## 3.3 Psychometric properties of the HLS₁₉-DIGI scales

Spearman correlations among the eight items measuring the HLS₁₉-DIGI-HI dimension of the HLS₁₉-DIGI (i.e., the ability to access, understand, appraise, and apply online health information) ranged between 0.37 and 0.66 for all countries combined. The same levels of correlation are found for individual countries (26).

Cronbach's alpha (α) for the HLS₁₉-DIGI-HI items are above 0.70 for all countries, with an average α of 0.83 for the dichotomized items and 0.89 for the polytomous items (Table 3), indicating a reliable scale. The two HLS₁₉-DIGI-INT items were highly correlated.

<table><caption>TABLE 2 Use of different digital resources to obtain health information (% of respondents for all 13 countries combined).</caption><thead><tr><td>Type of digital resources</td><td>Not relevant / do not know</td><td>&lt; once per week</td><td>1–3 days per week</td><td>4–6 days per week</td><td>Daily</td><td>More than once per day</td></tr></thead><tbody><tr><td>Internet (websites)</td><td>26.9</td><td>47.7</td><td>15.1</td><td>3.2</td><td>3.2</td><td>3.9</td></tr><tr><td>Social Media including online forums</td><td>42.6</td><td>39.3</td><td>9.0</td><td>2.6</td><td>3.1</td><td>3.4</td></tr><tr><td>A digital device related to health or health care</td><td>49.8</td><td>24.7</td><td>7.1</td><td>4.6</td><td>7.0</td><td>6.8</td></tr><tr><td>Health app on your mobile phone</td><td>52.6</td><td>24.8</td><td>7.2</td><td>3.9</td><td>7.0</td><td>4.5</td></tr><tr><td>Digital interaction with your health system</td><td>48.3</td><td>43.9</td><td>4.7</td><td>1.2</td><td>1.1</td><td>0.9</td></tr><tr><td>Other</td><td>70.7</td><td>24.2</td><td>2.2</td><td>1.0</td><td>0.9</td><td>0.9</td></tr></tbody></table>
<figure><img src="image_5.png" /><figcaption>FIGURE 2
Average Percentage Response Patterns (APRP) for the item set measuring the ability to access, understand, appraise, and apply online health information, for each country (equally weighted).</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>FIGURE 3
Average Percentage Response Patterns (APRP) for item set measuring the ability to interact with digital health devices, for each country (equally weighted).</figcaption></figure>

On average, the HLS₁₉-DIGI-HI and DIGI-INT scores are moderately correlated (with a Pearson correlation coefficient of *r* = 0.48 for the d-type score and *r* = 0.55 for the p-type score), but the level of correlation differs considerably across countries (from 0.21 to 0.60 for the d-type score and 0.24 to 0.68 for the p-type score).

Table 3 shows the fit indices for the single-factor confirmatory factor model for the HLS₁₉-DIGI-HI scale. The indices generally indicate an acceptable fit for all countries, although the standardized root mean square residual (SRMSR, assuming a 0.08 threshold value) is too high in three countries and the lower bound of the confidence interval of the root mean square error of approximation (RMSEA, with 0.05 as threshold) is too high in all countries, suggesting a possible misfit between the observed data and the single-factor model. However, the other fit indices (comparative fit index, Tucker-Lewis index, goodness of fit and adjusted goodness of fit index) overall indicate a sufficient fit between the observed covariance matrix and the model implied covariance matrix for all countries.

Using an amended sample size of *n* = 480 the data showed an overall sufficient fit to the Rasch model (χ² statistic) for data collected in Austria, Germany, Ireland, Norway, and Switzerland. We observed

acceptable fit for the data collected in Belgium and in the Czech Republic, and, with a further reduced sample size of *n* = 240, also for the data collected in Denmark, Hungary, Israel, and Portugal. A principal component analysis (PCA) of Rasch model residuals, combined with dependent t-tests to identify possible subscales, revealed that the HLS₁₉-DIGI-HI scale could be considered sufficiently unidimensional for most countries. The HLS₁₉-DIGI-HI scale was also considered well-targeted in most countries. The thresholds, and thus the response categories, were ordered and functioned well.

There is no significant statistical dependency between pairs of items, which means that no items are “too similar” and therefore redundant. However, item 5 “to judge whether health-related information is offered with commercial interests” tends to under-discriminate in most countries and thus does not strictly correspond to the latent trait that is underlying the scale. Some items displayed DIF for person factors, such as age and gender, but there was no consistent pattern across countries. Furthermore, the scale did not measure invariantly across countries, as the item threshold locations or item “difficulty order” varied between countries. The latter may be ascribed to DIF for country or language.

## 3.4 Content and concurrent validity of the HLS₁₉-DIGI scales

Both scales of the HLS₁₉-DIGI are supported by a theory-based model that justifies the selection of items for the measures. Table 4 shows the distribution of the DHL score measuring the ability to access, understand, appraise, and apply digital health information for each country. For the d-type score, the mean score across all countries (equally weighted) is 62.3 (on a scale from 0 to 100), varying between 41.8 (DE) and 78.7 (NO). For most countries, the distribution is left skewed with the 75% percentile starting at the maximum value, indicating a ceiling effect for the measure. For the p-type score, the mean score across all countries (equally weighted) is 56.6 (on a scale from 0–100), varying between 44.1 (DE) and 68.6 (NO). The p-scores are less skewed than the d-scores.

In terms of concurrent validity, Pearson correlations between the ability to access, understand, appraise, and apply digital health information measured by the HLS₁₉-DIGI-HI and general health literacy measured by the HLS₁₉-Q12 varies, for the d-type scores, between 0.44 (BE) and 0.67 (IL), with a mean of 0.53. For the p-type scores, the correlation coefficients range from 0.52 (BE, CH, HU) to 0.72 (IL) with a mean of 0.58. This confirms the assumption that both concepts are strongly related, but also sufficiently distinct to warrant separate measures. Across countries, the Spearman correlation between the scale and a score for the use of digital resources is ρ = 0.15, ranging from 0.04 (for BE) to 0.31 (for DE). So, for most countries, the perceived ease of using digital health information sources is moderately related to the use of such sources.

## 3.5 Determinants of DHL

Correlational analyses indicate that in most HLS₁₉ countries the score on the HLS₁₉-DIGI-HI scale measuring the ability to access, understand, appraise and apply digital health information is moderately correlated with having had training in a health profession (Spearman ρ = 0.14 for all countries, varying between 0.03 for AT and
<table><caption>TABLE 3 Internal consistencies (Cronbach alpha) and fit indices for the one-factor confirmatory factor model for HL-DIGI-8 items, for each country and mean of all countries (equally weighted).</caption><thead><tr><th>Fit index (Threshold value)</th><th>AT</th><th>BE</th><th>CH</th><th>CZ</th><th>DE</th><th>DK</th><th>FR</th><th>HU</th><th>IE</th><th>IL</th><th>NO</th><th>PT</th><th>SK</th><th>Mean</th></tr></thead><tbody><tr><td colspan="15">Dichotomized items</td></tr><tr><td>Cronbach α</td><td>0.81</td><td>0.86</td><td>0.85</td><td>0.82</td><td>0.83</td><td>0.86</td><td>0.86</td><td>0.79</td><td>0.79</td><td>0.83</td><td>0.77</td><td>0.83</td><td>0.87</td><td>0.83</td></tr><tr><td>SRMSR</td><td>0.07</td><td>0.12</td><td>0.08</td><td>0.07</td><td>0.07</td><td>0.09</td><td>0.06</td><td>0.10</td><td>0.06</td><td>0.08</td><td>0.07</td><td>0.07</td><td>0.07</td><td>0.08</td></tr><tr><td>RMSEA</td><td>0.08</td><td>0.13</td><td>0.09</td><td>0.07</td><td>0.08</td><td>0.11</td><td>0.07</td><td>0.10</td><td>0.06</td><td>0.08</td><td>0.06</td><td>0.08</td><td>0.08</td><td>0.08</td></tr><tr><td>RMSEA (p)</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.06</td><td>0.00</td><td>0.05</td><td>0.00</td><td>0.00</td><td>0.01</td></tr><tr><td>CFI</td><td>0.98</td><td>0.98</td><td>0.99</td><td>0.99</td><td>0.99</td><td>0.98</td><td>0.99</td><td>0.96</td><td>0.99</td><td>0.99</td><td>0.98</td><td>0.99</td><td>0.99</td><td>0.98</td></tr><tr><td>TLI</td><td>0.97</td><td>0.97</td><td>0.98</td><td>0.98</td><td>0.98</td><td>0.98</td><td>0.99</td><td>0.94</td><td>0.98</td><td>0.98</td><td>0.98</td><td>0.98</td><td>0.99</td><td>0.98</td></tr><tr><td>GFI</td><td>0.98</td><td>0.97</td><td>0.99</td><td>0.98</td><td>0.99</td><td>0.98</td><td>0.99</td><td>0.97</td><td>0.99</td><td>0.99</td><td>0.99</td><td>0.99</td><td>0.99</td><td>0.98</td></tr><tr><td>AGFI</td><td>0.97</td><td>0.95</td><td>0.98</td><td>0.97</td><td>0.98</td><td>0.97</td><td>0.98</td><td>0.94</td><td>0.98</td><td>0.97</td><td>0.98</td><td>0.98</td><td>0.98</td><td>0.97</td></tr><tr><td colspan="15">Polytomous items</td></tr><tr><td>Cronbach α</td><td>0.89</td><td>0.91</td><td>0.91</td><td>0.88</td><td>0.91</td><td>0.92</td><td>0.89</td><td>0.84</td><td>0.86</td><td>0.89</td><td>0.87</td><td>0.89</td><td>0.92</td><td>0.89</td></tr><tr><td>SRMSR</td><td>0.05</td><td>0.09</td><td>0.07</td><td>0.06</td><td>0.06</td><td>0.07</td><td>0.06</td><td>0.10</td><td>0.04</td><td>0.07</td><td>0.06</td><td>0.05</td><td>0.05</td><td>0.06</td></tr><tr><td>RMSEA</td><td>0.10</td><td>0.18</td><td>0.14</td><td>0.10</td><td>0.11</td><td>0.15</td><td>0.12</td><td>0.17</td><td>0.07</td><td>0.12</td><td>0.11</td><td>0.10</td><td>0.12</td><td>0.12</td></tr><tr><td>RMSEA (p)</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.05</td><td>0.00</td><td>0.00</td><td>0.00</td></tr><tr><td>CFI</td><td>0.99</td><td>0.98</td><td>0.99</td><td>0.99</td><td>0.99</td><td>0.99</td><td>0.99</td><td>0.96</td><td>0.99</td><td>0.99</td><td>0.98</td><td>0.99</td><td>0.99</td><td>0.99</td></tr><tr><td>TLI</td><td>0.99</td><td>0.98</td><td>0.99</td><td>0.98</td><td>0.99</td><td>0.99</td><td>0.98</td><td>0.94</td><td>0.99</td><td>0.98</td><td>0.98</td><td>0.99</td><td>0.99</td><td>0.98</td></tr><tr><td>GFI</td><td>0.99</td><td>0.99</td><td>0.99</td><td>0.99</td><td>0.99</td><td>0.99</td><td>0.99</td><td>0.97</td><td>0.99</td><td>0.99</td><td>0.99</td><td>0.99</td><td>0.99</td><td>0.99</td></tr><tr><td>AGFI</td><td>0.98</td><td>0.96</td><td>0.98</td><td>0.98</td><td>0.98</td><td>0.98</td><td>0.98</td><td>0.93</td><td>0.98</td><td>0.97</td><td>0.97</td><td>0.99</td><td>0.99</td><td>0.97</td></tr></tbody></table>

AT, Austria; BE, Belgium; CH, Switzerland; CZ, Czech Republic; DE, Germany; DK, Denmark; FR, France; HU, Hungary; IE, Ireland; IL, Israel; NO, Norway; PT, Portugal; SK, Slovakia; Cronbach α (> 0.7); SRMSR (≤ 0.08), standardized root-mean square residual; RMSEA (≤ 0.06), root-mean-square error of approximation; CFI (≥ 0.95), comparative fit index; TLI (≥ 0.95).
<table><caption>TABLE 4 Mean, median, quartiles, and standard deviation of the HLS19-DIGI-HI score, for each country and for all countries (equally weighted).</caption><thead><tr><th></th><th>Mean</th><th>SD</th><th>25th percentile</th><th>Median</th><th>75th percentile</th></tr></thead><tbody><tr><td colspan="6"><strong>Dichotomized items</strong></td></tr><tr><td>AT</td><td>70.1</td><td>29.2</td><td>50.0</td><td>75.0</td><td>100.0</td></tr><tr><td>BE</td><td>54.9</td><td>34.6</td><td>25.0</td><td>50.0</td><td>87.5</td></tr><tr><td>CH</td><td>55.7</td><td>33.0</td><td>25.0</td><td>62.5</td><td>87.5</td></tr><tr><td>CZ</td><td>59.3</td><td>31.4</td><td>37.5</td><td>62.5</td><td>87.5</td></tr><tr><td>DE</td><td>41.8</td><td>31.6</td><td>12.5</td><td>37.5</td><td>62.5</td></tr><tr><td>DK</td><td>67.2</td><td>32.2</td><td>37.5</td><td>75.0</td><td>100.0</td></tr><tr><td>FR</td><td>53.9</td><td>33.5</td><td>25.0</td><td>50.0</td><td>87.5</td></tr><tr><td>HU</td><td>71.8</td><td>26.9</td><td>62.5</td><td>75.0</td><td>100.0</td></tr><tr><td>IE</td><td>62.1</td><td>30.2</td><td>37.5</td><td>62.5</td><td>87.5</td></tr><tr><td>IL</td><td>62.7</td><td>31.0</td><td>37.5</td><td>62.5</td><td>87.5</td></tr><tr><td>NO</td><td>78.7</td><td>24.7</td><td>62.5</td><td>87.5</td><td>100.0</td></tr><tr><td>PT</td><td>74.0</td><td>28.7</td><td>57.1</td><td>85.7</td><td>100.0</td></tr><tr><td>SK</td><td>60.3</td><td>34.2</td><td>37.5</td><td>62.5</td><td>100.0</td></tr><tr><td>All</td><td>62.3</td><td>32.5</td><td>37.5</td><td>62.5</td><td>100.0</td></tr><tr><td colspan="6"><strong>Polytomous items</strong></td></tr><tr><td>AT</td><td>62.9</td><td>19.7</td><td>50.0</td><td>62.5</td><td>76.2</td></tr><tr><td>BE</td><td>52.0</td><td>21.2</td><td>37.5</td><td>54.2</td><td>66.7</td></tr><tr><td>CH</td><td>52.3</td><td>20.0</td><td>41.7</td><td>54.2</td><td>66.7</td></tr><tr><td>CZ</td><td>54.6</td><td>18.7</td><td>41.7</td><td>54.2</td><td>66.7</td></tr><tr><td>DE</td><td>44.1</td><td>21.6</td><td>33.3</td><td>45.8</td><td>58.3</td></tr><tr><td>DK</td><td>59.5</td><td>19.4</td><td>45.8</td><td>58.3</td><td>70.8</td></tr><tr><td>FR</td><td>51.0</td><td>18.9</td><td>37.5</td><td>50.0</td><td>66.7</td></tr><tr><td>HU</td><td>59.6</td><td>14.7</td><td>54.2</td><td>62.5</td><td>66.7</td></tr><tr><td>IE</td><td>58.9</td><td>21.9</td><td>45.8</td><td>58.3</td><td>75.0</td></tr><tr><td>IL</td><td>57.8</td><td>19.7</td><td>45.8</td><td>58.3</td><td>66.7</td></tr><tr><td>NO</td><td>68.6</td><td>17.0</td><td>58.3</td><td>66.7</td><td>79.2</td></tr><tr><td>PT</td><td>59.6</td><td>14.5</td><td>50.0</td><td>62.5</td><td>66.7</td></tr><tr><td>SK</td><td>55.4</td><td>20.6</td><td>41.7</td><td>54.2</td><td>66.7</td></tr><tr><td>All</td><td>56.6</td><td>20.2</td><td>45.8</td><td>58.3</td><td>66.7</td></tr></tbody></table>

AT, Austria; BE, Belgium; CH, Switzerland; CZ, Czech Republic; DE, Germany; DK, Denmark; FR, France; HU, Hungary; IE, Ireland; IL, Israel; NO, Norway; PT, Portugal; SK, Slovakia.

0.18 for SK). The same applies to self-perceived social status in society ($\rho$ = 0.13, ranging from 0.00 in AT to 0.25 in SK), age ($\rho$ = -0.11, ranging from 0.03 in IL to -0.25 in DE), and education level ($\rho$ = 0.07, varying from 0.03 in the CZ to 0.31 in SK). The relationship with education is more or less linear, meaning that the score on the HLS19-DIGI-HI scale increases with educational level. In contrast, the correlation with gender (overall $\rho$ = 0.01) is very low for all countries.

Supplementary Table 2 shows the deviation from the population mean d-score for the scale measuring the ability to access, understand, appraise, and apply digital health information for selected subpopulations that are potentially vulnerable or disadvantaged regarding digital health literacy. Notably, older people show, on average, the largest deviation from the general population (-12), followed by those with a (very) bad self-perceived health (-11.4), financial deprivation (-8.7), low education (-7.8), and low social status (-6.8). Those who often visit their GP/family doctor also score markedly lower (-6.9). A somewhat lower level of deviation from the general population score is seen for people with limitations due to

health problems (-4.7) and for those suffering from a long-term/chronic illness (-3.2). For most subpopulations the variation between countries is considerable.

To identify possible associations among the determinants of DHL, a series of regression models (Supplementary Table 3) were tested with the 8-item HLS19-DIGI-HI scale as the dependent variable. A first model including the core social variables of age, gender, education level, self-perceived social status and financial deprivation as independent variables explains 6% of the variance in DHL for all countries weighted equally, with the explained variance varying from 2% (BE) to 23% (SK). The best predictor of HLS19-DIGI-HI for all countries is financial deprivation (overall $\beta$ = -0.15; significant for 11 countries with $\beta$ values between -0.08 and -0.27), followed by age (overall $\beta$ = -0.13; significant for 6 countries with $\beta$ values between -0.15 and -0.26) and self-perceived social status (overall $\beta$ = 0.08; significant for 10 countries with $\beta$ values between 0.05 and 0.13). Education and gender have, on average, a smaller effect on the level of DHL, with lower overall $\beta$ values (0.03) and a significant effect in only
a few countries. As such, the results suggest a social gradient for DHL, albeit with an inconsistent pattern across countries. A second model (not shown in the tables), in which use of digital resources is added as an independent variable, gives a small increase of the explained variance to 7% (ranging between 2% in BE and 24% in SK). Again, financial deprivation (overall $\beta$ = -0.14), age (overall $\beta$ = -0.12) and social status (overall $\beta$ = 0.08) are the strongest predictors, along with use of digital sources (overall $\beta$ = 0.11), while gender and education level have little influence (overall $\beta$ = 0.02).

## 3.6 Outcomes of DHL

The importance of DHL for health can be inferred from its relationship with a number of health outcomes. Significant Spearman correlations are found between the scores on the HLS₁₉-DIGI-HI scale measuring the ability to access, understand, appraise, and apply digital health information and self-perceived health ($\rho$ = -0.18 for all countries, with $\rho$ varying between -0.10 in IE and -0.28 for DE), limitations due to illness or health problems ($\rho$ = 0.18 for all countries), and long-term illness or health problems ($\rho$ = -0.11 for all countries).

The contribution of DHL to health outcomes can be derived from a series of multiple regression analyses using the HLS₁₉-DIGI-HI score in combination with gender, age, education, self-perceived social status, and financial deprivation as independent variables (Supplementary Table 4). When self-perceived health is entered as the outcome variable, the model explains 15% of the variance for all countries combined ($R^2$ ranging between 10% in IE and 26% in SK). Although age (overall $\beta$ = 0.24, ranging from 0.07 to 0.38, significant for all countries except BE) and self-perceived social status (overall $\beta$ = -0.17, ranging from -0.29 to -0.09, significant for all countries) are stronger predictors of self-perceived health, DHL also significantly contributes to the regression in nine out of the 13 countries participating in the study. The overall $\beta$ = -0.10, with $\beta$ values for individual countries ranging from -0.05 to -0.14.

However, when more objective outcome measures are considered such as the reported number of GP/family doctor visits or specialist consultations, DHL becomes less important as a determinant. For GP/family doctor's visits, the model explains 6% of the variance for all countries combined ($R^2$ ranging between 4% in IL and 14% in DE). On par with financial deprivation (average $\beta$ = 0.08, significant for 10 countries with $\beta$ between -0.01 and 0.17), DHL is the third most important contributor to the regression model, with $\beta$ values ranging from -0.01 to -0.10 (overall $\beta$ = -0.08). For specialist consultations, the variance explained by the model is only 5% ($R^2$ ranging between 1 and 11%), and DHL's contribution to the regression is only significant in SK ($\beta$ = -0.08) and an overall $\beta$ of -0.07. Thus, the level of DHL has direct potential consequences for some forms of health service utilization, in some countries.

## 4 Discussion

DHL is gaining momentum in public health and health service research, policy and practice. The rapid development of digital health information sources and resources beyond the Internet has widened

the potential for the digitalization of health services, but also created a need to address the necessary digital skills of individuals to use these resources. Aligned with these developments, DHL refers to the individual, social, technical, critical, and analytic skills that a person needs in order to find, access, understand, evaluate, and apply digitally available health information (18). While existing research on DHL tends to take a narrow view of the topic, limiting it to finding and evaluating health-related information on the Internet, we detected a need to examine the subject more broadly in terms of the skills needed to navigate the digital space, access and use digital information sources such as health apps including wearables, social media, information websites, evaluate them, and take decisions on one's health based on the options offered. A systematic bibliometric analysis on DHL did not take into consideration the validation of psychometrics and the sub-skills embedded in the term DHL (31).

This article described the conceptual background, development, and validation of a further developed instrument to measure DHL at the population level and investigated its determinants and associations with health outcomes. The instrument, named HLS₁₉-DIGI, builds on the conceptual model, definition and dimensions of general health literacy proposed by the HLS-EU consortium (32) to measure the perceived ability of respondents to find, understand, appraise, and apply specifically digitally available health related information, and interact with digital information sources. Its validation is based on data of the European Health Literacy Survey 2019-2021 (HLS₁₉) undertaken by the WHO Action Network on Measuring Population and Organizational Health Literacy (M-POHL) (26), involving approximately 28,000 participants in 13 countries in measuring DHL.

The instrument consists of 10 items measuring two aspects of DHL: the ability to access, understand, appraise/validate, and apply online health information (HLS₁₉-DIGI-HI), and the ability to clearly formulate questions, opinions, thoughts, or feelings when interacting by typing or posting information on a digital device (HLS₁₉-DIGI-INT). With regard to the first aspect, our results demonstrate that the 8-item scale measuring this dimension is sufficiently robust and valid. The internal consistency, confirmatory factor analysis and Rasch analysis confirmed its unidimensional structure for the data of all countries. The reliability of the scale, based on Cronbach's alpha, is sufficiently high, with values ranging from 0.77-0.87 for the d-score and 0.84-0.92 for the p-score. The thresholds, and thus the response categories, are ordered and well-functioning. The correlation with general health literacy is high but not extreme ($r$ = 0.55 for p-type scores and 0.48 for d-type scores), suggesting that both scales measure parts of the same construct, but are independent enough to be treated as different entities. One item tends to discriminate somewhat poorly across countries and some items display DIF for country or language, which could limit international comparisons. Future research could investigate whether minor changes to items like, e.g., 6 “to visit different websites to check whether they provide similar information about a topic” or 5 “to judge whether the information is offered with commercial interests” (Figure 1) could improve the model fit. Also, in some countries, based on the d-type score the scale displays a ceiling effect, with more than 25% scoring the highest value.

Regarding the second component (HLS₁₉-DIGI-INT), the two items measuring difficulties in clearly formulating questions, opinions, thoughts or feelings when posting information on a digital device, are highly correlated, and, in some countries, do not differentiate well, suggesting that the items should be adjusted. The two items are, on
average, moderately correlated (d-type score *r* = 0.48 and p-type score *r* = 0.55), but this differs considerably between countries. In the next M-POHL Health Literacy Survey 2024–2026 (HLS₂₄), a refined measure may be considered for this aspect of DHL.

Using the HLS₁₉-DIGI-HI, it was possible to document the perceived difficulty to process digital health information at the population level. Thus, it was seen that a significant part of the population – between 22 and 58%, depending on the country – finds it difficult to find, access, understand, evaluate, and apply digitally available health information. On average, 8% of the respondents (percentages ranging from 2 to 18% for the different countries) found it very difficult, and another 25% found it difficult (percentages ranging from 20 to 40%).

The most difficult tasks are to judge/appraise whether the information is offered with commercial interests (item 5), or whether information is reliable (item 4). Using proper words or search queries to find information one is seeking or visiting different websites to check whether they provide similar information is considered less difficult. This resonates with results from other (health) literacy research that with the ubiquitous stream of information in today's society, the challenge is less to find information, but rather to discern whether it is correct, complete, and useful, aligned with the concept of critical health literacy (33).

In terms of its determinants, a social gradient for DHL is seen for all countries, but to a considerably different degree for different countries and indicators. The strongest social predictors of DHL are, on average, financial deprivation, age, and self-perceived social status in society. Perceived social status has acknowledged as a contribution to other aspects in a population's social gradient (34). Education and gender are less significant, and only in some countries. Regarding education, ISCED levels were handled as a continuous variable. Different ISCED levels refer to different intervals of “years of education,” and these intervals also differ somewhat across countries. Thus, in future studies the measure may best be continuous such as number of years in an educational setting. General health literacy is a very strong predictor of DHL for all countries, and use of digital resources for most countries. Older people, people with poor health, and people with a low education level are most probable to have low DHL skills, which confirms the findings of other studies (16). This finding supports the conclusion of studies that have stressed the importance of interventions that address DHL in socially disadvantaged groups (35).

Regarding its importance for health and health care use, it was seen that DHL correlates with self-perceived health and limitations due to long-term illness, and that in most countries it is a determinant of self-perceived health along with age and self-perceived social status, but that its impact on healthcare use (i.e., GP visits and specialist consultations) is generally less strong.

Future analyses based on the study can examine the association between DHL and health behavior indicators measured in HLS₁₉ (smoking, physical activity, BMI, alcohol, fruit and vegetable consumption).

## 4.1 Limitations

Differences in sampling and data collection limit the extent of country comparisons. Web-based surveys may have biased the results toward over-reporting of high DHL. For some countries, the

non-response rates for several items were markedly higher than for other HLS₁₉ measures, which may be partly due to the fact that people cannot evaluate something they do not do in everyday life, namely if they do not use the Internet to search for health information. As a result, the results may not be representative of all country samples. For example, in Ireland (IE), the group with missing data have a higher mean age than the total sample and a lower mean level of education. Furthermore, some of the countries collected data prior to the COVID-19 pandemic, others during the pandemic and some following the major pandemic waves. This diversity may reduce the extent to which participants in different countries can be compared, since the pandemic may have influenced the extent to which populations became digitally health literate. Yet, the measures across countries seems to behave rather uniformly, despite different data collecting modes, and the general picture of associations between determinants and DHL and with outcome measures also suggest that our conclusions are well supported. As is the case in all cross-sectional surveys association between variables does not infer causation. The next cycle of the M-POHL HLS survey can consider ways to mitigate these limitations.

## 5 Conclusions and implications

A compact, conceptually sound instrument to measure DHL was validated for 13 languages in 13 countries, showing acceptable psychometric properties. The study shows that considerable proportions of the general adult populations across 13 countries in Europe have limited DHL skills, that the level of DHL shows a social gradient, and that it is associated with indicators of health status. Professionals and policy makers in public health, health promotion and healthcare services should recognize the difficulties that large groups of society, especially the most vulnerable, have in accessing, understanding, appraising, validating and applying health information from digital sources and interacting with digital services. This challenge will become more critical as healthcare systems continue to transition into digital avenues of communication and information sharing with patients and communities (36). Addressing these difficulties should include organizational health literacy (OHL) action such as quality assurance of digital health resources (37), promoting critical DHL skills and expanding research to include populations from all ages in the life course. Finally, as artificial intelligence emerges and becomes more accessible for use in healthcare and public health (38), DHL skills will become increasingly important among the population-at-large to leave no one behind.

## Author's note

The authors are part of the WHO Action Network on Measuring Population and Organizational Health Literacy (M-POHL Network). The M-POHL Network is initiator of the HLS₁₉.

## Data availability statement

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found at: https://m-pohl.net/Design_Methods.
## Ethics statement

In each country, it has been ensured that ethical requirements have been met in accordance with the country/institutional ethics regulation and requirements. For more information about ethical considerations, data protection, and informed consent by country, please see the International Report on the methodology, results, and recommendations of the European Health Literacy Population Survey 2019–2021 (HLS19) of M-POHL (26).

## Author contributions

DL-Z: Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Supervision, Validation, Writing – original draft, Writing – review & editing, Project administration. SB: Conceptualization, Data curation, Funding acquisition, Investigation, Methodology, Writing – original draft, Writing – review & editing, Formal analysis. ÉB: Writing – original draft, Writing – review & editing, Investigation, Methodology. HB: Writing – original draft, Writing – review & editing, Investigation, Methodology. LB: Writing – original draft, Writing – review & editing. SMG: Writing – original draft, Writing – review & editing, Investigation, Methodology. HS: Writing – original draft, Writing – review & editing, Formal analysis, Investigation, Methodology. SG: Writing – original draft, Writing – review & editing, Investigation. RG: Conceptualization, Data curation, Formal analysis, Methodology, Writing – original draft, Writing – review & editing. LG: Writing – original draft, Writing – review & editing, Methodology, Investigation. ØG: Formal analysis, Methodology, Writing – original draft, Writing – review & editing. ZKl: Writing – original draft, Writing – review & editing, Investigation, Methodology. ZKu: Writing – original draft, Writing – review & editing, Investigation, Methodology. CL: Writing – original draft, Writing – review & editing, Formal analysis, Investigation, Methodology. TL: Writing – original draft, Writing – review & editing, Conceptualization, Data curation, Formal analysis, Investigation, Methodology. JM: Writing – original draft, Writing – review & editing, Investigation, Methodology. DM: Writing – original draft, Writing – review & editing, Investigation, Methodology. DS: Writing – original draft, Writing – review & editing, Investigation, Methodology, Supervision. CR: Writing – original draft, Writing – review & editing, Investigation. KS: Writing – original draft, Writing – review & editing, Conceptualization, Investigation, Methodology. CS: Methodology, Writing – original draft, Writing – review & editing, Conceptualization, Funding acquisition, Investigation, Supervision. MT: Writing – original draft, Writing – review & editing, Investigation, Methodology. MV: Writing – original draft, Writing – review & editing, Investigation, Methodology. JP: Conceptualization, Writing – original draft.

## Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This research received no external funding, but the data collection was supported either by ministries of health, university, public health

institutes, or insurance funds in the different countries. AT: The Austrian Health Literacy Survey was commissioned and financed by the Austrian Federal Health Agency and the Federation of Austrian Social Insurance Institutions. BE: Data collection for the Belgian study was funded by the MLOZ Health Insurance Company. BG: Medical University–Sofia, Faculty of Public Health and AGREEMENT No. D-125/24.06.2020, PROJECT No. 8418/21.11.2019, “GRANT-2020,” MU-Sofia. CH The Swiss Federal Office of Public Health FOPH with a financial contribution from the foundation Health Promotion Switzerland. CZ: Data collection was funded by (all seven) Czech health insurance funds. DE: The German study was funded by the German Federal Ministry of Health, grant number: Kapitel 1504 Titel 54401, ZMV I 1–2518 004 (HLS-GER 2). DK: The distribution of the questionnaire was funded by Aalborg University. FR: The research was supported by the National Public Health agency (Santé Publique France, 21DPPA040-0) and by Ligue contre le cancer (LIGUE2019). HU: The research was supported by the Ministry of Human Capacities, Hungary (IV/956-740 4/2020/EKF). The preparation of this paper was supported by the János Bolyai Research Scholarship of the Hungarian Academy of Sciences (BO/00933/22/5). IE: Department of Health, Dublin. IL: The Israel study was funded by the Israel Ministry of Health, Clalit Health Services and the University of Haifa Israel. IT: The Italian participation to the Action Network on Measuring Population and Organizational Health Literacy (M-POHL Network) was funded by the Istituto Superiore di Sanità, Rome, Italy; the implementation of the Health Literacy Survey-HLS₁₉ in Italy was funded by the National Centre for Disease Prevention and Control (CCM) of the Italian Ministry of Health and coordinated by the Department of cardiovascular, endocrine–metabolic diseases and aging of the Istituto Superiore di Sanità, Rome, Italy. NO: The Norwegian HLS₁₉ was commissioned and financed by the Norwegian Ministry of Health and Care Services. The Norwegian Directorate of Health funded the data collection and the administrative costs for the whole project, while Oslo Metropolitan University and Inland Norway University of Applied Sciences contributed with the scientific workforce. PT: Direção-Geral da Saúde, Lisbon. RU: The Russian study was supported by the Ministry of Health of the Russian Federation and WHO Europe. SI: The national survey of Health literacy in Slovenia took place within the framework of the project increasing health literacy in Slovenia-ZaPiS, which is co-financed by the Republic of Slovenia to the amount of 20% of the value and the European Union from the European Social Fund to the amount of 80% of the value. SK: Public Health Authority of the Slovak Republic, Bratislava.

## Acknowledgments

This article is dedicated to the loving memory of Professor Jürgen Pelikan, who initially conceptualized and founded the WHO Action Network on Measuring Population and Organizational Health Literacy (M-POHL) and was a driving force in promoting this study and publication. The authors would like to acknowledge The International Coordination Centre (ICC) of the HLS₁₉ Project for project coordination. We also want to thank all national HLS₁₉ team members who contributed to the HLS₁₉ Project by contributing to the
chapter on Digital Health Literacy in the International Report or and
providing country data for analyses.

Conflict of interest

The authors declare that the research was conducted in the
absence of any commercial or financial relationships that could
be construed as a potential conflict of interest.

Correction note

A correction has been made to this article. Details can be found
at: 10.3389/fpubh.2025.1472706.

References

1. Paige S, Stellson M, Krieger JL, Anderson-Lewis C, Cheong JW, Stopka C. Proposing a transactional model of eHealth literacy: concept analysis. *J Med Internet Res.* (2018) 20:e10175. doi: 10.2196/10175

2. Levin-Zamir D, Bertschi I. Media health literacy, eHealth literacy, and the role of the social environment in context. *Int J Environ Res Public Health.* (2018) 15:1643. doi: 10.3390/ijerph15081643

3. Visser M. Digital literacy and public policy through the library lens. *Maine Policy Rev.* (2013) 22:104-13. doi: 10.53558/WCUZ5068

4. World Health Organization. Digital health resolution. Geneva: World Health Assembly (2018).

5. Norman CD, Skinner HA. eHEALS: the eHealth literacy scale. *J Med Internet Res.* (2006) 8:e27. doi: 10.2196/jmir.8.4.e27

6. Karnoe A, Kayser L. How is eHealth literacy measured and what do the measurements tell us? A systematic review. *Knowl Manag E-Learn.* (2015) 7:576-600.

7. Faux-Nightingale A, Philp F, Chadwick D, Singh B, Pandyan A. Available tools to evaluate digital health literacy and engagement with eHealth resources: a scoping review. *Heliyon*. (2022) 8:e10380. doi: 10.1016/j.heliyon.2022.e10380

8. Livingstone S, Lemish D, Lim SS, Bulger M, Cabello P, Claro M, et al. Global perspectives on children's digital opportunities: an emerging research and policy agenda. *Pediatrics*. (2017) 140:S137-41. doi: 10.1542/peds.2016-1758S

9. Shu-Ching Y, Yi-Fang L, Chia-Hsun C, et al. The associations among individual factors, eHealth literacy, and health-promoting lifestyles among college students. *J Med Internet Res.* (2017) 19:e15. doi: 10.2196/jmir.5964

10. Leung A, Ko P, Chan KS, Chi I, Chow N. Searching health information via the web: Hong Kong Chinese older adults' experience. *Public Health Nurs.* (2007) 24:169-75. doi: 10.1111/j.1525-1446.2007.00621.x

11. Choi NG, Dinitto DM. The digital divide among low-income homebound older adults: internet use patterns, eHealth literacy, and attitudes toward computer/internet use. *J Med Internet Res.* (2013) 15:e93. doi: 10.2196/jmir.2645

12. Levin-Zamir D, Leung YMA, Dodson S, Rowlands G, Peres F, Uwamahoro N, et al. Health literacy in selected populations: individuals, families, and communities from the international and cultural perspective. *Inf Serv Use.* (2017) 37:131-51. doi: 10.3233/ISU-170834

13. Aaby A, Friis K, Christensen B, Rowlands G, Maindal HT. Health literacy is associated with health behaviour and self-reported health: a large population-based study in individuals with cardiovascular disease. *Eur J Prev Cardiol.* (2017) 24:1880-8. doi: 10.1177/2047487317729538

14. Aharony N, Goldman R. E-health literacy and the vaccination dilemma: an Israeli perspective. *Inf Res.* (2017) 22:751.

15. Neter E, Brainin E. eHealth literacy: extending the digital divide to the realm of health information. *J Med Internet Res.* (2012) 14:e19. doi: 10.2196/jmir.1619

16. van der Vaart R, Drossaert C. Development of the digital health literacy instrument: measuring a broad spectrum of health 1.0 and health 2.0 skills. *J Med Internet Res.* (2017) 19:e27. doi: 10.2196/jmir.6709

17. Neter E, Brainin E, Baron-Epel O. Group differences in health literacy are ameliorated in ehealth literacy. *Health Psychol Behav Med.* (2021) 9:480-97. doi: 10.1080/21642850.2021.1926256

18. Kolpatzik K, Mohrmann M, Zeeb H. Digitale Gesundheitskompetenz in Deutschland. Berlin: KomPart (2020).

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online
at: https://www.frontiersin.org/articles/10.3389/fpubh.2025.1472706/
full#supplementary-material

19. Bittlingmayer UH, Dadaczynski K, Sahrai D, van den Broucke S, Okan O. Digital health literacy—conceptual contextualization, measurement, and promotion. *Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz*. (2020) 63:176–84. doi: 10.1007/s00103-019-03087-6

20. Dadaczynski K, Okan O, Messer M, Leung AY, Rosário R, Darlington E, et al. Digital health literacy and web-based information-seeking behaviors of university students in Germany during the COVID-19 pandemic: cross-sectional survey study. *J Med Internet Res.* (2021) 23:e24097. doi: 10.2196/24097

21. Zrubka Z, Brito Fernandes Ó, Baji P, Hajdu O, Kovacs L, Kringos D, et al. Exploring eHealth literacy and patient-reported experiences with outpatient care in the Hungarian general adult population: cross-sectional study. *J Med Internet Res.* (2020) 22:e19013. doi: 10.2196/19013

22. European Commission. (2014) “European citizens’ digital health literacy” Flash Eurobarometer 404 Report. Available at: http://publications.europa.eu/resource/cellar/fd42f9e7-937c-41f3-bf03-4221b2db712b.0001.04/DOC_1 (Accessed July 1, 2024).

23. Schaeffer D, Berens EM, Vogt D, Gille S, Griese L, Klinger J, et al. Health literacy in Germany – findings of a representative follow-up survey. *Dtsch Arztebl Int.* (2021) 118:723-9. doi: 10.3238/arztebl.m2021.0310

24. Le C, Finbräten HS, Pettersen KS, Joranger P, Guttersrud Ø. Befolkningens helsekompetanse, del I the International Health Literacy Population Survey 2019–2021. Oslo, Norway: The Norwegian Directorate of Health (2021).

25. Levin-Zamir D, Laron M, Artom T, Baron-Epel O. Health literacy in Israel – HLS19 results of National Survey 2019-2021. Israel: Health Literacy Survey (2023).

26. The HLS19 Consortium of the WHO Action Network M-POHL. International report on the methodology, results, and recommendations of the European health literacy population survey 2019–2021 (HLS19) of M-POHL. Vienna: Austrian National Public Health Institute (2021).

27. Pelikan JM, Link T, Straßmayr C, Waldherr K, Alfers T, Bøggild H, et al. Measuring comprehensive, general health literacy in the general adult population: the development and validation of the HLS19-Q12 instrument in seventeen countries. *Int J Environ Res Public Health*. (2022) 19:14129. doi: 10.3390/ijerph192114129

28. Guttersrud Ø, Le C, Pettersen KS, Finbråten HS. Rasch analyses of data collected in 17 countries - a technical report to support decision-making within the M-POHL consortium. Vienna, Austria: International HLS19 (2023).

29. Griese L, Finbråten HS, Francisco R, De Gani SM, Griebler R, Guttersrud Ø, et al. HLS19-NAV—validation of a new instrument measuring navigational health literacy in eight European countries. *Int J Environ Res Public Health*. (2022) 19:13863. doi: 10.3390/ijerph192113863

30. Finbråten HS, Nowak P, Griebler R, Biró É, Vrdelja M, Charafeddine R, et al. The HLS19-COM-P, a new instrument for measuring communicative health literacy in interaction with physicians: development and validation in nine European countries. *Int J Environ Res Public Health*. (2022) 19:11592. doi: 10.3390/ijerph191811592

31. Yang K, Hu Y, Qi H. Digital health literacy: bibliometric analysis. *J Med Internet Res.* (2022) 24:e35816. doi: 10.2196/35816

32. Sørensen K, Van den Broucke S, Fullam J, Doyle G, Pelikan J, Slonska Z, et al. Health literacy and public health: a systematic review and integration of definitions and models. *BMC Public Health*. (2012) 12:1–13. doi: 10.1186/1471-2458-12-80
33. Abel T, McQueen D. Critical health literacy and the COVID-19 crisis. *Health Promot Int.* (2020) 35:1612-3. doi: 10.1093/heapro/daaa040

34. Adler NE, Epel ES, Castellazzo G, Ickovics JR. Relationship of subjective and objective social status with psychological and physiological functioning: preliminary data in healthy white women. *Health Psychol.* (2000) 19:586-92. doi: 10.1037/0278-6133.19.6.586

35. Cheng C, Beauchamp A, Elsworth GR, Osborne RH. Applying the electronic health literacy lens: systematic review of electronic health interventions targeted at socially disadvantaged groups. *J Med Internet Res.* (2020) 22:e18476. doi: 10.2196/18476

36. Palumbo R, Nicola C, Adinolfi P. Addressing health literacy in the digital domain: insights from a literature review. *Kybernetes*. (2022) 51:82-97. doi: 10.1108/K-07-2021-0547

37. Sentell T, Foss-Durant A, Patil U, Taira D, Paasche-Orlow MK, Trinacty CM. Organizational health literacy: opportunities for patient-centered care in the wake of COVID-19. *Qual Manag Health Care*. (2021) 30:49-60. doi: 10.1097/QMH. 0000000000000279

38. Nutbeam D, Lloyd JE. Understanding and responding to health literacy as a social determinant of health. *Annu Rev Public Health*. (2021) 42:159-73. doi: 10.1146/annurev-publhealth-090419-102529