# Evaluating a Public Health Information Service According to Users' Socioeconomic Position and Health Status: Protocol for a Cross-Sectional Study

Megane Estevez, Sandrine Domecq, Ilaria Montagni, Viviane Ramel

► To cite this version:

Megane Estevez, Sandrine Domecq, Ilaria Montagni, Viviane Ramel. Evaluating a Public Health Information Service According to Users' Socioeconomic Position and Health Status: Protocol for a Cross-Sectional Study. JMIR Research Protocols, 2023, 12, pp.e51123. 10.2196/51123. hal-04387385

HAL Id: hal-04387385

https://hal.science/hal-04387385v1

Submitted on 11 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Protocol

# Evaluating a Public Health Information Service According to Users' Socioeconomic Position and Health Status: Protocol for a Cross-Sectional Study

Mégane Estevez, MSc; Sandrine Domecq, MS; Ilaria Montagni, PhD; Viviane Ramel, MPH, PhD

Bordeaux University, Inserm, Bordeaux Population Health Research Center, U1219, Bordeaux, France

**Corresponding Author:**

Mégane Estevez, MSc
Bordeaux University, Inserm, Bordeaux Population Health Research Center, U1219
146 Rue Léo Saignat
Bordeaux, 33300
France
Phone: 33 0631690669
Email: megane.estevez@gmail.com

Abstract

**Background:** The increasing use of information technology in the field of health is supposed to promote users' empowerment but can also reinforce social inequalities. Some health authorities in various countries have developed mechanisms to offer accurate and relevant information to health care system users, often through health websites. However, the evaluation of these sociotechnical tools is inadequate, particularly with respect to differences and inequalities in use by social groups.

**Objective:** Our study aims to evaluate the access, understanding, appraisal, and use of the French website *Santé.fr* by users according to their socioeconomic position and perceived health status.

**Methods:** This cross-sectional study involves the entire French population to which *Santé.fr* is offered. Data will be collected through mixed methods, including a web-based questionnaire for quantitative data and interviews and focus groups for qualitative data. Collected data will cover users' access, understanding, appraisal, and use of *Santé.fr*, as well as sociodemographic and socioeconomic characteristics, health status, and digital health literacy. A validation of the dimensions of access, understanding, appraisal, and use of *Santé.fr* will be conducted, followed by principal component analysis and ascendant hierarchical classification based on the 2 main components of principal component analysis to characterize homogeneous users' profiles. Regression models will be used to investigate the relationships between each dimension and socioeconomic position and health status variables. NVivo 11 software (Lumivero) will be used to categorize interviewees' comments into preidentified themes or themes emerging from the discourse and compare them with the comments of various types of interviewees to understand the factors influencing people's access, understanding, appraisal, and use of *Santé.fr*.

**Results:** Recruitment is scheduled to begin in January 2024 and will conclude when the required number of participants is reached. Data collection is expected to be finalized approximately 7 months after recruitment, with the final data analysis programmed to be completed around December 2024.

**Conclusions:** This study would be the first in France and in Europe to evaluate a public health information service, in this case the *Santé.fr* website (the official website of the French Ministry of Health), according to users' socioeconomic position and health status. The study could discover issues related to inequalities in access to, and the use of, digital technologies for obtaining health information on the internet. Given that access to health information on the internet is crucial for health decision-making and empowerment, inequalities in access may have subsequent consequences on health inequalities among social categories. Therefore, it is important to ensure that all social categories have access to *Santé.fr*.

**International Registered Report Identifier (IRRID):** PRR1-10.2196/51123

(JMIR Res Protoc 2023;12:e51123) doi: 10.2196/51123

KEYWORDS

internet; health information seeking; literacy; social health inequalities; evaluation; digital health; public health; socioeconomic position; health status; user; empowerment; social inequality; mobile phone
# Introduction

## Digital Technologies and Health

The health sector is characterized by an exponential use of digital health technology that serves both for care and for coordination among professionals and facilities and for health care system users (eg, health information, appointments scheduling, and communication with caregivers). Digital health technologies are intended to enhance the autonomy of users, including patients and individuals who use these technologies to support the health of their relatives. This implies that users, whether for themselves or for their loved ones, should have access to health information that enables them to better manage health-related concerns.

Although digital health is promising in terms of technology, its increasing use can lead to unintended negative consequences [1-3]. Indeed, digital health technology can mitigate some geographic location-related barriers, but access to, and the use of, these technologies can be influenced by various factors, including socioeconomic position. Geographic location and socioeconomic position remain important factors influencing the use of digital health technologies.

Individuals' socioeconomic characteristics and their overall health status can affect how different social groups adopt digital health technologies [4]. Consequently, the growing digital divide may worsen existing social health inequalities. The COVID-19 pandemic has accelerated and amplified the digital divide [5]. The lockdown showed that many people were at a loss when faced with the necessity of carrying out web-based procedures. This health, social, and economic crisis has amplified the risks of digital exclusion. Existing digital tools, such as telemedicine and health websites, have been useful in enabling remote medical consultations or keeping up to date with the spread of the virus. By contrast, countless websites are designed according to technical criteria, rather than on the basis of the experience of users in all their diversity [5]. Therefore, it is important to carefully examine the unexpected, undesirable, and negative consequences of digital health to understand its real impact on people and ensure that public health policies address these issues [1-3].

## Social Gradient in Access to Digital Health

Studies highlight a social gradient in the access, use, and appraisal of digital health technologies in other countries [6,7], but data are lacking in France, where individuals belonging to the working classes have less access to computers and are less likely to search for health information on the internet [8].

More recently, using data from the *Baromètre Santé* study conducted between 2010 and 2017, French researchers showed that the profile of health information seekers versus nonseekers has remained consistent over time [9]. Seekers were more likely to be younger, more educated female individuals, with higher household incomes and holding executive positions. In 2017, as in 2014, general health websites remained the top source of information (38.6%), followed by social media and commercial websites, whereas institutional websites ranked as the third source (8.1%) [9]. These authors recommended that health

authorities should improve citizens' digital health literacy and provide reliable and accessible sources of web-based health information.

In response to this issue, some health authorities in different countries have progressively developed mechanisms to provide relevant, accurate, and useful information to health care system users, often through health information websites [10,11]. However, the use of these new sociotechnical tools has been poorly assessed, particularly in terms of differences and inequalities in use among social groups.

## Evaluation of Health Information Websites

The literature includes several studies that have evaluated the quality of health information websites. Some studies have primarily focused on specific information related to particular diseases [12-23] or prevention measures [24-28], whereas other studies have analyzed general health information websites [29-35].

Studies evaluating the quality of health information websites often use automated tools and manual evaluations by professionals. These assessments focus on various criteria, such as readability, the ease of use, reliability, and the quality of information. Tools such as Flesch-Kincaid [14,18,21,22,30,36-41], Flesch Reading Ease [12,18,21,28,30,36-38,41], Simple Measure of Gobbledygook [14,18,21,22,24,30,40], Gunning Fog Index [14,21,22], Coleman-Liau Index [14,21,22], and Automated Readability Index [14,21] measure reading comfort. The Health on the Net Foundation Code of Conduct assesses reliability [17,30,36,42], and the DISCERN tool evaluates health information quality [19,20,28,38,42,43]. The Lida instrument assesses design and content [12,19,37,40], whereas the Clear Communication Index, a tool developed by the Centers for Disease Control and Preventions, evaluates written health materials [32,44]. Some studies also include user perspectives to evaluate website usability and information quality [45,46].

Access to clear and understandable health information is important for informed decision-making, but many health websites are difficult to navigate and read, especially for those with low literacy levels [12,32,37,47]. This can lead to health inequalities [39] and delay the adoption of appropriate health behaviors. Although government websites are usually easier to read [14], improvements are still necessary. Website design and content can also present barriers to use [48]. Few studies focus on the evaluation of a single website, and evaluations often overlook users' sociodemographic characteristics, health status, and health literacy levels. Therefore, it is important to conduct evaluations that take into account these factors to identify any population groups facing barriers to accessing and understanding health information on these websites.

To make health information websites accessible and user-friendly, it is important to involve users in the evaluation process [32]. Understanding users' needs and preferences can ensure that the information provided is clear, relevant, and actionable [39]. It is crucial to assess issues related to the accessibility of health information to ensure that all individuals can take advantage of the wealth of health information available
on the internet, given the growing focus on patient empowerment in health [48].

## Santé.fr, the Website of the French Public Health Information Service

In France, the *Santé.fr* website is a public service website that provides general health information selected for its quality and usefulness. This website is under the Ministry of Health's responsibility, and its mission is to provide free information on health care as well as medical and social services to the public. The information is adapted and accessible to people with disabilities and aims to encourage citizen participation in health choices, improve patient management, and simplify the adoption of preventive behaviors.

Santé.fr serves as a reference health website and offers access to resource directories, prevention and health education information services, and more. It is accessible anytime and anywhere, making it a reliable and convenient source of health information.

As *Santé.fr* is an information and support tool for users, it is crucial to track and evaluate its use and identify any disparities among different population groups. Therefore, it is important to explore the experiences and evaluations of *Santé.fr* users to determine to what extent the information provided meets their needs and expectations. The objective is to ensure that *Santé.fr* proves to be a genuine advantage for users and does not exacerbate the existing social inequalities by promoting the use of a tool that is inaccessible or unsuitable for a portion of the population because disparities in the access, understanding, appraisal, and use of digital health technology can worsen social health inequalities, depending on individuals' social position.

## Objectives of the Study

The main objective of this study was to evaluate the access, understanding, appraisal, and use of *Santé.fr* by users according to their socioeconomic position and health status. Our specific objectives were to (1) determine users' profiles in terms of the access behavior, understanding, appraisal, and use of *Santé.fr* based on their socioeconomic position and health status; (2) examine the association between users' socioeconomic position and health status and their access, understanding, appraisal, and use of *Santé.fr*; and (3) determine the reasons why users of *Santé.fr* use this service.

# Methods

## Research Design

The *Évaluation du service public d'information en santé* (the *Santé.fr* website) selon le statut socio-économique de ses utilisateurs (evaluation of the public health information service [the *Santé.fr* website] according to the socioeconomic status of users) is a cross-sectional observational study of the French population using *Santé.fr*. It will use mixed methods, that is, it will combine quantitative and qualitative research methods [49]. The study period will last 1 year so that there will be enough time to collect the necessary quantitative and qualitative data.

## Study Population

The study target population is the entire population to which *Santé.fr* is offered, that is, the general French population. The source population is made up of all health care system users, whether or not they use *Santé.fr*; the respondents to this study; and the study sample. All inhabitants of metropolitan France and the *Départements et régions d'outre-mer et collectivités d'outre-mer* (overseas departments and regions and overseas communities) will be concerned by this study. The quantitative and qualitative study inclusion criterion is as follows: any adult (aged ≥18 y) living in metropolitan France or *Départements et régions d'outre-mer et collectivités d'outre-mer*.

## Recruitment

The quantitative study will include both users and nonusers of *Santé.fr* who visit the website and voluntarily complete the questionnaire. Before participation, all participants will receive comprehensive and adequate information through an information note. *Santé.fr* users are individuals who have previously consulted *Santé.fr* or are visiting it for the first time when completing the questionnaire. Nonusers of *Santé.fr* are individuals who, although aware of this public health information service, have never accessed it before coming across the questionnaire or were not previously familiar with it.

The questionnaire will be self-administered by *Santé.fr* users, using a computer-assisted web interviewing method (ie, completed on the web). An article on *Santé.fr* will be developed to present the research team and the study to participants and to share the external link to the questionnaire. A poster will be designed to encourage users to participate in the study. The article and the questionnaire will be displayed on *Santé.fr* at several navigation levels (eg, thumbnail on the home page, contextual message block, end of an article, and thematic folder) to increase access to the questionnaire. Users connecting to *Santé.fr* will be able to click directly on the poster and reach either the article or the questionnaire. Users will decide whether to participate in the study. A communication plan will be elaborated by the research team to disseminate the study as much as possible.

The qualitative study will include individuals, who may or may not be users of *Santé.fr*, with a variety of profiles, including different sexes, age groups, geographic origins, and socioeconomic positions. Specifically, socioeconomic position will be assessed based on criteria such as education level, occupational category, and income level.

Participants will be recruited through the following channels:

* Relatives or persons in the professional and personal environment of the research team will be requested to identify interested individuals.

* Snowball method: the research team members will ask some of their contacts to identify people they know from different geographic locations and ask them to agree to be contacted by the research team.

* Associations such as Emmaüs Connect, Point Information Mediation Multi Services, and La Mednum will be contacted to identify individuals with digital difficulties. Emmaüs Connect helps individuals experiencing social and
digital insecurity to access essential web-based tools. La Mednum helps to develop mediation and digital inclusion programs all over France, as does Point Information Mediation Multi Services.

* Assistance will be requested from the France Assos Santé network, an interassociative organization that represents health system patients and users and defends their interests, to identify patients or users of the health care system.

* Additional resources may be used.

For the quantitative study, a sample size calculation was performed. As we had no information on the proportion of Santé.fr users, we calculated the number of participants necessary according to the estimation formula in the case of an expected prevalence of 50% with a risk of error α of 5% and a precision of 5%. A sample of 385 people was deemed necessary. For the qualitative study, we estimate conducting 30 interviews, taking into account feasibility and the need for a wide range of respondent profiles.

# Data Collection

## Quantitative Study

### Overview

A questionnaire was developed according to the literature to collect the necessary quantitative and qualitative data to address the objectives. The 5-section questionnaire, described in detail in the following subsections, is available in Multimedia Appendix 1. A feasibility test of the questionnaire will be conducted to ensure that the questions are clear, simple, and understandable.

### Section 1: Access or Use

This section includes questions to determine whether users or nonusers have the necessary equipment (computer, tablet, smartphone, and internet connection) to access the internet and the general skills needed to use the internet effectively on these devices. The questions also explore their interest in using the internet to find health information and seek to understand how often they use Santé.fr for health information. To develop these questions, we used questions from the French *Baromètre Santé* periodic surveys and the Website Quality Evaluation Tool [50]. The *Baromètre Santé* surveys have been conducted since 1992 to better understand French people's knowledge, behavior, beliefs, and practices concerning health [51].

### Section 2: Understanding

This section consists of questions related to users' ability to find and understand health information on Santé.fr and their ability to evaluate, in general, health information presented on the internet. These indicators will help determine whether Santé.fr is accessible and user friendly. Some items from the eHealth Literacy Scale (eHEALS), discussed in the *eHealth Literacy* subsection, were used [52]. We also used questions from the *Baromètre Santé* periodic surveys [51], the DISCERN tool [53], a systematic literature review by Sun et al [54] on relevant criteria and indicators for evaluating the quality of health information on the internet, and a critical analysis of evaluation criteria for health websites [55].

## Section 3: Application

This section concerns the appraisal and adoption of Santé.fr by users, either for themselves or for relatives. These questions assess whether this website helps users to make informed decisions about their own health. The questions on the intention of users to reuse Santé.fr and to recommend it to relatives will allow us to evaluate their satisfaction and to assess their degree of confidence in the provided information. To develop these questions, we were inspired by the technology acceptance model [56] as well as items from the eHealth Impact Questionnaire [57] and items used in the study by Kang and An [33], which evaluated health-related sites for older adults and assessed whether the sites take into account factors that affect older adults' intention to use them for health information seeking.

## Section 4: Sociodemographic and Economic Characteristics

This section collects the respondents' sociodemographic and economic characteristics. The questions cover their indirect geographic origin or native area (and that of their parents), their job status (employed or not) and socioprofessional category or social position, their economic and financial situation (income, origin of income, social benefits, and self-declared financial situation), and their geographic place of residence (postal code and the name of the town of residence). These variables will make it possible to estimate the respondents' socioeconomic situation. These questions were inspired by data on socioeconomic situations collected from the French Constances study [58].

## Section 5: Health Status

This section includes questions to evaluate the respondents' perceived health status, addressing the general, physical, and psychological dimensions of health as well as the presence of an illness or chronic health problem. To develop these questions, we were inspired by the questions from the *Enfants et familles sans logement* (Children and families experiencing homelessness) survey [59] and the periodic *Baromètre Santé* surveys [51].

## eHealth Literacy

To calculate a digital health literacy score for each respondent and to be able to compare the total scores between Santé.fr users and nonusers, the eHEALS items are used [52]: in their original form for nonusers of Santé.fr and adapted for users of Santé.fr to match the digital tool being evaluated. The corresponding items are presented in Multimedia Appendix 2 and are distributed in the questionnaire's different sections.

## Qualitative Study

To explore users' and nonusers' perceptions regarding their access, understanding, appraisal, and use of Santé.fr, as well as their general use of the internet for searching and using health information, semistructured individual interviews will be conducted. The interview guide will be developed based on the questionnaire used in the quantitative study. Regarding the interview's organization, an initial telephone contact with the participant will allow us to briefly explain the study to them, obtain their consent to participate in the interview, and set up an appointment for the interview. The participant's main characteristics will also be collected during this first contact by
the interviewer to ensure a wide-range sample. If a participant agrees to participate in the study and has never consulted *Santé.fr*, the interviewer will ask them to navigate through the website before the interview and browse the health topics of interest. On the day of the interview, if the participant has been able to access *Santé.fr*, the interviewer will ask them to describe their overall experience and ask the interview guide questions. If a participant has not been able to access the website, the interviewer will ask them to explain why, which will be considered a result per se.

These interviews will take place via videoconference, telephone, or face-to-face, depending on the participants' preference, at a time and date of their choice.

Focus groups will also be conducted by the research team. The objective is to enable the participants to provide their reactions to the *Santé.fr* website. Participants will have to navigate the website and share their experience: what they think of *Santé.fr* in general, the difficulties or problems faced, the elements to be improved according to them, their ability to carry out a search and obtain answers to their questions, whether the navigation is easy, and so on. Participants may or may not already know each other and have varied sociodemographic profiles.

## Statistical Analysis

### Quantitative Analysis

### Descriptive Analyses

The sociodemographic and socioeconomic characteristics, health status, and digital health literacy score measured by the eHEALS will be described. The variables related to the access, understanding, appraisal, and use of *Santé.fr* will also be presented. Qualitative variables will be analyzed using frequencies and proportions, whereas the digital health literacy scores will be described using mean and SD as well as median and IQR.

A validation of the dimensions of access, understanding, appraisal, and use of *Santé.fr* will be conducted to ensure their

reliability and validity. This validation process will involve assessing the internal consistency of items as well as evaluating convergent and discriminant validity. To assess internal consistency, the Cronbach α coefficient will be calculated. In addition, correlation analysis will be conducted to examine the relationships among variables and to evaluate convergent and discriminant validity.

Principal component analysis will be performed for each dimension, using the variables specific to that dimension as active variables. Subsequently, ascendant hierarchical classification will be conducted based on the 2 components of principal component analysis to characterize homogeneous users' profiles based on their access, understanding, appraisal, and use of *Santé.fr*. The sociodemographic, socioeconomic, and health status variables will be used as illustrative variables to characterize users' profiles.

## Analytical Analyses

On the basis of the results of the dimension validation in the descriptive part of the statistical analyses, different approaches will be used for the analytical analyses. If the dimension (ie, access, understanding, appraisal, and use of *Santé.fr*) is validated, a quantitative score will be used as a continuous variable. However, if the dimension is not validated, a specific questionnaire variable related to that dimension will be used instead. Depending on the nature of the outcome variable, different regression models will be developed. If the outcome variable is continuous, a linear regression model will be conducted. If the outcome variable is binary, a logistic regression model will be performed. In the case of a variable with >2 categories, a multinomial logistic regression model will be applied. Regardless of the model used, the explanatory variables will include education level, socio-occupational category, and financial situation. Confounding variables such as sex, age, and place of residence will also be included in the analyses to account for potential biases. This approach follows an etiological explanatory perspective, for which a directed acyclic graph will be used (Figure 1).
<figure><img src="image_2.png" /><figcaption>**Figure 1.** Directed acyclic graph applied to the relationships between socioeconomic position and health status and the access, understanding, appraisal, and use of Santé.fr.</figcaption></figure>

All quantitative analyses will be conducted using RStudio (version 2023.03.1+446; Posit Software, PBC).

## Qualitative Analysis

The first step of the qualitative analysis will consist of transcribing all interviews and focus groups. The analysis grid will be constructed from the interview guide and the focus groups and completed inductively based on new items that may appear during the transcripts analysis. Using NVivo 11 software (Lumivero), the interviewees' comments will be classified within each preidentified theme or themes emerging from the discourse and compared among types of interviewees to understand the reasons why people access, understand, evaluate, and use (or do not) Santé.fr. A profile analysis of people with similar responses will be conducted to see whether any profile types emerge.

## Mixed Analysis

Quantitative and qualitative data will be collected simultaneously using the convergent parallel design approach. The analyses will then be combined to meet the overall objectives of the study.

## Ethical Considerations

The data collection and processing implemented in this research conforms to the *Commission nationale de l'informatique et des libertés* (National Commission of Informatics and Liberties) reference methodology MR-004 (research not involving human persons, studies, or evaluations in the health field).

An information note at the beginning of the questionnaire informs participants about the study, its objectives, its progress, the processing of personal data, and their rights. By completing the questionnaire, the participants implicitly agree to share their data for scientific purposes. For the interviews, the participants will give their oral consent.

Personal data will be processed to analyze the results of the research in relation to its objectives. The data collected will be recorded in a digital file that complies with several data security and confidentiality conditions. The data will be deidentified (ie, anonymized) using the following techniques: we have ensured that the nature of the response modalities is broad enough to prevent the recognition of the person who completed the questionnaire (such as age group and income group); we do not collect nominative data; and we assign a numerical identifier specific to the study to pinpoint the respondent that is known only to the research team.

No compensation will be offered to users for participating in the study.
## Results

Recruitment is scheduled to begin in January 2024 and will conclude when the required number of participants is reached. Data collection is expected to be finalized approximately 7 months after recruitment, with the final data analysis programmed to be completed around December 2024.

## Discussion

### Difficulties and Limitations of the Study

The first difficulty concerning the quantitative part of the study could be linked to the recruitment of participants, given that participation in the study will be on a voluntary basis. Despite our efforts to aim at a wide dissemination at several navigation levels on Santé.fr (eg, thumbnail on the home page, contextual message block, end of an article, and thematic folder) to multiply access to the questionnaire, the response rate could prove to be too low to allow statistical analyses to be conducted. For the time being, owing to the different waves of the COVID-19 pandemic, the current pop-in dedicated to screening and vaccination against COVID-19 and appearing on any page after a few minutes of navigation could not be removed and replaced by a message presenting the study. It would therefore be advisable to provide maximum visibility for the study on Santé.fr; for example, a pop-in would make it possible to make the study visible to each user and thus increase the chances of response. Moreover, as Santé.fr users prefer to access the website on their mobile phones, if the study does not appear on the home page on mobile phones, the number of respondents could be insufficient can be used.

By contrast, people who feel uncomfortable using the internet could also have difficulties accessing the questionnaire. A lack of motivation or time to complete the questionnaire as well as a low interest in digital health topics and research could decrease the response rate to the study. Compensation for participating in the study could give users a reason to complete the questionnaire.

For the qualitative study too, the lack of sufficiently diverse respondents could be a challenge. In addition, the risk of social desirability bias in the interviewees' responses to the interviewer could be an issue.

### Strength of the Study

This study would be the first of its kind in France and, more widely, in Europe to focus on the evaluation of a digital health information website according to users' socioeconomic position.

The study that we intend to set up is thus essential because it will make it possible to fill a gap in the existing literature. More precisely, it will allow us to discover the issues related to inequalities in access to, and the use of, digital technologies to obtain health information. This study will give an insight into the digital divide that accounts for the poor access to these digital health information services for people who do not have access to the necessary material conditions such as digital devices and an internet connection. This study will discover the social inequalities that affect the access, understanding, appraisal, and use of health information found on the internet, particularly among people with lower levels of education, income, and digital health literacy.

As access to health information on the internet is crucial for health decision-making and for people's empowerment, if inequalities in access are discovered by this study, then they may also be related to inequalities in the health status of social groups. It would therefore be important to ensure that all social groups have equal access to Santé.fr, a quality digital health information service offering reliable health information.

With regard to our research subject, using mixed methods is a relevant choice because it will allow us to combine quantitative methods (questionnaire) and qualitative methods (interviews and focus groups). The quantitative methods will make it possible to obtain an overview of access to Santé.fr and to measure specific indicators (access, understanding, appraisal, and use) that constitute the digital health literacy determinants. The combination of these 2 methods will give a complete and more accurate view of this evaluation through quantitative results and user experiences that may be influenced by users' socioeconomic position. Reasons for the use or lack of use can be obtained through the qualitative component of the study.

### Directions for Future Research

This study could serve as a feasibility study for collecting data on the French population's habits regarding the use of the internet to access health information, which could be incorporated into general national health surveys for regular monitoring purposes as well as Santé.fr follow-up surveys.

## Acknowledgments

The authors thank the public service of health information of the French Ministry of Health and Jérôme Wittwer from Bordeaux University. The authors also thank the Conseil Régional de Nouvelle-Aquitaine for its financial support (2017-1R30222 - 00013233) as well as the Bordeaux University.

## Authors' Contributions

ME, with IM's contribution and under the direction of VR, designed the protocol and the study design. ME, IM and VR elaborated the main conceptual ideas. ME, VR and IM developed the questionnaire and the interview guides. SD provided critical expertise on the statistical analysis plan. ME, VR and IM wrote the manuscript. All authors contributed to finalizing the manuscript and read and approved the final version. VR and IM supervised all the steps involved in elaborating the protocol and the manuscript. All authors have read and agreed to the published version of the manuscript.
## Conflicts of Interest

None declared.

## Multimedia Appendix 1

Study questionnaire.

[DOCX File, 40 KB-Multimedia Appendix 1]

## Multimedia Appendix 2

Original eHealth Literacy Scale and items adapted to this study.

[DOCX File, 23 KB-Multimedia Appendix 2]

## References

1. Bloomrosen M, Starren J, Lorenzi NM, Ash JS, Patel VL, Shortliffe EH. Anticipating and addressing the unintended consequences of health IT and policy: a report from the AMIA 2009 Health Policy Meeting. J Am Med Inform Assoc 2011;18(1):82-90 [FREE Full text] [doi: 10.1136/jamia.2010.007567] [Medline: 21169620]

2. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 2017 Nov 01;19(11):e367 [FREE Full text] [doi: 10.2196/jmir.8775] [Medline: 29092808]

3. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Analysing the role of complexity in explaining the fortunes of technology programmes: empirical application of the NASSS framework. BMC Med 2018 May 14;16(1):66 [FREE Full text] [doi: 10.1186/s12916-018-1050-6] [Medline: 29754584]

4. Lupton D. Digital Health: Critical and Cross-Disciplinary Perspectives. Milton Park, UK. Routledge; 2017.

5. Lucas JF. La COVID-19, acceleratrice et amplificatrice des fractures numériques. HAL Open Science. 2020. URL: https://hal.science/hal-03004991 [accessed 2022-11-23]

6. Latulippe K, Hamel C, Giroux D. Social health inequalities and eHealth: a literature review with qualitative synthesis of theoretical and empirical studies. J Med Internet Res 2017 Apr 27;19(4):e136 [FREE Full text] [doi: 10.2196/jmir.6731] [Medline: 28450271]

7. Ramel V. Les technologies numériques en santé face aux inégalités sociales et territoriales : une sociologie de l'action publique comparée. Thèse de doctorat. Université de Bordeaux. 2020. URL: https://tel.archives-ouvertes.fr/tel-02963193

8. Renahy E, Parizot I, Chauvin P. Health information seeking on the Internet: a double divide? Results from a representative survey in the Paris metropolitan area, France, 2005-2006. BMC Public Health 2008 Feb 21;8:69 [FREE Full text] [doi: 10.1186/1471-2458-8-69] [Medline: 18291024]

9. Ducrot P, Montagni I, Nguyen Thanh V, Serry AJ, Richard JB. Evolution of online health-related information seeking in France from 2010 to 2017: results from nationally representative surveys. J Med Internet Res 2021 Apr 14;23(4):e18799 [FREE Full text] [doi: 10.2196/18799] [Medline: 33851927]

10. Eysenbach G. Consumer health informatics. BMJ 2000 Jun 24;320(7251):1713-1716 [FREE Full text] [doi: 10.1136/bmj.320.7251.1713] [Medline: 10864552]

11. Madon S, Sahay S, Sudan R. E-government policy and health information systems implementation in Andhra Pradesh, India: need for articulation of linkages between the macro and the micro. Inf Soc 2007 Sep 27;23(5):327-344 [doi: 10.1080/01972240701572764]

12. Soobrah R, Clark SK. Your patient information website: how good is it? Colorectal Dis 2012 Mar;14(3):e90-e94 [doi: 10.1111/j.1463-1318.2011.02792.x] [Medline: 21883807]

13. Liebl P, Seilacher E, Koester MJ, Stellmanns J, Zell J, Hübner J. What cancer patients find in the internet: the visibility of evidence-based patient information - analysis of information on German websites. Oncol Res Treat 2015;38(5):212-218 [doi: 10.1159/000381739] [Medline: 25966768]

14. Purdy AC, Idriss A, Ahern S, Lin E, Elfenbein DM. Dr Google: the readability and accuracy of patient education websites for Graves' disease treatment. Surgery 2017 Nov;162(5):1148-1154 [doi: 10.1016/j.surg.2017.07.011] [Medline: 28864099]

15. Pereira Neto AF, Paolucci R, Daumas RP, Souza RV. Participatory evaluation of the quality of health information on the internet: the case of dengue sites. Cien Saude Colet 2017 Jun;22(6):1955-1968 [FREE Full text] [doi: 10.1590/1413-81232017226.04412016] [Medline: 28614515]

16. Zhu X, Smith RA, Parrott RL, Worthington AK. Understanding information sharing about rare diseases: an evaluation of the NIH's website on AATD. J Commun Healthc 2018 Apr;11(2):128-139 [doi: 10.1080/17538068.2018.1453434]

17. Bragazzi NL, Prasso G, Re TS, Zerbetto R, Del Puente G. A reliability and content analysis of Italian language anorexia nervosa-related websites. Risk Manag Healthc Policy 2019;12:145-151 [FREE Full text] [doi: 10.2147/RMHP.S193088] [Medline: 31534378]
18. Correa DJ, Milano L, Kwon CS, Jetté N, Dlugos D, Harte-Hargrove L, et al. Quantitative readability analysis of websites providing information on traumatic brain injury and epilepsy: a need for clear communication. Epilepsia 2020 Mar;61(3):528-538 [FREE Full text] [doi: 10.1111/epi.16446] [Medline: 32096225]

19. Sun W, Luo A, Bian Z, Zhao B, Liu P, Wang K, et al. Assessing the quality of online health information about breast cancer from Chinese language websites: quality assessment survey. JMIR Cancer 2021 Nov 18;7(4):e25783 [FREE Full text] [doi: 10.2196/25783] [Medline: 34792471]

20. Vivien A, Kowalski V, Chatellier A, Babin E, Bénateau H, Veyssière A. Information quality in general public French-speaking websites dedicated to oral cancer detection. Article in French. J Stomatol Oral Maxillofac Surg 2017 Feb;118(1):20-28 [doi: 10.1016/j.jormas.2016.10.003] [Medline: 28330570]

21. Castro-Sánchez E, Spanoudakis E, Holmes AH. Readability of ebola information on websites of public health agencies, United States, United Kingdom, Canada, Australia, and Europe. Emerg Infect Dis 2015 Jul;21(7):1217-1219 [FREE Full text] [doi: 10.3201/eid2107.141829] [Medline: 26079313]

22. Mishra V, Dexter JP. Comparison of readability of official public health information about COVID-19 on websites of international agencies and the governments of 15 countries. JAMA Netw Open 2020 Aug 03;3(8):e2018033 [FREE Full text] [doi: 10.1001/jamanetworkopen.2020.18033] [Medline: 32809028]

23. Momenipour A, Rojas-Murillo S, Murphy B, Pennathur P, Pennathur A. Usability of state public health department websites for communication during a pandemic: a heuristic evaluation. Int J Ind Ergon 2021 Nov;86:103216 [FREE Full text] [doi: 10.1016/j.ergon.2021.103216] [Medline: 34566224]

24. Seitz CM, Shiplo S, Filippini T, Kabir Z, Lennon JL, Fowler D. The reading level of government and voluntary health organization smoking cessation websites: a descriptive analysis. Am J Health Educ 2017 Sep 29;48(6):392-399 [doi: 10.1080/19325037.2017.1358122]

25. Silverman T, Asante N, van den Berg JJ. Quality of HIV websites with information about pre-exposure prophylaxis or treatment as prevention for men who have sex with men: systematic evaluation. JMIR Public Health Surveill 2018 Oct 16;4(4):e11384 [FREE Full text] [doi: 10.2196/11384] [Medline: 30327291]

26. Whiteley L, Healy MG, Lowery A, Haubrick KK, Brown LK. An evaluation and review of English language pre-exposure prophylaxis websites and YouTube videos. Int J STD AIDS 2020 Apr;31(5):460-466 [doi: 10.1177/0956462420905271] [Medline: 32131703]

27. Chong J. How much knowledge can they gain? women's information behavior on government health websites in the context of HIV/AIDS prevention. InterActions UCLA J Educ Inf Stud 2012 Jan;8(1) [FREE Full text] [doi: 10.5070/D481000826]

28. Ruan S, Raeside R, Singleton A, Redfern J, Partridge SR. Limited engaging and interactive online health information for adolescents: a systematic review of Australian websites. Health Commun 2021 May;36(6):764-773 [doi: 10.1080/10410236.2020.1712522] [Medline: 31964190]

29. Yucel G, Ozok AF. Quantifying ergonomic quality of governmental websites. Electron Gov Int J 2010;7(3):233-251 [doi: 10.1504/eg.2010.033590]

30. Risoldi Cochrane Z, Gregory P, Wilson A. Readability of consumer health information on the internet: a comparison of U.S. government-funded and commercially funded websites. J Health Commun 2012;17(9):1003-1010 [doi: 10.1080/10810730.2011.650823] [Medline: 22512714]

31. Conesa-Fuentes MC, Aguinaga-Ontoso E, Hernandez-Morante JJ. Evaluation of the quality of the general health information webpages in Spain: influence of page source. Inform Health Soc Care 2013 Dec;38(4):382-395 [doi: 10.3109/17538157.2013.812645] [Medline: 23957652]

32. Gagen DM, Kreps GL. An examination of the clarity of government health websites using the centers for disease control and prevention's clear communication index. Front Commun 2019 Nov 14;4:1-15 [doi: 10.3389/fcomm.2019.00060]

33. Kang H, An S. Do websites contain factors to aid older adults' adoption of health-related information and communication technology? J Commun Health 2020 May 28;13(2):89-101 [doi: 10.1080/17538068.2020.1761691]

34. Fernández-Díaz E, Iglesias-Sánchez PP, Jambrino-Maldonado C. Exploring WHO communication during the COVID 19 pandemic through the WHO website based on W3C guidelines: accessible for all? Int J Environ Res Public Health 2020 Aug 05;17(16):5663 [FREE Full text] [doi: 10.3390/ijerph17165663] [Medline: 32764480]

35. Yi YJ. Web accessibility of healthcare web sites of Korean government and public agencies: a user test for persons with visual impairment. Univ Access Inf Soc 2018 Jul 17;19:41-56 [doi: 10.1007/s10209-018-0625-5]

36. Thompson AE, Graydon SL. Patient-oriented methotrexate information sites on the internet: a review of completeness, accuracy, format, reliability, credibility, and readability. J Rheumatol 2009 Jan;36(1):41-49 [doi: 10.3899/jrheum.080430] [Medline: 19004031]

37. Gilbert M, Michelow W, Dulai J, Wexel D, Hart T, Young I, et al. Provision of online HIV-related information to gay, bisexual and other men who have sex with men: a health literacy-informed critical appraisal of Canadian agency websites. Sex Health 2019 Feb;16(1):39-46 [FREE Full text] [doi: 10.1071/SH18092] [Medline: 30620885]

38. Kaicker J, Borg Debono V, Dang W, Buckley N, Thabane L. Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument. BMC Med 2010 Oct 12;8:59 [FREE Full text] [doi: 10.1186/1741-7015-8-59] [Medline: 20939875]
39. English CL, Alden KR, Zomorodi M, Travers D, Ross MS. Evaluation of content on commonly used web sites about induction of labor and pain management during labor. MCN Am J Matern Child Nurs 2018;43(5):271-277 [doi: 10.1097/NMC.0000000000000455] [Medline: 30113406]

40. Visser K, Slattery M, Stewart V. Help or hinder? an assessment of the accessibility, usability, reliability and readability of disability funding website information for Australian mental health consumers. Health Soc Care Community 2021 Sep;29(5):1378-1390 [doi: 10.1111/hsc.13192] [Medline: 33051906]

41. Roberts M, Peterson LJ, Hyer K. The state of states' assisted living websites: information available to consumers. Gerontologist 2020 Jul 15;60(5):878-884 [doi: 10.1093/geront/gnz174] [Medline: 31930384]

42. Haute Autorité de Santé. Évaluation de la qualité des sites e-santé et de la qualité de l'information de santé diffusée sur Internet (Revue de la littérature des outils d'évaluation). Haute Autorité de Santé. 2007 May. URL: https://www.has-sante.fr/upload/docs/application/pdf/evaluation_qualite_site_sante_internet.pdf [accessed 2022-01-13]

43. Zuidema WP, Graumans MJ, Oosterhuis JW, van der Steeg AF, van Heurn E. The quality of web sites' health information on minimal invasive repair of pectus excavatum using the DISCERN instrument. Eur J Pediatr Surg 2021 Apr;31(2):157-163 [doi: 10.1055/s-0040-1710026] [Medline: 32396941]

44. The CDC clear communication index. Centers for Disease Control and Prevention. URL: https://www.cdc.gov/ccindex/index.html [accessed 2022-11-30]

45. Marton C. How women with mental health conditions evaluate the quality of information on mental health web sites: a qualitative approach. J Hosp Librariansh 2010 Jul 30;10(3):235-250 [doi: 10.1080/15323269.2010.491422]

46. Motoo Y, Yukawa K, Hisamura K, Arai I. Usability of the evidence-based Japanese integrative medicine (eJIM) information site: analysis with focus group discussion and internet survey on the general population. Integr Med Res 2018 Sep;7(3):238-247 [FREE Full text] [doi: 10.1016/j.imr.2018.03.002] [Medline: 30271712]

47. Lachance CR, Erby LA, Ford BM, Allen VC, Kaphingst KA. Informational content, literacy demands, and usability of websites offering health-related genetic tests directly to consumers. Genet Med 2010 May;12(5):304-312 [FREE Full text] [doi: 10.1097/GIM.0b013e3181dbd8b2] [Medline: 20386454]

48. Ibrahim NA, Seifi S. Assessing the accessibility of health web sites during the H1N1 pandemic. J Consum Health Internet 2014 Sep 03;18(3):211-225 [doi: 10.1080/15398285.2014.931770]

49. Guével MR, Pommier J. Recherche par les méthodes mixtes en santé publique: enjeux et illustration. Santé Publique 2012;24:23-38 [FREE Full text] [doi: 10.3917/spub.121.0023]

50. McInerney CR, Bird NJ. Assessing website quality in context: retrieving information about genetically modified food on the web. Inf Res Int Electron J 2005 Jan;10(2) [FREE Full text]

51. Baromètres de Santé publique France. Santé publique France. URL: https://www.santepubliquefrance.fr/etudes-et-enquetes/barometres-de-sante-publique-france [accessed 2023-02-23]

52. Norman CD, Skinner HA. eHEALS: the eHealth literacy scale. J Med Internet Res 2006 Nov 14;8(4):e27 [FREE Full text] [doi: 10.2196/jmir.8.4.e27] [Medline: 17213046]

53. Khazaal Y, Chatton A, Zullino D, Coquard O. Version brève du DISCERN, une voie d'amélioration de la recherche d'information médicale sur le net. Rev Med Suisse 2009 Sep 16;217:1819

54. Sun Y, Zhang Y, Gwizdka J, Trace CB. Consumer evaluation of the quality of online health information: systematic literature review of relevant criteria and indicators. J Med Internet Res 2019 May 02;21(5):e12522 [FREE Full text] [doi: 10.2196/12522] [Medline: 31045507]

55. Ayani S, Sadoughi F, Jabari R, Moulaei K, Ashrafi-Rizi H. Evaluation criteria for health websites: critical review. Front Health Inform 2020 Sep;9(1):44 [doi: 10.30699/fhi.v9i1.235]

56. Davis FD, Bagozzi RP, Warshaw PR. User acceptance of computer technology: a comparison of two theoretical models. Manag Sci 1989 Aug 01;35(8):982-1003 [doi: 10.1287/mnsc.35.8.982]

57. Kelly L, Ziebland S, Jenkinson C. Measuring the effects of online health information: scale validation for the e-health impact questionnaire. Patient Educ Couns 2015 Nov;98(11):1418-1424 [FREE Full text] [doi: 10.1016/j.pec.2015.06.008] [Medline: 26162953]

58. Zins M, Goldberg M, CONSTANCES team. The French CONSTANCES population-based cohort: design, inclusion and follow-up. Eur J Epidemiol 2015 Dec;30(12):1317-1328 [FREE Full text] [doi: 10.1007/s10654-015-0096-4] [Medline: 26520638]

59. Vandentorren S, Le Méner E, Oppenchaim N, Arnaud A, Jangal C, Caum C, et al. Characteristics and health of homeless families: the ENFAMS survey in the Paris region, France 2013. Eur J Public Health 2016 Feb;26(1):71-76 [doi: 10.1093/eurpub/ckv187] [Medline: 26511600]

Abbreviations

eHEALS: eHealth Literacy Scale
Edited by A Mavragani; submitted 21.07.23; peer-reviewed by SH Hong, A Anastasiou, S Pitoglou; comments to author 01.09.23; revised version received 12.10.23; accepted 14.10.23; published 24.11.23

Please cite as:

Estevez M, Domecq S, Montagni I, Ramel V

Evaluating a Public Health Information Service According to Users' Socioeconomic Position and Health Status: Protocol for a Cross-Sectional Study

JMIR Res Protoc 2023;12:e51123

URL: https://www.researchprotocols.org/2023/1/e51123

doi: 10.2196/51123

PMID: 37999943

©Mégane Estevez, Sandrine Domecq, Ilaria Montagni, Viviane Ramel. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 24.11.2023. This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.