DOI: 10.4121/db8f0218-4b17-46c7-b172-4a9cf1dab0d5.v1
title: Data underlying the publication: “It Should Be Relevant, Reliable and Feasible”: Introducing Face, an Instrument for Assessing the Face Validity of Choice Experiments

description: Face validity indicates to what extent participants are engaged in making choices; and understand and interpret the information presented to them in the study as intended by its designer. It is an important but often overlooked aspect of the overall validity of choice experiments and no comprehensive instruments for assessing it are available. Improving the design of choice experiments potentially improves the quality of participants’ responses, which increases the relevance and usability for policy and practice. In this study we developed and tested an instrument to assess the FAce validity of Choice Experiments (FACE) in a uniform, systematic manner. The instrument is based on nine components that are used to define face validity identified from literature: acceptance, clarity, completeness, familiarity, feasibility, legibility, relevance, sensitivity, and transparency. FACE covers these components in 14 questions with 5-point Likert scales on which participants can indicate their level of agreement. <br>1,020 participants completed the instrument following a discrete choice experiment on COVID-19 pandemic preparedness measures in the Netherlands. To evaluate and validate FACE, we used a four-step approach. First, we evaluated the internal consistency of the instrument using Cronbach’s Alpha. Second, we checked how suitable the data was for applying factor analysis. We evaluated the correlation matrix and its determinant, Barlett’s test of Sphericity and Kaiser-Meyer-Olkin’s (KMO) test. Third, we performed exploratory factor analysis to investigate how the components included in the instruments are, using principal component analysis with promax oblique rotations to account for expected correlation among factors. We used the Eigenvalue rule of λ ≥ 1.0 to retain factors. Finally, we converted the Likert scale answer categories into scores of 1 (fully disagree) to 5 (fully agree). Using these scores, we computed factor scores and factor-based scales for each participant, indicating how they related to each factor. We standardized the scores across all components, weighted these by score coefficients and summed them based on the regression method. Finally, we assessed the relationship between face validity scores and socio-demographic groups of participants using linear regression analyses with a dummy dependent variable that takes the value of 1 if the factor-based scale has a value below 25% of the distribution (i.e., lower face validity) and takes the value of 0 otherwise. We related it to respondents’ socio-demographic characteristics, experiment performance and attitude towards the presentation of the decision problem. <br>This first application of FACE showed that the face validity of a choice experiment was determined by whether participants considered its study design to be relevant, reliable and feasible. Moreover, we found that relevance and reliability were most strongly related to characteristics of the survey design, while feasibility was most strongly related to participants’ socio-demographic characteristics. Face validity was assessed high(er) by participants who were younger, male, lower educated, vaccinated against COVID-19, supportive of policy responses to a pandemic situation and sufficiently engaged in the experiment.