# Uses of the viable validity concept: A systematic scoping review

Césarine Sambou, Charlotte Decroix, Judith Martin-Fernandez, Linda Cambon, Francois Alla

► To cite this version:

Césarine Sambou, Charlotte Decroix, Judith Martin-Fernandez, Linda Cambon, Francois Alla. Uses of the viable validity concept: A systematic scoping review. Evaluation and Program Planning, 2024, 108, pp.102516. 10.1016/j.evalprogplan.2024.102516. hal-04801681

HAL Id: hal-04801681

https://hal.science/hal-04801681v1

Submitted on 25 Nov 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Uses of the viable validity concept: A systematic scoping review

Césarine Sambou<sup>a,*</sup>, Charlotte Decroix<sup>a</sup>, Judith Martin-Fernandez<sup>a</sup>, Linda Cambon<sup>a</sup>,
François Alla<sup>a,b</sup>

## ARTICLE INFO

**Keywords:**

Viable validity

Viability

Scoping review

Chen

Research traditions

Evaluation

## ABSTRACT

**Objective:** The concept of viable validity was first defined in 2010 within the framework of the integrative validity model. The concept has continued to evolve in the intervening years, and the purpose of this systematic scoping review is to describe and analyze the ways in which it has been deployed and appropriated by various research traditions.

**Methods:** We began by including all articles which cite Chen's original article "The bottom-up approach to integrative validity: a new perspective for program evaluation (Eval Program Plann. 2010;33(3):205-14) and/or contain the terms "viable validity" or "viable cogency," sourced from 5 databases (PubMed, Web of Science, Scopus, Psycinfo and ResearchGate).

**Results:** we selected and included 31 articles published between 2011 and 2022. These studies fall into three major research traditions (evaluation science, population health intervention research and humanities and social sciences), providing a broad overview of the conceptual mobilization of viable validity. Paradoxically, our literature reveals the concept of viable validity to be poorly operationalized and only partially mature, owing to a lack of consensus among the research traditions with regard to its definition, as well as the porous boundaries between this concept and adjacent concepts such as feasibility and acceptability.

**Conclusion:** Viable validity is a complex concept, and its operational application constitutes a major challenge for research into and evaluation of population health interventions.

## 1. Introduction

Some decades ago, Campbell & Stanley (Campbell & Stanley, 1963) proposed a model of validity which has had a profound influence on subsequent research and evaluation. Campbell's model distinguished between two types of validity: internal and external. Internal validity "is concerned with ascertaining whether or not, in a specific experimental case, the intervention made a difference," whereas external validity "wants to know if an experimental effect can be generalized, and if so to which populations, to which contexts, and using which processing and measuring variables." (Chen et al., 2014). In an article published in 2010, Chen (Chen, 2010) questions the value of this model when applied to population health interventions (PHI), widely regarded as a complex. That complexity is particularly evident in the role of the environment in producing health effects (Pawson & Tilley, 1997; Skivington et al., 2021). Researchers have identified numerous limitations which restrict

the applicability of Campbell's model to PHI research: an exaggerated emphasis on internal validity, failure to take account of the needs and perspectives of stakeholders in the real world, and a failure to take into consideration the practical and functional aspects of population health interventions, limiting their usefulness in real-world conditions (Chen & Garbe, 2011; Chen, 2010). Hence Chen's proposal to supplement internal and external validity with a third category, which he calls viable validity.

Viable validity invokes the perspective of the stakeholders involved in an intervention, exploring the capacity of the intervention to integrate with the system within which it is to be deployed, in the real world rather than the research environment. The concept seeks to ascertain whether stakeholders' experience of an intervention is "practical, affordable, appropriate, useful and measurable in the real world » (Chen, 2010). Practicality, in this context, refers to the capacity of an intervention to be adequately implemented by ordinary practicians, rather

<footnote><sup>a</sup> Bordeaux Population Health Research Center U 1219, Inserm - University of Bordeaux, Case 11, 146 rue Léon Saignat, Bordeaux Cedexx 33076, France</footnote>

<footnote><sup>b</sup> Bordeaux University Hospital -Saint-André Hospital, 1 rue Jean Burguet, Bordeaux 33000, France</footnote>

<footnote>* Corresponding author.</footnote>

<footnote>E-mail addresses: cesarine.sambou@u-bordeaux.fr (C. Sambou), charlotte.decroix@u-bordeaux.fr (C. Decroix), judith.martin-fernandez@u-bordeaux.fr (J. Martin-Fernandez), linda.cambon@u-bordeaux.fr (L. Cambon), francois.alla@u-bordeaux.fr (F. Alla).</footnote>
than researchers. An intervention can be regarded as appropriate or
adaptable if it can be integrated into the existing organization of the
actors involved, adapting to their standard organizational structures so
that the associated activities can be effectively coordinated. Afford-
ability refers to the capacity of an organization responsible for imple-
menting an intervention to recruit and/or retain ordinary participants
without financial compensation, and thus to bear the cost of the inter-
vention. An intervention can be regarded as useful if the stakeholders
involved notice and experience an improvement, in terms of attenuating
or resolving an existing problem. Measurability examines whether the
expected results of an intervention are clear and measurable, and if there
is a consistent logic which determines the capacity of the intervention to
deliver the expected results (Chen, 2010; Chen et al., 2014).

Chen argues that we should always begin by evaluating viable val-
idity before seeking to measure the efficacy of an intervention (Chen,
2010). To that end, pilot studies are of utmost importance when pre-
paring an intervention, as well as evaluation measures. Pilot studies seek
to gather information in order to prepare for deployment on a larger
scale. As Thabane and co-authors have recently shown (Thabane et al.,
2019), viability studies can help us to appreciate interventions in their
operational context.

Viability studies feature prominently in population health interven-
tion research (Cambon et al., 2019; Decroix et al., 2022; Thabane et al.,
2019). Further methodological and operational research is needed to
clarify this concept, and this is precisely the purpose of the VAPS pro-
gram (Viability Assessment of population health interventions within
Pilot Studies) (Decroix et al., 2023). The VAPS program seeks to
re-examine viability as defined by Chen, studying the concept with
reference to research in various fields of public health studies. To do so,
we need a better understanding of the ways in which researchers,
depending on their research paradigm, engage with the question of
integrating interventions into the systems within which they are inten-
ded to be deployed. On a theoretical level, the advantages of employing
the concept of viable validity when developing complex interventions
has already been demonstrated (Cambon et al., 2019; Chen, 2010;
Decroix et al., 2022; Thabane et al., 2019). Nevertheless, we need to
explore whether and how researchers have taken up the concept: who is
using it? How? From what perspectives? In this article we present the
results of a literature review encompassing the use and understanding of
the concept of viable validity.

Taking as our starting point the definition of the concept of viable
validity proposed by Chen in 2010, our objectives for this literature
review were: i) to identify the research traditions which have taken up
the concept of “viable validity” and the theoretical and methodological
approaches utilized; ii) to understand how different authors have
appropriated the concept, and from what perspectives (full or partial
mobilization of the five criteria, conceptual or operational perspectives);
iii) to grasp the positioning of the concept in relation to other, related
terms (feasibility, acceptability, longevity etc.).

## 2. Methods

In conducting this systematic scoping review and drafting the present
article, we have made use of the PRISMA Extension for Scoping Review
(PRISMA-ScR) checklist (Tricco et al., 2018).

### 2.1. *Identification of articles*

In the interests of achieving an interdisciplinary perspective, we
searched for articles in 5 different databases: PubMed, Web of Science,
Scopus, Psycinfo and ResearchGate. In order to identify pertinent arti-
cles, we made use of two search strategies. The first was to identify all
publications citing the original article by Chen: ‘The bottom-up approach
to integrative validity: a new perspective for program evaluation’ (Eval
Program Plann. 2010;33(3):205–14). This gave us a list of all articles in
which Chen’s article is cited (using the search function “cited by”).

Further to this, we searched the key words *viable validity*, * viable cogency* and *Evaluation* using the Boolean operators OR and AND to form a search equation: (“viable validity” OR “viable cogency”) AND (Evaluation).

### 2.2. Eligibility criteria

Inclusion and exclusion criteria were imposed in order to effectively
target our review. We thus included all articles: - written in English or
French and published since 2010 which utilize the concept of “viability”
and/or “viable validity” while making explicit reference to Chen; -
including relating terms such as feasibility, acceptability, transferability,
durability etc. The exclusion criteria were as follows:

- All studies which cited the source article but did not utilize the concept of “viability” and/or “viable validity”;

- Articles which did include the term “viable validity” but without actually engaging with questions of viability or transferability (for example, articles which “erroneously” cited the concept without including a definition, and without addressing the issue of transferability);

- Chapters in books and lectures delivered at conferences, since these
contributions are not “peer-reviewed”;

- Articles which were not available in their entirety.

### 2.3. *Selecting the articles*

We thus identified an initial list of 238 studies using Covidence®, an
online platform for systematic reviews which complies with the meth-
odological standards set out by the Cochrane Community. Several arti-
cles appeared more than once on the list, and these repetitions were
removed. We then conducted a first round of selection, identifying the
studies whose titles and abstracts clearly met our inclusion criteria.
These titles and abstracts were then subjected to independent evaluation
by two research engineers (C.S and C.D) in order to verify their eligi-
bility. If there was any disagreement, the two evaluators re-read the text
together to reach a consensus. A third evaluator (F.A) was consulted
whenever difficulties or questions arose during the process of selecting
eligible titles and abstracts, and during the in-depth analysis of the ar-
ticles. F.A. was also consulted whenever difficulties or questions arose
during the process of selecting and extracting the data.

### 2.4. Content analysis

The full text of the selected articles was analyzed using a specially
constructed analytical framework, comprising five main themes:

* Identification of articles for which data such as author name(s), title, contact details, year of publication, location of study, publication journal and article type were available;

* Role of viability in the scientific process, with a particular focus on the degree to which the concept of “viable validity” is utilized (e.g. is viability the main object of the article, or is the concept invoked in a short passage of the article when discussing a result?), and the perspective adopted (e.g. is viability the object of the research or evaluation and/or a method?);

* Conceptual/operational mobilization of the concept of viable val-
idity and related terms. We looked at the properties and meanings
attached to the concept of “viable validity” as well as related terms
such as feasibility, acceptability, longevity etc., mobilization (partial
or total) of criteria and definitions adopted for the criteria used;

* Analysis of the methods employed, looking at information such as data gathering and analysis tools, stakeholders involved (explicit or implied targets of the research) and the justification offered for methodological choices;
* Questions of viability and transferability/upscaling, for example discussions of Chen's approach and the conditions associated with the upscaling/transfer of interventions.

We analyzed the content of all of the articles using the concept of research traditions. A research tradition is defined as a set of interrelated studies based upon a consistent paradigm (methodological, theoretical or hypothetical approaches etc.) shared by a group of scientists at a given moment in time (Wong et al., 2013). As per Kuhn's definition (Kuhn, 1996), scientific research occurs within the framework of research traditions based on specific paradigms which embody the shared understanding that a scientific community has of its work, and its commitment to a set of rules and norms. According to (Ridde Valérie, 2021), a paradigm comprises four interconnected dimensions: a) epistemology (relation to data); b) methodology (methods, procedures, techniques); c) ontology (potential manipulation of the objects of the research or the evaluation) and d) teleology (the intentions and interests of researchers, the purpose of their evaluation). These four dimensions were used to identify pertinent articles and assign each to an appropriate research tradition.

This process of identification and categorization involved three steps: i) identifying, for each article, the applicable paradigm on the basis of our four dimensions (epistemology, methodology, ontology and teleology), ii) assigning each article to a domain or field of research (based on elements clearly declared or identifiable in the text and/or the object of study), iii) grouping together articles by domain or field of research and by paradigm to identify research traditions.

As such, for the purposes of this review, a research tradition may correspond to a research field/domain, a research object or an inter-disciplinary trend.

## 3. Results

The results of our searches in the five databases and our process of article selection are presented in the PRISMA diagram (Fig. 1) below.

A total of 31 articles met our eligibility criteria.

### 3.1. Description of the selected articles

The 31 articles selected for inclusion date from the period 2011–2022. The studies they describe were mainly conducted in USA and France. Among the 31 articles we identified:

Twelve are methodological articles, concerned with research and evaluation methodologies (Cambon & Alla, 2019; Cambon et al., 2012b; Feyaerts et al., 2017b; Harman & Azzam, 2017; Im et al., 2022; Schalock et al., 2017; Schalock et al., 2011; Spencer et al., 2013; Stevens et al., 2021; Thabane et al., 2019; Ton, 2012; Urban et al., 2014)

Twelve are empirical studies focusing on one or more interventions, whose primary objective is not to present the results of a viability study but do mention viability in at least part of the article (Borge et al., 2022; Chen & Turner, 2012; L. Durá et al., 2014; Feyaerts et al., 2017a; Keller et al., 2012; Kothari et al., 2012; Kuhnley & Cueva, 2011; Luesse & Contento, 2018; Martin et al., 2017b; Ribeiro Santiago & Colussi, 2018; Vallata et al., 2022; Vanderpool et al., 2011).

Five articles (Chen et al., 2014; Harshbarger et al., 2019; Inrig et al., 2017; O'Toole et al., 2015; Riches et al., 2020) are primarily devoted to presenting the results of viability studies on intervention(s) (regardless of title)

Two articles contained the protocols of viability studies (Beckerman-Hsu et al., 2020; Martin-Fernandez et al., 2022).

Table 1 contains descriptive information regarding the selected articles.

The studies identified as being methodological or empirical in nature generally mention viability or the concept of “viable validity” in their discussion section. In the five articles devoted to presenting the results of viability studies, viability is mentioned throughout the text. These five articles presenting the results of viability studies have helped to identify the challenges of assessing viable validity. One of the challenges of these viability studies is to give much more power and capacity to stakeholders in the implementation of programs and/or interventions before

<figure><img src="image_2.png" /><figcaption>Fig. 1. The flow diagram of the study selection.</figcaption></figure>
<table><caption>**Table 1**
Description of the selected articles.</caption><thead><tr><th>Authors</th><th>Year</th><th>Type of article</th><th>Definition of the concept of “viable validity” adopted</th></tr></thead><tbody><tr><td>Martin (Martin et al., 2017a)</td><td>2017</td><td>Empirical</td><td>Viable validity that findings are “practical, affordable, appropriate, useful and measurable in the real world.”</td></tr><tr><td>O'Toole (O'Toole et al., 2015)</td><td>2015</td><td>Viability</td><td>Describes the extent to which an intervention/evaluation tool is practical, affordable, appropriate, measurable and useful in the real world.</td></tr><tr><td>Vanderpool (Vanderpool et al., 2011)</td><td>2011</td><td>Empirical</td><td>Focuses on the perspectives and experiences of ordinary practitioners, with interventions tested in real environments.</td></tr><tr><td>Cambon (Cambon & Alla, 2019)</td><td>2019</td><td>Methodological</td><td>Analyzing viability is about checking that an intervention can become a routine presence in real environments. This is a prerequisite for efficacy studies, as recommended by Chen.</td></tr><tr><td>Martin-Fernandez (Martin-Fernandez et al., 2022)</td><td>2022</td><td>Protocol</td><td>Evaluates the extent to which an intervention is viable in the real world. Analyzing viability is also essential when studying transferability.</td></tr><tr><td>Schalock (Schalock et al., 2011)</td><td>2011</td><td>Methodological</td><td>The extent to which evaluation of efficacy can be generalized from a research setting to a real-world environment. Helps to bridge the gap between interventional research and practice.</td></tr><tr><td>Stevens (Stevens et al., 2021)</td><td>2021</td><td>Methodological</td><td>Sine qua non condition for the development of innovations. Analyzing the viability of innovations during pilot studies enables us to study their transferability.</td></tr><tr><td>Urban (Urban et al., 2014)</td><td>2014</td><td>Methodological</td><td>A pertinent type of validity because a program which satisfies the criteria of all the usual types of validity (construction, conclusion, internal, external) may still fail if viable validity is not present.</td></tr><tr><td>Feyaerts (Feyaerts et al., 2017b)</td><td>2017</td><td>Methodological</td><td>Refers to the perspectives and experiences of “stakeholders, in order to ascertain whether or not an intervention program is “practical, affordable, appropriate, measurable and useful in the real world.”</td></tr><tr><td>Chen (Chen & Turner, 2012)</td><td>2012</td><td>Empirical</td><td>Shows that stakeholders are responsible for organization and implementation, and concerned with the viability of interventions.</td></tr><tr><td>Kothari (Kothari et al., 2012)</td><td>2012</td><td>Empirical</td><td>Invites stakeholders to tacitly acknowledge the practicality, pertinence and acceptability of a public health program before</td></tr></tbody></table>

<table><caption>**Table 1 (continued)**</caption><thead><tr><th>Authors</th><th>Year</th><th>Type of article</th><th>Definition of the concept of “viable validity” adopted</th></tr></thead><tbody><tr><td>Ribeiro Santiago (Ribeiro Santiago & Colussi, 2018)</td><td>2018</td><td>Empirical</td><td>considering questions of efficacy.</td></tr><tr><td>Cambon (Cambon et al., 2012a)</td><td>2012</td><td>Methodological</td><td>Not present</td></tr><tr><td>Vallata (Vallata et al., 2022)</td><td>2021</td><td>Empirical</td><td>Evaluates the capacity of an intervention to recruit and/or retain ordinary people, and to be implemented to a satisfactory standard by ordinary practitioners.</td></tr><tr><td>Beckerman (Beckerman-Hsu et al., 2020)</td><td>2020</td><td>Protocol</td><td>An indispensable step in the process of translating research into practice.</td></tr><tr><td>Schalock (Schalock et al., 2017)</td><td>2017</td><td>Methodological</td><td>Not present</td></tr><tr><td>Spencer (Spencer et al., 2013)</td><td>2013</td><td>Methodological</td><td>Expands the concept of external validity, and focuses on the extent to which results can be generalized from research settings to real settings, and from one real setting to another.</td></tr><tr><td>Feyaerts (Feyaerts et al., 2017a)</td><td>2017</td><td>Empirical</td><td>Not present</td></tr><tr><td>Chen (Chen et al., 2014)</td><td>2014</td><td>Viability</td><td>The proof that an intervention is successful in the real world and enables political decision-makers to know whether or not a measure is “pertinent and usable” in practice, demonstrating its ability to solve a specific problem.</td></tr><tr><td>Thabane (Thabane et al., 2019)</td><td>2019</td><td>Methodological</td><td>The extent to which an intervention is capable of succeeding in the real world. The generalization process described herein uses viability as a catch-all encompassing the durability, feasibility and scope of an intervention.</td></tr><tr><td>Keller (Keller et al., 2012)</td><td>2012</td><td>Empirical</td><td>The extent to which an evaluation provides evidence that an intervention can succeed in the real world, and permits a pilot study to explore and understand how the intervention operates in that real context.</td></tr><tr><td>Harshbarger (Harshbarger et al., 2019)</td><td>2019</td><td>Viability</td><td>Is guided by the viewpoints and interests of stakeholders, and takes real-world participants into consideration in their natural environments, while also considering their mores and cultural values.</td></tr><tr><td>Ton (Ton, 2012)</td><td>2012</td><td>Methodological</td><td>Not present</td></tr><tr><td></td><td></td><td></td><td>Highlights the importance of verifying the practical feasibility of implementing interventions in real conditions, noting the risk factors which may cause practical implementation to fail.</td></tr></tbody></table>

(continued on next page)
<table><caption>**Table 1 (continued)**</caption><thead><tr><th>Authors</th><th>Year</th><th>Type of article</th><th>Definition of the concept of "viable validity" adopted</th></tr></thead><tbody><tr><td>Riches (Riches et al., 2020)</td><td>2020</td><td>Viability</td><td>Looks to the contributions of stakeholders to determine whether an intervention is affordable, practical, useful in the real world and capable of being implemented without the supervision of researchers. The viability of a program may not guarantee its validity; nonetheless, a weak viability rating means that an intervention has little chance of succeeding in its target context.</td></tr><tr><td>Im (Im et al., 2022)</td><td>2022</td><td>Methodological</td><td>An intervention's chances of surviving in the community, regardless of its efficacy, are increased if it can demonstrate that it is practical, compatible with the implementation capacities of community organizations, and acceptable for both clients and those responsible for implementation.</td></tr><tr><td>Kuhnley (Kuhnley & Cueva, 2011)</td><td>2011</td><td>Empirical</td><td>Not present</td></tr><tr><td>Harman (Harman & Azzam, 2017)</td><td>2017</td><td>Methodological</td><td></td></tr><tr><td>Luesse (Luesse & Contento, 2019)</td><td>2018</td><td>Empirical</td><td>Viable validity determines whether a program is practical, affordable, appropriate, measurable and useful when designing or testing new initiatives. The viability of an intervention (the evidence that it is appropriate, practical, affordable, and useful) is what ensures that the intervention being developed is directly informed by stakeholder input, and thus has more chance of being implemented</td></tr><tr><td>Borge (Borge et al., 2022)</td><td>2022</td><td>Empirical</td><td>The extent to which a program is practical, affordable, appropriate and useful in the real world Viability measures the extent to which an organization or county authority could make a significant contribution to BSPAN (Breast Screening & Patient Navigation) to combat breast cancer among uninsured and under-insured women in Texas.</td></tr><tr><td>Durá (Lucía Durá et al., 2014)</td><td>2014</td><td>Empirical</td><td></td></tr><tr><td>Inrig (Inrig et al., 2017)</td><td>2017</td><td>Viability</td><td></td></tr></tbody></table>

scaling up.

For Riches and colleagues (Riches et al., 2020), the assessment of viable validity has made it possible to identify, during the first phase, the variability of the commitment of the various stakeholders, particularly teachers. Most teachers wanted more power to implement activities mainly or entirely online, to streamline and standardize activities. This stakeholder desire resulted in a modification of the activities during the second phase toward a fully online delivery system. With the move online, teachers appreciated the activities with greater commitment. Thus, thanks to the viability assessment, the authors (Riches et al., 2020)

suggest that “the intervention could be designed to be delivered during an average class period with little to no teacher involvement”.

Inrig and colleagues (Inrig et al., 2016) agree with us. One of the aims of their study was to allow stakeholders to see and engage with their organization's and county's ability to contribute to the success of the BSPAN (Breast Screening & Patient Navigation) program in the real world. BSPAN is a program to expand rural access to breast cancer screening procedures for populations in five underserved counties in Texas. The viability assessment went a long way toward showing how “their organization or county was able to contribute significantly to BSPAN's efforts in breast cancer screening for uninsured and underinsured women in their county.” For example, it helped to reduce the likelihood that stakeholders in a county would reject the intervention simply because the program had not identified or explored factors that were critical to the county's perspective. Additionally, the viable validity assessment promoted consensus among program staff and stakeholders on the accuracy of the final designation of the county's capacity to conduct breast cancer awareness and screening activities.

The second issue identified was to bring about positive change for stakeholders, as shown by Chen and co-authors (Chen et al., 2014). According to these authors, the carbon monoxide alarm ordinance in Mecklenburg County, North Carolina has strengthened the capacity of different stakeholders (firefighters, emergency medical service personnel and the health department) to address the problem of CO poisoning by standardizing and routinizing procedures (such as building permit inspections etc.) and residents to comply with the law.

Only one study (Chen et al., 2014) empirically tests all five viability criteria. The other four “viability studies” only partially operationalize the concept of viability using selected criteria: useful (O'Toole et al., 2015; Riches et al., 2020), practical (Inrig et al., 2017; Riches et al., 2020), appropriate or adapted (Harshbarger et al., 2019; Inrig et al., 2017) and affordable (Riches et al., 2020). The measurability criterion was not operationalized in these four studies. The table below shows the different criteria mobilized in the articles presenting the results of viability studies, along with the various meanings attached to these criteria. (Table 2)

## 3.2. Description of research traditions making use of the concept of “viable validity”

Three research traditions have clearly engaged with the concept of viable validity. They share certain attributes (paradigm and research field/domain), and their boundaries are more or less agreed-upon. We name these three traditions as follows: evaluation science, PHIR (Population Health Intervention Research) and humanities and social sciences.

Evaluation science foregrounds the use of rigorous scientific methods to tackle questions of particular value to the field of evaluation (Donaldson, 2007). In practice, this approach regards program theory as an element of central importance. Population Health Intervention Research (PHIR) is defined as any research activity which uses scientific methods to produce knowledge pertaining to interventions, in the form of policies or programs, in the health sector or outside the health sector but with potential consequences for the health of a target population (Hawe & Potvin, 2009). PHIR seeks to improve both healthy and equal access to healthcare, tackling underlying social, cultural, economic or environmental conditions or directly engaging with health-related behaviors (Craig et al., 2008). The research tradition we define as “humanities and social sciences” includes a number of disciplines devoted to the study of human behavior in its myriad forms and expressions, both individual and collective. Within this tradition, the position adopted by researchers is of particular importance, and is a research topic in its own right (see (Feuerhahn, 2017). Table 3 describes these research traditions.

Above and beyond the integrative validity model, which is referenced in all of the articles in our selection, these three research traditions draw upon different theoretical approaches. Evaluation science,
<table><caption>**Table 2**

Viability criteria used and associated meanings.</caption><thead><tr><th rowspan="2">Articles</th><th colspan="5">Criteria utilized and associated meanings</th></tr><tr><th>Practical</th><th>Affordable</th><th>Appropriate</th><th>Useful</th><th>Evaluable</th></tr></thead><tbody><tr><td>Chen (Chen et al., 2014)</td><td>Ascertaining whether those responsible for implementation, and the members of the community, are capable of implementing an intervention to a satisfactory standard.</td><td>The capacity of an organization responsible for implementation to meet the cost of coordinating and implementing an intervention.</td><td>The capacity for an intervention to be coordinated or organized by local agencies and community organizations.</td><td>When stakeholders observe and experience progress in terms of alleviating or resolving an existing problem.</td><td>whether the expected results of an intervention are clear and measurable, and if there is a consistent logic which dictates the capacity of the intervention to deliver the expected results.</td></tr><tr><td>Harshbarger (Harshbarger et al., 2019)</td><td>No</td><td>No</td><td>Describes how the tool is generally integrated into workflows by clinical practitioners, from beginning to end of patient consultations.</td><td>No</td><td>No</td></tr><tr><td>Riches (Riches et al., 2020)</td><td>Referring to the ease of use of mobile devices used by pupils in class and at home.</td><td>The activities on offer were free of charge.</td><td>No</td><td>Identifying positive responses to the intervention, and noting major changes in life goals.</td><td>No</td></tr><tr><td>Inrig (Inrig et al., 2017)</td><td>Refers to the perception that the program fits well with the existing clinical information system and protocols.</td><td>No</td><td>Refers to the fact that the staff and participating women succeeded in running the program.</td><td>No</td><td>No</td></tr><tr><td>O'Toole (O'Toole et al., 2015)</td><td>No</td><td>No</td><td>No</td><td>An indicator measuring the improvement, enhanced well-being or problem resolution perceived by beneficiaries.</td><td>No</td></tr></tbody></table>

<table><caption>**Table 3**

Description of the research traditions and their theoretical and methodological approaches.</caption><thead><tr><th>Research traditions</th><th>Dominant paradigm</th><th>Number of articles</th><th>Types of articles</th><th>Methodological and theoretical approaches</th></tr></thead><tbody><tr><td>Evaluation science</td><td>(Post)positivist-evolutionist</td><td>10 articles:<ul><li>8 (post)positivist (Chen et al., 2014; Harman & Azzam, 2017; Keller et al., 2012; Martin et al., 2017a; Schalock et al., 2017; Schalock et al., 2011; Spencer et al., 2013; Ton, 2012);</li><li>2 evolutionist (Riches et al., 2020; Urban et al., 2014)</li></ul></td><td><ul><li>6 methodological articles (Harman & Azzam, 2017; Schalock et al., 2017; Schalock et al., 2011; Spencer et al., 2013; Ton, 2012; Urban et al., 2014);</li><li>2 viability articles (Chen et al., 2014; Riches et al., 2020);</li><li>2 empirical articles (Keller et al., 2012; Martin et al., 2017a)</li></ul></td><td>Mixed methods: qualitative (interviews, observations, study of documents, crowdsourcing, literature review) and quantitative (questionnaires, surveys).<br>Theoretical approaches: evolutionary evaluation and exhibited generalization</td></tr><tr><td>PHIR</td><td>Constructivist-pragmatic-transformational</td><td>11 articles:<ul><li>5 constructivist (Feyaerts et al., 2017a; Harshbarger et al., 2019; Thabane et al., 2019; Vallata et al., 2022; Vanderpool et al., 2011);</li><li>4 pragmatic (Borge et al., 2022; Chen & Turner, 2012; Inrig et al., 2017; Martin-Fernandez et al., 2022) and</li><li>2 transformational (Cambon & Alla, 2019; Stevens et al., 2021)</li></ul></td><td><ul><li>5 empirical articles (Borge et al., 2022; Chen & Turner, 2012; Feyaerts et al., 2017a; Vallata et al., 2022; Vanderpool et al., 2011);</li><li>3 methodological articles (Cambon & Alla, 2019; Stevens et al., 2021; Thabane et al., 2019);</li><li>2 viability articles (Harshbarger et al., 2019; Inrig et al., 2017)</li></ul></td><td>Mixed methods: qualitative (interviews, observations, case studies, workshops for formal presentations, discussion and co-construction, seminars) and quantitative (questionnaires, surveys).<br>Theoretical approaches: realistic approach, consensus approach</td></tr><tr><td>Humanities and social sciences</td><td>Constructivist-pragmatic</td><td>10 articles:<ul><li>7 constructivist (Cambon et al., 2012a; L. Durá et al., 2014; Feyaerts et al., 2017b; Kothari et al., 2012; Kuhnley & Cueva, 2011; Luesse & Contento, 2019; O'Toole et al., 2015) et</li><li>3 pragmatic (Beckerman-Hsu et al., 2020; Im et al., 2022; Ribeiro Santiago & Colussi, 2018)</li></ul></td><td><ul><li>5 empirical articles (L. Durá et al., 2014; Kothari et al., 2012; Kuhnley & Cueva, 2011; Luesse & Contento, 2018; Ribeiro Santiago & Colussi, 2018);</li><li>3 methodological articles (Cambon et al., 2012a; Feyaerts et al., 2017b; Im et al., 2022)</li><li>1 article on viability (O'Toole et al., 2015)</li><li>1 article on protocol (Beckerman-Hsu et al., 2020)</li></ul></td><td>Mixed methods: qualitative (interviews, observations, focus groups, in person and remote meetings, literature review),<br>quantitative (self-completed questionnaires, controlled randomized testing of social support) and participatory methodology.<br>Theoretical approaches: analytical theory involving stakeholders and social science theories.</td></tr></tbody></table>

for instance, makes use of two main theoretical approaches. The first is the theory of evolutionary evaluation, which is rooted in evolutionary theory and holds that programs alternate between phases of evolution and phases of evaluation, each aligned with different forms of validity (Urban et al., 2014). The second approach prioritizes theory-based evaluations, with Chen's work featuring prominently. This often

includes using Chen's exhibited generalization approach to resolve problems of external validity and transferability (Chen et al., 2014). In the PHIR domain, most studies make use of theory-based evaluations, particularly the realistic approach. The realistic approach seeks to understand how an intervention in a given context succeeds or fails in activating the mechanisms which generate effects (Pawson, 1997). The
humanities and social sciences research tradition is more concerned with social science-indebted and stakeholder theories, mobilizing the knowledge of the stakeholders involved in selecting activities and producing results.

All of these research traditions make use of qualitative or mixed methods (quantitative and qualitative) (Creswell & Plano Clark, 2011). The use of consensus methods is particularly common in PHIR. The consensus approach consists of democratically steering the opinion of a working group towards a consensus solution (Fink et al., 1984).

Regarding definitions, viable validity does not present consensual definitions. The concept lends itself to various interpretations by the different research traditions. Evaluation science offers a more fundamental definition of viability, based on notions of validity (Urban et al., 2014), evidence (Martin et al., 2017a) and generalization (Chen et al., 2014). PHIR adopts a more operational definition, and viability is often compared and contrasted with transferability (Cambon et al., 2012a; Martin-Fernandez et al., 2022; Stevens et al., 2021), defined as the extent to which the measured efficacy of an intervention can be reproduced in a different context (Wang et al., 2006). Several authors (Cambon et al., 2012a; Chen, 2010) argue that analyzing viable validity is an essential responsibility of evaluations. Research in the humanities and social sciences, meanwhile, invokes notions of community (Kuhnley & Cueva, 2011), experience (Feyaerts et al., 2017b), cultural values (L. Durá et al., 2014) and ordinary people (Cambon et al., 2012a) when seeking to define the concept of viable validity. Articles discussing the results of viability studies are found in all three of these research traditions: two in the evaluation science group (Chen et al., 2014; Riches et al., 2020), two in PHIR (Harshbarger et al., 2019; Inrig et al., 2017) and one in humanities and social sciences (O'Toole et al., 2015).

### 3.3. Positioning the concept of viable validity with reference to related terms

The articles in our corpus employ a variety of related terms, including feasibility, acceptability, durability, sustainability and adaptability. Feasibility is the term encountered most frequently, present in all three research traditions (Table 4).

In several of these articles (Ribeiro Santiago & Colussi, 2018), (Harshbarger et al., 2019), (Ton, 2012), we noted the lack of a clear distinction between the concepts of viability, feasibility and acceptability. For example, Ton (2012) conflates viability with practical feasibility. Another article devoted to the results of a viability study (Harshbarger et al., 2019) addresses these related concepts, defining: (i) feasibility as the “extent to which clinical staff are able to successfully administer or use Positive Health Checks (PHC) in intensely busy clinical settings;” (ii) acceptability as “the perception held by stakeholders that the implementation of PHC is pleasant or satisfactory, and that it helps the work of the clinics.” (Harshbarger et al., 2019) These definitions, which are similar to the definition of viability, serve to illustrate that the boundaries between these concepts remain vague.

## 4. Discussion

### 4.1. Viable validity is primarily utilized at the theoretical level

Writing in 2010, Chen (Chen, 2010) fine-tuned Campbell's original distinction between internal and external validity by adding a third dimension, which he called “viable validity.” Our documentary analysis provides a broader overview of the current usage of the concept of viable validity. The results of our literature review suggest that viable validity is used in a predominantly theoretical manner in three main research traditions: evaluation science, PHIC and humanities and social sciences.

At the theoretical level, certain authors acknowledge the need to conduct viability studies on potential interventions before worrying about efficacy. Studying viable validity can allow researchers to explore and understand how an intervention fits into its context, avoiding the pitfall of attempting to assess efficacy “in a vacuum” (Thabane et al., 2019). Furthermore, taking viability into consideration when designing studies can provide evaluators with important data regarding the capacity of stakeholders to implement a program without outside help. For example, for Feyaerts and his co-authors (Feyaerts et al., 2017a), whose specific frame of reference is the city of Brussels' policy for integrating immigrants, the most important criterion for decision-makers is not whether a measure is “scientifically credible,” but rather whether it is “pertinent and usable” in practice, and “useful” in terms of promoting integration. A lack of viability can thus lead to the suspension or failure of a newly launched intervention (Im et al., 2022; Urban et al., 2014). In spite of the high stakes associated with viability, and its strong conceptual maturity, viable validity is still under-operationalized at time of writing.

### 4.2. Operational stakes and challenges

Few researchers have successfully operationalized the concept of viable validity, as witnessed by the relative dearth of articles presenting results of viability studies, and the frequently partial mobilization of the concept (e.g. incomplete mobilization of the viability criteria, viability studies demonstrating scant interest in the real world). The lack of articles operationalizing the concept would suggest that viable validity was a new and largely unknown concept; and yet, it was conceptualized in 2010. These operationalization struggles might be partly explained by the largely theoretical nature of viable validity: there are no practical validation tools available, and few empirically demonstrated results (the first empirical results of a viability study to be published were those presented by (Chen et al., 2014)). These shortcomings raise questions as to the maturity of viable validity as a concept. According to Morse et al. and Pfadenhauer et al. (J. Mitchell Morse et al., 1996; Pfadenhauer et al., 2015), in order for a concept to be considered mature there must be a clear consensus on its definition, with a precise description of its key characteristics, necessary conditions and proven results, as well as clearly defined boundaries distinguishing it from other concepts.

The results of this literature review suggest that these criteria have not yet been met, casting doubt upon the maturity of viability as a concept. Firstly, in order to be mature a concept requires a relatively

<table><caption>**Table 4**
Use of related terms in the research traditions.</caption><thead><tr><th rowspan="2">Research traditions</th><th colspan="5">Related terms</th></tr><tr><th>Feasibility</th><th>Acceptability</th><th>Durability</th><th>Sustainability</th><th>Adaptability</th></tr></thead><tbody><tr><td>Evaluation science</td><td>Yes (Harman & Azzam, 2017; Ton, 2012)</td><td>No</td><td>Yes (Chen et al., 2014; Keller et al., 2012)</td><td>No</td><td>No</td></tr><tr><td>PHIR</td><td>Yes (Borge et al., 2022; Chen & Turner, 2012; Feyaerts et al., 2017a; Harshbarger et al., 2019; Martin-Fernandez et al., 2022; Stevens et al., 2021)</td><td>Yes (Harshbarger et al., 2019; Stevens et al., 2021; Thabane et al., 2019)</td><td>Yes (Inrig et al., 2017; Thabane et al., 2019)</td><td>Yes (Stevens et al., 2021)</td><td>Yes (Vallata et al., 2022)</td></tr><tr><td>Humanities and social sciences</td><td>Yes (Beckerman-Hsu et al., 2020; Im et al., 2022; Ribeiro Santiago & Colussi, 2018)</td><td>Yes (L. Durá et al., 2014; Kuhnley & Cueva, 2011)</td><td>Yes (Im et al., 2022; Ribeiro Santiago & Colussi, 2018)</td><td>No</td><td>No</td></tr></tbody></table>
consistent and stable definition, shared by the authors that invoke it (J. Mitchell *Morse et al., 1996; Pfadenhauer et al., 2015*). Although more than thirty articles draw upon Chen's work (Chen, 2010), each research tradition has its own definition of viability. Furthermore, the five articles we identified which contain results from viability studies all use different definitions of viability (indeed, only the study led by Chen himself included measurability as a criterion (Chen et al., 2014)). The malleability of the concept is evident in Chen's work too, since in 2023 he replaced the measurability criterion with a new criterion he calls "cultural compatibility." It is worth noting that cultural considerations were taken on board in the 2012 study conducted by Keller et al., whose definition of viability includes "the mores and cultural values of participants." (Keller et al., 2012) These discrepancies indicate a lack of consensus regarding the definition of viable validity as a concept. Secondly, our literature review reveals a degree of ambiguity regarding the boundaries between viable validity and related concepts such as feasibility and acceptability. Comparative analysis of the use of these terms in relation to the concept of "viable validity" within the different research traditions suggests that they are perceived as being very close to viability; in some cases, the terms are used interchangeably (Harshbarger et al., 2019; Ribeiro Santiago & Colussi, 2018; Ton, 2012).

One challenge frequently besetting the operational use of the concept is the difficulties inherent to operationalizing all of the viability criteria within the same study, particularly the criterion of measurability. However, taking the example of (Riches et al., 2020), even if they do not mobilize all of the viability criteria their intervention displays a certain form of viable validity by showing itself to be affordable, practical and useful in tackling lack of motivation among students. All of which might lead us to wonder whether the viability of an intervention is really such a binary affair. It is not always necessary to consider all of the criteria when assessing the viability of an intervention. Moreover, all of the criteria are open to different analytical approaches depending on the stakeholders involved and the scale of the analysis (micro-system, meso-system or macro-system). Viability is situated, and depends upon the perceptions, experiences and diversity of stakeholders, as well as the multidimensional character of the criteria and the nature of the results obtained (Decroix, 2024).

While some of the viability criteria are indeed operationalized in the existing literature, it seems abundantly clear that viable validity is not yet fully mature as a concept, due to the lack of a widely shared definition, some clear limitations and a low level of operationalization. These factors could restrict the future development of viable validity as a practical tool, which requires conceptual clarity.

### 4.3. Strengths and limitations of this literature review

This literature review has certain strengths as well as certain limitations. It offers a broad overview of the use of viability as a concept in the existing literature. The subject of our review is the concept of viability as defined by Chen in 2010, engaging with the tradition of theory-based evaluations. Nevertheless, many authors working in the field of population health have conducted pilot studies with objectives very similar to those of viability studies, without invoking Chen and without mentioning viability in their titles (e.g. studies focusing on the feasibility or acceptability of interventions) (Saillour-Glénisson & Salmi, 2023; Skivington et al., 2021). Further research is needed to study the frameworks and concepts developed by other authors for the purpose of evaluating complex interventions, in order to better understand what sets the concept of viability apart from related concepts.

In this respect, the research traditions identified by this review may be of use. The diverse array of databases consulted is one of the strengths of this study, allowing us to include research publications from various disciplines connected with population health. There is certainly room for debate regarding the research traditions we identify, in terms of their boundaries and the different scales at which they operate (discipline, field of study and/or object of study). Nonetheless, the identification of

these three research traditions is a real asset of this review. These three research traditions can provide a working basis for identifying other authors who have used concepts and frameworks closely related to that proposed by Chen.

## 5. Conclusion

This literature review demonstrates that viable validity is a complex concept, and one that is underused in spite of its high degree of conceptualization. The complexity of its definition and its criteria may explain the fact that the latter are often only partially utilized. But even if a study does not set out to test all the stipulated criteria of viable validity, it is important to consider the conditions which determine the viability of an intervention in the real world, and at every step in the intervention life cycle. It seems clear from the existing literature that the operationalization of "viable validity" as a concept remains a major challenge for research into and evaluation of intervention programs. The concept of viability needs to achieve greater maturity in the sense described by Morse (Morse et al., 1996a,b) in order to facilitate its empirical application in evaluative research.

## Funding acknowledgement

La ligue contre le cancer Convention N°AAPPVN2021.LCC/FA (Projet de Recherche Prévention Ligue) and ANR (Agence Nationale de la Recherche) Projet N°ANR-21-CE36-0004-01

## CRediT authorship contribution statement

**François Alla:** Validation, Supervision. **Linda Cambon:** Validation, Supervision. **Judith Martin-Fernandez:** Writing – review & editing, Methodology, Conceptualization. **Charlotte Decroix:** Writing – original draft, Supervision, Methodology, Formal analysis. **Césarine Sambou:** Writing – review & editing, Writing – original draft, Methodology, Formal analysis, Conceptualization.

## References

Beckerman-Hsu, J. P., Aftosmes-Tobio, A., Gavarkovs, A., Kitos, N., Figueroa, R., Kalyoncu, Z. B., Lansburg, K., Yu, X., Kazik, C., Vigilante, A., Leonard, J., Torrico, M., Jurkowski, J. M., & Davison, K. K. (2020). Communities for Healthy Living (CHL) a community-based intervention to prevent obesity in low-income preschool children: Process evaluation protocol [Article]. *Trials*, *21*(1), 674. https://doi.org/10.1186/s13063-020-04571-0

Borge, C., Larsen, M., Osborne, R., Engebretsen, E., Andersen, M., Holter, I., & Wahl, A. (2022). How to co-design a health literacy-informed intervention based on a needs assessment study in chronic obstructive pulmonary disease. *BMJ Open*, *12*, Article e063022. https://doi.org/10.1136/bmjopen-2022-063022

Cambon, L., & Alla, F. (2019). Current challenges in population health intervention research [Review]. *Journal of Epidemiology and Community Health*, *73*(11), 990-992. https://doi.org/10.1136/jech-2019-212225

Cambon, L., Minary, L., Ridde, V., & Alla, F. (2012a). Transferability of interventions in health education: a review. *BMC Public Health*, *12*, 497. https://doi.org/10.1186/1471-2458-12-497

Cambon, L., Minary, L., Ridde, V., & Alla, F. (2012b). Transferability of interventions in health education: a review. *BMC Public Health*, *12*. https://doi.org/10.1186/1471-2458-12-497

Cambon, L., Terral, P., & Alla, F. (2019). From intervention to interventional system: towards greater theorization in population health intervention research. *BMC Public Health*, *19*(1), 339. https://doi.org/10.1186/s12889-019-6663-y

Campbell D.T. Stanley J. (1963). Experimental and quasi-experimental designs for research, Cengage Learning 1st edition Chicago. 89-pages.

Chen, H., & Garbe, P. (2011). Assessing program outcomes from the bottom-up approach: An innovative perspective to outcome evaluation. *New Directions for Evaluation*, *2011*, 93-106. https://doi.org/10.1002/ev.368

Chen, H. T. (2010). The bottom-up approach to integrative validity: A new perspective for program evaluation. *Evaluation and Program Planning*, *33*(3), 205-214. https://doi.org/10.1016/j.evalprogplan.2009.10.002

Chen, H. T., & Turner, N. C. (2012). Formal Theory Versus Stakeholder Theory: New Insights From a Tobacco-Focused Prevention Program Evaluation [Article]. *American Journal of Evaluation*, *33*(3), 395-413. https://doi.org/10.1177/1098214012442802

Chen, H. T., Yip, F., Lavonas, E. J., Iqbal, S., Turner, N., Cobb, B., & Garbe, P. (2014). Using the exhibited generalization approach to evaluate a carbon monoxide alarm
ordinance [Article]. *Evaluation and Program Planning*, **47**, 35-44. https://doi.org/10.1016/j.evalprogplan.2014.06.003

Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008). Developing and Evaluating Complex Interventions: The New Medical Research Council Guidance. *BMJ (Clinical Research ed)*, **337**, a1655. https://doi.org/10.1016/j.ijnurstu.2012.09.010

Creswell, J. W., & Plano Clark, V. L. (2011). *Designing and conducting mixed methods research (Second ed.).* Los Angeles: SAGE Publications.

Decroix, C. (2024). Du développement d'une intervention complexe en santé des populations à sa mise à l'échelle: aspects conceptuels et méthodologiques ses études de viabilité, applications dans le champs de la petite enfance. In U. d. Bordeaux (Ed.), (pp. 269).

Decroix, C., Kervran, C., Cambon, L., & Alla, F. (2022). Fitting health promotion research with real-life conditions: Viability evaluation. In *In Global Handbook of Health Promotion Research*, 1 pp. 625-640. Springer International Publishing. https://doi.org/10.1007/978-3-030-97212-7_41

Decroix, C., Martin-Fernandez, J., Cambon, L., Ridde, V., & Alla, F. (2023). Les défis de l'interdisciplinarité pour la recherche interventionnelle en santé des populations: le cas de la recherche VAPS. *Recherches Qualitatives.*

Donaldson, S. (2007). *Program Theory-Driven Evaluation Science: Strategies and Applications*. https://doi.org/10.4324/9780203809730

Durá, L., Left, L., & Singhal, A. (2014). What counts? For Whom? Cultural beacons and unexpected areas of programmatic impact [1]. *Evaluation and Program Planning*, **44**. https://doi.org/10.1016/j.evalprogplan.2014.01.001

Feuerhahn, W. (2017). Les sciences humaines et sociales: Des disciplines du contexte? *Revue d'histoire des sciences humaines*, **30**, 7-29. https://doi.org/10.4000/rhsh.493

Yeaerts, G., Degueruy, M., Deboosere, P., & De Spiegelaere, M. (2017a). Analysis of the decision-support function of policy assessment in real-world policy making in the field of poverty and social inequalities. Case study on migrant integration policies in the Brussels-Capital Region [Article]. *Environmental Impact Assessment Review*, **67**, 40-48. https://doi.org/10.1016/j.eiar.2017.08.007

Yeaerts, G., Degueruy, M., Deboosere, P., & De Spiegelaere, M. (2017b). Exploration of the functions of health impact assessment in real-world policymaking in the field of social health inequality: towards a conception of conceptual learning [Article]. *Global Health Promotion*, **24**(2), 16-24. https://doi.org/10.1177/1757975916679918

Fink, A., Kosecoff, J., Chassin, M., & Brook, R. H. (1984). Consensus methods: Characteristics and guidelines for use. *American Journal of Public Health*, **74**(9), 979-983. https://doi.org/10.2105/ajph.74.9.979

Harman, E., & Azzam, T. (2017). Towards Program Theory Validation: Crowdsourcing the qualitative analysis of participant experiences. *Evaluation and Program Planning*, **66**. https://doi.org/10.1016/j.evalprogplan.2017.08.008

Harshbarger, C., Burrus, O., Zulkiewicz, B. A., Ortiz, A. M., Galindo, C. A., Garner, B. R., Furberg, R. D., & Lewis, M. A. (2019). Implementing web-based interventions in HIV primary care clinics: Pilot implementation evaluation of positive health check [Article]. *JMIR Formative Research*, **3**(2), Article e10688. https://doi.org/10.2196/10688

Hawe, P., & Potvin, L. (2009). What is population health intervention research? *Canadian Journal of Public Health*, **100**(1). https://www.ncbi.nlm.nih.gov/pubmed/19263977. Suppl I8-14.

Im, G. H., Shin, D., & Park, S. (2022). Suggesting a policy-driven approach to validation in the context of the Test of Proficiency in Korean (TOPIK). *Current Issues in Language Planning*, **23**(2), 214-232. https://doi.org/10.1080/14664208.2021.1984674

Inrig, S., Higashi, R., Tiro, J., Argenbright, K., & Lee, S. (2016). Assessing local capacity to expand rural breast cancer screening and patient navigation: An iterative mixed-method tool. *Evaluation and Program Planning*, **61**. https://doi.org/10.1016/j.evalprogplan.2016.11.006

Inrig, S. J., Higashi, R. T., Tiro, J. A., Argenbright, K. E., & Lee, S. J. (2017). Assessing local capacity to expand rural breast cancer screening and patient navigation: An iterative mixed-method tool. *Evaluation and Program Planning*, **61**, 113-124. https://doi.org/10.1016/j.evalprogplan.2016.11.006

Keller, C., Records, K., Coe, K., Ainsworth, B., López, S. V., Nagle-Williams, A., & Permana, P. (2012). Promotoras' roles in integrative validity and treatment fidelity efforts in randomized controlled trials [Article]. *Family and Community Health*, **35**(2), 120-129. https://doi.org/10.1097/FCH.0b013e31824650a6

Kothari, A., Rudman, D., Dobbins, M., Rouse, M., Sibbald, S., & Edwards, N. (2012). The use of tacit and explicit knowledge in public health: A qualitative study [Article]. *Implementation Science*, **7**(1), 20. https://doi.org/10.1186/1748-5908-7-20

Kuhn, T. S. (1996). *The Structure of Scientific Revolutions*. University of Chicago Press. (https://books.google.fr/books?id=xnj5401VuFMC).

Kuhnley, R., & Cueva, M. (2011). Learning about cancer has brightened my light: Cancer education for alaska community health aides and community health practitioners (CHA/Ps) [Article]. *Journal of Cancer Education*, **26**(3), 522-529. https://doi.org/10.1007/s13187-011-0207-1

Luesse, H., & Contento, I. (2018). Context considerations for developing the in defense of food nutrition education curriculum. *Journal of Nutrition Education and Behavior*, **51**. https://doi.org/10.1016/j.jneb.2018.10.010

Luesse, H. B., & Contento, I. R. (2019). Context considerations for developing the in defense of food nutrition education curriculum [Article]. *Journal of Nutrition Education and Behavior*, **51**(3), 370-378. https://doi.org/10.1016/j.jneb.2018.10.010

Martin-Fernandez, J., Stevens, N., Moriceau, S., Serre, F., Blanc, H., Latourte, E., Auriacombe, M., & Cambon, L. (2022). Realist evaluation of the impact, viability and transferability of an alcohol harm reduction support programme based on mental health recovery: The Vitae study protocol (Article) *BMJ Open*, **12**(8), Article e065361. https://doi.org/10.1136/bmjopen-2022-065361.

Martin, W., Wharf Higgins, J., Pauly, B., & MacDonald, M. (2017a). layers of translation" - Evidence literacy in public health practice: A qualitative secondary analysis [Article]. *BMC Public Health*, **17**(1), 803. https://doi.org/10.1186/s12889-017-4837-z

Martin, W., Wharf Higgins, J., Pauly, B. B., & MacDonald, M. (2017b). Layers of translation" - evidence literacy in public health practice: a qualitative secondary analysis. *BMC Public Health*, **17**(1), 803. https://doi.org/10.1186/s12889-017-4837-z

Morse, J. M., Hupecy, J. E., Mitcham, C., & Lenz, E. R. (1996a). Concept analysis in nursing research: a critical appraisal. *Scholarly Inquiry for Nursing Practice*, **10**(3), 253-277.

Morse, J. M., Mitcham, C., Hupecy, J. E., & Tason, M. C. (1996b). Criteria for concept evaluation. *Journal of advanced nursing*, **24**(2), 385-390. https://doi.org/10.1046/j.1365-2648.1996.18022.x

O'Toole, J. K., Klein, M. D., McLinden, D., Sucharew, H., & DeWitt, T. G. (2015). A pilot study of the creation and implementation of a teaching development assessment tool. *Journal of Graduate Medical Education*, **7**(4), 638-642. https://doi.org/10.4300/jgme-d-14-00439.1

Pawson, R., & Tilley, N. (1997). *Realistic evaluation* (1st edition). SAGE Publications Ltd, 256.

Pfadenhauer, L. M., Mozygemba, K., Gerhardus, A., Hofmann, B., Booth, A., Lysdahl, K. B., Tummers, M., Burns, J., & Rehfuß, E. A. (2015). Context and implementation: A concept analysis towards conceptual maturity. *Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen*, **109**(2), 103-114. https://doi.org/10.1016/j.zefz.2015.01.004

Ribeiro Santiago, P. H., & Colussi, C. F. (2018). Feasibility evaluation of a mindfulness-based intervention for primary care professionals: Proposal of an evaluative model [Article]. *Complementary Therapies in Clinical Practice*, **31**, 57-63. https://doi.org/10.1016/j.ctcp.2018.01.013

Riches, B. R., Benavides, C. M., & Dubon, V. X. (2020). Development of a fostering purpose intervention [Article]. *Evaluation and Program Planning*, **83**, Article 101857. https://doi.org/10.1016/j.evalprogplan.2020.101857

Ridde Valérie, D.T., Devaux-Spatarakis Agathe, Revillard Anne. (2021). *Introduction: la pluralité des approches paradigmatiques*. In Q. E. s. e. b. commun (Ed.), *Evaluation. Fondements, controverses, perspectives*.

Sailbour-Glénisson, F., & Salmi, L. R. (2023). Évaluation des effets d'une intervention complexe. *Revue d'épidémiologie et de Santé Publique*, **71**(2), Article 101377. https://doi.org/10.1016/j.respe.2022.06.002

Schalock, R. L., Gomez, L. E., Verdugo, M. A., & Claes, C. (2017). Evidence and evidence-based practices: Are we there yet? [Article]. *Intellectual and Developmental Disabilities*, **55**(2), 112-119. https://doi.org/10.1352/1934-9556-55.2.112

Schalock, R. L., Verdugo, M. A., & Gomez, L. E. (2011). Evidence-based practices in the field of intellectual and developmental disabilities: An international consensus approach [Article]. *Evaluation and Program Planning*, **34**(3), 273-282. https://doi.org/10.1016/j.evalprogplan.2010.10.004

Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N., French, D. P., McIntosh, E., Petticrew, M., Yrcroft-Malone, J., White, M., & Moore, L. (2021). A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance. *BMJ*, **374**, n2061. https://doi.org/10.1136/bmj.n2061

Spencer, L. M., Schooley, M. W., Anderson, L. A., Kochitztzy, C. S., DeGroff, A. S., Devlin, H. M., & Mercer, S. L. (2013). Seeking best practices: A conceptual framework for planning and improving evidence-based practices [Article]. *Preventing Chronic Disease*, **10**(12), Article 130186. https://doi.org/10.5888/pcd10.130186

Stevens, N., Cambon, L., & Alla, F. (2021). Innovative transformation of the health system through a preventive transition [Article]. *Revue d'épidémiologie et de Santé Publique*, **69**(4), 235-240. https://doi.org/10.1016/j.respe.2021.04.138

Thabane, L., Cambon, L., Potvin, L., Pommier, J., Kivits, J., Minary, L., Nour, K., Blaise, P., Charlesworth, J., & Alla, F. (2019). Population health intervention research: What is the place for pilot studies? *Trials*, **20**(1), 309. https://doi.org/10.1186/s13063-019-3422-4

Ton, G. (2012). The mixing of methods: A three-step process for improving rigour in impact evaluations. *Evaluation*, **18**(1), 5-25. https://doi.org/10.1177/135689011431506

Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., & Straus, S. E. (2018). PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and Explanation. *Annals of Internal Medye*, **169**(7), 467-473. https://doi.org/10.7326/m18-0850

Urban, J. B., Hargraves, M., & Trochim, W. M. (2014). Evolutionary Evaluation: Implications for evaluators, researchers, practitioners, funders and the evidence-based program mandate. *Evaluation and Program Planning*, **45**, 127-139. https://doi.org/10.1016/j.evalprogplan.2014.03.011

Vallata, A., Cadeville, M., Kanski, C., & Alla, F. (2022). TABADO 2: une stratégie d'accompagnement au sevrage tabagique des adolescents en milieu scolaire [Article]. *Global Health Promotion*, **29**(1), 130-138. https://doi.org/10.1177/1757975921997721

Vanderpool, R. C., Gainor, S. J., Conn, M. E., Spencer, C., Allen, A. R., & Kennedy, S. (2011). Adapting and implementing evidence-based cancer education interventions in rural Appalachia: Real world experiences and challenges [Article]. *Rural and Remote Health*, **11**(4), Article, 1807. (https://www.scopus.com/inward/record uri)
eid=2-s2.0-84856620634&partnerID=40&md5=9af9ddc78dd0545fe647e5a6
e79b7abe).

Wang, S., Moss, J. R., & Hiller, J. E. (2006). Applicability and transferability of
interventions in evidence-based public health. *Health Promotion International*, 21(1),
76-83. https://doi.org/10.1093/heapro/dai025

Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013). RAMESES publication standards: Meta-narrative reviews. *BMC Medicine*, *11*, 20. https://doi.org/10.1186/1741-7015-11-20

Sambou Césarine: is a Prevention research engineer in the field of Public Health at the university of Bordeaux since September 2022. She holds a doctorate in Health and Migration Anthropology (2018) and a Master 2 in International Public Health (2012). The subject of her doctoral thesis was "Malaria of return. An anthropology of malaria risk among migrant travelers from sub-Saharan Africa in Bordeaux." Her work focuses on polypathology, the aging process, chronic diseases, the study of viability in evaluation.

Decroix Charlotte: is a post-doctoral researcher in public health. She holds a Masters in administration of public health policies and program, and a University degree in qualitative health research. The subject of her doctoral thesis was: "From the development of a complex public health intervention to its scaling up: conceptual and methodological aspects of viability studies, application in the field of early childhood"

Martin-Fernandez Judith: is a Doctor of Public Health with a Master’s degree in Sociology and Psychology. Thanks to her triple academic background, she has a multidisciplinary approach and expertise. Her areas of methodological expertise include qualitative research, mixed methods, social epidemiology, and evaluation of complex prevention interventions. These skills are mobilized on various themes: prevention, adolescents, harm reduction, addiction, health inequalities, and food insecurity.

Cambon Linda: is an academic teacher-researcher, Public health PhD habilitated to su-
pervise research (HDR). She is responsible of MéRISP axis (PHIR methods) in PHARes team
(translational research in population health), linked to the BPH (Bordeaux Population
Health - Inserm U1219 - Bordeaux University). She also holds the Prevention ISPED
(Institute of Public Health, Epidemiology and Development) Chair. Her research focuses on
theory-driven evaluations in the field of Prevention comprising in particular Trans-
ferability questions and upscaling process.

Alla François: is an academic teacher-researcher, PhD and PH (Hospital Practitioner) in Public Health. He is co-director of ISPED (Bordeaux University Institute of Public Health) and Head of the Methodological Support and Innovation in Prevention Service (SSMIP). Its main areas of research and expertise are health policies, interventions and organizations, particularly in the fields of prevention and health promotion.