# Population health intervention research: what is the place for pilot studies?

Lehana Thabane, Linda Cambon, Louise M Potvin, Jeanine Pommier, Joëlle Kivits, Laetitia Minary, Kareen Nour, Pierre Blaise, Julie Charlesworth, François Alla

► To cite this version:

Lehana Thabane, Linda Cambon, Louise M Potvin, Jeanine Pommier, Joëlle Kivits, et al.. Population health intervention research: what is the place for pilot studies?. Trials, 2019, 20 (1), pp.309. 10.1186/s13063-019-3422-4 . hal-02499959

HAL Id: hal-02499959

https://hal.science/hal-02499959

Submitted on 24 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
COMMENTARY

Open Access

<figure><img src="image_1.png" /></figure>

Check for
updates

Population health intervention research:
what is the place for pilot studies?

Lehana Thabane¹, Linda Cambon<fnref n="2" />, Louise Potvin³, Jeanine Pommier⁴, Joëlle Kivits⁵, Laetitia Minary⁵, Kareen Nour³, Pierre Blaise⁶, Julie Charlesworth⁷, François Alla<fnref n="2" />* and Discussion Panel

**Abstract**

**Background:** An international workshop on population health intervention research (PHIR) was organized to foster exchanges between experts from different disciplines and different fields.

**Aims:** This paper aims to summarize the discussions around one of the issues addressed: the place or role of pilot studies in PHIR. Pilot studies are well-established in biomedical research, but the situation is more ambiguous for PHIR, in which a pilot study could refer to different purposes.

**Methods:** The workshop included formal presentations of participants and moderated discussions. An oral synthesis was carried out by a rapporteur to validate by expert consensus the key points of the discussion and the recommendations. All discussions have been recorded and fully transcribed.

**Discussion:** PHIR generally addresses complex interventions. Thus, numerous tasks may be required to inform the intervention and test different aspects of its design and implementation. While in clinical research the pilot study mainly concerns the preparation of the trial, in PHIR the pilot study focuses on the preparation of both the intervention and the trial. In particular, pilot studies in PHIR could be used for viability evaluation and theory development.

**Recommendations from the workshop participants:** The following recommendations were generated by consensus from the workshop discussions: i) terms need to be clarified for PHIR; ii) reporting and publication should be standardized and transparency should be promoted; iii) the objectives and research questions should drive the methods used and be clearly stated; iv) a pilot study is generally needed for complex intervention evaluation and for research-designed programs; and v) for field-designed programs, it is important to integrate evaluability assessments as pilot studies.

**Conclusion:** Pilot studies play an important role in intervention development and evaluation. In particular, they contribute to a better understanding of the mechanisms of intervention and the conditions of its applicability and transferability. Pilot studies could therefore facilitate evidence-based decisions about design and conduct of main studies aimed to generate evidence to inform public health policy.

**Keywords:** Process evaluation, Complex intervention, Intervention research, Pilot studies

**Introduction and problem statements**

Population health intervention research (PHIR) can be
defined as the use of scientific research methods to pro-
duce knowledge about policy and program interventions,
whether or not they are conducted in the context of the
health system/sector, and have the potential to make an
impact at the population level [1]. There is a growing

interest in PHIR at the present time; indeed, it has
become a priority for health systems [2].

Population health interventions are generally, but not
necessarily, considered as complex interventions, that is,
"made up of various interconnecting parts" [3]. These
interventions can also be considered as complex because
of the influence of the context on their results [4]. The
Medical Research Council Guidance provided recom-
mendations to guide researchers in designing, develop-
ing, and evaluating complex health interventions, and
more specifically process evaluation [5].

<footnote>* Correspondence: francois.alla@u-bordeaux.fr</footnote>

<footnote n="2">²Université de Bordeaux, Bordeaux, France
Full list of author information is available at the end of the article</footnote>
COMMENTARY

Open Access

<figure><img src="image_2.png" /></figure>

Check for
updates

Population health intervention research:
what is the place for pilot studies?

Lehana Thabane¹, Linda Cambon<fnref n="2" />, Louise Potvin³, Jeanine Pommier⁴, Joëlle Kivits⁵, Laetitia Minary⁵, Kareen Nour³, Pierre Blaise⁶, Julie Charlesworth⁷, François Alla<fnref n="2" />* and Discussion Panel

**Abstract**

**Background:** An international workshop on population health intervention research (PHIR) was organized to foster exchanges between experts from different disciplines and different fields.

**Aims:** This paper aims to summarize the discussions around one of the issues addressed: the place or role of pilot studies in PHIR. Pilot studies are well-established in biomedical research, but the situation is more ambiguous for PHIR, in which a pilot study could refer to different purposes.

**Methods:** The workshop included formal presentations of participants and moderated discussions. An oral synthesis was carried out by a rapporteur to validate by expert consensus the key points of the discussion and the recommendations. All discussions have been recorded and fully transcribed.

**Discussion:** PHIR generally addresses complex interventions. Thus, numerous tasks may be required to inform the intervention and test different aspects of its design and implementation. While in clinical research the pilot study mainly concerns the preparation of the trial, in PHIR the pilot study focuses on the preparation of both the intervention and the trial. In particular, pilot studies in PHIR could be used for viability evaluation and theory development.

**Recommendations from the workshop participants:** The following recommendations were generated by consensus from the workshop discussions: i) terms need to be clarified for PHIR; ii) reporting and publication should be standardized and transparency should be promoted; iii) the objectives and research questions should drive the methods used and be clearly stated; iv) a pilot study is generally needed for complex intervention evaluation and for research-designed programs; and v) for field-designed programs, it is important to integrate evaluability assessments as pilot studies.

**Conclusion:** Pilot studies play an important role in intervention development and evaluation. In particular, they contribute to a better understanding of the mechanisms of intervention and the conditions of its applicability and transferability. Pilot studies could therefore facilitate evidence-based decisions about design and conduct of main studies aimed to generate evidence to inform public health policy.

**Keywords:** Process evaluation, Complex intervention, Intervention research, Pilot studies

**Introduction and problem statements**

Population health intervention research (PHIR) can be
defined as the use of scientific research methods to pro-
duce knowledge about policy and program interventions,
whether or not they are conducted in the context of the
health system/sector, and have the potential to make an
impact at the population level [1]. There is a growing

interest in PHIR at the present time; indeed, it has
become a priority for health systems [2].

Population health interventions are generally, but not
necessarily, considered as complex interventions, that is,
"made up of various interconnecting parts" [3]. These
interventions can also be considered as complex because
of the influence of the context on their results [4]. The
Medical Research Council Guidance provided recom-
mendations to guide researchers in designing, develop-
ing, and evaluating complex health interventions, and
more specifically process evaluation [5].

<footnote>* Correspondence: francois.alla@u-bordeaux.fr</footnote>

<footnote n="2">²Université de Bordeaux, Bordeaux, France
Full list of author information is available at the end of the article</footnote>
Despite progress and increasing interest in complex interventions since the MRC definitions in 2000, they continue to present significant challenges for research communities, practitioners, and decision-makers. In particular, issues relating to the transferability of results are critical because the environment and conditions of implementation are themselves determinants of the intervention outcomes [6].

While the MRC guidance on process evaluation [5] represents a key milestone, methods and tools to conduct evaluations need to be refined and there are many outstanding challenges and questions. There is a need not only to develop methods, tools, and practical guidance for researchers seeking to implement the latest MRC guidance, but also to clarify some underlying paradigms and to operationalize the overall research approach, from conceptualization to the dissemination of an intervention.

In France, where PHIR is well developed, the National Coordinated Action for Intervention Research (Action coordonnée pour la recherche interventionnelle en santé publique; ACRISP) was created in 2015 to support the development of research that is both scientifically accurate and useful to practitioners and policymakers; to promote the sharing of experience between researchers, practitioners, and policymakers; to encourage conceptual and methodological reflections; and to make proposals in terms of organization of research, regulation, and funding. In November 2016, ACRISP organized an international workshop bringing together some of the world's leading experts and researchers. The objective was to promote exchanges between researchers from different disciplines due to the complexity of the field, which requires an interdisciplinary approach. The workshop provided an opportunity to share experiences and learning between researchers from various fields, such as clinical research, health services research, and population health intervention research. The invited researchers were particularly interested in methodological research (most of them had published methodological papers). Some of the key issues in PHIR were addressed. The presentations and discussions, in three successive sessions, covered various themes. One of them was the place of pilot studies in PHIR.

The term 'pilot study' is well-established in biomedical research and pilot studies for phase III trials are routinely performed in many areas of clinical research [7]. The situation is more ambiguous for PHIR, in which a pilot study could refer to different purposes. Yet, in this field, the complexity of the intervention, the necessary flexibility of its implementation in order to fit with the context, and the influence of its contextual parameters—individual, social, organizational, physical, etc.—require numerous preparatory tasks prior to the evaluation,

influencing both the way the evaluation will be conducted and its results. Such tasks therefore have to be considered in pilot studies.

This article aims to share and synthesize the discussions, work, and recommendations put forward by experts during this workshop. Practically, the aim is to clarify what kind of pilot studies should be used in PHIR and for which objectives, through an account of the different questions raised and the consensual answers the experts provided. It is not intended to be a systematic synthesis of science or to bring new data, but to be a milestone for a common basis for discussion between researchers from different disciplines and fields.

## Methods

The workshop was prepared by LT and FA (preparation of a program and compilation of a bibliographic file). It included formal presentations of participants and discussions moderated by LT. At the end of the workshop, an oral synthesis was carried out by a rapporteur (JP) to validate by expert consensus the key points of the discussion and the recommendations. All the discussions and the validated synthesis were recorded. The recording has been fully transcribed. A first draft of the paper was prepared by JC and FA from this material, then corrected and validated by LT, then by all of the coauthors (who all participated in the debates).

## Discussion and key arguments

### What is a pilot and/or feasibility study?

There are many different definitions of pilot studies. These definitions have in common the concept of doing something on a smaller scale with the intent of gathering information to do something on a larger scale. However, a small-sized study is not necessarily in itself a pilot study. There are issues regarding language and terminology. Synonymous terms include feasibility study, vanguard study, pre-study, and first phase. Multiple terms are used without clear distinctions, e.g. pilot work, pilot study, feasibility study, pilot trial [8].

The terms pilot and feasibility study have often been used interchangeably [7]. There is no consensus on their definitions. Some people consider the two to be the same; others think they are different [9]. Some have used an overarching definition of feasibility studies, with pilot studies being a subset: "a feasibility study asks whether something can be done, should we proceed with it, and if so, how. A pilot study asks the same questions but also has a specific design feature: in a pilot study a future study, or part of a future study, is conducted on a smaller scale" [10].

Moreover, in current practice, there are questions about how to describe 'preparatory trials', i.e., the work done before starting a study. In some circumstances
Despite progress and increasing interest in complex interventions since the MRC definitions in 2000, they continue to present significant challenges for research communities, practitioners, and decision-makers. In particular, issues relating to the transferability of results are critical because the environment and conditions of implementation are themselves determinants of the intervention outcomes [6].

While the MRC guidance on process evaluation [5] represents a key milestone, methods and tools to conduct evaluations need to be refined and there are many outstanding challenges and questions. There is a need not only to develop methods, tools, and practical guidance for researchers seeking to implement the latest MRC guidance, but also to clarify some underlying paradigms and to operationalize the overall research approach, from conceptualization to the dissemination of an intervention.

In France, where PHIR is well developed, the National Coordinated Action for Intervention Research (Action coordonnée pour la recherche interventionnelle en santé publique; ACRISP) was created in 2015 to support the development of research that is both scientifically accurate and useful to practitioners and policymakers; to promote the sharing of experience between researchers, practitioners, and policymakers; to encourage conceptual and methodological reflections; and to make proposals in terms of organization of research, regulation, and funding. In November 2016, ACRISP organized an international workshop bringing together some of the world's leading experts and researchers. The objective was to promote exchanges between researchers from different disciplines due to the complexity of the field, which requires an interdisciplinary approach. The workshop provided an opportunity to share experiences and learning between researchers from various fields, such as clinical research, health services research, and population health intervention research. The invited researchers were particularly interested in methodological research (most of them had published methodological papers). Some of the key issues in PHIR were addressed. The presentations and discussions, in three successive sessions, covered various themes. One of them was the place of pilot studies in PHIR.

The term 'pilot study' is well-established in biomedical research and pilot studies for phase III trials are routinely performed in many areas of clinical research [7]. The situation is more ambiguous for PHIR, in which a pilot study could refer to different purposes. Yet, in this field, the complexity of the intervention, the necessary flexibility of its implementation in order to fit with the context, and the influence of its contextual parameters—individual, social, organizational, physical, etc.—require numerous preparatory tasks prior to the evaluation,

influencing both the way the evaluation will be conducted and its results. Such tasks therefore have to be considered in pilot studies.

This article aims to share and synthesize the discussions, work, and recommendations put forward by experts during this workshop. Practically, the aim is to clarify what kind of pilot studies should be used in PHIR and for which objectives, through an account of the different questions raised and the consensual answers the experts provided. It is not intended to be a systematic synthesis of science or to bring new data, but to be a milestone for a common basis for discussion between researchers from different disciplines and fields.

## Methods

The workshop was prepared by LT and FA (preparation of a program and compilation of a bibliographic file). It included formal presentations of participants and discussions moderated by LT. At the end of the workshop, an oral synthesis was carried out by a rapporteur (JP) to validate by expert consensus the key points of the discussion and the recommendations. All the discussions and the validated synthesis were recorded. The recording has been fully transcribed. A first draft of the paper was prepared by JC and FA from this material, then corrected and validated by LT, then by all of the coauthors (who all participated in the debates).

## Discussion and key arguments

### What is a pilot and/or feasibility study?

There are many different definitions of pilot studies. These definitions have in common the concept of doing something on a smaller scale with the intent of gathering information to do something on a larger scale. However, a small-sized study is not necessarily in itself a pilot study. There are issues regarding language and terminology. Synonymous terms include feasibility study, vanguard study, pre-study, and first phase. Multiple terms are used without clear distinctions, e.g. pilot work, pilot study, feasibility study, pilot trial [8].

The terms pilot and feasibility study have often been used interchangeably [7]. There is no consensus on their definitions. Some people consider the two to be the same; others think they are different [9]. Some have used an overarching definition of feasibility studies, with pilot studies being a subset: "a feasibility study asks whether something can be done, should we proceed with it, and if so, how. A pilot study asks the same questions but also has a specific design feature: in a pilot study a future study, or part of a future study, is conducted on a smaller scale" [10].

Moreover, in current practice, there are questions about how to describe 'preparatory trials', i.e., the work done before starting a study. In some circumstances
these may be covered by the general term feasibility
assessment/trial. On the other hand, a phase that is in a
study assessing uncertain aspects in order to refine the
design, but is part of the same study, would be better
described as an ‘internal pilot’ [11].

Pilot trials differ from princeps trials in their aims and objectives, focusing on assessing feasibility rather than effectiveness or efficacy. They are usually designed to support the development of a future definitive trial and generally provide a good way to assess the feasibility of the main trial. However, they are not usually well-described, the feasibility objectives are usually unclear, and the rationale for trialing feasibility is often missing. The analytical strategies on how to analyze feasibility outcomes lack clarity and feasibility criteria are often missing [12].

Many pilot studies are not published or even reported. This is sometimes because inappropriate theory produces negative results (pilot trials are not designed to produce statistical significance). They are sometimes not considered important enough to be published in the same way as the main trial. However, it has been suggested that they should be scrutinized in the same way as full trials and that they should require registration. Information should be publicly available—for those who participated and also to ensure that any mistakes are not repeated. We are aware that there are steps in place to improve this, as well as requirements imposed by funders to publish protocols and registration requirements prior to release of research funds.

It was noted that progress is being made to improve
standardization and reporting of pilots. A CONSORT ex-
tension specifically for pilot trials is now available [12].
The development of this extension was motivated by the
increasing number of studies described as feasibility or
pilot studies and by research identifying weaknesses in
their reporting and conduct. There is also a journal specif-
ically for publishing pilot and feasibility studies (https://
pilotfeasibilitystudies.biomedcentral.com/).

Are the terms and methods for pilot studies in
biomedical models also applicable to PHIR?

The types of models and approaches in the biomedical
field may not be suitable for other types of interventions.
It is useful at this point to try to see where terminologies
might be appropriate and where they might not. The
choice of these kinds of approach in PHIR requires an
open mind and a more flexible outlook.

In particular, in PHIR complex interventions need to
be addressed. Thus, before testing the evaluation, as in a
traditional pilot or feasibility study, many tasks may be
necessary to inform the intervention and test different
aspects of its design and implementation. In other

words, while in clinical research the pilot study mainly
concerns the preparation of the trial, in PHIR a pilot
study focuses on the preparation of both the interven-
tion and the trial/evaluation.

Which pilot study for field-designed programs?

We must distinguish programs designed and imple-
mented by researchers from those built by the actors
and decision-makers. In this latter case, researchers do
not intervene but observe (i.e., natural experimental
designs). The consensus is that a pilot study is generally
needed for complex intervention evaluation for de novo
research-designed programs. However, the place of a
pilot study in real-life programs is less clear. In these
cases, the concept of pilot study as defined in clinical
research does not make any sense and the goal is more
to ensure the evaluability of an ongoing program than
the feasibility of research.

The concept of evaluability assessment developed by
evaluators over the past 30 years can provide useful
information on the role of pilot studies in these kinds of
PHIR. Indeed, in 1980, Rutman proposed evaluability
assessment [13], which can now be considered to essen-
tially cover ideas addressing program relevance and
questions such as the relevancy of the program accord-
ing to targets, the existing theory of change, the respon-
sivity of the implementation to local conditions, the
nature of activities, the influence of the context on the
results, etc. A report on publications of evaluability
assessment commissioned by the Department for Inter-
national Development in the UK [14] concluded that an
evaluability assessment addresses what can be learnt
from studying a program, in principle and in practice. Is
the program based on known and tested theory? In prac-
tice, is it worth it and can it be done? In practice, evalua-
tors have to describe the objectives, logic, and activities
of the program with the aim of investigating its credibil-
ity, feasibility, sustainability, and acceptability.

Moreover, contrary to research-designed programs,
field-designed programs have a genealogy: a previous
version of this program was implemented before. In real
life, the genealogy of programs may be traced through
their previous iterations.

Therefore, this evaluability improves the understanding
of how the intervention is set in context and refines the
expected outcomes to be assessed in the efficacy study. It
could constitute a subject for a pilot study in order to im-
prove the program and prepare the efficacy study.

Which pilot study for research-based interventions?

In research-based intervention, there are many reasons
to conduct pilot work in PHIR, including both evalu-
ation and intervention aspects.
these may be covered by the general term feasibility
assessment/trial. On the other hand, a phase that is in a
study assessing uncertain aspects in order to refine the
design, but is part of the same study, would be better
described as an ‘internal pilot’ [11].

Pilot trials differ from princeps trials in their aims and objectives, focusing on assessing feasibility rather than effectiveness or efficacy. They are usually designed to support the development of a future definitive trial and generally provide a good way to assess the feasibility of the main trial. However, they are not usually well-described, the feasibility objectives are usually unclear, and the rationale for trialing feasibility is often missing. The analytical strategies on how to analyze feasibility outcomes lack clarity and feasibility criteria are often missing [12].

Many pilot studies are not published or even reported. This is sometimes because inappropriate theory produces negative results (pilot trials are not designed to produce statistical significance). They are sometimes not considered important enough to be published in the same way as the main trial. However, it has been suggested that they should be scrutinized in the same way as full trials and that they should require registration. Information should be publicly available—for those who participated and also to ensure that any mistakes are not repeated. We are aware that there are steps in place to improve this, as well as requirements imposed by funders to publish protocols and registration requirements prior to release of research funds.

It was noted that progress is being made to improve
standardization and reporting of pilots. A CONSORT ex-
tension specifically for pilot trials is now available [12].
The development of this extension was motivated by the
increasing number of studies described as feasibility or
pilot studies and by research identifying weaknesses in
their reporting and conduct. There is also a journal specif-
ically for publishing pilot and feasibility studies (https://
pilotfeasibilitystudies.biomedcentral.com/).

Are the terms and methods for pilot studies in
biomedical models also applicable to PHIR?

The types of models and approaches in the biomedical
field may not be suitable for other types of interventions.
It is useful at this point to try to see where terminologies
might be appropriate and where they might not. The
choice of these kinds of approach in PHIR requires an
open mind and a more flexible outlook.

In particular, in PHIR complex interventions need to
be addressed. Thus, before testing the evaluation, as in a
traditional pilot or feasibility study, many tasks may be
necessary to inform the intervention and test different
aspects of its design and implementation. In other

words, while in clinical research the pilot study mainly
concerns the preparation of the trial, in PHIR a pilot
study focuses on the preparation of both the interven-
tion and the trial/evaluation.

Which pilot study for field-designed programs?

We must distinguish programs designed and imple-
mented by researchers from those built by the actors
and decision-makers. In this latter case, researchers do
not intervene but observe (i.e., natural experimental
designs). The consensus is that a pilot study is generally
needed for complex intervention evaluation for de novo
research-designed programs. However, the place of a
pilot study in real-life programs is less clear. In these
cases, the concept of pilot study as defined in clinical
research does not make any sense and the goal is more
to ensure the evaluability of an ongoing program than
the feasibility of research.

The concept of evaluability assessment developed by
evaluators over the past 30 years can provide useful
information on the role of pilot studies in these kinds of
PHIR. Indeed, in 1980, Rutman proposed evaluability
assessment [13], which can now be considered to essen-
tially cover ideas addressing program relevance and
questions such as the relevancy of the program accord-
ing to targets, the existing theory of change, the respon-
sivity of the implementation to local conditions, the
nature of activities, the influence of the context on the
results, etc. A report on publications of evaluability
assessment commissioned by the Department for Inter-
national Development in the UK [14] concluded that an
evaluability assessment addresses what can be learnt
from studying a program, in principle and in practice. Is
the program based on known and tested theory? In prac-
tice, is it worth it and can it be done? In practice, evalua-
tors have to describe the objectives, logic, and activities
of the program with the aim of investigating its credibil-
ity, feasibility, sustainability, and acceptability.

Moreover, contrary to research-designed programs,
field-designed programs have a genealogy: a previous
version of this program was implemented before. In real
life, the genealogy of programs may be traced through
their previous iterations.

Therefore, this evaluability improves the understanding
of how the intervention is set in context and refines the
expected outcomes to be assessed in the efficacy study. It
could constitute a subject for a pilot study in order to im-
prove the program and prepare the efficacy study.

Which pilot study for research-based interventions?

In research-based intervention, there are many reasons
to conduct pilot work in PHIR, including both evalu-
ation and intervention aspects.
Similar to clinical research, a pilot study could assess the feasibility of procedures and methods, recruiting for a main trial and retaining subjects in a trial, collecting complete data, etc. Pilot work may also be used to gather preliminary feedback on an intervention and what might be refined for the main trial going forward, and also to measure adherence to protocol.

However, in complex interventions other key objec-
tives could be considered.

Chen [15] recommends addressing the “viability valid-
ity” of an intervention as the first step in an evaluation
process. He has defined this as “the extent to which an
evaluation provides evidence that an intervention is suc-
cessful in the real world”. This refers to the practical,
affordable, suitable, evaluable, and helpful aspects of an
intervention in the real world. So, it actually refers to
the way an intervention fits with its context. In the
viability study, the aim is to answer these questions:
“Can it recruit and/or retain ordinary clients? Can it be
adequately implemented by ordinary implementers? Is it
suitable for ordinary implementing organizations to co-
ordinate intervention-related activities? Is it affordable,
is it evaluable and does it enable ordinary clients and
other stakeholders to view and experience how well it
solves the problem?” [15]. This viability study could take
place within pilot studies to explore and understand
how the intervention fits with the context. According to
Chen, without a viable evaluation, we risk addressing the
efficacy of an ‘off ground’ intervention that cannot be
transferred to another context or scaled up. Overall, he
has proposed to use an integrative validity approach, by
firstly proceeding to a viability study, secondly carrying
out an effectiveness study, taking into account the real
conditions of implementation, and lastly (and if neces-
sary) performing an efficacy study or directly proceeding
to dissemination.

Moreover, like an evaluability assessment, a pilot study
could also concern the reasons for the intervention in
order to help to define the expected results. It could con-
tribute to informing the design of the best intervention to
fit the context, according to the best evidence on what
works and how, along with contextual parameters, as rec-
ommended in theory-based intervention [15, 16]. Indeed,
according to the intervention-based paradigm, all inter-
ventions are based on implicit or explicit theories. The
principle of theory-based intervention is to make explicit
what underlies the intervention: which activities, and
under which conditions, produce which mechanisms of
change, in turn producing the expected results. Making
this underlying theory explicit is necessary to ensure that
the intervention delivered is relevant to the production of
the expected results. There are different methods to pro-
duce a theory-based intervention, but all of them cross
scientific evidence, theoretical works, and stakeholders’

expertise in a participative and multidisciplinary way. This
work could be an integral part of a pilot study.

Hence, in the framework of research-based interven-
tion, a pilot study could assess the feasibility of the trial,
but also contribute to the design of the theory-based
intervention and assess its conditions of viability in the
context. Knowledge would be provided on what will be
evaluated (through a comprehensive explanation of what
the intervention attempts to change by a theory-based
approach), how the intervention has to be conducted in
order to fit with the context and to be transferable (with
the viability study), and how to conduct the effectiveness
study (with the classic pilot study as in clinical research).
A pilot study pursuing all of these objectives (feasibility,
expected outcomes, investigation, and contextual fac-
tors) could enable practitioners to hypothesize what may
be potentially effective in the context and under which
conditions. Once this has been established, it is relevant
to test the ‘real’ efficacy and effectiveness in the specific
real world in which the intervention is to be imple-
mented. This could be likened to phases I/II in drug
development in the clinical field, whereby an interven-
tion is tailored/adapted and refined to optimize its suc-
cess in the main trial.

To conclude the discussion, from an intervention
perspective in public health programs, pilots could be
used in many ways: to test the intervention's activity
or strategy, to identify its most important compo-
nents, to test ways of operating/administrating an
intervention, or to explore how intervention protocols
are followed by practitioners and clients. Pilot studies
might be also the preferred option when they can
focus on an in-depth evaluation of a specific aspect of
an innovation. Moreover, pilot studies could provide a
practical option if the funding for a randomized con-
trolled trial is not available. Indeed, the proposal of a
pilot may be helpful in negotiations with funding
agencies and may enable a more rapid start, which
could be followed by a quick transition to the full
phase. Another advantage is that generally a full PHIR
can last years. However, policymakers generally want
answers quickly. A well-conducted pilot study could
provide a quick answer that is useful for decision-
making. This answer may lead, for example, to the de-
cision of whether or not to implement the interven-
tion, and how to implement it. Full research can then
be done concurrently with the deployment of the
intervention. Thus, in PHIR, a pilot study does not
delay the decision but may advance it. Finally, in pilot
studies in biomedical research, there are criteria to in-
dicate whether to proceed with the full trial. The guid-
ing principles for establishing the criteria to decide
when to progress to the main study in PHIR are yet to
be defined.
Similar to clinical research, a pilot study could assess the feasibility of procedures and methods, recruiting for a main trial and retaining subjects in a trial, collecting complete data, etc. Pilot work may also be used to gather preliminary feedback on an intervention and what might be refined for the main trial going forward, and also to measure adherence to protocol.

However, in complex interventions other key objectives could be considered.

Chen [15] recommends addressing the “viability validity” of an intervention as the first step in an evaluation process. He has defined this as “the extent to which an evaluation provides evidence that an intervention is successful in the real world”. This refers to the practical, affordable, suitable, evaluable, and helpful aspects of an intervention in the real world. So, it actually refers to the way an intervention fits with its context. In the viability study, the aim is to answer these questions: “Can it recruit and/or retain ordinary clients? Can it be adequately implemented by ordinary implementers? Is it suitable for ordinary implementing organizations to coordinate intervention-related activities? Is it affordable, is it evaluable and does it enable ordinary clients and other stakeholders to view and experience how well it solves the problem?” [15]. This viability study could take place within pilot studies to explore and understand how the intervention fits with the context. According to Chen, without a viable evaluation, we risk addressing the efficacy of an ‘off ground’ intervention that cannot be transferred to another context or scaled up. Overall, he has proposed to use an integrative validity approach, by firstly proceeding to a viability study, secondly carrying out an effectiveness study, taking into account the real conditions of implementation, and lastly (and if necessary) performing an efficacy study or directly proceeding to dissemination.

Moreover, like an evaluability assessment, a pilot study could also concern the reasons for the intervention in order to help to define the expected results. It could contribute to informing the design of the best intervention to fit the context, according to the best evidence on what works and how, along with contextual parameters, as recommended in theory-based intervention [15, 16]. Indeed, according to the intervention-based paradigm, all interventions are based on implicit or explicit theories. The principle of theory-based intervention is to make explicit what underlies the intervention: which activities, and under which conditions, produce which mechanisms of change, in turn producing the expected results. Making this underlying theory explicit is necessary to ensure that the intervention delivered is relevant to the production of the expected results. There are different methods to produce a theory-based intervention, but all of them cross scientific evidence, theoretical works, and stakeholders’

expertise in a participative and multidisciplinary way. This work could be an integral part of a pilot study.

Hence, in the framework of research-based intervention, a pilot study could assess the feasibility of the trial, but also contribute to the design of the theory-based intervention and assess its conditions of viability in the context. Knowledge would be provided on what will be evaluated (through a comprehensive explanation of what the intervention attempts to change by a theory-based approach), how the intervention has to be conducted in order to fit with the context and to be transferable (with the viability study), and how to conduct the effectiveness study (with the classic pilot study as in clinical research). A pilot study pursuing all of these objectives (feasibility, expected outcomes, investigation, and contextual factors) could enable practitioners to hypothesize what may be potentially effective in the context and under which conditions. Once this has been established, it is relevant to test the ‘real’ efficacy and effectiveness in the specific real world in which the intervention is to be implemented. This could be likened to phases I/II in drug development in the clinical field, whereby an intervention is tailored/adapted and refined to optimize its success in the main trial.

To conclude the discussion, from an intervention perspective in public health programs, pilots could be used in many ways: to test the intervention’s activity or strategy, to identify its most important components, to test ways of operating/administrating an intervention, or to explore how intervention protocols are followed by practitioners and clients. Pilot studies might be also the preferred option when they can focus on an in-depth evaluation of a specific aspect of an innovation. Moreover, pilot studies could provide a practical option if the funding for a randomized controlled trial is not available. Indeed, the proposal of a pilot may be helpful in negotiations with funding agencies and may enable a more rapid start, which could be followed by a quick transition to the full phase. Another advantage is that generally a full PHIR can last years. However, policymakers generally want answers quickly. A well-conducted pilot study could provide a quick answer that is useful for decision-making. This answer may lead, for example, to the decision of whether or not to implement the intervention, and how to implement it. Full research can then be done concurrently with the deployment of the intervention. Thus, in PHIR, a pilot study does not delay the decision but may advance it. Finally, in pilot studies in biomedical research, there are criteria to indicate whether to proceed with the full trial. The guiding principles for establishing the criteria to decide when to progress to the main study in PHIR are yet to be defined.
## Position statements, further research directions, and recommendations

Considering these discussions, the experts at the international workshop organized by ACRISP defined five recommendations for pilot studies in PHIR.

### Recommendation 1: Clarifying the terms for PHIR

In practice, different researchers appear to have preferences for different terms. It is therefore important that the terminology is consistent and understandable by all. It is not always possible or even desirable to have a strict definition of words. However, terminology should at least be explicitly stated at some stage, especially in a multidisciplinary field such as PHIR.

This clarification is a sine qua non condition to encourage, advocate, and train in the use of pilot studies in this field, more relevant in PHIR than in others because of the complexity of interventions.

### Recommendation 2: Standardizing reporting and publication

The need for high standards in conduct and reporting is just as important for pilot work as for the main studies.

This would improve quality and consistency, promote transparency, and facilitate understanding. The availability of a larger number of high-quality examples would enable further improvements for the conduct of such studies and provide valuable results for other workers in this field.

The Consolidated Standards of Reporting Trials (CONSORT) statement is a guideline designed to improve the transparency and quality of the reporting of randomized controlled trials (RCTs) [17]. Adherence to CONSORT recommendations for RCTs is already recommended by many journals in their instructions to authors. Weaknesses in the reporting and conduct of pilot trials have been identified [7, 10]. The CONSORT guideline has therefore recently been extended to provide recommendations and a checklist for randomized pilot and feasibility trials conducted before the main RCTs [12]. Although the extension covers different terminologies (pilot, feasibility, trial, study), it does not apply to internal pilot studies that are built into the design of a main trial, or to non-randomized pilot and feasibility studies. However, much of what is presented could be adapted to apply to these types of pilot or feasibility studies or similar types of trial.

Qualitative research is often used alongside other methods to assess feasibility and there are variations in how this is used and how it is reported. Some advice and references to other guidelines for qualitative studies are provided [12].

Adherence to guidelines would be helpful for other researchers designing studies in the future and would

facilitate the evaluation of manuscripts by peer reviewers and editors.

The growing number of studies described as feasibility or pilot studies is reflected by the recent introduction of new journals specifically covering this important area of work.

### Recommendation 3: Clearly defining the different objectives of the pilot study in reports and publications in PHIR

There are many types of pilot studies with many different objectives.

The objectives and research questions should drive the methods used and they need to be clearly stated for the purpose of evaluation of the intervention and for process evaluation. It is also a sine qua non condition to encourage, advocate, and train in the use of pilot studies in PHIR.

### Recommendation 4: Encouraging complete pilot studies for de novo research-designed complex interventions

As detailed, pilot studies could prepare for both the evaluation and the intervention.

In addition to testing the feasibility of the trial, the pilot study can be used to test the intervention's activity or strategy, to validate components and intervention theory, to test a way of operating/administrating an intervention, to explore how intervention protocols are followed by practitioners and clients, and to assess viability.

They serve to improve the intervention which will be delivered and hence improve the results expected by exploring how the deliverable works.

But this approach has to be structured and supported. It could be interesting to design and promote a guideline explaining the different aspects to be studied, how and in which stages of the intervention and evaluation development, and encouraging funders to support pilot studies.

### Recommendation 5: Integrating evaluability assessments as pilot studies for field-designed programs in PHIR

As a field-designed program has not initially been designed for the purpose of knowledge generation, it is important to assess to what extent suitable and reliable data can be extracted when the program is run. This is the purpose of evaluability assessment.

The evaluability assessment goes further than merely providing information on whether a program can be evaluated or not. It is used to describe the objectives, logic, and activities of the program with the aim of investigating its credibility, feasibility, sustainability, and acceptability. It ensures that there is an interest in evaluating this program.
**Position statements, further research directions, and recommendations**

Considering these discussions, the experts at the international workshop organized by ACRISP defined five recommendations for pilot studies in PHIR.

**Recommendation 1: Clarifying the terms for PHIR**

In practice, different researchers appear to have preferences for different terms. It is therefore important that the terminology is consistent and understandable by all. It is not always possible or even desirable to have a strict definition of words. However, terminology should at least be explicitly stated at some stage, especially in a multidisciplinary field such as PHIR.

This clarification is a sine qua non condition to encourage, advocate, and train in the use of pilot studies in this field, more relevant in PHIR than in others because of the complexity of interventions.

**Recommendation 2: Standardizing reporting and publication**

The need for high standards in conduct and reporting is just as important for pilot work as for the main studies.

This would improve quality and consistency, promote transparency, and facilitate understanding. The availability of a larger number of high-quality examples would enable further improvements for the conduct of such studies and provide valuable results for other workers in this field.

The Consolidated Standards of Reporting Trials (CONSORT) statement is a guideline designed to improve the transparency and quality of the reporting of randomized controlled trials (RCTs) [17]. Adherence to CONSORT recommendations for RCTs is already recommended by many journals in their instructions to authors. Weaknesses in the reporting and conduct of pilot trials have been identified [7, 10]. The CONSORT guideline has therefore recently been extended to provide recommendations and a checklist for randomized pilot and feasibility trials conducted before the main RCTs [12]. Although the extension covers different terminologies (pilot, feasibility, trial, study), it does not apply to internal pilot studies that are built into the design of a main trial, or to non-randomized pilot and feasibility studies. However, much of what is presented could be adapted to apply to these types of pilot or feasibility studies or similar types of trial.

Qualitative research is often used alongside other methods to assess feasibility and there are variations in how this is used and how it is reported. Some advice and references to other guidelines for qualitative studies are provided [12].

Adherence to guidelines would be helpful for other researchers designing studies in the future and would

facilitate the evaluation of manuscripts by peer reviewers and editors.

The growing number of studies described as feasibility or pilot studies is reflected by the recent introduction of new journals specifically covering this important area of work.

**Recommendation 3: Clearly defining the different objectives of the pilot study in reports and publications in PHIR**

There are many types of pilot studies with many different objectives.

The objectives and research questions should drive the methods used and they need to be clearly stated for the purpose of evaluation of the intervention and for process evaluation. It is also a sine qua non condition to encourage, advocate, and train in the use of pilot studies in PHIR.

**Recommendation 4: Encouraging complete pilot studies for de novo research-designed complex interventions**

As detailed, pilot studies could prepare for both the evaluation and the intervention.

In addition to testing the feasibility of the trial, the pilot study can be used to test the intervention's activity or strategy, to validate components and intervention theory, to test a way of operating/administrating an intervention, to explore how intervention protocols are followed by practitioners and clients, and to assess viability.

They serve to improve the intervention which will be delivered and hence improve the results expected by exploring how the deliverable works.

But this approach has to be structured and supported. It could be interesting to design and promote a guideline explaining the different aspects to be studied, how and in which stages of the intervention and evaluation development, and encouraging funders to support pilot studies.

**Recommendation 5: Integrating evaluability assessments as pilot studies for field-designed programs in PHIR**

As a field-designed program has not initially been designed for the purpose of knowledge generation, it is important to assess to what extent suitable and reliable data can be extracted when the program is run. This is the purpose of evaluability assessment.

The evaluability assessment goes further than merely providing information on whether a program can be evaluated or not. It is used to describe the objectives, logic, and activities of the program with the aim of investigating its credibility, feasibility, sustainability, and acceptability. It ensures that there is an interest in evaluating this program.
## Conclusions

Pilot studies play an important role in intervention and PHIR. There are many types of pilot studies with many different objectives. They are of value in understanding the feasibility of the intervention and optimizing its design and evaluation. The questions to be addressed should drive the approach and methods. When considering the place of pilot studies in process evaluation for PHIR there are many challenges, including terminology. Although pilot studies fit well with the biomedical research model, for PHIR this model and approach are not always appropriate. The consensus was that a pilot study is generally appropriate for a de-novo research-designed program whereas evaluability assessment may be more appropriate for real-life programs.

The increasing use of pilot studies in complex interventions and the sharing of information and experience across many different disciplines, through better reporting, should provide a better understanding of the place and value of pilot studies in PHIR. On the one hand, pilot studies increase the appropriateness of the investigation methods (i.e., quality of research methods). On the other hand, they allow intervention mapping (feasibility, components, contextual factors, fit for real conditions), which is needed for the knowledge transfer process. These two conditions for evidence-based decision-making and pilot studies could thus facilitate evidence-based decisions in public health policy.

## Acknowledgements

Discussion panel: François Alla, Pierre Arwidson, Pierre Blaise, Christopher Bonell, Isabelle Boutron, Linda Cambon, Rona Campbell, Patrizia Carrieri, Franck Chauvin, François Dabis, Nancy Edwards, Christine Ferron, Marie-Renée Guevel, Nadir Kellou, Joëlle Kivits, Antony Lacouture, Thierry Lang, Susan Michie, Laëtitia Minary, Graham Moore, Grégory Ninot, Kareen Nour, Jeanine Pommier, Louise Potvin, Lehana Thabane.

## Authors' contributions

All authors contributed to the workshop and provided intellectual content. FA and JC drafted the manuscript; the other authors provided critical feedback. All authors approved the final version.

## Funding

The workshop was supported by partners of the National Coordinated Action for Intervention Research: AVIESAN, INCA, ANRS, IReSP.

## Availability of data and materials

Not applicable.

## Ethics approval and consent to participate

Not applicable.

## Consent for publication

Not applicable.

## Competing interests

The authors declare that they have no competing interests.

## Author details

Received: 17 August 2018 Accepted: 10 May 2019

Published online: 30 May 2019

## References

1. Hawe P, Potvin L. What is population health intervention research? Can J Public Health. 2009;100(1):Suppl I8-14.

2. Alla F, Cambon L. Transformation of health systems: contribution of population health intervention research. Lancet Public Health. 2017;2(12):e539.

3. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321(7262):694-6.

4. Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol. 2009;43(3-4):267-76.

5. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O'Cathain A, Tinati T, Wight D, Baird J. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.

6. Cambon L, Minary L, Ridde V, Alla F. Transferability of interventions in health education: a review. BMC Public Health. 2012;12:497.

7. Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, Robson R, Thabane M, Giangregorio L, Goldsmith CH. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10:1.

8. Arnold DM, Burns KE, Adhikari NK, Kho ME, Meade MO, Cook DJ, et al. The design and interpretation of pilot trials in clinical research in critical care. Crit Care Med. 2009;37(1 suppl):S69-74.

9. Whitehead AL, Sully GO, Campbell MJ. Pilot and feasibility studies: is there a difference from each other and from a randomised controlled trial? Contemp Clin Trials. 2014;38:130-3.

10. Eldridge SM, Lancaster GA, Campbell MJ, Thabane L, Hopewell S, Coleman CL, et al. Defining feasibility and pilot studies in preparation for randomised controlled trials: development of a conceptual framework. PLoS One. 2016; 11(3):e0150205.

11. Wittes J, Brittain E. The role of internal pilot studies in increasing the efficiency of clinical trials. Stat Med. 1990;9:65-71.

12. Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, et al. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. BMJ. 2016;355:i5239.

13. Rutman L. Planning useful evaluations: evaluability assessment. Beverly Hills: Sage Publications; 1980.

14. Davies C. Planning evaluability assessments: a synthesis of the literature with recommendations. In: Report of a study commissioned by the Department for International Development; 2013. Available at: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/248656/wp40-planning-eval-assessments.pdf. Accessed 1 June 2018.

15. Chen HT. The bottom-up approach to integrative validity: a new perspective for program evaluation. Eval Program Plann. 2010;33:205-14.

16. Weiss CH. How can theory-based evaluation make greater headway? Eval Rev. 1997;21(4):501-24.

17. Schulz KF, Altman DG, Moher D, for the CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332.

## Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Ready to submit your research? Choose BMC and benefit from:

* fast, convenient online submission

* thorough peer review by experienced researchers in your field

* rapid publication on acceptance

* support for research data, including large and complex data types

* gold Open Access which fosters wider collaboration and increased citations

* maximum visibility for your research: over 100M website views per year

At BMC, research is always in progress.

Learn more biomedcentral.com/submissions

<figure><img src="image_2.png" /></figure>

<footnote n="1">¹McMaster University, Hamilton, Canada. ²Université de Bordeaux, Bordeaux, France. ³Université de Montréal, Montreal, Canada. ⁴European Center on Disease Prevention and Control, Stockholm, Sweden. ⁵Université de Lorraine, Nancy, France. ⁶ARS Pays de Loire, Nantes, France. ⁷A Tree of Life Sciences Ltd, Manchester, UK.</footnote>
## Conclusions

Pilot studies play an important role in intervention and PHIR. There are many types of pilot studies with many different objectives. They are of value in understanding the feasibility of the intervention and optimizing its design and evaluation. The questions to be addressed should drive the approach and methods. When considering the place of pilot studies in process evaluation for PHIR there are many challenges, including terminology. Although pilot studies fit well with the biomedical research model, for PHIR this model and approach are not always appropriate. The consensus was that a pilot study is generally appropriate for a de-novo research-designed program whereas evaluability assessment may be more appropriate for real-life programs.

The increasing use of pilot studies in complex interventions and the sharing of information and experience across many different disciplines, through better reporting, should provide a better understanding of the place and value of pilot studies in PHIR. On the one hand, pilot studies increase the appropriateness of the investigation methods (i.e., quality of research methods). On the other hand, they allow intervention mapping (feasibility, components, contextual factors, fit for real conditions), which is needed for the knowledge transfer process. These two conditions for evidence-based decision-making and pilot studies could thus facilitate evidence-based decisions in public health policy.

## Acknowledgements

Discussion panel: François Alla, Pierre Arwisdson, Pierre Blaise, Christopher Bonell, Isabelle Boutron, Linda Cambon, Rona Campbell, Patrizia Carrieri, Franck Chauvin, François Dabis, Nancy Edwards, Christine Ferron, Marie-Renée Guevel, Nadir Kellou, Joëlle Kivits, Antony Lacouture, Thierry Lang, Susan Michie, Laëtitia Minary, Graham Moore, Grégory Ninot, Kareen Nour, Jeanine Pommier, Louise Potvin, Lehana Thabane.

## Authors' contributions

All authors contributed to the workshop and provided intellectual content. FA and JC drafted the manuscript; the other authors provided critical feedback. All authors approved the final version.

## Funding

The workshop was supported by partners of the National Coordinated Action for Intervention Research: AVIESAN, INCA, ANRS, IReSP.

## Availability of data and materials

Not applicable.

## Ethics approval and consent to participate

Not applicable.

## Consent for publication

Not applicable.

## Competing interests

The authors declare that they have no competing interests.

## Author details

Received: 17 August 2018 Accepted: 10 May 2019

Published online: 30 May 2019

## References

1. Hawe P, Potvin L. What is population health intervention research? Can J Public Health. 2009;100(1):Suppl I8-14.

2. Alla F, Cambon L. Transformation of health systems: contribution of population health intervention research. Lancet Public Health. 2017;2(12):e539.

3. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321(7262):694-6.

4. Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol. 2009;43(3-4):267-76.

5. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O'Cathain A, Tinati T, Wight D, Baird J. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.

6. Cambon L, Minary L, Ridde V, Alla F. Transferability of interventions in health education: a review. BMC Public Health. 2012;12:497.

7. Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, Robson R, Thabane M, Giangregorio L, Goldsmith CH. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10:1.

8. Arnold DM, Burns KE, Adhikari NK, Kho ME, Meade MO, Cook DJ, et al. The design and interpretation of pilot trials in clinical research in critical care. Crit Care Med. 2009;37(1 suppl):S69-74.

9. Whitehead AL, Sully GO, Campbell MJ. Pilot and feasibility studies: is there a difference from each other and from a randomised controlled trial? Contemp Clin Trials. 2014;38:130-3.

10. Eldridge SM, Lancaster GA, Campbell MJ, Thabane L, Hopewell S, Coleman CL, et al. Defining feasibility and pilot studies in preparation for randomised controlled trials: development of a conceptual framework. PLoS One. 2016; 11(3):e0150205.

11. Wittes J, Brittain E. The role of internal pilot studies in increasing the efficiency of clinical trials. Stat Med. 1990;9:65-71.

12. Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, et al. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. BMJ. 2016;355:i5239.

13. Rutman L. Planning useful evaluations: evaluability assessment. Beverly Hills: Sage Publications; 1980.

14. Davies R. Planning evaluability assessments: a synthesis of the literature with recommendations. In: Report of a study commissioned by the Department for International Development; 2013. Available at: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/248656/wp40-planning-eval-assessments.pdf. Accessed 1 June 2018.

15. Chen HT. The bottom-up approach to integrative validity: a new perspective for program evaluation. Eval Program Plann. 2010;33:205-14.

16. Weiss CH. How can theory-based evaluation make greater headway? Eval Rev. 1997;21(4):501-24.

17. Schulz KF, Altman DG, Moher D, for the CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332.

## Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

### Ready to submit your research? Choose BMC and benefit from:

* fast, convenient online submission

* thorough peer review by experienced researchers in your field

* rapid publication on acceptance

* support for research data, including large and complex data types

* gold Open Access which fosters wider collaboration and increased citations

* maximum visibility for your research: over 100M website views per year

At BMC, research is always in progress.

Learn more biomedcentral.com/submissions

<figure><img src="image_3.png" /></figure>

<footnote n="1">¹McMaster University, Hamilton, Canada. ²Université de Bordeaux, Bordeaux, France. ³Université de Montréal, Montreal, Canada. ⁴European Center on Disease Prevention and Control, Stockholm, Sweden. ⁵Université de Lorraine, Nancy, France. ⁶ARS Pays de Loire, Nantes, France. ⁷A Tree of Life Sciences Ltd, Manchester, UK.</footnote>