QUALITATIVE ME THODS IN
IMPLEMENTATION SCIENCEOFI OHEALTHQ UA L I TAT I V E M E T H O DU.S.
S I NDEPARTMENTI M PL EM EN TATN S C I ENANDC E HUMAN SERVICES | NATIONAL INSTITUTES OF HEALTH

1

PREFACE
In recent years, the field of implementation science has significantly grown, both in the theories and frameworks
used to explain it and in the available findings used to understand and influence the adoption, implementationand sustainment of evidence-based practices (and the de-implementation of ineffective or harmful practices).
Capacity has greatly increased for our investigators to conduct studies that reflect the complexity and dynamism of
improving care within clinical, community and public health settings. Improvements have accrued in the range of
research designs used, the methods to collect data, and analytical strategies used to make sense of implementation
processes and outcomes. From its beginnings, implementation science has recognized the importance of both
qualitative and quantitative data and throughout its development, leading scholars have called for mixed methods
approaches to ensure maximal return from each investigation.
Despite clear recognition of the value of qualitative research methods within implementation studies, there remains
confusion as to which methods to use, how to best use them, and what guidance investigators should follow intheir application. The Qualitative Research in Implementation Science (QualRIS) group was formed to reduce thisconfusion. Our investigators have moved from viewing qualitative research methods as an add-on to the major
components of a study (i.e. addition of focus groups to address exploratory work on provider factors affecting
implementation outcomes) to a necessary but often underspecified core of a study. In this paper, the QualRIS
group explains multiple ways in which qualitative methods can be effectively used to answer a range of high priority
implementation science questions, and describes resources that are available to support the community. While
the field will continue to progress in its methodologies, this report offers a snapshot of how qualitative methods in
implementation science have advanced and where they can go from here. We hope this serves as a helpful resource
to investigators and that the report’s recommendations are taken on so that the next generation of implementation
science studies include the best qualitative research approaches to fit each question.
We are grateful to the QualRIS team for their substantial effort to review progress and offer guidance to the field,
and to all those who seek to integrate qualitative research methods to strengthen their implementation science
investigations.

Sincerely,David A Chambers
Deputy Director for Implementation Science
Division of Cancer Control and Population Sciences
National Cancer Institute

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

TABLE OF CONTENT S
I.

Introduction

01

II.

Approach02

III.

Traditional Qualitative Research Methods: An Overview

03

IV.

Qualitative Research Methods in Implementation Sciencein Health and Health Care: An Overview

04

Value of Qualitative Methods in Implementation Science

05

V.

A. Elicit Stakeholder-Centered Perspectives
B. Inform Design and Implementation
C. Understand Contexts Across Diverse Settings
D. Provide Documentation and Encourage Reflection
on Implementation Processes
E. Gain Insight Into Implementation EffectivenessF. Understand Mechanisms of Change
G. Contribute to Theoretical Development

VI.

Methodological Considerations for the Use of
Qualitative Methods in Implementation Science

09

A. Sampling
B. Timing of Data Collection
C. Roles of Qualitative Research Teams in Implementation Science
D. Data Collection
E. Data Analysis
F. Presentation of Qualitative Findings

VII.

Collecting and Analyzing Qualitative Data with Rigor

14

VIII.

Need and Opportunities for Innovation in
Use of Qualitative Methods in Implementation Science

16

IX.

Conclusion

18

X.

Acknowledgements

19

References

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

20I . INTRODUC TION
Implementation science is a rapidly developing field dedicated to putting evidence-based practices to use inreal-world settings (Brownson et al., 2018; Glasgow et al., 2003; Kessler et al., 2012). Implementation science has
been defined as, the scientific study of methods to promote the integration of research findings and evidence-based
interventions into healthcare practice and policy. (See NIH PAR-16-237, PAR-18-007, and PAR-18-017 athttps://cancercontrol.cancer.gov/funding_apply.html#is.) The field extends beyond healthcare into health behavior andpromotion, including public and population health (Leeman et al., 2017), as well as education, policy, and other fields.
The evidence-based interventions that are the focus of implementation science in healthcare include programs,
practices, guidelines, and public and organizational policies that are intended to improve the health of individuals andpopulations (Rabin et al., 2008). To meet its overarching aim, implementation science seeks to identify factors andstrategies that influence or predict adoption, adaptation, implementation (or de-implementation), sustained use, andscale-up and spread of evidence-based interventions.
Implementation science relies on qualitative and quantitative methods—or mixed methods—to understand thestrategies and processes needed to implement an evidence-based change, the outcomes the change produces, thebarriers and facilitators to implementing the change, and how all of these may vary across different types of contexts.
Qualitative research methods are widely used in implementation science (Palinkas, 2014; Weiner et al., 2011). The
power of mixed methods in yielding broader responses to research questions has been demonstrated in numerous
areas of inquiry, and its advantages and challenges have been widely discussed (Zhang & Watanabe-Galloway, 2014;
Hadi & Closs, 2016; Bishop, 2015). The use of mixed methods in implementation science has gained much recent
attention (Tonkin-Crine et al., 2016; Palinkas et al., 2015; Palinkas et al., 2011; Beidas et al., 2014).
The uses of qualitative methods in mixed methods research vary according to the study design and can be quite
complex. The discussion of design in mixed methods studies is beyond the scope of this paper. We will not discuss
mixed methods specifically, but, rather, refer the reader to some notable resources on the subject (Creswell, 2015;
Creswell et al., 2011; Creswell & Plano-Clark, 2011).
While there is a great deal written on standards for qualitative research in general, there are aspects of doing
qualitative research in the context of studying implementation that may make the norms and guidance for bestpractices in this use quite different. For example, demand for timely or rapid turnaround of results, need for multiple
stakeholder perspectives, diversity of contexts across implementation sites, and contextual changes over time all pose
unique challenges to the agility and rigor of qualitative methods. This paper is an initial step to provide guidance to thefield about the use of qualitative methods in implementation science. Our target audiences include implementation
researchers with limited to no experience with qualitative research methods, as well as experienced qualitative
researchers who are learning about implementation science and potentially partnering with implementation scientists.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

1

II . APPROACH
In 2015, the National Cancer Institute’s (NCI’s) Implementation Science Team convened a group of experts in
implementation science and/or qualitative research, to develop guidance for the use of qualitative methods in
implementation science. The Qualitative Research in Implementation Science (QUALRIS) group conducted its work
via teleconference, email, and as an NCI online learning community (https://researchtoreality.cancer.gov). The groupconsulted literature on best practices in qualitative methods and drew upon members’ own extensive experiencein adapting these methods for implementation science. Following this iterative process, the group determined focal
areas to examine, presented preliminary work to scientific audiences (Hamilton et al., 2015), and then drafted this white
paper. After several revisions, the draft white paper was reviewed by the group’s advisory panel, consisting of fiveleaders in implementation science and qualitative methods: Ross Brownson, Cam Escoffery, Russell E. Glasgow, Anne
Sales, and Bryan J. Weiner.
After briefly describing traditional qualitative research methods, we (1) describe the role of qualitative research inhealth-related implementation science; (2) offer guidelines for conducting qualitative research in implementation
science; (3) identify areas of need and opportunities for the development of innovative qualitative methods; and (4)provide resources for those wishing to employ or learn more about qualitative methods in implementation science.
Below is a list of QUALRIS group members and their affiliations.
Deborah Cohen, Ph.D.
Department of Family Medicine
Oregon Health and Science University

Jennifer Leeman, DrPH, MDIV
School of Nursing
University of North Carolina, Chapel Hill

Benjamin F. Crabtree, Ph.D.
Department of Family Medicine and Community Health
Rutgers Robert Wood Johnson Medical School

Deborah K. Padgett, Ph.D., M.P.H.
Silver School of Social WorkNew York University

Laura Damschroder, M.S., M.P.H.
VA Ann Arbor Center for Clinical Management Research
PeRsonalizing Options through Veteran Engagement(PrOVE) QUERI, Ann Arbor, Michigan

Lawrence Palinkas, Ph.D.
School of Social Work
University of Southern California

Alison B. Hamilton, Ph.D., M.P.H.
Center for the Study of Healthcare Innovation,
Implementation and Policy
VA Greater Los Angeles Healthcare System and
Department of Psychiatry and Biobehavioral Sciences
University of California, Los Angeles
Suzanne Heurtin-Roberts, Ph.D., M.A., M.S.W.
Implementation Science Team, Division of Cancer
Control and Population Sciences
National Institutes of Health/National Cancer Institute

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

Borsika Rabin, Ph.D., M.P.H., PharmD
Center of Excellence for Stress and Mental Health, VA
San Diego and Department of Family Medicine andPublic Health, School of Medicine
University of California San Diego and Triple Aim QUERI,
Eastern Colorado Healthcare System, Denver, Colorado
Heather Schacht Reisinger, Ph.D.
Iowa City VA Health Care System andCarver College of Medicine, University of Iowa

2

III . TR ADITIONAL QUALITATIVE RES E ARCH
ME THODS : AN OVERVIEW
Qualitative methods generally emerged from the social sciences, specifically anthropology and sociology, and tend
to focus on approaches for studying human behavior and experience. These attributes are particularly salient for
implementation science because of its focus on understanding how implementation processes influence and are
influenced by dynamic contextual factors, which requires study of human behaviors and experiences, individually
and within collectives (e.g., a healthcare organization or setting). Qualitative methods do not depend upon numericalmanipulation of data, although basic frequencies and summations may be used to present data, provide contextual
information, and transpose data for use in mixed methods analysis.
The diverse forms of qualitative methods do, for the most part, share common elements and perspectives (Crabtree &Miller, 1999; Padgett, 2008; Pope & Mays, 2006; Southam-Gerow & Dorsey, 2014). Traditionally, such methods seek
in-depth understanding, and may be used to generate hypotheses. Qualitative methods are typically inductive andnaturalistic; in other words, data are drawn from people, often in their natural settings. Qualitative methods, to varying
degrees, seek interpretation of social phenomena, rather than their objective representation. Qualitative methods intend
to give a holistic, full representation of complex, dynamic social circumstances reality (Padgett, 2016; Pope, Mays, &Popay, 2006; Damschroder et al., 2009; Kitson et al., 2017). Traditional qualitative research approaches have typicallyinvolved deep engagement of researchers in naturalistic settings and with participants over significant periods of time.
The immersive and iterative nature of qualitative research allows for the comprehension of the dynamism and complexityof aspects of everyday life.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

3

IV. QUALITATIVE RES E ARCH ME THODS IN
IMPLEMENTATION SCIENCE IN HE ALTH
AND HE ALTH C ARE: AN OVERVIEW
Implementation science is action- and change-oriented, with a focus on understanding what is changing in relation
to the intervention and implementation strategies; how, when, where, and why changes happen (or not); and who is
involved in the changes. Studies in implementation science usually occur over a relatively short period of time, witha clear before, during, and after. (See Brear et al., 2018; Ivankova, 2015; and Meyer, 2000, for discussions of action
research and qualitative methods.) With this backdrop in mind, our team has identified nine key features of the use ofqualitative methods within implementation science:1.

In contrast to a more constructivist, exploratory, inductive paradigm, qualitative methods in implementation
science tend to be more positivist and deductive in nature, with their use increasingly guided by theories andorganized by one or more implementation models or frameworks.

2. Qualitative methods in implementation science tend to be practical and targeted, geared more toward
explaining specific implementation-related phenomena.
3. Implementation science involving qualitative methods is inherently multidisciplinary rather thandiscipline-specific.
4. Implementation science typically integrates qualitative and quantitative methods to strengthen findings, often
in the context of complex mixed methods designs rather than using qualitative methods alone.
5. Implementation science using qualitative methods typically involve qualitative methods teams rather thansolo researchers.
6. Qualitative data collection in implementation science is often time-limited, with data collection occurring at
single or punctuated time points (e.g., pre-, mid-, and post-implementation), often in rapid fashion rather thancontinuous data collection over an extended period, a hallmark of traditional qualitative methods.
7.

Engagement in implementation science tends to be targeted toward multiple stakeholders (e.g., providers,administrators) in multiple, diverse settings (e.g., several health clinics) rather than traditional long-term, indepth engagement within a community.

8. Qualitative data analysis in implementation science tends to be targeted to specific a priori research questions,is usually more rapid, and is often integrated with quantitative analyses. This contrasts withmore traditional qualitative analysis, which tends to be phenomenologically oriented with an emphasis
on reflexivity.
9. In addition to contributing to traditional outlets such as the scientific literature, implementation science seeks todisseminate both qualitative and quantitative findings through “grey” literature, including interim reports, policy
and operations briefs, workshops and trainings, online resources, site profiles, and implementation metrics
that may be more readily accessed by practitioners.
Implementation science is a young, evolving, and complex field of research. Its researchers are confronted
with several challenges, including facing and embracing the complexity and dynamic nature of interventions,
implementation processes, and intersecting implementation contexts and their interactive effects on outcomes(Abbott et al., 2014; Azar et al., 2015; Brives et al., 2016; Brownson et al., 2012; Damschroder et al., 2009;
Glasgow, 2008; Glasgow et al., 2012).

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

4

V. VALUE OF QUALITATIVE ME THODS IN
IMPLEMENTATION SCIENCE
Here we review several ways in which qualitative methods are valuable in implementation research
conducted to date.
A. Elicit Stakeholder-Centered Perspectives
Qualitative methods are used to elicit the perspectives of those who have a “stake” in implementation and who areresponsible for “healthcare-related decisions that can be informed by research evidence,” i.e., stakeholders, such asadministrators, providers, staff, patients, clients, families, and community members (Martinez et al., 2016; Concannon
et al., 2012). Understanding the perspectives of those involved in and/or affected by implementation of an intervention
is essential to ensuring that interventions address problems recognized as high priority and use approaches that arefeasible and acceptable within real-world practice contexts (Chalmers & Glasziou, 2009; Glasgow et al., 2012; Tunis etal., 2003). For example, Leeman and colleagues (2014) conducted in-depth interviews with public health practitioners
working in obesity prevention to understand factors that influenced the adoption and implementation of evidence-based
policy, systems, environmental change interventions at the state and local levels. Kane and colleagues (2014) also
conducted interviews with providers to assess perspectives on feasibility and acceptability of new guidelines. Qualitative
researchers often bring a participatory perspective, especially when focused on implementing practices to serve
marginalized populations, relying on partnerships with a wide range of stakeholders to achieve study goals.

B. Inform Design and Implementation
Qualitative studies have been used to inform the design and implementation of interventions. For example, Proctor andcolleagues (2007) conducted a qualitative pilot study to document the perspectives of agency directors on the challengeof implementing evidence-based practices in community mental health agencies prior to the development and testing ofa specific implementation intervention. Using a participatory research approach (see Reason & Bradbury, 2008), Elseyand colleagues (2016) conducted qualitative semi-structured interviews and focus groups in Nepal to understand patientand health worker knowledge of tobacco and patients’ motivation to quit. Findings were used to inform the design andimplementation of a smoking cessation intervention. A Norwegian research team used a series of individual and groupinterviews with various stakeholders to inform the implementation of tailored interventions to improve the treatment ofelderly patients with depression (Aakhus et al., 2015). The stakeholders, including health professionals, researchers, andrelatives of elderly patients with depression, were asked to brainstorm interventions to improve adherence to treatment
guidelines. These qualitative findings were used to test targeted versus non-targeted interventions to improve adherencein an implementation trial. Semi-structured interviews with mental health practitioners were used to examine the barriers
to using a smartphone application to enhance patient compliance to evidence-based psychotherapy for post-traumaticstress disorder (PTSD) (Reger et al., 2017). Based on these interviews, a targeted intervention was developed toincrease use of the smartphone application among PTSD therapists.

C. Understand Contexts Across Diverse Settings
One of the most frequently cited reasons for using qualitative methods in implementation science is to understandcontextual factors that influence implementation, such as variations in uptake (Bekelman et al., 2016; Cadogan etal., 2016; Cohen et al., 2016; Elsey et al., 2016; Lord et al., 2016; Marshall et al., 2008; Marty et al., 2008; Rapp etal., 2010; Sommerbakk et al., 2016; Varsi et al., 2015; Damschroder et al., 2017). Since contexts can differ markedly
across implementation sites, understanding contexts and their relation to implementation is of critical importance.
See Section VI/B below. Qualitative methods are well suited to explore how context, actions, and people contributeto successful (or failed) implementation. Researchers have examined factors that influence implementation of the

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

5life of study or initiative, from pre-implementation (Koenig et al., 2016), to implementation (Moeckli et al., 2013), tosustainability (Colon-Emeric et al., 2016). For example, researchers (Swain et al., 2009) studied implementation of an
illness management and recovery program for people with severe mental illness in community mental health settings.
Qualitative methods were used to assess perceived barriers and facilitators of implementation such as leadership,organizational culture, training, and staff and supervision. The analysis showed that several cross-site factors worked
synergistically to influence implementation (Whitley et al., 2009). Freedman and colleagues (2012) supplemented survey
data with in-depth interviews to identify factors critical to a health center’s readiness to implement a farmers’ market.
They found several factors to guide researchers and health center staff in development of a farmer’s market to improveaccess to healthy foods. Damschroder and colleagues (2009) have used qualitative assessments, guided by theConsolidated Framework for Implementation Research (CFIR), to understand variations in implementation success andprovide recommendations for scaling up implementation of programs (Damschroder & Lowery, 2013; Damschroder,
Reardon, AuYoung et al., 2017; Damschroder, Reardon, Sperber et al., 2017). Internationally, researchers have used
qualitative methods to encapsulate findings in conceptual models of system transformations (Bate et al., 2008; Lukas etal. 2007), and have highlighted context as an active agent influencing the implementation and not just a backdrop to it(Dopson & Fitzgerald, 2006).

D. Provide Documentation and Encourage Reflection on Implementation Processes
Another value of qualitative methods is to document and reflect on implementation processes. For example, Aarons andPalinkas (Aarons & Palinkas, 2007; Palinkas & Aarons, 2009) collected qualitative data through annual interviews andfocus groups to assess the process of implementation of Safe Care®, an intervention designed to reduce child neglect
and out-of-home placements of neglected children into foster care. Hoagwood and colleagues (2007) used a casestudy of an individual child to describe the process of implementation of an evidence-based, trauma-focused, cognitivebehavioral therapy for treatment of symptoms of PTSD in children living in New York City in the aftermath of the World
Trade Center attack on September 11, 2001.
Qualitative methods such as ethnography are well-suited to capture implementation micro-processes, that is, atthe level of individual interactions (Nilsson et al., 2018). Furthermore, fidelity to intervention use and implementation
strategies and their adaptations can be documented and better understood by qualitative inquiry, as can adoption of
improved practices (Bunger et al., 2017; Cohen et al., 2008). Such inquiry can inform development of and contribution
to the adaptome, an aggregate body of knowledge documenting “adaptations to interventions and their impacts
on implementation, service, and health outcomes” as proposed by Chambers and Norton (2016). This also raises
considerations of flexibility and fidelity during adaptation (Cohen & Crabtree, 2008).

E. Gain Insight Into Implementation EffectivenessQualitative methods are used to study the effectiveness of implementation. Proctor and colleagues (2011) created ataxonomy of implementation outcomes to consider when assessing implementation effectiveness. Many implementation
outcomes are well-suited for evaluation via qualitative methods. Furthermore, qualitative methods are often used
when valid measures are not available. (See Rabin et al., 2016, for measurement resources in implementation). The
effectiveness of implementation strategies, (i.e., approaches employed to integrate an intervention into a new practicesetting; Powell et al., 2012; Powell et al., 2017; Powell et al., 2015), also requires evaluation, often best accomplished withqualitative methods. Wozniak and colleagues (2015) evaluated the implementation of an efficacious collaborative caremodel or patients with diabetes and depression in Canada. Researchers used qualitative interviews among staff andsystematic documentation of the implementation to assess effectiveness. The qualitative data analysis identified factors
such as lack of training in team operations, physician-nurse relations, and local physician culture as barriers to optimal
collaborative care that the model stipulated.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

6Rapp and co-authors (2008) used a mix of qualitative data collection techniques to identify and evaluate strategies that
contributed to successful implementation in six community mental health centers (CMHCs). Data collection techniques
included text analysis (e.g., notes, memos, reports), formal and informal interviews, and direct observation. Data
was coded according to 26 a priori themes and included in comprehensive site reports for each of the six CMHCs.
Through an iterative analytic process, investigators and key stakeholders blinded to the original a priori categories,
created a provisional coding guide. This was used to develop a final coding scheme applied to all six implementationsites. Eight strategies were found that contributed to successful implementation in the six CMHCs.

F. Understand Mechanisms of Change
Qualitative methods can be employed to understand mechanisms of change and the relationship(s) between thetheorized and actual mechanisms of change (Kazdin, 2007) by identifying and explaining which aspects of theintervention or its implementation result in change. Now, “mechanism of change” can be defined variously and one
study’s mechanism may be another study’s moderator or intermediate outcome. A discussion of this variation isbeyond our scope, but in this section, we will describe some uses of qualitative methods to understand change. (Forfurther perspectives on change, see Lewis et al., 2018; Dalkin et al., 2015; May, 2013; Rycroft-Malone et al., 2012).
Following a continuous quality improvement (CQI) effort for care of Indigenous Australians, Schierhout and colleagues(2013) explored the patterns of change in service delivery in 36 widely-varying health centers that had participatedin the CQI project. Using a mixed-methods “realist” approach (see Rycroft-Malone, 2012; Dalkin et al., 2015),
implementation researchers conducted key informant interviews to elicit interpretations about cross-site variation’srole in producing service-delivery variation. Inductive analysis led to fine grained hypotheses of change and further
data were collected though clinical audits, project monitoring reports and workshops with key informants involved in
implementation. The team abstracted potential patterns of context, mechanism, and outcomes to develop a working
theory of change for program strengthening.
Qualitative methods can help us to understand how interventions (change) can be sustained (Cohen et al., 2016;
Palinkas et al., 2016; Colon-Emeric et al., 2016). For example, in a study examining how to sustain complex
interventions in long-term care, 15 focus groups were conducted with 83 staff who had participated in the intervention
trial (Colon-Emeric et al., 2016). Participants gave perceptions of intervention salience and sustainability. Analysis
produced insights about intervention features, organizational context, and external support offering concretesuggestions on how to maintain changes brought about by interventions.
Damschroder, Reardon, AuYoung, and colleagues (2017) mapped qualitatively-identified barriers and facilitators,
to specific types of outcomes defined by RE-AIM to identify factors that may moderate the intervention’s effects
on outcomes. This work highlights the key role qualitative methods can play in explicating multiple potential causal
pathways for further testing (Forman et al., 2017).

G. Contribute to Theoretical Development
There is ample description, cataloguing, and discussion of conceptualization, that is, theory, frameworks, andmodels in implementation science (Nilsen, 2015; Chambers, 2014; Tabak et al., 2012; Davies et al., 2010; May et al.,2009; Eccles et al., 2005) but less agreement on terminology. According to Tabak and colleagues (2018), theory andframeworks can both be considered to belong to the category of models. Davidoff and colleagues (2015) recognize
multiple levels of theory, grand, mid-level, and lower. They propose that lower or “program” level theories are perhaps
the most useful in improvement work. Similarly, Chambers (2014) points out that rather than “classical” theory, thesemore immediate-level models or frameworks are useful in formulating specific research questions and explaining
dissemination and implementation processes across diverse stakeholders and settings.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

7Researchers have explicitly challenged the implementation science community to contribute to advancing existing
theoretical frameworks to stimulate new theoretical developments (Damschroder et al., 2009; Kitson et al., 2008).
Theory provides an efficient way to build the knowledge base through shared language and concepts that arefoundational for guiding data collection, measurement, analyses, and interpretation, and creates a natural bridge ofcomparison to other studies using the same theories (Colquhoun et al., 2014; Foy et al., 2011). Qualitative approaches
can advance conceptual development in multiple ways.
Qualitative methods can inform the development of conceptual models or frameworks through grounded methods(Corbin & Strauss, 2008; Glaser & Strauss, 1967). For example, Zazzali and colleagues (2008) developed a conceptualmodel of implementation of Functional Family Therapy using qualitative data elicited through semi-structured interviews
with 15 program administrators. Their model could then be tested using quantitative methods. Blasinsky et al. (2006)
developed a rating scale to construct predictors of program outcomes and sustainability of a collaborative careintervention to assist older adults suffering from major depression or dysthymia based on qualitative data obtained
from semi-structured interviews. Woltmann et al. (2008) created categories of turnover and designations of positive,
negative, and mixed influence of turnover on outcomes using qualitative data obtained through interviews with staff,
clinic directors, and consultant trainers in their study of staff turnover during implementation of evidence-based
practices in mental healthcare.
Testable theories of implementation within organizations have been evaluated using qualitative methods. For
example, Klein and colleagues’ model of implementation (Klein & Sorra, 1996; Klein et al., 2001) was first tested in a
manufacturing setting. This model was then adapted for clinical settings and assessed for applicability and usefulness
in two different studies using qualitative data (Helfrich et al., 2007; Damschroder et al., 2011) collected through semistructured interviews of key stakeholders. Both studies recommended refinements to the model based on theirrespective qualitative findings. The adapted model can be further tested in more settings and verified using a full rangeof methods including qualitative, quantitative, and/or mixed methods.
Highly cited implementation frameworks have been developed based on foundational, empirical qualitative data
including the Promoting Action on Research Implementation in Health Services (PARIHS) (Kitson, 2008) andNormalization Process Theory (NPT) (May, 2006; May et al., 2009). PARIHS developers further developed theirframework into i-PARIHS, based on a growing knowledge base (Harvey & Kitson, 2015), and May (2013) further
extended NPT to create a generalized, mid-range theory of implementation based, in part, on accumulating qualitative
findings from several researchers using NPT. Because of the complex and dynamic interplay between implementation
processes, places, and people, qualitative methods are often used in conjunction with frameworks to guide evaluation
or execution of implementation. For example, a systematic review of published studies using the CFIR, revealed that 23
of 26 eligible studies used qualitative methods (Kirk et al. 2016).

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

8VI . ME THODOLO GIC AL CON SIDER ATION S
FOR THE US E OF QUALITATIVE ME THODSIN IMPLEMENTATION SCIENCE
When using qualitative methods in implementation science, several considerations should be kept
in mind, including sampling procedures, the timing of data collection, the role of the qualitative
researcher, procedures for collection and analysis of qualitative data, and presentation of findings.
Each of these considerations is examined briefly below.
A. Sampling
Sampling of participants in qualitative studies is generally purposive in nature (Palinkas et al., 2015). Random samplingis occasionally used if participants are involved in quantitative studies as well or if there is a need to generalize study
results in a manner that cannot be accomplished with maximum variation or other forms of purposive sampling. In
implementation science, participants are often sampled based on their level of participation in the implementation
process and role in the organization to maximize diversity of perspectives (Colon-Emeric et al., 2016). Bekelman andcolleagues (2016) conducted stratified purposeful sampling to capture the range of perspectives from providers ofdifferent disciplines who provide or have patients enrolled in outpatient palliative care. They also recruited leaders who
were likely to decide whether and how to adopt and sustain outpatient palliative care in their settings.
Implementation scientists may also select the entire population of those implementing or receiving the intervention,
particularly when the population is small, such as in a feasibility pilot study of an implementation strategy. Sample size
may also be limited by geographic setting, such as population defined by a census tract. To be sure, size is not themost important sampling issue, depth or data saturation is the goal. The state of saturation is reached by sampling to
the point of redundancy or repetition, when data collection in a particular sample yields no new information (Crabtree &Miller, 1999; Francis et al., 2010).
Samples may be selected from any population of interest, not only individual persons but of care delivery sites,
political or economic institutions, geographic entities, and so on. Focusing on case study analyses, Sax et al. (2013)
purposefully sampled six hospitals from a larger study of 15 hospitals involved in an infection control implementation
project across Europe. The investigators used criteria sampling to choose three “high potential for change” hospitals
and three “low potential for change” hospitals to examine the process of uptake of the intervention designed to reduce
hospital infections in intensive care units. This sampling strategy was based upon preliminary qualitative results.
Researchers then used key informant interviews, ethnographic observation, written protocol collection, guidelinecollection, and photography to collect data relevant to infection control. Preliminary analysis of data found 15 themes
relevant to successful implementation of infection control practices (Casillas et al., 2013) across the hospitals and factors
influencing leadership dynamics in infection control (Clack et al., 2013).

B. Timing of Data Collection
Due to implementation science’s orientation toward change, qualitative researchers must pay particular attention to
the dynamic nature of the contexts in which they are working (Chambers et al., 2013). See Section V/C above. Time isa central factor to consider when conducting qualitative research in implementation science. Early in implementation
studies, qualitative researchers are often called upon to assess the organizational or community context in which
the intervention will be implemented. This may require semi-structured interviews with the key stakeholders who will
be involved or impacted by the intervention to examine the “baseline” context of a setting, asking questions suchas, what is the stakeholders’ current practice in relation to the target outcome? (Stewart et al., 2010; Nutting et al.,Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

92010). Observations of the community or clinical setting are frequently conducted to assess the interactions of the
stakeholders and whether their behavioral practices are consistent with the perceptions they articulated in interviews.
Observations can be especially helpful in documenting current processes (Nilsson et al., 2018).
The baseline assessment of context is then used to make key decisions in the next stage of the study. Qualitative
findings are often used to choose the most appropriate implementation strategies or, at the very least, to tailor strategies
to settings and actors. The timing of when the intervention must be implemented, therefore, impacts how the baseline
data will be analyzed. With a typically shorter turnaround time, qualitative researchers in implementation science haveturned toward rapid analysis techniques, including template analysis, in which key concepts are identified in advance tostructure and focus the inquiry (Reger et al., 2017; Hamilton, 2013; Fox et al., 2016).
Qualitative researchers are often asked to conduct formative evaluations of implementation processes, which may entail
documentation of implementation strategies as they are used in real time. Qualitative researchers may also be asked toprovide feedback as the implementation process is occurring to aid in the change process, asking questions such as,
what are the major facilitators and barriers to implementation, and what could be changed to improve implementationand sustainability?
Finally, qualitative research may be conducted at the end of an implementation study to understand and explain both
implementation and effectiveness outcomes. It may also be used to assess and consider the reactions of adopters of
the intervention as well as stakeholders who resisted adoption.
An issue related to timing is the iterative nature of qualitative and mixed methods research in implementation science.
Understanding and facilitating implementation and sustainment of innovative and evidence-based practices is acontinuous learning process. With multiple time points of data collection and attention towards change, this canmake harnessing the iterative nature of qualitative research more difficult. This also raises the issue of when qualitative
methods “cross over” and become part of the implementation strategies, e.g., when key stakeholder interviews become
mechanisms of education about and engagement in an intervention (Hamilton et al., 2017).

C. Roles of Qualitative Research Teams in Implementation Science
Implementation science is a truly interdisciplinary field. The health-related implementation science team generally
includes clinical experts, social-behavioral scientists, biostatisticians, economists, intervention developers, andorganizational scientists (Fernald & Duclos, 2005; Guest & MacQueen, 2008). Qualitative researchers need to consider
this team as they develop interview and observational guides as well as when they decide who will conduct theinterviews, and who will participate in the analysis process (Damschroder & Lowery, 2013). The qualitative expert
will need to lead these activities while managing the interdisciplinary team’s involvement. Leading a multidisciplinary
team can be a challenging undertaking (see Vogel et al., 2014; and Hesse-Biber, 2016; Crabtree et al., 1994, on teamscience). The challenge can be amplified when the qualitative expert is not the principal investigator. See Section VIII.

D. Data Collection
Several different data collection techniques have been used in implementation science, such as one-on-one interviews,
focus groups, observation, and archival or document analysis
Interviews can be conducted either in-person, by telephone, or electronic media (e.g., Skype) and are ideally digitally
recorded and transcribed verbatim for analysis. (Mealer & Jones, 2014; Kothari et al., 2012; DiCiccio-Bloom & Crabtree,2006), Also used in implementation science are semi-structured interviews or elicitation methods techniques that utilize
a quasi-statistical perspective. For example, Palinkas and colleagues (2016) reported using a free list exercise to elicit

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

10participants’ conceptions of the term sustainment, the elements of their programs they wished to sustain, and therequirements for sustaining these elements. Responses were weighted based on order of presentation and comparedby source of program funding.
Focus groups, also called group interviews, are widely used in health research and in implementation science(Reichstadt et al. 2007; Tausch & Menold, 2016; Kitzinger, 2006; Krueger & Casey, 2000; Thomas et al., 1992). The
advantage of the group interview approach is that it capitalizes on within group interaction to stimulate thought resulting
in rich detailed accounts of the focal topic (Asbury 1995). Focus groups are usually held on-site, gathering participantsin a common locality (Howard et al., 2017) but they can also be held online. DuBois and colleagues (2015) conducted
online focus groups about decision-making among gay and bisexual adolescent males. Focus groups tend to be
popular with researchers not only because of the results they generate, but also because they frequently offer a more
economic use of financial and spatial resources as well as time. However, there are drawbacks and limitations to focus
groups and the decision to use focus groups or not really depends upon the information being sought. Carey and Smith(1994) noted the major pitfall of the focus group technique is the potential impact of censoring and conforming behavior
in group interaction. Participants can be strongly influenced by each other, potentially narrowing the range of ideas
obtained. Also, participants may feel uncomfortable to express a sincerely held idea or opinion, which calls into question
the accuracy of results (Smithson, 2000).
Observation often occurs during site visits, which are conducted for a variety of reasons, including assessing local
contexts and observing the nature and intensity of how interventions are being implemented (Cohen et al., 2016; Fox
et al., 2016; Palinkas et al., 2016). Data gleaned from observation is typically in the form of field notes, which may be
more structured in implementation science than in traditional ethnographic research, in order to facilitate more rapid
or targeted analysis. When conducted over a period of time, such observations capture the interactions and behaviors
of participants in their natural settings (e.g., a hospital or community-based organization). For example, Escoffery andcolleagues (2009) made annual observations of 40 swimming pools during the five-year, mixed-methods implementation
study of an evidence based skin cancer prevention program along with qualitative interviews and quantitative surveys.
Archival analysis entails the collection of documents, including meeting minutes, logs, policies, guidelines, and healthrecords (Turner et al., 2016; Lessard et al., 2016). Pelcastre-Villafuerte and colleagues (2014) evaluated the processes
and performance of Project “Casas” to deliver culturally and linguistically appropriate services to indigenous womenin four Mexican states. Data were collected through extraction from archived regulatory and program documents aswell as semi-structured interviews. The study not only evaluated the past performance of the project, it also provided
information for the improvement of the existing program sites, as well as more cost-effective implementation of themodel in new sites. Other archival data include diaries and videotapes used to document implementation experiences(Cohen et al., 2016; Elsey et al., 2016), videotaping, and photography (e.g., photo voice) (Kowitt et al., 2015).
Implementation science typically includes multiple data sources to aid in triangulation. For example, Kennedy andcolleagues (2016) employed a longitudinal case study design in which ethnographic methods comprising video, nonparticipant observation of intervention delivery, and qualitative interviews at baseline, six and 12 months were used
to identify processes and dynamics of delivering a social network intervention and to capture individual outcomes of
the use of a web-based tool. Shea et al. (2014) studied the feasibility of implementing a virtual tumor board programthat connected community-based clinicians to academic tumor boards. Methods included observation of virtual
tumor board cases using a structured observation checklist and semi-structured interviews with participants. In an
implementation study of behavioral health counseling in primary care practices, Cohen and colleagues (2011) used visit
notes, documents, interviews, and online implementation diaries in a comparative analysis of nine different interventions
in nine practice-based research networks. They found practice health risk assessment and brief counseling, coupledwith referral and outreach to a valued and known counseling resource, to be the best way to consistently coordinate
and encourage follow-through for health behavior counseling.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

11E. Data Analysis
A commonly used analytic approach to qualitative data in implementation science is thematic analysis (Hsieh & Shannon,2009). There are several variations of thematic analysis, ranging from techniques that are largely deductive to techniquesthat are largely inductive, with substantial variation in between. Reflecting more of a deductive approach to qualitative
data analysis, several studies (Bajunirwe et al., 2016; Elsey et al., 2016) have employed a framework or templateapproach (Gale et al., 2013; Parkinson et al., 2016; Srivristava & Thomson, 2009) to data analysis. See Section VIII. Suchapproaches are theory- or framework-driven and involve mapping data or emergent themes onto a priori domains,themes, or categories. Cadogan and colleagues (2016) used a framework analysis approach to map emergent themes
onto the Theoretical Domains Framework (Michie et al., 2014). Several studies (Elsey et al., 2016; Kennedy et al., 2016;
Nordmark et al., 2016) have used NPT (May & Finch, 2009) to provide conceptual orientation for exploratory data analysis
of complex interventions.
One of the better known examples of a framework approach is the use of the CFIR (Damschroder et al., 2009) toguide the analysis of data collected through semi-structured interviews. For example, Bekelman and colleagues (2016)
conducted semi-structured interviews, using the CFIR domains to assess barriers and facilitators to the adoption andscale-up of outpatient palliative care. Varsi and colleagues (2015) used the CFIR to examine barriers and facilitators forthe implementation of an internet-based patient-provider communication service (Forman et al., 2017). Hamilton andcolleagues (2018) used CFIR post-hoc to examine the impact of external context on implementation of an evidencebased prevention intervention (Hamilton et al., 2018).
Content analysis typically involves identifying pre-selected incidents or references in the data (Padgett, 2016). Frequently
used with archival materials, content analysis does not entail a search for deeper meanings and interpretations.
Padwa and colleagues (2016) conducted a directed content analysis of qualitative data using the Conceptual
Model of Evidence-Based Practice Implementation in Public Service Sectors (Aarons et al., 2011) to understand theimplementation of integrated behavioral health protocols in primary care settings.
Since theoretical frameworks are generally expected to drive implementation science (Tabak et al., 2012), highly
inductive approaches such as grounded theory (Charmaz, 2006) may not always be a good fit for analyzing qualitative
data gathered in implementation science (Davidoff et al., 2015). In grounded theory, preconceived conceptual categories
must “earn their way” into the findings, rather than drive the data collection effort (as in implementation science).
However, principles of grounded theory can still be useful in informing implementation-related qualitative analysis (e.g.,
being data-driven).
Perhaps the most common strategy for data analysis is to combine inductive and deductive approaches. For instance,
Mozaffar et al. (2016) conducted a thematic inductive and deductive analysis, using a biography of artifact perspective
to move beyond a single timeframe and a “localist” perspective and to examine both technical and nontechnicalaspects of computerized hospital physician order entry and clinical decision-support systems. Colon-Emeric et al.
(2016) analyzed interview transcripts using framework analysis of a priori concepts, combined with inductive analyses.
Sommerbakk and colleagues (2016) used a combination of thematic analysis using an inductive approach andtheoretical thematic approach, applying codes to Grol & Wensing’s (2004) multilevel model of barriers and facilitators.
Using a combination of inductive and deductive approaches, analysts can draw on ideas emerging from the empirical
data as well as frameworks and existing literature (Bradley, Curry, & Devers, 2007).
Whether inductive, deductive, or a combination of the two approaches, thematic analysis usually involves the processof constant comparison (Strauss & Corbin, 1998) to construct themes based on a taxonomy of codes. Themes may
be compared across organizations and stakeholder groups (Kane et al., 2016). In the rapid analytic approach used by
Hamilton (2013), main topics (domains) are drawn from interview and focus group guides, and a summary template is

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

12developed and used to summarize transcripts (e.g., Fox et al., 2016). Summaries are analyzed using matrix analysis andkey actionable findings (e.g., the variable use of implementation strategies) are shared with the implementation team toguide further implementation in real time, particularly during the course of phased designs such as in a Hybrid Type II
Study (Curran et al., 2012). This rapid approach can inform more in-depth hybrid analytic approaches, described above.
In terms of assessing interrater reliability during data analyses, coding independently then meeting as a group to reachconsensus appears to be the most common strategy (Bekelman et al., 2016; Cadogan et al., 2016; Lessard et al., 2016;
Lord et al., 2016; Martinez et al., 2016). During these meetings, investigators compare how the same texts are coded bydifferent coders and resolve discrepancies in terminology and assignment of codes (Fox et al., 2016). Some qualitative
studies (Damschroder & Lowery, 2013; Lord et al., 2016) make explicit use of a consensus-based directed content
analysis approach (Hill, 2012). In a few instances, this comparison is quantified by calculating percent agreement or
kappa statistics in assignment of codes (Palinkas et al., 2013; Saldana, 2014). Occasionally, a qualitative expert not
integrally involved in the study is employed to review or “audit” the process as an effort to maximize validity of findings(Damschroder & Lowery, 2013). (For a thorough discussion of team-based qualitative analysis see Guest & McQueen,
2008, and consult section VI/c, above.)

F. Presentation of Qualitative Findings
It is a challenge to present depth of qualitative analysis in the limited space allowed for journal articles. Most
implementation science presents qualitative results organized by themes illustrated with direct quotations from study
participants, an approach that is accessible to readers (Sandelowski & Leeman, 2012). Results should not merely becomprised of a list of themes, but should specify which themes were emergent and which were a priori, and whenrelevant, how themes are related to one another. Although verbatim quotations are commonly used to document
the existence or character of a theme, they should also provide a level of eloquence and insight not possible witha summary of the theme. Usually, one quote is sufficient to make a specific point, unless the point itself contains
important nuances or contrasts that require more than one quotation to illustrate.
When using quotes, it is advisable to link them to a participant identification number so that readers can be assured
that different participants are being quoted. When multiple stakeholders are involved, identifying their role (e.g., hospitaladministrator, nurse, physician), if permissible, helps to contextualize what is being said and by whom. In addition, formulti-site studies, it is useful to use identifiers to distinguish between sites and demonstrate contextual patterns within
and across sites and/or experimental conditions, if relevant.
Although themes and associated quotes typically form the narrative structure of the findings, tables have also beenused to organize the presentation of themes with illustrative quotes. Tables have the advantage of presenting theresults in a condensed form, enabling authors to publish in journals that have more stringent word limitations. Elsey andcolleagues (2016) used qualitative data to construct a table of barriers and facilitators to implementation using NPT (Mayet al., 2009). Also, Guetterman, Fetters, and Creswell (2015) investigated mixed methods displays and discussed what
each display type communicated and represented.
A third form of presentation of study results is the use of figures and diagrams that illustrate relationships among themes
identified from analyses of qualitative data. Palinkas and colleagues (2008) used a figure to illustrate relationships among
themes and subthemes representing predictors and outcomes of short-term implementation of three evidence-based
treatments for depression, anxiety, and behavioral problems in 8- to 13-year-old youths. Lessard and colleagues(2016) used figures to depict a conceptual framework of facilitation categories and themes, the relationship between
implementation-oriented facilitation roles and support-oriented facilitation roles, and the dynamic interrelations between
groups of actors and/or between actors in the same group of facilitators. Elsey and colleagues (2016) used qualitative
data to construct a figure outlining patient flow in a proposed smoking cessation intervention.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

13VII . COLLEC TING AND ANALY Z INGQUALITATIVE DATA WITH RIGOR
Qualitative methods encompass diverse approaches, yet they share a need to foster standards of quality andmethodological rigor (Lincoln & Guba, 1985; Tracy, 2010). Given differences in paradigmatic or epistemologicalorientations, qualitative researchers may adhere to a non-prescriptive interpretive approach or they may prefer asomewhat more structured or formalized approach (post-positivist) to determining quality. Most qualitative researchers
in health services and implementation science opt for explicit standards without sacrificing the flexibility that is thehallmark of qualitative inquiry.
It is helpful to distinguish between criteria applied to completed qualitative studies (evaluative standards) and specificstrategies or actions undertaken as part of the study’s design and procedures. This distinction, beginning with a
discussion of evaluative standards and moving on to recommending specific strategies for rigor, will comprise thissection. Evaluation standards are an imperative from the beginning to insure all members of the team agree upon theprocedures and basic techniques that will be used to determine assess outcomes (Frankel, 1999; Cohen & Crabtree,2008). Such standards have become especially salient with the rise of qualitative systematic reviews where thediversity of methods and approaches makes consensus a challenging but necessary goal (Tracy, 2010).
The provision of evaluative standards specific to qualitative inquiry began with Lincoln and Guba (1985) and later gave
rise to standards in various disciplines, including medicine and healthcare (Mays & Pope, 1995). In 2008, the Robert
Wood Johnson Foundation’s Qualitative Research Guidelines Project reviewed the extant literature and promulgatedguidelines (www.qualres.org/index.html) for qualitative researchers. These guidelines begin with an overall indication ofquality, known as trustworthiness, meaning the qualitative study inspires confidence in the ways it was conducted andthe data were interpreted.
The Cochrane Collaboration has assembled a list of checklists and software tools that might be applied to assess thequality of qualitative studies, and advocates a focus on four core principles: credibility, transferability, dependability,
and confirmability. Qualitative findings are “transferable” by yielding new concepts or frames of reference that canapply to different settings. Examples of the type of criteria that might be applied include, assessing (1) the qualitative
methods employed, which often differ from the terms authors use to name their methods; (2) the relevance of sources(informants, documents, etc.) to research questions; (3) efforts employed to ensure reliability in data analysis; (4)consideration of how findings relate to context; and (5) extent to which sample size and composition are adequately
described and sufficient to support the findings (Hannes, 2011). Qualitative research methods value “thick and rich”
description (see Robert Wood Johnson Foundation at www.qualres.org/HomeThic-3697.html).
Qualitative researchers do not try to fully eliminate subjective researcher bias from a study. In other words, they accept
that scientific inquiry cannot be value-free. This contrasts with much quantitative research, which seeks to remove
subjective bias and control for or eliminate other sources of bias. However, diligent qualitative researchers seek tominimize bias, where possible, by identifying their biases early on and reckoning with them throughout a study. Thisis accomplished through “reflexivity,” or maintaining vigilance and mindfulness during the study, acknowledging one’s
biases and reflecting on them in a way that lends credence to the findings.
Some qualitative researchers believe that they need not (and should not) adhere to quantitative standards of internaland external validity as these would compromise the very nature and value of qualitative methods. However, qualitative
researchers do not share complete consensus on evaluative standards nor the optimal means of meeting thesestandards. We suggest that the best approach for methodological integrity is for implementation researchers toarticulate the standards they use for trustworthiness and be transparent in describing the methods used and rationale
for their use. Once the study is complete, the researchers are accountable for demonstrating that their chosen
standards were met in the design, conduct, and analysis.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

14Enhancing trustworthiness implies use of strategies for rigor throughout the study (Padgett, 2016). Unlike evaluative
standards that inform the study and establish criteria for evaluating its overall quality, strategies for rigor are specifictechniques or actions taken before and during the course of the study. When it comes to rigor strategies, there are no
rigid formulaic expectations but there is a menu of options from which the researcher can choose, as befits the study
design. Below is a list of the most commonly used strategies and what they entail.
•

Triangulation of data—using more than one source of data (e.g., interview transcripts, field notes, archival
material, survey results).

•

Prolonged engagement—with study participants and in study settings, via multiple interviews, extended
periods of observations, etc.

•

Member checking—returning to participants to verify or validate portions of the data or interpretations. Thiscan be especially important in community-based research as well as patient-centered research.

•

Audit trail—the use of analytic memos, journals, and other means of documenting the study’s procedures toenhance transparency of methods.

•

Peer debriefing and support—meeting regularly with the study team to share study findings and processes
to identify biases as well as support one another.

•

Negative or deviant case analysis—seeking alternative explanations in the data to avoid foreclosing
analyses and conclusions prematurely.

For implementation science that incorporates qualitative methods, some, but not necessarily all, of these strategies
would be appropriate. For example, prolonged engagement, the sine qua non of ethnography, is more challenging in
implementation science, but not impossible. In the European hospital study by Sax and colleagues (2013), prolonged
engagement entailed devoting two days to ethnographic site visits to hospitals. Their rationale stated, “Staying for twodays rather than just one allows us to take advantage of the exposure re-exposure effect. This helps to counter the
Hawthorne effect,” (Sax et al., 2013) wherein individuals modify their behavior in response to their awareness of being
observed (McCambridge et al., 2014).

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

15

VIII . NEED AND OPPORTUNITIES FORINNOVATION IN US E OF QUALITATIVE
ME THODS IN IMPLEMENTATION SCIENCE
Use of qualitative methods in implementation science reflects practices consistent with their use in the broader fieldsof health services and the social and behavioral sciences. However, the demand for such methods in implementation
science represent both needs and opportunities for using existing methods in innovative ways or developing entirely
new methods. First, though team-based qualitative methods are commonly used in implementation science (e.g.,
Reger et al., 2017; Damschroder & Lowery, 2013; Moeckli et al., 2013), there is a need to bring greater transparency
to and documentation of team-based analysis (Guest & MacQueen, 2007). See Section VI. Most qualitative studies of
implementation describe holding meetings that are designed to reach consensus in coding of data or construction ofthemes. However, it is unclear what constitutes consensus (e.g., is it 100 percent agreement on all decisions, greaterthan 50 percent agreement), and whether reaching consensus represents a uniform process with a uniform outcome
such that it can be replicated within the same study or compared from one study to the next.
Second, we need to continue to strengthen tools and techniques for conducting rapid qualitative assessment andanalysis. Rapid assessment procedures (RAP) are frequently used in program planning and evaluation (Cifuentes etal., 2006; Scrimshaw & Hurtado, 1987; Scrimshaw et al., 1991). RAP is generally characterized by a small researchteam and compressed time frame, in which specifically described and quite limited research questions are addressedusing multiple sources of qualitative data such as observation and interviews (Beebe, 2001; Utarini et al., 2001). RAP
is especially appropriate for implementation research because it (1) reduces the time required to conduct the study,
thereby reducing the gap between research and practice, and (2) is more likely to involve a partnership between
researchers and practitioners, enabling both groups to contribute to study results and to learn from one another inthe process. Innovations in the use of RAP might include developing standardized procedures for training research
assistants, use of social media and other forms of technology for data collection and analysis, and developing quality
control procedures for insuring consistency and rigor in activities conducted by teams of researchers and practitioners.
(See also Ash et al., 2008, for a variation on RAP.)
The template or framework approach mentioned earlier is well-suited to RAP as it provides a ready-made structure forthe inquiry based upon key constructs identified in advance (Padgett, 2016). Pioneered by Crabtree and Miller (1999)
and Miles and Huberman (1994), the template approach was used by Hamilton (2013) to quickly abstract urgently
needed data from a broader, deeper data pool following a predetermined template. More specification of the templateapproach and its utility is needed in future implementation science. See Section VI.
Third, it is important that qualitative methodologists explore methods of qualitative data collection and analysis notcommonly used in implementation science (Jennings, Sandelowski, & Higgins, 2013). These include phenomenologicalmethods to understand the lived experience (van Manen, 2015) of implementation. As well, elicitation techniques suchas pile sorts, photo elicitation interviews, semantic frames, and rank order tasks represent more structured, quasistatistical ways to assess stakeholder perceptions of implementation goals, requirements, barriers, facilitators, andoutcomes.
Fourth, there is a need to contribute to the development of a common language when conducting qualitative research
in implementation science. Semantic consistency is needed for clear communication and meaningful comparisons,especially in a multidisciplinary field such as implementation science. Work on pragmatic, consistent, intelligible
terminology to describe qualitative methods in implementation science is much needed (Colquhohn et al., 2014;
Rabin et al., 2008). For example, qualitative reports in implementation science should be transparent and explicit indescribing the methods used, e.g., clarifying how grounded theory informed one’s analytic approach. They should also
describe what and when strategies for rigor were used to enhance trustworthiness (Padgett, 2016).

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

16Fifth, a field that is as context-dependent as implementation science needs to develop meaningful approaches forcross-context comparison and synthesis of qualitative data. Too much emphasis on standardization can be counter tothe intent and value of qualitative inquiry. Still, a theoretically sound and demonstrably useful approach to contextualcomparisons of qualitative data for the specific purpose of informing implementation science is much needed.
There are currently several methods that synthesize and compare qualitative research findings such as meta-synthesis,
meta-ethnography, qualitative synthesis, and others. Leeman et al. (2010) used realist synthesis methods to identify
and summarize context and process related evidence that might advance understanding of factors that influence theimplementation of antiretroviral therapy adherence interventions. All contend with the challenge that qualitative data
is inherently contextual and that qualitative methodologists do not always agree on the uniform standards of qualitythat such syntheses necessitate (Padgett, 2016). This tension between maintaining contextualization and diversityof approach while trying to synthesize findings across very different implementation sites and qualitative studies
represents an ongoing challenge in implementation science (see Benzer et al., 2013, on between and within sitevariation). The emphasis by some researchers on technology of synthesis or meta-analysis at the expense of attention
to epistemology and conceptualization has been cautioned against by several authors (Sandelowski et al., 2006;
Bondas & Hall, 2007; Finfgeld-Connett, 2016; Ludvigsen et al., 2016; Thorne, 2017). The use of these approaches
in implementation science are important to increase the utility of qualitative studies, yet as the authors cited above
have noted, this must be based on theoretically sound and solid methodological research. Certainly, there is muchmethodological work to be done.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

17

IX . CONCLUSION
Rigorous qualitative methods are essential to the field of implementation science. Many implementation researchers
have limited or no experience in qualitative methods, despite the wide use of these methods in the field. Furthermore,
many qualitative experts have little to no knowledge of the sphere and unique challenges of implementation science,
despite the centrality of qualitative research in implementation science. Hence, the QUALRIS group has offered its
collective knowledge and experience to describe the value of qualitative methods, methodological considerations,typical approaches to the use of qualitative methods, and needs and opportunities for innovation.
This manuscript has notable limitations. The authors have tried to communicate at a level accessible to researchers
new to qualitative approaches, avoiding jargon and advanced methodological concepts. The paper is not intended
to be a comprehensive rendering of the difficulties and dilemmas encountered in the conduct of qualitative inquiry
in implementation science. It does not systematically review the multitude of studies that have encountered andattempted to address the challenges posed. Another important limitation is the paper’s tendency to focus on
implementation science conducted within the United States. This likely reflects the research emphases of the authors
and limits to the group’s time and resources, which prohibited a broader international scope. The authors urge otherresearchers to expand this focus to a global exploration of how qualitative methods are used in implementation
science, the difficulties encountered, and solutions that have been generated.
There is a pressing need for methodological innovations to meet the challenges for the rigorous use of qualitative
methods in implementation science. There is also great opportunity to advance both qualitative methodology andimplementation science in pursuing such innovations. This can be facilitated by an implementation science-wide
conversation about these topics. We have offered here some perspectives on the value and use of qualitative methods
in implementation science. We invite the larger research community to contribute to the dialogue.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

18

X . ACKNOWLEDGEMENT S
The members of the QUALRIS group would like to sincerely thank our group of advisors (listed below) for their
exceptional contributions to this manuscript. They read several iterations of this paper, giving generously of theirtime, intellect, and experience. Our advisors’ thoughtful comments and discussion helped us to greatly improve our
final product.
Ross Brownson, Ph.D.
Brown School
School of Medicine
Washington University in St. LouisBryan J. Weiner, Ph.D.
Department of Global Health
Department of Health Services
University of Washington

Cam Escoffery, Ph.D., M.P.H.
Rollins School of Public Health
Emory University

Anne Sales, Ph.D., R.N., M.S.N.
University of Michigan Medical School
VA Ann Arbor Healthcare System

Russell E. Glasgow, Ph.D.
School of Medicine
University of Colorado
We would also like to thank David Chambers who first gave his support for the idea of this effort. We thank him forhis guidance and forbearance through the non-linear writing process.
We wish to thank Kaelin Rapport and Jordan Tompkins for their assistance with the literature review andsubstantive consultations.
Finally, thanks go to NCI’s Implementation Science Team for their feedback and support through the production ofthis paper.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

19

REFERENCES
Aakhus, E., Granlund, I., Oxman, A. D., & Flottorp, S. A. (2015). Tailoring interventions to implement recommendations for the treatment of elderly
patients with depression: A qualitative study. International Journal of Mental Health Systems, 9(1), 36.
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service
sectors. Administration and Policy in Mental Health, 38, 4-23.
Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration
and Policy in Mental Health and Mental Health Services Research, 34, 411-419.
Abbott, P. A., Foster, J., Marin, H., de F., & Dykes, P. C. (2014). Complexity and the science of implementation in health IT—Knowledge gaps andfuture visions. International Journal of Medical Informatics, 83(7), e12-22.
Asbury, J. E. (1995). Overview of focus group research. Qualitative Health Research 5(4), 414-420.
Ash, J. S., Sittig, D. F., McMullen, C. K., Guappone, K., Dykstra, R., & Carpenter, J. (2008). A rapid assessment process for clinical informatics
interventions. AMIA Annual Symposium Proceedings, 26-30.
Azar, J., Adams, N., & Boustani, M. (2015). The Indiana University Center for Healthcare Innovation and Implementation Science: Bridging
healthcare research and delivery to build a learning healthcare system. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen,
109(2), 138-143.
Bajunirwe, F., Tumwebaze, F., Abongomera, G., Akakimpa, D., Kityo, C., & Mugyenyi, P. N. (2016). Identification of gaps for implementation sciencein the HIV prevention, care and treatment cascade; A qualitative study in 19 districts in Uganda. BMC Research Notes, 9, 217.
Bate, P., Mendel, P., & Robert, G. (2008). Organizing for quality: The improvement journeys of leading hospitals in Europe and the United States.
NY: Radcliffe Publishing.
Beebe, J. (2001). Rapid assessment process: An introduction. Walnut Creek, CA: Altamira Press.
Beidas, R. S., Wolk, C. L., Walsh, L. M., Evans, A. C., Hurford, M. O., & Barg, F. K. (2014). A complementary marriage of perspectives:
Understanding organizational social context using mixed methods. Implementation Science, 9(1), 1-15, 175.
Bekelman, D. B., Borsika, R. A., Nowels, C. T., Sahay, A., Heidenreich, P. A., Fischer, S. M., & Main, D. S. (2016). Barriers and facilitators to scaling
up outpatient palliative care. Journal of Palliative Medicine, 19(4), 456-459. doi:10.1089/jpm.2015.0280
Benzer, J. K., Beehler, S., Cramer, I. E., Mohr, D. C., Charns, M. P., & Burgess, J. F. (2013). Between and within-site variation in qualitative
implementation research. Implementation Science, 8, 4.
Bishop, F. L. (2015). Using mixed methods research designs in health psychology: An illustrated discussion from a pragmatist perspective. British
Journal of Health Psychology, 20(1), 5-20.
Blasinsky, M., Goldman, H. H., & Unützer, J. (2006). Project IMPACT: A report on barriers and facilitators to sustainability. Administration and Policyin Mental Health and Mental Health Services, 33, 718-729.
Bondas, T., & Hall, E. O. C. (2007). Challenges in approaching metasynthesis research. Qualitative Health Research, 17(1), 113-121.
Bradley, E. H., Curry, L. A., & Devers, K. J. (2007). Qualitative data analysis for health services research: Developing taxonomy, themes and theory.
Health Services Research, 42, 1758-1772.
Brear, M. R., Shabangu, P. N., Fisher, J. R., Hammarberg, K., Keleher, H. M., & Livingstone, C. (2018, April 1). Health capability deprivations ina rural Swazi community: Understanding complexity with theoretically informed, qualitatively driven, mixed-method design, participatory action
research. Qualitative Health Research. doi: 10.1177/1049732318768236
Brives, C., Le Marcis, F., & Sanabria, E. (2016). What’s in a context? Tenses and tensions in evidence-based medicine. Medical Anthropology,
35(5), 369-376.
Brownson, R. C., Colditz, G. A., & Proctor, E. K. (2018) Dissemination and implementation research in health: Translating science to practice (2nded.) NY: Oxford University Press.
Brownson, R. C., Dreisinger, M., Colditz, G. A., & Proctor, E. K. (2012). The path forward in dissemination and implementation research. New York,
NY: Oxford University Press.
Bunger, A. C., Powell, B. J., Robertson, H. A., MacDowell, H. A., Birken, S. A., & Shea, C. (2017). Tracking implementation strategies: A descriptionof a practical approach and early findings. Health Research Policy and Systems, 15(1), 15.
Cadogan, S. L., McHugh, S. M., Bradley, C. P., Browne, J. P., & Cahill, M. R. (2016). General practitioner views on the determinants of test ordering:
A theory-based qualitative approach to the development of an intervention to improve immunoglobulin requests in primary care. Implementation
Science, 11(1), 102.
Carey, M. A., & Smith, M. W. (1994). Capturing the group effect in focus groups: A special concern in analysis. Qualitative Health Research, 4(1),123-127.
Casillas, A., Zingg, W., Clack, L., Jantarada, F., da L., Touveneau, S., & Sax, H. (2013). Infection control through in depth eyes: A thematic study of
infection control practices across six European hospitals. Poster from 23rd European Congress of Clinical Microbiology. Berlin, Germany.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

20REFERENCES
Chalmers, I., & Glasziou, P. (2009). Avoidable waste in the production and reporting of research evidence. Lancet, 374(9683), 86-89.
Chambers, D. A. (2014). Guiding theory for dissemination and implementation research: A reflection on models used in research and practice. In
R. S. Beidas & P.C. Kendall (Eds.), Child and Adolescent Therapy: Dissemination and Implementation of Empirically Supported Treatments. New
York: Oxford University Press.
Chambers, D. A., Feero, W. G., & Khoury, M. J. (2016). Convergence of implementation science, precision medicine, and the learning health caresystem: A new model for biomedical research. JAMA, 315(18), 1941-1942.
Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013). The dynamic sustainability framework: Addressing the paradox of sustainment amid
ongoing change. Implementation Science, 8, 117.
Chambers, D. A., & Norton, W. E. (2016). The adaptome: Advancing the science of intervention adaptation. American Journal of Preventive
Medicine, 51(4 Suppl 2), S124-131.
Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis. London, UK: Sage.
Cifuentes, E., Alamo, U., Kendall, T., Brunkard, J., & Scrimshaw, S. (2006). Rapid assessment procedures in environmental sanitation research: A
case study from the northern border of Mexico. Canadian Journal of Public Health, 97(1), 24-28.
Clack, L., Casillas, A., Touveneau, S., Jantarada, F. da L., Zingg, W., & Sax, H. (2013). Factors influencing leadership dynamics in the context
of infection control: A cross-case analysis of six European hospitals. 19ème Journée Romande d’Hygiène Hospitalière (19th Romandy Day of
Hospital Hygiene). Montreux, Switzerland.
Cohen, D. J., Balasubramanian, B. A., Isaacson, N. F., Clark, E. C., Etz, R. S., & Crabtree, B. F. (2011). Coordination of health behavior counselingin primary care. Annals of Family Medicine, 9(5), 406-415.
Cohen, D. J., Balasubramanian, B. A., Gordon, L., Marino, M., Ono. S., Solberg, L. I., & Cresswell, J. (2016). A national evaluation of adissemination and implementation initiative to enhance primary care practice capacity and improve cardiovascular disease care: The ESCALATES
study protocol. Implementation Science, 11, 86.
Cohen, D. J. & Crabtree, B. F. (2008). Evaluative criteria for qualitative research in health care: Controversies and recommendations. Annals of
Family Medicine, 6(4), 331-339.
Cohen, D. J., Crabtree, B. F., Etz, R. S., Balasubramanian, B. A., Donahue, K., Leviton, L. C., . . . Green, L. W. (2008). Fidelity vs. flexibility:
Translating evidence-based research into practice. American Journal of Preventive Medicine, 35(5 Suppl), S381-389.
Colon-Emeric, C., Toles, M., Cary, M. P., Batchelor-Murphy, M., Yap, T., Song, Y., . . . Anderson, R. A. (2016). Sustaining complex interventions inlong-term care: a qualitative study of direct care staff and managers. Implement Science, 11, 94.
Colquhoun, H., Leeman, J., Michie, S., Lokker, C., Bragge, P., Hempel, S., & Grimshaw, J. (2014). Towards a common terminology: A simplified
framework of interventions to promote and integrate evidence into health practices, systems, and policies. Implementation Science, 9, 51.
Concannon, T. W., Meissner, P., Grunbaum, J. A., McElwee, N., Guise, J. M., Santa, J., . . . Leslie, L. K. (2012). A new taxonomy for stakeholder
engagement in patient-centered outcomes research. Journal of General Internal Medicine, 27, 985–991.
Corbin, J. M. & Strauss, A. (2008). Basics of qualitative research: Techniques and procedures for developing grounded theory (3rd ed.). Thousand
Oaks, CA: Sage Publications.
Crabtree, B. F. & Miller, W. (Eds.). (1999). Doing qualitative research (2nd ed.). London, UK: Sage.
Crabtree, B. F., Miller, W., Addison, R., Gilchrist, V. & Kuzel, A. (Eds.). (1994). Exploring collaborative research. Thousand Oaks, CA: Sage.
Creswell, J. W. (2015). A concise introduction to mixed-methods research. Thousand Oaks, CA: Sage Publications.
Creswell, J. W., Klassen, A. C., Clark, V. L. P., & Smith, K. C. for the Office of Behavioral and Social Sciences Research, National Institutes of
Health. (2011). Best practices for mixed methods research in the health sciences. Retrieved March 26, 2018 from https://obssr.od.nih.gov/training/
mixed-methods-research/
Creswell, J. W., & Clark, V. L. P. (2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.
Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012). Effectiveness-implementation hybrid designs: Combining elements ofclinical effectiveness and implementation research to enhance public health impact. Medical Care, 50(3), 217-226.
Dalkin, S. M., Greenhalgh, J., Jones, D., Cunningham, B., & Lhussier, M. (2015). What’s in a mechanism? Development of a key concept in realist
evaluation. Implementation Science, 10(1), 49.
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services
research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50.
Damschroder, L. J., Goodrich, D. E., Robinson, C. H., Fletcher, C. E., & Lowery, J. C. (2011). A systematic exploration of differences in contextual
factors related to implementing the MOVE! weight management program in VA: A mixed methods study. BMC Health Services Research, 11(1), 248.
Damschroder, L. J., & Lowery, J. C. (2013). Evaluation of a large-scale weight management program using the consolidated framework for
implementation research (CFIR). Implementation Science, 8, 51.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

21

REFERENCES
Damschroder, L. J., Reardon, C. M., AuYoung, M., Moin, T., Datta, S. K., Sparks, J. B., . . . Richardson, C. R. (2017). Implementation findings
from a hybrid III implementation-effectiveness trial of the Diabetes Prevention Program (DPP) in the Veterans Health Administration (VHA).
Implementation Science, 12(1), 94.
Damschroder, L. J., Reardon, C. M., Sperber, N., Robinson, C. H., Fickel, J. J., & Oddone, E. Z. (2017). Implementation evaluation of the TelephoneLifestyle Coaching (TLC) program: Organizational factors associated with successful implementation. Translational Behavioral Medicine, 7(2), 233-241.
Davidoff, F., Dixon-Woods, M., Leviton, L., & Michie, S. (2015). Demystifying theory and its use in improvement. BMJ Quality & Safe, 24(3), 228-238.
Davies, P., Walker, A., & Grimshaw, J. (2010). A systematic review of the use of theory in the design and implementation strategies andinterpretation of the results of rigorous evaluations. Implementation Science, 5.
DiCicco-Bloom, B., & Crabtree, B. F. (2006). The qualitative research interview. Medical Education, 40(4), 314-321.
Dopson, S., & Fitzgerald, L. (2006). The active role of context. In S. Dopson & L. Fitzgerald (Eds.), Knowledge to action? Evidence-based healthcare in context (p. 223). Oxford, UK: Oxford University Press.
DuBois, L. Z., Macapagal, K. R., Rivera, Z., Prescott, T. L., Ybarra, M. L., & Mustanski, B. (2015). To have sex or not to have sex? An online focus
group study of sexual decision making among sexually experienced and inexperienced gay and bisexual adolescent men. Archives of Sexual
Behavior, 44(7), 2027-2040.
Eccles, M., Grimshaw, J., Walker, A., Johnston, M., & Pitts, N. (2005). Changing the behavior of healthcare professionals: The use of theory inpromoting the uptake of research findings. Journal of Clinical Epidemiology, 58, 107-112.
Elsey, H., Khanal, S., Manandhar, S., Sah, D., Baral, S. C., Siddiqi, K., & Newell, J. N. (2016). Understanding implementation and feasibility of
tobacco cessation in routine primary care in Nepal: A mixed methods study. Implementation Science, 11, 104.
Escoffery, C., Glanz, K., Hall, D., & Elliott, T. (2009). A multi-method process evaluation of a skin cancer prevention diffusion study. Evaluation & the
Health Professions, 32(2), 184-203.
Fernald, D. H., & Duclos, C. W. (2005). Enhance your team-based qualitative research. Annals of Family Medicine, 3(4), 360-364.
Finfgeld-Connett, D. (2016). The future of theory-generating meta-synthesis research. Qualitative Health Research, 26(3), 291–293.
Forman, J., Heisler, M., Damschroder, L. J., Kaselitz, E., & Kerr, E. A. (2017). Development and application of the RE-AIM QuEST mixed methods
framework for program evaluation. Preventive Medicine Report, 6, 322-328.
Fox, A. B., Hamilton, A. B., Frayne, S. N., Wiltsey-Stirman, S., Bean-Mayberry, B., Carney, D., . . . Vogt, D. (2016). Effectiveness of an evidencebased quality improvement approach to cultural competence training: The Veterans Affairs’ “Caring for Women Veterans” program. Journal for
Continuing Education in the Health Professions, 36, 96-103.
Foy, R., Ovretveit, J., Shekelle, P. G., Pronovost, P. J., Taylor, S. L., Dy, S., . . . Wachter, R. M. (2011). The role of theory in research to develop andevaluate the implementation of patient safety practices. BMJ Quality & Safety, 20(5), 453-459.
Francis, J. J., Johnston, M., Robertson, C., Glidewell, L., Entwistle, V., Eccles, M. P., & Grimshaw, J. M. (2010). What is an adequate sample size?
Operationalizing data saturation for theory-based interview studies. Psychology & Health, 25(1), 1229-1245.
Frankel, R. (1999). Standards of qualitative research. In B. F. Crabtree & W. L. Miller (eds.), Doing qualitative research (2nd ed.) (pp. 333-346).
Thousand Oaks, CA: Sage.
Freedman, D. A., Whiteside, Y. O., Brandt, H. M., Young, V., Friedman, D. B., & Hebert, J. R. (2012). Assessing readiness for establishing a farmers’market at a community health center. Journal of Community Health, 37(1), 80-88.
Gale, N. K., Heath, G., Cameron, E., Rashid, S., & Redwood, S. (2013). Using the framework method for the analysis of qualitative data in multidisciplinary health research. BMC Medical Research Methodology, 13, 117.
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. New York: Aldine de Gruyter.
Glasgow, R. E. (2008). What types of evidence are most needed to advance behavioral medicine? Annals of Behavioral Medicine, 35(1), 19-25.
Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003). Why don’t we see more translation of health promotion research to practice? Rethinking
the efficacy-to-effectiveness transition. American Journal of Public Health, 93(8), 1261-1267.
Glasgow, R. E., Vinson, C., Chambers, D., Khoury, M. J., Kaplan, R. M., & Hunter, C. (2012). National Institutes of Health approaches todissemination and implementation science: Current and future directions. American Journal of Public Health 102(7), 1274-1281.
Grol, R., & Wensing, M. (2004). What drives change? Barriers to and incentives for achieving evidence-based practice. Medical Journal ofAustralia, 180(6 Suppl), S57–60.
Guest, G., & MacQueen, K. (2008). Handbook for team-based qualitative research. Lanham, MD: Altamira Press.
Guetterman, T. C., Fetters, M. D., & Creswell, J. W. (2015). Integrating quantitative and qualitative results in health science mixed methods researchthrough joint displays. Annals of Family Medicine, 13(6), 554-561.
Hadi, M. A., & Closs, S. J. (2016). Applications of mixed-methods methodology in clinical pharmacy research. International Journal of ClinicalPharmacy, 38(3), 635-640.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

22

REFERENCES
Hamilton, A. B. (2013, December 11). Qualitative methods in rapid turn-around health services research. Retrieved from https://www.hsrd.
research.va.gov/for_researchers/cyber_seminars/archives/780-notes.pdf
Hamilton, A. B., Cohen, D., Heurtin-Roberts, S., Padgett, D., Palinkas, L., & Rabin, B. (2015, December). The value of qualitative methods in
implementation research. Oral presentation from the 8th Annual Conference on the Science of Dissemination and Implementation in Health.
Washington, DC.
Hamilton, A. B., Mittman, B. S., Campbell, D., Hutchinson, C., Liu, H., Moss, N. J., & Wyatt, G. E. (2018). Understanding the impact of external
context on community-based implementation of an evidence-based HIV risk reduction intervention. BMC Health Services Research, 18(1), 11.
Hamilton, A. B., Brunner, J., Cain, C., Chuang, E., Luger, T., Canelo, I., Rubenstein, L., & Yano, E. M. (2017). Engaging multilevel stakeholders in an
implementation trial of evidence-based quality improvement in VA women’s health primary care. Translational Behavioral Medicine, 7(3), 478-485.
Hamilton, A. B., Cohen, A. N., Glover, D. L., Whelan, F., Chemerinski, E., McNagny K. P., . . . Young, A. S. (2013). Implementation of evidencebased employment services in specialty mental health. Health Services Research, 48(6 Pt 2), 2224-2244.
Hannes, K. (2011). Critical appraisal of qualitative research. In J. Noyes, A. Booth, K. Hannes, A. Harden, J. Harris, S. Lewin, & C. Lockwood(Eds.), Supplementary guidance for inclusion of qualitative research in Cochrane systematic reviews of interventions. Retrieved from http://cqrmg.
cochrane.org/supplemental-handbook-guidanceHarvey, G., & Kitson, A. (2015). Implementing evidence-based practice in healthcare: A facilitation guide. New York, NY: Routledge.
Harvey, G., & Kitson, A. (2016). PARIHS revisited: From heuristic to integrated framework for the successful implementation of knowledge into
practice. Implementation Science, 11, 33.
Helfrich, C. D., Weiner, B. J., McKinney, M. M., & Minasian, L. (2007). Determinants of implementation effectiveness: Adapting a framework forcomplex innovations. Medical Care Research and Review, 64(3), 279-303.
Hesse-Biber, S. (2016). Doing interdisciplinary mixed methods health care research: Working the boundaries, tensions, and synergistic potential of
team-based research. Qualitative Health Research, 26(5), 649-658.
American Psychological Association. (2012). Consensual qualitative research: A practical resource for investigating social science phenomena.
Washington, DC: Hill, C. E.
Hoagwood, K. E., Vogel, J. M., Levitt, J. M., D’Amico, P. J., Paisner, W. I., & Kaplan, S. J. (2007). Implementing an evidence-based trauma treatment
in a state system after September 11: the CATS Project. Journal of the American Academy of Child Adolescent Psychiatry, 46(6), 773-779.
Howard, B. N., Van Dorn, R., Myers, B. J., Zule, W. A., Browne, F. A., Carney, T., & Wechsberg, W. M. (2017). Barriers and facilitators to
implementing an evidence-based woman-focused intervention in South African health services. BMC Health Services Research, 17(1), 746.
Hsieh, H. F., & Shannon, S. E. (2009). Three approaches to qualitative content analysis. Qualitative Health Research, 15, 1277-1288.
Ivankova, N. V. (2015). Mixed methods applications in action research. Thousand Oaks, CA: Sage Publications.
Jennings, B.M., Sandelowski, M., & Higgins, M.K. (2013). Turning over patient turnover: An ethnographic study of admissions, discharges, andtransfers. Research in Nursing & Health, 36, 554-566.
Kane, H., Lewis, M. A., Williams, P. A., & Kahwati, L. C. (2014). Using qualitative comparative analysis to understand and quantify translation andimplementation. Translational Behavioral Medicine, 4(2), 201-208.
Kane, J. C., Adaku, A., Nakku, J., Odokonyero, R., Okello, J., Musisi, S., & Toi, W. A. (2016). Challenges for the implementation of World HealthOrganization guidelines for acute stress, PTSD, and bereavement: A qualitative study. Implementation Science, 11, 36.
Kazdin, A. (2007). Mediators and mechanisms of change in psychotherapy research. Annual Review Clinical Psychology, 3(1), 1-27.
Kennedy, A., Vassilev. I., Jame, E., & Rogers, A. (2016). Implementing a social network intervention designed to enhance and diversify support forpeople with long-term conditions. A qualitative study. Implementation Science, 11, 27.
Kessler, R. S., Purcell, E. P., Glasgow, R. E., Klesges, L. M., Benkeser, R. M., & Peek, C. J. (2013). What does it mean to “employ” the RE-AIM
model? Evaluation & the Health Professions, 36(1), 44-66.
Kirk, M. A., Kelley, C., Yankey, N., Birken, S. A., Abadie. B., & Damschroder, L. J. (2016) A systematic review of the use of the Consolidated
Framework for Implementation Research. Implementation Science, 11(1), 1-13.
Kitson, A., Brook, A., Harvey, G., Jordan, Z., Marshal, R., O’Shea, R., & Wilson, D. (2017). Using complexity and network concepts to inform
healthcare knowledge translation. International Journal of Health Policy and Management, 6, 1-13.
Kitson, A., Rycroft-Malone, J., Harvey, G., McCormack, B., Seers, K., & Titchen, A. (2008). Evaluating the successful implementation of evidence
into practice using the PARiHS framework: Theoretical and practical challenges. Implementation Science, 3, 1.
Kitzinger, J. (2006). Focus groups. In C. Pope & N. Mays (Eds.), Qualitative research in health care (3rd ed.) (pp. 21-31). Malden, MA: Blackwell.
Klein, K. J., Conn, A. B., & Sorra, J. S. (2001). Implementing computerized technology: An organizational analysis. Journal of Applied Psychology,
86(5), 811-824.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

23

REFERENCES
Klein, K. J., & Sorra, J. S. (1996). The challenge of innovation implementation. The Academy of Management Review, 21(4), 1055-1080.
Koenig, C. J., Abraham, T., Zamora, K. A., Hill, C., Kelly. P. A., Uddo, M., Hamilton, M., Pyne, J. M., & Seal, K. H. (2016). Pre-implementation
strategies to adapt and implement a veteran peer coaching intervention to improve mental health treatment engagement among rural veterans.
Journal of Rural Health, 32(4), 418-428.
Kothari, A., Rudman, D., Dobbins, M., Rouse, M., Sibbald, S., & Edwards, N. (2012). The use of tacit and explicit knowledge in public health: aqualitative study. Implementation Science, 7, 20
Kowitt, S., Woods-Jaeger, B., Lomas, J., Taggart, T., Thayer, L., Sutton, S., & Lightfoot, A. F. (2015). Using Photovoice to understand barriers to andfacilitators of cardiovascular health among African American adults and adolescents, North Carolina, 2011-2012. Preventing Chronic Disease, 12, E164.
Krueger, R. A., & Casey, M. A. (1994). Focus groups: A practical guide for applied research (2nd ed.). Thousand Oaks, CA: Sage.
Leeman, J., Chang, Y. K., Lee, E. J., Voils, C., Crandell, J. L., & Sandelowski, M. (2010). Implementation of antiretroviral therapy adherence
interventions: A realist synthesis of evidence. Journal of Advanced Nursing, 66, 1915-1930.
Leeman, J., Myers, A., Grant, J. C., Wangen, M., & Queen, T. L. (2017). Implementation strategies to promote community-engaged efforts tocounter tobacco marketing at the point of sale. Translational Behavioral Medicine, 7, 405-414.
Leeman, J., Teal, R., Jernigan, J., Reed, J. H., Farris, R., & Ammerman, A. (2014). What evidence and support do state-level public healthpractitioners need to address obesity prevention. American Journal of Health Promotion, 28(3), 189-196.
Lessard, S., Barei, C., Lalonde, L., Duhamel, F., Hudon, E., Goudreau, J., & Levesque, L. (2016). External facilitators and interprofessional
facilitation teams: A qualitative study of their roles in supporting practice change. Implementation Science, 11, 97.
Lewis, C. C., Klasnja, P., Powell, B. J., Lyon, A. R., Tuzzio. L., Jones, S., Walsh-Bailey, C., & Weiner, B. (2018). From classification to causality:
Advancing understanding of mechanisms of change in implementation science. Frontiers in Public Health, 6(136).
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic Inquiry. Newbury Park, CA: Sage Publications.
Lord, S., Moore, S. K., Ramsey, A., Dinauer, S., & Johnson, K. (2016). Implementation of a substance use recovery support mobile phone app incommunity settings: Qualitative study of clinician and staff perspectives of facilitators and barriers. JMIR Mental Health, 3(2).
Ludvigsen, M. S., Hall, E. O. C., Meyer, G., Fegran, L., Aagaard, H., & Uhrenfeldt, L. (2016) Using Sandelowski and Barroso’s meta-synthesis
method in advancing qualitative evidence. Qualitative Health Research, 26(3), 320–329.
Lukas, C. V., Holmes, S. K., Cohen, A. B., Restuccia. J., Cramer, I. E., Shwartz, M., & Charns, M. P. (2007). Transformational change in health caresystems: An organizational model. Health Care Management Review, 32(4), 309-320.
Marshall, T., Rapp, C. A., Becker, D. R., & Bond, G. R. (2008). Key factors for implementing supported employment. Psychiatric Services, 59, 886-892.
Martinez, R. N., Hogan, T. P., Lones, K., Balbale, S., Scholten, J., Bidelspach, D., & Smith, B. M. (2016). Evaluation and treatment of mild traumaticbrain injury through the implementation of clinical video telehealth: Provider perspectives from the Veterans Health Administration. Physical
Medicine and Rehabilitation Journal, 9, 231-240
Marty, D., Rapp, C., McHugo, G., & Whitley, R. (2008). Factors influencing consumer outcome monitoring in implementation of evidence-based
practices: results from the National EBP Implementation Project. Administration and Policy in Mental Health, 35, 204-211.
May, C., & Finch, T. (2009). Implementation, embedding and integration: An outline of normalization process theory. Sociology, 43, 535-554.
May, C. (2013). Towards a general theory of implementation. Implementation Science, 8, 18.
May, C. (2006). A rational model for assessing and evaluating complex interventions in health care. BMC Health Services Research, 6, 86.
May, C., Mair, F., Finch, T., MacFarlane, A., Dowrick, C., Treweek, S., . . . Montori, V. M. (2009). Development of a theory of implementation andintegration: Normalization Process Theory. Implementation Science, 4, 29.
Mays, N., & Pope, C. (1995). Rigour and qualitative research. BMJ, 311(6997): 109-112.
McCambridge, J., Witton, J., & Elbourne, D. R. (2014). Systematic review of the Hawthorne effect: New concepts are needed to study research
participation effects. Journal of Clinical Epidemiology, 67(3), 267-277.
Mealer, M., & Jones, J. (2014). Methodological and ethical issues related to qualitative telephone interviews on sensitive topics. Nurse Researcher,21(4), 32-37.
Meyer, J. (2000). Using qualitative methods in health-related action research. BMJ, 320, 178-181.
Michie, S., Atkins, L., & West, R. (2014). The behavior change wheel: A guide to designing interventions. London, UK: Silverback Publishing.
Miles, M. B., & Hubermann, M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks, CA: Sage Publications.
Moeckli, J., Cram, P., Cunningham, C., & Reisinger, H. S. (2013). Staff acceptance of a telemedicine intensive care unit program: A qualitative
study. Journal of Critical Care, 28(6), 890-901.
Morgan, D. L. (1997). Focus groups as qualitative research. Newbury Park, CA: Sage.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

24

REFERENCES
Mozaffar, H., Cresswell, K. M., Lee, L., Williams, R., & Sheikh, A. (2016). Taxonomy of delays in the implementation of hospital computerized
physician order entry and clinical decision support systems for prescribing: A longitudinal qualitative study. BMC Medical Informatics and Decision
Making, 16, 25.
National Institutes of Health. (2018). PARs Dissemination and Implementation Research in Health for R01 (Clinical Trial Optional; PAR-18-007), R21(Clinical Trial Optional; PAR-17-017), R03 (PAR-16-237). Retrieved July 5, 2018 from https://cancercontrol.cancer.gov/funding_apply.html#is
Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation Science, 10, 53.
Nilsson, G., Hansson, K., Tiberg, I., & Hallström, I. (2018). How dislocation and professional anxiety influence readiness for change during theimplementation of hospital-based home care for children newly diagnosed with diabetes—An ethnographic analysis of the logic of workplacechange. BMC Health Services Research, 18(1), 61.
Nordmark, S., Zingmark, K., & Lindberg, I. (2016). Process evaluation of discharge planning implementation in healthcare using normalization
process theory. BMC Medical Informatics and Decision Making, 16, 48.
Nutting, P. A., Crabtree, B. F., Miller, W. L., Stewart, E. E., Stange, K. C., & Jaén, C. R. (2010). Journey to the patient-centered medical home: aqualitative analysis of the experiences of practices in the National Demonstration Project. Annals of Family Medicine, 2010(8 Suppl 1), S45-56,
S92. Erratum in Annals of Family Medicine, 8(4), 369.
Padgett, D. (2016). Qualitative methods in social work research (3rd ed.). Thousand Oaks, CA: Sage.
Padwa, H., Teruya, C., Tran, E., Lovinger, K., Antonini, V. P., Overholt, C. C., & Urada, D. (2016). The implementation of integrated behavioral healthprotocols in primary care settings in project care. Journal of Substance Abuse Treatment, 62, 74-83.
Palinkas, L. A. (2014). Qualitative and mixed methods in mental health services and implementation research. Journal of Clinical Child andAdolescent Psychology, 43(6), 851-861. doi: 10.1080/15374416.2014.910791
Palinkas, L. A., & Aarons, G. A. (2009). A view from the top: Executive and management challenges in a statewide implementation of an evidencebased practice to reduce child neglect. International Journal of Child Health and Human Development, 2, 47-55.
Palinkas, L. A.., Aarons, G. A., Horwitz, S. M., Chamberlain, P., Hurlburt, M., & Landsverk, J. (2011). Mixed method designs in implementation
research. Administration and Policy in Mental Health, 38, 44-53.
Palinkas, L. A., Horwitz, S. M., Green, C. A., Wisdom, J. P., Duan, N., & Hoagwood, K. (2015). Purposeful sampling for qualitative data collection
and analysis in mixed method implementation research. Administration and Policy in Mental Health, 42(5), 533-544.
Palinkas, L. A., Schoenwald, S. K., Hoagwood, K., Landsverk, J., Chorpita, B. R., Weisz, J. R., & MacArthur Research Network on Youth Mental
Health. (2008). An ethnographic study of implementation of evidence-based treatments in child mental health: First steps. Psychiatric Services,59(7), 738-746.
Palinkas, L. A., Spear, S. E., Mendon, S. J., Villamar, J., Valente, T., Chou, C. P., . . . Brown, C. H. (2016). Measuring sustainment of prevention
programs and initiatives. Implementation Science, 11, 95.
Palinkas, L. A., Weisz, J. R., Chorpita, B. R., Levine, B., Garland, A.F., Hoagwood, K. E., & Landsverk, J. (2013). Continued use of evidence-based
treatments after a randomized controlled effectiveness trial: A qualitative study. Psychiatric Services, 64(11), 1110-1118.
Parkinson, S., Eatough, V., Holmes, J., Stapley, E., & Midgley, N. (2016). Framework analysis: a worked example of a study exploring young
people’s experiences of depression. Qualitative Research in Psychology, 13(2), 109-129.
Pelcastre-Villafuerte, B., Ruiz, M., Meneses, S., Amaya, C., Márquez, M., Taboada, A., & Careaga, K. (2014). Community-based health care forindigenous women in Mexico: A qualitative evaluation. International Journal for Equity in Health, 13(1), 2.
Pope, C., Mays, N., & Popay, J. (2006). How can we synthesize qualitative and quantitative evidence for healthcare policy-makers and managers?
Healthcare Management Forum, 19(1), 27-31.
Powell, B. J., Beidas, R. S., Lewis, C. C., Aarons, G. A., McMillen, J. C., Proctor, E. K., & Mandell, D. S. (2017) Methods to improve the selection
and tailoring of implementation strategies. Journal of Behavioral Health Services Research, 44(2), 177-194.
Powell, B. J., McMillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T., Bunger, A. C., . . . York, J. L. (2012). A compilation of strategies for
implementing clinical innovations in health and mental health. Medical Care Research and Review, 69(2), 123-157.
Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., . . . Kirchner, J. E. (2015). A refined compilation of
implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10, 21.
Proctor, E. K., Knudsen, K. J., Fedoracivius, N., Hovmand, P., Rosen, A., & Perron, B. (2007). Implementation of evidence-based practice incommunity behavioral health: Agency director perspectives. Administration and Policy in Mental Health, 34, 479-488.
Proctor, E. K., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G. A., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation
research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38, 65-76.
Rabin, B. A., Brownson, R., Haire-Joshu, D., Kreuter, M., & Weaver, N. (2008). A glossary for dissemination and implementation research in health.
Journal of Public Health Management and Practice, 14, 117-123.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

25

REFERENCES
Rabin, B. A., Lewis, C. C., Norton, W. E., Neta, G., Chambers, D., Tobin, J. N., . . . Glasgow, R. E. (2016). Measurement resources fordissemination and implementation research in health. Implementation Science, 11, 42.
Rapp, C. A., Etzel-Wise, D., Marty, D., Coffman, M., Carlson, L., Asher, D., . . . Holter M. (2010). Barriers to evidence-based practice
implementation: Results of a qualitative study. Community Mental Health Journal, 46(2), 112-118.
Rapp, C. A., Etzel-Wise, D., Marty, D., Coffman, M., Carlson, L., Asher, D., . . . Whitley, R. (2008). Evidence-based practice implementation
strategies: Results of a qualitative study. Community Mental Health Journal, 44(3), 213-224.
Reason, P., & Bradbury, H. (2008). Handbook of action research. Thousand Oaks, CA: Sage.
Reger, G. M., Browne, K. C., Campellone, T. R., Simons, C., Kuhn, E., Fortney, J., . . . Reisinger, H. S. (2017). Barriers and facilitators to mobile
application use during PTSD treatment: Clinician adoption of PE coach. Professional Psychology: Research and Practice, 48(6), 510-517.
Reichstadt, J., Depp, C. A., Palinkas, L. A., Folsom, D. P., & Jeste, D. V. (2007). Building blocks of successful aging: A focus group study of older
adults’ perceived contributors to successful aging. American Journal of Geriatric Psychiatry, 15(3), 194-201.
Ritchie, J., & Lewis, J. (2003). Qualitative research practice: A guide for social science students & researchers. Thousand Oaks, CA: Sage.
Robert Wood Johnson Foundation. (2018). Qualitative Research Guidelines Project. Retrieved from http://www.qualres.org/HomeThic-3697.html
Rychetnik, L., Frommer, M., Hawe, P., & Shiell, A. (2002). Criteria for evaluating evidence on public health interventions. Journal of Epidemiology
and Community Health, 56, 119-127.
Rycroft-Malone, J., McCormack, B., Hutchinson, A. M., DeCorby, K., Bucknall, T. K., Kent, B., . . . Wilson V. (2012). Realist synthesis: illustrating themethod for implementation research. Implementation Science, 7(1), 33.
Saldana, L. (2014). The stages of implementation completion for evidence-based practice: Protocol for a mixed methods study. Implementation
Science, 9, 43.
Sandelowski, M., Trimble, F., Woodard, E. K., & Barroso, J. (2006). From synthesis to script: Transforming qualitative research findings for use inpractice. Qualitative Health Research, 16(10), 1350-1370.
Sandelowski, M., & Leeman, J. (2012). Writing usable qualitative health research findings. Qualitative Health Research, 22(10), 1404-1413.
Sax, H., Clack, L., Touveneau, S., Jantarad, F. da L., Pittet, D., Zingg, W., & PROHIBIT study group. (2013). Implementation of infection control bestpractice in intensive care units throughout Europe: A mixed-method evaluation study. Implementation Science, 8, 24.
Schierhout, G., Hains, J., Si, D., Kennedy, C., Cox, R., Kwedza, R., . . . Bailie, R. (2013). Evaluating the effectiveness of a multifaceted, multilevel
continuous quality improvement program in primary health care: Developing a realist theory of change. Implementation Science, 8(1), 119.
Scrimshaw, S. C., Carballo, M., Ramos, L., & Blair, B. A. (1991). The AIDS Rapid Anthropological Assessment Procedures: A tool for healtheducation planning and evaluation. Health Education Quarterly, 18(1), 111-123.
Scrimshaw, S. C. M., & Hurtado, E. (1987). Rapid assessment procedures for nutrition and primary health care: Anthropological approaches toimproving programme effectiveness. Los Angeles, CA: UCLA Latin American Center.
Shea, C. M., Teal, R., Haynes-Maslow, L., McIntyre, M., Weiner, B. J., Wheeler, S. B., . . . Shea, T. C. (2014). Assessing the feasibility of a virtual
tumor board program: a case study. Journal of Healthcare Management, 59(3), 177-193.
Smithson, J. (2000). Using and analysing focus groups: Limitations and possibilities. International Journal of Social Research Methodology, 3(2), 103-119.
Sommerbakk, R., Haugen, D. F., Tjora, A., Kaasa, S., & Hjermstad, M. J. (2016). Barriers to and facilitators for implementing quality improvementsin palliative care—Results for a qualitative interview study in Norway. BMC Palliative Care, 15, 61.
Southam-Gerow, M. A., & Dorsey, S. (2014). Qualitative and mixed methods research in dissemination and implementation science: Introduction to
the special issue. Journal of Clinical Child & Adolescent Psychology, 43(6), 845-850.
Srivistava, A., & Thomson, S. B. (2009) Framework analysis: A qualitative methodology for applied policy research. Journal of Administration and
Governance, 4, 72-79.
Stewart, E. E., Nutting, P. A., Crabtree, B. F., Stange, K. C., Miller, W. L., & Jaén, C. R. (2010) Implementing the patient-centered medical home:
Observation and description of the national demonstration project. Annals of Family Medicine, 8(Suppl 1), S21-32; S92. Erratum in Annals of Family
Medicine, 8(4), 369.
Strauss, A., & Corbin, J. M. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory. Thousand Oaks,
CA: Sage.
Swain, K., Whitley, R., McHugo, G. J., & Drake, R. E. (2009). The sustainability of evidence-based practices in routine mental health agencies.
Community Mental Health Journal, 46(2), 119-129.
Tabak, R. G., Chambers, D. A., Hook, M., & Brownson, R. C. (2018). The conceptual basis for dissemination and implementation research:
Lessons from existing models and frameworks. In R. C. Brownson, G. A. Colditz, & E. K. Proctor (Eds.), Dissemination and implementation
research in health: Translating science into practice (pp. 73-88). New York, NY: Oxford University Press.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

26

REFERENCES
Tabak, R. G., Khoong, E. C., Chambers, D. A., & Brownson, R. C. (2012) Bridging research and practice: Models for dissemination andimplementation research. American Journal on Preventative Medicine, 43, 337–350.
Tausch, A. P., & Menold, N. (2016). Methodological aspects of focus groups in health research: Results of qualitative interviews with focus groupmoderators. Global Qualitative Nursing Research, 3, 1–12.
Thomas, S., Steven, I., Browning, C., Dickens, E., Eckermann, L., Carey, L., & Pollard, S. (1992). Focus groups in health research: A methodologicalreview. Annual Review of Health Social Science, 2(1), 7-20.
Thorne, S. (2017). Metasynthetic madness: What kind of monster have we created? Qualitative Health Research, 27(1), 2-12.
Tonkin-Crine, S., Anthierens, S., Hood, K., Yardley, L., Cals, J. W., Francis, N.A., . . . Little, P. (2016). Discrepancies between qualitative andquantitative evaluation of randomised controlled trial results: Achieving clarity through mixed methods triangulation. Implementation Science, 11(1) 66.
Tracy, S. J. (2010). Qualitative quality: Eight ‘big tent’ criteria for excellent qualitative research. Qualitative Inquiry, 16(10), 837-851.
Tunis, S. R., Stryer, D. B., & Clancy, C. M. (2003). Practical clinical trials: Increasing the value of clinical research for decision making in clinical andhealth policy. JAMA, 290(12), 1624-1632.
Turner, S., Morris, S., Sheringham, J., Hudson, E., & Fulop, N. J. (2016). Study protocol: Decisions in health care to introduce or diffuse innovations
using evidence (DECIDE). Implementation Science, 1, 48.
Utarini, A., Winkvist, A., & Pelto, G. H. (2001). Appraising studies in health using rapid assessment procedures (RAP): Eleven critical criteria,
Human Organization 60(4), 390-400.
van Manen, M. (2015). Researching lived experience: Human science for an action sensitive pedagogy (2nd ed). New York, NY: Routledge.
Varsi, C., Ekstedt, M., Gammon, D., & Ruland, C. M. (2015). Using the Consolidated Framework for Implementation Research to identify barriers
and facilitators for the implementation of an internet-based patient-provider communication service in five settings: A qualitative study. Journal of
Medical Internet Research, 17(11), e262.
Vogel, A. L., Stipelman, B. A., Hall, K. L., Nebeling, L., Stokols, D., & Spruijt-Metz, D. (2014). Pioneering the transdisciplinary team science
approach: Lessons learned from National Cancer Institute grantees. Journal of Translational Medicine & Epidemiology, 2(2).
Weiner, B. J., Amick, H. R., Lund, J. L., Lee, S. Y., & Hoff, T. J. (2011). Use of qualitative methods in published health services and management
research: A 10-year review. Medical Care Research and Review, 68(1), 3-33.
Whitley, R., Gingerich, S., Lutz, W. J., & Mueser, K. T. (2009). Implementing the illness management and recovery program in community mental
health settings: Facilitators and barriers. Psychiatric Services 60, 202-209.
Woltmann E. M., Whitley, R., McHugo, G. J., Brunette, M., Torrey, W. C., Coots, L., Lynde, D., & Drake, R. E. (2008). The role of staff turnover in theimplementation of evidence-based practices in health care. Psychiatric Services, 59, 732-737.
Wozniak, L., Soprovich, A., Rees, S., Al Sayah, F., Majumdar, S. R., & Johnson, J. A. (2015). Contextualizing the effectiveness of a collaborative
care model for primary care patients with diabetes and depression (Teamcare): A qualitative assessment using RE-AIM. Canadian Journal of
Diabetes, 39(Suppl 3), S83-91.
Zazzali, J. L., Sherbourne, C., Hoagwood, K. E., Greene, D., Bigley, M. F., & Sexton, T. L. (2008). The adoption and implementation of an evidence
based practice in child and family mental health services organizations: A pilot study of functional family therapy in New York State. Administration
and Policy in Mental Health, 35, 38-49.
Zhang, W., & Watanabe-Galloway, S. (2014). Using mixed methods effectively in prevention science: Designs, procedures, and examples.
Prevention Science: The Official Journal of the Society for Prevention Research, 15(5), 654-662.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E

27