November 1, 2012

Measurement Issues in 

Implementation Science

Cara C. Lewis, Ruben G. Martinez, & Kate A. Comtois
Please do not distribute slides without permission

Agenda

1. The importance of measurement in D&I

2. 10 Key Measurement Issues in D&I

3. Preliminary Results of SIRC Systematic 

Review of Instruments

4. Q & A
Importance of Measurement

• “Science is measurement”, but measurement is 

not necessarily science

• Strong measurement (i.e., good psychometrics) 

allows us to confidently interpret our findings as 

valid 

• Achenbach (2005) said, “Without evidence based 

assessment, evidence based treatment may be 

like a magnificent house with no foundation.” (p. 

547)

Messick, 1988; Siegel, 1964
Importance of Measurement

Measurement...

1. Hinges on careful design to maximize internal validity

2. Allows for evaluation of our D&I efforts, sets stage for 

comparative effectiveness (Proctor & Brownson, 2012)

3. Enables cross study comparison

4. Facilitates building a D&I knowledge base 

5. Allows for understanding the mechanisms and 

approaches responsible for change to ultimately 

improve public health impact (NIH, PAR RO1)

• Critical for understanding the core effective components

• Important for determining incremental utility of components
Importance of Measurement

Unfortunately,

• Grimshaw et al., (2006) reported that D&I 

strategy effectiveness, and what leads to 

effectiveness, cannot yet easily be evaluated 

because of measurement issues
Ten Key Measurement Issues in D&I

1. The role of psychometric 

properties 

2. Frameworks, construct 

identification & definitions

3. What to measure, when, and 

at what level?

4. Need for communication and 

instrumentation

5. “Praiseworthy rush” 

challenges teams to create 

“homegrown” instruments

6. Instrument specificity and 

adaptation

7. Shared method bias & 

pitfalls of self-reports

8. Mixed methods

9. Practicality versus 

burdensomeness

10. Need for decision-making 

tools

Reference Handout
Seattle Implementation Research Conference

Seattleimplementation.org

• NIMH-funded biennial conference series

• Mission: Promote rigorous implementation 

research methodology

• Top priority for our core team: 

• Measurement Issues

• Embarked on a systematic review of D&I 

instruments

• (N ~ 450)
#1 Psychometric Properties

APA, 1999; Cook & Beckman, 2006; Messick, 1989; Proctor & Brownson, 2012

• Reliability is necessary but not sufficient component 

of validity 

• Reliability: 

• is the reproducibility or consistency of scores from 

one assessment to another

• Indicates items of a particular construct deliver 

consistent scores
#1 Psychometric Properties

APA, 1999; Cook & Beckman, 2006; Messick, 1989; Proctor & Brownson, 2012

• Validity: addresses whether interpretations are wellgrounded and meaningful

• Construct: degree to which instrument measures 

what it purportedly measures

• Five sources of evidence to support validity:

1. Content

2. Response process

3. Internal structure

4. Relations to other variables

5. Consequences 
#1 Psychometric Properties

Proctor & Brownson, 2012

• Criterion-Related: degree to which 

instrument scores correlate with other 

instrument scores (typically gold standard)

• Gold standards do not exist for majority of 

constructs in this field

• Predictive Validity: degree to which instrument 

scores correlate with scores on established 

instrument administered at some point in future

• Need to understand what constructs in each stage 

predicts constructs or “outcomes” in subsequent or 

later stages
#2 Frameworks, Constructs, Definitions

Tabak, Khoong, Chambers, & Brownson, 2012

• 61 Models (i.e., theories & frameworks)

• Rated according to: (1) “D” vs “I”, (2) Socioeconomical

framework level

• Evidence synthesis results:

• Broad models are most common and tended to be “D” focused; 

Definition: “contain constructs that are more loosely 

outlined/defined allowing researchers greater flexibility to apply 

to a wide array of D&I activities”

– Majority of models (n = 25) categorized as “3” on 1=“broad” to 

5=“operational” scale

• Majority emphasized community (n = 52) and organization (n = 

59) with only 8 addressing policy
CFIR Domains Construct

Characteristics of Individuals Knowledge & Beliefs about the Intervention

Individual Stage of Change

Individual Identification with Organization

Other Personal Attributes

Self-Efficacy

Inner Setting Culture

Implementation Climate

Networks and Communications

Readiness for Implementation

Structural Characteristics

Intervention Characteristics Adaptability

Complexity

Cost

Design Quality and Packaging

Evidence Strength and Quality

Intervention Source

Relative Advantage

Trialability

Outer Setting Cosmopolitanism

External Policy and Incentives

Patient Needs and Resources

Peer Pressure

Process Engaging

Executing

Planning

Reflecting and Evaluating

* Note. CFIR = Consolidated Framework for 

Implementation Research. Adapted from 

Damschroder, Aron, Keither, Kirsch, Alexander, 

& Lowery, 2009 

http://www.implementationscience.com/content

/4/1/50/additional/

Consolidated 

Framework for 

Implementation 

Research
* Note. Implementation Outcomes Framework; 

Adapted from Proctor, Landsverk, Aarons, 

Chambers, Glisson, & Mittman, 2009 

Implementation Outcomes 

Framework

Construct

Service Outcomes Effectiveness

Efficiency

Equity

Patient-Centeredness

Safety

Timeliness

Client Outcomes Function

Satisfaction

Symptomology

Implementation Outcomes Acceptability

Adoption

Appropriateness

Feasibility

Fidelity

Penetration

Sustainability

Cost

Implementation 

Outcomes Framework
http://wiki.cfirwiki.net/index.php?title=Main_Page

• Inconsistent, or lack of, definitions provided

• Need a consensual common language (Michie, Fixsen, 

Grimshaw, & Eccles, 2009)

• Validity of instrument hinges on construct, requiring clear definition as 

first step (Cook & Beckman, 2006)

• Consoldiated Framework for Implementation Research Wiki

• Goal: to provide online collaborative space to refine and 

establish terms and definitions related to D&I

• To promote consistent use of these terms and definitions

• To provide a foundation on which a knowledge-base of 

findings related to implementation can be developed.

#2 Frameworks, Constructs, Definitions
• Inconsistent identification and evaluation of 

constructs limits cross-study comparison

• Construct definitions and synonyms

• How different are:

• Appropriateness

– Perceived Fit

– Fitness

– Relevance

– Compatibility

– Suitability

– Usefulness

– Practicality

– Applicability

#2 Frameworks, Constructs, Definitions

Lewis, Borntrager, Martinez, Weiner, Barwick, & Comtois, under review; Tabak, Khoong, Chambers, & 

Brownson, 2012
Proctor & Brownson, 2012

• Important to present a model of the relation 

between constructs you intend to evaluate 

in D&I project

• E.g.,

• Implementation success (low) = f of 

effectiveness (high/low) + acceptability 

(moderate) + cost (high) + sustainability (low)

#2 Frameworks, Constructs, Definitions
#3 What to measure, when?

• CFIR constructs = predictors, moderators, and mediators 

(Damschroder et al., 2009)

• Implementation Outcomes = dependent variables 

(Proctor et al., 2010) 

• Implicated at several stages of implementation

• RE-AIM framework constructs (e.g., reach, adoption, 

implementation, maintenance) = mediators

(Rabin, Brownson, Haire-Joshu, Kreuter, & Weaver, 2008)

• More recently Aarons, Hurlburt, & Horwitz (2011) as well 

as Wandersman, Chien, & Katz (2012) present models 

that help us to know what to measure when
Proctor & Brownson, 2012

• Current state of the field

• Studies not consistent (in terms of time frame) as to 

when constructs are measured

• Others not reporting during what stage of 

implementation the instruments are administered

#3 What to measure, when?
#3 What to measure, when & at what level

• Many of the conceptual papers have intentionally laid out 

the levels of analysis at which D&I constructs are 

relevant

• Individual

• Consumer, Provider, Management/Supervisor, Administration

• Organization

• Community

• System

• Policy

Tabak, Khoong, Chambers, & Brownson, 2012
Aarons, Hurlburt, & Horwitz 2011; Martinez & Lewis, in prep

Temporal heuristic for 

thinking through 

measurement issues at 

multiple stakeholder levels
Aarons, Hurlburt, & Horwitz 2011; Martinez & Lewis, in prep; Proctor & Brownson, 2012

Temporal heuristic for 

thinking through 

measurement issues at 

multiple stakeholder levels 

and across multiple stages 

of an implementation
Aarons, Hurlburt, & Horwitz 2011; Martinez & Lewis, in prep

Temporal heuristic for 

thinking through 

measurement issues at 

multiple stakeholder 

levels and across 

multiple stages of an 

implementation, when 

stakeholders are not 

aligned within the 

process
#4 Need for Communication and 

Instrumentation

• SIRC aims to collect 

instruments ‘in 

development’ to avoid 

unnecessarily 

redundant efforts

• 54% of constructs are 

of high priority for 

instrument 

development

High Priority (<5 instruments) Medium Priority (5-10 instruments) Low Priority (10+ instruments)

Design Quality and Packaging (0) RI: Leadership Engagement (5) Safety (11)

Engaging: Formally Appointed 

Internal Implementation (0) Penetration (5)

Networks & Communications 

(12)

Engaging: Champions (0) Evidence Strength and Quality (6) Satisfaction (client) (12)

Executing (0) Appropriateness (Applicability) (7) Feasibility (15)

External Policy & Incentives (0)

Patient Needs and Resources (Needs 

Assessment) (7) IC: Learning Climate (15)

IC: Goals and Feedback (0) Individual Stage of Change (7)

Readiness for Implementation 

(RI) (18)

Intervention Source (0) Sustainability (8) Client Satisfaction (19)

Peer Pressure (0)

Relative Advantage(Innovativeness) 

(9)

Reflecting and Evaluating 

(Workshop evaluation) (20)

Adaptability (1)

Individual Identification with 

Organization (9)

Combined (Functioning , 

Context) (21)

Cosmopolitanism (1) Culture (21)

Effectiveness (1) Implementation Climate (IC) (23)

Engaging (1)

Adoption (Uptake, Knowledge 

Translation) (24)

Engaging: External Change Agents 

(1) Planning(Conceptualization) (28)

IC: Compatability (1)

Other Personal 

Attibutes(Demographics) (33)

IC: Relative Priority (1) Acceptability (52)

IC: Tension for Change (2)

Knowledge & Beliefs about the 

Intervention (Attitudes, Beliefs, 

Perceived Barriers) (56)

RI: Available Resources (2)

RI: Access to Knowledge and 

Information (Basic Knowledge) (2)

Structural Characteristics (2)

Cost (3)

Complexity (3)

Engaging: Opinion Leaders (3)

Trialability (3)

Feasibility (Transferability, 

Disseminability) (4)

IC: Organizational Incentives & 

Rewards (4)

Self-Efficacy (4)

Lewis, Borntrager, Martinez, Weiner, Barwick, & 

Comtois, under review
#5 ‘Homegrown’ Instruments

Lewis, Borntrager, Martinez, Weiner, Barwick, & Comtois, under review; Weiner, 2009

• Due to “praiseworthy rush” to implement evidence 

based interventions, we have observed an influx in 

‘homegrown’ instruments

• ‘Homegrown’ loosely defined: Developed in haste 

without systematically using theory, not engaging in 

the necessary steps of appropriate instrument 

development, notably without conducting tests of 

psychometrics

• 41% of SIRC’s Implementation Outcomes repository (N = 

92) fall within this definition of ‘homegrown’
#5 Stages of Instrument Development

Haynes, Richard, & Kubany, 1995

• Item generation

• Borrow from related instruments with good 

psychometrics

• Review literature relevant to construct

• Discuss with working group

• Subject to expert review

• Establish rating scheme

• Pilot instrument

• Solicit suggestions for modification

• Refine and narrow item pool

• Subject to second expert review
• Establish Psychometrics

• Exploratory factor analysis on random halves of a 

large sample, EFA followed by CFA to assess 

structural validity

• Examine: internal consistency

• Evaluate instrument with respect to already 

established instruments to assess:

• convergent, divergent, concurrent validity

#5 Stages of Instrument Development

Haynes, Richard, & Kubany, 1995
#6 Instrument Specificity and Adaptation

• Certain constructs tend to necessarily be 

intervention-, population-, or setting-specific

• Intervention-specific: Fidelity

• Population-specific: Client outcomes

• Setting-specific: Patient needs and resources

• Makes it challenging to compare across studies

• Countless research teams are adapting instruments 

• In ways that affect psychometrics

• Without delineating how it has been adapted
#7 Shared Method Bias and Pitfalls of 

Self-reports

• Shared method bias particularly problematic when 

predictor and criterion data obtained from same 

person

• Capitalize on opportunities for direct observation or 

independent assessor as opposed to relying on 

self-report

• Minimize burden

Podsakoff, MacKenzie, Lee, & Poksakoff, 2003
#7 Shared Method Bias and Pitfalls of 

Self-reports

• Self-report is less accurate for particular D&I 

constructs such as adherence 

• Self-report 

• Provides restricted range of content

• Can be influenced by intentional false reporting or 

presentation bias

• Subject to inattentive responding

• Cognitive or memory limits

• Differential responding due to unintentional item 

ambiguity

Donaldson, Rutledge, & Ashley, 2004; Kimberly & Cook, 2008
#8 Mixed Methods: What is the role of 

this in D&I?

Palinkas, Aarons, Horwitz, Chamberlain, Hurlburt, & Landsverk, 2011

• To provide better understanding than 

either approach alone

• Qualitative: exploratory, to gain a deeper 

understanding

• Quantitative: test hypotheses

• Range of mixed method designs from 

simple to complex, dependent on stage 

of implementation
#9 Practicality versus Burdensomeness

http://cancercontrol.cancer.gov/brp/gem.html

• Factors to consider in terms of practicality

• Cost

• Accessibility (in the public domain?)

• Length/burdensomeness

• User friendly

• Applicable

• Scoring
#10 Need for Decision Making Tools

• Some constructs (e.g., culture, n = 21) have several 

instruments from which to select, 3 of which have 

the same name

• Factors influencing instrument choice

• Theory

• Previous Research

• Psychometrics

• Practicality of instruments

• Grid-Enabled Measures Project

• SIRC Systematic Review of Instruments Project

Lewis, Borntrager, Martinez, Weiner, Barwick, & Comtois, under review
Goals

Step 1

Instrument Repository

Step 3

Consensus 

Battery

Step 2

Systematic 

Review

Evidence Based 

Assessment
SIRC Systematic Review of Instruments

Seattleimplementation.org

• Build capacity

• University of Montana, North Carolina, Indiana

• Task Force of approximately 60 members

• Information gathering: 

• Systematic review of constructs

• Systematic review of specific instruments 
SIRC Systematic Review of Instruments

Seattleimplementation.org

• Evidence Based Assessment Criteria

• Hunsley & Mash, 2010

• Terwee, Bot, De Boer, et al., 2007

• Core criteria

• Validity – structural + predictive

• Reliability – internal consistency

• Usability – number of items

• If at least “good” (score of 3 on 5-point scale) on 

core then additional criteria applied

• Norms, responsiveness (i.e., sensitivity to change)
SIRC Instrument Review Project

http://www.seattleimplementation.org/sircprojects/sirc-instrument-project/
Lewis, Martinez, & Comtois, in prep

Measurement Issues Summary Handout

Theoretical Empirical Psychometric Practical

1. Psychometric Properties

Carefully define constructs to be sure that core 

features/ components are made clear

Be certain to have the sample size 

necessary to maximize evaluation of 

psychometrics particularly with new 

instrument

Evaluate and explicitly report both 

reliability and validity

Prioritize instruments with strong 

psychometrics (and fewest number of items as a 

marker of practicality)

2. Frameworks, Construct 

Identification & Definitions

Identify a theory to guide the project and 

contribute to evaluation of said theory through 

careful measurement; Use shared/consensus 

definitions of constructs wherever possible

Evaluate models of D&I to determine 

relation between constructs; Determine if 

nuanced terms are distinct constructs or 

synonyms

Contribute to the literature by using 

previously established instruments when 

conceptually and theoretically 

appropriate

Consider measuring the key constructs relevant 

to the D&I effort with a focus on mechanisms of 

change

3. What to measure, when, 

and at what level?

Determine at which level of analysis each 

construct is implicated; Use theory and 

empirical evidence to inform what should be 

measured when

Report exactly at what stage/time and level 

of analysis each construct was measured

Carefully consider the role of predictive 

validity (and concurrent, convergent, 

divergent)

Carefully define D&I model being tested to be 

clear on top priority constructs to measure

4. Need for Communication 

& Instrumentation

Search carefully for instruments tapping 

synonyms for constructs of interest; Avoid 

modifying scales

If scale modification is necessary report 

specifically how it was modified

Carefully report all potentially meaningful 

statistics (e.g., norms, internal 

consistency) for the specific sample

Consider sharing psychometrically valid 

instruments with GEM or SIRC initiatives

5. “Homegrown” 

Instruments versus 

Rigorous Test Development

Carefully define constructs to be sure that core 

features/ components are made clear

Consider engaging a test developer if high 

priority construct without established 

instrumentation identified

Evaluate and report psychometrics of 

“homegrown” instruments if 

administered

Avoid use of “homegrown” instruments for core 

constructs

6. Instrument Specificity 

versus Adaptation

Consider assessing core D&I constructs in 

reproducible ways to test and inform theory 

development

If scale modification is necessary report 

specifically how it was modified

Conduct psychometric analyses (e.g., to 

assess structural validity) whenever 

possible

Choose to adapt an instrument if possible over 

generating entirely new one

7. Shared Method Bias & 

Pitfalls of Relying Solely on 

Self-Report

Try to assess constructs that tap higher levels of 

analysis (e.g., organizational readiness) 

appropriately as opposed to drawing inferences 

about individual perspectives

Consider observations in addition to 

surveys

Understand the influence of 

psychometrics on the variables in the 

study

Consider alternative ways to measure 

constructs in addition to self-report to reduce 

problem of common method bias, or measure 

at different time points

8. The Role of Mixed 

Methods

Contribute to theory development by 

employing mixed methods to deepen our 

understanding of less established constructs

Consider the specific role and utility of 

qualitative and quantitative data 

depending on the stage of the project and 

the design

Employ experts, particularly with respect 

to qualitative data analysis to strengthen 

inferences made

Carefully define role and purpose to maximize 

utility of mixed method design

9. Practicality to Promote 

Stakeholder Engagement

Consider only assessing core D&I constructs 

theorized to be essential mechanisms of change 

relevant to the question/study , potentially 

based on previous research

Consider using instruments that have wellestablished psychometrics to maximize the 

potential of the study to contribute to 

knowledge base & be confident in 

interpretations made

Do not ignore psychometrics in favor of 

practicality because psychometrics reflect 

the capacity to accurately interpret scores

Strive for developing the shortest instruments 

possible; Identify psychometrically sound 

alternatives when cost prohibits use of wellestablished instrument

10. Decision-Making Aids

Carefully define constructs to be sure that core 

features/ components are made clear and the 

instrument selected taps the intended features

Consider using instruments that have been 

well-established to further contribute to 

the knowledge base

Assess psychometrics of “competing” 

instruments and use the most reliable 

and valid

Consider prioritizing strong psychometrics 

wherever possible
Acknowledgements

• SIRC Core Team

• Cameo Borntrager

• Bryan Weiner

• TRIP Lab
Measurement Issues Lewis, Martinez, & Comtois, 2012

Theoretical Empirical Psychometric Practical Notes

1. Psychometric 

Properties

Carefully define constructs to 

be sure that core features/ 

components are made clear

Be certain to have the 

sample size necessary to 

maximize evaluation of 

psychometrics particularly 

with new instrument

Evaluate and explicitly 

report both reliability and 

validity

Prioritize instruments with 

strong psychometrics (and 

fewest number of items as a 

marker of practicality)

2. Frameworks, 

Construct 

Identification & 

Definitions

Identify a theory to guide the

project and contribute to 

evaluation of said theory 

through careful measurement; 

Use shared/consensus 

definitions of constructs

wherever possible

Evaluate models of D&I to 

determine relation 

between constructs; 

Determine if nuanced 

terms are distinct 

constructs or synonyms

Contribute to the 

literature by using 

previously established 

instruments when 

conceptually and 

theoretically appropriate

Consider measuring the key 

constructs relevant to the D&I 

effort with a focus on 

mechanisms of change

3. What to 

measure, when, 

and at what 

level?

Determine at which level of 

analysis each construct is 

implicated; Use theory and 

empirical evidence to inform 

what should be measured 

when

Report exactly at what 

stage/time and level of 

analysis each construct was 

measured

Carefully consider the role 

of predictive validity (and 

concurrent, convergent, 

divergent)

Carefully define D&I model 

being tested to be clear on top 

priority constructs to measure

4. Need for 

Communication 

& 

Instrumentation

Search carefully for 

instruments tapping synonyms 

for constructs of interest; 

Avoid modifying scales

If scale modification is 

necessary report

specifically how it was 

modified

Carefully report all 

potentially meaningful 

statistics (e.g., norms, 

internal consistency) for 

the specific sample

Consider sharing 

psychometrically valid 

instruments with GEM or SIRC 

initiatives

5. “Homegrown” 

Instruments

versus Rigorous 

Test 

Development

Carefully define constructs to 

be sure that core features/ 

components are made clear

Consider engaging a test 

developer if high priority 

construct without 

established 

instrumentation identified

Evaluate and report 

psychometrics of 

“homegrown” instruments 

if administered

Avoid use of “homegrown” 

instruments for core 

constructs

Notes
Measurement Issues Lewis, Martinez, & Comtois, 2012

6. Instrument 

Specificity versus 

Adaptation

Consider assessing core D&I 

constructs in reproducible 

ways to test and inform theory 

development

If scale modification is 

necessary report 

specifically how it was 

modified

Conduct psychometric 

analyses (e.g., to assess 

structural validity) 

whenever possible

Choose to adapt an 

instrument if possible over 

generating entirely new one

7. Shared 

Method Bias & 

Pitfalls of Relying 

Solely on SelfReport

Try to assess constructs that 

tap higher levels of analysis 

(e.g., organizational readiness) 

appropriately as opposed to 

drawing inferences about

individual perspectives

Consider observations in 

addition to surveys

Understand the influence 

of psychometrics on the 

variables in the study

Consider alternative ways to 

measure constructs in addition 

to self-report to reduce 

problem of common method

bias, or measure at different 

time points

8. The Role of 

Mixed Methods

Contribute to theory 

development by employing 

mixed methods to deepen our 

understanding of less 

established constructs

Consider the specific role 

and utility of qualitative 

and quantitative data 

depending on the stage of 

the project and the design

Employ experts, 

particularly with respect 

to qualitative data analysis 

to strengthen inferences 

made

Carefully define role and 

purpose to maximize utility of 

mixed method design

9. Practicality to 

Promote 

Stakeholder 

Engagement

Consider only assessing core 

D&I constructs theorized to be 

essential mechanisms of 

change relevant to the

question/study , potentially 

based on previous research

Consider using instruments 

that have well-established 

psychometrics to maximize 

the potential of the study 

to contribute to knowledge 

base & be confident in 

interpretations made

Do not ignore 

psychometrics in favor of 

practicality because 

psychometrics reflect the

capacity to accurately 

interpret scores

Strive for developing the 

shortest instruments possible; 

Identify psychometrically 

sound alternatives when cost 

prohibits use of wellestablished instrument

10. DecisionMaking Aids

Carefully define constructs to 

be sure that core features/ 

components are made clear

and the instrument selected 

taps the intended features

Consider using instruments 

that have been wellestablished to further 

contribute to the 

knowledge base

Assess psychometrics of 

“competing” instruments 

and use the most reliable 

and valid

Consider prioritizing strong 

psychometrics wherever 

possible

Notes
Measurement Issues in Implementation Science November 1, 2012

Suggested Readings/Reference List

Aarons, Gregory A., Hurlburt, Michael, & Horwitz, Sarah M.C. (2011). Advancing a conceptual model of 

evidence-based practice implementation in public service sectors. Administration and Policy in Mental 

Health and Mental Health Services Research, 38(1), 4-23. 

Achenbach, Thomas M. (2005). Advancing assessment of children and adolescents: commentary on 

evidence-based assessment of child and adolescent disorders. Journal of Clinical Child and Adolescent 

Psychology, 34(3), 541-547. 

Beidas, Rinad S., Mehta, Tara, Atkins, Marc, Solomon, Bonnie, & Merz, Jenna. (2011). Dissemination and 

Implementation Science in Mental Health Services: Research Methods Dissemination and 

Implementation Science: Research Methods.

Chamberlain, Patricia, Brown, C. Hendricks, & Saldana, Lisa. (2011). Observational measure of

implementation progress in community based settings: The Stages of implementation completion (SIC). 

Implementation Science, 6(1), 116. 

Cook, David A., & Beckman, Thomas J. (2006). Current concepts in validity and reliability for 

psychometric instruments: theory and application. The American Journal of Medicine, 119(2), 166.e167-

166.e116. 

Damschroder, Laura J., Aron, David C., Keith, Rosalind E., Kirsh, Susan R., Alexander, Jeffrey A., & Lowery, 

Julie C. (2009). Fostering implementation of health services research findings into practice: a 

consolidated framework for advancing implementation science. Implementation Science, 4(1), 1-15. 

Donaldson, Nancy E., Rutledge, Dana N., & Ashley, Jeffrey. (2004). Outcomes of adoption: measuring 

evidence uptake by individuals and organizations. Worldviews on Evidence‐Based Nursing, 1, S41-S52. 

Greenhalgh, Trisha, Robert, Glenn, Macfarlane, Fraser, Bate, Paul, & Kyriakidou, Olivia. (2004). Diffusion 

of innovations in service organizations: systematic review and recommendations. Milbank Quarterly, 

82(4), 581-629. 

Haynes, Stephen N., Richard, David C.S., & Kubany, Edward S. (1995). Content validity in psychological 

assessment: A functional approach to concepts and methods. Psychological Assessment, 7(3), 238-247. 

Hunsley, John, & Mash, Eric J. (2010). Evidence-based assessment. In D. H. Barlow (Ed.), The Oxford 

Handbook of Clinical Psychology (pp. 76): Oxford University Press.

Kimberly, John, & Cook, Joan M. (2008). Organizational measurement and the implementation of 

innovations in mental health services. Administration and Policy in Mental Health and Mental Health 

Services Research, 35(1-2), 11-20. 

Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13-103). New York: 

Macmillan.
Measurement Issues in Implementation Science November 1, 2012

Michie, Susan, Fixsen, Dean L., Grimshaw, Jeremy M., & Eccles, Martin P. (2009). Specifying and 

reporting complex behaviour change interventions: the need for a scientific method. Implement 

Science, 4(40), 40. 

Palinkas, Lawrence A., Aarons, Gregory A., Horwitz, Sarah, Chamberlain, Patricia, Hurlburt, Michael, & 

Landsverk, John. (2011). Mixed method designs in implementation research. Administration and Policy 

in Mental Health and Mental Health Services Research, 38(1), 44-53. 

Podsakoff, Phillip M., MacKenzie, Scott B., Lee, Jeong-Yeon, & Podsakoff, Nathan P. (2003). Common 

method biases in behavioral research: a critical review of the literature and recommended remedies. 

Journal of applied psychology, 88(5), 879-903. 

Proctor, Enola K., & Brownson, Ross C. (2012). Measurement Issues in Dissemination and 

Implementation Research. In R. C. Brownson, G. A. Colditz & E. K. Proctor (Eds.), Dissemination and 

Implementation Research in Health: Translating Science to Practice (pp. 261-280): Oxford University 

Press.

Proctor, Enola K., Landsverk, John, Aarons, Gregory, Chambers, David, Glisson, Charles, & Mittman, 

Brian. (2009). Implementation research in mental health services: An emerging science with conceptual, 

methodological, and training challenges. Administration and Policy in Mental Health and Mental Health 

Services Research, 36(1), 24-34. 

Rabin, Borsika A., Brownson, Ross C., Haire-Joshu, Debra, Kreuter, Matthew W., & Weaver, Nancy L. 

(2008). A glossary for dissemination and implementation research in health. Journal of Public Health 

Management and Practice, 14(2), 117-123. 

Siegel, A.E. (1964). Sidney Siegel: a memoir Messick and Brayfield (eds.).

Tabak, Rachel G., Khoong, Elaine C., Chambers, David A., & Brownson, Ross C. (2012). Bridging Research 

and Practice: Models for Dissemination and Implementation Research. American Journal of Preventive 

Medicine, 43(3), 337-350. 

Terwee, Caroline B., Bot, Sandra D.M., De Boer, Michael R., van der Windt, Danielle A.W.M., Knol, Diek 

L., Dekker, Joost, . . . De Vet, Henrica C.W. (2007). Quality criteria were proposed for measurement 

properties of health status questionnaires. Journal of Clinical Epidemiology, 60(1), 34-42.

Wandersman, Abraham, Chien, Victoria H., & Katz, Jason. (2012). Toward an evidence-based system for 

innovation support for implementing innovations with quality: tools, training, technical assistance, and 

quality assurance/quality improvement. American journal of community psychology, 1-15.

Weiner, Bryan J. (2009). A theory of organizational readiness for change. Implementation Science, 4(1), 

67.