Evaluating Evidence-Based Quality Improvement of Comprehensive Women's Health Care Implementation in Low-Performing VA Facilities (PEC 16-352)
Phase: NA
Conditions: Comprehensive Care
Sponsor: VA Office of Research and Development

PEX 16-002 
 
Evaluating Evidence-Based Quality Improvement of Comprehensive Women's Health Care 
Implementation in Low-Performing VA Facilities 
 
NCT03238417 
 
December 1, 2016 

02. Specific Aims 
Women Veterans (WVs) continue to represent the fastest growing segment of new users of the VA 
healthcare system, doubling their numbers in the past decade, with 10% of VA users being women by 2018.  
Their physical and mental health (MH) needs have proven to be more complex than their male counterparts, 
including a high rate of military sexual trauma (MST) that places them at greater risk of depression, anxiety 
and posttraumatic stress disorder (PTSD), while also complicating medical management and requiring 
gender-sensitive care environments. Most VA providers also have minimal exposure to women, creating 
deficits in needed knowledge and experience.  Historical gaps in VA provision of women’s health (WH) care 
have also resulted in WVs being more likely to be referred to community providers, adding to their care 
fragmentation and contributing to persistent gender disparities in VA quality of care and patient experience.   
These trends have required multiple initiatives to change VA culture to be more gender-sensitive, set 
minimum standards for training and proficiency of designated WH providers, and delineate features of 
acceptable primary care clinic models that integrate gender-specific and MH services in “one-stop 
shopping” approaches. VA Women’s Health Services (WHS) has led these efforts and set forth an updated 
VA policy on “Health Care Services for Women Veterans” (VHA Handbook 1330.01, May 2010), which 
seeks to systematically improve WVs’ access to comprehensive healthcare services delivered by proficient 
providers and staff in environments that ensure WVs’ safety, security and dignity. In the same year, WHS 
launched an annual Women’s Assessment Tool for Comprehensive Health (WATCH) to evaluate Handbook 
implementation, and an external evaluation comprised of site visits to over 84% of VA medical centers 
(VAMCs). Together, these evaluation activities have documented substantial progress and informed 
strategic planning and decision-making in terms of policies and resources needed to improve VA WH 
programs nationwide.   
However, the evaluations also found that traditional top-down policy implementation—even when leveraged 
by evaluation feedback and multilevel WH champions at the clinic, VAMC, VISN and national levels—has 
not been uniformly successful in achieving the tenets of VA policy on comprehensive WH care. WHS has 
begun to identify consistently low-performing VA facilities that would benefit from more focused 
organizational interventions. Based on previous success using evidence-based quality improvement (EBQI) 
as an implementation strategy that fosters local engagement and multilevel stakeholder activation around 
improvement goals, WHS will rollout EBQI in low-performing VA facilities starting in FY17. WHS’s EBQI 
approach, which will be implemented by an already-approved contractor, builds directly on the bundle of 
activities being tested in one of the five WH CREATE studies (CRE 12-026), which itself was based on 
EBQI efforts in multiple VA randomized trials.  These activities include local site visits designed to achieve 
consensus on QI targets, multilevel stakeholder engagement, external practice facilitation, local EBQI team 
training, and formative feedback, for which technical specifications have already been developed.    
We have worked closely with WHS to design a convergent parallel mixed methods evaluation in the context 
of a dynamic waitlisted design to evaluate the effectiveness of EBQI implementation on achievement of 
comprehensive WH care in low-performing VA facilities.  Building on our substantial prior experience 
implementing and evaluating EBQI in VA, we propose the following specific aims: 
Aim #1:  To evaluate barriers and facilitators to achieving delivery of comprehensive women’s health care in 
the identified low-performing VA facilities; 
Aim #2: To evaluate the effectiveness of EBQI in supporting low-performing VA facilities achieve improved: 
a. Organizational features (e.g., level of comprehensive services available, care coordination 
arrangements, PACT features implemented, environment of care improvements); 
b. Provider/staff attitudes (e.g., improved gender awareness, WH knowledge and practice);  
c. Quality of care and patient experience among WV patients; and, 
Aim #3:  To evaluate contextual factors, local implementation processes, and organizational changes in the 
participating facilities over time. 
Evaluation results will inform strategies for optimizing future policy deployment and multilevel engagement 
strategies with the field, while also informing best practice diffusion. The focus on low-performing VAs will 
offer new insights, as these less-studied facilities may require uniquely concentrated and/or tailored efforts. 

02a. RESEARCH PLAN (EVALUATION PLAN) 
A. BACKGROUND  
A1. Gaps in Comprehensive Care for WVs. Historically plagued by gaps in safety and privacy for women 
in VA facilities originally designed for men, with a workforce with little to no exposure to female patients, the 
VA has faced significant challenges in meeting WVs’ complex care needs.1,2,,3 Ensuring access to gender-
specific care and a full complement of reproductive and gynecologic health services has also contributed to 
higher rates of community referrals among WVs, further fragmenting their care.4,5,6  These and other 
challenges have led to persistent gender disparities in VA care quality and patient experience.7,8 
Over a decade ago VA stood up a handful of comprehensive WH centers in response to Government 
Accounting Office findings of gaps in WVs’ care, and subsequent legislation.9 WH clinics grew 8-fold over 
the next decade.10 However, only a fraction delivered comprehensive services like the original model 
programs, many focusing on gender-specific exams to help increase VA breast and cervical cancer 
screening rates.11 Nonetheless, adoption of WH clinic models was associated with higher preventive 
practices and higher WV ratings of access, continuity, coordination, and satisfaction.12,13 
A2. VA Policy Aimed to Respond to Gaps in Care. In 2010, VA WHS launched a 5-year strategic plan to 
redesign the nation’s healthcare delivery system for WVs, with guidance on required use of designated WH 
providers (DWHPs) for primary care (PC), acceptable PC model features with integrated MH and gender-
specific care, and a set of required comprehensive services (VHA Handbook 1330.01, May 2010). Added 
were explicit expectations that all VA care environments must afford WVs a sense of safety/security, supply 
same-gender providers (if preferred), and address gender-sensitivity across the spectrum of VA services. 
A3. Evaluation of Comprehensive Care Policy Implementation. WHS contracted with a private sector 
firm from 2010-15 to gauge progress towards full Handbook implementation through site visits that included 
interviews with approximately 20 individuals/groups, case discussions, facility tours, and a review of 
program-related documents. Each VAMC’s capabilities were then scored (from “needs development” to 
“highly developed”) through focused team consensus sessions across 33 Handbook-related domains. Site 
visits concluded with closing conferences with leadership, followed by Site Visit Summary Reports and 
Score Cards for each site. WHS received dashboard updates for distinctively high- and low-end performers 
on program components and capabilities by facility and VISN.   
Major problematic findings included gaps in organizational structure (e.g., WV Program Managers being 
assigned collateral duties, precluding focus on policy-mandated roles and responsibilities), inadequate 
oversight or planning, major gaps in privacy/security in exam rooms, limited cross-coverage for key WH 
roles, environments not welcoming to women, gaps in comprehensive care in community-based outpatient 
clinics, staffing insufficiency (including below-standard staffing ratios, high panel sizes, limited WH 
experience/training), and limited efforts to improve cultural competency and gender sensitivity of staff.  
A4. WHS/HSR&D Collaboration to Accelerate Comprehensive Care Implementation. WHS has long 
been funding the applicant HSR&D investigators and their colleagues to conduct evaluation activities of 
WVs’ health conditions and use (e.g., WV Sourcebooks), needs and experiences (e.g., National Survey of 
WVs), and evaluations of VA WH programs (e.g., telegynecology). In2012, WHS funded the applicant team 
to conduct a national expert panel to define gender-sensitive comprehensive care to improve its 
measurement.14 WHS then worked with us as primary partner for the WH CREATE, a partnered 
collaborative research initiative funded by VA HSR&D to use research to further accelerate implementation 
of comprehensive care for WVs.15 All five CREATE studies launched in early 2013, including a national 
organizational survey of comprehensive care arrangements based on the panel results, and a 12-VAMC 
trial of an EBQI approach to tailoring Patient Aligned Care Teams (PACT) to the needs of WVs. The WH 
CREATE is an integral initiative supported by the VA WH Research Network (WHRN), which the Blueprint 
for Excellence cited as central to accelerating innovations in VA WH care (Transformational Strategy 7.2.g).   
A5. WHS Main Evaluation Goals. WHS seeks a partnered evaluation of their rollout of EBQI as a new 
strategy for improving quality of WH care in consistently low-performing VA facilities. Building on early 
evidence of EBQI’s promise in activating local teams and leadership around WH PACT improvements, 
WHS asked for technical specifications to enable contracting for EBQI. They then asked us to adapt our WH 
PACT EBQI evaluation methods—specifically, key stakeholder (KS) interviews, provider/staff surveys, and 

analysis of secondary data—to determine the ways in which EBQI may help low-performing VAs improve 
quality of care.16 In our WH PACT study, these methods and measures have revealed new information on 
WVs’ needs and experiences, elucidated implementation barriers, and helped identify actionable 
provider/staff attitudes and knowledge gaps.  
A6. Conceptual Framework for Evaluation. We have adapted our conceptual framework from the WH 
PACT trial for the proposed evaluation (Figure below).16 In this evaluation, a contractor working under 
technical specifications for EBQI (far left column) will: 1) convene facility-level stakeholder meetings; 2) 
facilitate local facility-level QI team design meetings; and 3) provide external practice facilitation through 
within- and across-facility QI collaboration calls, QI data feedback and QI training/education. Initial results of 
EBQI implementation will include local QI actions (e.g., strategic project activities, structured QI proposals, 
and multilevel review in advance of conduct), and improved provider and staff QI orientation, WH 
knowledge/awareness, and 
gender awareness (middle 
column, top). These actions 
will occur in the context of 
each VAMC’s leadership 
support, local resources, pre-
EBQI WH care model and 
staffing, pre-EBQI provider 
and staff QI and WH 
experience, awareness and 
attitudes, as well as area 
characteristics (e.g., 
urban/rural location) (middle 
column, bottom).  
Our evaluation activities map directly to this conceptual framework using a formative evaluation framework 
designed to identify the potential and actual influences on progress and effectiveness of implementation 
efforts.17 For example, we will evaluate the EBQI contractor’s implementation of EBQI methods (first 
column) under Aim #1 as a developmental evaluation stage (e.g., degree of less-than-best practice, 
determinants of current practice, barriers/facilitators and feasibility/perceived utility).  Under Aim #2, we will 
conduct a progress-focused evaluation of EBQI effectiveness on achievement of comprehensive care (right 
column), monitoring impacts and indicators of progress toward goals, with feedback to WHS and the EBQI 
contractor. Aim #3 will cover two types of evaluation. First, we will conduct an implementation-focused 
evaluation (also known as process evaluation) to examine discrepancies between EBQI implementation 
plans and how the EBQI contractor actually operationalizes them, helps identify influences we might 
otherwise have not considered, will enable us to describe experiences of those undergoing EBQI 
implementation, and will consider the context in which facilities participate (12-month key stakeholder 
interviews [§B4a] will be especially important). Second, we will use results from all of the other evaluation 
stages to conduct an interpretive evaluation (24-month key stakeholder interviews will be key here). 
B. METHODS (EVALUATION PLAN) 
B1. Evaluation Plan Overview 
We will conduct a convergent parallel mixed methods evaluation17,18 in the context of a dynamic waitlisted 
design to evaluate the effectiveness of EBQI implementation on achievement of comprehensive WH care in 
low-performing VA facilities. For Aim #1 (developmental evaluation), we will rely on key stakeholder 
interviews (§B4a), organizational surveys (§B4b), and administrative data (§B4d) at baseline and ahead of 
each phased implementation launch of another wave of VA facilities. For Aim #2 (progress-focused 
evaluation), we will use organizational surveys (§B4b), provider/staff surveys (§B4c), and secondary data 
on WVs’ quality of care and patient experience (§B4d). For Aim #3 (implementation-focused & 
interpretive evaluations), we will re-interview Aim #1 key stakeholders, with a focus on implementation 
processes and organizational changes at 12 and 24 months (§B4a). 
B2. Evaluation Design   

The choice of a dynamic waitlisted design is based on WHS’s plans for staged EBQI implementation at no 
more than eight VA facilities per year for the next three years for a total of 24 facilities. This randomized 
“roll-out” implementation design has very good statistical properties, including higher power than traditional 
wait-listed designs19 and less vulnerability to external, 
uncontrolled factors.20 EBQI implementation will 
commence in Year 1 (FY17) in eight randomly 
selected facilities (Group 1), with the remaining 16 
facilities serving as waitlisted controls. In FY18, eight 
of these waitlisted controls will be randomly selected 
for EBQI implementation (Group 2), leaving the 
remaining eight as waitlisted controls in FY18, which 
will begin EBQI implementation in Year 3 (Group 3).   
B3. Study Population and Sampling Strategy   
In the first 30 days of FY17, with input from WHS and the evaluation team, the EBQI contractor will analyze 
prior site visit assessment data (Booz-Allen-Hamilton FY16 report), local VA facility self-assessment data 
(WATCH) (Appendix 2), and selected performance measures to identify the lowest-performing VAs (bottom 
25-30%) of previously assessed sites nationwide and by VISN. In FY17Q1, WHS will make the final site 
selection decision (n=24 VA facilities) that will be the focus of EBQI implementation and evaluation. 
B4. Data Sources, Measures and Data Collection Approaches  
The proposed data sources are a direct result of WHS input on and preferences for evaluation design, and 
their experience with these sources under the WH CREATE and previous evaluations we have conducted 
with and for WHS. In particular, the evaluation will substantively benefit from use of existing instruments 
from the WH CREATE, including the national VA organizational survey modules noted above (CRE 12-038) 
(including national secondary data on quality metrics and patient experience) and the WH PACT trial (CRE 
12-026) (including KS interview guides and provider/staff surveys). Below we describe each evaluation data 
source’s sampling plans, measures, and data collection procedures. 
B4a. Key Stakeholders (KS). Semi-structured qualitative interviews will be conducted at baseline for all 
sites (Aim #1), as well as immediately prior to the EBQI launch of Groups 2 and 3. Interviews will be 
conducted at all sites at 12 months post-EBQI launch; in Group 1, interviews will also be conducted at 24 
months. Sampling Plan:  For Aim #1, we will work with the EBQI contractor to identify facility-level KS (e.g., 
quadrad member, PC director, WH medical director, WVPM, QI/system redesign lead; n~6/facility) and 
VISN-level KS (e.g., VISN director, VISN Lead WVPM, VISN system redesign lead; n~4/VISN). We will also 
interview WHS leaders and EBQI contractor personnel to evaluate leadership and implementation 
processes. We will adapt KS selection based on the QI targets established at initial EBQI site visits (e.g., 
add a MH director if the QI project targets MH). For Aim #3, we will seek to re-interview the same KS as in 
Aim #1, but will pursue replacement personnel in the event of turnover and/or position changes over time. 
Measures: The Aim #1 baseline interview guide (Appendix 3) includes questions about the structure and 
delivery of usual care for WVs, barriers and facilitators to achieving delivery of comprehensive WH care, 
what (if any) improvements are underway in WH and/or for WVs, familiarity with performance metrics, 
access to metrics by gender, experience with QI, local culture, perceptions of the care environment, and 
engagement. The Aim #3 12- and 24-month interviews will assess any changes in care for WVs (staffing, 
structure, etc.), details of completed/in progress QI projects, perspectives on critical components of EBQI, 
and anticipated sustainability of local improvements and QI methods. Data Collection: Under the 
supervision of Dr. Hamilton (Co-PI), KS interviews will be conducted by telephone, recorded and 
professionally transcribed. Transcripts will be reviewed and edited for accuracy. 
B4b. Organizational Surveys. We will use key informant organizational surveys at baseline, 12- and 24-
months among the 24 participating VA facilities, in addition to annual administered WATCH surveys from 
WHS (Aim #2a). For Year 1 (FY17), we will rely on national organizational survey modules that will be 
fielded in Quarter 1 (Q1) as part of a WH CREATE study (Rose & Yano, Co-PIs, CRE 12-038) (Appendix 
4). For Years 2 and 3 (FY18-19), we will re-administer the same surveys, adapting selected domains in 
relation to EBQI targets of participating VAs. Sampling Plan: All 24 participating VA facilities, including 
modules for PC directors, WH medical directors, MH directors and WV Program Managers (WVPMs). We 

will use the tailored design method to identify named respondents for each module.21 Additional key 
informants (e.g., chief of medicine) may be considered in subsequent waves if local EBQI targets warrant it. 
Measures: We will include measures of leadership support,22 local resources (e.g., sufficiency of time, 
personnel, equipment),23 practice structure (e.g., WH care model, staff mix, referral arrangements), service 
availability,24 care coordination arrangements (within and outside VA), ability to engage in QI (e.g., barriers 
to QI, data access by gender), gender-sensitivity of environment (e.g., privacy), local challenges (e.g., 
provider shortages, hiring difficulties, practice chaos),25,26 facility type (e.g., size, academic affiliation, 
urban/rural), and EBQI activities.27 Data Collection: We will field surveys through REDCap, a VA-approved 
web survey vendor.  
B4c. VA Providers/Staff Surveys. We will use web-based provider/staff surveys at baseline, 12- and 24-
months. Sampling Plan: We will obtain lists of local providers and staff using the same protocol used 
previously to identify PC and WH providers/staff, drawing a census from Primary Care Management Module 
(PCMM) data for each participating facility (including PACT teamlets and larger PACT teams [e.g., 
pharmacists, social workers]). In the 12-VA WH PACT trial, this approach yielded 775 providers and staff 
across urban and rural VAMCs; we anticipate identifying roughly twice as many personnel in the 24-facility 
evaluation. If other work units are prioritized by WHS (e.g., MH provider/staff), we will adapt the same 
approach but using VA PAID and other administrative sources, with local sample validation. Measures: We 
will include measures of EBQI exposure/participation (e.g., awareness, hours spent, local buy-in), QI 
orientation/culture (e.g., perceived cooperation among managers/providers/staff, communication 
effectiveness, culture fostering flexibility, participative decision-making),28,29,30 gender sensitivity (e.g., 
awareness, knowledge, attitudes, self-assessment of WH proficiency),31 practice context (e.g., leadership 
norms, organizational readiness to change, job satisfaction, burnout),32,33,34 and provider/staff 
characteristics (e.g., age, gender, race, ethnicity, staff type, clinician type, DWHP, proportion of WVs in 
panel/clinic, board certification, years in VA) (Appendix 5). Data Collection: Consistent with prior 
procedures, we will send pre-notification emails, then email invites with an embedded survey weblink, Q&A, 
and endorsement letters from WHS and local leadership. Email reminders will include a hardcopy option.  
B4d. VA Administrative Data. We will pull secondary data on WV-specific VA quality of care and patient 
experience, in addition to utilization patterns and other administrative data. Sampling Plan:  Each VA data 
source (e.g., chart-based External Peer Review Program [EPRP] quality metrics; Survey of Healthcare 
Experiences for Patients [SHEPs]) has its own sampling plan, limiting our latitude and also statistical power 
as WVs are not currently sufficiently over-sampled at the facility-level. For quality metrics, we will address 
this by applying existing programming code from WHS for generating EPRP-like measures on the 
population of WVs seen in each participating facility from the Corporate Data Warehouse (CDW). We will 
also obtain comparable CDW-derived metrics from the National PACT Evaluation team for PACT Compass-
like measures (e.g., average wait time, non-face-to-face visit use). For survey metrics, we will explore 
aggregated SHEPs data by Group (1/2/3) and over time, and examine contextual differences in the sample 
of WVs seen in the 24 participating facilities compared to VAs of similar facility complexity. Because the 
sample of low-performing facilities has not yet been identified, we cannot as yet estimate available sample 
sizes. Measures:  CDW-derived metrics will include process measures of quality for diabetes and 
cardiovascular disease (e.g., lipid screening) care and intermediate outcome measures (e.g., glycemic and 
lipid control). SHEPs metrics will include access, continuity, coordination, courtesy and overall satisfaction 
with VA care. PACT Compass-like measures include additional measures of access (e.g., average wait 
time, % enrolled in PC-MH integration [PC-MHI]), continuity (% of visits with PACT team providers), 
coordination of care (e.g., emergency room use) and non-face-to-face access (e.g., secure messaging; 
telephone visits; group visits). We will include utilization measures on PC, WH, MH, specialty care visit 
rates, in addition to hospital and emergency room use each year. We will also include area measures (e.g., 
urban/rural location, academic affiliation, facility complexity score). Data Abstraction: We will pull data from 
CDW through VINCI and/or obtain data elements directly from WHS for use in evaluation.  
B5. Evaluation Analysis Plan 
B5a. Qualitative Analyses (Aims #1 and #3). Analysis of KS interviews will initially focus on data 
reduction35 to use findings rapidly and iteratively to inform WHS oversight, support EBQI implementation, 
and prepare deliverables. Specifically, using an approach designed by this component lead (Hamilton) 
(§C1), all interviews will be summarized using a template informed by the interview guide topics (Appendix 

3). Summaries will be organized into matrices to compare and contrast findings across roles, sites, and 
levels (e.g., facility, VISN). In our extensive experience, this data reduction approach is sufficient for 
generating formative feedback reports to sites and presentations to partners and other stakeholders. The 
approach will also help generate a preliminary codebook, to be used for more in-depth analysis of the KS 
interviews using ATLAS.ti, a qualitative data analysis software program that allows for fluid “interaction” of 
data across types and sources. Initially, a top-level codebook will be developed for the baseline interviews 
based on the semi-structured interview guide.36 Using a constant comparison analytic approach, this 
codebook will be elaborated upon based on emergent themes and adjusted as each round of interviews is 
completed.37 Interviews will be compared within facility, across facilities, and over time. These approaches 
and groupings are easily facilitated within ATLAS.ti, which has the capacity to group data in multiple ways 
and which allows for maximum efficiency in navigating a complex narrative dataset. In baseline transcripts, 
we will identify commonly shared knowledge about the structure and delivery of WH care, barriers and 
facilitators to delivering comprehensive care, as well as pre-implementation knowledge of and experience 
with QI. In 12- and 24-month KS interview data, we will identify factors that facilitated and impeded EBQI 
and what changes (if any) were observed in WH care delivery and quality. Consistent with our 
implementation-focused evaluation in the WH-PACT trial, we will explore which EBQI components are of 
particular value in improving care and examine clinic and provider characteristics associated with varying 
levels of EBQI effectiveness and achievement of comprehensive care. 
B5b. Quantitative Analyses (Aim #2). To assess effectiveness of EBQI implementation on comprehensive 
care achievement, we will use data from the organizational surveys (§B4b), provider/staff surveys (§B4c) 
and administrative data on quality and patient experiences (§B4d). Dependent Variables: We will examine 
multiple outcome measures as dependent variables:  1) multiple individual measures of comprehensive 
care achievement, including levels of WH service availability (as noted in VHA Handbook 1330.01), 
integration of and access to gender-specific and MH care, and other related measures that capture different 
domains of comprehensiveness; 2) gender-sensitive care delivery, including organizational and 
provider/staff level measures; and 3) quality of care and patient experience measures. For comprehensive 
care achievement, we will include as dependent variables the individual measures, and we will also 
examine approaches to creating an aggregated ordinal score of the individual measures. We will prioritize 
the final set of dependent variables in consultation with WHS. Independent Variables: The primary 
regressors of interest will be EBQI exposure (i.e., level of implementation) and time. We will examine the 
potential moderating effects of practice context and provider/staff knowledge/attitudes (e.g., determine EBQI 
effects in high vs. low leadership support sites). We will use multiple linear or logistic regression to evaluate 
EBQI effectiveness. Where appropriate we will adjust for covariates, account for clustering of patients by 
site, and mitigate bias due to non-response or loss to follow-up through the use of enrollment/attrition 
weights. Covariates used for adjustment will include patient factors (e.g., facility casemix, proportion of WVs 
seen), provider/staff factors (e.g., DWHP availability, readiness), and organizational factors (e.g., resource 
sufficiency, facility size). Clustering by site will be accounted for by fitting hierarchical regression models 
with random intercepts for the sites (e.g., using SAS PROC Mixed).38  We will evaluate the goodness-of-fit 
of a given regression model using standard diagnostics (e.g., Mallow’s statistic (Cp)).39 To adjust for 
potential non-response bias and loss to follow-up over time for the provider/staff survey samples, we will 
apply enrollment weights using available characteristics of eligible providers/staff and attrition or “inverse 
probability of inclusion” weights estimated using an appropriately specified logistic regression model.40 We 
will use multiple imputation methods to replace missing values among covariates41 with hot-deck methods 
used for imputation as needed.42 We will estimate site-level effects using the hierarchical regression models 
with random intercepts for sites. While our sample of sites (24) is small for the estimation site-level effects, 
EBQI trials of fewer sites have noted significant effects.43 
B6. Potential Obstacles and Solutions  
B6a. Timing of Data Collection. The evaluation’s launch will be reliant on identification of low-performing 
VAs and WHS’s introduction of EBQI plans with facility and VISN leadership. Once navigated, we will brief 
leaders of participating VAs regarding the evaluation, work with locally identified contacts, and begin sample 
development (KS, provider/staff). KS interviews will begin as soon as practicable, as will baseline 
administrative data extraction. Provider/staff surveys, however, will have to undergo review by the National 
Center for Organizational Development’s Organizational Assessment Sub-Committee (OASC) as the 
evaluation meets the criteria of VA employees across 10+ sites. While no FY17 blackout dates have yet 

been published, we will be required to accommodate them. As with prior work, we will also submit the 
surveys for national Union review, with a 30-day review period. 
B6b. Timing and Unit of Randomization. The EBQI contractor was recently approved (8/22/16), enabling 
an early start on site selection. WHS has agreed to allow us to randomly allocate up to eight VA facilities of 
the 24 low-performers in Year 1, up to eight VAs of the remaining 16 in Year 2, with the remainder getting 
EBQI exposure in Year 3. If the EBQI contractor does not meet contract deliverables, evaluation activities 
may be delayed to accommodate their processes. Fortunately, the contractor (Atlas Research) has a strong 
track record with VA and specifically WHS. We plan to proceed with the evaluation even if not all 
contracted-for activities occur, enabling us to still address our evaluation aims. Low-performing facilities 
may also suffer from leadership gaps, provider and staff burnout, and other structural and management 
issues that may complicate their engagement in evaluation activities. We have experience working with 
VHA, VISN and VAMC leaders in multilevel research and evaluation, but will consider site replacement, 
depending on the circumstances.  
B6c. Survey Feasibility and Subject Burden. Based on prior experience, we anticipate high participation 
rates in the KS interviews (96% completed interviews in the WH PACT trial), chiefly in light of the caliber of 
interviewers, the comfort and trust they routinely engender, and the open offer to not record sensitive 
dialogue as needed. Even so, we plan on 30-minute rather than 60-minute interviews to reduce subject 
burden while also increasing feasibility of rapid deliverable generation. We have achieved 85-100% 
response rates (RRs) on our past PC, WH and MH organizational surveys,44,45,46,47 but lower rates for 
subspecialists (e.g., average 60% RR for cardiology, neurology, oncology chiefs).48 We have divided the 
organizational surveys that will be used here into modules to minimize respondent burden for any one key 
informant, while ensuring that the most knowledgeable respondent is asked. We anticipate the lowest RRs 
for providers and staff, given competing demands, but will use approaches similar to those used by the 
VISN 22 PACT Demo Lab (>60% RR), including local leader survey endorsements, incentives/perquisites (if 
available), and aggregated feedback.  
C. RESEARCH TEAM AND RELEVANT EXPERIENCE  
C1. Key Participants. Elizabeth Yano, PhD, MSPH will serve as the proposed WHS/QUERI PEC Director.  
A healthcare epidemiologist, Dr. Yano has conducted numerous VA program evaluations since the 1980s 
and taught program evaluation in the UCLA Executive MPH program for 15 years. She directs the VA 
HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy, which is the only HSR&D 
Center with WH as a primary focus area, and WHRN (SDR 10-012), to accelerate intervention, 
implementation and high-impact (I3) research. This work led to development of the WH CREATE (§A4), for 
which she serves as Director and PI of the WH PACT trial noted above (CRE 12-026). Alison Hamilton, 
PhD, MPH, is a nationally known medical anthropologist and expert in implementation evaluation. She is the 
Center’s Associate Director for Implementation Science and directs the EMPOWER QUERI (QUE 15-272), 
including leadership of a stepped-wedge trial to gender-tailor PC-MHI. She led implementation evaluations 
for collaborative care models for schizophrenia (EQUIP) and depression (Hep-TIDES), and the VISN 22 
PACT Demo Lab, all of which used EBQI methods. She is also a PI of the WHRN (focused on multilevel 
stakeholder engagement) and a CREATE study on determinants of WVs’ attrition, using a convergent 
parallel mixed methods design (CRE 12-019). Danielle Rose, PhD is a health services researcher with 
expertise in organizational theory and organizational determinants of VA quality. She is PI of the WH 
CREATE study of impacts of variations in comprehensive care on WVs’ quality and experience (CRE 12-
038), and leads medical home neighborhood evaluation in the VISN 22 PACT Demo Lab. Sabine Oishi, 
PhD, MSPH, is a psychologist with extensive survey expertise and leadership of partnered VA evaluations 
of WVs’ needs around PC-MHI, trauma, and eating disorders, including national assessments of gender-
sensitive MH care arrangements (MHS) and WVPM needs (WHS). Katherine Hoggatt, PhD, is an 
epidemiologist and VA HSR&D Career Development Awardee focused on substance use disorders (SUD) 
treatment. She co-leads secondary analyses of VA data on alcohol screening, and is PI of a new HSR&D 
IIR to improve validity of VA SUD performance measurement. She is also Co-I leading secondary analyses 
of VA EPRP and SHEP data for the Office of Health Equity (OHE)/PEC at the Center.   
C2. Experience with Partnerships with VA National Program Offices. Dr. Yano’s entire VA tenure (since 
1989) has continuously involved partnered research and evaluation with National Program Offices, starting 
with evaluation of VA’s first demonstration project on VA PC global academic care teams, which informed 

VA’s National PC Directive (1994). She then led the national evaluation of its implementation (1996), and 
launched her HSR&D career by evaluating PC practice features associated with VA quality (MPC 97-012, 
1997-99). She continued her partnership with the VA PC Office and then Office of Quality & Performance to 
evaluate organizational influences on quality using the VHA Clinical Practice Organizational Survey linked 
to VA performance measures (2005-07) and National VA PC Survey (2008-09). She evaluated VA’s quality 
transformation to generate lessons for evidence-based management (IIR 06-087, 2007-10), and used the 
PC practice data to evaluate determinants of colorectal cancer screening (CRS 02-163), smoking cessation 
treatment (IIR 04-308), racial-ethnic disparities in quality (IAA 08-087), and medical home impacts pre-
PACT (IIR 09-082). She led the PC Program Office’s efforts to better understand factors underlying medical 
home readiness by analyzing ACP Medical Home Builder scores for over 850 VA PC practices in 2009 and 
2011. She later co-led the PC Program Office funded VISN 22 PACT Demonstration Lab, treating the entire 
Lab as a stepped wedge trial of EBQI’s impacts on PACT innovations, provider/staff outcomes and quality. 
Dr. Yano’s VA WHS partnership has a comparable track, starting with her Under Secretary-funded survey of 
WVs’ health programs (2001), and an evaluation comparing/contrasting VA and Dept. of Health & Human 
Services’ WH centers (2003). She pioneered use of expert panel methods to engage Program Office 
leaders in the review and policy feedback on HSR&D-funded studies of WVs’ ambulatory care use (GEN 
00-082), impact of practice structure on WVs’ quality of care (IIR 04-036), and impacts of structural changes 
on quality over time (IAE 07-170). Dr. Yano’s partnered work evolved into annual WHS-funded multi-project 
agreements since 2009 for a broad base of evaluation projects, including leadership of a national VA expert 
panel on gender-sensitive comprehensive care, strategies for tailoring PC-MH integration to WVs’ needs, 
among others. WHS also funds a CREATE-level program manager and EBQI work in the WH PACT trial. 
Dr. Yano’s evaluation team has similar partnered evaluation expertise, including Drs. Hamilton (WHS, MHS, 
National Center for Disease Prevention and Health Promotion (NCP), Office of Patient-Centered Care & 
Cultural Transformation [OPCC&CT]), Rose (PC, WHS), Oishi (WHS, MHS), and Hoggatt (OHE). Ms.  
Ismelda Canelo has been project director for partnered evaluations for 17+ years. 
C3. Roles & Tasks of each Evaluation Team Member.  Dr. Yano will direct the PEC, overseeing aims’ 
achievement and working directly with WHS and the EBQI contractor on all evaluation phases. Dr. Hamilton 
will oversee the interviews under Aims #1 and 3, consistent with her collaboration with Dr. Yano on the WH-
PACT trial and VISN 22 PACT Demo Lab, and will serve as Co-PI. She will finalize the KS interview guides; 
guide sample development; lead selected interviews; supervise qualitative staff (Chrystal, Dyer); and 
oversee and participate in coding, quality review and analysis. Dr. Rose will oversee organizational survey 
design, administration, and oversee analysis (Brunner) (Aim #2a). Dr. Oishi will lead the provider/staff 
survey work, with support for web-survey procedures (Simon) and sample development and survey analysis 
(Than) (Aim #2b).  Dr. Hoggatt will lead administrative data acquisition and analysis with the support of a 
statistical analyst (Brunner) (Aim #2c). All leads will participate in report and manuscript preparation in their 
respective areas. Ms. Canelo will serve as the PEC project director, organizing work tasks laid out in the 
Gantt chart (Appendix 6), coordinating project meetings and managing sample development, data 
collection procedures, data cleaning and management, as well as transcription services, with assistance of 
the research assistant (Chow). Work will be coordinated through weekly PEC staff meetings and biweekly 
data collection and later analysis meetings for the qualitative and quantitative teams. 
C4. Local Resources Supporting Evaluation Team.  The PEC will reside organizationally within CSHIIP, 
while benefiting from the infrastructure, professional networks, and research teams of the VA WHRN (Yano, 
Hamilton, Frayne, PIs), WH CREATE (Yano, Director), EMPOWER QUERI Program (Hamilton, Director), 
and the Office of Health Equity PEC (Hoggatt, Co-I). This collaborative context will enable cross-coverage 
and technical support to PEC team members. CSHIIP is heavily leveraged by partner-funded initiatives as 
well as two QUERI Programs (EMPOWER and Care Coordination), providing an unusually strong 
environment of implementation scientists brought together in biweekly work-in-progress seminars. CSHIIP 
has also launched Implementation and Improvement Cores supporting development and testing of 
dissemination tools and EBQI implementation strategies in the context of VAMC and VISN level initiatives.  
D. MANAGEMENT PLAN AND TIMELINE  
D1. Management Plan. The core team (Yano, Hamilton, Rose, Oishi, Hoggatt, Canelo, Brunner) will meet 
weekly, with biweekly Aim-specific meetings focused on evaluation sub-aims (e.g., KS interviews). Monthly 

WHS calls already underway will continue throughout the PEC’s evaluation period, with additional emails 
and calls arranged to meet Partner information needs and obtain WHS review/input. We will also participate 
in regular WHS calls with the EBQI contractor (see Organizational Chart in Appendix 7). Drs. Yano and 
Hamilton will attend 1-2 of the initial in-person EBQI meetings and join initial site engagement calls.  
D2. Timeline/Deliverables.  We will deliver written quarterly reports and briefings to WHS throughout the 
evaluation, consistent with current practices in annual WHS Memoranda of Understanding (MOUs). The 
FY17 MOU has already been finalized and will be signed before QUERI PEC review but for a reduced 
scope of work. Deliverables (below) will be appended to the MOU in Quarter 1 (Q1) if the PEC is awarded.  
FY17:  In Q1, we will work with WHS and the EBQI contractor to finalize operational criteria for classifying 
low-performing VA facilities. We will review the draft KS interview guide (Appendix 3) and provider/staff 
surveys (Appendix 5) with WHS, incorporating WHS and other subject matter experts’ feedback as 
requested. WHS has already approved the organizational survey modules for FY17 (Appendix 4); these 
will be fielded in Q1. WHS will launch facility leadership introductions (including notification of evaluation 
plans), affirming participation, after which we will randomly allocate sites to EBQI (n=8) or waitlisted control 
(n=16). We will support contractor training in EBQI technical specifications in collaboration with WHS, 
document training, and develop a template for use in monitoring EBQI procedures used.   
In Q2, WHS and the EBQI contractor will identify local champions and evaluation contacts for our use in 
identifying KS and launching baseline interviews across all sites. Organizational survey data will be cleaned 
and a report of initial frequencies provided to WHS. Provider/staff surveys will be submitted to OASC when 
released by WHS, after which we will submit them for national Union review; endorsement letters will be 
drafted for WHS review, and adapted to fit local contexts and needs, while we simultaneously obtain and 
begin validating sample lists. We will also pull and analyze the administrative data for participating sites. 
In Q3, we will deliver KS summaries to WHS, while beginning to conduct the more in-depth ATLAS.ti 
analyses. We will finalize the provider/staff surveys following OASC and Union review, program them in 
REDCap, and launch survey administration by the end of Q3. We will conduct an organizational analysis of 
the 24 participating low-performers compared to higher performers of similar complexity.  
In Q4, we will provide results for Aim #1 (barriers/facilitators) to WHS in the form of presentations and 
summary reports for each group of sites. We will finish provider/staff survey follow-up, with concurrent data 
cleaning and quality checks, followed by analysis and reporting to WHS of baseline evaluation findings for 
Aim #2. We will generate an annual report for WHS, summarizing evaluation findings across all data 
sources for FY17. 
FY18:  In Q1, we will randomly allocate the next 8 VA facilities to EBQI, update the KS sample, and launch 
the second wave of KS interviews, focusing first on baseline for newly allocated EBQI VAs. We will restart 
the OASC and national Union review procedures, adapting to FY18 blackout dates, and launch the second 
wave of the organizational surveys. 
In Q2, we will continue with KS interviews, including WHS and EBQI contractor interviews and extending 
the KS sample to accommodate EBQI targets in Group 1 (e.g., MH leads). We will update the provider/staff 
survey sample lists, and work with site-level evaluation contacts on sample validation and updated cover 
letters as needed (i.e., to accommodate leadership changes). By the end of Q2, we will deliver KS 
summaries and organizational survey findings. We will launch the second wave provider/staff surveys in Q2. 
We will also pull a second year of administrative data, and provide WHS with a summary report on changes. 
In Q3, we will finish the provider/staff survey wave, clean the data and deliver an initial summary report. KS 
analysis will continue. KS contrasts between Year 1-2 will be reported, in addition to early findings regarding 
Aim #3 contextual factors and implementation processes based on both waves. Additional organizational 
analyses with linked administrative data for Aim #2 will also be reported (e.g., variations in resource 
sufficiency by large and small low-performers).   
In Q4, organizational changes based on KS interviews and organizational surveys between Years 1 and 2 
overall and by EBQI vs. waitlisted controls will be reported to WHS. We will deliver a second annual report 
for WHS, summarizing findings across evaluation data sources for FY18, with notable FY17 comparisons. 
FY19:  In Q1, the EBQI contractor will launch EBQI in the remaining low-performing VA facilities. We will 
again update the KS sample as needed, and launch the third and final wave of KS interviews and again 

extend the KS sample to accommodate EBQI targets in Groups 1 and 2 that will have had two and one year 
of EBQI, respectively. We will start with the baseline KS interviews for Group 3, followed by the KS follow-
up interviews for Groups 1 and 2, which may carry over into Q2. We will again repeat OASC and Union 
review, and launch the third and final organizational survey wave.  
In Q2, we will complete the KS interviews, and initiate rapid KS analysis. We will update the provider/staff 
sample for the third wave, repeat sample validation steps with participating sites, and prepare endorsement 
letters once again, and launch the survey by the end of Q2. By the end of Q2, we will also provide WHS 
with the third wave of KS summaries and an organizational survey summary in the quarterly report.  
In Q3, we will continue conducting the more detailed qualitative coding for the annual report and completion 
of Aim #3 by Q4. We will deliver a summary of organizational changes over all three survey waves to WHS, 
as well as an initial provider/staff survey summary for the third and final wave. KS contrasts by year will be 
highlighted as well. We will pull and analyze administrative data, aiming for as many months of data as 
possible for the summative evaluation as part of Aim #2. 
In Q4, we will report time trends related to Aim #1 (developmental evaluation), as well as summative results 
for Aim #2 (progress-focused evaluation) and Aim #3 (implementation-focused and interpretive evaluation) 
to WHS. We will include an appraisal of EBQI’s potential utility for related WHS initiatives and its 
sustainability and spread. 
In addition to these partner-focused deliverables, we plan to submit the protocol to Implementation Science 
for publication, as we did with the WH PACT EBQI trial, and pursue other traditional dissemination routes 
through scientific journals and presentations. We will also prepare brief summaries for WHS e-newsletters 
to the field when appropriate, provide briefings to participating sites’ leadership at the VAMC and VISN 
levels on findings over the course of the evaluation, and offer national briefings to other relevant Program 
Offices (e.g., MHS, PC, NCP, OPCC&CT) based on evaluation findings and WHS recommendations and 
requests. With WHS, we will also disseminate findings to WVs, through existing WV Councils affiliated with 
WHRN and the WH CREATE, consistent with our Veteran engagement work.    
 
 

REFERENCES 
                                                          
1 U.S. General Accounting Office.  Actions needed to ensure that female veterans have equal access to VA 
benefits.  Washington DC, Pub. No. GAO/HRD-82-98, 1982. 
2 U.S. General Accounting Office.  VA health care for women, progress made in providing services to 
women veterans.  Washington DC, Pub. No. GAO/HEHS-99-38, 1999. 
3 Veterans Health Administration.  Sourcebook:  Women veterans in the Veterans Health Administration.  
Volume 1:  Sociodemographic Characteristics and Use of VHA Care.  Washington DC, December 2010. 
4 Bean-Mayberry B, Chang CC, McNeil M, Hayes P, Scholle SH.  Comprehensive care for women veterans:  
Indicators of dual use of VA and non-VA providers.  J Am Med Womens Assoc. 2004;59(3):192-7. 
5 Bean-Mayberry BA, Chang CH, McNeil MA, Scholle SH.  Assuring high quality primary care for women: 
Predictors of success. Women’s Health Issues. 2006;16(1):22-29. 
6 McNeil M, Hayes P.  Women’s health care in the VA system:  Another “patchwork quilt.” Women’s Health 
Issues.  2003;13(2):47-49 
7 Bean-Mayberry B, Yano EM, Mor MK, Bayliss NK, Xu X, Fine MJ.  Does sex influence immunization status 
for influenza and pneumonia in older veterans?  J Am Geriatr Soc. 2009;57(8):1427-1432. 
8Farmer M, Rose D, Riopelle D, Lanto A, Yano EM.  Gender differences in smoking and smoking cessation 
treatment:  An examination of the organizational features related to care.  Women’s Health Issues.  
2011;21(4 Suppl):S182-S189. 
 
9 Yano EM, Washington DL, Goldzweig C, Caffrey C, Turner CL.  The organization and delivery of women’s 
health care in Department of Veterans Affairs medical centers.  Women’s Health Issues.  2003;13:55-61. 
10 Yano EM, Goldzweig C, Canelo I, Washington DL. Diffusion of innovation in women’s health care 
delivery: VA adoption of women’s health clinics.  Women’s Health Issues, 2006;16(5):226-35. 
11 Goldzweig C, Parkerton PH, Washington DL, Lanto AB, Yano EM. Primary care practice and facility 
quality orientation: Influence on breast and cervical cancer screening rates. Am J Managed Care, 
2004;10:265-72. 
12 Bean-Mayberry BA, Chang CC, McNeil MA, Whittle J, Hayes PM, Scholle SH.  Patient satisfaction in 
women’s clinics versus traditional primary care clinics in the Veterans Health Administration.  J Gen Intern 
Med. 2003;18(3):175-181. 
13 VA Under Secretary for Health Workgroup on Provision of Primary Care to Women Veterans.  VA Office 
of Public Health & Environmental Hazards (Women Veterans Health Strategic Health Care Group):  
Washington DC, November 2008. [Yano EM, Workgroup Member and Senior Author] 
14 deKleijn M, Lagro-Janssen AL, Canelo I, Yano EM. Creating a roadmap for delivering gender-sensitive 
comprehensive care for women Veterans: results of a national expert panel. Med Care. 2015 Apr;53(4 
Suppl 1):S156-64. 
15 Yano EM.  A partnered research initiative to accelerate implementation of comprehensive care for women 
Veterans:  The VA Women’s Health CREATE.  Med Care. 2015 Apr;53(4 Suppl 1):S10-14. 
16 Yano EM, Darling J, Hamilton AB, Canelo I, Chuang E, Meredith LS, Rubenstein LV.  Cluster randomized 
trial of a multilevel evidence-based quality improvement approach to tailoring VA Patient Aligned Care 
Teams to the needs of women Veterans [Protocol Paper].  Implement Sci.  2016;11(1):101. 
17 Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, Kimmel B, Sharp ND, Smith JL. 
The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 
2006 Feb;21 Suppl 2:S1-8. 
18 Curry LA, Krumholz HM, O'Cathain A, Plano Clark VL, Cherlin E, Bradley EH. Mixed methods in 
biomedical and health services research. Circ Cardiovasc Qual Outcomes. 2013 Jan 1;6(1):119-23. 
19 Brown CH, Wyman PA, Guo J, Pena J.  Dynamic wait-listed designs for randomized trials:  New designs 
for prevention of youth suicide.  Clin Trials. 2006;3(3):259-0271. 
20 Chamberlain P, Brown CH, Saldana L, et al.  Engaging and recruiting counties in an experiment on 
implementing evidence-based practice in California.  Adm Policy Ment Health.  2008;35(4):250-260. 
21 Dillman DA, Smyth JD, Christian LM. Internet, Mail, and Mixed Mode Surveys:  The Tailored Design 
Method (3rd Edition). Hoboken, New Jersey:  John Wiley & Sons, 2009.  
22 Reddy S, Rose D, Burgess JF, Charns M, Yano EM.  The role of organizational factors in the provision of 
comprehensive women’s health care in the Veterans Health Administration.  Medical Care. 2016:[in press]. 
 

                                                                                                                                                                                                                  
23 Soban LM, Yano EM.  The impact of primary care resource sufficiency on prevention performance.  J 
Ambul Care Manag. 2005;28(3):231-243. 
24 Washington DL, Caffrey C, Goldzweig C, Simon B, Yano EM.  Availability of comprehensive women’s 
health care through Department of Veterans Affairs medical centers.  Women’s Health Issues. 
2003;13(2):50-54. 
25 Farmer M, Rose D, Canelo I, Stark R, Schectman G, Rubenstein LV, Yano EM.  Primary care challenges 
facing practices aiming to implement patient-centered medical homes. J Gen Intern Med. 2014;29(Suppl 
2):S555-S562. 
26 Williams ES, Manwell LB, Konrad TR, Linzer M.  The relationship of organizational culture, stress, 
satisfaction, and burnout with physician-reported error and suboptimal patient care: Results from the MEMO 
study. Health Care Manage Rev. 2007;32(3):203-12. 
27 Rubenstein LV, Stockdale SE, Sapir N, Altman L, Dresselhaus T, Salem-Schatz S, Vivell S, Ovretveit J, 
Hamilton AB, Yano EM.  A patient-centered primary care practice approach using evidence-based quality 
improvement:  Rationale, methods, and early assessment of implementation.  J Gen Intern Med. 2014 
Jul;29(Suppl 2):S589-S597.   
28 Garvin DA, Edmondson AC, Gino F. Is yours a learning organization? Harvard Bus Rev. 2008;86(3):109. 
29 Ohman-Strickland, PA, Orzano AJ, Nutting PA, Dickinson WP, Scott-Cawiezell J, Hahn K, Gibel M, 
Crabtree BF.  Measuring organizational attributes of primary care practices: Development of a new 
instrument. Health Serv Res.  2007;42(3p1):1257-1273. 
30 Caldwell DF, Chatman J, O’Reilly CA, Ormiston M, Lapiz M.  Implementing strategic change in a health 
care system” Health Care Manage Rev.  2008;33(2):124-133. 
31 Meredith LS, Wang Y, Okunogbe A, Bergman AA, Canelo I, Darling J, Yano EM.  Primary care provider 
experiences with the patient-centered medical home for women Veterans.  Women’s Health Issues, under 
review. 
32 Maslach C, Jackson SE. The measurement of experienced burnout. J Occup Behav. 1981;2:99–113. 
33 Holt D.  Readiness for organizational change: The systematic development of a scale. J Applied Behav 
Sci.  2007;43(2):232. 
34 Helfrich CD, Li YF, Sharp ND, Sales AE.  Organizational readiness to change assessment (ORCA): 
Development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) 
framework.  Implement Sci.  2009;4:38. 
35 Miles MB, Huberman AM. Qualitative Data Analysis: An Expanded Sourcebook (2nd Edition).  Thousand 
Oaks, CA: Sage Publications. 1994, p. 11. 
36 Fereday J, Muir-Cochrane E. Demonstrating rigor using thematic analysis: A hybrid approach of inductive 
and deductive coding and theme development. International journal of qualitative methods. 2006;5(1):80-92. 
37 Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing 
taxonomy, themes, and theory. Health services research. 2007;42(4):1758-72. 
38 Huber PJ.  The behavior of maximum likelihood estimates under nonstandard conditions.  Fifth Berkeley 
Symposium on Mathematical Statistics and Probability, 1967:221. 
39 Azen R, Budescu DV, Reiser B.  Criticality of predictors in multiple regression.  Br J Math Stat Psychol.  
2001;54(Pt 2):201-225. 
40 Groves RM, Dillman DA, Eltinge JL, Little RJA.  Survey nonresponse.  Wiley Series in Probability & 
Statistics.  New York:  John Wiley & Sons, Inc.  2002. 
41 Arnold AM, Kronmal RA.  Multiple imputation of baseline data in the cardiovascular health study.  Am J 
Epidemiol.  2003;157(1):74-84. 
42 Gmel G.  Imputation of missing values in the case of a multiple item instrument measuring alcohol 
consumption.  Stat Med.  2001;20(15):2369-2381. 
43 Hamilton AB, Cohen AN, Glover DL, Whelan F, Chermerinski E, McNagny KP, Mullins D, Reist C, 
Schubert M, Young AS. Implementation of evidence-based employment services in specialty mental health. 
Health Serv Res 2013;48(6 Pt 2):2224-44. 
44 Yano EM, Simon BF, Lanto AB, Rubenstein LV.  The evolution of changes in primary care delivery 
underlying the Veterans Health Administration’s quality transformation.  Am J Public Health. 
2007;97(12):2151-2159. 

                                                                                                                                                                                                                  
45 Washington DL, Bean-Mayberry B, Mitchell MN, Riopelle D, Yano EM. Tailoring VA primary care to 
women veterans:  Association with patient-rated quality and satisfaction.  Women’s Health Issues.  
2011;21(4 Suppl):S112-S119.   
46 Cordasco KM, Zephyrin LC, Kessler CS, Mallard M, Canelo I, Rubenstein LV, Yano EM.  An inventory of 
VHA emergency departments:  Resources and processes for caring for women.  J Gen Intern Med. 
2013;28(Suppl 2):S583-S590.   
47 Oishi SM, Rose D, Washington DL, Bean-Mayberry B, Yano EM.  Organization of women’s mental health 
services in the Veterans Health Administration. Women’s Health Issues.  2011;21(4 Suppl):S130-S137. 
48 Oishi SM, Marshall N, Hamilton AB Yano EM, Lerner B, Scheuner MT.  Assessing multilevel determinants 
of adoption and implementation of genomic medicine:  An organizational mixed-methods approach.  Genet 
Med. 2015 Nov;17(11):919-926.