Controlled Trial of Tele-Support and Education for Womens Health Care in CBOCs
Conditions: Metrorrhagia, Urinary Incontinence, Remote Consultation, Videoconferencing, Women's Health, Menopause
Sponsor: VA Office of Research and Development

Version 12-15-15 
 
1 
 
Controlled Trial of Tele-Support and Education for Women’s Health Care in CBOCs 
 
 
PI: Donna L. Washington, MD, MPH;  
Co-PI: Kristina M. Cordasco, MD, MPH, MSHS 
 
NCT01918072 
 
 
 
 
 
Study Protocol and Statistical Analysis Plan 
 
Revision Date: 12/15/15 
 
 
 

Version 12-15-15 
 
2 
 
Background 
Women Veterans are a rapidly growing minority among VA patients.  While entitled to receive care equivalent 
to their male counterparts, achievement of this goal is often complicated by a VA primary care workforce that 
has limited exposure to caring for women and, for patients in community-based outpatient clinics (CBOCs), 
significant distances to sources of women’s health (WH) expertise.  This proposal aims to test a provider 
education and interactive communication intervention for improving WH care in CBOCs.   
It is particularly challenging to deliver comprehensive WH care (i.e. the full spectrum of primary, specialty, and 
mental health care, both gender-specific and gender-neutral) in CBOCs.  CBOCs were developed to improve 
access to VA primary care (PC), reducing distances Veterans need to travel to receive basic services.1 They 
were never set up to handle the complexity of services needed by the rapidly growing and younger population 
of new women Veteran users (e.g, reproductive health care, post traumatic stress disorder, sequelae of military 
sexual trauma).2  Such care requires expertise and resources that are difficult to incorporate into small 
practices.3,4  Concordantly, the majority of CBOCs currently do not have women’s health specialty clinics and 
struggle to offer on-site PC services specific to women’s needs.5-8 Therefore, CBOC patients often must travel 
significant distances to VA Medical Centers (VAMCs) or use healthcare settings outside the VA to access WH 
expertise.9  They also experience care fragmentation due to provider and site discontinuity.  Further, with 
CBOC PC providers’ (PCPs) small caseloads of women Veterans, and therefore limited ongoing exposure to a 
depth and breadth of WH issues, it is difficult for CBOC providers to maintain their WH knowledge and skills.10 
Recent data demonstrates that women Veterans give lower ratings for the WH skills of PCPs at small caseload 
CBOCs, compared with PCPs at large VA sites with WH centers.8 
Although we know that improved provider knowledge and skills are essential for provider behavior change (and 
therefore, improved care), we also know that they generally are not sufficient.26  However, mechanisms that 
build on provider knowledge and skills have been shown to be effective in promoting and supporting provider 
behavior change.26  One such mechanism is interactive communication, defined as the “timely, two-way 
exchange of pertinent clinical information, directly between primary care and specialist physicians.”27   
Interactive communication is common and effective in improving care in settings where it is supported. PCPs 
highly value direct interaction with specialists (rather than communication via chart notes).28   When such 
interactions are done informally, they are referred to as “curbside consultations.”  A cross-sectional survey of 
PCPs found that 70% had participated in at least one curbside consultation in the previous week.28 A meta-
analysis showed that diabetes and mental health interventions that improve PCP-specialist interactive 
communication improve patient outcomes.27  
In recognition of these issues, the VA launched efforts aimed to improve WH knowledge and skills of VA PCPs, 
with an emphasis on CBOCs.  First, the VA established “Designated Women’s Health Providers” (DWHPs) in 
CBOCS to consolidate the care of CBOC women Veterans to 1-2 providers, thereby increasing these 
providers’ WH caseloads.  Second, the VA instituted intensive one-time training opportunities—“WH mini-
residencies”—for DWHPs.11  However, substantial evidence demonstrates that one-time medical education is 
not sufficient, as knowledge attenuates over time.12 Serial education re-enforcement over time is needed to 
produce and maintain long-term gains in knowledge or skills.13 Furthermore, education alone is not sufficient to 
change provider behavior, and is not likely to improve care quality.  Additional interventions, such as 
enhancements in communication between PCPs and specialists, are necessary to achieve quality gains.14 
 
Research Aims 
We propose to test an intervention for DWHP Support to improve CBOC delivery of comprehensive WH care. 
DWHP Support will be a combination of: 1) advanced WH serial patient-based education that exposes 
DWHPs, over time, to a depth and breadth of WH cases and issues; and 2) interactive communication 
between CBOC DWHPs and VAMC-based WH specialists for “just in time” support of  DWHP WH care.  It will 
be technology-supported and delivered non-face-to-face.    
This proposal features Electronic Consults and DWHP-Specialist Clinical Tele-Videoconferencing (CTV), two 
potential mechanisms for improving and maintaining DWHP knowledge and skills and increasing 
communication between CBOC-based DWHPs and WH specialists. Technology-based care models have been 
proposed as being potentially efficient, cost-efficient (or even cost-reducing) mechanisms of overcoming 
specialty physician shortages in certain geographic regions (e.g., rural areas), providing continuing medical 

Version 12-15-15 
 
3 
 
education, and improving care quality.29-31   Two such models, electronic-consults (“e-consults”) and grouped 
PCP-Specialist Clinical Tele-Videoconferencing (CTV), are foci of this proposal.   
Electronic consults (“e-consults) are asynchronous PC–specialist communication via an electronic medium 
(e.g., secure messaging, e-mail, or computerized note entry).32  It formalizes and allows for long-distance 
“curbside consultations.”  E-consults allow PCPs to obtain access to the clinical expertise of a specialist 
without the patient having a separate appointment with the specialist, thereby eliminating patient travel and 
wait times. The PCP submits a consultation request for a defined clinical question pertaining to a patient. The 
specialist reviews the patient’s chart, as needed, and provides an answer electronically, requests more 
information, or advises that the patient should receive an in-person consultation (so that the specialist can 
perform an enhanced examination or perform a diagnostic or therapeutic intervention).   
Another complementary approach that has been used successfully to build PCP knowledge and skills and enhance 
provider communication is group PCP-specialist clinical tele-videoconferencing (CTV) sessions. These sessions 
provide participating PCPs with serial patient-based advanced education by exposing them, over time, to a depth 
and breadth of cases and issues within a defined clinical area. Prior to each session, PCPs submit data on patients 
about whom the PCPs are seeking diagnostic and/or therapeutic management advice from a multidisciplinary 
specialist team.  The team reviews the submitted information prior to the session.   During the scheduled session, a 
specialist team at a “hub” facility and PCPs at multiple “spoke” locations see and hear each other (via CTV) and 
engage in a real-time discussion about the management of the submitted patient cases.  The specialist team, in 
addition to giving recommendations, discusses the evidence and/or rationale behind the recommendations, thereby 
increasing the session’s educational value.  The PCP who submitted the consult, as well as other PCPs participating 
in the session, interact with the specialist team, asking clarifying questions.  The specialist team also provides a 
brief didactic relevant to the cases discussed.  Such sessions have been shown, in one setting and condition, to 
improve PCP capacity to manage complex patients and positively affect patient care outcomes.  Specifically, 
University of New Mexico implemented such a program for management of patients with hepatitis C, and, in a 
controlled trial, showed that PCPs who engaged in group CTV sessions with hepatologists improved patient access 
and achieved clinical outcomes comparable to those of the hepatologists.38,39 
We hypothesize that DWHP Support will improve the quality and efficiency of WH care in CBOCs.  
Our specific aims are:   
Aim #1: To evaluate the effect of DWHP Support on WH care quality and efficiency, using a 
stepped wedge design; 
Aim #2: To explore the impact of DWHP Support in changing DWHP behavior and self-rated 
WH knowledge, skills, and self-efficacy; 
Aim #3: To assess attitudes about DWHP Support and its use, specialist time for its 
implementation, and other features that could influence DWHP Support’s effectiveness, 
sustainability and spread; 
Aim #4: To develop tools to measure quality of WH care in VA.  
This proposed research project was developed in partnership with VA Patient Care Services’ Women’s Health 
Services (WHS) and Specialty Care Services (SCS), VISN 22, and VA Greater Los Angeles Healthcare 
System.  It will impact VA practice and policies in women Veterans’ health care, access, and quality (HSR&D 
priority areas) as well as other specialty areas.  Further, it will advance science in both WH care quality 
measurement and the delivery of health care through non-face-to-face interactions.   
 
Research Design and Methods 
 
Study Design and Overview: We propose to conduct a prospective, modified stepped wedge trial of DWHP 
Support for DWHPs in healthcare systems that participate in Gynecology SCAN-ECHO and have gynecology 
e-consults. To accommodate facility and VISN needs, we will work with our facility and women’s health 
partners to stage the assignments for the stepped wedge rollout (i.e., it will not be randomized).  We will 
introduce the intervention throughout one healthcare system at each step, with the intervention being directed 
to all DWHPs in that healthcare system. 
 
 We will use a mixed methods analytic approach to measure the intervention’s effect, while also evaluating the 

Version 12-15-15 
 
4 
 
implementation processes and use of the intervention. Therefore, while testing DWHP Support effectiveness, 
this study will also advance our understanding of the context for its implementation, which can support its 
broad implementation in VA, as well as inform future trials. An advantage of our modified stepped wedge 
approach is that our intervention will occur in the “real world”, i.e., not in a highly controlled experimental 
setting; this enhances the potential external validity of the intervention and thereby increases the possibility 
that the intervention could feasibly be spread to other VISNs, should it prove effective. 
 
Setting and Participants: DWHP Support will be implemented in healthcare systems in VISN 22 and at other 
VA sites nationally. VISN 22 encompasses Southern California and the Las Vegas area. It has a racially and 
ethnically diverse women Veteran population, with 68% non-Hispanic White, 5% non-Hispanic Black, 21% 
Hispanic, and 6% other racial/ethnic groups.45  The participants will be DWHPs in those settings.   
DWHP Identification: We will identify DWHPs in two ways.  First, we will contact the Women’s Health Medical 
Director for each healthcare system, to obtain a current list of DWHPs.  Second, in FY12, with funding from the 
WHS (the national WH strategic planning group), the Women’s Health Evaluation Initiative (WHEI; PI: Frayne) 
is generating and validating a list of DWHPs by VA site. To identify our sample of DWHPs, at the start of this 
study, we will use these data sources to obtain the list of participating site DWHPs and their provider ID 
numbers. We will adhere to VA policy regarding secure data transfer to obtain this list. 
Patient Identification: To identify patients who received care from  participating site DWHPs, we will use the 
Corporate Data Warehouse (CDW) – national patient-level administrative data for VHA-provided health care 
encounters This was data previously contained in the VHA Medical SAS (MedSAS) Outpatient Events (SE) 
and Visits (SF) datasets.47 We will obtain this data through VINCI, VA’s platform for accessing CDW for 
research. 
 
Intervention: DWHP Support will be comprised of two components. [1] Gynecology (Women’s Health) E-
consults: This is asynchronous PC consultation primarily via electronic communication; the telephone may also 
be used if needed. The DWHP will electronically message a gynecologist for expert guidance on WH issues 
that are within the scope of practice for the PC setting (i.e., medical and gynecologic in the Handbook 
1330.01), but about which the DWHP needs additional guidance. The PC consultation component, adapted 
from the SCS’s e-consult program, formalizes and allows for long-distance “curbside” consultations that are 
often used by PCPs in larger, multi-disciplinary settings. All e-consults will be answered within three business 
days. The gynecologists, who will have dedicated time protected, through clinical funds, to answer these 
consults, will review the patient’s chart as needed, and provide an answer electronically. The DWHP is 
responsible for acting on the recommendations and/or submitting a request for an in-person consultation, if this 
is the recommendation or they are not comfortable implementing the care plan.  The DWHP can also 
electronically ask the gynecologist clarifying or additional questions, if needed.  If necessary, phone calls 
between the DWHP and gynecologist will be arranged.  [2] SCAN-ECHO:  In these monthly group interactive 
case consultation conferences, held via CTV, cases are submitted to the multidisciplinary team by DWHP 
participants, with defined clinical questions pertaining to a patient. Cases will be selected by the gynecologist 
based on their educational merit; those not selected will be answered electronically or by telephone (and 
thereby become e-consults). The gynecologist discusses the case with the patient’s DWHP, with other DWHPs 
observing. For each session, the gynecologist also gives a 15-minute didactic related to the discussed case. 
Each tele-video conference thereby, in addition to providing expert consultation for the care of that patient, 
educates all the participating DWHPs on their care of patients with similar conditions. Participants receive 
continuing medical education (CME) credits for each SCAN-ECHO session.  
 
Usual Care: Usual care in VA CBOCs includes availability of patient referral for in-person specialist 
consultation (typically at the parent VAMC); access to web-based tools such as UpToDate,50 a clinical decision 
support system to which the VA subscribes; and access to monthly WH cyber-seminars.  
 
Overview of Data Sources, Data Collection Time Points, and Key Measures: The data sources, data 
collection time points, and key measures to achieve our specific aims are discussed by specific aim, and  

Version 12-15-15 
 
5 
 
summarized in Table 1 below.  
 
Table 1: Data sources, data collection time points, and key measures by research specific aim 
 
Data sources 
Data collection 
time point(s) 
Key Measures 
Aim 1: Effect of DWHP Support on care quality and efficiency  
Electronic medical records 
Baseline & 
throughout 
implementation* 
Quality of Care: Proportion of care that is guideline-adherent; 
Efficiency of Care: Proportion of in-person specialist visits that 
are preventable 
Aim 2: Impact in changing provider behavior, self-rated WH knowledge, skills, and self-efficacy  
E-consult referring provider brief 
survey  
After each e-
consult 
Change in patient management for patient who is the subject of 
the SCAN/e-consult; change in provider knowledge 
DWHP self-assessment tool  
Baseline and 
during 
implementation 
DWHP self-rated WH knowledge, skills, and self-efficacy 
Aim 3: Attitudes about DWHP Support, its use, specialist time for implementation, other features  
Semi-structured interviews with 
DWHPs 
Baseline and 
during 
implementation 
Familiarity with SCAN-ECHO programs. Attitudes and beliefs 
regarding intervention; motivation and opportunity for its use. 
Familiarity with DWHP Support (reach and adoption) and other 
SCAN-ECHO programs; Influence of acceptability, usability, 
barriers and facilitators on use; Motivation and opportunity for 
sustained use 
Aim 4: Develop tools to measure quality of WH care in VA 
Clinical practice guidelines 
To commence  at 
start of study 
Not applicable [chart abstraction templates will be produced] 
*Electronic records from the baseline time periods will be assessed retrospectively  
 
Specific Aim #1: To evaluate the effect of DWHP Support on WH care quality and efficiency using a stepped 
wedge design with sites participating in DWHP Support.  
We hypothesize that DWHP Support will improve the quality and efficiency of WH care in CBOCs. We expect 
that these effects may be influenced by baseline characteristics of the DWHPs (e.g., time in practice), patient 
conditions and characteristics, and practice characteristics, as well as DWHPs’ use of the intervention.  For this 
aim, our analytic sample will be limited to sites participating in DWHP Support outside of GLA, given that 
DWHP support was piloted in GLA and therefore does not have a recent “control” period (i.e., GLA was 
contaminated by the pilot).  We will, however, use GLA data to pilot our electronic medical record (EMR) data 
abstraction procedures. We will also collect GLA e-consult data because this data were not contaminated by 
the piloting of SCAN-ECHO at GLA.   
 
Dependent Variables: The main outcome measures will be quality of care defined by adherence to practice 
guidelines. We will assess quality of care and efficiency of care using the VA electronic medical records (EMR) 
of all female patients seen by DWHPs during the data collection period. With the modified stepped wedge 
design, each participating site healthcare system is contributing data to both intervention and control periods, 
thereby addressing comparability concerns.  We expect to identify multiple episodes of care per DWHP over 
the combined data collection periods, where we define an episode of care as the care initiated by the DWHP 
for a WH condition, irrespective of whether it occurs over one or multiple encounters, and is face-to-face or 
not.54   
Electronic medical record (EMR) data abstraction: To assemble the relevant data from the EMR using the VHA 
Health Information Access (HIA) program’s Data Use Agreement protocols, we will use the Corporate Data 
Warehouse (CDW), accessed through VINCI and discussed above and, as needed, the Compensation and 

Version 12-15-15 
 
6 
 
Pension Record Interchange (CAPRI), which provides direct access to VistA medical records (Computerized 
Patient Record System, CPRS) from multiple VistA systems with a single access/verify code51-53 Through the 
VINCI system, we will obtain from CDW lists of all patients seen within participating site healthcare systems 
within our data collection periods.   We will maintain these lists in VINCI workspace.   From these lists, we will 
identify patients seen by DWHPs so that we can describe characteristics of their patient populations.  We will 
then identify the subset of DWHP female patients with the study tracer conditions for quality assessment. To 
do this, we will use ICD9 codes, CPT codes, and, for some conditions, consult requests, medications 
prescribed, and/or laboratory tests ordered, in combination with the previously described DWHP list.   For 
these patients, we will collect patient demographics, provider(s) associated with each encounter, clinic stop 
codes (which identifies the clinic type), and all EMR notes, orders, consult requests, and test results (e.g., 
laboratory, pathology, radiology)  for our data collection periods. This data will be downloaded to the VA GLA 
Research Server. 
EMR de-identification for blinded review: Nurse abstractors and implicit reviewers will be blinded to whether the 
episodes of care that they are reviewing are from the intervention or control periods. During an initial training 
period, all records will be reviewed by supervising research staff. After this initial training period, for the 
duration of the data collection period, records will be randomly spot-checked to verify adherence to the 
protocol. 
Quality of Care – Guideline-adherent care classification procedure: We will conduct medical record abstraction 
to identify a sample of episodes of care for tracer clinical conditions.  We will categorize care as guideline-
adherent versus not guideline-adherent. The tracer conditions will be evaluation/management of: abnormal 
uterine bleeding;55-58 menopause management, and urinary incontinence.69 To determine if the care is or is not 
guideline-adherent, we will develop quality measurement tools.  The methods we will use to develop these 
tools are described below in Specific Aim #4. 
Trained chart abstractors will review each chart to complete the chart abstraction templates with the 
information requested. They will not be making a direct determination of care quality or efficiency, but rather 
obtaining the information for a computerized algorithm that will do this.  A WH specialist will be available to 
them to answer questions that may arise. We will develop a database (using REDcap software) so that data 
may be entered directly into REDcap via a study computer. Senior research staff who are clinicians will 
randomly spot-check chart abstraction data, throughout the data collection period, to verify adherence to study 
protocols. 
 
Independent Variables: DWHP characteristics may influence the quality and efficiency of the care that they 
provide, as well as their response to the intervention. We will extract DWHP characteristics from the Personnel 
and Accounting Integrated Data System (PAID) Master File, which contains the human resources data for all 
VA employees, including provider gender, age, education and licensure (MD, NP, PA), and medical specialty.71   
We will obtain characteristics of patients seen by the DWHP in the study period from VA administrative data. 
This data will include: female caseload (# of uniques) and workload (# of encounters), female percent of total 
caseload and workload. We will derive percent of female workload by age category (<45, 45-64, 65+, 
corresponding to reproductive age group, age for emergence of chronic conditions, and older). 
 
Aim #1 Statistical Analysis: We will calculate descriptive statistics of the proportion of CBOC women’s 
primary care that is guideline-adherent. We will examine each of these descriptive statistics by clinical 
condition, and conduct z-tests for differences in proportions between conditions. We will also evaluate a series 
of logistic regressions to test for association of the composite measure of quality with each of the various 
independent variables in order to build a predictive model. Because of the hierarchical nature of the data 
(episode of care within patient within DWHP), a generalized linear model (GLM) approach using a logit link will 
be used to estimate and test the significance of the regression coefficients. 
For the analysis of the intervention effect over time, we will conduct a modified stepped-wedge analysis. This 
analysis takes into account the hierarchical nature of the data, with episodes of care occurring over time, and 
being nested within DWHPs. In the stepped wedge analysis, calendar time is divided into steps, corresponding 
to the time periods when providers are initially exposed to the intervention.  

Version 12-15-15 
 
7 
 
 
Sample Size and Power Calculations: The power calculations are predicated on the goal of evaluating the 
performance of DWHPs through the episodes of care they see. There are two levels of clustering: DWHPs 
within healthcare systems and episodes of care within DWHPs. We are making the assumption that there is 
little or no correlation between the DWHPs. As a result, we performed the calculations via the single level of 
clustering within DWHPs, with the episode of care as the unit of analysis. We will assume two values for the 
intra-cluster correlation coefficient (ICC): 0.03 and 0.05. The latter is relatively large compared to results that 
have been reported in the literature.72 The calculations assume a two-tailed 5% significance level and power of 
80% to detect a difference between the study arms (with the study arm referring to intervention versus control, 
and including all data collection in each of those study periods, respectively). The statistical test involved is a 
simple z-test for the comparison of two proportions, but the actual analysis will be more sophisticated using 
covariates and adjusting for the actual number of episodes seen by a given DWHP.  Thus, it can be assumed 
that the sample size calculations presented here are conservative. 
With a baseline estimate from our preliminary data that 67% of episodes of care are fully proficient, then with 
25 episodes of care per DWHP, we could detect small to medium effects of the intervention (see Table 2). If 
the baseline proficiency was 80%, then we could detect a medium size intervention effect, whereas if the 
baseline proficiency was 50%, we could detect a very small effect. If we increased the number of eligible 
episodes of care from which to measure proficiency to 50 per DWHP, then with a baseline proficiency of 67%, 
we could detect a small intervention effect. The number of available DWHPs (approximately 20 per study arm) 
and episodes of care provides us with >80% power.  
 
Table 2: Number of DWHPs needed to detect the effects in each scenario with 80% power 
 
 
 
 
 
 
*study arm refers to intervention versus control, and includes all data collection  
 
Aim #2:   
We hypothesize that DWHP Support will increase DWHPs’ practice expertise, and thereby modify their 
practice behavior, resulting in improved quality and efficiency (the focus of Aim 1). To test this hypothesis, we 
will collect process outcomes for the components of the intervention and their effects on patient care with a 
SCAN/e-consult referring provider survey (here after referred to as e-consult survey).  
 
Provider Behavior: e-consult referring DWHP brief surveys: We will obtain the names of participating 
providers who initiate gynecology e-consults by obtaining from CDW, through VINCI, a list of all participating 
site gynecology e-consults.  After each e-consult has been completed, we will email a request for the providers 
who initiated the consult to complete a brief survey about the encounter, including if and how it impacted their 
management plan. More specifically, we will ask: to what extent the consult altered the diagnostic, patient 
counseling, and/or treatment plans for this patient; the provider’s confidence in their ability to implement the 
recommended diagnostic, patient counseling and/or treatment plans; how much, if at all, the provider learned 
something from the consult; and how, if at all, the consult affected the provider’s plans to refer the patient for 
an in-person consultation. For the referring provider surveys, we will use the question about effect on the 
provider’s plans to refer the patient to separately estimate number of avoidable consults averted by the 
availability of e-consults. An open-ended question, asking for additional comments, will also be included. The 
e-consult referring provider survey will be administered online (using Inquisite software) and responses will be 
confidential.  
 
% of episodes of care with fully 
proficient care quality and efficiency 
    Control    Intervention (effect size) 
25 episodes per 
DWHPper study arm* 
ICC=0.03      ICC=0.05 
50 episodes per DWHPper 
study arm* 
ICC=0.03      ICC=0.05 
80% 
90% (medium) 
14 
18 
10 
14 
67% 
83% (medium) 
8 
10 
6 
8 
67% 
77% (small) 
22 
28 
16 
22 
50% 
63% (very small) 
16 
20 
12 
16 

Version 12-15-15 
 
8 
 
Self-rated WH knowledge, skills, and self-efficacy: DWHP self-assessment tool: Prior to the start of DWHP 
Support, or early in its implementation, and again after at least 12 months of use by study providers, we will 
administer a self-assessment tool to all DWHPs to assess their self-rated WH knowledge, skills, and self-
efficacy in delivery of WH care.  The self-efficacy assessment is modeled after the WVHSHG mini-residency 
self-evaluation tool, which examined 15 clinical content areas using a 5-point Likert scale. The 12-month self-
efficacy assessment will be expanded to add items for clinical content areas covered during the SCAN 
sessions. We will also assess perceived change in self-efficacy in WH delivery, and perceptions about 
influences on this change. We will generate summary descriptive statistics for each of the data fields assessed. 
 
Aim #3: To assess attitudes about DWHP Support and its use, specialist time for its implementation, and other 
features that could influence DWHP Support’s effectiveness, sustainability and spread.  
Because we are interested in affecting CBOC DWHP behavior—as well as knowledge, skills, and self-
efficacy—it is necessary to develop an understanding of the context in which the changes are to occur and the 
resource needs that could impact maintenance of DWHP Support. Accordingly, our evaluation includes 
baseline assessment of DWHP providers, a process evaluation of the intervention, and a post-intervention 
assessment. We do so guided by the RE-AIM framework – a comprehensive framework designed to evaluate 
the impact of multilevel interventions in real-world settings.74  RE-AIM has as a central tenet that the ultimate 
impact of an intervention is due to a combination of its reach, effectiveness, adoption, implementation, and 
maintenance. Aim 3 examines reach, adoption, and implementation of DWHP Support. Reach refers to the 
participation rate among those approached, which we will document. Barriers to participation (e.g., scheduling) 
may impact reach, and will be assessed. Adoption concerns the percentage and representativeness of settings 
that will conduct a given program. Adoption is impacted by several organizational factors, e.g., level of 
resources and expertise required, and how similar the proposed intervention is to current practices. We will 
measure potential variables influencing adoption of practice standards. Hypothesized effects of DWHP Support 
include reduced need for in-person specialty consultation, and fewer in-person visits associated with specialty 
consultations; therefore, we will calculate the number of SCAN/e-consults needed to avert one in-person 
specialist visit. Implementation refers to intervention integrity, or the quality and consistency of delivery. We will 
conduct a process evaluation to provide a context for the effectiveness findings, and to define resources and 
other factors that influence implementation.  
Semi-structured interviews will be conducted with all DWHPs to assess their attitudes and beliefs regarding the 
proposed intervention (platforms and content), and motivation and opportunity for its use.  Use of other care 
delivery and educational supports and participant characteristics will be assessed during the interview. We will 
also assess prior exposure to SCAN-ECHO clinics and electronic consultations for other clinical areas.  All 
interviews will be digitally recorded and professionally transcribed. For respondents who are interviewed after 
the start of the intervention, we will ask about exposure to SCAN and e-consults for other clinical areas.  With 
an eye toward potential maintenance of the intervention, participants will be asked to provide 
recommendations for sustainability and spread.   
 
Qualitative Data Analysis: All digital recordings of semi-structured interviews will be sent to Keystrokes, a VA-
approved medical transcription company, via a secure website for verbatim transcription. Transcripts will be 
reviewed and edited for accuracy by study staff. Analysis of the qualitative data will be conducted using 
ATLAS.ti, a qualitative data analysis software program that allows for fluid “interaction” of data across types 
and sources. Initially, a top-level codebook will be developed for the baseline interviews based on the interview 
guide.75  Using a constant comparison analytic approach, this codebook will be elaborated upon based on 
emergent themes, and it will be adjusted as each round of interviews is reviewed. Interviews will be compared 
across DWHPs, and over time. 
In the pre-intervention transcripts (i.e., baseline assessment), we will identify commonly shared knowledge, 
attitudes, and beliefs related to the intervention’s potential for effectiveness as well as motivation and 
opportunity to use the intervention. We will synthesize this information to create baseline summaries. In post-
intervention interview data, we will assess adoption, focusing on factors facilitating and impeding 
implementation of the intervention, and strengths and weaknesses of the intervention as implemented. In 
conjunction with logs that will be maintained during the intervention, we will assess the extent to which the 

Version 12-15-15 
 
9 
 
intervention was utilized by whom (i.e., “reach”), and we will identify which components and/or content was 
most salient for participants. We will examine whether (and why) this differs among DWHPs.  
 
 
Aim #4: To develop tools to measure quality of WH care in VA.  
To complete the work in Aim 1, we will need to develop tools to measure the quality of WH care in VA. VA 
monitors two gender-specific quality indicators as part of its annual External Peer Review Program 
performance monitoring – timely receipt of breast and cervical cancer screening. Though relatively easy to 
measure, there is little variation in these quality indicators across VA sites. 76 Therefore, we have selected 
alternate gynecology tracer conditions for our quality assessment.  Explicit evidence-based quality guidelines 
exist for each of the selected tracer conditions (abnormal uterine bleeding;55-58 counseling in patients 
prescribed teratogenic medications, contraception counseling and management;60,61 preconception 
counseling;66-68 menopause management, and urinary incontinence.69). The goal of Aim 4 is to operationalize 
these guidelines into quality measurement tools. The process for achieving this aim consists of four steps – 
translating guidelines, defining CPRS data sources and elements, developing EMR abstraction forms, and 
creating computer programs based on specifications. 77  Serial testing and revising will be done within each 
step.  
 
Translating Guidelines:  We will first construct, modify, or augment the algorithm for each guideline.  Next, 
we will translate each guideline into a series of binary “if/then” statements.78 We will also define terms as 
needed for clarifying these if/then statements. Two investigators will independently draft these “if/then” 
statements for each guideline and then reconcile their statements, discussing differences, to form for each 
guideline one draft series of if/then statements. To validate these if/then statements, these drafts and the 
guidelines and algorithms associated with them will then be given to members of our quality measurement tool 
validation, who will comment on our interpretation of the guidelines (to create the algorithm and specify terms) 
and our if/then statements, and then back-translate them to reconstruct quality guidelines.79 If there are 
discrepancies between the versions reconstructed by our panel members, and the actual guidelines, we will 
investigate the source of these discrepancies and modify/clarify our if/then statements as needed. We will do 
this serially until we have a final set of validated if/then statements and associated definitions.   
 
Defining CPRS data sources and elements:  We will next define the data sources and elements for each 
if/then statement.77 For each source, we will define the elements to abstract.  To reduce the complexity of the 
quality assessment process, we will choose a subset of if/then statements for selected aspects of care with 
data sources that are more easily assessable in the electronic medical record (rather than attempt to collect 
data sources for the universe of if/then statements). Once we define all the data sources and elements for 
each if/then statement within a series, we will test them by reviewing the EMRs of a sample of approximately 
10-20 patients with the conditions to verify our data sources and elements, revising as needed.  In order to 
identify a sample of 10-20 patients with each selected  condition, the VINCI data manager will pull the 
electronic medical records of VA Greater Los Angeles women patients from  FY09 to FY13 (approximately 
40,000 records), as described above. These records will be stored on the VINCI server. Using ICD-9 and CPT 
codes, while working on the VINCI server platform, we will identify the 10-20 patients with the conditions and 
download these electronic medical records to the VA GLA Research Server to develop the chart abstraction 
forms.     
 
Developing chart abstraction forms: We will then develop chart abstraction forms for each guideline.80  We 
will develop a series of questions that query for these data elements, specifying data sources. We will develop 
rules for each question. For example, again referring to figure 4, we will develop a rule for what needs to be in 
the chart to indicate that the provider “referred the patient to a surgeon” (e.g., submitted a consult request to 
General Surgery).  The form, and its data elements, and abstraction process instructions  will be developed 
iteratively   with two nurse abstractors, we will test our form, rules and guidelines. With this test, we will ask the 
nurse abstractors to first review the form and guidelines, letting us know where it needs to be revised for 

Version 12-15-15 
 
10 
 
clarity. After we make this initial revision, we will ask the nurse abstractor to use the form to abstract data from 
the EMRs of 10-20 patients (identified via the same process as the initial 10-20 patients, using ICD9 and CPT 
codes while working on the VINCI server platform), asking questions when needed and noting difficulties. We 
will revise the form based on this feedback and then re-test serially until all problems are resolved.    We will 
also develop training materials for our nurse abstractors. 
 
Creating computer programs to operationalize specifications: Finally, we will create a program, using 
STATA 11, to operationalize the specifications and algorithms. As needed by our if/then statements, this will 
include variable construction, such as converting linear variables (e.g. age) to bivariates/categorical (e.g., >30 
years of age or <30). Then we will develop programs to operationalize the algorithms formed with the if/then 
statements to make a final determination of whether or not the care was guideline adherent (as a yes/no 
determination). For each episode of care, all relevant if/then statements must reflect the guideline-adherent 
action in order for that episode of care to be classified as guideline-adherent.  For cases in which the program 
determines the care to not be adherent, the program will also specify where in the algorithm the care failed to 
be guideline adherent. We will test and validate this program for a sample of 25 cases in each condition, by 
having an investigator who is blinded to the computer determination, review the information entered into the 
abstraction form and determine if the care was adherent, and if not, identify where it was non-adherent. We will 
compare these manual determinations to those of the computer, correcting any errors in the programming that 
may be revealed by the discordances.  The investigator will also have both the EMR chart extract and the 
completed chart abstraction form, to determine if all of the pertinent information is in the form to allow him/her 
to make an accurate decision.     
 
Implicit review forms: Guided by the RAND DRG project’s implicit review form, we will develop a structured 
implicit review form that elicits judgments from reviewers for defined items for each non-tracer condition 
episode of care. An example of an item is “rate the quality of care for this episode of care on a 5-point scale.”  
Baseline encounter data will be available at the start of the project for use in developing this form.  This form 
will also request information about the factors that influenced the quality rating, and this will be coded for all 
implicit review forms.   

Version 12-15-15 
 
11 
 
REFERENCES 
1) 
Chapko MK, Borowsky SJ, Fortney JC, Hedeen AN, Hoegle M, Maciejewski ML, VanDeusen Lukas C. 
Evaluation of the Department of Veterans Affairs community-based outpatient clinics. Med Care. 2002 
Jul;40(7):555-60. 
2) 
Haskell SG, Gordon KS, Mattocks K, et al.  Gender differences in rates of depression, PTSD, pain, 
obesity, and military sexual trauma among Connecticut War Veterans of Iraq and Afghanistan. J Women’s 
Health (Larchmt). 2010 Feb;19(2):267-71. 
3) 
Cordasco KM, Zephyrin LC, Keesler CS, Mallard M, Canelo I, Rubenstein LV, Yano EM.  2011 VHA 
Emergency Services For Women (ESW) Survey, Final Report. 2012, in press 
4) 
Washington DL, Yano EM, Goldzweig C, Simon B. VA emergency health care for women: condition – 
critical or stable?  Women’s Health Issues. 2006 May-Jun;16(3):133-8. 
5) 
Yano EM, Washington DL, Goldzweig C, Caffrey C, Turner C. The organization and delivery of 
women's health care in Department of Veterans Affairs Medical Centers. Women’s Health Issues. 2003 Mar-
Apr;13(2):55-61.  
6) 
Bean-Mayberry BA, Yano EM, Caffrey CD, Altman L, Washington DL. Organizational characteristics 
associated with the availability of women’s health clinics for primary care in the Veterans Health Administration.  
Mil Med. 2007 Aug;172(8):824-8. 
7) 
U.S. Government Accountability Office (GAO). VA Health Care: VA has taken steps to make services 
available to women Veterans, but needs to revise key policies and improve oversight processes. GAO-10-287, 
Washington, DC, March 2010. Available at www.gao.gov.  
8) 
Washington DL, Bean-Mayberry B, Mitchell MN, Riopelle D, Yano EM. Tailoring VA Primary Care to 
Women Veterans: Association with patient-rated quality and satisfaction. Women’s Health Issues. 
2011;21(4S):S112-9. 
9) 
Washington DL, Yano EM. Women Veterans’ Ambulatory Care Use Project, Phase II: Final Report. 
Los Angeles, CA: VA HSR&D (IAE-06-083), 2010. Available at: 
http://www.hsrd.research.va.gov/research/abstracts.cfm?Project_ID=2141696839 
10) 
Landon BE, Wilson IB, Wenger NS, Cohn SE, Fichtenbaum CJ, Bozzette SA, Shapiro MF, Cleary PD.  
Specialty Training and Specialization Among Physicians Who Treat HIV/AIDS in the United States.  JGIM 2002 
Jan; 17(1):12-22 
11) 
VA Women Veterans Strategic Health Care Group and Patient Care Services. VHA Training Providers in 
Health Care for Women Veterans. http://www.simlearn.va.gov/feature/feature11_09.asp,  Accessed December 
4th, 2011. 
12) 
Grimshaw J, Shirran L, Thomas R, et al.  Changing Provider Behavior: An Overview of Systematic 
Reviews of Interventions. Medical Care. 2001; Vol. 39, No. 8, Supplement II pp. II2-II45. 
13) 
Mansouri M, Lockyer J.  A meta-analysis of continuing medical education effectiveness.  Journal of 
Continuing Education in the Health Professions.  2007 Winter; 27(1): 6-15. 
14) 
Davis DA; Thomson, MA; Oxman AD; Haynes RB.  Changing Physician Performance: A Systematic 
Review of the Effect of Continuing Medical Education Strategies.  JAMA. 1995;274:700-705. 
15) 
Frayne SM, Phibbs C, Friedman SA, et al. Sourcebook: Women Veterans in the Veterans Health 
Administration. Volume 1. Sociodemographic Characteristics and Utilization of VHA Care. Washington DC: 
Department of Veterans Affairs; 2010. 
16) 
Yano EM, Washington DL, Bean-Mayberry B. Impact of Practice Structure on Quality of Care for 
Women Veterans (Phase 2): Final Report. Los Angeles, CA: VA HSR&D (IAE-07-170), 2009. 
17) 
Washington DL, Bean-Mayberry B, Riopelle D, Yano EM. Access to care for Women Veterans: 
Delayed healthcare and unmet need. J Gen Intern Med.  2011 Nov; 26 (Suppl 2): S655-61. 
18) 
Department of Veterans Affairs. Veterans Health Administration Policy for Planning and Activating 
Community Based Outpatient Clinics. VHA Directive 97-036. July 1997:16. 
19) 
Washington DL, Yano EM, Simon B, Sun S. To use or not to use: what influences why women 
Veterans choose VA health care. J Gen Intern Med. 2006;21:S11-18. 
20) 
Washington DL, Kleimann S, Michelini AN, Kleimann KM, Canning M. Women Veterans’ perceptions 
and decision-making about Veterans Affairs Health Care. Mil Med. 2007;172(8):812-7. 
21) 
Saultz JW and Lochner J.  Interpersonal Continuity of Care and Care Outcomes:  A Critical Review.  
Annals of Family Medicine.  2005 Mar; 3(2):159-166 

Version 12-15-15 
 
12 
 
22) 
Mainous AG and Gill JM.  The Importance of Continuity of Care in the Likelihood of Future 
Hospitalization:  Is Site of Care Equivalent to a Primary Clinician?  Am J Public Health 1998; 88:1539-1541 
23) 
Haggerty JL, Reid RK, Freeman GK, Starfield BH, Adair CE, McKendry R.  Continuity of care:  a 
multidisciplinary review.  BMJ 2003; 327(7425):1219-1221 
24) 
VA Women Veterans Strategic Health Care Group and Patient Care Services. VHA Training Providers in 
Health Care for Women Veterans. http://www.simlearn.va.gov/feature/feature11_09.asp,  Accessed December 
4th, 2011. 
25) 
Baier L, Yee EFT, Veet L, Bonnema R, Hayes P. Re-educating VA Primary Care Providers: A program 
to improve care for women Veterans of childbearing age. VA HSR&D Women’s Health Conference. July 2010. 
26) 
Rubenstein LV, Mittman BS, Yano EM, Mulrow CD.  From Understanding Health Care Provider 
Behavior to Improving Health Care: The QUERI Framework for Quality Improvement.  Medical Care 2000 Jun; 
38(6):129-141 
27) 
Foy R, Ovretveit J, Shekelle PG, Pronovost PJ, Taylor SL, Dy S, Hempel S, McDonald KM, Rubenstein 
LV, Wachter RM.  The role of theory in research to develop and evaluate the implementation of patient safety 
practices.  BMJ Qual Saf. 2011 May;20(5):453-9. Epub 2011 Feb 11. 
28) 
Berendsen AJ, Benneker WH, Schuling J, Rijkers-Koorn N, Slaets JP, Meyboom-de Jong B. 
Collaboration with general practitioners: preferences of medical specialists--a qualitative study. BMC Health 
Serv Res. 2006 Dec 4;6:155.  
29) 
Smith Anthony C, Bensink M, Armfield N, Stillman J, Caffery L. Telemedicine and rural health care 
applications. Journal of Postgraduate Medicine.  2005; 51(4):286-293. 
30) 
Charles BL. Telemedicine can lower costs and improve access. Healthcare Finanacial Management. 
2000 Apr;54(4):66-9 
31) 
Speedie SM, Ferguson AS, Sanders J, Doarn CR. Telehealth: the promise of new care delivery models. 
Telemed J E Health. 2008 Nov; 14(9): 964-7. 
32) 
Horner K, Wagner E, Tufano J. Electronic consultations between primary and specialty care clinicians: 
early insights. Commonwealth Fund Issue Brief, pub. 1554, vol. 23. 2011 October. Available at: 
http://www.commonwealthfund.org/Publications/Issue-Briefs/2011/Oct/Electronic-Consultations.aspx. 
33) 
Rushakoff RJ, Woeber KA. Evaluation of a “formal” endocrinology curbside consultation service: advice 
by means of internet, fax, and telephone. Endocr Pract. 2003 Mar-Apr;9(2):124-7. 
34) 
Wegner SE, Humble CG, Feaganes J, Stiles AD. Estimated savings from paid telephone consultations 
between subspecialists and primary care physicians. Pediatrics 2008;122:e1136-40. 
35) 
Kim-Hwang JE, Chen AH, Bell DS, Guzman D, Yee HF, Kushel MB. Evaluating electronic referrals for 
specialty care at a public hospital. J Gen Intern Med. 2010;25(10):1123-8. 
36) 
Harno K, Paavola T, Carlson C, Viikinkoski P. Patient referral by telemedicine: effectiveness and cost 
analysis of an Intranet system. Journal Of Telemedicine And Telecare  2000;6(6):320-329. 
37) 
Jaatinen P, Aarnio P, Remes J, Hannukainen J, Köymäri-Seilonen T. Teleconsultation as a replacement 
for referral to an outpatient clinic. Journal Of Telemedicine And Telecare. 2002;8(2):102-106. 
38) 
Arora S, Thornton K, Murata G, et.al. Outcomes of treatment for Hepatitis C virus infection by primary 
care providers. N Engl J Med 2011;364:2199-2207. 
39) 
Arora S, Kalishman S, Dion D, et.al. Partnering urban academic medical centers and rural primary care 
clinicians to provide complex chronic disease care. Health Affairs 2011;30(6):1176-84. 
40) 
Department of Veterans Affairs (2010, February 4). Budget request for FY2011 and FY2012, Statement 
of The Honorable Eric K. Shinseki, House Committee on Veterans Affairs. Available at: 
http://democrats.veterans.house.gov/hearings/Testimony.aspx?TID=72714&Newsid=507&Name=The%20Hon
orable%20Eric%20K.%20Shinseki.  
41) 
Agency for Healthcare Research and Quality.   FEDERAL PATIENT CENTERED MEDICAL HOME 
(PCMH) COLLABORATIVE.  Available at: http: www.pcmh.ahrq.gov Accessed Dec 5th, 2011. 
42) 
Perednia DA and Allen A.  Telemedicine Technology and Clinical Applications.  JAMA 1995;273(6):483-
488. 
43) 
Department of Veterans Affairs. Report of the Under Secretary for Health Workgroup: Provision of 
Primary Care to Women Veterans. Office of Public Health and Environmental Hazards, Women Veterans 
Health Strategic Health Care Group. Washington, DC, November 2008. 
44) 
Zephyrin L, FY10 VHA Gynecology Physician Productivity and Staffing report, Women Veterans Health 
Strategic Health Care Group. Washington, DC, May 2011. 

Version 12-15-15 
 
13 
 
45) 
Washington DL, Yano EM, Bean-Mayberry B, Mitchell M, Sun S, Canning M, Yosef J, Riopelle 
D.  Assessment of the health care needs and barriers to VA use experienced by Women Veterans.  
Unpublished data. 
46) 
Primary Care Management Module User Manual. Available at: 
http://www4.va.gov/vdl/documents/Clinical/Pri_Care_Mgmnt_Module_(PCMM)/pcmmug.pdf.   
47) 
VA Information Resource Center (VIReC), VHA Medical SAS Datasets – Documentation. Available at: 
http://www.virec.research.va.gov/DataSourcesName/Medical-SAS-Datasets/SASdocumentation.htm.   
48) 
Kirchner J, Edlund CN, Henderson K, Daily L, Parker LE, Fortney JC. Using a multi-level approach to 
implement a primary care mental health (PCMH) program. Families, Systems, & Health. 2010;28(2):161-74. 
49) 
Hamilton-Brown A, Cohen AN, Chinman MJ, Kessler C, Young AS. EQUIP: implementing chronic care 
principles and applying formative evaluation methods to improve care for schizophrenia: QUERI Series. 
Implement Sci. 2008 Feb 15;3:9. 
50) 
UpToDate, available at http://www.uptodate.com/index 
51) 
Department of Veterans Affairs, Health Information Access – HIA. Available at: http://vaww4.va.gov/hia/. 
52) 
Department of Veterans Affairs, Compensation Pension Records Interchange (CAPRI). Available at: 
http://www.va.gov/vdl/application.asp?appid=133 .  – links to user manual and release notes 
53) 
Department of Veterans Affairs, Office of Enterprise Development, VistA Compensation and Pension 
Record Interchange (CAPRI) GUI User Manual, June 2011. Available at; 
http://www.va.gov/vdl/documents/Financial_Admin/CAPRI/dvba_27_p000_um.pdf.  
54) 
McGlynn EA, Kerr EA, Adams J, Keesey J, Asch SM. Quality of health care for women – a 
demonstration of the Quality Assessment Tools System. Med Care 2003;41:616-25. 
55) 
ACOG Committee on Gynecology. Practice Bulletin 14. Management of Anovulatory Bleeding. ACOG 
Compendium of Selected Publications 2009. Also published in Obstet Gynecol 2000; 95(3). Reaffirmed in 
2009. 
56) 
Kaiser Permanente Southern California. Acute uterine bleeding unrelated to pregnancy. Pasadena (CA): 
Kaiser Permanente Southern California; 2006 Aug. Guideline Summary NGC-5180 available at: National 
Guideline Clearinghouse, http://www.guideline.gov.  
57) 
Amann M, Anguino H, Bauman RA, Cheung ML, Harris S, Kennedy J, et.al. Chronic abnormal uterine 
bleeding in nongravid women. Pasadena (CA): Kaiser Permanente Southern California;2006 Dec. Guideline 
Summary NGC-5687 available at: National Guideline Clearinghouse, http://www.guideline.gov. 
58) 
Amann M, Anguino H, Bauman RA, Cheung ML, Harris S, Kennedy J, et.al. Postmenopausal uterine 
bleeding. Pasadena (CA): Kaiser Permanente Southern California; 2006 Dec. Guideline Summary NGC-5688 
available at: National Guideline Clearinghouse, http://www.guideline.gov. 
59) 
Institute for Clinical Systems Improvement. Health Care Guideline: Diagnosis of Breast Disease. 2010 
Jan. 13th Edition. Available at: http://www.icsi.org/improvement_resources.  
60) 
ACOG Committee on Gynecology. Practice Bulletin 73. Use of Hormonal Contraception in Women with 
Coexisting Medical Conditions. ACOG Compendium of Selected Publications 2009. Also published in Obstet 
Gynecol 2006; 107:1453-72. Reaffirmed in 2008. 
61) 
Centers for Disease Control and Prevention. U.S. Medical Eligibility Criteria for Contraceptive Use, 
2010. Adapted from the World Health Organization Medical Eligibility Criteria for Contraceptive Use, 4th edition. 
MMWR. 2010 Jun 18;59(RR-4):1-85. 
62) 
Nelson HD, Haney EM, Dana T, Bougatsos C, Chou R. Screening for Osteoporosis: An Update for the 
U.S. Preventive Services Task Force. Ann Intern Med. 2010;153:99-111. 
63) 
Lewiecki EM, Watts NB. New Guidelines for the Prevention and Treatment of Osteoporosis. Southern 
Medical Journal. 2009;102:175-9. 
64) 
Hamdy RC, Baim S, Broy SB, Lewiecki EM, Morgan SL, Tanner SB, Williamson HF. Algorithm for the 
Management of Osteoporosis. Southern Medical Journal. 2010;103:1009-15. 
65) 
North American Menopause Society. Management of osteoporosis in postmenopausal women: 2010 
position statement of The North American Menopause Society. Menopause 2010 Jan-Feb;17(1):25-54. 
Guideline Summary NGC-7590 available at: National Guideline Clearinghouse, http://www.guideline.gov. 
66) 
ACOG Committee on Gynecologic Practice. Committee Opinion 313. The importance of preconception 
care in the continuum of Women’s Health Care. ACOG Compendium of Selected Publications 2009. Also 
published in Obstet Gynecol 2005; 106:665-6. Reaffirmed in 2009. 
67) 
Lu MC. Recommendations for Preconception Care. Am Fam Physician 2007;76:397-400. 

Version 12-15-15 
 
14 
 
68) 
Centers for Disease Control and Prevention. Recommendations to Improve Preconception Health and 
Health Care – United States. MMWR. 2006 Apr 21;55(RR-6):1-23. 
69) 
ACOG Committee on Gynecology. Practice Bulletin 63. Urinary Incontinence in Women. ACOG 
Compendium of Selected Publications 2009. Also published in Obstet Gynecol 2005; 105:1533-45. Reaffirmed 
in 2011. 
70) 
Fleiss, J. L., Levin, B. and Paik, M. C. (2004) The Measurement of Interrater Agreement, in Statistical 
Methods for Rates and Proportions, Third Edition, John Wiley & Sons, Inc., Hoboken, NJ, USA. 
71) 
Shane AC, Phibbs CS. Guidebook for Research Use of PAID Data. Guidebook. Menlo Park, CA. VA 
Palo Alto, Health Economics Resource Center; 2010. 
72) 
Campbell MK, Steen IN, Grimshaw JM, et.al. Design and statistical issues in implementation research. 
In Thorsen T, Makela M (eds). Changing Professional Practice. Copenhagen: DSI: Danish Institute for Health 
Services Research and Development, 1999. 
73) 
Cook TD, Campbell DT. Quasi-Experimentation: Design & Analysis Issues for Field Settings. Houghton 
Mifflin Co., Boston, 1979. 
74) 
Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: 
The RE-AIM Framework. Am J Public Health. 1999;89:1322-1327. 
75) 
Glaser BG, Strauss AL. The discovery of grounded theory: Strategies for Qualitative Research. New 
York NY: Aldine Publishing; 1967. 
76) 
Goldzweig CL, Parkerton PH, Washington DL, Lanto AB, Yano EM. Primary care practice and facility 
quality orientation: influence on breast and cervical cancer screening rates. Am J Manag Care. 2004 
Apr;10(4):265-72 
77) 
Pronovost PJ, Miller MR, Dorman T, Berenholtz SM, Rubin H.  Developing and implementing measures 
of quality of care in the intensive care unit. Curr Opin Crit Care 2001, 7:297–303 
78) 
Shekelle PG, MacLean CH, Morton SC, Wenger NS. Acove quality indicators. Ann Intern Med 2001 Oct 
16;135(8 Pt 2):653-67. 
79) 
Greenfield S, Cretin S, Worthman L.  Comparison of criteria maps to a criteria list in quality of care 
assessment for patients with chest pain:  the relation of each to outcome.  Med Care 1981: 19:255-272 
80) 
Rubenstein LC, Kosecoff J, Kahn KL, et al.  Medical Record Abstraction Form and Guidelines for 
Assessing Quality of Care for Hospitalized Patients with Cerebrovascular Accident.  Santa Monica, Calif: 
RAND Corp; 1988.  N2802-HCFA.  
81) 
Knowles M. Adult Learning. In RL Craig (Ed.), The ASTD Training and Development Handbook. New 
York: McGraw-Hill Publishers, 1996, pp 253-264. 
82) 
Coppus SFPJ, Emparanza JI, Hadley J, et al. A clinically integrated curriculum in evidence-based 
medicine for just-in-time learning through on-the-job training: The EU-EBM project. BMC Med Educ. 2007;7:46. 
83) 
Duffy TM, Jonassen DH. Preface to Constructivism and the Technology of Instruction: A Conversation. 
Hillsdale, NJ: Laurence Erlbaum & Associates, 1992. 
84) 
Kirkpatrick DL, Kirkpatrick JD. Evaluating Training Programs: The Four Levels. 3rd ed. San Francisco; 
Berrett-Koehler Publishers, 2006. 
85) 
Washington DL, Caffrey C, Goldzweig C, Simon B, Yano EM. Availability of comprehensive women's 
health care through Department of Veterans Affairs Medical Centers. Women’s Health Issues. 2003;13(2):50-
54.