Evaluating Community Health Centers' Adoption of a New Global Capitation Payment
Conditions: Chronic Diseases
Sponsor: Oregon Health and Science University

IRB #00009455 
Revised: 6/1/16 
 
Oregon Health & Science University 
RESEARCH PROTOCOL 
 
Protocol Title: 
Evaluating Community Health Centers’ Adoption of a New Global Capitation Payment (eCHANGE) 
 
Principal Investigator: 
John Heintzman, MD 
Assistant Professor, Oregon Health & Science University Department of Family Medicine  
3181 SW Sam Jackson Park Rd, Mailcode: FM 
Portland, OR 97239 
 
heintzma@ohsu.edu 
 
Sponsor: 
Robert Wood Johnson Foundation, grant #71125 
Agency for Healthcare Research and Quality (AHRQ)  
 
Specific Aims: 
Aim 1: Assess pre-post Alternative Payment Methodology (APM) changes in internal services utilization 
and quality of clinical care delivered in intervention sites, as compared with control sites. 
 
Hypothesis 1a: Patterns of internal services utilization (e.g., face-to-face visits, e-mail visits) will change 
among intervention clinics’ patients from pre- to post-APM, while control clinics’ patients’ rates will not 
change. 
Hypothesis 1b: Intervention site patients will achieve greater pre-post improvements in quality of care 
delivered internally, as compared to the rates of improvement for patients in control clinics. 
 
Aim 2: Measure pre-post changes in patients’ utilization of external services (e.g., emergency department) 
and overall costs to the Medicaid program among patients from intervention clinics, as compared to 
patients from control clinics. 
 
Hypothesis 2a: Expenditures attributed to external services will decrease for patients from intervention 
clinics, but these expenditures will remain the same or increase for patients in the control clinics.  
Hypothesis 2b: Total Medicaid expenditures will decrease for patients from intervention clinics, while 
expenditures for patients from control clinics will remain the same or increase.  
 
Aim 3: Study the change processes associated with APM implementation, including the organizational, 
workflow, and service delivery changes made in the Community Health Centers (CHCs) during and after 
APM implementation.  
 
Hypothesis 3a: APM clinics will make fundamental pre-post changes in patterns of clinical care delivery 
and staff workflow.  
Hypothesis 3b: There will be variability among the intervention clinics in what changes are successfully 
achieved, associated with vision, approach, strategic planning, and other clinic characteristics. 
 
 

 
 
We will utilize a mixed-methods approach, drawing upon both qualitative and quantitative data from 
clinic site visits, clinic surveys, key informant interviews, follow-up calls, Medicaid administrative 
claims, and the OCHIN electronic health record (EHR).  
 
Background and Significance:  
The common practice of reimbursing primary care providers through a fee-for-service model hampers 
efforts to develop patient-centered medical homes: it rewards a high volume of face-to-face clinic visits, 
which does not necessarily lead to value, quality, and patient-centered care. This is especially problematic 
in Community Health Centers (CHCs), which are currently paid based on the Prospective Payment 
System (PPS) rate, a fee-for-service model. To reform this payment model in CHCs, Oregon developed 
an Alternative Payment Methodology (APM) that converts PPS (Medicaid revenue) to a prospective, 
capitated per-member per-month rate. The Oregon Primary Care Association (OPCA) is providing 
technical assistance to the APM clinics, with funding from the Robert Wood Johnson Foundation. The 
APM project seeks to (i) incentivize providing value over volume of visits, (ii) support comprehensive 
treatment modalities for patients with complex needs, and (iii) allow CHCs to focus more of their 
resources on population health initiatives.  
 
APM is being implemented in several phases. Three CHC organizations are participating in Phase I of the 
APM demonstration project, which kicked off on March 1, 2013. Five additional CHC joined Phase II 
July 1, 2014 or October 1, 2014. Phase III began July 1, 2015 with three additional CHC organizations. 
Each CHC organization may include several clinics. All but one of the APM CHCs are members of the 
OCHIN community health information network practice-based research network, with a single, shared 
EHR. In this study, we propose to evaluate the APM clinics and/or patients matched to an equal number 
of control clinics and/or patients that are part of the OCHIN network. 
 
Preliminary Studies / Progress Report: 
For over a decade, we have worked with policy makers to evaluate the impact of Oregon Health Plan 
(OHP) policy changes. We learned about problems families experienced when they lost insurance, 
barriers faced when accessing care, disparities in care, and the importance of health insurance coverage 
and a usual source of care.1-24 Once we identified these barriers, we worked with families, community 
advocacy organizations, and policy makers to improve access to public health insurance in Oregon. State 
policies to expand and improve coverage programs were informed by this research. Specifically, we 
found that even small co-payments led to increased uninsurance and unmet care needs.7,23,25-28 We learned 
that children with short coverage gaps had disproportionate rates of unmet need,10 and uninsured children 
had even higher rates of unmet need.2,4 Our findings influenced changes, including: the elimination of co-
payments for Medicaid beneficiaries, expansions in the OHP, and the passage of a state referendum which 
expanded Oregon’s Children’s Health Insurance Program (CHIP). Our work also informed policies that 
made public insurance more accessible (e.g., shortened waiting periods). Our finding that parents with 
public insurance were more likely than those with private coverage to have continuously insured children 
informed initiatives to expand public coverage to more Oregon parents.11 Many other states requested 
copies of publications based on these findings, and this work was highlighted in policy briefs at the 
national level.29  
 
Research Design and Methods 
 
Subjects and Experimental Design: 
 
This is a clinic- or patient-level matched prospective study design, coupled with a difference-in-

differences (DID) approach. The sample is Oregon CHCs participating in the APM demonstration project 
implemented by the state of Oregon (implementation clinics and/or patients) and an equal number of 
control clinics and/or patients.  
 
We will interact with approximately 40 clinic staff subjects at each of the Phase I clinic sites, and 
approximately 10 staff subjects at the OPCA office, aged 18-100 years.  
 
We will also assemble a dataset of EHR data on approximately 400,000 patients aged 2-64 from the 
implementation clinics and/or patients and control clinics and/or patients. All Phase I study sites have 
agreed to take part in the initial qualitative data collection portion of this study and have agreed to $1000 
compensation at the organization level for their participation. We will select control sites and/or patients 
based on matching algorithms to ensure selected controls are similar to implementation clinics and/or 
their patients. 
 
We will conduct up to two site visits to intervention clinics to observe practice changes that occurred post 
APM-implementation. We will also conduct preliminary qualitative data collection in the APM sites, 
including a practice survey and key informant interviews. Due to a shortage of funding for the initial 
baseline data collection, and because the Phase II and III sites have limited involvement in the study, they 
will not receive reimbursement for study participation.  
 
Baseline qualitative data collection 
 
Box 1: Qualitative Activities, Subjects and Recruitment 
Study Site Name 
Study Activities 
Approximate Number 
of Clinic Staff Subjects 
at Each Site 
Recruitment Strategy 
Phase I APM Sites: 
Virginia Garcia – 
Beaverton, Cornelius, 
Hillsboro, McMinnville 
(several clinics may be 
included at each of these 
locations) 
  
Mosaic Medical – Bend, 
Prineville, Madras 
(several clinics may be 
included at each of these 
locations) 
 
OHSU Richmond (several 
clinics may be included 
from this location) 
Pre-site visit telephone 
interview 
1 
Clinic “point-person” 
Phone interview after 
monthly OPCA meeting, 
if needed 
2 
Clinic “point-person” 
Staff interviews 
10 
Referred by clinic “point-
person” and identified by 
study staff during site 
visit 
Practice Change 
Assessment Questionnaire 
20 
Clinic staff meeting 
attendees 
Oregon Primary Care 
Association Office 
Observe monthly 
meetings 
10 
Monthly project meeting 
attendees 
Phase II APM Sites: 
 
OHSU Scappoose 
(several clinics may be 
included from this 
location) 
 
Coastal Family Health 
Telephone interview 
1 
Clinic “point-person” 
Phone interview after 
monthly OPCA meeting, 
if needed 
2 
Clinic “point-person” 
Practice Change 
Assessment Questionnaire 
20 
Clinic staff meeting 
attendees 

Center (several clinics 
may be included from this 
location) 
 
Benton County Health 
Department - Benton, 
East Linn, Monroe, 
Lincoln (several clinics 
may be included at each 
of these locations) 
 
Multnomah County 
Health Department- 
Northeast, East County, 
Rockwood, Mid County, 
Southeast, North Portland 
(several clinics may be 
included at each of these 
locations) 
 
Yakima Valley 
Farmworkers Clinic - 
Hermiston, Woodburn, 
Portland, Salem (clinics 
from this CHC 
organization who do not 
have an OCHIN Epic 
EHR may not be included 
in this study) 
 
Site visits. We will conduct a site visit in implementation CHCs to collect data from staff. The visit will 
collect data to help us observe and understand APM-related processes and workflows, and assess factors 
unique to each clinic that facilitate or hinder change. Trained study staff will collect data during the site 
visits, which will last approximately 3-5 days at each of the APM sites. We will use the multi-method 
assessment process (MAP),30 as we have successfully done in other studies.31-33 Using MAP’s iterative 
process, field researchers will gather data, synthesize information and select new avenues to probe in a 
cycle. As the data expand and lead to deeper insights, new key informants may be tapped and new data 
sources mined. This iterative process is central to obtaining rapid and substantive knowledge in field 
research. At each site visit, the team will spend the first 2-3 days intensively observing activities. During 
the visits, we will focus on areas that clinics targeted for change, observing and describing tasks, 
workflow and clinical operations; team members will also talk informally with clinic staff, as these 
opportunistic conversations may elicit different information than what is captured in formal interviews.  
 
In addition, we will conduct semi-structured interviews with a diverse group of clinic staff. These 
interviews will be designed to explore experiences with the APM model and the change process. We will 
obtain in-depth information about what they did (or did not do) as a result of APM and why. We will have 
regular follow-up phone calls with these key informants, roughly once a month, to ask about specific 
changes. Team members will prepare field notes and debrief to share observations and strategize for 
additional visits. After the site visit, the field team will debrief with the full team to share findings, get 
feedback, review the interview list, and refine the interview guide, as needed. If needed, we will continue 
to conduct interviews and observations with a focus on filling knowledge gaps for 1-2 final days. A total 
of 8-10 interviews will be conducted per CHC; we will schedule the interviews as convenient for 
participants.  

 
 
CHC surveys. At APM sites, we will survey CHC staff using the Practice Change Assessment 
Questionnaire to assess staff attitudes about change and facilitators and barriers to change at the clinic. 
The study team will conduct the surveys at a clinic meeting and personally follow-up with non-
responders.  
 
Statistical considerations  
Inclusion Criteria. Qualitative data collection will target current staff at the APM implementation clinics, 
aged 18-100. 
 
Data management. Study staff will assemble a database of all data elements. Qualitative data will be 
catalogued in a spreadsheet. Field notes, hand-written on site, will be expanded and put into electronic 
form by the researchers. Interviews will be digitally recorded and professionally transcribed. Recordings, 
field notes, summaries of follow-up phone calls, transcripts and digital copies of all collected artifacts will 
be kept on a secure network. Access to digital files will be password protected. We will use Atlas.ti 
qualitative software to aid this process. Surveys will be designed to scan electronically into a REDCap 
database, checked for accuracy by study staff, and exported to statistical software for analysis.. 
 
Analysis Plan. We will begin qualitative analyses shortly after data collection commences. The study 
team will meet regularly to review all data, and listen to and discuss key segments of the recorded 
interviews. Analyzing qualitative data as it is collected is crucial to monitoring data quality, refining 
observation and interview guides, and monitoring theme saturation. This ongoing process will be used to 
track emerging themes and create a coding template to be used for more in-depth analysis. We will follow 
the 5-phase analysis strategy described by Miller and Crabtree (describing, organizing, connecting, 
corroborating/ legitimizing, representing).34,35 We will use an immersion-crystallization approach in 
which the team reads and discusses the data for each clinic (immersion) to identify key findings 
(crystallization);35,36 we will do this twice, first to identify key themes within each “case” CHC, and then 
to identify cross-case finding. 
 
The survey data and interview transcripts will be coded using the 5-phase analysis strategy explained 
above. We will also analyze differences between the Phase I and Phase II clinics’ experiences with APM 
implementation. 
 
Quantitative data collection 
 
We will compile EHR and Medicaid claims datasets that include data from up to 3 years pre-APM 
implementation, during implementation, and ≥3 years post.   
 
Aims 1 and 2 Evaluation Summary 
Evaluate effect of 
APM intervention 
on: 
Study Time 
Periods  
Data Sources 
Inclusion 
Criteria 
Analyses  
 
Aim 1. Internal 
services utilization 
and quality of care 
delivered; 
 
Aims 2 & 3. 
Utilization of and 
spending on services 
outside of the CHC, 
 
Up to 3 years pre-
APM 
implementation 
;  
 
≥3-years post-
APM 
implementation 
;  
 
1. Internal 
services: 
  
OCHIN EHR 
data  
 
2. External 
services and 
expenditures: 
 
1. Total clinic 
population:  
established 
patients at 
intervention and 
control clinics 
aged 2-64 
 
2. Medicaid 
 
Clinic level analysis using the DID 
analysis and patient level analysis 
using Generalized Estimating 
Equation models, to accommodate 
serial and intra-clinic correlations, 
categorical and continuous 
covariates, fixed and time-dependent 
covariates. Two-part models will be 
used to accommodate distributions 

total Medicaid 
spending.  
 
 
 
(We will assess 
how long 
implementation 
takes and include 
it as transition 
time to our 
analyses)  
 
OCHIN EHR 
data; Medicaid 
administrative 
data (including 
intervention 
CHCs’ per-
member per-
month capitated 
payment rates) 
Population: 
Medicaid-
enrolled patients 
at intervention 
and control 
clinics aged 2-64 
of healthcare spending. Matching 
methods will be utilized to reduce 
the observed bias and to adjust for 
imbalances between intervention and 
control clinics and/or patients. 
 
 
Data Sources 
EHR data. We will assemble a dataset of EHR data elements including all demographics, healthcare 
utilization, diagnoses, treatment, and health insurance coverage start and stop dates for all established 
patients age 2-64 for patients of APM clinics and their matched controls. We will include data from the 
following dates: no earlier than January 1, 2010 – December 31, 2018. The EHR is centrally hosted and 
managed at OCHIN and shared across all study CHC clinics. OCHIN maintains this central data 
warehouse, and the data are standardized across all OCHIN CHCs and regularly checked for accuracy. 
 
Medicaid administrative data. These data will provide information on external services received and costs 
incurred. Oregon’s Medicaid recipients are assigned unique individual identification (ID) numbers, 
facilitating data linkages across multiple databases, including the OCHIN EHR data. We will use claims 
and cost data from Medicaid. 
 
Linkage between EHR and Medicaid administrative data. Project staff will use state ID numbers to create 
individual-level linkages between OCHIN EHR and Medicaid data..37-39 All transfers will be done 
securely. A de-identified dataset will be transferred to OHSU. The lead research analyst at OHSU has 
obtained security clearances to work with OCHIN’s data team to assist with preparing the dataset to 
transfer to OHSU.  
 
This study is being conducted in collaboration with OCHIN. The State of Oregon data will be shared with 
OCHIN under a Data Access Agreement. OCHIN will then share Medicaid and OCHIN EHR data with 
OHSU under a Data Use Agreement. 
 
Data included in the datasets Once the Medicaid-OCHIN EHR link is created, names, addresses, social 
security numbers and ID #s will be removed by the OCHIN analysts and a study ID will be given to the 
records that does not stem from any identifiers prior to sending the datasets to OHSU. OCHIN will 
maintain, but not share with OHSU the crosswalk that links the study ID to the identified data.  
 
Inclusion criteria. OCHIN’s EHR data includes internal services utilization and care quality on all patients 
seen, regardless of insurance type (Aim 1). We will assess the APM’s impact on the entire clinic 
population (aged 2-64) as well as relevant subpopulations (e.g., Medicaid-insured, uninsured, APM 
patients). For some comparisons, the denominator will be patients with clinic contact. Patients in a 
clinic’s panel will not necessarily be the same individuals in the pre- vs. post-APM period. Because APM 
might result in fewer face-to-face visits, we will work with our expert advisors to determine what 
“counts” as contact with the clinic. For some comparisons, we will create a longitudinal cohort of the 
same patients over time. Assessment of external services utilization and overall costs will focus on the 
Medicaid-insured (Aim 2).  
 
Study variables. We adapted Aday and Andersen’s model of health care utilization  to help us understand 
the interplay between predisposing factors, enabling / hindering factors, and need for care, their impact 

on utilization of CHC services and other recommended care, and the resulting effect on external services 
utilization and overall cost of care. The primary independent variable is whether or not the patient was 
exposed to the APM by being a patient at an intervention vs. control site (an enabling / hindering factor). 
The primary dependent variables of utilization of internal / external services and quality of care delivered 
are based on (i) the Medicaid initial core sets of health quality measures for adults,43,44 and (ii) pediatric 
care quality measures developed by AHRQ’s National Advisory Council Subcommittee on Children’s 
Healthcare Quality Measures for Medicaid and CHIP Programs, pursuant to the Children’s Health 
Insurance Program Reauthorization Act. We will select measures that are: 1) identified by Oregon’s 
Medicaid program and/or those on the list of Clinical Quality Measures and other EHR incentive 
programs;45,46 2) feasibly measured with EHR or Medicaid claims data (our team has created methods 
for adapting quality measures based on claims data for use in OCHIN’s EHR data47,48); 3) 
representative across age groups and gender; 4) representative of treatment levels (e.g., prevention, acute 
and chronic condition care); and 5) relevant to CHC populations. With guidance from an expert advisory 
group, we will select and define final specifications and modify them to accommodate calculation over 
the pre-post periods. Assessment of internal services utilization and quality can be done using OCHIN’s 
EHR data; assessment of external services utilization will use EHR data linked with Medicaid 
administrative data.   
 
Statistical analyses. We will conduct analyses at the clinic- and patient-level. The clinic-level analyses are 
typically performed in cluster-randomized trials, and the individual-level analyses are typically performed 
in observational studies in order to account for socio-demographic differences among clinics.  
 
Clinic-level: We will summarize baseline measures using descriptive statistics and data visualization 
methods (e.g., histograms, scatter plots) to assess how baseline data differs. To compare pre- and post-
APM rates, we will assess data in pre- and post-APM time periods, using regression methods to estimate 
pre- vs. post-intervention change for each clinic. We will use a Generalized Estimating Equation (GEE) 
approach to adjust for serial correlation and other potential confounders. The pre- vs. post-APM changes 
will be calculated with GEE logit regression between intervention and control CHCs (DID analysis). 
Additional analyses to account for  differences among APM phases and temporal inclusion may be used. 
 
Patient-level: The primary independent variable is whether a patient belongs to an intervention or control 
CHC. Other covariates include predisposing factors and need for services. We will use GEE models49,50 
which offer flexible regression modeling to accommodate different sources of correlations (serial, intra-
clinic, and intra-family), categorical and continuous covariates, and fixed and time-dependent covariates. 
These methods offer a wide range of parametric distributions to model the dependent variables, including 
logistic regression (binary data), beta regression (percent data), Poisson regression (count data), and 
Gaussian regression (normally distributed data). For example, to assess a change in quality rates, a GEE 
logistic regression model can analyze quality of care received in pre- and post-APM periods as a function 
of whether a patient belongs to an intervention or control CHC and other possible confounders (e.g., 
predisposing factors; need for services). Serial, and intra-clinic correlations will be modeled, and 
empirical robust standard errors will be applied, to provide valid inference.51 As an exploratory secondary 
analysis, we will evaluate the impact of the APM intervention on total number of recommended services 
received. The distribution of the total services received will be examined first before selecting a specific 
model to use for the analysis. Analytic models will be refined through an iterative process, guided by the 
hypotheses, conceptual model, and preliminary analyses. If there are significant patient differences 
between the intervention and control sites, we may use propensity score methods in the analysis phase to 
reduce the observed bias and adjust for imbalances between intervention and control clinics.52  
 
Econometric analyses. Using difference-in-differences methods, we will calculate the average pre-post 
APM difference in spending of patients in APM intervention clinics, subtracted by the average difference 
among patients in comparison clinics. Changes in spending after APM implementation in the intervention 

clinics vs. spending changes in the control clinics will “net out” any secular changes not related to APM. 
Any remaining significant differences in outcome – the DID – are attributed to the APM intervention. 
These findings will help to assess whether the APM intervention can help reduce costs, and if so, whether 
the reduction comes from lower spending at the CHC level or through reductions in potentially 
unnecessary or inefficient use of specialty and hospital services. We note that CHCs are not financially at 
risk for services outside of the CHC and thus do not have incentives to act as gatekeepers to care. Thus, 
reductions in spending for specialty services and other external services would most likely reflect 
improvements in internal care delivery. 
 
Analytically, the difference-in-differences estimate is represented by the marginal effect on the interaction 
between a dummy variable indicating that the observations occur in the post-APM period and a dummy 
variable indicating the individual’s attribution to an APM intervention clinic. We will also include 
patient-level variables to account for predisposing factors and need for services. A variety of issues must 
be addressed in our DID estimation: 
 
Definition of external services. External services will be defined as any care occurring outside of the 
CHC, e.g., emergency department visits, hospitalizations, and other ambulatory care that takes place in 
settings other than the CHC.  
 
Accounting for clinic-level effects. Our study will focus on the “treatment-effect” of the APM: how does 
the transition to a capitated payment system affect health care spending? Since this is our main focus, our 
approach may include use of clinic-level fixed effects to net out any time-invariant differences in clinic-
level cost outcomes. However, recognizing the potential interest in clinic-level effects, in secondary 
analyses we will test hierarchical (mixed) models to more specifically model (i) clinic-level variation and 
(ii) patient-level variation.  
 
Modeling health care costs. As in many analyses of a patient’s health care costs, in any given year, many 
patients will have no visits or expenditures. Thus, our dependent variable will have a cluster of 
observations at zero. We will use a well-validated approach for modeling this phenomenon: the 2-part 
model.53 Part 1 will use a logistic regression to estimate the probability of any expenditure. Part 2 will 
focus on individuals with non-zero expenditures. We will use recent literature to guide the appropriate 
estimation approach, taking into account the potentially skewed distribution of the dependent variable.54,55 
 
Accounting for all expenditures. Estimates of changes in spending must account for all expenditures, 
including capitated and fee-for-service (FFS) payments. Of note, Oregon’s Medicaid managed care 
organizations (MCOs) provide encounter data to the state Medicaid agencies in lieu of FFS claims, a 
common practice in many states. These data are assessed for general accuracy and consistency and have 
been used by states to set managed care payment rates. In order to attach “paid” amounts to these 
encounter claims, we will impute a “paid” amount across all FFS and MCO claims (that is, any external 
services for which there would be payment outside of the APM capitation rate) based on the median FFS 
payment. The study team has used this approach previously and has extensive experience in the data 
structures, elements, values, means of combining records and methods for aggregating and measuring 
service dimensions.25,56 In summary, creation of the expenditure variable will be based on a blend of 
actual FFS payments, recorded capitation rates, and, where necessary, imputed expenditures that have 
served as the actuarial basis for rate setting in Oregon. 
 
Modeling interaction terms. In non-linear models (e.g., the 2-part model) the interpretation of the 
interaction effect cannot be directly calculated from the coefficient on the interaction term.57 Our 
approach will be to estimate the 2-part model, save those coefficients, and use empirical simulation (also 
known as “recycled predictions”) methods to determine the APM’s average effect using bootstrapping to 
derive 95% confidence intervals.  

 
Accounting for repeated observations. We will observe outcomes for each individual in the pre- and post-
APM periods. We will consider several options for modeling the error component, including random 
effects and generalized estimating equations, and the block bootstrap. 
 
Human Subjects Considerations: 
 
Potential Risks, Protection from Risks, and Risk/Benefit Discussion 
 
Site Visits: Conducted at CHC practices. No personally identifiable information will be collected during 
these visits. 
 
Staff Interviews: Interviews will be recorded. Research team members will follow IRB consent and 
privacy protocols at all times. Transcripts from completed interviews will be coded with a unique 
identifier and kept secure in a locked file cabinet when not in use. All computer files containing unique 
identifier links will be password protected, accessible only to relevant study team personnel. 
   
Research team members will follow IRB privacy protocols at all times.  All recordings will be kept secure 
in a locked file cabinet when not in use and destroyed once the study is complete. All computer files will 
be password protected, accessible only to relevant study team personnel. All electronic files will be 
maintained for 2 years after the conclusion of the study, including all data analyses. After this time period, 
all paper and computer data files will be destroyed or will be used for ongoing research. In the latter 
instance, the principal investigator will obtain IRB approval.      
 
Follow Up Phone Calls with Key Informants: Interviews will be recorded. Research team members will 
follow IRB consent and privacy protocols at all times. Summaries and transcripts of these calls will be 
coded with a unique identifier and kept secure in a locked file cabinet when not in use. All computer files 
containing unique identifier links will be password protected, accessible only to relevant study team 
personnel. 
   
Research team members will follow IRB privacy protocols at all times. All recordings, transcripts, and 
computer files will be stored on a password protected drive, accessible only to relevant study team 
personnel. All electronic files will be maintained for 2 years after the conclusion of the study, including 
all data analyses. After this time period, all paper and computer data files will be destroyed or will be 
used for ongoing research. The principal investigator will obtain IRB approval before using data from this 
study for other research. 
 
Observation of Project Meetings: Conducted with project leads from each clinic organization and the 
Oregon Primary Care Association. No personally identifiable information will be collected during 
monthly meetings.  
 
EHR Abstraction: Conducted with the records of patients aged 2-64 who receive health care at 
intervention and control CHCs. All EHR data from the OCHIN member clinics are stored securely at 
OCHIN, each clinic has business use agreements with OCHIN to handle and manage PHI from their 
clinical data. OCHIN data analysts will be responsible for secure transfer of data. When all data linkages 
have been completed, the data will be de-identified (OCHIN will maintain a crosswalk of identifiers, but 
it will not be shared with OHSU).  
 
Potential Risks 
 

Potential risk to Practices: For practices, this is a low risk study. We will be observing practice operations 
and implementation of the APM. This observation does not pose a risk to the practice, as all of the data 
we collect, via survey and fieldnotes, is immediately de-identified. 
 
Potential risk to CHC staff:  The CHC staff interviews and CHC staff survey pose minimal risk to 
practice members because the survey itself does not contain any personal information about the practice 
members. It is possible that someone in the practice could use information about a respondent’s role (e.g., 
Office Manager) to identify that person and his or her responses. For this reason, our study team collects 
these data personally, and the data we collect is completely confidential, and only used by the study team. 
If data is reported to practices, we only report these data in aggregate so that it is not possible to link an 
individual and his or her responses. 
 
Potential Risk to Patients: Our protocol and procedures for collecting data and conducting record 
abstraction ensure that we are not collecting any information that could be used to identify patients that 
participate in this study. The EHR data collection holds a minimal risk of the small possibility of 
disclosure of legally protected health information (PHI) or personal information. We will ensure proper 
protections are in place to prevent any disclosure. 
 
Adequacy of Protection Against Risks 
 
a. Recruitment and Informed Consent  
IRB and HIPAA approval will be obtained for all study steps. 
 
Full explanation of the study will be given to practice leaders. We will ask practice leaders to explain the 
study to practice staff. An information sheet will be used to provide information about the study to 
potential subjects prior to qualitative data collection.   
 
We are applying for a Waiver of Authorization for the EHR abstraction.  
 
b. Protection Against Risk In the Study 
All researchers are trained and follow federal HIPAA regulations, which require specific protocols for the 
transferring, storage, and reporting of protected health information. The data we analyze at OHSU will 
not contain any personal identifiers. In the manner described above, each patient whose EHR data we use 
will be assigned a unique subject identification. All data will be stored on secure servers as well as secure, 
password protected computers in the locked offices. Any signed consent forms will be stored in a locked 
office. 
 
All of the data we collect will be securely exported under the direct supervision of the investigator and co-
investigators. Secure communications by electronic mail are also provided using secure messaging 
software. In addition, all data will be computerized and managed on OCHIN HIPAA-compliant 
computers. All data will be stored and backed up password-protected secure servers. The research team 
will meet at regular intervals to identify and discuss any data collection and management issues that may 
arise. 
 
Potential Benefits of the Proposed Research to Subjects and Others  
 
This study will greatly enhance our understanding of how to improve primary care delivery and readiness 
to change in primary care clinics. Participating clinics will benefit from study findings that help identify 
barriers and facilitators of successful APM implementation.      
 
Importance of the Knowledge to be Gained  

 
This study has the potential to greatly improve primary care delivery and patient health in CHCs by: 
 
 
Filling a critical knowledge gap: traditional fee-for-service reimbursement models are not achieving 
improvements in population health at reduced cost, so testing alternative payment methods is imperative;   
 
Measuring the real-time effects of health policy changes on care delivery in vulnerable populations;  
 
Using mixed methods to evaluate an APM “natural experiment” in CHCs, including using CHCs’ 
electronic health record (EHR) data to assess process and outcomes of a policy reform;  
 
Engaging CHCs in a study that will inform ongoing / future policy and practice changes; 
 
Informing the potential dissemination of alternative payment methods to more CHCs and other 
primary care settings; and 
 
Determining what types of practice change can be affected with payment reforms – information 
crucial to future efforts to combine payment reform with other forms of technical assistance.
 
Inclusion of Children  
 
Data from children and adults aged 2-64 from CHCs will be utilized for the quantitative portion of this study. 
CHCs care for a large proportion of racial/ethnic minority and low-income populations. We will also involve 
adults who are CHC staff in study site visits and interviews; we will not be interacting directly with children.   
 
Literature Cited: 
 
1. 
Angier H, DeVoe JE, Tillotson CJ, Wallace LS. Changes in US Family Health Insurance Coverage Patterns: 
Comparing 2003 to 2008. Family Medicine. 2013;45(1):26-32. 
2. 
Gold R, Nichols G, Muench J, et al. Implementing an Integrated Care Setting’s Diabetes QI Initiative in Safety Net 
Clinics: A Practice-Based Randomized Trial. Paper presented at: 5th Annual NIH conference on the Science of 
Dissemination and implementation: Science at the Crossroads2012; Bethesda, MD. 
3. 
Gold R, Muench J, Hill C, et al. Collaborative development of a randomized study to adapt a diabetes quality 
improvement initiative for federally qualified health centers. Journal of Health Care for the Poor & Underserved. 
Aug 2012;23(3 Suppl):236-246. 
4. 
DeVoe JE, Petering R, Krois L. A Usual Source of Care:  Supplement or Substitute for Health Insurance Among 
Low-Income Children?  Recent Findings from Oregon. Med Care. 2008;46(10):1041-1048. 
5. 
Angus L, Devoe J. Evidence that the citizenship mandate curtailed participation in Oregon's Medicaid family 
planning program. Health Aff. Apr 2010;29(4):690-698. 
6. 
Bauer J, Angus L, Fischler N, Rosenberg KD, Gipson TF, Devoe J. The impact of citizenship documentation 
requirements on access to medicaid for pregnant women in Oregon. Maternal and child health journal. Aug 
2011;15(6):753-758. 
7. 
Carlson M, DeVoe J, Wright B. Short-term impacts of coverage loss in a Medicaid population:  early results from a 
prospective cohort study of the Oregon Health Plan (OHP) Annals of family medicine. 2006;4(4):391-398. 
8. 
DeVoe J, Fryer G, Phillips R, Green L. Receipt of preventive care among adults: insurance status and usual source 
of care. American journal of public health. 2003;93:786-791. 
9. 
DeVoe J. The unsustainable US health care system: a blueprint for change. Annals of family medicine. May-Jun 
2008;6(3):263-266. 
10. 
DeVoe JE, Graham A, Krois L, Smith J, Fairbrother GL. "Mind the gap" in children's health insurance coverage: 
does the length of a child’s coverage gap matter? Ambulatory Pediatrics. 2008;8(2):129-134. 
11. 
DeVoe JE, Krois L, Edlund C, Smith J, Carlson NE. Uninsured but Eligible Children: Are Their Parents Insured? 
Recent Findings From Oregon. Med Care. 2008;46(1):3. 
12. 
DeVoe JE, Krois L, Edlund T, Smith J, Carlson NE. Uninsurance among children whose parents are losing 
Medicaid coverage: Results from a statewide survey of Oregon families. Health Serv Res. Feb 2008;43(1 Pt 2):401-
418. 
13. 
DeVoe JE, Krois L, Stenger R. Do children in rural areas still have different access to health care? Results from a 
statewide survey of Oregon's food stamp population. The Journal of rural health : official journal of the American 
Rural Health Association and the National Rural Health Care Association. Winter 2009;25(1):1-7. 

 
 
12 
14. 
DeVoe JE, Tillotson CJ, Wallace LS, Selph S, Graham A, Angier H. Comparing types of health insurance for 
children: a public option versus a private option. Med Care. Sep 2011;49(9):818-827. 
15. 
DeVoe JE, Ray M, Krois L, Carlson MJ. Uncertain health insurance coverage and unmet children's health care 
needs. Fam Med. Feb 2010;42(2):121-132. 
16. 
DeVoe JE, Tillotson C, Wallace L. Uninsured children with insured parents. Journal of the American Medical 
Association. 2008;300(16):1904-1913. 
17. 
Angier H, Wiggins N, Gregg J, Gold R, Devoe JE. Engagement of Community Health Workers In a Health Policy 
Research Project Through a Novel Technique: The Community Retreat. Paper presented at: North American 
Primary Care Research Group2011; Banff, Alberta. 
18. 
Willson JM, Wallace LS, DeVoe JE. Are State Medicaid Application Enrollment Forms Readable? Journal of 
Healthcare for the Poor and Underserved. 2009;20(423-431). 
19. 
Hansen JS, Wallace LS, DeVoe JE. How readable are Spanish-language Medicaid applications? Journal of 
immigrant and minority health / Center for Minority Public Health. Apr 2011;13(2):293-298. 
20. 
DeVoe JE, Tillotson CJ, Wallace LS, Lesko SE, Pandhi N. Is health insurance enough? A usual source of care may 
be more important to ensure a child receives preventive health counseling. Maternal and child health journal. Feb 
2012;16(2):306-315. 
21. 
Wallace LS, DeVoe JE, Hansen JS. Assessment of Children's Public Health Insurance Program enrollment 
applications: a health literacy perspective. Journal of pediatric health care : official publication of National 
Association of Pediatric Nurse Associates & Practitioners. Mar-Apr 2011;25(2):133-137. 
22. 
DeVoe JE, Westfall N, Crocker S, et al. Why Do Some Eligible Families Forego Public Insurance for Their 
Children? A Qualitative Analysis. Family Medicine. 2012;44(1):39-46. 
23. 
Wright B, Carlson M, Edlund C, DeVoe J, Gallia C, Smith J. The impact of increased cost sharing on Medicaid 
enrollees. Health Affairs. 2005;24(4):1106-1116. 
24. 
Yamauchi M, Carlson MJ, Wright BJ, Angier H, Devoe JE. Does Health Insurance Continuity Among Low-income 
Adults Impact Their Children's Insurance Coverage? Maternal and child health journal. Feb 23 2012. 
25. 
Wallace N, McConnell KJ, Gallia C, Smith J. How Effective are Co-Payments in Reducing Expenditures for Low-
Income Adult Medicaid Beneficiaries? Experience from the Oregon Health Plan. Health services research. 
2008;43(2):515-530. 
26. 
Handel DA, McConnell KJ, Wallace N, Gallia C. How Much Does Emergency Department Use Affect the Cost of 
Medicaid Programs? Annals of Emergency Medicine. 2008;51(5):614-621. 
27. 
Lowe RA, McConnell KJ, Vogt ME, Smith JA. Impact of Medicaid cutbacks on emergency department use: the 
Oregon experience. Annals of Emergency Medicine. Dec 2008;52(6):626-634. 
28. 
Wallace NT, McConnell KJ, Gallia CA, Edlund TD. Benefit policy and disenrollment of adult Medicaid 
beneficiaries from the Oregon health plan. Journal of Health Care for the Poor & Underserved. Nov 
2010;21(4):1382-1394. 
29. 
Ku L. Collateral Damage: Children Lose Coverage When Their Parents Lose Health Insurance. Washington DC: 
Center on Budget and Policy Priorities;2007. 
30. 
Crabtree BF, Miller WL, Stange KC. Understanding practice from the ground up. J Fam Pract. Oct 
2001;50(10):881-887. 
31. 
Cohen D, McDaniel RR, Jr., Crabtree BF, et al. A practice change model for quality improvement in primary care 
practice. J Healthc Manag. May-Jun 2004;49(3):155-168; discussion 169-170. 
32. 
Cohen DJ, Balasubramanian BA, Isaacson NF, Clark EC, Etz RS, Crabtree BF. Coordination of health behavior 
counseling in primary care. Annals of family medicine. 2011;July(9):5. 
33. 
Cohen DJ, Flocke SA, Lawson P, Casucci B. How teachable moments are co-constructed to delivery health behavior 
advice during primary care visits. American Academy of Communication in Health Care International Conference. 
South Carolina2007. 
34. 
Crabtree BF, Miller WL. Doing Qualitative Research, 2nd Ed. Thousand Oaks, CA: Sage Publications, Inc.; 1999. 
35. 
Miller WL, Crabtree BF. The dance of interpretation. Doing Qualitative Research, 2nd Ed. Thousand Oaks, CA: 
Sage Publications, Inc.; 1999:127-143. 
36. 
Borkan J. Immersion/crystallization. In: Crabtree BF, Miller WL, eds. Doing qualitative research. 2nd ed. Thousand 
Oaks, CA: Sage Publications, Inc.; 1999:179-194. 
37. 
Gold R. The Guide to Conducting Research in the OCHIN Practice Management Data. Portland, OR: Kaiser 
Permanente Center for Health Research;2007. 
38. 
Gold R, DeVoe J, Shah A, Chauvie S. Insurance continuity and receipt of diabetes preventive care in Oregon's 
CHCs. Med Care. 2008;47(4):431-439. 
39. 
DeVoe JE, Gold R, McIntire P, Puro J, Chauvie S, Gallia CA. Electronic health records vs Medicaid claims: 
completeness of diabetes preventive care data in community health centers. Annals of family medicine. Jul-Aug 
2011;9(4):351-358. 
40. 
Hayes RJ, Alexander NDE, Bennett S, Cousens SN. Design and analysis issues in cluster-randomized trials of 

 
 
13 
interventions against infectious diseases. Statistical Methods in Medical Research. 2000;9:95-116. 
41. 
US Department of Health & Human Services. Community Health Centers and the Affordable Care Act in 2011: 
Increasing Access to Affordable, Cost Effective, High Quality Care. 2011; 
http://www.healthcare.gov/news/factsheets/2011/08/communityhealthcenters08092011a.html. Accessed July 20, 
2012. 
42. 
The White House. The Obama Administration and Community Health Centers. Washington, D.C.2012. 
43. 
US Department of Health and Human Services. Initial core set of health care quality measures for Medicaid-eligible 
adults. Federal Register. 2012;77(2):286-291. 
44. 
Centers for Medicare & Medicaid Services. Initial Core Set of Health Care Quality Measures for Adults Enrolled in 
Medicaid (Medicaid Core Set): Technical Specifications and Resource Manual for Fiscal Year 2013. 2013; 
http://www.medicaid.gov/Medicaid-CHIP-Program-Information/By-Topics/Quality-of-Care/Downloads/Medicaid-
Adult-Core-Set-Manual.pdf. Accessed March 8, 2013. 
45. 
Centers for Medicare & Medicaid Services. Eligible Professional Clinical Quality Measures (CQMs). 2011-2012; 
http://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/CQM_Through_2013.html. 
Accessed March 8, 2013. 
46. 
Oregon Metrics and Steering Committee. Oregon Metrics and Scoring Committee. 2013; 
http://www.oregon.gov/oha/Documents/MetricsScoringCommitteeMaterials020113.pdf. Accessed March 4, 2013. 
47. 
Gold R, Angier H, Mangione-Smith R, et al. Feasibility of Evaluating the CHIPRA Care Quality Measures in 
Electronic Health Record Data. Pediatrics. Jul 2012;130(1):139-149. 
48. 
Casciato A, Angier H, Milano C, Gideonse N, Gold R, DeVoe JE. A practice-based approach to demonstrate the use 
of pediatric quality measures in outpatient electronic health record data. Journal of the American Board of Family 
Medicine. 2012;25(5):686-693. 
49. 
Dean C, Nielsen J. Generalized linear mixed models: a review and some extensions. Lifetime Data Analysis. 
2007;13(4):497-512. 
50. 
Lipsitz SR, Laird NM, Harrington DP. Generalized estimating equations for correlated binary data: using the odds 
ratio as a measure of association. Biometrika. 1991;78(1):153-160. 
51. 
Kauermann G, Carroll RJ. A note on the efficiency of sandwich covariance matrix estimation. Journal of the 
American Statistical Association. 2001;95(456):1387-1396. 
52. 
Luo Z, Gardiner JC, Bradley CJ. Applying propensity score methods in medical research: pitfalls and prospects. 
Medical Care Research and Review. 2010;67(5):528-554. 
53. 
Duan NH, Manning WG, Morris CN, Newhouse JP. A Comparison of Alternative Models for Demand for Medical 
Care. Journal of Business & Economic Statistics. 1983;1(2):115-126. 
54. 
Manning WG. The logged dependent variable, heteroscedasticity, and the retransformation problem. J Health Econ. 
Jun 1998;17(3):283-295. 
55. 
Manning WG, Mullahy J. Estimating log models: to transform or not to transform? J Health Econ. Jul 
2001;20(4):461-494. 
56. 
McConnell KJ, Wallace NT, Gallia CA, Smith JA. Effect of Eliminating Behavioral Health Benefits for Selected 
Medicaid Enrollees. Health services research. 2008;43(4):1348-1365. 
57. Ai CR, Norton EC. Interaction terms in logit an probit models. Economics Letters. 2003;80(1):123-129.