Post Affordable Care Act: Evaluation of Community Health Centers
Conditions: Chronic Disease
Sponsor: Oregon Health and Science University

 
1 
 
 
 
 
 
 
 
 
 
A Study Manual for: 
 
The PACE Study 
 
Post Affordable Care Act: Evaluation of Community Health Centers 
  
AHRQ  R01HS024270 
NCT 02657499 
 
January 1. 2019  
 
 
A partnership between: 
Oregon Health & Science University 
OCHIN, Inc. 
 
 
 

 
2 
Manual of Procedures Summary ................................................................................................. 3 
Overview of Study ...................................................................................................................... 4 
Funding ................................................................................................................................... 4 
Table 1. Study Staff. ............................................................................................................ 4 
ClinicalTrials.gov Protocol Submission ................................................................................... 4 
Specific Aims.......................................................................................................................... 4 
Data Sources ........................................................................................................................... 5 
OCHIN EHR Data. ........................................................................................................... 5 
Linkage between EHR and Medicaid administrative data. ................................................ 5 
Study Population and Study Period ......................................................................................... 5 
Table 2. Study population stratified by expansion status and state ....................................... 6 
Stakeholder Engagement ......................................................................................................... 6 
Data Analyses ............................................................................................................................. 7 
Project Bibliography ................................................................................................................. 21 
Regulatory Requirements .......................................................................................................... 25 
Institutional Review Board (IRB) .......................................................................................... 25 
Approval ........................................................................................................................... 25 
Protocol ............................................................................................................................. 25 
Data Use Agreements and Amendments ................................................................................ 25 
Table 3. Data Use Agreements .......................................................................................... 25 
Decision Log ............................................................................................................................ 26 
Table 4. Decision Log ....................................................................................................... 26 
Appendix 1. IRB Approval ....................................................................................................... 29 
Appendix 2. Project Protocol .................................................................................................... 31 
Appendix 3. Data Use Agreements ............................................................................................ 37 
 
 
 
 
 
 
 
 
 

 
3 
Manual of Procedures Summary 
This manual of procedures provides an overview of the PACE project, and is meant to serve as a 
resource to facilitate the replication of this study in other settings. An overview of the study 
describes the project aims, as well as information about the study team, funding, and data 
sources. Further sections describe the regulatory aspects of the study, major project decisions, 
and the analyses conducted. 
 

 
4 
Overview of Study 
Funding 
This work was supported by the Agency for Healthcare Research and Quality grant number 
R01HS024270, and was funded September 2015 through September 2017. 
Table 1. Study Staff. 
Name 
Project Role 
Institution 
Jennifer DeVoe, MD, DPhil 
Principal Investigator 
OHSU 
Heather Angier, PhD, MPH  
Co-Project Director, Co-Investigator 
OHSU 
Nathalie Huguet, PhD 
Co-Project Director, Co-Investigator 
OHSU 
Miguel Marino, PhD 
Co-Investigator, Biostatistician 
OHSU 
Lewis Raynor, PhD, MS, MPH 
Co-Investigator 
OCHIN 
Megan Hoopes, MPH 
Data Analyst -  Site PI (Year 2- NCE) 
OCHIN 
Heather Holderness, MPH 
Project Manager 
OHSU 
Nate Warren, MPH  
Project Manager 
OCHIN 
Jean O’Malley, MPH 
Data Analyst 
OHSU 
Rachel Springer, MS 
Data Analyst 
OHSU 
Steele Valenzuela, MS 
Data Analyst 
OHSU 
David Ezekiel-Herrera, MS 
Data Analyst 
OHSU 
Teresa Schmidt, MS 
Data Analyst 
OCHIN 
Pedro Rivera-Ortega 
Data Programmer Year 1 
OCHIN 
Allison O’Neil 
Data Analyst Year 1 
OCHIN 
Carlyn Hood 
Project Manager Year 1 
OCHIN 
Kaleb Keaton 
Project Manager Year 1 
OHSU  
 
ClinicalTrials.gov Protocol Submission 
ClinicalTrials.gov Identifier: NCT02657499 
 
Specific Aims 
Aim 1. Compare pre-post health insurance status, primary care, mental health, and dental 
visits, and receipt of preventive services, as well as changes in payer mix among ADVANCE 
CHCs in states that did and did not expand Medicaid.  Hypothesis 1a: CHCs in expansion states 
will experience an increase in overall visits and visits paid by Medicaid, relative to non-
expansion states. Hypothesis 1b: The percentage of Medicaid visits in CHCs' overall payer mix 
will increase more significantly in expansion-states relative to non-expansion states.  
 
Aim 2. Examine pre-post utilization of CHC services (including receipt of preventive services) 
by newly insured compared to already insured and uninsured patients. Hypothesis 2: Newly 
insured patients will have a significant increase in overall CHC visits compared to (a) already 
insured and (b) uninsured patients.  
 
Aim 3: Measure pre-post changes in overall utilization of healthcare services and Oregon 
Medicaid expenditures among newly insured compared to already insured patients. 

 
5 
Hypothesis 3a: There will be an increase in utilization of CHC services and a decrease in use of 
services external to the CHCs (e.g., emergency department) for newly insured patients, but 
these utilization rates will remain the same for already insured patients. Hypothesis 3b: There 
will be an initial increase, then plateau, in overall Medicaid expenditures among newly insured 
relative to already insured patients.  
 
Data Sources 
ADVANCE EHR Data. We used electronic health record (EHR) data from the Accelerating Data 
Value Across a National Community Health Center Network (ADVANCE) clinical data research 
network (CDRN) data warehouse. The ADVANCE CDRN is a unique ‘community laboratory’ for 
research with underrepresented populations that includes patients receiving care in safety net 
clinics. The ADVANCE CDRN data warehouse includes integrated longitudinal EHR data from 
several organizations. Data from two ADVANCE partners, OCHIN and Health Choice Network 
(HCN), were used in the PACE study. From the ADVANCE CDRN, we included CHC members ‘live’ 
on their EHR as of January 1, 2013: 225 primary care CHCs in 10 states that expanded Medicaid 
as of 1/1/2014 (California, Hawaii, Maryland, Minnesota, New Mexico, Ohio, Oregon, Rhode 
Island, Washington, and Wisconsin) and 134 primary care CHCs in 6 non-expansion states 
(Florida, Kansas, Missouri, North Carolina, Texas, Montana). We included Wisconsin as an 
expansion state because although they did not expand Medicaid to 138% FPL they opened 
enrollment to adults with eligibility criteria of 100% FPL and therefore behaved more like an 
expansion state. Montana did not expand Medicaid until after our study period (expanded 
1/1/2016), which included a one-year pre- (1/1/13-12/31/13) and a two-year post-period 
(1/1/2014-12/31/2015). Analyses included >4 million ambulatory visits among patients aged 
19-64 (expansion states: 2.6 million visits from 499,719 patients; non-expansion states: 1.5 
million visits from 370,600 patients).  
Linkage between EHR and Medicaid administrative data. Oregon Medicaid administrative data 
was linked to OCHIN Oregon clinics in the ADVANCE CDRN to assess the receipt of healthcare 
services outside the Oregon CHCs and all healthcare expenditures. 
 
Study Population and Study Period 
The overall study population was adults ages 19-64 throughout the study period, 2012-2015. 
Included patients had ≥1 ambulatory visit at an eligible facility in the study period. Included 
facilities provided adult primary care and were ‘live’ on OCHIN’s or HCN’s EHR system by the 
start of the study period (1/1/2012). EHR data were extracted on all eligible patients, 
encounters, and facilities.  
Overall, study analyses included >4 million ambulatory visits among patients aged 19-64 
(expansion states: 2.6 million visits from 499,719 patients; non-expansion states: 1.5 million 
visits from 370,600 patients).  
 

 
6 
We examined differences in utilization among patients newly gaining Medicaid coverage (newly 
insured), as compared to utilization among individuals who are already insured by Medicaid 
(already insured) and those who remain uninsured (uninsured). Additionally, in subpopulation 
analyses restricted to the state of Oregon (an expansion state), we link EHR data from Oregon 
CHC clinics to Oregon’s Medicaid administrative claims data to investigate post-expansion 
overall healthcare utilization and Medicaid expenditures among the newly insured as compared 
to the already insured. Inclusion and exclusion criteria were modified for some analyses; see 
below for details on specific paper analyses.  
 
Table 2. Study population stratified by expansion status and state 
 
2014 Expansion status 
State 
N facilities 
N patients 
Expansion 
California 
48 
163,686 
Expansion 
Hawaii 
2 
10,075 
Expansion 
Maryland 
24 
15,297 
Expansion 
Minnesota 
4 
12,725 
Expansion 
New Mexico 
25 
63,488 
Expansion 
Nevada 
3 
2,326 
Expansion 
Ohio 
15 
55,537 
Expansion 
Oregon 
121 
261,166 
Expansion 
Rhode Island 
59 
38,946 
Expansion 
Washington 
6 
24,585 
Non-expansion 
Alaska*  
1 
3,820 
Non-expansion 
Florida 
169 
412,138 
Non-expansion 
Indiana**  
10 
16,200 
Non-expansion 
Kansas 
9 
8,047 
Non-expansion 
Missouri 
3 
16,461 
Non-expansion 
Montana 
6 
13,155 
Non-expansion 
North Carolina 
21 
39,726 
Non-expansion 
Texas 
5 
11,383 
Non-expansion 
Wisconsin*** 
8 
20,593 
*Alaska expanded Medicaid to 138% FPL on 9/1/2015 and was excluded from 
some analyses; **Indiana expanded Medicaid to 138% FPL on 2/1/2015 and was 
excluded from some analyses; ***Wisconsin expanded Medicaid from 0% to 100% 
on 1/1/2014 and therefore more closely resembled an ACA expansion state. 
Wisconsin was considered an expansion state in analyses. 
 
Stakeholder Engagement 
The project team engaged a Patient Investigator (Kaye Dickerson) and a Clinician Investigator 
(Dr. Andrew Suchocki, MD) throughout the duration of the project. These stakeholders advised 
the project team during the data analysis, manuscript development, and dissemination phases 
of the project. Their involvement was facilitated through ongoing training about the research 
process, collaboration with research team about data analyses challenges, data interpretation, 

 
7 
and manuscript development, and dissemination of research aims through presentations and 
blogging. 
 
 
 
Data Analyses  
Medicaid Expansion Produces Long-Term Impact on Insurance Coverage Rates in Community 
Health Centers 
Full Citation 
Huguet, N., Hoopes, M. J., Angier, H., Marino, M., Holderness, H., 
& DeVoe, J. E. (2017) Medicaid Expansion Produces Long-Term 
Impact on Insurance Coverage Rates in Community Health 
Centers. J Prim Care Community Health. 8(4):206-212.     
Objective 
Assess change in insurance coverage rates pre- and post-ACA in 
Medicaid expansion and non-expansion states for community 
health center patients. 
Study Population 
N=875,571 patients; Inclusion: ages 19-64 with ≥ 1 visit between 
2012 and 2015. Setting: 412 primary care CHCs in 9 expansion 
and 4 non-expansion states. 
Exclusion Criteria 
Pregnant women 
Independent 
Variables 
Whether state had taken Medicaid expansion as of January 1, 
2014. 
Dependent 
Variables 
Health care delivery: rates of billed encounters (all, primary 
care, new patient, established patient), receipt of preventive 
services (preventive medicine visits, immunizations, medications 
ordered). 
 
1. Insurance type: uninsured, Medicaid-insured, privately- 
insured. 
Covariates 
Clinic-level: distributions of sex, age, race/ethnicity, primary 
language, federal poverty level. 
 
State-level: marketplace type, 2013 minimum wage, 2013 
uninsured rate, 2013 unemployment rate. 
Analysis Plan 
Post- versus pre-expansion rate ratios within expansion group; 
difference-in-difference ratios (comparing post- versus pre-
period changes between expansion states vs non-expansion 
states); and second year post- (2015) versus first year post-ACA 
(2014) rate ratios within expansion group. 
Data Set Name 
OCHIN: e:\sasroot\PPDA\P_Huguet_24mo 
Code Location 
OCHIN SAS server 
 
 

 
8 
 
 
Oregon Medicaid Expenditures After the 2014 Affordable Care Act Medicaid Expansion: Over-
time Differences Among New, Returning, and Continuously Insured Enrollees 
Full Citation 
Springer, R., Marino, M., O'Malley, J. P., Lindner, S., Huguet, N., 
& DeVoe, J. E. (2018). Oregon Medicaid Expenditures After the 
2014 Affordable Care Act Medicaid Expansion: Over-time 
Differences Among New, Returning, and Continuously Insured 
Enrollees. Med Care, 56(5), 394-402.  
Objective 
To assess health care expenditures among Medicaid 
enrollees in the 24 months after Oregon’s 2014 Medicaid 
expansions and examine whether expenditure patterns were 
different among the newly, returning, and continuously insured. 
Study Population 
Oregon adult Medicaid beneficiaries insured continuously 
from 2014 to 2015 who were either newly, returning, or 
continuously insured. 
Exclusion Criteria 
Our study population included adults aged 19-64 insured by 
Oregon Medicaid continuously from January 2014 through 
December 2015. We excluded patients without coverage on 
1/1/2014 and those with coverage gaps during the study period. 
We excluded patients with dual Medicaid and Medicare 
eligibility and whose 2014-2015 eligibility did not depend on FPL 
(e.g. pregnant women). Of 622,513 adults aged 19-64 with any 
2014 Medicaid enrollment, 230,602 (37%) remained in our 
sample. 
Independent 
Variables 
Insurance group, time, and their interaction terms, mean 
monthly expenditures. 
Dependent 
Variables 
standardized expenditures for inpatient care, prescription drugs, 
and total outpatient care in addition to subdivisions of 
outpatient: outpatient claims were classified as either 
emergency department, dental, mental and behavioral health, 
primary care evaluation, management, and 
procedures, specialist evaluation, management, and 
procedures, and primary care/specialist imaging or tests based 
on a hierarchical system involving 
procedure code, provider type, provider specialty, place of 
service, and associated costs. 
Covariates 
The patient’s age, sex, racial and ethnic background, rural 
setting, and comorbidity level as assessed by the enhanced 
Charlson comorbidity index score. 
Analysis Plan 
Retrospective cohort study using inverse-propensity weights to 
adjust for differences between groups. We summarized enrollee 

 
9 
utilization outcomes as mean monthly expenditures, running a 
separate (two-part) hurdle model for each type. 
Data Set Name 
NPAT, INPATIENT_Q42015, OUTPAT, OUTPATIENT_2015Q4, RX , 
RX_Q42015, SAMPLE, SAMPLE_with_weights 
 
FY_2015_FR_AND_CN_Table 5, Nov 2013 Fee Schedule – Excel, 
January 2014 Behavioral Health Fee Schedule, January 2014 
Mental Health Fee Schedule – Excel 
Code Location 
Box at OHSU  
 
 
 
 Medicaid Coverage Accuracy in Electronic Health Records 
Full Citation 
Marino, M., H. Angier, S. Valenzuela, M. Hoopes, M. Killerby, B. 
E. Blackburn, N. Huguet, J. Heintzman, B. Hatch, J. P. O' Malley 
and J. E. DeVoe (2018). Medicaid Coverage Accuracy in 
Electronic Health Records. Prev Med Reports 11: 297–304. 
Objective 
To evaluate the validity of EHR data for monitoring longitudinal 
Medicaid coverage and assess variation by patient 
demographics, visit types, and clinic characteristics. 
Study Population 
Oregon Medicaid patients aged 18-64 years with a Medicaid ID 
or Medicaid insurance recorded in the OCHIN EHR ≥1 billed 
health care visit (excluding dental) from 184 Oregon CHCs in the 
OCHIN network linked to state Medicaid records during the 
study time period (1/1/2013–12/31/2014). N=135,514. 
Exclusion Criteria 
Patients with no Medicaid record on EHR who also did not have 
a Medicaid ID. 
Independent 
Variables 
Insurance cohort: (1) continuously Medicaid (Medicaid recorded 
at all visits); (2) continuously not Medicaid (Medicaid not 
recorded at any visit (patients could have Medicare, private, 
VA/Military, worker’s compensation, or no coverage); (3) Gained 
Medicaid: Medicaid not recorded for all visits in 2013 and 
Medicaid recorded at all visits in 2014; and (4) Discontinuously 
Medicaid: Any combination of visit coverage that did not follow 
the definitions above. 
Dependent 
Variables 
The primary outcome was agreement between EHR and 
Medicaid data in assigning patients to one of the four cohorts. 
Covariates 
Patient demographics (sex, age, race, ethnicity, language, 
household income represented as % FPL, urban/rural, number of 
common chronic conditions, and number of encounters),  visit 
and provider types, and clinic-specific (department type and 
customers of OCHIN's billing service) characteristics. We 
included the following common chronic conditions assessed 

 
10 
from diagnostic codes in the EHR: hypertension, diabetes, 
coronary artery disease, lipid disorder, and asthma/chronic 
obstructive pulmonary disorder. 
Analysis Plan 
Visit-level: agreement, prevalence-adjusted bias-adjusted kappa 
(PABAK), sensitivity, specificity, positive predictive value (PPV), 
and negative predictive value (NPV).  
 
Patient-level: 4x4 cross-tabulation, of insurance cohorts; two-
stage logistic regression. 
Data Set Name 
PPDA Projects à InsuranceValidation à Data à CLEAN à 
patients.csv, patient_pre.csv, patient_post.csv, patient_ch.csv, 
encs_pre.csv, encs_post.csv, agree_no_final.csv 
 
Code is included under: PPDA Projects à InsuranceValidation à 
Final Code 
Code Location 
 Box at OHSU  
 
 
In Low-Income Latino Patients, Post-Affordable Care Act Insurance Disparities May Be 
Reduced Even More than Broader National Estimates: Evidence from Oregon 
Full Citation 
Heintzman, J., S. R. Bailey, J. DeVoe, S. Cowburn, T. Kapka, T. V. 
Duong and M. Marino (2017). In Low-Income Latino Patients, 
Post-Affordable Care Act Insurance Disparities May Be Reduced 
Even More than Broader National Estimates: Evidence from 
Oregon. J Racial Ethn Health Disparities 4(3): 329-336. 
Objective 
To evaluate the association of Hispanic/Latino ethnicity and 
Spanish language preference with insurance status among a 
cohort of patients accessing community health centers (CHCs) in 
Oregon; prior to ACA implementation (2013) and 1-year post-
ACA (2014). 
 
To determine if disparities in insurance coverage existed by 
race/ethnicity and language preference prior to ACA. If so, were 
such disparities mitigated by ACA implementation. 
Study Population 
N=49,392; Adults aged 21-79 in Oregon community health 
centers (CHCs) who had low income (<100% FPL), and at least 1 
primary care encounter at a study clinic in 2009-2013, had to 
have at least 1 face to face primary care encounter.  
Exclusion Criteria 
 
Independent 
Variables 
Race/ethnicity, preferred language  
Dependent 
Variables 
Insurance status: Medicaid, Medicare, private insurance, 
uninsured. 

 
11 
Categorized patients into the Medicaid and Medicare categories 
if patient had one or more encounters with these insurances.  
Covariates 
Age at beginning of study, sex, total number of primary care 
office visits in study period. 
Analysis Plan 
Chi-square for patient demographics; ANOVA to compare 
patient demographics by insurance group; GEE to account for 
correlation of patients nested with CHCs; DID/GEE to assess 
changes in odds of uninsured from pre-to post-ACA between 
Hispanic/Latino/Spanish, Hispanic/Latino/English, and non-
Hispanic White cohorts. 
Data Set Name 
PPDA Projects à PACE à HeintzmanACA à 2014data.xls, 
heintzman_k_post_aca_comparison.xls 
 
Code is included under: PPDA Projects à PACE à Heintzman 
ACA à Code 
Code Location 
Box at OHSU  
 
 
Medicaid's Impact on Chronic Disease Biomarkers: A Cohort Study of Community Health 
Center Patients.           
Full Citation 
Hatch, B., M. Marino, M. Killerby, H. Angier, S. R. Bailey, J. 
Heintzman, J. P. O' Malley and J. E. DeVoe (2017). Medicaid's 
Impact on Chronic Disease Biomarkers: A Cohort Study of 
Community Health Center Patients.  J Gen Intern Med 32(8): 
940-947. 
Objective 
To assess changes in biomarkers of chronic disease among 
community health center (CHC) patients who gained Medicaid 
coverage with the Oregon Medicaid expansion (2008–2011). 
Study Population 
OCHIN Medicaid patients within 3 chronic disease cohorts: 
patients with diabetes (N=608); patients with hypertension 
patients (N=1244); patients with hyperlipidemia patients 
(N=546). 
 
Patients gained Medicaid coverage between 2008-2011: 
Patients aged 1-+64 years; Non-pregnant, non-deceased; 
Uncontrolled biomarker result in EHR within 6 months 
before/after gained Medicaid coverage. 
Exclusion Criteria 
Pregnant women 
Independent 
Variables 
Insurance status, time, “spline” (slope change at 6 months) 
interactions between time and insurance and spline and 
insurance. 
Dependent 
Variables 
Biomarkers: 
1. HbA1c 

 
12 
2. Systolic blood pressure 
3. Diastolic blood pressure  
4. Lipids 
Covariates 
federal poverty level, comorbidities 
Analysis Plan 
Time to event assessing time from uncontrolled baseline 
measurement to follow up controlled measurement; 
Longitudinal analysis modeling the mean biomarker value over 
time; Logistic regression to estimate the odds of having a 
disease-specific medication ordered during study.  
Data Set Name 
PPDA Projects à PACE à CARDIACBiomarker à Data à all the 
files in there 
 
Code is under: PPDA Projects à PACE à CARDIACBiomarker à 
Rcode 
Code Location 
Box at OHSU  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
13 
Uninsured Primary Care Visit Disparities Under the Affordable Care Act 
Full Citation 
Angier, H., M. Hoopes, M. Marino, N. Huguet, E. A. Jacobs, J. 
Heintzman, H. Holderness, C. M. Hood and J. E. DeVoe (2017). 
Uninsured Primary Care Visit Disparities Under the Affordable 
Care Act. Ann Fam Med 15(5): 434-442. 
Objective 
To assess changes in insurance coverage at community health 
center (CHC) visits after the Patient Protection and Affordable 
Care Act (ACA) Medicaid expansion by race and ethnicity 
Study Population 
1. N=870,319 
2. 359 CHCs 
3. CHC patients aged 19-64 years 
4. Expansion states as of 1-1-14: California, 
5. Hawaii, Maryland, Minnesota, New Mexico, Ohio, 
Oregon, Rhode Island, Washington, and Wisconsin 
6. Non-expansion states as of 1-1-14: Florida, Kansas, 
Missouri, North Carolina, Texas, Montana 
Exclusion Criteria 
Pregnant women 
Independent 
Variables 
Expansion status (Medicaid expansion vs. non-expansion)  
Dependent 
Variables 
Health insurance type at each visit (Medicaid-insured, uninsured, 
privately-insured) 
Covariates 
 
Analysis Plan 
1. Difference-in-difference (DID) 
2. Difference-in-difference-in-difference (DDD) 
3. GEE models  
Data Set Name 
HA_disp_analysis_long.sas7bdat 
 
HA_disp_monthly_analysis_set.sas7bdat 
Code Location 
OCHIN archive server 
 
 

 
14 
A Cohort Study of Public Health Insurance Coverage Loss among Oregon Adolescents 
Full Citation 
Angier, H., C. J. Tillotson, L. S. Wallace, M. Marino, J. P. O' Malley, 
A. Sumic, L. Baker, C. Nelson, N. Huguet, A. Suchocki, H. 
Holderness and J. E. DeVoe (2018). A Cohort Study of Public 
Health Insurance Coverage Loss among Oregon Adolescents. J 
Health Dispar Res 11(1). 
Objective 
The purpose of this study was to identify the sociodemographics 
of adolescents transitioning to young adulthood who lost public 
health insurance which could inform future health policies for 
this vulnerable population. 
Study Population 
Oregon adolescents (17-19 years of age) with public coverage 
[January 1, 2011-December 31, 2013 (n=51,988). 
Exclusion Criteria 
Missing information on urban/rural residency status (n=4,025) 
and/or family income (n=973). 
Independent 
Variables 
Sociodemographic characteristics: age, sex, race, ethnicity, 
language, percentage of federal poverty level based on family 
income and size, history of a healthcare visit to a community 
health center included in the OCHIN network during the study 
period (2011-2013), and urbanicity. 
Dependent 
Variables 
OHP Coverage 
Covariates 
 
Analysis Plan 
Time to Event: to determine association of coverage loss with 
sociodemographic characteristics. 
Data Set Name 
analysis_nogaps_01202016.sas7bdat 
dmapochin13_nogaps_01202016.sas7bdat 
 
Code is under: 
R:\Active projects\IMPACCT\Reports\Data analysis and 
results\Teen Transition\Carrie\SAS Code and Programs\5d_MV 
Models nogaps 01112016.sas 
Code Location 
OCHIN archive server  
 
A New Role for Primary Care Teams in the United States After “Obamacare”: Track and 
Improve Health Insurance Coverage Rates. 
Full Citation 
DeVoe, J. E., H. Angier, M. Hoopes and R. Gold (2016). A New 
Role for Primary Care Teams in the United States After 
“Obamacare”: Track and Improve Health Insurance Coverage 
Rates. Family Medicine and Community Health 4(4): 63-67(65). 
Objective 
Describe efforts to longitudinally track health insurance rates 
using data from the electronic health record (EHR) of a primary 
care network and to use these data to support practice-based 
insurance outreach and assistance.  

 
15 
Study Population 
OCHIN member clinics and patients.  
Exclusion Criteria 
Not an OCHIN member clinic or patient. 
Independent 
Variables 
NA 
Dependent 
Variables 
Insurance coverage in EHR. 
Covariates 
NA 
Analysis Plan 
NA 
Data Set Name 
NA 
Code Location 
NA 
 
 
 
The Impact of the Affordable Care Act Medicaid expansion on visit rates for a patient 
population with diabetes or pre-diabetes in safety net health centers 
Full Citation 
Huguet, N., R. Springer, M. Marino, H. Angier, M. Hoopes, H. 
Holderness and J. E. DeVoe. (2018) The Impact of the Affordable 
Care Act Medicaid expansion on visit rates for a patient 
population with diabetes or pre-diabetes in safety net health 
centers. J Am Board Fam Med. 31(6):905-916 
Objective 
To compare clinic-level uninsured, Medicaid-insured, and 
privately insured visit rates within and between expansion and 
non-expansion states prior to and after the ACA Medicaid 
expansion among the three cohorts of patient populations; and, 
to assess whether there was a change in clinic-level overall, 
primary care, preventive care visits, and diabetes screening rates 
in expansion versus non-expansion states from pre- to post-ACA 
Medicaid expansion. 
Study Population 
198 primary care community health centers: Non-pregnant 
patients aged 19-64 with ≥1 ambulatory visit between 
01/01/2012-12/31/2015 (n=483,912 in expansion states; 
n=388,466 in non-expansion states). 
Exclusion Criteria 
Pregnant women  
Independent 
Variables 
Expansion status  
Dependent 
Variables 
1. Insurance coverage 
2. Healthcare services  
Covariates 
Health insurance status, sociodemographic variables, 
urban/rural, state-level factors.  
Analysis Plan 
• Rate Ratios 
• Difference in Difference 
• GEE/Poison models with sandwich variance estimators 
• Clustered models by CHC 

 
16 
• Exchangeable covariance structure  
• Sensitivity analysis  
Data Set Name 
STATES, ELIGIBLE FACILITIES, ENCOUNTERS_AV1215_pipe, 
PAT_CONSORT_08312016, FAC_URBRUR 
Code Location 
BOX  at OHSU 
 
Prescription Opioid Use Patterns, Use Disorder Diagnoses, and Addiction Treatment Receipt 
after the 2014 Medicaid Expansion 
Full Citation 
Springer, R., M. Marino, S. Bailey, H. Angier, J. O’Malley, M. 
Hoopes, S. Lindner, J. DeVoe and N. Huguet. Prescription Opioid 
Use Patterns, Use Disorder Diagnoses, and Addiction 
Treatment Receipt after the 2014 Medicaid Expansion. 
Resubmitted 12/18 to Addiction. 
Objective 
To examine rates of opioid prescribing, Opioid Use Disorder 
(OUD) prevalence, and OUD treatment among Oregon Medicaid 
enrollees after the ACA Medicaid expansion; and to understand 
how insurance history and dose type are related to OUD 
diagnosis rates, as well as whether patients with OUD in the 
expansion population (of new enrollees and those ‘returning’ 
after coverage gaps) had different MAT participation rates than 
those previously continuously enrolled. 
Study Population 
Adults aged 19+64; continuously insured by Oregon Medicaid 
1/2014-12/31/2015. N=225,295. 
Exclusion Criteria 
Patients with coverage gaps during study period; dual eligibles; 
hospice care; cancer diagnosis other than non-melanoma skin 
cancer. 
Independent 
Variables 
1. Insurance group  
2. Episode type  
Dependent 
Variables 
1. Opioid filled prescription  
2. Documented diagnosis of OUD  
3. Chronic opioid use 
4. Receipt of medication assisted treatment 
Covariates 
Age, sex, race, urban/rural, co-morbidities, zip code, federal 
poverty level, employment, and number of episodes 
Analysis Plan 
1. Inverse propensity weighting  
2. Absolute standardized mean difference  
3. Prevalences 
4. Logistic regression  
5. Cox proportional hazard 
6. Multinomial logistic regression  
Data Set Name 
OUTPAT, OUTPATIENT_2015Q4, RX , RX_Q42015, 
SAMPLE_FLAGGED_FOR_EXPORT 

 
17 
 
hospice_providers, mat_codes, opioid_ndcs_2016 
Code Location 
Box at OHSU  
 
 
Following uninsured patients through Medicaid Expansion: healthcare utilization and 
diagnosed conditions 
Full Citation 
Huguet, N., S. Valenzuela, M. Marino, H. Angier, B. Hatch, M. 
Hoopes and J. E. DeVoe. Following uninsured patients 
through Medicaid Expansion: healthcare utilization and 
diagnosed conditions. Submitted Ann Fam Med 11/18. 
Objective 
Assess healthcare utilization and diagnosed health conditions 
among a cohort of community health center patients who were 
uninsured pre-ACA and follow them 24 months post-Affordable 
Care Act (ACA). 
Study Population 
All non-pregnant patients without insurance pre-ACA, aged 19-
64 during the study period, with ≥1 ambulatory visit pre-ACA 
from 300 primary care CHCs ‘live’ on their EHR system as of 
1/1/2012 in 11 states. N=138,246. 
Exclusion Criteria 
Pregnant women, insurance coverage prior to ACA. 
Independent 
Variables 
1. Insurance group  
Dependent 
Variables 
1. Healthcare utilization 
2. Health conditions 
Covariates 
Age, sex, language, race/ethnicity, federal poverty level, and 
clinic location  
Analysis Plan 
1. Adjusted predicted probability  
2. Mixed effects ordinal logistic regression  
3. Random effects to account for clustering visits  
Data Set Name 
PAT_CONSORT_08312016 
encounters_AV1215_pipe 
FAC_URBRUR 
STATES 
PPDA_RUCA_CHARLSON 
PPDA_PREEXISTING_CONDITIONS 
Code is under: PPDA Projects -> Who_Remains_Uninsured -> 
draft4.Rmd 
Code Location 
Box at OHSU  
 
 
 
 
 

 
18 
Utilization of Community Health Centers in Medicaid Expansion and Non-expansion States, 
2013-2014. 
Full Citation 
Hoopes, M. J., H. Angier, R. Gold, S. R. Bailey, N. Huguet, M. 
Marino and J. E. DeVoe (2016). Utilization of Community Health 
Centers in Medicaid Expansion and Non-expansion States, 
2013-2014. J Ambul Care Manage 39(4): 290-298. 
Objective 
Assess CHC utilization a full year before and after the 
implementation of Affordable Care Act (ACA) Medicaid 
expansions (24 months) and describes changes in visit type 
before and after implementation. 
Study Population 
Adults 19-64 years, billed visits in pre expansion (1/1/13-
12/31/13) and post expansion (1/1/14-12/31/14) 
OCHIN member clinics. N=401,988 patients. 
Exclusion Criteria 
Pregnant women 
Independent 
Variables 
Expansion status  
Dependent 
Variables 
1. Insurance status  
2. Visit type  
Covariates 
Sex, age, race/ethnicity, urban/rural, federal poverty level, 
minimum wage, unemployment, insurance exchange type. 
Analysis Plan 
1. GEE 
2. Poisson models 
Data Set Name 
OCHIN: e:\sasroot\CATCHUP\ACA 12 month 
Code Location 
OCHIN SAS server  
 
Effect of Gaining Insurance Coverage on Smoking Cessation in Community Health Centers: A 
Cohort Study. 
Full Citation 
Bailey, S. R., M. J. Hoopes, M. Marino, J. Heintzman, J. P. 
O'Malley, B. Hatch, H. Angier, S. P. Fortmann and J. E. DeVoe 
(2016). Effect of Gaining Insurance Coverage on Smoking 
Cessation in Community Health Centers: A Cohort Study. J Gen 
Intern Med 31(10): 1198-1205. 
Objective 
To determine if uninsured community health center (CHC) 
patients who gained Medicaid coverage experienced greater 
primary care utilization, receive more cessation medication 
orders, and achieve higher quit rates, compared to continuously 
uninsured smokers. 
Study Population 
CHC patients aged 19–64 years who gained Oregon Medicaid 
coverage between 2008 and 2011 after being uninsured for ≥ 6 
months and who maintained this insurance for ≥ 6 months. 
Cohort of ‘current every day smoker’ or ‘current some day 
smoker.’ 

 
19 
Exclusion Criteria 
Non-smoking, pregnant, non-living, private insurance, Medicare 
insurance.  
Independent 
Variables 
Insurance Status  
Dependent 
Variables 
1. ‘Quit’ smoking status (yes/no) 
2. Smoking cessation medication ordered (yes/no) 
Covariates 
Usual source of care after in follow up period. 
Analysis Plan 
1. Propensity score matching  
2. Between group differences 
3. GEE for clustering 
Data Set Name 
Smoking_model_set_PSmatches.sas7bdat 
Code Location 
OCHIN archive server  
 
 
 
Observational Study of Racial/Ethnic Disparities in Health Insurance and Differences in Visit 
Type for a Population of Patients with Diabetes after Medicaid Expansion. 
Full Citation 
Angier, H., D. Ezekiel-Herrera, M. Marino, M. Hoopes, E. A. 
Jacobs, J. E. DeVoe and N. Huguet (In press). Observational 
Study of Racial/Ethnic Disparities in Health Insurance and 
Differences in Visit Type for a Population of Patients with 
Diabetes after Medicaid Expansion. J Health Care Poor 
Underserved. 
Objective 
To understand the impact of the Affordable Care Act (ACA) 
Medicaid expansion on racial/ethnic health insurance disparities 
and differences in visit type for a population of patients with 
diabetes, by assessing changes pre- versus post-ACA in 
expansion compared with non-expansion states. 
Study Population 
Adults aged 19-64, diabetes diagnosis any time during study 
period, CHCs in ADVANCE network with EHRs live as of January 
2013. 
Exclusion Criteria 
Pregnant women 
Independent 
Variables 
Expansion status  
Dependent 
Variables 
1. Health insurance status 
2. Healthcare visit types  
a. Total 
b. Type 
Covariates 
Sex, age, FPL, Charlson comorbidity index score, state-level 
factors 
Analysis Plan 
1. Incidence rates 
2. GEE Poisson models  
3. Difference in Difference  

 
20 
Data Set Name 
Analytic Data: PPDA Projects à PACE à Diabetes Disparities à 
Analytic Data 
 
Code: PPDA Projects à PACE à Diabetes Disparities à Code 
Code Location 
Box at OHSU   
 
 
 
 
 
 Assessing the prevalence of pre-existing conditions among community health center 
patients. 
Full Citation 
Huguet, N., H. Angier, M. J. Hoopes, M. Marino, J. Heintzman, T. 
Schmidt and J. E. DeVoe. Assessing the prevalence of pre-
existing conditions among community health center patients. 
Medical Care. Submitted 9/18 
Objective 
To assess the prevalence of pre-existing conditions pre- versus 
post-Affordable Care Act (ACA) for community health center 
(CHC) patients who gained insurance 
Study Population 
386 CHCs in 19 states; 78,059 non-pregnant patients aged 19-64 
uninsured at their last visit pre-ACA. 
Exclusion Criteria 
Pregnant women  
Independent 
Variables 
Medicaid expansion state status  
Dependent 
Variables 
Prevalence and types of pre-existing conditions by insurance 
status and race/ethnicity. 
Covariates 
 
Analysis Plan 
1. Prevalence 
2. Within-group prevalence  
3. GEE 
Data Set Name 
OCHIN: e:\sasroot\PPDA\Huguet_preexist cond paper 
Code Location 
OCHIN SAS server  
 
 

 
21 
Project Bibliography  
Manuscripts: 
 
1. Huguet N, Hoopes MJ, Angier H, Marino M, Holderness H, DeVoe JE. Medicaid Expansion Produces 
Long-Term Impact on Insurance Coverage Rates in Community Health Centers. J Prim Care 
Community Health. 2017;8(4):206-212. PMC5665709. 
 
2. Angier H, Hoopes M, Marino M, Huguet N, Jacobs EA, Heintzman J, et al. Uninsured Primary Care 
Visit Disparities Under the Affordable Care Act. Annals of Family Medicine. 2017;15(5):434-42. 
PMC5593726. 
 
3. Hoopes MJ, Angier H, Gold R, Bailey SR, Huguet N, Marino M, et al. Utilization of Community Health 
Centers in Medicaid Expansion and Nonexpansion States, 2013-2014. The Journal of Ambulatory 
Care Management. 2016;39(4):290-8. PMC4942402. 
 
4. Springer R, Marino M, O'Malley JP, Lindner S, Huguet N, DeVoe JE. Oregon Medicaid Expenditures 
After the 2014 Affordable Care Act Medicaid Expansion: Over-time Differences Among New, 
Returning, and Continuously Insured Enrollees. Med Care. 2018;56(5):394-402. PMC5893375.   
 
5. Marino M, Angier H, Valenzuela S, Hoopes M, Killerby M, Blackburn BE, et al. Medicaid Coverage 
Accuracy in Electronic Health Records. Prev Med Reports. 2018;11:297–304. PMC6082971. 
 
6. Hatch B, Marino M, Killerby M, Angier H, Bailey SR, Heintzman J, et al. Medicaid's Impact on Chronic 
Disease Biomarkers: A Cohort Study of Community Health Center Patients. Journal of General 
Internal Medicine. 2017;32(8):940-7. PMC5515790. 
 
7. Huguet N, Springer R, Marino M, Angier H, Hoopes M, Holderness H, et al. The impact of the 
Affordable Care Act Medicaid expansion on visit rates for diabetes in safety net health centers. 
Journal of the American Board of Family Medicine. 2018;31(6):905-16. PMID30413546 
8. Angier H, Ezekiel-Herrera D, Marino M, Hoopes M, Jacobs EA, DeVoe JE, et al. Observational Study of 
Racial/Ethnic Disparities in Health Insurance and Differences in Visit Type for a Population of 
Patients with Diabetes after Medicaid Expansion. J Health Care Poor Underserved. In Press. 
9. Angier H, Tillotson CJ, Wallace LS, Marino M, O' Malley JP, Sumic A, et al. A Cohort Study of Public 
Health Insurance Coverage Loss among Oregon Adolescents. J Health Dispar Res. 2018;11(1): 74-84. 
10. Bailey SR, Hoopes MJ, Marino M, Heintzman J, O'Malley JP, Hatch B, et al. Effect of Gaining 
Insurance Coverage on Smoking Cessation in Community Health Centers: A Cohort Study. J Gen 
Intern Med. 2016;31(10):1198-205. PMC5023615. 
11. DeVoe JE, Angier H, Hoopes M, Gold R. A New Role for Primary Care Teams in the United States 
After “Obamacare:” Track and Improve Health Insurance Coverage Rates. Family Medicine and 
Community Health. 2016;4(4):63-7(5). PMC5617364. 
12. Heintzman J, Bailey SR, DeVoe J, Cowburn S, Kapka T, Duong TV, et al. In Low-Income Latino 
Patients, Post-Affordable Care Act Insurance Disparities May Be Reduced Even More than Broader 

 
22 
National Estimates: Evidence from Oregon. J Racial Ethn Health Disparities. 2017;4(3):329-36. 
PMC5075278. 
Submitted Manuscripts: 
 
Springer R, Marino M, Bailey S, et al. Prescription Opioid Use Patterns, Use Disorder Diagnoses, and 
Addiction Treatment Receipt after the 2014 Medicaid Expansion. Addiction. Revised and Resubmitted 
11/2018. 
 
Huguet N, Valenzuela S, Marino M, et al. Following uninsured patients through Medicaid Expansion: 
healthcare utilization and diagnosed conditions. Annals of Family Medicine. Revised and Resubmitted 
11/2018. 
 
Huguet N, Angier H, Hoopes MJ, et al. Assessing the prevalence of pre-existing conditions among 
community health center patients. Medical Care. Submitted 9/2018. 
 
Bailey SR, Marino M, Ezekiel-Herrera D, et al. Tobacco Cessation in Affordable Care Act Medicaid 
Expansion States versus Non-Expansion States. Nicotine & Tobacco Research. Submitted 12/2018. 
 
Presentations: 
 
Valenzuela S, Huguet N, Springer R, Ezekiel-Herrera D, O' Malley JP, Marino M. Navigating Large-Scale 
Forest Plots Using R and Shiny. Poster Presentation presented at American Statistical Society Conference 
on Statistical Practice; Feb 15-17, 2018, 2017; Portland, OR. 
 
Springer R, Raynor L, Marino M, et al. Effects of Medicaid Expansion on Oregon Medicaid Patient 
Expenditures from 2014-2015. Oral Presentation on Completed Research presented at 45th annual 
meeting of the North American Primary Care Research Group; Nov 17-21, 2017; Montreal, Canada. 
 
Springer R, Marino M, Bailey SR, et al. Opioid Medication Expenditures in Oregon Medicaid Enrollees 
after the ACA Expansion. Poster presentation presented at AcademyHealth Annual Research Meeting; 
June 24-26, 2018; Seattle, WA. 
 
Springer R, Marino M, Bailey S, et al. Opioid Prescription Expenditures by Oregon Medicaid Enrollees 
after the 2014 Medicaid Expansion. Oral presentation presented at 46th Annual Meeting of the North 
American Primary Care Research Group; Nov, 2018; Chicago, IL. 
 
Marino M. Assessing the Impact of Health Policy Changes in Primary Care. Joint Statistical Meeting J, 
Vancouver, Canada. Assessing the Impact of Health Policy Changes in Primary Care. Oral presnetation 
presented at Joint Statistical Meeting; July 28-August 2, 2018; Vancouver, Canada. 
 
Marino M, Valenzuela S, Raynor L, et al. Validation of Medicaid Coverage in Electronic Health Records to 
Monitor Health Reform. WNAR Conference; June 27, 2017, 2017. 
 
Marino M, Valenzuela S, Hoopes M, DeVoe JE. A Two-Stage Logistic Regression Model to Identify 
Covariates that are Predictive of Agreement Between Insurance Sources. Oral Presentation presented at 
10th International Chinese Statistical Association International Conference; Dec. 19-22, 2016; Shanghai, 
P. R. China. 

 
23 
 
Marino M, Killerby M, Hatch B, et al. Change in chronic disease biomarkers over time: The role of 
Medicaid Expansion. Oral Presentation presented at 44th annual meeting of the North American 
Primary Care Research Group; 2016; Colorado Springs, CO. 
 
Marino M, Killerby M, Angier H, et al. Using Electronic Health Records for Accurate Adult Medicaid 
Coverage Continuity. 44th annual meeting of the North American Primary Care Research Group; 2016; 
Colorado Springs, CO. 
 
Marino M, Huguet N, Springer R, Angier H, Hoopes M, DeVoe JE. Studying the Impact of the Affordable 
Care Act Using Electronic Health Records. Oral Presentation presented at 12th International Conference 
on Health Policy Statistics; Jan 10-12, 2018, 2018; Charleston, SC. 
 
Marino M, Angier H, Valenzuela S, et al. The Impact of the Affordable Care Act on Diabetes Biomarker 
Control. Oral Presentation presented at AcademyHealth Annual Research Meeting; June 24-26, 2018, 
2018; Seattle, WA. 
 
Marino M, Angier H, Springer R, et al. The Affordable Care Act - Effects of Medicaid on Diabetes 
Biomarker Control. Oral Presentation on Completed Research presented at 45th annual meeting of the 
North American Primary Care Research Group; Nov 17-21, 2017; Montreal, Canada. 
 
Marino M, Angier H, Hoopes M, et al. Combining Historical Data and Propensity Score Methods in 
Observational Studies to Improve Internal Validity. Poster Presentation presented at American Statistical 
Association Conference on Statistical Practice; Feb 15-17, 2018, 2017; Portland, OR. 
 
Marino M. Using Electronic Health Records for Observational Cancer Studies. Invited Speaker presented 
at OHSU Knight Cancer Biostatistics Symposium; May, 2018, 2018; Portland, OR. 
 
Krollenbrock A, Baker D, Crosland K, et al. Developing a multi-level model for patient engagement in a 
Practice-based Research Network. 8th International Conference on Patient- and Family- Centered Care; 
June 2018, 2017; Baltimore, MD. 
 
Huguet N, Valenzuela S, Marino M, et al. Who remained uninsured after Medicaid expansion? Oral 
Presentation on Completed Research presented at 45th annual meeting of the North American Primary 
Care Research Group; Nov 17-21, 2017; Montreal, Canada. 
 
Hoopes M, Schmidt T, Winters K, et al. Prevalence and Characteristics of Cancer Survivors in Outpatient 
Safety Net Clinics. Poster Presentation presented at AcademyHealth Annual Research Meeting; June 24-
26, 2018, 2018; Seattle, WA. 
 
Hoopes M, Angier H, Suchocki A, Dickerson K, DeVoe JE. Impacts of the Affordable Care Act Insurance 
Expansions on Community Health Centers: Current and Future Directions. OCHIN Learning Forum; April 
14-16, 2016; Portland, OR. 
 
Hoopes M, Angier H, Gold R, et al. Impact of the First Year of Affordable Care Act Insurance Expansions 
on Community Health Center Encounters. 43rd annual meeting of the North American Primary Care 
Research Group; October 24-28, 2015; Cancun, Mexico. 
 

 
24 
Holderness H, O'Malley JP, Marino M, et al. Initial Source of Outpatient Services after Implementation of 
the Affordable Care Act Medicaid Expansion in Oregon. Poster presentation presented at OHSU 
Research Week; 2018, 2018; Portland, OR. 
 
Holderness H, O'Malley JP, Marino M, et al. Initial Source of Outpatient Services after Implementation of 
the Affordable Care Act Medicaid Expansion in Oregon. Poster Presentation presented at 
AcademyHealth Annual Research Meeting; 2018; Seattle, WA. 
 
Holderness H, O' Malley JP, Huguet N, et al. Site of First Primary Care Visit by Medicaid Enrollees after 
Implementation of the Affordable Care Act Medicaid Expansion. Oral Presentation presented at 46th 
Annual Meeting of the North American Primary Care Research Group; Nov 9-13, 2018, 2018; Chicago, IL. 
 
Hatch B, Tillotson C, Hoopes M, et al. Clinic factors associated with receipt of needed preventive 
services. 44th annual meeting of the North American Primary Care Research Group; 2016; Colorado 
Springs, CO. 
 
Ezekiel-Herrera D, Angier H, Springer R, et al. Modeling and Interpreting Three-Way Interactions for 
Disparities Research. Poster Presentation presented at American Statistical Association Conference on 
Statistical Practice; Feb 15-17, 2018, Rejected 2017; Portland, OR. 
 
DeVoe JE. Using Electronic Health Record Data from Community Health Centers to measure Utilization 
One Year Before and After Affordable Care Act Insurance Expansions in the United States. 3rd 
International Primary Health Care Reform Conference; March 17, 2016; Brisbane, Australia. 
 
DeVoe JE. Innovation and Discovery in Primary Care: Creating 21st Century Laboratories and Classrooms 
for Improving Health. Society for Teachers of Family Medicine Annual Spring Conference; May 2, 2016; 
Minneapolis, MN. 
 
DeVoe JE. Health Policy Roundtable: U.S. Healthcare System in Transition. Health Policy Roundtable, 
Australian Commonwealth Department of Health; March 22, 2016; Canberra, Australia. 
 
DeVoe JE. Healthcare Policy Seminar Series: U.S. Healthcare System in Transition: Before and After 
Obamacare. Healthcare Policy Seminar series, Australian Commonwealth Department of Health; March 
22, 2016; Canberra, Australia. 
 
Angier H, Huguet N, Hoopes M, et al. Prevalence and Type of Pre-existing Conditions Before and After 
the Affordable Care Act among Vulnerable Populations. Oral Presentation on Completed Research 
presented at 46th Annual Meeting of the North American Primary Care Research Group; Nov 9-13, 2018, 
2018; Chicago, IL. 
 
Angier H, Hoopes M, Marino M, Heintzman J, Huguet N, DeVoe JE. Impact of the Affordable Care Act on 
Racial and Ethnic Disparities in Medicaid Expansion vs. Non-Expansion States. 44th annual meeting of 
the North American Primary Care Research Group; 2016; Colorado Springs, CO. 
 
 

 
25 
 
Regulatory Requirements 
Institutional Review Board (IRB) 
 
A separate Regulatory Binder containing all IRB and other regulatory documents is maintained 
by the OHSU Project Manager. Refer to this binder for more details about regulatory 
documents.  
Approval  
See Appendix 1 
Protocol 
See Appendix 2 
 
Data Use Agreements and Amendments 
Table 3. Data Use Agreements 
Parties 
Document 
Purpose 
Execution 
OCHIN and OHSU 
DUA 
Allows OCHIN to share limited data 
set with OHSU 
4-6-16 
OCHIN and HCN 
DUA 
Allows OCHIN to share HCN data 
found in the ADVANCE CDM with 
OCHIN as a limited data set 
6-3-16 
OCHIN and OHSU 
DUA 
Amendment 
Adds Community Vital Signs data to 
original DUA 
5-22-17 
OCHIN and Oregon 
DAA 
Allows OCHIN to use state-owned 
Medicaid data 
6-20-16 
 
See Appendix 3 for full agreements. 
 

 
26 
Decision Log 
Table 4. Decision Log 
Study 
Influence 
Date of 
Decision 
Decision 
Context 
Decision 
PACE 
Insurance 
Validation 
Paper 
(Miguel) 
03/29/16 
PPD 
Meeting 
Will exclude Medicare/Medicaid dual eligibles 
(since we cannot determine who they are in the 
ADVANCE data). 
PACE 
Insurance 
Validation 
Paper 
(Miguel) 
03/29/16 
PPD 
Meeting 
Will exclude visits coded as being paid by another 
public payor – these don’t provide information 
about Medicaid coverage. 
PACE 
Data 
Methods 
01/12/16 
PPD 
Meeting 
Data Pull will include all states, however down the 
road if there are complications based on a state’s 
data they will be excluded. 
PACE 
Bio 
Marker 
Paper 
(Miguel) 
02/23/16 
PPD 
Meeting 
Analysts will use Oregon Medicaid administrative 
data to create insurance cohorts to validate 
ADVANCE visit based insurance groups. 
PPD 
Definition
s 
03/15/16 
PPD 
Meeting 
Patient state and clinic state will be given, 
classified as expansion vs non expansion. Patient 
state will be state at the patient’s most recent 
encounter, not historical state at date of 
encounter. 
PPD 
Data 
Methods 
03/25/16 
Journal 
Club 
Currently, the methods used within the 
"Propensity score weighting with multilevel data" 
article by Li, Zaslavsky and Landrum will be applied 
to all papers. 
PPD 
Data 
Methods 
04/01/16 
Allison/ 
Jean/ 
Megan 
Base data pull: Clinic inclusion criteria 
OCHIN: dept type=primary care, public health, or 
medical specialty specified as internal med, peds, 
or women's health. School-based health centers 
(SBHCs) included. Health Choice network: site 
type=CHC, peds, SBHC, or hospital specified as 
outpatient clinic. Exclude based on facility names 
including radiology, mental, behavioral, substance, 
dental, pharmacy, or lab. 
Facility type categorized into: primary care, public 
health, pediatrics, women's health, mobile, 
specialty, and SBHC. 

 
27 
PPD 
Data 
Methods 
04/01/16 
Allison/ 
Jean/ 
Megan 
Base data pull: the following algorithm was used to 
assign go-live dates to HCN facilities: 
- PM golive = earliest encounter date per facility 
that is >=1/1/2003 (earliest actual go-live date for 
an HCN facility is in 2003) 
- Identify earliest vitals date that is >=1/1/2003 --> 
if earliest vitals date is >= PM golive date then EHR 
golive=earliest vitals date; else if earliest vitals date 
< PM golive then EHR golive=PM golive; else if no 
vitals date, then EHR golive = null. 
PPDA 
Data 
Methods 
04/25/16 
Analyst 
Meeting 
Blood glucose and HbA1c both in consideration for 
diagnosis of diabetes (included in Nichols 
algorithm). 
PPDA 
Data 
Methods 
05/02/16 
Email 
‘Encounters’ in the OCHIN system can include non- 
face-to-face contacts such as telephone visits, 
mychart updates, etc. In past studies, we have 
typically considered ‘visits’ to be the face-to-face 
and in-clinic subset of ‘Encounters’, though this 
definition has varied across studies/analyses. At 
times ‘encounter’ and ‘visit’ has been used 
interchangeably. 
PPDA 
Data 
Methods 
05/02/16 
Email 
Other Ambulatory Visit (OA): Includes other non-
overnight ambulatory encounters such as dental 
visits, hospice visits, home health visits, skilled 
nursing visits, other non-hospital visits, as well as 
telemedicine, telephone and email consultations. 
May also include "lab only" visits (when a lab is 
ordered outside of a patient visit), "pharmacy 
only" (e.g., when a patient has a refill ordered 
without a face-to-face visit), "imaging only", etc. 
PPDA 
Data 
Methods 
05/02/16 
Email 
Ambulatory Visit (AV): Includes visits at outpatient 
clinics, physician offices, same day/ambulatory 
surgery centers, urgent care facilities, and other 
same-day ambulatory hospital encounters, but 
excludes emergency department encounters. Not 
limited to medical visits (e.g., includes 
mental/behavioral health visits). 
PPDA 
Data 
Methods 
05/02/16 
Email 
If additional encounter details, such as provider 
type, facility type, level of service, etc., are used in 
the inclusion/exclusion criteria, they should be 
clearly specified in the outlines (i.e. “OB qualifying 
encounter types”: face-to-face with x diagnoses, y 
procedures, provider type(s), etc.). 

 
28 
PPDA 
Data 
Document
ation 
05/09/16 
Analyst 
Meeting 
High-level patient inclusion/exclusion criteria for 
each paper will be tracked in columns in 
Manuscript Tracker; more detailed documentation 
(e.g., 'readme' and data dictionaries) will be 
provided with each study dataset and included as 
an attachment in the manuscript tracker. 
PPDA 
Data 
inclusion 
06/06/16 
Analyst 
Meeting 
Residents of states that are not included in study 
(but were seen at eligible facilities in one of our 19 
states) will be excluded. 
PPDA 
Data 
Methods 
06/13/16 
Analyst 
Meeting 
There may be data errors in the provider_type 
variable in the encounters file (i.e., an AV visit may 
have dentist listed as provider type – which one is 
correct?) – safer to use AV count based on 
ADVANCE data team’s large amount of work on 
this categorization. 
PPDA 
Data 
Definition 
Email 
PACE: Pre=2013, Post=2017/PREVENTD:  Pre=2012, 
Post=2020 
PPDA 
Data 
inclusion 
06/16/16 
Manuscript 
Meeting 
Keep continually insured, uninsured in pre and mix 
of all insured visits, track how many visits were 
paid by Medicaid (will be able to isolate based on 
persons insurance status), throw out public 
insurance/Medicare patients. Patient with 
continual private insurance in periods will be 
continuously insured. 
PPDA 
Data 
inclusion 
06/16/16 
Manuscript 
Meeting 
Biomarker change could change based on 
preexisting condition in Medicare group, Patients 
with Medicare record will be excluded (all papers). 
PPDA 
Data 
analysis 
8/18/16 
Manuscript 
Meeting 
Include WI as an expansion state in all analyses.  
PPDA 
Data 
Methods 
06/14/17 
Manuscript 
meeting 
HCN data for "smoking" not specific enough; using 
tobacco-only analysis instead of separating out 
type of Tobacco use (unless able to code further). 
PPDA 
Data 
Methods 
06/14/17 
Manuscript 
meeting 
Collect pre-AND post- Charlson Comorbidity Index 
score (using problem list) 
PPDA 
Data 
analysis 
1/4/18 
PPDA 
Manuscript 
Meeting 
ADVANCE data from 2017 to be pulled February 
2018. 
PPDA 
Data 
analysis 
2/14/18 
Other 
Refer to as “Enhanced Charlson Comorbidity 
Index” and cite: Charlson ME, Charlson RE, 
Marinopoulos SS, Briggs WM, Hollenberg JP. The 
Charlson Comorbidity Index is adapted to predict 
costs of chronic disease in primary care patients. J 
Clin Epidemiol. 2008; 61:1234-40. 

 
29 
 
 
 
 
 
 
Appendix 1. IRB Approval 
 
APPROVAL OF SUBMISSION 
May 1, 2017 
 
Dear Investigator: 
On 5/1/2017, the IRB reviewed the following submission: 
IRB ID: IRB00011858 
MOD or CR ID: MOD00007213 
Type of Review: Expedited-Minor Modification 
Title of Study: Impacts of the Affordable Care Act 
Title of modification Protocol Update 
Principal Investigator: Jennifer Devoe 
Funding: Name: DHHS CDCP, PPQ #: 1007159, Funding Source: 
U18 DP006116; Name: DHHS NIH Natl Cancer Inst, PPQ 
#: 1007752, Funding Source: pending; Name: DHHS 
Agency for Hlth Care Policy & Rsch, PPQ #: 1006693, 
Funding Source: R01HS024270 
IND, IDE, or HDE: None 
The IRB granted final approval on 5/1/2017.  The study is approved until 3/30/2018. 

 
30 
Review Category:  Expedited-Minor Modification 
Copies of all approved documents are available in the study's Final Documents (far right column 
under the documents tab) list in the eIRB.  Any additional documents that require an IRB 
signature (e.g. IIAs and IAAs) will be posted when signed.  If this applies to your study, you will 
receive a notification when these additional signed documents are available. 
Ongoing IRB submission requirements: 
• Six to ten weeks before the expiration date, you are to submit a continuing review to 
request continuing approval. 
• Any changes to the project must be submitted for IRB approval prior to implementation. 
• Reportable New Information must be submitted per OHSU policy. 
• You must submit a continuing review to close the study when your research is 
completed. 
Guidelines for Study Conduct 
In conducting this study, you are required to follow the guidelines in the document entitled, 
"Roles and Responsibilities in the Conduct of Research and Administration of Sponsored 
Projects," as well as all other applicable OHSU IRB Policies and Procedures. 
Requirements under HIPAA 
If your study involves the collection, use, or disclosure of Protected Health Information (PHI), 
you must comply with all applicable requirements under HIPAA. See the HIPAA and Research 
website and the Information Privacy and Security website for more information. 
IRB Compliance 
The OHSU IRB (FWA00000161; IRB00000471) complies with 45 CFR Part 46, 21 CFR Parts 50 and 
56, and other federal and Oregon laws and regulations, as applicable, as well as ICH-GCP codes 
3.1-3.4, which outline Responsibilities, Composition, Functions, and Operations, Procedures, 
and Records of the IRB. 
Sincerely, 
The OHSU IRB Office 
 
 
 
 

 
31 
Appendix 2. Project Protocol 
 
Protocol Title 
Impacts of the Affordable Care Act 
 
 
Objectives 
The purpose of this study is to understand the impacts of the Affordable Care Act (ACA) in a 
safety net population. We will assess the following in states that expanded Medicaid compared 
to those that did not or among states that expanded Medicaid: 
 
• Pre-post rates of health insurance status and payer mix 
• Pre-post access to healthcare (i.e., visits, provider workforce, receipt of recommended 
healthcare services, receipt of chronic disease specific care and outcomes; receipt of 
preventive care, appropriate care for specialized populations such as cancer survivors 
and patients with multimorbidity/complex healthcare needs)  
• Pre-post healthcare expenditures 
• Pre-post changes in health disparities (e.g., low-income, racial and ethnic minorities, 
rural versus urban) 
 
We will look at differences in access to healthcare, expenditures, and disparities among those 
who gained coverage in the post-period, those who already had coverage, those with 
intermittent coverage, and those who remained uninsured. We will also try to understand how 
aggregated measures of social determinants of health constructed from community-level 
geocoded data (i.e., fast food availability, parks, income distribution, and racial/ethnic 
breakdown) from publicly available sources, such as the US Census Bureau, change the impact 
of disparities, access, and receipt of healthcare in relation to gaining or not gaining health 
insurance. 
 
We propose to investigate how complex, and early-onset diseases affect the receipt of 
healthcare services later in life. Some of these analyses will focus on sub-group populations 
including patients with a history of cancer and those with multiple or complex chronic 
conditions. To accomplish these analyses, we will examine data from patients of all ages as 
many childhood cancers have a high likelihood of survival, and it is important to identify these 
patients as they are likely to have additional healthcare needs throughout their lifespan.  
 
 
Background 
Health insurance facilitates access to care and reduces unmet healthcare needs,1-4 yet 47 
million Americans had no coverage in 2012.5 The ACA, the largest healthcare-related legislation 
in the United States (US) since Medicare’s establishment in 1966, was enacted with the goal of 
expanding coverage to all citizens and legal residents.6 The ACA increases opportunities to gain 
health insurance, including expansions in Medicaid coverage to individuals earning ≤ 138% of 
the federal poverty level (FPL) and the establishment of health insurance marketplaces. In 
2012, the Supreme Court ruled that states were not legally required to implement Medicaid 

 
32 
expansions, and those opting out could not be penalized.7 As of December 2014, 27 states (and 
D.C.) implemented the expansion;8 the number of states is increasing. Many persons directly 
affected by these expansions are seen at community health centers (CHCs), which comprise 
much of our nation’s healthcare ‘safety net’ and serve a rapidly increasing number of patients;9 
in 2012, 36% of CHC patients were uninsured.10 The effect of the ACA on health insurance 
coverage and access to health care services is unknown, thus it is imperative to assess its impact. 
 
 
Study Design  
Secondary data analysis.  
 
We will use electronic health record (EHR) data from the Accelerating Data Value Across a 
National Community Health Center Network (ADVANCE) clinical data research network (CDRN) 
data warehouse and Oregon Medicaid administrative data.  
 
The ADVANCE CDRN is a unique ‘community laboratory’ for research with underrepresented 
populations that includes patients receiving care in safety net clinics. The ADVANCE CDRN data 
warehouse includes integrated longitudinal outpatient EHR data from several organizations, 
including OCHIN, Health Choice Network, Fenway Health, and Legacy Health.  
 
Oregon Medicaid administrative data will be linked to OCHIN Oregon clinics in the ADVANCE 
CDRN to assess the receipt of healthcare services outside the ADVANCE network clinics and all 
healthcare expenditures. 
 
Additionally, we will partner with the state cancer registries of one or more states to complete 
probabilistic linkage between our patient population and reported cancer cases. It is currently 
unknown how well personal cancer history and related data are documented in the outpatient 
EHR (i.e., data contained in the ADVANCE CDRN), thus our ability to identify and characterize 
cancer survivors may be incomplete. A linkage with one or more state cancer registries will 
supplement the CDRN data by comprehensively identifying cancer survivors in our cohort. This 
will allow us to test and validate our EHR-based algorithms, and to evaluate post-cancer care 
services provided by community health centers and the impact of insurance coverage for this 
population.  
 
 
Study Population 
a) Number of Subjects 
Approximately 2.3 million total possible subjects. All analyses will contain a subset of these 
subjects. 
 
b) Vulnerable Populations 
Children ages 0-18 will be included for methods development and analyses related to 
childhood cancer and long-term care for cancer survivors. There is little literature about 
childhood cancers in vulnerable populations (e.g., economically marginalized, uninsured, 
or Medicaid insured) and associated disparities. Furthermore, adults with a history of 
childhood or adolescent cancer may require special follow-up care and screening 

 
33 
throughout their adult lives, thus we wish to have access to their EHR records from earlier 
ages where available. 
 
c) Setting 
This research will be conducted in collaboration with OCHIN. OCHIN requests the OHSU IRB 
serve as the IRB of record for this study. 
 
d) Recruitment Methods and Consent Process 
All patients with data in the ADVANCE CDRN may be included in this study. Subsets may 
include only OCHIN EHR data, only Oregon OCHIN EHR data, only persons with certain 
diseases or conditions (e.g., cancer, diabetes, obesity), etc. For analysis of expenditures, 
Oregon OCHIN clinics from the ADVANCE CDRN will be linked to Oregon Medicaid 
administrative data using Medicaid identification number; subjects found in both datasets 
will be included. Similarly, for cancer registry linkage analyses only patients identified in 
both datasets will be included in analyses. 
 
The ADVANCE data was collected under a waiver of authorization as the data warehouse 
poses very little risk to patients, and it was not practical to consent the number of patients 
included in the dataset.  
 
 
Procedures 
Secondary data analysis will be conducted. 
 
 
Data and Specimens 
a) Sharing of Results with Subjects 
Results will not be directly shared with subjects. 
 
b) Data and Specimen Banking 
No specimens are being collected. Data is not being banked for future research.  
 
 
Data Analysis 
We will summarize baseline measures using descriptive statistics and data visualization 
methods (e.g., histograms, scatter plots) to characterize baseline data across clinic and state 
groups. We will estimate the differences in access and receipt of recommended health care. We 
will also assess health care expenditures. Our primary methodological approach will utilize 
difference-in-differences (DID) methodology. The DID approach has been frequently used by 
health economists and health services researchers to account for potential secular effects and 
changing policies that would affect both expansion and non-expansion states over time, while 
adjusting for potential confounders.11-16 
 
We will implement state random effects in clinic-level analyses and both clinic and state 
random effects in patient-level analyses to control for correlation of observations nested in 
clusters (e.g., individuals nested in clinics which are nested in states). We recognize that the 
assumption for random effects may not be met and we will also assess the robustness of our 

 
34 
assumptions by running models that treat clinics and states as fixed effects. We will formally 
test the validity of the random effects using the standard Hausman test,17 as well as assessing 
overall qualitative differences that may arise between the random and fixed effects models. 
 
We will use GLMM models,18 which offer flexible regression modeling to accommodate 
different sources of correlations (serial, intra-clinic, and intra-state), categorical and continuous 
covariates, and fixed and time-dependent covariates. These methods offer a wide range of 
parametric distributions to model the dependent variables, including logistic regression (binary 
data), beta regression (percent data), Poisson regression (count data), and Gaussian regression 
(normally distributed data). The distribution of the outcomes of interest will be examined 
before selecting an analysis model; specific models will be refined through an iterative process. 
 
Potential variables 
included in the analyses Description 
Age 
Categories 
Sex 
Male/female, other 
Race 
White, American Indian/Alaskan Native, Asian, Black, Native 
Hawaiian/Pacific Islander, Other 
Ethnicity 
Hispanic/non-Hispanic 
Household Income 
Federal poverty level; overall income; % FPL  
Language Preference 
English, Spanish, other, etc. 
Clinic Information 
Rural/urban; panel size; panel demographics; provider characteristics 
Medicaid Expansion 
Status 
ACA Medicaid expansion state/non-expansion state; date of expansion 
Visits 
Frequency and type of visits 
Healthcare Need  
Special need because of a chronic condition 
Insurance 
Coverage status and type at visit; changes in status and type over time; 
% covered over time 
Continuity of Care 
% of visits at the same site and with same provider; Continuity of Care 
Index 
Service Utilization 
Number and types of all billed encounters overall and yearly; all services 
received; disease specific services received; preventive services 
received; amount of time services are delayed 
Expenditures 
Categories; overall 
Health outcomes 
Biomarkers such as HbA1c, blood pressure, body mass index, 
cholesterol measurements, etc 
Diagnoses 
Number and type of chronic and acute conditions 
Community Vital Signs 
Community markers of social determinants of health (e.g., fast food 
availability, parks, income distribution, racial/ethnic breakdown, 
healthy food availability, income distribution, homeownership) 
Cancer registry 
elements 
Date of diagnosis, primary type, stage, sequence, histology, behavior, 
grade, insurance at diagnosis, family history, treatment, vital status, 
date of death 

 
35 
 
 
Privacy, Confidentiality and Data Security 
We will obtain EHR data for patients included in the ADVANCE CDRN data warehouse. Data 
has already been transferred to the ADVANCE data warehouse from OCHIN, HCN, Fenway 
Health, and Legacy Health. The ADVANCE data are stored securely at OCHIN. All of the 
protections currently in place for the ADVANCE data warehouse will apply to this study. We 
will request the Oregon Medicaid administrative data from the Oregon Health Authority 
and collect it under a Data Authorization Agreement. The data transferred from OCHIN to 
OHSU will be de-identified and shared under a Data Use Agreement. The dataset will be de-
identified by assigning each patient a unique subject identification that is not based on any 
personal identifiers. All data will be stored on secure servers and password protected 
computers. All data will be computerized and managed on HIPAA-compliant computers. All 
data work will be done on password-protected, HIPAA-compliant computers. All data will be 
stored and backed up on password-protected secure servers. All reports will describe 
results in aggregate form only.  
 
For cancer registry linkage activities, we will access personal identifiers sufficient for 
completing probabilistic matching; direct identifiers will be stripped from all datasets and 
replaced with anonymized patient identifiers at the completion of linkage and for the 
duration of analyses. Data access and linkage procedures will be governed by Data Access 
Agreements between each relevant state agency and OCHIN. Any data transferred from 
OCHIN to OHSU will fall under the same protocol as outline in the Data Use Agreement 
between these two institutions.  
 
 Risks and Benefits  
a) Risks to Subjects 
There is a small risk of a loss of confidentiality. 
 
b) Potential Benefits to Subjects 
There is no direct benefit to the subject for taking part in this study. 
References 
1. 
Asplin BR, Rhodes KV, Levy H, et al. Insurance status and access to urgent ambulatory 
care follow-up appointments. JAMA. 2005;294(10):1248-1254. 
2. 
Smolderen KG, Spertus JA, Nallamothu BK, et al. Health care insurance, financial 
concerns in accessing care, and delays to hospital presentation in acute myocardial 
infarction. JAMA. 2010;303(14):1392-1400. 
3. 
Burstin HR, Lipsitz SR, Brennan TA. Socioeconomic status and risk for substandard 
medical care. JAMA. 1992;268(17):2383-2387. 
4. 
Bindman AB, Grumbach K, Osmond D, et al. Preventable hospitalizations and access to 
health care. JAMA. 1995;274(4):305-311. 
5. 
Kaiser Commission on Medicaid and the Uninsured. Key Facts about Health Insurance on 
the Eve of Health Reform. Menlo Park, CA.2013. 

 
36 
6. 
Henry J. Kaiser Family Foundation. Summary of the Affordable Care Act. Menlo Park, 
CA2013. 
7. 
Supreme Court of the United States. National Federation of Independent Business v 
Sebelius. 2012; http://www.supremecourt.gov/opinions/11pdf/11-393c3a2.pdf. 
Accessed January 15, 2015. 
8. 
The Henry J Kaiser Familty Foundation. Status of state action on the Medicaid expansion 
decision. 2014; http://kff.org/health-reform/state-indicator/state-activity-around-
expanding-medicaid-under-the-affordable-care-act/. Accessed January 7, 2015. 
9. 
Morgan D. Health centers for poor, uninsured see ranks swell. 2012; 
http://www.reuters.com/article/2012/05/01/us-usa-healthcare-centers-
idUSBRE8401JL20120501?feedType=RSS&feedName=everything&virtualBrandChannel=
11563. Accessed January 15, 2015. 
10. 
National Association of Community Health Centers. A Sketch Of Community Health 
Centers, Chart Book 2014. Bethesda, MD. 2014. 
11. 
Bertrand M, Duflo E, Mullainathan S. How Much Should We Trust Differences-in-
Differences Estimates? The Quarterly Journal of Economics. 2004;119(1):249-275. 
12. 
Friedberg MW, Schneider EC, Rosenthal MB, Volpp KG, Werner RM. Association 
between participation in a multipayer medical home intervention and changes in 
quality, utilization, and costs of care. JAMA. 2014;311(8):815-825. 
13. 
Higgins S, Chawla R, Colombo C, Snyder R, Nigam S. Medical homes and cost and 
utilization among high-risk patients. The American journal of managed care. 
2014;20(3):e61-71. 
14. 
Werner RM, Duggan M, Duey K, Zhu J, Stuart EA. The patient-centered medical home: 
an evaluation of a single private payer demonstration in New Jersey. Medical care. 
2013;51(6):487-493. 
15. 
Werner RM, Konetzka RT, Polsky D. The effect of pay-for-performance in nursing homes: 
evidence from state Medicaid programs. Health services research. 2013;48(4):1393-
1414. 
16. 
Donald S, Lang K. Inference with Difference-in-Differences and Other Panel Data. The 
Review of Economics and Statistics. 2007;89(2):221-233. 
17. 
Hausman JA. Specification tests in econometrics. Econometrica. 1978;46:1251-1271. 
18. 
Dean CB, Nielsen JD. Generalized linear mixed models: a review and some extensions. 
Lifetime Data Anal. 2007;13(4):497-512. 
 

 
37 
Appendix 3. Data Use Agreements 
 

 
38 
 

 
39 
 

 
40 
 

 
41 
 

 
42 
 

 
43 
 

 
44 
 

 
45 
 

 
46 
 

 
47 
 
 

 
48 
 

 
49 
 

 
50 
 

 
51 
 

 
52 
 

 
53 
 

 
54 
 

 
55 
 

 
56 
 

 
57