The Alabama Healthy Marriage and Relationship Education Initiative
Phase: NA
Conditions: Healthy Marriage, Healthy Relationship
Sponsor: Auburn University

Impact Evaluation 
Analysis Plan Template 
for HMRF Grantees 
Alabama Healthy
Marriage and Relationship 
Education Initiative 
Analysis Plan
February 27, 2020
NCT03158714

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
1 
Impact Evaluation Analysis Plan Template for HMRF Grantees 
According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not 
required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB 
control number for this collection is 0970-0356; this number is valid through 6/30/2021.  Public reporting burden for 
this collection of information is estimated to average 8 hours, including the time for reviewing instructions, gathering 
and maintaining the data needed, reviewing the collection of information, and revising it. This collection of 
information is voluntary for individuals, but the information is required from Grantees. 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
2 
Impact Evaluation Analysis Plan: “The Evaluation of the Alabama Healthy Marriage and 
Relationship Education Initiative (AHMREI)” 
Grantee 
Grantee Name: Auburn University, Auburn, AL 
Project Lead: Francesca Adler-Baeder, Ph.D. 
Email address: francesca@auburn.edu 
Evaluator 
Evaluator’s Organization: Auburn University, Auburn, AL 
Evaluator Lead: Julianne McGill, Ph.D. 
Email address: mclanjm@auburn.edu 
 
 
 
 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
3 
IMPACT EVALUATION ANALYSIS PLAN 
THE EVALUATION OF THE ALABAMA HEALTHY MARRIAGE AND 
RELATIONSHIP EDUCATION INITIATIVE (AHMREI) 
A. Research questions 
This section presents the primary and secondary research questions that will be assessed in 
the impact evaluation of ELEVATE and Couples Connecting Mindfully (CCM). 
1. Primary research questions: Program Impact 
1. What is the impact of Elevate when compared to the no-program control group on the 
change in couple relationship skills from baseline to the immediate post-program follow-
up? 
2. What is the impact of Couples Connecting Mindfully when compared to the no-program 
control group on change in couple relationship skills from baseline to the immediate 
post-program follow-up? 
3. What is the impact of Elevate when compared to the no-program control group on 
change in individual mental health from baseline to the six-month follow-up?  
4. What is the impact of Couples Connecting Mindfully when compared to the no-program 
control group on change in individual mental health from baseline to the six-month 
follow-up?  
5. What is the impact of Elevate when compared to the no-program control group on 
change in couple satisfaction from baseline to the six-month follow-up? 
6. What is the impact of Couples Connecting Mindfully when compared to the no-program 
control group on change in couple satisfaction from baseline to the six-month follow-
up? 
2. Secondary research questions 
Program Impact: 
1) What is the impact of Elevate when compared to the no-program control group on 
change in individual mental health from baseline to the one-year follow-up?  
2) What is the impact of Couples Connecting Mindfully when compared to the no-
program control group on change in individual mental health from baseline to the 
one-year follow-up?  
3) What is the impact of Elevate when compared to the no-program control group on 
change in relationship satisfaction from baseline to the one-year follow-up?  
4) What is the impact of Couples Connecting Mindfully when compared to the no-
program control group on change in relationship satisfaction from baseline to the 
one-year follow-up?  
 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
4 
Process of change: 
In order to explore our logic model assumption that immediate change in program targets (i.e. 
couple relationship skills) immediately following program participation leads to longer-term 
outcomes in couple functioning we will examine a path model. See Figure 1.  
5) Do changes immediately following Elevate participation in couple relationship skills 
predict couple satisfaction at 6 months, accounting for baseline?   
6) Do changes immediately following Couples Connecting Mindfully participation in 
couple relationship skills predict couple satisfaction at 6 months, accounting for 
baseline?   
   
B. Description of the intended intervention and counterfactual condition 
This section provides a brief description of the intervention being evaluated. 
1. Intervention condition(s) 
Intended Components: The intervention consists of participation in either the Elevate 
curriculum or the Couples Connecting Mindfully (CCM) curriculum.  The couples education 
interventions are multi-session programs written for delivery to couples in community-based 
educational settings. Each curriculum involves 6-week group-based classes in which couples 
receive content on relationship skills.  
Intended Content: The content of the curricula is focused on key relationship skills and 
information that promotes couple quality and stability (see The National Extension Relationship 
and Marriage Education Model www.nermen.org). These include: self-care, intentionality, 
conflict and stress management, development of intimate knowledge of partner, development of 
a couple identity, caring couple behaviors, and development of supportive social connections. 
Planned Dosage and Implementation Schedule: Both curricula are explicitly research-
informed and consist of 6 sessions delivered in separate consecutive weekly sessions of 
approximately 2 hours each. This implementation schedule was consistent across the 10 
implementation sites. 
Intended Delivery: Programs are delivered in accessible community-based facilities that 
support a productive learning environment.  Trained facilitators (male/female teams) deliver the 
programs.  
Target Population: Each program is broadly offered in the 10 different implementation sites 
and are intended to serve couples aged 19 and older in self-defined committed couple 
relationships.  
Education and Training of Staff: Most facilitators have a background in family services 
and/or education and all have a minimum of a Bachelor’s degree.  Specifically, 73% of 
facilitators have a bachelor’s degree; 24% have a Master’s degree; and 3% have a doctorate. 
Each facilitator is provided a comprehensive 2-day training on program curriculum by program 
developers prior to implementation. Facilitators then complete fidelity checklists following each 
program session to track program components that were implemented each week. Classes were 
also routinely monitored by project staff for fidelity and quality. Ongoing technical assistance 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
5 
was provided to facilitators by scheduled phone calls and webinars with the project and 
Evaluation Staff to discuss program implementation, evaluation design, data collection, 
participant engagement, and evaluation feedback.  
2. Counterfactual condition 
Those assigned to the control group receive a list of family resources in the area. This occurs 
at time of random assignment and is sent via email. The location-specific resource list was 
developed by the staff at each site so they are specific to the community of study participants. 
The resource list included sources for employment, housing, child services, health and well-
being, education, addiction treatment, legal services, and other multi-need services. The partner 
agencies’ staff was able to update the resource list throughout recruitment. No other programs 
and services are provided to them as part of the intervention study.   
Table 1. Description of intended intervention and counterfactual components 
and target populations 
Component 
Curriculum and content 
Dosage and schedule 
Delivery 
Target Population 
Intervention 
Relationship 
skills 
sessions 
CCM: Healthy relationships 
curriculum: communication skills; 
relationship skills; mindfulness-
based stress relief skills 
ELEVATE: Healthy relationships 
curriculum; communication skills; 
relationship skills, self-care skills 
6 sessions (2 hours each)  
Group lessons 
provided at the 
intervention’s facilities 
by two trained 
facilitators in every 
session 
Adult couples in a 
self-defined 
committed couple 
relationship 
Counterfactual 
Location-
Specific 
Resource 
ListN/A 
Control respondents were 
provided location specific 
resource lists. Resources 
covered: employment/work, 
housing, child services, mental & 
physical health services, 
education, addiction treatment & 
support, legal services, and multi-
need services.   
Received at random 
assignment  
Sent via email  
Adult couples in a 
self-defined 
committed couple 
relationship 
 
Table 2. Staff training and development to support intervention and 
counterfactual components  
Component 
Education and initial training of staff 
Ongoing training of staff 
Intervention 
Relationship skills 
sessions 
Facilitators are male and female and hold at least a 
bachelor’s degree and received four days of initial 
training (2 days per curriculum).  
Facilitators participated in regular technical 
assistance calls and webinars with project and 
evaluation staff. Observations and feedback 
sessions were also utilized. 
Counterfactual 
Location-Specific 
Resource List 
Staff at partner agencies developed the resource 
list for their specific community.  
Staff at partner agencies were able to update the 
resource list before each cohort recruitment period. 
 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
6 
C. Study design 
This section provides a brief description of the study design and the process for creating 
intervention and comparison groups. 
1. Sample formation 
Couples were recruited into our study in five separate cohorts over two and a half years. 
Recruitment began in August 2016 and was completed in January 2018. Approximately a month 
before the beginning of programming for each cohort, recruitment across the 10 implementation 
sites began. The implementation sites are: 1) Alabama Cooperative Extension System in Elmore 
County, 2) Circle of Care, 3) Family Guidance Center, 4) Family Success Center, 5) Hope Place, 
6) IMPACT Family Counseling, 7) Auburn University, 8) Parents and Children Together 
(PACT), 9) Sylacauga Alliance for Family Enhancement (SAFE), and 10) Tuscaloosa’s One 
Place (TOP).  
Recruitment of couples into the impact evaluation occurred through distribution of flyers 
(approved by the Auburn University Institutional Review Board) at community agencies, such as 
family resources centers, local churches, libraries, and other settings where community members 
were likely to frequent. Additionally, flyer information was posted online via the AHMREI 
Facebook page and website. 
All adults who indicated they were in a committed couple relationship (married or 
nonmarried) and who expected to attend as a couple if selected for a program group were eligible 
to participate in the local impact evaluation study. Anyone under age 19 and adult individuals 
who intended to attend as singles were not eligible to participate in the impact evaluation. 
a. Random assignment process 
Couples were randomly assigned if at least one of the individuals in the couple completed 
the baseline survey. The evaluation lead conducted the random assignment by implementation 
site. Couple IDs developed by the nFORM system were entered into separate site-specific SPSS 
files by evaluation staff. The evaluation staff utilized the SPSS random number generator to 
arbitrarily code the couples as a 1 (Elevate), 2 (Couples Connecting Mindfully), or 3 (control) in 
order to randomly assign the study participants. Following completion of random assignment, the 
evaluation staff notified each study participant via an email with their assignment. For those in 
the program groups, their email included times and location of the assigned class. Each step of 
the random assignement process was implemented by the evaluation staff.  
2. Data collection 
The following data collection procedures were approved by the Auburn University 
Institutional Review Board (IRB) that oversees Human Subjects protection in research. The 
original approval was provided on August 15, 2016 and is still currently up to date after yearly 
renewals. When couples or a member of a couple expressed interest in participating, we 
collected basic enrollment information and further explained the nature of the evaluation project 
using the recruitment script. We explained random assignment procedures, reminding couples 
they had a 2 out of 3 chance to be assigned to one of the program groups and a 1 out of 3 chance 
they would be assigned to the control (no program) condition.  We explained they would be 
asked to complete a series of surveys on the web-based system, Qualtrics, at the following 
times: baseline, immediate post- program (8 weeks after baseline), 6-month follow-up to 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
7 
baseline, 1-year follow-up and 2-year follow up. We explained that participants complete the 
surveys individually rather than as a couple. We offered the option of filling out these surveys 
through a link provided via email or by coming to the agency/center to complete on a tablet in 
the rare case that a participant does not have personal access to technology. We also explained 
the participant compensation plan: participants who completed a survey received $50 each at 
baseline, immediate post-program, 6-month, and 1-year, and $100 at 2-year follow-up.  
 
Once a couple agreed to participate, they provided contact information (first name, last 
name, birthdate, address, phone, email) to a staff member/educator in order to generate a 
participant profile in the nFORM system. Enrollment took place both on the phone and in-
person. Information was provided for each member of the couple, resulting in an individual 
profile, and both a participant and couple ID in the nFORM system were generated. 
Immediately following enrollment in the nFORM system, each participant indicated their 
willingness to participate in the study by reading and signing the informed consent letter 
approved by the Auburn University Institutional Review Board. 
Two weeks prior to program start, all study participants who signed and returned an 
informed consent letter were emailed a link to a baseline Qualtrics survey to be completed 
within a week. One week before the start of classes, participants who completed (or who had a 
partner complete) a baseline survey were randomly assigned by site (block random 
assignment), using a random number generator based on the couple ID (generated upon 
enrollment through nForm) to either one of the two program groups or to the comparison group.  
Participants in the program groups also were asked at their first class to complete the 
nForm surveys (i.e., applicant characteristics and Entrance surveys). All participants who 
received programming were asked to complete the nForm Exit survey at the last class meeting. 
Note:  the data from the Entrance and Exit surveys are not included in our impact evaluation.  
All impact study participants (program groups and controls) were emailed a link to the 
immediate post-program Qualtrics survey within one week after the class series ending.  
This survey process was repeated at the six month, one year, and two year marks after 
baseline survey completion.  Participants were initially contacted one month prior to the data 
collection timepoint and given until one month after the data collection timepoint to complete 
the survey. Several prompts and reminders to complete the survey were sent during the “open” 
period.  
The evaluation staff at the central A.U. office tracks data as it is collected and initiates 
compensation via mailed checks within the week of receiving a completed survey from a 
participant.  
Efforts to reduce attrition after enrollment: 
Following enrollment in the study, the following engagement strategies were utilized: 
• Weekly emails and/or texts to remind participants of study activities and deadlines, such 
as receipt of the baseline survey and class start dates for those assigned to program 
groups 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
8 
• Email and/or text throughout the week of survey completion until survey was submitted 
• An email and/or text one day prior to class start for the program groups 
Engagement strategies to retain participants in the study for the 6-month, 1-year, and 
2-year follow up include: 
a. Birthday card emailed to participant 1 week prior to birthday. 
b. Anniversary card emailed to participant 1 week prior to anniversary date (anniversary 
date information was collected on the pre-program survey). 
c. Receipt of “save the date” (email, mail, and/or text) 1 week prior to emailing follow-up 
survey links. 
d. Reminder texts and emails during the “open” period for survey completion that included 
offering scheduled appointments for coming to agency to complete on iPad there.  
 
Table 3. Key features of the data collection  
 
Data source 
Timing of data 
collection 
Mode of data 
collection 
Party responsible for 
data collection 
Start and end 
date of data 
collection 
Intervention 
Intervention 
group study 
participants 
Enrollment (baseline)  
 
Online Qualtrics 
survey 
nFORM Applicant 
Characteristics 
Survey & 
Entrance Survey 
Evaluation Staff  
Implementation Staff 
September 2016 
through January 
2018 
 
 
End of intervention (6 
weeks after program 
start) 
Online Qualtrics 
survey 
nFORM Exit 
Survey 
Evaluation Staff 
Implementation Staff 
October 2016 
through March 
2018 
 
 
6 month follow-up 
 
Online Qualtrics 
survey 
Evaluation staff 
March 2017 
through August 
2018 
 
 
1 year follow-up 
Online Qualtrics 
survey 
Evaluation Staff 
September 2017 
through February 
2019 
 
 
2 year follow-up 
Online Qualtrics 
survey 
Evaluation Staff 
September 2018 
through February 
2020 
Counterfactual 
Control group 
study 
participants 
Enrollment (baseline) 
Online Qualtrics 
survey 
Evaluation staff 
September 2016 
through January 
2018 
 
 
End of intervention (6 
weeks after program 
start) 
Online Qualtrics 
survey 
 
Evaluation Staff 
October 2016 
through March 
2018 
 
 
6 month follow-up 
 
Online Qualtrics 
survey 
Evaluation staff 
March 2017 
through August 
2018 
 
 
1 year follow-up 
Online Qualtrics 
survey 
Evaluation Staff 
September 2017 
through February 
2019 
 
 
2 year follow-up 
Online Qualtrics 
survey 
Evaluation Staff 
September 2018 
through February 
2020 
 
3. CONSORT diagram 
[CONSORT diagrams are attached in the appendices.] 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
9 
D. Analysis 
1. Outcome measures 
Primary: 
Couple relationship skills will be measured using 32 items from the Couple Relationship 
Skills Inventory (CRSI).  This measure was constructed to match the core relationship skills and 
predictors of couple quality emphasized in the HMRE programs provided.  Items are taken from 
several established and validated social science measures assessing commitment and intentionality 
in the relationship, intimate knowledge about partner, sense of friendship and togetherness, caring 
behaviors, conflict management behaviors, and connections to support outside the relationship. 
We have recently conducted validity tests of the factor structure (i.e., confirmatory factor analyses) 
and have evidence of validity and reliability of the full measure for our sample. Response anchors 
range from 1 (Very strongly disagree) to 7 (Very strongly agree). Composite scores will be created 
at the individual level since they reflect the individual’s level of skills. Higher scores indicate 
greater couple relationship skills. Cronbach’s alpha for men (α = .91) and women (α = .92) indicate 
excellent reliability. 
 
Individual mental health will be measured using the SF-12 Mental Component Summary 
score (Sanda, Wei, & Litwin, 2002). Items are asked on differing scales (e.g., 1 to 3 or 1 to 5) with 
different anchor responses (e.g., all of the time to none of the time, or not at all to extremely). 
Following the SF-12 scoring instructions, item responses are first standardized, then summed, and 
standardized further by adding 60.75781, per the SF-12 scoring instructions. Higher scores indicate 
better individual mental health. 
 
Relationship satisfaction will be measured using an abbreviated version of the Couple 
Satisfaction Index (Funk & Rogge, 2007) utilized in previously published studies.  The 3 items 
are: “I have a warm and comfortable relationship with my partner,” “How rewarding is your 
relationship with your partner?,” and “In general, how satisfied are you with your relationship?”  
Response anchors range from 0 (Not at all) to 5 (Completely). Composite scores will be created 
based on responses from each member of the couple individually and from both members of the 
couple combined to represent a couple level variable.  Higher scores indicate higher relationship 
satisfaction. Cronbach’s alpha for men (α = .85) and women (α = .89) indicate good reliability. 
 
Table 4. Description of outcome measures used to answer impact analysis 
primary research questions 
Outcome name 
i.Description of the outcome measure 
ii.Source of the 
measure 
iii.Timing of measure 
Couple 
Relationship Skills  
The outcome measure will be an individual composite 
score of 32 questions asked on a 7-point Likert scale.  
Qualtrics survey 
Baseline and 6 weeks 
after baseline  
Mental Health 
The outcome measure will be an individual score 
calculated from 12 questions per the SF-12 instructions.  
Qualtrics survey 
Baseline and 6-month 
follow-up 
Relationship 
Satisfaction 
The outcome measure will be the couples’ composite 
score from 3 questions on a 5-point Likert scale.  
Qualtrics survey 
Baseline and 6-month 
follow-up 
Participant Group 
The predictor measure is based on participant group (1) 
or control group (0).  
Evaluation Team 
Records 
Random Assignment 
Secondary: 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
10 
 The outcomes measures for the secondary research questions are the same as the measures 
used in the primary research question. See above for details on those measures.  
 
Table 5. Description of outcome measures used to answer impact analysis 
secondary research questions 
Outcome name 
iv. 
Description of the outcome measure 
v. 
Source 
of the 
measure 
vi. 
Timing of 
measure 
Couple 
Relationship Skills  
The outcome measure will be an individual composite 
score of 32 questions asked on a 7-point Likert scale.  
Qualtrics survey 
1-year follow-up  
Mental Health 
The outcome measure will be an individual score 
calculated from 12 questions per the SF-12 instructions.  
Qualtrics survey 
1-year follow-up 
Relationship 
Satisfaction 
The outcome measure will be the couples’ composite 
score from 3 questions on a 5-point Likert scale.  
Qualtrics survey 
1-year follow-up 
Participant Group 
The predictor measure is based on participant group (1) 
or control group (0).  
Evaluation Team 
Records 
Random Assignment 
 
2. Data preparation 
Survey responses for each cohort across each time point will be downloaded from Qualtrics 
into separate SPSS databases. These databases will be merged based on participant individual ID 
in order to create a master dataset with survey responses from each of the five cohorts across 
each of the five time points. Basic descriptives will be run on all variables of interest to assess 
for outliers or impossible scores on each scale.  
Furthermore, consistency in demographic data will be assessed across participants and 
timepoints. Evaluation staff will use a master code list containing participant IDs and 
demographic data and nFORM to verify accuracy in demographic data across participants and 
timepoints. Responses that are obvious keystroke errors of respondents based on master code list 
information and inconsistency with other timepoint reports will be corrected.  We will also 
assess couple demographics reported between dyads for agreement.  If a couple disagrees on 
marital status (i.e., one reports married and one reports nonmarried), we will defer to the 
nonmarried report. 
Data will be assessed for patterns of missingness using a thorough plan that explores percent 
of missingness by item, by composite score, and by respondent. We will explore at each 
timepoint whether those missing data within treatment and control groups differ systematically 
from those who do respond in order to determine whether there is a risk of generalizability bias.  
CONSORT diagrams will clearly describe the number of couples who consented to be in the 
study, who complete the baseline assessment, who were randomly assigned to the program and 
control groups, and who responded to the immediate post-program and each follow-up survey.   
 
Another table will be completed to illustrate the sample characteristics of the groups, as well 
as the means and standard deviations of the measures of interest, at baseline and at follow-up 
points.  Statistical tests will be conducted (t-tests for continuous variables and chi square 
difference tests for categorical variables) to determine whether there are differences between 
respondents in the control group and respondents in the program group. Additionally, the 
Hedge’s g effect size for group differences will be calculated based on the analytic sample. 
 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
11 
Another table will describe differences in measured baseline characteristics of those who 
respond at each timepoint and those who do not. We will conduct statistical tests of between-
group differences. Hedge’s g effect size for group differences will be calculated. Patterns of 
missingness between treatment groups also will be assessed in order to determine whether there 
is a risk of causal validity bias, which results from systematic differences between responders in 
the treatment and control groups.   
 
After assessing overall and differential attrition, we will examine patterns of missingness by 
case and item. Our plan includes several approaches to addressing missingness, depending on the 
findings from the initial examination of missingness. If it is determined that we have a very low 
rate of missing values, there are no systematic differences between respondents and 
nonrespondents within and between groups, and loss of those cases would not affect power in 
our planned analyses, cases with missing values will be deleted in listwise fashion. Additionally, 
to address item nonresponse and to create scale scores, mean imputation methods will be used. 
Specifically, when an individual participant has responded to 80% of the items that make up the 
scale, they will be retained for the analyses and the mean of the items will be imputed. This will 
be done separately for treatment and control group members.  
 
3. Analytic sample 
The analytic sample will differ based on the research question because response rates are 
different at immediate post-program and at 6-month follow-up.  The current numbers presented 
in the CONSORT diagram illustrates the number of respondents at each time point who 
completed the survey (i.e., prior to imputation). We anticipate using the procedures described in 
the previous section for addressing missing data to maximize the retention of cases and for 
ensuring valid comparability of the treatment and control groups; however, because we are using 
mean imputation for those who respond to 80% or more of the measure, these numbers may 
change slightly based on the outcome of interest. Attendance at the program for those assigned to 
the program condition is not a condition of inclusion in the analytic sample (see explanation 
below for the Intent to Treat design).   
For primary research questions #1 & #2 we will utilize 753 couple data sets over 2 time 
points (baseline and immediate post-program). Specifically, 249 couples randomly assigned to 
Elevate, 249 couples randomly assigned to CCM, and 255 couples randomly assigned to the 
control condition responded to the immediate post-program survey. For primary research 
questions #3-6 we will utilize 671 couple data sets over 3 time points (baseline, immediate post-
program, and 6-month follow-up). Specifically, 221 couples randomly assigned to Elevate, 231 
couples randomly assigned to CCM, and 219 couples randomly assigned to the control condition 
responded at the 6-month follow-up. 
Assessment of baseline equivalence 
Differences among groups on baseline demographics (e.g., race, age, income, marital status, 
parent status, and education level) and baseline levels of the outcomes of interest will be assessed 
using independent t-tests for continuous variables and crosstabs with chi-squares for categorical 
variables wherein statistical significance in differences is determined at p < .05.  Additionally, 
the Hedge’s g effect size for group differences will be calculated based on the analytic sample. 
These tests of difference will be completed with non-imputed data.  

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
12 
Furthermore, there was no condition crossover.  That is, no control respondents participated 
in the HMRE program during the study period. Although not necessarily considered cross-over 
or contamination of services, 10% of control respondents did seek and/or receive services that 
are always available in the community at the immediate post-program survey, 15% at the 6-
month follow-up, and 18% at the 1-year follow-up. Because case management is offered to all 
HMRE program participants, involvement in community services was higher for those randomly 
assigned to a program group. Specifically, 34% of program participants sought or received 
services available in the community immediately following the program, 36% at the 6-month 
follow-up, and 37% at the 1-year follow-up. This information was gathered from questions on 
the Qualtrics surveys administered by the evaluation team asking about services outside of the 
intervention offerings. Regardless of whether other services were accessed during the study 
period, all control respondents will be included in the control group for analyses to maintain an 
intent to treat model for analyses.   
4. Analytic approach 
 
We will use an “Intent to Treat” (ITT) approach, which is the most rigorous approach to 
evaluation and yields the most reliable efficacy results for testing a program in a “real world” 
setting (OPRE, 2010; Weiss & Jacobs, 2008; Wood et al., 2014). Group comparisons are based on 
assignment to one of the three possible groups, rather than completion. All 6 primary questions 
center on the assessment of program impact (or treatment effect) of program participation.  
Questions 1 and 2 will include a test of program impact on immediate change (approximately 6 
weeks after baseline) in couple relationship skills for Elevate and for CCM, the primary areas of 
skills-training for the intervention.  We will utilize multi-level modeling to assess differences 
between groups at the immediate post-program timepoint. With this method, we can account for 
baseline levels of the skills and account for the nesting of the individual within the couple. To test 
for program impact at the 6-month mark of the other two primary outcomes (questions 3-6) we 
will employ recommended methods for randomized trials. Although most studies of Couple 
Relationship Education using comparison groups and included in meta-analyses have used 
repeated measures analysis of covariance to test for treatment effects, suggestions are that growth 
curve modeling as a variant of mixed-effects models (or random effects, random-regression) is a 
more powerful technique than the repeated-measures analysis of covariance (e.g., Schultz, Cown 
& Cown, 2006; Gueorguieva & Krystal, 2004; Amato, 2014). The results of multilevel Growth 
Curve Model can provide information on differences at timepoints between groups, as well as the 
program impact on rates of change. We will model these statistical analyses with a Bayesian 
approach in Stan.  
 
 
We will use SPSS and Mplus (Muthén & Muthén, 1998-2014), and the COMPLEX 
specification, which corrects for the non-independence of cases by clustering analyses by 
couple/site ID. Group differences across the 2 groups (i.e., one of the two program groups and the 
control group) will be tested by constraining means to be equal and using the chi-square difference 
test to determine if variant and invariant models are significantly different. Findings are considered 
statistically significant based on p < .05, two-tailed test. Cohen’s effect size d will be calculated to 
evaluate effect size, .10 = small, .25 = medium, .40 = large (Cohen, 1988).  
 
 
Sensitivity analyses will include testing alternative methods/approaches to ensure the 
robustness of the results and the appropriateness of the analytic approach taken. In couple datasets 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
13 
it is common to test results separately for men and women to compare to the results for methods 
that use the individual nested with the couple. We will also compare results in which both 
individual and cluster level covariates are used and results in which only individual covariates are 
use. Even though random assignment was used, we will fit models with and without covariates to 
test whether it is necessary to control for differences that may exist at baseline between groups.  
5. Additional planned analyses 
Secondary Research Questions. In addition to the primary research questions central to the 
local impact evaluation, there are several other important questions we plan to explore in order to 
more fully understand the experience of participation in HMRE programs at our sites. 
To address secondary research question #1-4, we will employ the same multi-level 
modeling methods described above to further test program impact  of program particitipation.  
To address secondary research question #5, whether changes immediately following 
program participation in couple relationship skills predicts couple satisfaction at 6 month 
follow-up, accounting for baseline level, a path model will be fit to test the change on change 
pathway. Structural equation modeling (SEM) techniques allow us to test the “spillover” 
hypothesis. Goodness of fit indices will be assessed to establish fit of data to the model.  
 
 
 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
14 
 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
15 
 
 
 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
16 
Appendix A: Primary Research Question Measures  
Couple Relationship Skills 
 
Please rate how strongly you disagree or agree with each of the following: 
 
Very 
Strongly 
Disagree 
Strongly 
Disagree 
Disagree 
Mixed 
Agree 
Strongly 
Agree 
Very 
Strongly 
Agree 
9. I want this relationship to stay 
strong no matter what rough times 
we encounter 
 
 
 
 
 
 
 
10. I commit effort every day to 
making my relationship work 
 
 
 
 
 
 
 
11. I always think about how my 
choices could affect my 
relationship 
 
 
 
 
 
 
 
12. I always make an effort to 
focus on my partner's strengths 
 
 
 
 
 
 
 
 
Please rate how strongly you disagree or agree with how well each statement describes you: 
 
Very 
Strongly 
Disagree 
Strongly 
Disagree 
Disagree 
Mixed 
Agree 
Strongly 
Agree 
Very 
Strongly 
Agree 
13. I know my partner's current 
life stresses 
 
 
 
 
 
 
 
14. I know some of my partner's 
major aspirations and hopes in life 
 
 
 
 
 
 
 
15. I know my partner's current 
major worries 
 
 
 
 
 
 
 
16. I know my partner pretty well 
 
 
 
 
 
 
 
 
 
 
 
Please rate how strongly you disagree or agree with each of the following: 
 
Very 
Strongly 
Disagree 
Strongly 
Disagree 
Disagree 
Mixed 
Agree 
Strongly 
Agree 
Very 
Strongly 
Agree 
1. I have the power to manage the 
challenges in my life. 
 
 
 
 
 
 
 
2. I ask for help from others when 
needed. 
 
 
 
 
 
 
 
3. I recognize my strengths. 
 
 
 
 
 
 
 
4. I manage the stress in my life. 
 
 
 
 
 
 
 
5. I eat healthy meals every day. 
 
 
 
 
 
 
 
6. I exercise at least 3 or more 
times a week. 
 
 
 
 
 
 
 
7. I get 7-8 quality hours of sleep 
every night. 
 
 
 
 
 
 
 
8. I have quiet time for myself 
every day. 
 
 
 
 
 
 
 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
17 
In the past month, how often would you say the following events occurred between you and your partner? 
 
Never 
Less than 
once a 
month 
Once or 
twice a 
month 
Once or 
twice a 
week 
3-4 
times a 
week 
Once 
a 
day 
More 
often than 
once a day 
17. Had a stimulating exchange of 
ideas 
 
 
 
 
 
 
 
18. Engage in and/or talk about 
outside interests together 
 
 
 
 
 
 
 
19. Make time to touch base with 
each other 
 
 
 
 
 
 
 
 
On average, how often in the past month did you: 
 
Never 
Less than 
once a 
month 
Once or 
twice a 
month 
Once or 
twice a 
week 
3-4 
times a 
week 
Once 
a 
day 
More 
often than 
once a day 
20. Say "I love you" to your 
partner 
 
 
 
 
 
 
 
21. Initiate physical affection with 
you partner (e.g., kiss, hug) 
 
 
 
 
 
 
 
22. Share emotions, feelings, or 
problems with your partner 
 
 
 
 
 
 
 
23. Tell my partner things I 
appreciate about him/her and how 
much I care for him/her 
 
 
 
 
 
 
 
 
First, rate how strongly you disagree or agree with how well each statement describes you during the past month in 
a typical disagreement: 
 
Very 
Strongly 
Disagree 
Strongly 
Disagree 
Disagree Mixed Agree Strongly 
Agree 
Very 
Strongly 
Agree 
24. I am able to see my partner's point of 
view and really understand it, even if I don't 
agree 
 
 
 
 
 
 
 
25. When things "get heated" I suggest we 
take a break to calm down 
 
 
 
 
 
 
 
26. I can easily forgive my partner 
 
 
 
 
 
 
 
27. I shout or yell at my partner 
 
 
 
 
 
 
 
28. I blame, accuse, or criticize my partner 
 
 
 
 
 
 
 
 
Please rate how strongly you disagree or agree with how well each statement describes you and your partner: 
 
Very 
Strongly 
Disagree 
Strongly 
Disagree 
Disagree 
Mixed 
Agree 
Strongly 
Agree 
Very 
Strongly 
Agree 
29. Many of our friends are 
friends of both of us 
 
 
 
 
 
 
 
30. We know people who care 
about us and our relationship 
 
 
 
 
 
 
 
31. If we were to need help getting 
by or encountered a crisis, we 
would have friends and family to 
rely on 
 
 
 
 
 
 
 
32. As a couple, we try to help 
others in need 
 
 
 
 
 
 
 

ANALYSIS PLAN FOR HMRF LOCAL IMPACT EVALUATIONS: [AUBURN UNIVERSITY] 
 
 
 
18 
Individual Mental Health 
 
Excellent 
Very 
Good 
Good 
Fair 
Poor 
1. In general, would you say your health is: 
 
 
 
 
 
The following questions are about activities you might do in a typical day. Does your health now limit you in 
these activities? If so, how much? 
 
Yes, a lot 
limited 
Yes, 
limited a 
little 
No, not 
limited at 
all 
2. Moderate activities, such as moving a table, pushing a vacuum, 
bowling, or playing golf 
 
 
 
3. Climbing several flight of stairs 
 
 
 
During the past month, how much of the time have you had any of the following problems with your work or 
other regular daily activities as a result of your physical health? 
 
All of the 
time 
Most of 
the 
time 
Some of 
the time 
A little of 
the time 
None of 
the time 
4. Accomplishing less than you would like 
 
 
 
 
 
5. Did work or other activities less carefully 
than usual 
 
 
 
 
 
During the past month, how much of the time have you had any of the following problems with your work or 
other regular daily activities as a result of any emotional problems? 
6. Accomplishing less than you would like 
 
 
 
 
 
7. Did work or other activities less carefully 
than usual 
 
 
 
 
 
How much of the time in the past month… 
8. Have you felt calm and peaceful? 
 
 
 
 
 
9. Did you have a lot of energy? 
 
 
 
 
 
10. Have you felt downhearted and depressed? 
 
 
 
 
 
11. How much of the time has your physical 
health or emotional problems interfered with 
social activities (like visiting friends, relatives, 
etc.)? 
 
 
 
 
 
 
Not at all 
A little 
Bit 
Moderately 
Quite a 
bit 
Extremely 
12. During the past month, how much did pain 
interfere with your normal work (including 
both work outside the home and housework)? 
 
 
 
 
 
Relationship Satisfaction 
 
Not at All 
A little 
Somewhat 
Mostly 
Almost 
Completely 
1. I have a warm and comfortable 
relationship with my partner.  
 
 
 
 
 
 
2. How rewarding is your 
relationship with your partner?  
 
 
 
 
 
 
3. How satisfied are you with your 
relationship?