Descriptive Study of the Family Champions Project
Phase: NA
Conditions: Fathers, Mothers
Sponsor: Midwest Evaluation & Research

 
 
Family Champions Project 
Study Protocol 
Clinicaltrials.gov ID: NCT05298345 
September 4, 2024 

 
 
1 
Descriptive Evaluation Analysis Plan 
The Evaluation of the Family Champions Project in Dallas, TX 
A. Description of the intended intervention(s)  
Anthem Strong Families’ Family Champions Project (FCP) offers skill-building primary curriculum and 
support services to program participants. The evidence-based TYRO Leadership and TYRO Core 
Communication curricula, totaling 18 hours, are offered as group classes by trained facilitators in which 
low-income adults learn skills related to healthy relationships and financial wellbeing. Support services, 
including case management and additional, optional classes through Anthem’s Mini-Clinic series, are 
available to all FCP participants. 
The main intervention being tested in this descriptive evaluation is the multistep continuous quality 
improvement (CQI) cycle created by Midwest Evaluation and Research (MER) that aims to improve FCP 
program implementation. The content and components of each step in the process are described below. 
1. Step 1: Real-Time Performance Assessment Reports (prepared and presented bi-weekly by 
evaluation lead and/or data manager) 
a. 
Report description - use key indicators to track overall performance trends. 
b. Purpose –  
i. Provide real time monitoring of data entry and progress toward key metrics 
ii. Familiarize CQI team staff with metrics that will be monitored and discussed 
throughout the lifetime of the project and in subsequent reports (Steps 2-5) 
c. 
Fidelity tracking – number of CQI team meetings scheduled and held where Step 1 CQI 
reports are presented to the CQI team 
d. Dosage tracking -  
i. Intended vs. actual amount received – attendance of all CQI Team members at 
meetings for Step 1 Reports (i.e., number, names, roles) documented on CQI Team 
Tracking Sheet 
e. Performance Intervention Tracking – CQI Tracking Sheet for Performance Issues and 
Interventions (ongoing entry and revision as needed) 
f. 
Feasibility Assessment –  
i. Focus on utility and user-friendliness of Step 1 report 
ii. CQI Team Survey for Step 1: administered each program year to permanent staff 
on the CQI Team 
2. Step 2: Performance Intervention Priorities Report (prepared and presented annually in 
quarter 2 of each grant year by evaluation lead) 
a. 
Report description – compare overall performance trends by service site 
b. Purpose – target improvement efforts by setting site-specific priorities for interventions 
c. 
Fidelity tracking -   
i. Intended amount of CQI to be offered based on intended report timeline 
ii. Actual amount offered (# CQI Team meetings scheduled for Step 2 Reports on 
Outlook, # CQI Team meetings where Step 2 CQI Report was presented to CQI 
Team, # Step 2 reports created) 

 
 
2 
d. Dosage tracking –  
i. Intended amount received – attendance of all members at CQI Team meetings for 
Step 2 Reports (i.e., number, names, roles) 
ii. Actual amount received – CQI Tracking Sheet for Team Attendance (See Appendix 
B) for Step 2 Reports 
e. Feasibility Assessment –  
i. Focus on utility and user-friendliness of Step 2 report 
ii. CQI Team Survey for Step 2: administered each program year to staff on the CQI 
Team 
iii. Family Champions Project staff survey for Step 2: administered each program year 
to all frontline staff who help to develop targeted, site-specific interventions 
during staff meetings with leadership and with the CQI team on a rotating basis 
3. Step 3: Performance Intervention Strategy Report (prepared for each service site in quarter 2 
of each grant year by evaluation lead) 
a. 
Report description – use key indicators to track performance trends for each site 
individually 
b. Purpose – establish a performance intervention agenda for each site 
c. 
Fidelity tracking - 
i. Intended amount to be offered based on intended report timeline 
ii. Actual amount offered (# CQI Team meetings scheduled on Outlook, # CQI Team 
meetings where Step 3 CQI Report presented to CQI Team, # Step 3 reports 
created) 
d. Dosage tracking -   
i. Intended amount received – attendance of all members at CQI Team meetings for 
Step 3 Reports (i.e., number, names, roles) 
ii. Actual amount received – CQI Tracking Sheet for Team Attendance for Step 3 
Reports 
e. Feasibility assessment –  
i. Focus on utility and user-friendliness of Step 3 report 
ii. CQI Team Survey for Step 3: administered each program year to staff on the CQI 
Team 
4. Step 4: Performance Intervention Results Reports (prepared annually in quarter 4 of each 
grant year by evaluation lead) 
a. 
Report description – examine performance trends before and after an intervention at each 
site, looking for improvements and comparing to the relevant standards 
b. Purpose – assess results of interventions on performance outputs to determine whether 
improvements were made for the relevant standard 
c. 
Fidelity tracking - 
i. Intended amount to be offered based on intended report timeline 
ii. Actual amount offered (# CQI Team meetings scheduled on Outlook, # CQI Team 
meetings where Step 4 CQI Report presented to CQI Team, # Step 4 reports 
created) 
d. Dosage tracking -   
i. Intended amount received – attendance of all members at CQI Team meetings for 
Step 4 Reports (i.e., number, names, roles) 

 
 
3 
ii. Actual amount received – CQI Tracking Sheet for Team Attendance for Step 4 
Reports 
e. Feasibility assessment –   
i. Focus on utility and user-friendliness of Step 4 report 
ii. CQI Team Survey for Step 4: administered each program year to staff on the CQI 
Team 
iii. FCP Staff Survey for Step 4: administered each program year to all frontline staff 
who develop performance interventions with the CQI team 
Table 1. Description of intended intervention components, content, dosage and implementation 
schedule, delivery, and focal populations 
Components 
Content 
Dosage and schedule 
Delivery 
Focal population 
CQI Process Step 1 Monitoring overall and 
recent program 
performance using 
nFORM operational 
reports, local 
evaluation survey 
data, and the query 
tool 
Reports are prepared 
bi-weekly and 
discussed at bi-weekly, 
hour-long CQI team 
meetings 
Virtual CQI team 
meetings are held bi-
weekly on Tuesday 
mornings; the 
evaluation lead 
facilitates meetings, 
and Step 1 reports 
are presented by the 
evaluation lead 
and/or data manager 
CQI team members 
(program director, 
program manager, 
evaluation lead, data 
manager(s)) 
CQI Process Step 2 Monitoring the same 
performance metrics 
as presented in the 
Step 1 report, but 
broken down by 
service site using 
nFORM operational 
reports, local 
evaluation survey 
data, and the query 
tool 
Reports are prepared 
annually in the second 
quarter of each grant 
year and discussed at 
standing CQI team 
meetings and program 
staff meetings 
Existing CQI team 
meetings are used to 
present and discuss 
Step 2 report findings; 
the evaluation lead 
facilitates the 
meetings and 
presents the Step 2 
reports to the CQI 
team. Following the 
CQI team discussion, 
program leadership 
shares the Step 2 
report findings with 
front line staff during 
staff meetings to 
solicit ideas for 
interventions 
CQI team members 
(program director, 
program manager, 
evaluation lead, data 
manager(s); rotating 
front line staff 
member); Anthem 
program staff (case 
managers, facilitators, 
recruitment and 
retention specialists) 
who work at the sites 
being discussed in 
the Step 2 reports 
CQI Process Step 3 Offering the same 
conclusions as the 
Step 2 report but 
focusing on one 
service site per Step 3 
report; uses nFORM 
operational reports, 
local evaluation survey 
data, and the query 
tool 
Reports are prepared 
annually in the second 
quarter of each grant 
year and discussed at 
standing CQI team 
meetings 
Step 3 reports are 
sent to the program 
director who 
distributes each site’s 
Step 3 report to the 
relevant program staff 
who work at that site 
CQI team members 
(program director, 
program manager, 
evaluation lead, data 
manager(s)); program 
staff for specific sites 
(case managers, 
facilitators) 

 
 
4 
Components 
Content 
Dosage and schedule 
Delivery 
Focal population 
CQI Process Step 4 Assessing the results 
of targeted 
interventions that were 
put in place at specific 
sites based on the 
findings of the Step 2 
and Step 3 reports 
Reports are prepared 
annually in the fourth 
quarter of each grant 
year and discussed at 
standing CQI team 
meetings 
Existing CQI team 
meetings are used to 
present and discuss 
Step 4 report findings 
(e.g., did the targeted 
metrics improve after 
implementing 
interventions? Does 
the metric now meet 
relevant standards? 
Were there any 
challenges with the 
implementation of 
interventions? How 
did frontline staff, 
leadership, and 
facility staff find the 
intervention? ); the 
evaluation lead 
facilitates the CQI 
team meetings and 
presents the Step 4 
reports to the CQI 
team. Program 
leadership share Step 
4 report findings with 
frontline staff in staff 
meetings. 
CQI team members 
(program director, 
program manager, 
evaluation lead, data 
manager(s)); program 
staff (case managers, 
facilitators, 
recruitment and 
retention specialists) 
who are able to 1) 
provide feedback on 
challenges and 
successes with the 
implementation of the 
intended intervention, 
2) see the results of 
the intervention 
Relationship skills 
workshops 
Healthy relationships 
curriculum (TYRO 
Core Communication; 
TYRO Leadership) 
A total of 18 hours, 
with six two-hour 
sessions occurring 
weekly (TYRO 
Leadership) plus two 
three-hour sessions 
occurring weekly 
(TYRO Core 
Communication) 
Group lessons 
provided online via 
Zoom or in-person at 
service sites; two 
trained facilitators 
lead every session 
Individuals in the 
greater Dallas-Fort 
Worth area with low 
income 
Table 2. Staff education and training (initial and ongoing)  
Component 
Education and initial training 
Ongoing training 
 
 
 
Relationship skills 
workshop 
All facilitators, case managers, intake 
specialists, and other members of the 
staff must complete the workshop as a 
participant before entering the certified 
TYRO Facilitator training component. 
Once the training is complete, each 
facilitator observes an experienced staff 
Facilitators complete an annual refresher 
process to ensure curriculum fidelity. That 
process includes observations by the 
Program Manager, Program Director, or 
another qualified designee. We attempt 
to provide opportunities to become re-
certified every two to three years.  Other 

 
 
5 
Component 
Education and initial training 
Ongoing training 
 
 
 
prior to undergoing observation of their 
group facilitation. Once completed, they 
are considered a certified TYRO 
curriculum facilitator. 
facilitation or case management training 
is offered through targeted webinars or 
staff development sessions. 
CQI process 
At the beginning of the grant cycle, Anthem 
held a 3-day training to discuss the specifics 
of the grant and evaluation, and the purpose 
and process of CQI were discussed in detail at 
this initial training. As new staff are 
onboarded, they receive an explanation of the 
purpose and process of CQI, both from 
program leadership and from more 
experienced staff. 
The purpose and process of CQI is reinforced 
on an ongoing basis. Starting with more 
experienced staff who fully understand their 
day-to-day roles, frontline staff are invited to 
sit in on CQI team meetings on a quarterly 
rotating basis to help them understand the 
value of the CQI process and how their role 
fits into the larger picture. CQI report findings 
are discussed with staff at weekly staff 
meetings. 
B. Outcomes study 
This section describes the research questions, evaluation enrollment process, data collection procedures, 
outcome measures, and analytic approach for the outcomes study. 
1. Research questions 
RQ1.1: Did the multistep CQI process support changes in primary workshop dosage, program attrition 
rates, or survey response rates at the program level? 
RQ2: Did meaningful improvements in primary workshop dosage, survey response rates, or program 
attrition rates vary by service site for sites that focused on improving the same metrics? 
RQ3 [secondary]: Do participants report healthier partner relationship attitude outcomes after program 
participation, as compared to baseline? 
RQ4 [secondary]: Was level of dosage associated with reporting heathier partner relationship attitude 
outcomes after program participation, as compared to baseline? 
2. Outcomes evaluation enrollment 
This study was initially approved by Solutions IRB on 03/29/2021 and has been annually reviewed and 
approved on 03/15/2022, 03/07/2023, and 3/11/2024. Additionally, revisions to the study were submitted 
as amendments and were approved on 05/17/2022, 02/15/2023, and 08/29/2023. 
Participants for the program, and thus the study, are recruited primarily from community partnerships, 
walk-ins to Anthem Strong Families, radio and social media advertising, and word of mouth referrals. 
Program service locations have varied throughout the implementation of the program but have included: 

 
 
6 
• 
Schools 
o 
Stephen C. Foster Elementary School 
o 
Herbert Marcus Elementary School 
o 
Burnet Elementary School 
• 
Treatment Centers / Transitional Facilities 
o 
Soul’s Harbor Treatment Center 
o 
Free Man House Transitional Living Facility 
o 
Shurrun’s House Sober Living Facility 
o 
Gateway Foundation Treatment Center 
• 
Vickery Meadow Youth Development Foundation 
• 
Buckner Children and Family Services, Family Hope Center at Bachman Lake 
• 
Virtual services provided via Zoom 
The projected enrollment and desired sample size for the study is 800 participants. The final report will 
include a breakdown of the number of sites, number of staff at each site, and number of participants at 
each site. 
To be eligible for the program, and thus the study, participants must be 1) low-income individuals, 2) at 
least 18 years of age, 3) with no open criminal cases (can be deferred), and 4) able to speak and 
understand English or Spanish. 
Consent for enrollment into the evaluation happens during program orientation, prior to the start of the 
first primary workshop. A consent form is provided to participants, and a script is read by either the 
facilitator or the data manager, explaining the study activities and the potential risks and benefits. 
Participants sign and date the consent form and return it to the data manager, who logs the record in a 
consent spreadsheet on MER’s secure Dropbox. 
Enrollment into the study began April 22, 2021 and is estimated to conclude February 15, 2025. 
3. Data sources and data collection 
The data source for the program implementation and performance outcome measures is nFORM 
administrative data. Program staff are responsible for entering and updating client and workshop 
information, including workshop attendance and current client status, on an ongoing basis. These data are 
summarized bi-weekly in Step 1 CQI reports with metrics at the program level and annually in Step 2-4 CQI 
reports with metrics at the site level to monitor trends in program enrollment, workshop dosage, program 
attrition rates, and survey response rates.  
The data source for participant outcomes related to healthy partner relationship attitudes is the nFORM 

 
 
7 
Healthy Marriage Adult Program survey, administered at orientation before the first primary workshop 
(entrance survey) and at the final primary workshop session (exit survey). 
Table 3. Sources of data to address the research questions  
Data source 
Timing of data collection 
Mode of data collection 
Start and end date of data 
collection 
nFORM 
administrative data 
Ongoing 
Program staff enter and 
update data related to 
enrollment, attendance, and 
client status 
Start: April 2021 
End: April 2025 
nFORM Healthy 
Marriage Adult 
Program Survey 
At orientation before the first 
primary workshop (entrance 
survey) and at the end of the final 
primary workshop session (exit 
survey) 
Online via tablet or on 
paper as necessary 
Start: April 2021 
End: April 2025 
4. Outcome measures 
The main outcomes of interest addressed by the multistep CQI process are primary workshop dosage, 
participant attrition rates, and nFORM survey response rates, as these are key program performance 
metrics with targets set by the federal funders. The first research question looks at changes in each of these 
metrics over time overall across all service sites (program level), and the second research question looks at 
changes in each of these metrics over time broken down by service site for sites that focused on the same 
metrics. This will involve an assessment of any changes in metrics after intervention strategies were 
implemented, including whether performance targets were reached and whether improvements were 
meaningfully large. Potential ceiling and floor effects are mitigated during the process of selecting which 
metrics to attempt to improve via intervention strategies (i.e., metrics that are already at or close to targets 
are not selected for intervention); however, potential ceiling and floor effects will also be addressed by 
providing baseline data for metrics targeted for interventions at each service site. 
The secondary research questions about participant-level attitude changes after program participation are 
assessed by constructing a composite measure calculated as the average of five nFORM survey items 
related to partner trust and intimacy. If more than one survey item within the composite is missing for a 
respondent, the value of the outcome measure will be set to missing.  
Table 4. Outcomes used to answer the research questions 
Research 
question  
Outcome name 
Description of the outcome measure 
Source of the 
measure 
Timing of measure 
RQ1 
Dosage 
The outcome measure is the percent of 
target primary workshop hours received 
by program participants 
nFORM operational 
report: Primary 
Workshop 
Participation Detail 
Attendance is 
recorded after each 
workshop session 
RQ1 
Attrition 
The outcome measure is the percent of 
program participants who leave the 
program without completing at least 90% 
of primary workshop hours 
nFORM operational 
report: Client Status 
Summary 
Ongoing 

 
 
8 
Research 
question  
Outcome name 
Description of the outcome measure 
Source of the 
measure 
Timing of measure 
RQ1 
Survey response 
rates 
The outcome measure is the percent of 
eligible participants who completed 
nFORM entrance and exit surveys 
nFORM operational 
report: Survey 
Completion 
Summary 
Ongoing 
RQ2 
Dosage 
The outcome measure is the percent of 
target primary workshop hours received 
by program participants at each service 
site 
nFORM operational 
report: Primary 
Workshop 
Participation Detail 
Attendance is 
recorded after each 
workshop session 
RQ2 
Attrition 
The outcome measure is the percent of 
program participants who leave the 
program without completing at least 90% 
of primary workshop hours at each 
service site 
nFORM operational 
report: Client Status 
Summary 
Ongoing 
RQ2 
Survey response 
rates 
The outcome measure is the percent of 
eligible participants who completed 
nFORM entrance and exit surveys at 
each service site 
nFORM operational 
report: Survey 
Completion 
Summary 
Ongoing 
RQ3 
Healthy partner 
relationship 
attitudes 
The outcome measure is a scale (value 
range 1-5) calculated as the average of 
five survey items measuring trust and 
intimacy 
nFORM Healthy 
Marriage Adult 
Program survey 
D10.a-e (entrance 
and exit) 
During orientation 
(entrance) and after 
the final workshop 
session (exit) 
RQ4 
Healthy partner 
relationship 
attitudes 
The outcome measure is a scale (value 
range 1-5) calculated as the average of 
five survey items measuring trust and 
intimacy 
nFORM Healthy 
Marriage Adult 
Program survey 
D10.a-e (entrance 
and exit) 
During orientation 
(entrance) and after 
the final workshop 
session (exit) 
A. Implementation study 
This section describes the research questions, the data used to answer the research questions, and the 
methods used to analyze the data and describe the findings of the process or implementation study. 
1. Research questions 
The research questions for this implementation study relate to fidelity to, and engagement with, the 
planned multistep CQI process. The implementation study seeks to understand how closely the CQI 
process followed the original outlined plan and how understandable, useful, and engaging program staff 
found each step of the CQI process. Additionally, the implementation study seeks to understand what 
contextual factors affected the implementation of the multistep CQI process and to identify best practices 
and lessons learned about implementing a CQI process that could be shared with the broader field. 
Table 5. Research questions for each implementation element 
Implementation element 
Research question 
Fidelity 
• To what extent were the five steps of the CQI process carried out as intended 
each grant year? 
• What were the unplanned adaptations to the key CQI process components? 

 
 
9 
Implementation element 
Research question 
Engagement 
• How understandable did CQI team members, and program staff broadly for Step 2 
and Step 4, find the reports for each step of the CQI process? 
• How engaged were CQI team members and program staff in each step of the CQI 
process? 
Context 
• How well did the implementation of the multistep CQI process go for program and 
evaluation staff? 
• What external events affected the implementation of the multistep CQI process? 
• What are some best practices and lessons learned for improving the multistep CQI 
process? 
2. Implementation evaluation enrollment 
The implementation study was initially approved along with the outcome study by Solutions IRB on 
03/29/2021 and has been annually reviewed and approved three times on 03/15/2022, 03/07/2023, and 
3/11/2024. Additionally, revisions to the study were submitted as amendments and were approved on 
05/17/2022, 02/15/2023, and 08/29/2023. 
The sample for the implementation evaluation is made up of staff from Anthem Strong Families. The final 
report will include information about staff demographics, role types, and participation levels. Permanent 
members of the CQI team are surveyed after each step of the CQI process has been completed, and the 
program staff more broadly are surveyed after reviewing Step 2, 3, and 4 reports that are relevant to the 
service sites where they work with participants. Staff are also interviewed during annual site visits. 
Because the data sources for the implementation evaluation are staff satisfaction surveys that are not 
administered to program participants, no consent is collected, and no incentives are offered for survey 
completion. 
3. Data sources and data collection 
The data sources for the implementation evaluation are CQI process tracking logs (See Appendix C), staff 
satisfaction surveys administered on Qualtrics (see Appendix A), and staff interviews. 
The evaluation lead and data manager from Midwest Evaluation and Research are responsible for tracking 
and logging the delivery of interventions and reports that are associated with each step of the multistep 
CQI process. When program staff decide on a targeted intervention to implement at a given site, 
evaluators document when (with which cohort, or on which specific date) and where (which service site or 
sites) the intervention is being put into place. The date that each CQI report is delivered is also 
documented. Additionally, CQI meeting attendance is tracked on an ongoing basis after each bi-weekly 
meeting to assess CQI team engagement with the process. 
The evaluation lead is responsible for creating and administering staff satisfaction surveys for each step of 
the multistep CQI process. After reviewing Step 2, 3, and 4 reports annually with the CQI team, anonymous 
satisfaction surveys are delivered to CQI team members by the evaluation lead using an anonymous 
Qualtrics link via email. Although Step 1 reports are delivered bi-weekly to CQI team members, satisfaction 
surveys for the Step 1 reports are sent out annually. After the Program Director presents Step 2 and Step 4 
annual reports with relevant program staff during staff meetings, an anonymous satisfaction survey is sent 

 
 
10 
to staff members using an anonymous Qualtrics link via email. Satisfaction survey instruments can be 
found in Appendix A. 
Additionally, staff interviews during annual site visits provide more information about lessons learned, best 
practices, how CQI findings are being applied to program implementation, and suggestions for improving 
the CQI multistep process. MER evaluation staff will also reflect at the end of the project on successes and 
challenges related to the implementation of the multistep CQI process to inform future CQI efforts. 
Table 6. Data for addressing the implementation research questions 
Implementation 
element 
Research question 
Data source 
Timing and frequency 
of data collection 
Party responsible for 
data collection 
Fidelity 
To what extent were 
the five steps of the 
CQI process carried 
out as intended 
each grant year? 
CQI process tracking log 
(See Appendix C) 
Updated after each step 
of a CQI cycle 
Evaluation lead and/or 
data manager 
Fidelity 
What were the 
unplanned 
adaptations to key 
CQI process 
components? 
CQI process tracking log 
(See Appendix C) 
Ad hoc 
Evaluation lead and/or 
data manager 
Engagement 
How understandable 
did CQI team 
members, and 
program staff 
broadly for Step 2 
and Step 4, find the 
reports for each step 
of the CQI process? 
CQI team satisfaction 
surveys; staff 
satisfaction surveys 
(See Appendix A) 
Annually after the 
completion of each step 
of the CQI process has 
been completed 
Evaluation lead 
Engagement 
How engaged were 
CQI team members 
and program staff in 
each step of the CQI 
process? 
CQI team satisfaction 
surveys; staff 
satisfaction surveys 
(See Appendix A) 
Annually after the 
completion of each step 
of the CQI process has 
been completed 
Evaluation lead and/or 
data manager 
Engagement 
How engaged were 
CQI team members 
in meetings? 
CQI team meeting 
attendance logs (See 
Appendix B) 
Bi-weekly after each 
CQI team meeting 
Evaluation lead 
Context 
• How well did the 
implementation of 
the multistep CQI 
process go for 
program and 
evaluation staff? 
 
Staff interviews; 
evaluator retrospective 
Annually during site 
visits 
Evaluation staff 
Context 
What external 
events affected the 
implementation of 
the multistep CQI 
process? 
Staff interviews; 
evaluator retrospective 
Annually during site 
visits 
Evaluation staff 

 
 
11 
Implementation 
element 
Research question 
Data source 
Timing and frequency 
of data collection 
Party responsible for 
data collection 
Context 
What are some best 
practices and 
lessons learned for 
improving the 
multistep CQI 
process? 
Staff interviews; 
evaluator retrospective 
Annually during site 
visits 
Evaluation staff