Detroit Parent Network 

Funded by the Corporation for National and Community Service 

Social Innovation Fund 

CFDA #94.019

Grant #11SIHMI001

Final Evaluation Report

August 31, 2015

5475 Morgan Rd.,

Ypsilanti, MI 48197 

www.evaluationstrategies.net
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

1

For questions regarding this report contact: 

Kimberly Browning, PhD 

kbrowning@comcast.net

(734) 476- 5039
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

2

Table of Contents

Table of Contents ......................................................................................................................... 2

Executive Summary..................................................................................................................... 3

Project Overview.......................................................................................................................... 5

Study Context ........................................................................................................................... 6

Evaluation Activities to Date (December 1, 2014 – April 30, 2015)....................................... 9

Summary of Activities ............................................................................................................. 9

SIF Implementation Reporting................................................................................................. 13

Implementation Study Context............................................................................................ 13

Implementation Dimensions ................................................................................................ 18

Implementation Data Collection and Measurement......................................................... 22

Implementation Analysis ...................................................................................................... 27

Implementation Findings...................................................................................................... 27

Outcomes/Impact Reporting................................................................................................... 29

Program Delivery................................................................................................................... 29

Counterfactual Condition ..................................................................................................... 33

Data Collection & Measurement.......................................................................................... 34

Analysis ................................................................................................................................... 36

Implementation Findings Outcomes................................................................................... 38

Attachment 1: Pathways to Literacy Tasks and Timelines ................................................. 50

Attachment 2: Pathways to Literacy Evaluation Measures ................................................ 53

Attachment 3: IRB Approved Updated Consents ................................................................ 56

Table of References .................................................................................................................... 82
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

3

Executive Summary 

The Detroit Parent Network Pathways to Literacy project sought to improve the literacy 

skills of children aged 1.5 – 5.5 years in low-income communities of Detroit, Michigan, 

including Central/North End and Osborn. Pathways to Literacy Model intended to 

impact the school readiness of urban, low-income children by increasing caregiver 

knowledge, skills, and efficacy in promoting a literacy rich environment and facilitating 

emergent literacy skills. There were three primary components to the Pathways to 

Literacy Model: 1) Home-based parent visitation services using a Pathways to Literacy

curriculum; 2) Detroit Parent Network Pathways to Literacy play groups, and 3) Detroit 

Parent Network’s Child Development workshops.

The Pathways to Literacy project was originally designed to reach a moderate level of 

evidence by Year 5; during the first four years of the project, evidence would have been

at a preliminary level. The project prematurely terminated in Year 3 due to reduction in 

time allowed to complete the project (in 2016 compared to 2017 planned originally, 

difficulty with finding a match, and staff changes. Seventy-four participants 

(caregivers, each with a participating child) were enrolled during the formative phase, 

and an additional 65 were enrolled in spring 2015 before project was discontinued. 

Originally, the project was designed to use a person-level randomized control trial to 

compare a group of 300 caregivers receiving the Pathways to Literacy Model to a 

delayed control group of an additional 300 caregivers. Key outcomes for caregivers 

were measured at baseline, 3, and 6 months from program entry and included: 1) 

increased knowledge of literacy; 2) improved engagement in dialogic reading; 3) 

increased frequency of reading to their child; 4) improved home literacy environments, 

and 5) increased parenting skills. As a result of these caregiver changes, children were 

hypothesized to increase language and literacy skills as measured at baseline and 6 

months from the program entry.

This report provides an overview of the project activities, participants, data collection, 

and some assessments collected for the duration of the project (Years 1-3):

 Usability Phase- Twelve caregivers were enrolled into the Pathways to Literacy 

program during the usability phase (pilot) to monitor implementation and work 

through issues with workflow, recruitment, instrumentation, and coordination of 

the Pathways to Literacy Model components. The pilot occurred between 

December 1, 2013 and April 30, 2014 (see a separate report describing pilot 

findings). 

 Formative phase (Cohort 1)- from December 1, 2014 to April 30, 2015 (i.e., 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

4

Quarters 1 and 2, Year 3). During the formative phase of the Pathways to 

Literacy project, data were collected from 74 caregivers and 74 children using the 

following instruments: Preschool Language Survey 5 (PLS-5), Adult-Child 

Interaction Reading Inventory (ACIRI), Child/Home Early Language and 

Literacy Observation (CHELLO), and a Caregiver Survey. Staff completed a total 

of 332 assessments (not counting enrollment paperwork). In addition, 28 sessions 

(17 home visits, 8 workshops, and 3 play groups) were observed by supervisors 

and members of the Quality Assurance team to assess fidelity of implementation 

of the Pathways to Literacy Model components. 

 Full Implementation (i.e., Cohort 2)- Full implementation of the Pathways to 

Literacy Model began in January 2015 and ended in August 2015. During the 

full implementation phase of the Pathways to Literacy Project (Quarters 2-4, Year 

3), baseline data were collected from 65 caregivers and 65 children using the 

instruments described above. At the end of August 2015, staff completed a total 

of 244 assessments (not including enrollment paperwork) for Cohort 2 baseline 

data.

Cohort 1 pre-post data showed promising preliminary findings. 

 Cohort 1 data shows all the PLS 5 standard score means to be in the expected 

direction with most approaching significant difference between the two groups 

(see Exhibits 1-3). The PLS 5 standard mean scores increased from baseline to 6-

month follow-up for both the PTL Curriculum and Control groups. However, the 

mean scores increased more markedly from baseline to 6-month follow-up for 

the PTL Curriculum group than for the Control group. More data are needed to 

document significance. 

 The PLS 5 Total Language mean standard scores were significantly higher in the 

PTL Curriculum group (M=102.00) than in the control group (M=94.57) by about 

half a standard deviation (see Exhibit 3). Percentile ranks were higher for both 

PLS 5 Auditory Comprehension subtest (56 in the PTL curriculum group 

compared to 42 in the control group) and PLS 5 Expressive Communication 

subtest (54 in the PTL curriculum group compared to 36 in the control group) in 

the PTL Curriculum group. 

 For the PTL Curriculum group, both Adult Behavior mean scores (see Exhibit 4) 

and Child Behavior mean scores (see Exhibit 5) on the ACIRI significantly 

increased from baseline to 3-month follow-up and decreased slightly from 3-

month follow-up to 6-month follow-up. The 6-month follow-up average score 

was significantly higher than the original average baseline score for the PTL 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

5

Curriculum group. For the Control group, Adult Behavior mean scores on the 

ACIRI did not change from baseline to 3-month follow-up and decreased from 3-

month follow-up to 6-month follow-up. 

 For four literacy areas assessed by CHELLO (book area, book use, writing 

materials, and toys) the PTL Curriculum group showed significant improvement 

in total scores (see Exhibits 6-10 for more details) as compared to the total scores 

in the control group. Both time and condition contributed to changes in scores 

for CHELLO. For technology – interaction between time and condition was not 

significant, but the time main effect was. When averaging over condition, 

Technology total scores on the CHELLO were significantly higher from baseline 

to later assessment scores. 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

6

Project Overview

Study Context 

In an effort to address the relationship between poverty, parenting literacy intervention, 

and children’s academic outcomes this Detroit Parent Network Pathways to Literacy 

project targeted 600 (300 in each of the control and treatment groups) low-income 

caregivers with children aged 1.5 – 5.5 years in the Central/North End, Osborn, and 

other communities of Detroit, Michigan. Pathways to Literacy Project’s goal was to

demonstrate the ability of a targeted parenting intervention to impact the school 

readiness of urban, low-income children, most specifically their literacy skills by 

increasing caregiver knowledge, skills, and efficacy in promoting a literacy rich 

environment and facilitating early childhood emergent literacy skills. Pathways to

Literacy grew out of previous research on dialogic reading aloud. It consisted of the 

three primary components: 1) Home-based parent visitation services; 2) Detroit Parent 

Network Pathways to Literacy play groups, and 3) Detroit Parent Network’s Child 

Development workshops. The fourth component was dropped from the model after the 

usability phase due to difficulty in getting books on time. It was determined that the 

change did not affect the Pathways to Literacy Model – or on the ability to track fidelity 

to this model component – because literacy skills and behaviors were demonstrated 

using books a caregiver had at home.

The Pathways to Literacy project was originally designed to reach a moderate level of 

evidence by Year 5; during the first four years of the project, evidence would have been

at a preliminary level. The project prematurely terminated in Year 3 due to reduction in 

time allowed to complete the project (in 2016 compared to 2017 planned originally, 

difficulty with finding a match, and staff changes). 

Originally, the project was designed to use a person-level randomized control trial to 

compare a group of 300 caregivers receiving the Pathways to Literacy Model to a 

delayed control group of an additional 300 caregivers. We hypothesized that the 

Pathways to Literacy Model (through the use of in-home literacy focused visits, play

groups, and community workshops for families) would result in better home literacy 

environments, improved interactions during parent-child reading, and increased scores 

on the core child development domains (Partridge, 2010). Specific measures were

gathered at both the child- and caregiver-levels. Child outcomes were measured using 

the Preschool Language Scale 5 (PLS-5) (Zimmerman, Steiner, & Pond, 2011). Caregiver 

literacy knowledge and the home environment were measured using a knowledge test 

and the Child Home Early Languages and Literacy Observation (CHELLO) (Neuman & 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

7

Koh, 2007). Engagement in reading was assessed using the Adult/Child Interactive 

Reading Inventory (ACIRI) (DeBruin-Parecki, 2006). Specific measures were gathered at 

both the child- and caregiver-levels. Child outcomes were measured using the 

Preschool Language Scale 5 (PLS-5) (Zimmerman, Steiner, & Pond, 2011). Caregiver 

literacy knowledge and the home environment were measured using a knowledge test 

and the child/Home Early Languages and Literacy Observation (CHELLO) (Neuman & 

Koh, 2007). Engagement in reading was assessed using the Adult/Child Interactive 

Reading Inventory (ACIRI) (DeBruin-Parecki, 2006). 

Key outcomes for caregivers were measured at baseline, 3, and 6 months from program 

entry and included: 1) increased knowledge of literacy; 2) improved engagement in 

dialogic reading; 3) increased frequency of reading to their child; 4) improved home 

literacy environments, and 5) increased parenting skills. As a result of these caregiver 

changes, children were hypothesized to increase language and literacy skills as 

measured at baseline and 6 months from the program entry.

Multivariate Analyses of Variance were used to model the outcomes over time 

including interactions between the time variable and treatment condition. Pre-post data 

were only available for Cohort 1 (i.e., formative phase). Seventy-four participants 

(caregivers, each with a participating child) were enrolled during the formative phase, 

and an additional 65 were enrolled in spring 2015 before project was discontinued. This 

project used a person-level randomized control trial to compare a group of about 30

caregivers receiving the Pathways to Literacy Model (for experimental group 

participants, n=29 for PLS 5, n=21 for ACIRI, and n=19 for CHELLO) to a group of 

about 15 caregivers not receiving the program (delayed control group) (for control 

group participants, n=16 for PLS 5, n=15 for ACIRI, and n=16 for CHELLO). All 

caregivers were randomly assigned to one of the two conditions after they had

expressed an interest in participating in the study. While planned that after the six 

month follow up assessments, the control group participants would receive the 

program in summer 2015, Cohort 1 control group participants did not receive an 

intervention due to the project being terminated.

This project had four distinct phases following the National Implementation Research 

Network framework: 

1. A development/planning phase during which Evaluation Strategies staff in 

collaboration with Detroit Parent Network staff finalized the Pathways to 

Literacy Model components and an evaluation plan (including a consent and 

instrumentation), developed a database, and data collection procedures; training 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

8

as well as IRB submission took place (Year 1); 

2. A usability implementation during which the team developed and tested specific 

hypotheses related to recruitment, implementation, randomization procedures, 

etc. (December 1, 2013 and April 30, 2014); 

3. A formative implementation during which an initial number of participants were 

randomized (n=74) and the Pathways to Literacy Model was implemented 

(Quarters 1 and 2, Year 3); and 

4. A full implementation during which the implementation started on a full sample; 

244 pre- assessments took place. Full implementation began in January 2015 and 

ended in August 2015 (Year 3). 

This report provides an overview of the project activities, participants, data collection, 

and some assessments collected for the duration of the project (Years 1-3): 

 Usability Phase- Twelve caregivers were enrolled into the Pathways to Literacy 

program during the usability phase (pilot) to monitor implementation and work 

through issues with workflow, recruitment, instrumentation, and coordination of 

the Pathways to Literacy Model components. The pilot occurred between 

December 1, 2013 and April 30, 2014 (see a separate report describing pilot 

findings). 

 Formative phase (Cohort 1)- from December 1, 2014 to April 30, 2015 (i.e., 

Quarters 1 and 2, Year 3). During the formative phase of the Pathways to 

Literacy project, data were collected from 74 caregivers and 74 children using the 

following instruments: Preschool Language Survey 5 (PLS-5), Adult-Child 

Interaction Reading Inventory (ACIRI), Child/Home Early Language and 

Literacy Observation (CHELLO), and a Caregiver Survey. Staff completed a total 

of 332 assessments (not counting enrollment paperwork). In addition, 28 sessions 

(17 home visits, 8 workshops, and 3 play groups) were observed by supervisors 

and members of the Quality Assurance team to assess fidelity of implementation 

of the Pathways to Literacy Model components. 

 Full Implementation (i.e., Cohort 2)- Full implementation of the Pathways to 

Literacy Model began in January 2015 and ended in August 2015. During the 

full implementation phase of the Pathways to Literacy Project (Quarters 2-4, Year 

3), 141 Cohort 2 participants were pre-enrolled as of summer 2015 with 65 of 

those participants signing consents and participating in baseline assessments. At 

the end of August 2015, staff completed a total of 244 assessments (not including 

enrollment paperwork) for Cohort 2 baseline data. 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

9

Evaluation Activities to Date (December 1, 2014 – August 30, 2015) 

Summary of Activities 

Evaluation Strategies team provided consultation and evaluation services to Detroit 

Parent Network – Pathways to Literacy staff. Evaluation Strategies’ staff (Dr. Browning 

and Dr. Malofeeva) stayed the same throughout the study. Activities during this 

reporting period included:

Meetings, Emails, and Phone Conversations: 

 Thirty in-person meetings between December 1, 2014 and August 30, 2015 took 

place to monitor implementation and work through issues with workflow, 

recruitment, coordination of the Pathways to Literacy Model components, 

development of the Pathways to Literacy /Detroit Parent Network database, and 

staff training on evaluation components; 

 Twenty-eight phone conversations between Evaluation Strategies and Detroit 

Parent Network staff took place during this period; 

 Six hundred sixty-two email exchanges occurred between Evaluation Strategies 

and Detroit Parent Network staff, and 

 Evaluation Strategies staff participated in monthly United Way for Southeastern 

Michigan (UWSEM) meetings for SIF evaluators.

Trainings: 

 Evaluation Strategies staff routinely provided hands on database demonstrations

for Detroit Parent Network staff. 

 Evaluation Strategies provided evaluation training to DPN on data collection 

instruments (i.e., CHELLO, ACIRI). 

 Evaluation Strategies and DPN discussed additional trainings and requirements 

for each training. 

Database and Data: 

 Worked with DPN to identify procedures that allowed for better data 

management, tracking, and report distribution.

 Updates were made to the database as needed and included additional features 

requested by DPN, such as being able to search by id, being able to see certain 

fields, reports being produced by additional variables.

 Evaluation Strategies provided assistance with assessment data and incentive 

tracking. 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

10

 Evaluation Strategies worked with DPN to correct errors in data entry and/or 

missing info.

 Evaluation Strategies/DPN prepared data to submit to UWSEM.

Reports:

 Evaluation Strategies developed additional Quality Assurance Reports (as a part 

of the database) that aided in Quality Assurance processes.

 Evaluation Strategies developed, solicited feedback, and submitted a semi-annual 

report to UWSEM (dated May 2015).

 Evaluation Strategies reviewed reports DPN submitted to UWSEM as a part of its 

reporting requirements.

 Evaluation Strategies team developed an impact report (June 2015).

 Evaluation Strategies team developed a final evaluation report.

Tasks:

 Evaluation Strategies evaluation team distributed the weekly Quality Assurance 

Reports.

 Evaluation Strategies evaluation team discussed the implementation of the 

requirement for workshops and play groups together with DPN and United Way 

of Southeastern Michigan.

 Evaluation Strategies evaluation team and DPN discussed timelines and 3 month 

assessments.

 Evaluation Strategies assisted with electronic links for Caregiver interviews.

 Evaluation Strategies worked with DPN to monitor caseloads and timelines.

 Evaluation Strategies and DPN discussed timelines and 6 month assessments. 

 Evaluation Strategies worked with United Way and Ty Partridge to update power 

analyses regarding sample size. The target sample size was reduced from 800 to 

600 participants. 

 Evaluation Strategies provided information to DPN on subcontractors to support 

DPN with staffing issues.

 Evaluation Strategies team provided extensive feedback to implement with 

fidelity. 

 The Data tracking manual was updated to document procedures and protocols.

 Staff discussed procedures to use during recruitment to maximize response.

 Evaluation Strategies team discussed project termination and steps to take to bring 

it to a closure.
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

11

 Evaluation Strategies team developed a detailed caseload 2 year plan per UWSEM 

request (May 2015).

 Evaluation Strategies team cleaned and analyzed Cohort 1 and 2 data.

Program problems/challenges:

 There were ongoing discussions with United Way of Southeastern Michigan about 

implementation of workshops and play groups. It was decided that workshops 

and play groups would continue to be offered with the same dosage. 

 Timeline issues. Since February 2015, the project was behind schedule. In April 

2015 USWEM notified DPN that SIF funding would terminate a year earlier, in 

2016. A new timeline to complete tasks was requested and produced. It became

clear that a reduction in time would not allow the project to serve all participants 

to reach a moderate level of evidence. UWSEM in collaboration with DPN made a 

decision in summer 2015 to terminate this project.

 Staffing issues. One full-time staff member left the project in spring 2015. There 

were not enough staff to implement and administer assessments esp. with reduced 

timeline (a year shorter than originally planned). To address this issue, DPN hired 

student workers and reassigned existing staff to provide additional support. In 

addition, DPN trained additional volunteers and staff. With a reduced timeline 

originally budgeted staffing loads plus volunteers DPN was able to obtain were 

not enough to be able to complete the full scale of work needed to bring this 

project to completion.

 Volunteer involvement. DPN is known in the community for its strong work with 

volunteers. Using volunteers as the staffing strategy for assessments was not 

possible for a randomized control trial with rigorous evaluation requirements. 

 There were challenges with recruitment for Cohort 2. Not enough participants 

were enrolled in Cohort 2 (141 participants were pre-enrolled as of summer 2015 

with 65 of those participants signing consents and participating in baseline 

assessments). During pre-enrollment participants were asked for initial 

commitment prior to full enrollment. While initially giving approval, a number of 

participants (n=90) decided not to continue with the project. This was prior to 

signing a consent form and prior to any assessments and randomization.

 Additionally, a new power analysis was conducted to determine if a smaller 

sample size would yield an appropriate power. It was determined that a sample 

size of 600 participants, as compared to the original target sample of 800

participants, would still yield an appropriate power. DPN received approval to 

service 600 participants in April 2015.
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

12

 Finding Match dollars was difficult.

 Data collection. Outsourcing data collection to volunteers (e.g., PLS 5) was outside 

the scope of skills those volunteers possessed. Evaluation Strategies had multiple 

discussions with DPN about the skill levels required to complete most 

assessments. Evaluation Strategies had a number of trained data collectors on staff 

to conduct these assessments, but DPN chose not to go that route.
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

13

SIF Implementation Reporting 

Implementation Study Context 

1.1 Program delivery 

timeline

An original proposed timeline is included in Attachment 1.

This project began full implementation in June 2014. The IRB 

application was re-approved on May 16, 2014 and an annual 

approval was received in November 2014. Pilot phase began 

January 2014 and was completed March 31, 2014. Cohort 2 

data collection started in January 2015. Project was 

terminated in August 2015.

1.2 Program 

beneficiaries

Detroit Parent Network Pathways to Literacy project 

originally targeted 800 (400 in the experimental group and 

400 in the control group) participants. After obtaining 

missing data rate in Cohort 1, it was determined that a 

sample size of 600 participants, as compared to the original 

target sample of 800 participants, would still yield an 

appropriate power. DPN received approval to service 600 

participants in April 2015. 

Detroit Parent Network Pathways to Literacy project targeted 

74 (36 in the experimental group and 38 in the control group)

Cohort 1 low-income caregivers with children aged 1.5 – 5.5 

years in the Central/Northend, Osborn, and other 

communities of Detroit, Michigan. For Cohort 2, 141 

participants were pre-enrolled as of summer 2015 with 65 of 

those participants signing consents and participating in 

baseline assessments before the project was terminated.

1.3 Program 

components/

activities 

Three key components of the Pathways to Literacy Model 

are: 

1. Home-based parent visitation services targeting caregiver 

literacy knowledge and practices (10 home visits);

2. Detroit Parent Network Pathways to Literacy play groups

(2 play groups), and

3. Detroit Parent Network’s Child Development workshops

(3 workshops).
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

The fourth Pathways to Literacy Component (monthly 

Imagination Library books) was dropped after the usability 

phase due to difficulty in getting books on time. This change 

did not have any impact on the Pathways to Literacy Model –

or on the ability to track fidelity to this model component –

since books a caregiver has at home are being used instead by 

the home visitor to demonstrate literacy skills and behaviors.

1.4 Program outputs Service Delivery

1. Characteristics of the caregivers enrolled in the Pathways 

to Literacy Program (family composition, ages of 

caregivers, ethnicity, primary language spoken in the 

home, caregiver educational/literacy level, caregiver 

employment status, income)

2. Characteristics of the children enrolled in the Pathways to 

Literacy Program (ages of children, ethnicity, gender)

3. Caregiver Satisfaction with the Pathways to Literacy

Program

4. Number of participants who disenroll and reasons for 

disenrollment.

Programmatic 

1. Characteristics of program staff (demographics, degrees, 

specialized training)

2. Types of training staff received 

3. Barriers to implementation

4. Pathways to Literacy staff caseloads

5. Number of service worker supervision hours by program 

administration

6. Number of continuing education hours of family service 

worker and program administrators

Assessments

1. Percentage of clients with complete baseline data

2. The nature and types of services both groups receive 

outside of the Pathways to Literacy intervention

Home Visits

1. Quality of the relationship between the home visitor and 

caregiver as reported by home visitor and caregiver

2. Number/frequency of home visits received

14
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

15

3. Length of home visits

4. Content of home visits

5. Location of the home visit

6. Observation-based quality of the home visits is being

tracked by collecting data on the following home visitor 

characteristics/behaviors: % of time spent on various 

home visitation activities, preparedness, connection to the 

previous visit, discussing the activity from the last visit, 

discusing the child’s experience with the activiy from the 

last visit, facilitated the literacy activity as a parent-child 

experience, modeling reading a book, reviewing 

extension activities, reviewing activities for other age 

groups, encouraging caregiver to participate in dialogic 

book reading, reviewing the objective of the visit, 

facilitate parent’s reflection of the child’s experience, 

review appropriate literacy skills and activities, revisit 

next steps, incorporate family’s language and culture, 

encourage engaging in book reading beyond the visit, 

provide a book and other supportive materials. 

Pathways to Literacy Play Groups

1. The level of caregiver and child engagement

2. The number/frequency of groups attended 

3. Length of Pathways to Literacy play groups

4. Content of attended groups 

5. Observation-based quality of the groups is being tracked 

by collecting data on the following facilitator 

characteristics: engagement in the opening activity, 

describing the timeline of the play group, planning

acivities that support the focus on the play group, 

explanation of the focus of the play group, interact with 

child and caregiver to support the focus of the play

group, and appear relaxed and confident when 

facilitating.

Pathways to Literacy Child Development Workshops

1. The level of participant engagement 

2. The number/frequency of workshops

3. Length of workshops

4. Content of workshops
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

16

5. Observation-based quality of the workshops is being 

tracked by collecting data on the following facilitator 

characteristics: engagement in the opening activity, 

explanation of the goal, main points, and application of 

the workshop, opportunity to practice the message of the 

workshop, connect the workshop to participant’s 

children, have a participant make a plan for 

implementation, appear relaxed and confident when 

implementing.

1.5 Program 

outcomes/impacts

Key outcomes for caregivers and children were measured at 

baseline, 3, and 6 months from the program entry and 

included: 

1. Increased knowledge of literacy (Caregiver Survey) ; 

2. Improved engagement in dialogic reading (ACIRI);

3. Increased frequency of reading to their child (Caregiver 

Survey);

4. Improved home literacy environments (CHELLO); and

5. Child improved auditory comprehension and expressive 

communication skills (PLS 5- at baseline and at 6 months). 

1.6 Impact study 

design (including 

comparison group 

details, if 

appropriate)

This evaluation followed the same 74 Cohort 1 caregivers 

from entry into the Pathways to Literacy Program through 6 

months from the program entry (with delayed treatment 

control group getting intervention after 6 months without 

any further assessments). It utilized an experimental design 

with randomization at the caregiver level as a means to 

determine the efficacy of the Pathways to Literacy Model. For 

Cohort 1, it compared a group of 36 caregivers receiving the 

Pathways to Literacy Model to a group of 38 caregivers not 

receiving the program (delayed-treatment control group). Of 

the 74 Cohort 1 participants, full pre-post data were available 

for 35 to 46 participants depending on the assessment. This 

project used a person-level randomized control trial to 

compare a group of about 30 caregivers receiving the 

Pathways to Literacy Model (for experimental group 

participants, n=29 for PLS 5, n=21 for ACIRI, and n=19 for 

CHELLO) to a group of about 15 caregivers not receiving the 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

17

program (delayed control group) (for control group 

participants, n=16 for PLS 5, n=15 for ACIRI, and n=16 for 

CHELLO). All caregivers were randomly assigned to one of 

the two conditions after they had expressed an interest in 

participating in the study. While planned that after the six 

month follow up assessments, the control group participants 

would receive the program in summer 2015, Cohort 1 control 

group participants did not receive an intervention due to the 

project being terminated. The study tested the claim that the 

Model (Pathways to Literacy) was the reason for the 

difference between the two groups. All children ages 1.5-5.5 

who were at least 6 months from kindergarten entry were

eligible to participate in the study. The order of activities 

followed the following sequence: eligibility screening, preenrollment, consent, baseline data collection, and random 

assignment. Reasons for sample loss were documented. 

A. Research Questions

1. Impact Research Questions

a. Confirmatory Research Questions

1. At 3 months from the program entry, are

the caregivers who are randomly assigned

to the experimental group significantly

superior to caregivers randomly assigned to

the delayed control group on the following

primary caregiver-level outcomes of

interest:

a. Parental knowledge of child development

b. Frequency of reading to their child

c. Engagement in reading

d. Home literacy environment.

2. For caregivers in the experimental group, are

Pathways to Literacy treatment gains

maintained at a follow-up time period (6

months from the program entry).

3. At 6 months from the program entry, are

the children whose caregivers are in the

Pathways to Literacy experimental group
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

18

significantly superior to those in the 

delayed-treatment control group on the

following primary child-level outcomes of

interest:

a. Language, and literacy skills.

2. Exploratory Research Questions

1. Will outcomes at post treatment be

significantly and positively correlated with

positive alliance between home visitor

and participant and dosage?

2. Will participant engagement be positively

correlated with positive alliance between

home visitor and participant and

treatment gains?

Implementation Dimensions

2.1 Fidelity to program Implementation evaluation was to ensure that the Pathways 

implementation to Literacy Model was implemented as designed. It was 

developed to provide vital information about fidelity. 

1. Implementation Research Questions

One of the keys to a strong program evaluation was

being able to link program outcomes to program 

activities as directly as possible.

a. Service Delivery Implementation Questions

1. What are the characteristics of caregivers who 

enroll in Pathways to Literacy services?

2. What are the characteristics of children who enroll 

in Pathways to Literacy services?

3. How satisfied are participants with the Pathways 

to Literacy program?

4. Why and how many participants dis-enroll from 

the program?

b. Programmatic Implementation Questions

1. What are the characteristics/demographics of the 

staff of the Pathways to Literacy program?

2. What types of training have the Pathways to 

Literacy staff received?
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

19

c.

3. What types of barriers to implementation do staff 

encounter while implementing the Pathways to 

Literacy Model?

4. What are Pathways to Literacy’ staff caseloads?

5. How many supervision hours do administrators 

provide?

6. What is the number of continuing education hours 

that family service workers and program 

administrators engage in?

Assessment Implementation Questions?

1. What percentage of caregivers have complete 

baseline data information?

d.

2. What types and amounts of services do both 

control and treatment groups receive outside of 

their participation in the Pathways to Literacy

program?

Home Visit Implementation Questions

1. What is the quality of the relationship between the 

home visitor and the caregiver?

2. How many and how often do participants receive

home visits?

3. What is the length of the home visits?

4. What is the content of the home visits?

e.

f.

5. Do coaches implement all the activities required 

by the Pathways to Literacy Curriculum?

Pathways to Literacy Implementation Questions

1. What is the quality of the relationship between the 

Pathways to Literacy staff and caregivers?

2. What is the number/frequency of groups 

attended by caregivers?

3. What is the length of Pathways to Literacy play 

groups?

4. What is the content of attended play groups?

Pathways to Literacy Workshop Implementation 

Questions

1. What is the number and frequency of workshops?

2. What is the length of workshops?
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

20

3. What is the content of workshops?

4. What is the quality of the workshops?

The second phase (formative evaluation) expanded

implementation of the Pathways to Literacy Model to 

include additional 65 caregivers and the evaluation included

randomization and outcome evaluation. Quality Assurance

processes, tested and refined during the usability phase, 

were administered on a weekly (not a monthly basis as 

originally proposed due to staff needing more support) 

basis during a weekly Quality Assurance call and through 

feedback mechanisms (supervision, Quality Assurance

Reports). Weekly Implementation Reports developed by the 

members of the Quality Assurance Team provided timely 

data-driven feedback to all Pathways to Literacy staff 

implementing the Pathways to Literacy Model. Areas of 

deviation were identified and recommendations for 

program adjustment were made. 

2.2 Program exposure 

(or dosage)

1. Ten Pathways to Literacy home visits following the 

Pathways to Literacy Model lasting approximately one 

hour each.

2.

3.

Two (at a minimum) Pathways to Literacy play groups.

Three child development workshops.

2.3 Quality of program 

delivery

Quality of program delivery was measured using:

1. Workshop Fidelity Observation Form

2. Play group Fidelity Observation Form, and 

3. Home Visit Fidelity Observation Form.

The quality of the relationship between the home visitor and 

the participant was measured in the following way:

1. Working Alliance Inventory completed by the caregiver, 

and

2. Working Alliance Inventory completed by the home 

visitor.

2.4 Program 

participant 

responsiveness

Program participant responsiveness was measured using 

the following fidelity measures:

1. Workshop Fidelity Observation Form had one item on 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

21

2.

participant engagement during the workshop measured 

on a scale from 1 (engaged less than 10% of the time) to 5

(engaged 75 to 100% of the time).

Play group Fidelity Observation Form had 2 items on 

engagement: parent engagement (measured on a scale 

from 1 (engaged less than 10% of the time) to 5 (engaged 

75 to 100% of the time)) and child engagement 

(measured on a scale from 1 (engaged less than 10% of 

the time) to 5 (engaged 75 to 100% of the time)).

2.5 Program 

differentiation

DPN staff and evaluation staff had multiple discussions 

about the Core Components that guided the development of 

measures, data collection instruments, workflow, analysis, 

and feedback regarding implementation fidelity. These core 

components represented the most essential and 

indispensable components of Pathways to Literacy Model. 

The Pathways to Literacy core components were derived 

from the following sources:

 SEP documents that were developed and reviewed by 

Pathways to Literacy administrators and Evaluation 

Strategies staff;

 The implementation science literature1 provided 

guidance on the theoretical model, and

 Evaluation Strategies staff identified the core 

components based on discussions with Detroit Parent 

Network and the Quality Assurance committee during 

Year 1 and Year 2 meetings and the usability phase of the 

study.

During the full implementation phase, staff implemented 

the 3 required components included in the Pathways to 

Literacy intervention. 

 

1Fixsen, D.L.,Naoom, S.F., Blasé, K.A., Friedman, R.M., & Wallace, F. (2005). Implementation research: A Synthesis of the 

literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation 

Research Network (FMHI Publication No 231). 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

22

2.6 Participant DPN collected participant satisfaction data on a continuous 

satisfaction basis every December at their Annual Meeting. Some 

caregivers submitted their satisfaction ratings indicating 

they had received Pathways to Literacy services in the past. 

Since only a very small number of SIF participants took part 

in the satisfaction survey, DPN decided not to pull this data 

separately for SIF analyses.

Implementation Data Collection and Measurement

3.1 Amount of data 

collected (e.g., 

observations, 

surveys, records). 

Sample for data 

collection (e.g., size, 

demographic 

composition, 

representativeness 

of sample to all 

personnel/ 

participants)

Number of Subjects: Located in SW Detroit, the DPN

Pathways to Literacy program served 74 children and their 

caregivers through this study in Cohort 1. In addition, 65

caregivers were enrolled and pre- assessed in Cohort 2 (244 

assessments were completed in January- August 2015).

Those families with children older than one year and a half 

and younger than five and a half or who were at least 9 

months from kindergarten entry and resided in Detroit 

qualified for services if they were considered low-income. 

As of August 2015 a total of 233 participants initially agreed

to participate in the study; 92 participants in Cohort 1 and 

141 in Cohort 2. 74 of the 92 participants in Cohort 1 signed 

consents and completed full baseline assessments 

(enrollment paperwork, ACIRI, Caregiver Survey, CHELLO, 

and PLS 5) and 65 of the 141 participants in Cohort 2 signed 

consents and started baseline assessments. Fifty-one 

participants in Cohort 2 had baseline assessments competed 

by August 2015. 

Only after all of the assessments were completed, 

participants in Cohort 1 (N=74) were randomized to one of 

the two conditions. In Cohort 1, 36 participants were 

randomly assigned to Group A (experimental group), and 

38 were randomly assigned to Group B (control group). All 

participants who were randomized in Cohort 1 were 

followed up for 3 and 6 month assessments. For Cohort 1, 

those with incomplete baseline data (n=18) did not 

participate in the study due to such reasons as no longer 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

23

interested, moved, no longer had time, no further response, 

or the phone number was not valid. Of the Cohort 2 

participants who signed consents, 14 had incomplete 

baseline data. Seventy-seven participants in Cohort 2 were 

pre-enrolled (i.e., indicated interest) but did not yet sign 

consents in August 2015. 

Cohort 1

Baseline data were collected from 74 caregivers and 74 

children using the following instruments: Preschool 

Language Survey 5 (PLS-5), Adult-Child Interaction 

Reading Inventory (ACIRI), Child/Home Early Language 

and Literacy Observation (CHELLO), and a Caregiver 

Survey. 

The following baseline Cohort 1 assessments were 

completed as of August 2015:

 ACIRI-74 (n=0, 0% missing)

 Caregiver Survey -74 (n=0, 0% missing)

 PLS-5 – 74 (n=0, 0% missing)

 Chello -74 (n=0, 0% missing)

The following 3 month Cohort 1 assessments were 

completed as of August 2015:

 ACIRI-59 (n=15, 20% missing)

 Caregiver Survey -61 (n=13, 18% missing)

 Chello -60 (n=14, 19% missing)

As of August 2015, the following 6 month Cohort 1 

assessments were completed:

 ACIRI-57 with n=17, 23% missing

 Caregiver Survey -57 with n=17, 23% missing

 Chello -59 with n=15, 20% missing

 PLS-5 – 58 with n=16, 22% missing.

Cohort 2

Baseline data were collected from 65 caregivers and 65 

children using the following instruments: Preschool 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

24

Language Survey 5 (PLS-5), Adult-Child Interaction 

Reading Inventory (ACIRI), Child/Home Early Language 

and Literacy Observation (CHELLO), and a Caregiver 

Survey. 

The following baseline assessments were completed with 

Cohort 2 participants as of August 2015:

 ACIRI-62 with n=3, 5% missing

 Caregiver Survey -64 with n=1, 2% missing

 Chello -64 with n=1,2% missing

 PLS 5-52 with n=13, 20% missing.

3.2 Sample for data 

collection

Below we present on analyses and findings from Cohort 1 

participants’ data. Since Cohort 2 baseline data collection 

was not complete and due to none of the Cohort 2 

participants receiving PTL curriculum, these data are not 

reported here. All analyses below, including pre-posttest 

comparisons, include Cohort 1 data only.

Gender of Subjects: Three of the 74 caregivers in Cohort 1

were male (32 females in Group A and 38 females in Group 

B, 3 males in Group A and 0 males in Group B). A Fisher’s 

exact test could not be calculated due to small expected 

values in some cells. All staff, including supervisors, were 

female.

Age of Subjects: All of the recruited caregivers involved 

children who were older than one year and a half and less 

than five and a half at the beginning of the study. Cohort 1 

caregivers’ average age was 31.5 years for Group A and 34.4 

for Group B (the difference in age (continuous variable) 

between the two groups was not significant, t(68)=-1.43, 

p>.05).

Racial and Ethnic Origin: In Cohort 1, 47 (63.5%) of the 74 

caregivers were African-American. Twenty-one caregivers

(28%) were Hispanic. All Fisher’s exact tests were nonsignificant (p>.05).

Inclusion Criteria: Eligibility was restricted to caregivers of 

children aged one year and a half to under the age under 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

25

five and a half and those who live in Detroit. Caregivers 

who passed the screening (child age requirement and 

residency in Detroit) and who themselves consented to 

participation and who gave their child consent to participate 

were invited to participate in the study. Caregivers were 

randomly assigned to the treatment (Group A) or the 

control (Group B) group. If two children with the same 

caregiver wanted to join the study, only one was assessed. 

The younger child was chosen. 

Exclusion Criteria: Caregivers who did not satisfy the four 

screening criteria were excluded from participation in the 

study. Caregivers who declined to offer consent for their 

participation or their child’s participation were not included 

in the study. Additionally, pregnant women were not 

eligible to participate. 

3.3 Description of data Assessments were being conducted through surveys 

collection methods (Caregiver Survey, observation (CHELLO, ACIRI)), and 

(e.g., surveys, direct child assessment methods (PLS 5) by the coaches. It 

observations, was estimated each child assessment required about an hour 

interviews, focus to an hour an a half per assessment. Additional measures 

groups, coding of required another home visit. Most baseline assessments 

existing data) were done in two sessions. The mode of data collection was 

the same in two groups. 

3.4 Description of data A complete list of measures and method used for collection 

collection is located in Attachment 2. Identical data were collected 

procedures (e.g., from both the treatment and control groups. 

who collected the Exhibit 1 presents the workflow of the Pathways to Literacy 

data and how) Project. 

Check Study Baseline Eligibility Criteria, Study Consent Randomization Assessment Pre-enrollment

6 Month Follow3 Month Follow- Up PTL for the PTL Intervention Up Child Outcomes Control Group

Collected
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

26

Upon completion of eligibility testing and the obtaining of 

study consent, baseline data were collected by coaches from 

all study participants. Then participants were randomly 

assigned to either the control or treatment group. 

Caregivers in both treatment and control groups were 

assessed again at 3-, and 6 months. If there was more than 

one child under 54 months of age, one child was randomly 

chosen to be assessed for the duration of the intervention. 

Child assessments occured at baseline and six months. 

Pathways to Literacy staff were responsible for all outcomes 

and some implementation data collection as well as for data 

entry (an internal database is being utilized). Evaluation 

Strategies’ staff participated in some data collection, data 

tracking, training staff on data collection, data entry, quality 

assurance, and coordination of data collection and data 

processing. For both groups, all assessments were 

conducted by coaches. In addition, for Cohort 1 supervisors 

and evalaution staff observed 28 sessions (3%) to establish 

fidelity of implementation. These sessions were chosen 

randomly.

3.5 Measures used for Collected data included the following:

each dimension, Child: Basic child demographic information was collected. including target Child outcomes were measured by the PLS-5. 

levels if 

appropriate Caregiver: Caregiver data collected included educational 

attainment, marital, economic status (see Attachment 2). 

ACIRI was used to assess caregiver-child reading. Caregiver 

literacy knowledge and the home environment were 

measured using a knowledge test which was developed 

based upon the Pathways to Literacy curriculum and the 

Child/Home Early Language and Literacy Observation 

(CHELLO). The evaluator developed the knowledge 

assessment tool (based on the Pathways to Literacy

curriculum) and 3 fidelity measures.

Using these measures and methods, data were collected 

through either self-report or observation. 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

27

Implementation Analysis

4.1 Type of analysis Descriptive statistics

4.2 Analysis 

procedure/steps

For data analyses, Evaluation Strategies is conducting the 

following steps:

a. Clean the data (check for missing data and accuracies);

b. Determine the data distributions of the major variables 

for the analysis (i.e., frequency distributions, histograms, 

central tendencies, skewedness, etc.);

c. Adjust the analytic plan so the analysis is appropriate to 

the types of data;

d. Create syntax for re-coding of variables if needed, for 

example to aggregate data, or re-code variables to 

address uneven distributions, etc.;

e. If scales are used, calculate scale scores and determine 

scale reliability, and conduct item analysis to assess 

empirical validity;

f. Conduct major analyses based on type of data, for 

example, correlational or inferential statistics;

g. Write and present reports on findings and 

recommendations; and

h. Consult on continuous improvement, design changes, 

and corrective action.

Implementation Findings

5.1 Implementation 

findings

The following services were provided by members of the 

Detroit Parent Network staff:

Usability (pilot) 0 Total number of workshops offered 

Total number of caregivers attending workshops 0

Total number of play groups 1

Total number of caregivers attending play groups 1

Total number of home visits 31
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

28

Total number of caregivers receiving home visits 13

Cohort 1 88

Total number of workshops offered 

Total number of caregivers attending workshops 26 (Group A)

Total number of play groups offered 69

Total number of caregivers attending play groups 25 (Group A)

Total number of home visits (including assessment 651 visits) 

Total number of caregivers receiving home visits 74

Cohort 2

Total number of workshops offered from 0

December 1, 2014 to August 30, 2015

Total number of caregivers attending workshops 0 (Group A)

Total number of play groups offered from 0 December 1, 2014 to April 30, 2015

Total number of caregivers attending play groups 0 (Group A)

Total number of home visits (including assessment 152 visits) from December 1, 2014 to August 30, 2015

Total number of caregivers receiving home visits 65

We want to note that Cohort 2 curriculum home visits, play 

groups, and workshops were not started by August 30, 

2015.

In addition, the following services were provided during 

this study:

 Phone calls -1290

 Text messages-17

 Other (e.g., letters)-65
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

29

Below are the number and type of services provided by 

cohort.

 Usability

 Phone calls -76

 Text messages-0

 Other (e.g., letters)-34

Cohort 1

 Phone calls -727

 Text messages-16

 Other (e.g., letters)-34

Cohort 2

 Phone calls -487

 Text messages-1

 Other (e.g., letters)-9

In addition, 28 sessions (17 home visits, 8 workshops, and 3 

play groups) were observed by supervisors and members of 

the Quality Assurance team to assess fidelity of 

implementation of the Pathways to Literacy Model 

components. 

5.2 Lessons learned Lessons learned from Cohort 1 included:

1. Documenting procedures and protocols

for consistency.

2. Transportation continued to be a barrier 

Detroit.

was important 

to services in 

Outcomes/Impact Reporting 

Program Delivery 

1.1 Number of Three key components of the Pathways to Literacy Model 

program units (e.g., were: 

sessions, events, 1. Home-based parent visitation services targeting classes)/outputs caregiver literacy knowledge and practices (10 home 

visits);

2. Detroit Parent Network Pathways to Literacy play 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

30

groups (2 play groups), and

3. Detroit Parent Network’s Child Development 

workshops (3 workshops). 

Program delivery began in June 2014 (recruitment and 

baseline assessments). Baseline assessments were completed 

in August 2014. Model implementation began in August

2014. Pathways to Literacy Model implementation was 

completed in November 2014. Full implementation began in 

January 2015 (Quarter 2, Year 3). Data were analyzed to 

track fidelity of implementation. Outcome data for Cohort 1 

were analyzed using MANOVA.

1.2 Number of See response to 5.1 above

program units (e.g., 

sessions, events, 

classes)/outputs 

The following services were provided by members of the 

Detroit Parent Network staff to Group B:

Total number of workshops offered 0

Total amount of caregivers attending workshops 0

Total number of play groups offered 0

Total number of caregivers attending play groups 0

Total number of home visits (not including 

0 assessment visits)

Total number of caregivers receiving home visits 0

Therefore, as planned Group B did not receive the 

PTL curriculum. We want to note that Cohort 2 

home visits, play groups and workshops were not 

started by August 30, 2015.

1.3 Quality DPN collected participant satisfaction data on a continuous 

of/satisfaction with basis every December at their Annual Meeting. Some 

program delivery caregivers submitted their satisfaction ratings indicating 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

31

they had received Pathways to Literacy services in the past. 

Since the number of SIF participants who participated in the 

satisfaction survey was small, their responses were not 

analyzed separately.

1.4 Date program 

delivery 

began/ended

For Cohort 1, program delivery began in June 2014 (Quarter 

3, Year 2). Baseline assessments were completed in August 

2014. Model implementation began in August 2014. 

Pathways to Literacy Model implementation was completed 

in November 2014. Full implementation began in January 

2015. For Cohort 2, program delivery did not begin.

1.5 Number of 

program 

participants/partici

pation rate

As of August 2015 a total of 233 participants initially agreed 

to participate in the study; 92 participants in Cohort 1 and 

141 in Cohort 2. 74 of the 92 participants in Cohort 1 signed 

consents and completed full baseline assessments 

(enrollment paperwork, ACIRI, Caregiver Survey, CHELLO, 

and PLS 5) and 65 of the 141 participants in Cohort 2 signed 

consents and started baseline assessments. Fifty-one 

participants in Cohort 2 had baseline assessments competed 

by August 2015. 

Only after all of the assessments were completed, 

participants in Cohort 1 (N=74) were randomized to one of 

the two conditions. In Cohort 1, 36 participants were 

randomly assigned to Group A (experimental group), and 

38 were randomly assigned to Group B (control group). All 

participants who were randomized in Cohort 1 were 

followed up for 3 and 6 month assessments. For Cohort 1, 

those with incomplete baseline data (n=18) did not 

participate in the study due to such reasons as no longer 

interested, moved, no longer had time, no further response, 

or the phone number was not valid. Of the Cohort 2 

participants who signed consents, 14 had incomplete 

baseline data. Seventy-seven participants in Cohort 2 were 

pre-enrolled (i.e., indicated interest) but did not yet sign 

consents in August 2015. 

Cohort 1
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

32

During the formative phase of the Pathways to Literacy

Project, baseline data were collected from 74 caregivers and 

74 children using the following instruments: Preschool 

Language Survey 5 (PLS-5), Adult-Child Interaction 

Reading Inventory (ACIRI), Child/Home Early Language 

and Literacy Observation (CHELLO), and a Caregiver 

Survey. Staff completed a total of 759 assessments. In 

addition, 28 sessions (17 home visits, 8 workshops, and 3 

play groups) were observed by supervisors and members of 

the Quality Assurance Team to assess fidelity of 

implementation of the Pathways to Literacy Model 

components.

Cohort 2

During the full implementation phase of the Pathways to 

Literacy Project (Quarter 2, Year 3), Cohort 2 baseline data 

were collected from 65 caregivers and 65 children using the 

instruments described above. Staff completed a total of 244

assessments (not including enrollment paperwork).

The following measures were implemented to address

differential attrition in the delayed intervention group:

1. At recruitment participants presented the project as a 9-

month project.

2. We trained Detroit Parent Network staff to present the

two conditions in a manner that was appealing to both

groups no matter the assignment. A script is presented 

below:

“ The study participation will involve three assessment

visits and participation in groups and home visits for

three months. Some caregivers will start groups and

home visits sooner and others are able to start in six

months since we cannot service everyone at once. You

will not be able to choose who starts sooner or later.

Everyone will get the assessment visits at the same time 

points.”

3. We excluded a number of assessment points from the

previous version of the design from both groups in 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

33

order to not overwhelm the participants.

4. We actively collected information on additional

contacts, offer $10 incentives if a participant reported

moving to a new address, and monitored the quality of

the home visitor- caregiver working relationship.

1.6 Demographic Preliminary info is available as a response to item 1.6 above.

characteristics of 

participants

Counterfactual Condition

2.1 Final Randomization was carried out through the database Detroit 

randomization Parent Network used for the study (using the random

process number generator). Evaluation Strategies’ staff worked with a

computer programmer to develop a Random Assignment

button generator that assigned enrollees to one of the two

groups using the random number generator. The button was

available after the baseline data collection was complete. The

Random Assignment button was deactivated once hit and

after randomly assigning an enrollee. Staff did not have an 

ability to manipulate the Random Assignment button.

In a person-level randomized control trial one caregiver per

family was randomly assigned to one of the two conditions

with the primary goal of detecting treatment effects after

baseline data collection took place.

2.2 Baseline Cohort 1 

equivalence Gender of Subjects: Three of the 74 caregivers in Cohort 1 analysis were male (32 females in Group A and 38 females in Group 

B, 3 males in Group A and 0 males in Group B). A Fisher’s 

exact test could not be calculated due to small expected 

values in some cells. All staff, including supervisors, were 

female.

Age of Subjects: All of the recruited caregivers involved 

children who were older than one year and a half and less 

than five and a half at the beginning of the study. Cohort 1 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

34

caregivers’ average age was 31.5 years for Group A and 34.4 

for Group B (the difference in age (continuous variable) 

between the two groups was not significant, t(68)=-1.43, 

p>.05). 

Racial and Ethnic Origin: In Cohort 1, 47 (63.5%) of the 74 

caregivers were African-American. Twenty-one caregivers 

(28%) were Hispanic. All Fisher’s exact tests were non 

significant (p>.05).

Preliminary analysis show no significant differences between

groups in gender, age (in months), and ethnicity. 

2.3 Differential 

attrition analysis

See responses to section 4.1 above.

We monitored retention rates and utilized various retention 

strategies in a proactive way. We actively tracked caregivers 

who were trying to drop out. In addition to getting 

information on three contacts, we established check in with 

caregivers, and worked with program staff to regularly 

assess the intent to move to a different location. 

2.4 Matching 

diagnostic 

statistics 

N/A

2.5 Description of 

counterfactual 

condition

Delayed treatment control group did not receive services 

with Cohort 2 participants due to premature project 

termination.

Data Collection & Measurement

3.1 Amount of data Program delivery began in June 2014 (recruitment and 

collected baseline assessments). Baseline assessments were completed 

in August 2014. Model implementation began in August

2014. Pathways to Literacy Model implementation was 

completed in November 2014. Full implementation began in 

January 2015 (Quarter 2, Year 3). Project was terminated in 

August 2015.
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

35

The types of data collected are described in detail in 

Implementation Data Collection and Measurement, section 

3.1 above.

3.2 Description of 

data collection 

methods

(See responses to 3.3) 

A complete list of measures and method used for collection is 

presented in Attachment 2. Assessments were conducted 

through interview, observation, and direct child assessment 

methods by the coaches. It was estimated each child 

assessment required about 45 minutes to an hour per 

assessment. These were completed in one session or, if 

necessary, two depending upon circumstances. The mode of 

data collection was the same in two groups.

3.3 Description of 

data collection 

procedures

(See responses to 3.4) 

As of August 2015 a total of 233 participants initially agreed 

to participate in the study; 92 participants in Cohort 1 and 

141 in Cohort 2. 74 of the 92 participants in Cohort 1 signed 

consents and completed full baseline assessments 

(enrollment paperwork, ACIRI, Caregiver Survey, CHELLO, 

and PLS 5) and 65 of the 141 participants in Cohort 2 signed 

consents and started baseline assessments. Fifty-one 

participants in Cohort 2 had baseline assessments competed 

by August 2015 before the project was terminated. 

Only after all of the assessments were completed, 

participants in Cohort 1 (N=74) were randomized to one of 

the two conditions. In Cohort 1, 36 participants were 

randomly assigned to Group A (experimental group), and 38 

were randomly assigned to Group B (control group). For 

Cohort 1, those with incomplete baseline data (n=18) did not 

participate in the study due to such reasons as no longer 

interested, moved, no longer had time, no further response, 

or the phone number was not valid. 

Of the Cohort 2 65 participants who signed consents, 14 had 

incomplete baseline data. Seventy-seven participants in 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

36

Cohort 2 were pre-enrolled (i.e., indicated interest) but did 

not yet sign consents in August 2015. 

All participants who were randomized (Cohort 1) were 

followed up for 3 and 6 month assessments. Participants in 

Group A, but not in Group B, received the Pathways to 

Literacy Model immediately after the baseline data 

collection.

Identical data were collected from both the treatment and 

control groups. Upon completion of screening, baseline data 

were collected by coaches from all study participants. Then 

participants were randomly assigned to either the control or 

treatment group. Caregivers in both treatment and control 

groups were assessed again at 3-, and 6 months. If there was 

more than one child under 54 months of age, one child was

randomly chosen to be assessed for the duration of the 

intervention. Child assessments occurred at baseline and 6 

months.

Pathways to Literacy staff were responsible for all outcomes 

and some implementation data collection as well as for data 

entry (internal database will be utilized). Evaluation 

Strategies’ staff participated in some data collection, data 

tracking, training providers on data collection, data entry, 

quality assurance, and coordination of data collection and 

data processing. For both groups, all assessments were 

conducted by coaches. In addition, supervisors and 

evaluation staff observed 28 sessions (home visits, 

workshops, and play groups) that were chosen randomly to 

establish fidelity of implementation.

3.4 Measure 

validation results

Currently not available

Analysis

4.1 Type of analysis Program delivery began in June 2014 (recruitment and 

baseline assessments). Baseline assessments were completed 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

37

in August 2014. Model implementation began in August

2014. Pathways to Literacy Model implementation was 

completed in November 2014. Full implementation began in 

January 2015 (Quarter 2, Year 3). Outcome/impact data will 

be fully analyzed in Year 5 as low n for Cohort 1 will not 

make analyses meaningful.

4.2 Power analysis 

findings

The original study design involved recruiting 800 

participants who would be divided into 400 participants per 

treatment or delayed control group condition. Power 

analyses were conducted again in March 2015 to determine if 

a smaller sample size would yield an appropriate power. It 

was concluded that a sample size of 600 participants who 

would be divided into 300 participants per treatment or 

delayed control group condition would yield sufficient 

power. 

Here are the assumptions used to calculate minimum 

detectable differences in the comparison of the Pathways to 

Literacy Model vs. a delayed treatment control condition:

1. Total sample size: 600, divided into 2 groups of 300 

participants per group.

2. Analysis sample size: After 20% attrition, there are n=192

participants per group.

3. Power: 80%

4. Number of comparisons: 1 (Since there are only two 

groups)

5. Alpha: Since there are only two groups, we use two 

tailed tests with alpha = .05.

6. Variance explained: 50% as an estimate of posttest 

variance that can be explained by the pretest and 

covariates is used - this will reduce the between-person 

variation and increase the precision of the estimate of the 

treatment.

In a balanced design with a proposed sample size with alpha 

at .05, power of .8, MDES for continuous outcomes is 

approximately .20. 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

38

We originally arrived at 800 by assuming small effect sizes (.2 

or .15) and factoring in attrition at 20% at two time 

points. With a Full Information Maximum Likelihood 

(FIML) estimation or multiple imputation procedure to 

address missing data we might not even need the 20% 

originally factored into the power analyses. The updated 

proposed sample size is 600.

4.3 Missing data 

analysis findings

Program delivery began in June 2014 (recruitment and 

baseline assessments). Model implementation began in 

August 2014. Full implementation began in January 2015 

(Quarter 2, Year 3). Data were analyzed to track fidelity of 

implementation. Outcome data will be fully analyzed in Year 

5.

Implementation Findings Outcomes

5.1 Implementation Cohort 1: Program delivery began in June 2014 (recruitment 

findings and baseline assessments for Cohort 1). Baseline 

assessments were completed in August 2014. Model 

implementation began in August 2014. Pathways to Literacy

Model implementation was completed in November 2014. 

Full implementation began January 2015 (Quarter 2, Year 3). 

Fourteen different Quality Assurance reports track fidelity 

of implementation info in real time (through the database).

This info was discussed with staff during weekly evaluation 

calls.

Fidelity of implementation was tracked through reviewing 

administrative records, staff interviews, and observations. 

Quality Assurance Reports were developed on a more 

frequent basis than originally intended (weekly as opposed 

to monthly). Rapid feedback was provided through

additional Quality Assurance reports through the database 

that allowed staff continuously and effortlessly monitor 

certain implementation characteristics (e.g., frequency of 

home visits by client, length of services, etc.).

The following services were provided during this study:
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

39

 Phone calls -1290

 Text messages-17

 Home visits-834

 Other (e.g., letters)-65

Below are the number and type of services provided by 

cohort.

 Usability

 Phone calls -76

 Text messages-0

 Home visits-31

 Other (e.g., letters)-34

Cohort 1

 Phone calls -727

 Text messages-16

 Home visits-651

 Other (e.g., letters)-34

Cohort 2

 Phone calls -487

 Text messages-1

 Home visits-152

 Other (e.g., letters)-9

5.2 Outcomes Program delivery began in June 2014 (recruitment and 

baseline assessments). Baseline assessments were completed 

in August 2014. Model implementation began in August

2014. Pathways to Literacy Model implementation was 

completed in November 2014. Full implementation began in 

January 2015 (Quarter 2, Year 3). 

Preliminary Findings Using Cohort 1 Data

 PLS 5 standard scores have a mean of 100 and a 

standard deviation of 15. Scores between + or -1.5 

standard deviation from the mean are considered to be 

within the average range. Cohort 1 data shows all the 

PLS 5 standard score means to be in the expected 

direction with most approaching significant difference 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

40

between the two groups (see Exhibits 1-3 below). The 

PLS 5 standard mean scores increased from baseline to 

6-month follow-up for both the PTL Curriculum and 

Control groups. However, the mean scores increased 

more markedly from baseline to 6-month follow-up for 

the PTL Curriculum group than for the Control group:

 Auditory Comprehension: Time main effectF(1,43)=3.32, p=.075; Condition main effect -

F(1,43)=3.68, p=.062; Interaction between time and 

condition- F(1,43)=.39, p>.05). More data are needed to 

document statistical significance.

 Expressive Communication: a significant Time main 

effect- F(1,43)=9.74, p<.01; a significant Condition main 

effect - F(1,43)=4.25, p>.05; Interaction between time

and condition- F(1,432.57, p<.05.

 The PLS 5 Total Language mean standard scores were 

significantly higher in the PTL Curriculum group 

(M=102.00) than in the control group (M=94.57) by 

about half a standard deviation (a significant Condition 

main effect, F(1, 38)=4.73, p<.05, see Exhibit 3). 

Percentile ranks indicate the percentage of children in 

an age group who score at or below a given score. 

Percentile ranks were higher for both PLS 5 Auditory 

Comprehension subtest (on average, 56 in the PTL 

curriculum group compared to 42 in the control group) 

and PLS 5 Expressive Communication subtest (on 

average, 54 in the PTL curriculum group compared to 

36 in the control group) in the PTL Curriculum group.

Exhibit 1. PLS 5- Auditory Comprehension Standard Scores
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

41

100.17

95.56

106.66

98.75

90

95

100

105

110

PTL Curriculum (n=29) Control (n=16)

PLS 5 Mean Scores for Auditory 

Comprehension Standard Score at 

Baseline and 6-month Follow-up for PTL 

Curriculum and Control Groups

Baseline 6-month Follow-up

Exhibit 2. PLS 5- Expressive Communication

PLS 5 Mean Scores for Expressive 

Communication at Baseline and 6-month 

Follow-up for PTL Curriculum and Control 

Groups

110 108.07

105

100

95

96.75 95.83

92.81

90

85

PTL Curriculum (n=29) Control (n=16)

Baseline 6-month Follow-up

Exhibit 3. PLS 5- Total Language
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

42

 Both time and condition contributed to changes in 

scores for ACIRI (i.e., an interaction between time and 

condition was significant, F(2,33)=38.13, p<.001). For the 

PTL Curriculum group, both Adult Behavior mean 

scores (see Exhibit 4) and Child Behavior mean scores 

(see Exhibit 5) on the ACIRI significantly increased from 

baseline to 3-month follow-up (only in the experimental 

condition, (Adult Behavior-t(30)=-8.81, p<.001; Child 

Behavior- (t(30)=-6.49, p<.001) and significantly 

decreased from 3-month follow-up to 6-month followup (Adult Behavior-t(20)=2.99, p<.01; Child Behavior- 

(t(20)=3.10, p<.01). The 6-month follow-up average 

score was significantly higher than the original average 

baseline score for the PTL Curriculum group at 3 

months (Adult Behavior-t(57)=10.18, p<.001; Child 

Behavior- (t(57)=6.83, p<.001)) and 6 months (Adult 

Behavior-t(36)=11.55, p<.001; Child Behavior- 

(t(57)=8.58, p<.001)), but not at baseline (Adult 

Behavior-t(72)=-.703, p>.05; Child Behavior- (t(72)=-.47, 

p>.05)). For the Control group, Adult Behavior mean 

scores on the ACIRI did not change from baseline to 3-

month follow-up (Adult Behavior-t(27)=1.92, p>.05; 

Child Behavior- (t(27)=1.58, p>.05) and significantly 

decreased from 3-month follow-up to 6-month follow97.96

94.21

106.04

94.93

85

90

95

100

105

110

PTL Curriculum (n=26) Control (n=14)

PLS 5 Total Language Mean Scores at 

Baseline and 6-month Follow-up for PTL 

Curriculum and Control Groups

Baseline 6-month Follow-up
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

43

up (Adult Behavior-t(14)=4.92, p<.001; Child Behavior- 

(t(14)=2.77, p<.05). 

Exhibit 4. ACIRI - Adult Behavior 

1.21 1.46

2.27

1.26

2.06

0.71

0

0.5

1

1.5

2

2.5

PTL Curriculum (n=21) Control (n=15)

ACIRI Adult Behavior Mean Scores at 

Baseline, 3-month Follow-up, and 6-

month Follow-up for PTL Curriculum and 

Control Groups

Baseline 3-month Follow-up 6-month Follow-up

Exhibit 5. ACIRI - Child Behavior
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

44

 For four literacy areas assessed by CHELLO (book area, 

book use, writing materials, and toys) the PTL 

Curriculum group showed significant improvement in 

total scores (see Exhibits 6-10 for more details) as 

compared to the total scores in the control group. Both 

time and condition contributed to changes in scores for 

CHELLO (i.e., an interaction between time and 

condition was significant, book area- F(2,32)=36.03, 

p<.001; book use- F(2,32)=7.88, p<.01; writing materials 

–F(2,32)=14.05, p<.001; toys- F(2,32)=6.60, p<.01). For 

technology – interaction between time and condition 

was not significant (F(2,32)=.53, p>.05), but the time 

main effect was (F(2,32)=3.38, p<.05). When averaging 

over condition, Technology total scores on the CHELLO 

were significantly higher from baseline to later 

assessment scores. 

Exhibit 6. CHELLO - Book Area

1.02 1.40

2.03

1.10

1.74

0.63

0

0.5

1

1.5

2

2.5

PTL Curriculum (n=21) Control (n=15)

ACIRI Child Behavior Mean Scores at 

Baseline, 3-month Follow-up, and 6-month 

Follow-up for PTL Curriculum and Control 

Groups

Baseline 3-month Follow-up 6-month Follow-up
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

45

1.68

3.06

4.11

2.63

4.47

0.88

0

1

2

3

4

5

PTL Curriculum (n=19) Control (n=16)

CHELLO Book Area Total Scores at Baseline, 

3-month Follow-up, and 6-month Follow-up 

for PTL Curriculum and Control Groups

Baseline 3-month Follow-up 6-month Follow-up

An interaction between time (Baseline, 3 months, 6 months) 

and condition (PTL Curriculum, Control) was significant for 

the Book Area total scores (F(2,32)=36.03, p<.001). For the 

PTL Curriculum group, Book Area total scores on the 

CHELLO increased notably from baseline to 3-month 

follow-up (t(30)=-8.42, p<.001) and did not significantly 

improve from 3-month follow-up to 6-month follow-up

(t(18)=-1.24, p>.05). For the Control group, Book Area total 

scores on the CHELLO did not significantly change from 

baseline to 3-month follow-up (t(28)=.86, p>.05) and 

significantly decreased from 3-month follow-up to 6-month 

follow-up (t(15)=4.34, p<.001). At each time point except 

baseline (t(72)=-.64, p>.05), scores for the PTL Curriculum 

group were significantly higher than for the control group. 

Exhibit 7. CHELLO - Book Use
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

46

5.05

6.13

7.00

6.19 6.53

4.69

0

2

4

6

8

PTL Curriculum (n=19) Control (n=16)

CHELLO Book Use Total Scores at Baseline, 

3-month Follow-up, and 6-month Followup for PTL Curriculum and Control Groups

Baseline 3-month Follow-up 6-month Follow-up

An interaction between time and condition was significant 

for these Book Use total scores (F(2,32)=7.88, p<.01). For the 

PTL Curriculum group, Book Use total scores on the 

CHELLO significantly increased from baseline to 3-month 

follow-up (t(30)=-5.17, p<.001) and did not change from 3-

month follow-up to 6-month follow-up (t(18)=1.58, p>.05). 

The 6-month follow-up score was significantly higher than 

the original baseline score for the PTL Curriculum group. 

For the Control group, Book Use total scores on the 

CHELLO stayed about the same from baseline to 3-month 

follow-up (t(28)=-.90, p>.05). and significantly decreased 

from 3-month follow-up to 6-month follow-up (t(15)=2.27, 

p<.05 for both). At each time point except baseline 

(t(72)=.51, p>.05), scores for the PTL Curriculum group were 

significantly higher than for the control group. 

Exhibit 8. CHELLO - Writing Materials
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

47

3.32

4.13

4.63

3.25

4.84

2.88

0

1

2

3

4

5

6

PTL Curriculum (n=19) Control (n=16)

CHELLO Writing Materials Total Scores at 

Baseline, 3-month Follow-up, and 6-

month Follow-up for PTL Curriculum and 

Control Groups

Baseline 3-month Follow-up 6-month Follow-up

An interaction between time (Baseline, 3 months, 6 months) 

and condition (PTL Curriculum, Control) was significant for 

the Writing Materials total scores (F(2,32)=14.05, p<.001). 

For the PTL Curriculum group, Writing Materials total 

scores on the CHELLO increased significantly from baseline 

to 3-month follow-up (t(30)=-4.85, p<.001) and changed 

further from 3-month follow-up to 6-month follow-up

(t(19)=-4.14, p<.001). For the Control group, Writing 

Materials total scores on the CHELLO did not significantly 

change from baseline to 3-month follow-up (t(28)=.31, 

p>.05) and from 3-month follow-up to 6-month follow-up

(t(15)=.66, p>.05). At each time point except baseline 

(t(72)=-.01, p>.05), scores for the PTL Curriculum group 

were significantly higher than for the control group. 

Exhibit 9. CHELLO - Toys
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

48

2.21

2.38

2.63

1.75

2.47

1.31

0

0.5

1

1.5

2

2.5

3

PTL Curriculum (n=19) Control (n=16)

CHELLO Toys Total Scores at Baseline, 3-

month Follow-up, and 6-month Follow-up 

for PTL Curriculum and Control Groups

Baseline 3-month Follow-up 6-month Follow-up

For the PTL Curriculum group, an interaction between time 

and condition was significant (F(2,32)=6.60, p<.01). For PTL 

Curriculum Group, Toys total scores on the CHELLO 

significantly increased from baseline to 3-month follow-up 

(t(30)=2.16, p<.05) and did not significantly change from 3-

month follow-up to 6-month follow-up (t(18)=.77, p>.05). 

The 6-month follow-up score was the same as the original 

baseline score for the PTL Curriculum group (t(19)=-1.16, 

p>.05). For the Control group, Toys total scores on the 

CHELLO decreased markedly from baseline to 3-month 

follow-up (t(28)=3.09, p<.01) and did not significantly 

change from 3-month follow-up to 6-month follow-up

(t(15)=1.13, p>.05). At each time point except baseline 

(t(72)=.67, p>.05), scores for the PTL Curriculum group were 

significantly higher than for the control group. 

Exhibit 10. CHELLO - Technology 
Detroit Parent Network: Pathways to Literacy

August 2015 Final Evaluation Report

49

CHELLO Technology Total Scores at 

Baseline, 3-month Follow-up, and 6-month 

Follow-up for PTL Curriculum and Control 

Groups

2 1.79

1.47 1.50

1.37 1.38 1.5

1.13

1

0.5

0

PTL Curriculum (n=19) Control (n=16)

Baseline 3-month Follow-up 6-month Follow-up

Across both groups, Technology total scores on the 

CHELLO increased from baseline to 3-month follow-up and 

decreased from 3-month follow-up to 6-month follow-up. 

When averaging over condition, Technology total scores on 

the CHELLO were significantly higher from baseline to 

later assessment scores (time main effect, F(2, 32)=3.38,

p<.05).

5.3 Preliminary 

impact findings 

This study was terminated prematurely. Only Cohort 1 data 

were collected pre-post.

5.4 Impacts This study was terminated prematurely. Only Cohort 1 data 

were collected pre-post.

5.5 Lessons learned Quality Assurance Process was in place to monitor 

implementation of the Pathways to Literacy Model. 

Additional feedback was provided on a continuous basis 

through the database. Weekly Quality Assurance Reports 

monitored missing data, ways data were coded, how data

were tracked, and consistency in ways to document 

information.
Attachment 1:

Pathways to Literacy Tasks and Timeline

50

Attachment 1: Pathways to Literacy Tasks and Timelines 

Year 1 Year 2 Year 3 Year 4 Year 5

Activity Q

1

Q

2

Q

3

Q

4

Q

1

Q

2

Q

3

Q

4

Q

1

Q

2

Q

3

Q

4

Q

1

Q

2

Q

3

Q

4

Q

1

Q

2

Q

3

Q

4

General Development/Planning

Finalize components of the model X X X X X

Finalize evaluation plan X X X X X

Consent development X X X X

Instrumentation X X X X X

Develop data collection procedures X X X X

Develop data collection training X X X X X

Document translation into Spanish X X X X X

Initial IRB X X

Develop implementation 

instrumentation X X X X

Database development X X X X X

IMPLEMENTATION PHASE 1: USABILITY (THE PILOT STUDY)

Usability Implementation X X X

Consent training X X X

Instrument Adjustment X X
Attachment 1:

Pathways to Literacy Tasks and Timeline

51

Year 1 Year 2 Year 3 Year 4 Year 5

Activity Q

1

Q

2

Q

3

Q

4

Q

1

Q

2

Q

3

Q

4

Q

1

Q

2

Q

3

Q

4

Q

1

Q

2

Q

3

Q

4

Q

1

Q

2

Q

3

Q

4

Procedure Adjustment X X X

IMPLEMENTATION PHASE 2: FORMATIVE PHASE

Formative Implementation (n=100) X X

Addition of control group: 

Randomization X X X X X X X X X X

IMPLEMNATION PHASE 3: FULL IMPLEMENTATION

Full Implementation (n=600) X X X X X X X X X X X X

3 and 6 month follow ups (caregiver) X X X X X X X X X X X X X

6 month follow up (child) X X X X X X X X X X X

ON-GOING TASKS

Consent tracking X X X X X X X X X X X X X X

Randomization tracking X X X X X X X X X X X X

IRB Updates X X X X

Data entry X X X X X X X X X X X X X X X

Track and locate missing participants X X X X X X X X X X X X X X X

Implementation analysis/QA X X X X X X X X X X X X X X X X

Bi-Monthly Project Management 

Meetings X X X X X X X X X X X X X X X X X X X X
Attachment 1:

Pathways to Literacy Tasks and Timeline

52

Year 1 Year 2 Year 3 Year 4 Year 5

Activity Q

1

Q

2

Q

3

Q

4

Q

1

Q

2

Q

3

Q

4

Q

1

Q

2

Q

3

Q

4

Q

1

Q

2

Q

3

Q

4

Q

1

Q

2

Q

3

Q

4

Analyses X X X X

Semi-Annual Reports X X X X

Annual Reports X X X X X

Dissemination X X X X X X X X X X X X X X X X

Utilization-focused Implementation 

Reports X X X X X X X X X
53

Attachment 2: Pathways to Literacy Evaluation Measures

Measure Description Method of Collection Technical Information

Child Outcomes

Preschool 

Language 

Survey5(Zimmerman, 

Seiner, & Pond, 

2011)

The PreSchool Language Scale, 

Fifth Edition (PLS-5) is an 

individually administered test 

for identifying children from 

birth through 7.11 years. It 

assesses: 

 Language

 Articulation

 Connected Speech

 Social/Interpersonal 

Communication Skills

 Stuttering

 Voice

Individually administered by 

trained home visitor. Contains 

Pointing or verbal response to 

pictures and objects. 

Completion Time: 45-60 

minutes 

Reviews on PLS 5 are under review, PLS 4 data is follows:

The reliability of PLS-4 was estimated using test-retest 

reliability (data that show that PLS-4 scores are dependable 

and stable across repeated administrations), internal 

consistency (data that show tasks in PLS-4 are homogeneous), 

and inter-rater reliability (data that show scoring is objective 

and consistent across examiners). The test-retest stability 

coefficients ranged between .82 and .95 for the subscale scores 

and .90 to .97 for the Total Language Score. The internal 

consistency reliability coefficients range from .66 to .96 (for 

most ages the coefficients are .81 and higher). The inter-rater 

reliability study included 15 scorers who scored the 

Expressive Communication subtest on 100 protocols selected

from the standardization sample. Each protocol was scored by 

two different scorers. The percentage of agreement between 

scorers was 99% and the correlation between the Expressive 

Communication scores was .99.

Internal Structure. The internal consistency of the subscales 

were examined for evidence of high homogeneity. The internal 

structure of the PLS–4 was also examined—the correlation 

between the two subscales (Auditory Comprehension and 

Expressive Communication) across ages was .80. 

Relationships with Other Variables. A clinical validity study 

was conducted with a sample of 150 children (75 with a 

language disorder, 75 typically developing children). 

Sensitivity and specificity information for PLS-4 scores for 

children in this study are: 

 Auditory Comprehension Sensitivity .80 Specificity .92

 Expressive Communication Sensitivity .77 Specificity .84

 Total Language Score Sensitivity .80 Specificity .88






Auditory

Comprehension subtest was .65; the correlation of the PLS3/PLS-4

Expressive Communication subtest was .79.

Parent Outcomes and Home Environment

Child/Home 

Early Language 

and Literacy 

Observation 

(CHELLO) 

(Neuman & 

Koh, 2007)

The CHELLO examines 

language and literacy practices 

specific to the contextual 

features of family and homebased child care settings 

(Neuman, Dwyer, & Koh, 

2007). The CHELLO is 

composed of two 

interdependent research tools: 

The Literacy Environment 

Checklist, and the Observation 

and Provider Interview. The 

Literacy Checklist measures 

the presence or absence of 22 

items in the environment, 

including the accessibility of 

books, writing materials, and 

displays of children’s work. 

Observation by Pathways to 

Literacy coaches. Checklist 

contains 22 items and can be 

completed in 10 minutes. 

Reliability: Inter-Rater Reliability 91%.

Internal Consistency: Cronbach’s Alpha .82.

Internal Correlations ranged from .34 to .97.

Concurrent Validity: The CHELLO correlated significantly 

with children’s language growth (as measured by the PPVT (r

= .36, p<.01), phonological skills (as measured by the PALS 

nursery rhyme (r = .25, p< .05)), and ability to do languageoriented math problems (as measured by the WoodcockJohnson Applied problems test (r = .28, p<.05)). 

The Adult Child 

Interactive 

Reading 

Inventory 

(ACIRI)

(DeBruinParecki, 2006)

Grounded in scientifically 

based reading research and 

extensively field tested, ACIRI

 Measures what's 

important. ACIRI assesses 

both adult and child 

behaviors in three categories 

that research has identified as 

critical: Enhancing Attention 

to Text, Promoting 

Interactive Reading and 

Supporting Comprehension, 

and Using Literacy Strategies.

Observation by Pathways to 

Literacy coaches. 

Not Available

54
55

 Takes just 15–20 minutes for 

users to observe the adult 

and child reading together, 

assess them jointly using 12 

key reading behaviors, and 

score the assessment with the 

simple, easy-to-use form.

 Includes intervention 

activities. ACIRI is much 

more than an assessment. For 

each behavior evaluated, 

users will get tips on 

explaining the behavior to 

adults, plus two fun, 

photocopied activities to help 

promote the behavior: a class 

activity and a take-home 

activity. Lists of 

recommended children's 

books to use with the 

activities are also included

Parental 

Literacy 

Knowledge Test

Measure of knowledge 

attained regarding literacy. 

 

Caregiver Survey Not Applicable. 

Survey created by Evaluation Strategies based upon content of 

the Pathways to Literacy Curriculum.

Parent/Guardian Life Course

Educational 

attainment

Attending school, high school 

graduate, general educational 

development recipient (GED) 

or postsecondary education

Caregiver Survey; Pathways 

to Literacy Central Intake 

Form

Not Applicable

Marital status Married with or without 

partner in the home or single 

with or without partner in 

home

Caregiver Survey; Pathways 

to Literacy Central Intake 

Form

Not Applicable

Economic status Job training, employment, total 

household income, or reliance 

on government benefits

Caregiver Survey; Pathways 

to Literacy Central Intake 

Form

Not Applicable
56

Attachment 3: IRB Approved Updated Consents
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

Table of References 

Institute of Education Sciences. (2013). What Works Clearninghouse: Procedures and 

Standards Handbook. 2.1. Retrieved 11/06/2013, 2013, from 

http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standa

rds_handbook.pdf

Neuman, S. B., J., D., & Koh, S. (2007). Child/ Home Early Language and Literacy 

Observation (CHELLO) Boston, MA Brookes Publishing.

U.S. Census Bureau 2010 American Community Survey. (2010).