**Guidance to ECEPD Grantees**

**Clarifying Information Submitted in FY 2007 Performance Reports**

Grantees should include the following information in their evaluation
reports to assist the program office in determining the rigor of local
evaluations and, for rigorous evaluations, what can be said about the
impact of ECEPD grantee projects. Grantees should submit this report
along with their Annual Performance Report (APR). The evaluation report
needs to include the following detailed information regarding the
intervention and the evaluation design, implementation, and findings. If
a grantee's evaluation is not complete, the grantee should provide as
much information as possible on the following items with each APR.

Note to reader: The evaluation for Project MENTOR was based upon the
plan approved in the original grant application. Although the project
evaluation did measure changes in provider knowledge and practice as
well as child-level outcomes, MENTOR staff did [not]{.underline}
administer the required GPRA evaluation instruments (which were
instituted after the commencement of the project).

**Intervention**

-   Describe the professional development (PD) intervention as
    implemented including the setting, content, and delivery (i.e.,
    curriculum, provider, duration, intensity, and implementation
    fidelity).

This information is also described in the evaluation report.
**Setting**: The 40 hours of professional development occurred in the
classrooms of participating providers --some participants were
center-based preschool providers while other participants were family
childcare providers. For center-based providers, training occurred in
their classrooms or small group settings at their centers. Training for
family childcare providers took place in their homes during or after
business hours. **Content/Delivery:** The focus of the professional
development was determined by each provider's pre-project assessment and
needs identified jointly by the provider and mentor. All providers
received training on language development, classroom environment and
instructional strategies to support early literacy development. However,
the materials/curriculum and training strategies (coaching,
demonstrations, etc.) varied depending upon provider need. Providers
also received copies of the Community-based Learning Modules.

**Evaluation design**

Present the final evaluation questions.

Evaluation questions are included in the final evaluation report.

-   Describe the evaluation design, indicating whether it is an
    experimental, quasi-experimental, or other study. For experimental
    and quasi-experimental designs, describe how treatment and
    control/comparison groups were assigned or matched. For "other
    studies," explain the rationale i.e. why an experimental or
    quasi-experimental study was not conducted and describe the details
    of the evaluation design.

The evaluation utilized a quasi-experimental design. For participating
providers, changes in knowledge and practice were measured using
pre/post-treatment classroom observation tool (administered by mentors),
mentors' logs, and provider surveys and interviews. There was no control
group of providers.

Child-level outcomes were measured only for kindergarten-bound children
(a relatively small percentage of children in the care of participating
providers). A comparison sample of kindergarten-bound children who had
not participated in a center-based preschool program was obtained in
collaboration with participating school districts.

-   For experimental and quasi-experimental evaluations describe
    services received (if any) by the control/comparison group including
    the setting, content, and delivery of services. If other designs
    were implemented in lieu of an experimental and quasi-experimental
    evaluation, explain what was done.

There was no comparison group of providers. A pre/post design was used
to measure changes in provider knowledge and practice. The attached
final evaluation report describes instruments, data collection
procedures and provider outcomes.

For child-level outcomes, the comparison group consisted of
kindergarten-bound children who had not participated in any formal
center-based preschool (these children may have been in family child
care settings that did not participate in the project). The assessments
of children in the comparison group were conducted during summer
pre-Kindergarten camps offered by two school districts that participated
in the project.

-   Describe the size of the sampling frame, and how the study's sample
    was selected. Provide the number of centers, classrooms, teachers,
    and/or children selected for each group in the study.

All providers who participated in Project MENTOR were invited to
participate in evaluation activities; however providers who declined to
participate still received services. The final evaluation report lists
the total number of centers, family child care homes, and project
participants. The report also lists the number and age ranges of
children in the care of project providers. Only kindergarten-bound
children with positive parental consent (a small percentage of the total
number of children in the care of participating providers) participated
in the assessments. Parental consent was also solicited for comparison
children who attended a summer pre-Kindergarten camp.

-   Describe all teacher and student outcome measures used in the study
    (GPRA and non-GPRA), including evidence that the instruments used
    are reliable and valid.

The project did not use the required instruments to measure teacher or
student outcomes because the specified instruments were not identified
when the project was funded. The evaluation report describes the
instruments that were approved by Dr. Michael Kamil when the evaluation
design was finalized in November 2004 and were used to measure teacher
and student outcomes.

**Evaluation implementation**

-   For experimental and quasi-experimental evaluations, compare the
    characteristics between the treatment and control/comparison groups
    to show that there were no systematic differences at baseline. If
    there were systematic differences, describe those differences and
    how they were addressed in the analysis.

Not applicable. The kindergarten-bound comparison group was a
convenience sample that was demographically similar to children in the
care of participating providers. No baseline assessments were conducted
on any children in the project.

-   Discuss the timing and procedures used for data collection. For
    experimental and quasi-experimental evaluations, discuss whether the
    data collection for the treatment and control/comparison groups used
    the same procedures and was conducted at the same (relative) times.

Early literacy assessments were conducted in May- June on all
kindergarten-bound children (with parental consent) in the care of
participating providers. Comparison group assessments were conducted in
June-July of each program year.

-   Provide attrition rates (percentage of teachers and children who
    participated in the pre-tests but not post-tests) and response rates
    (the percentage of teachers and children for whom there are data for
    each instrument).

The evaluation report describes the dropout rate for providers. A total
of 311 providers participated in the project and 104 have pre/post
classroom observation data. Of the approximately 2,308 children in the
care of participating providers, 170 were kindergarten-bound and 128
were included in the child-level spring assessments. The attached
evaluation report also describes the evaluation instruments for teachers
and target children.

-   Describe how the data were analyzed for each outcome. Be specific
    about the statistical techniques used. For regression analyses,
    describe the specified model including covariates. For hierarchical
    linear models (HLM) also identify the levels. Provide the type of
    statistical test used to determine significance, and describe how
    effect sizes were calculated.

Four instruments were used to measure changes in ECE provider practice:
(1) pre/post classroom observation scale (the Early Literacy Environment
and Practices Rating Scale- ELEP), (2) End-of-project structured
interviews with participants, (3) participant surveys, and (4) mentor
coaching logs documenting qualitative changes in provider practice. ELEP
baseline and follow-up ratings were available for 104 providers. The
classroom environment scale yields ordinal level data, and the results
were presented as the percentage of providers scoring at one of five
performance levels. The post intervention performance target was to
increase the percentage of providers scoring at or above 3 of the
five-point scale. Results for each subscale are presented in the final
evaluation report. A chi square test indicated that providers made
statistically significant gains from pre to post; however, because
providers' pre-treatment scores were so low, many providers failed to
score 3 or higher on the post-observation (a score that indicates the
provider meets high quality standards for professional practice).

Child-level outcomes were measured only for kindergarten-bound children
with parental consent. Child outcome data were presented as average
correct for some items and percent of children proficient on other
items. The evaluation report contains a description of how child-level
data were analyzed.

-   Describe any problems in implementing the evaluation design and
    lessons learned and how they were addressed.

The overall challenge in implementing the evaluation was that the
project was designed around professional development needs of
participating providers, resulting in varying types of treatments
received by participating providers. The work conditions of providers
served varied tremendously (some were family childcare providers; others
were private center-based preschool teachers). Even among the preschool
teachers there was variation in the curriculum utilized, identified
professional development needs at the beginning of the project, and
demographic characteristics of children served. Because the professional
development was guided by provider needs, providers received treatments
with differing content focus, length, and delivery strategies (e.g.,
modeling, demonstrations, feedback, etc.).

**Evaluation findings**

-   For each outcome measure, present statistics including, but not
    limited to:

    -   Means (by treatment status), indicate if they were
        regression-adjusted,

    -   Standard deviations (by treatment status),

    -   Sample size (by treatment status),

    -   Statistical significance (e.g., p-value), and

    -   Effect size estimates (i.e., the magnitude of impact).

The final evaluation report presents evaluation findings. The means were
not regression-adjusted.

-   State what the evaluation results say about the intervention\'s
    effectiveness and how success was defined.

Mentor logs and post-training reflections and provider interviews
provide evidence of the impact project services had upon the practices
of participating providers. Success was modestly defined as any
improvement in practice. Although the majority of providers made
significant progress, they still fell short of meeting standards of
professional practice that would result in the desired child-level
outcomes.

-   Describe factors and circumstances that may account for the
    intervention\'s effect (or lack thereof). For example, if the
    comparison group was exposed to similar services provided to the
    treatment that may diminish the observed differences between the
    groups.

See evaluation report and comments above

-   If experimental or quasi-experimental evaluation designs were not
    implemented, provide statistics for other evaluation designs.

Not applicable