**U.S. Department of Education**

**Office of Educational Research and Improvement**

**Preschool Curriculum Evaluation Research Grants**

**Preapplication Meeting Summary**

**January 24, 2002**

**1:00−4:00 P.M.**

[1:00**−**1:20 Welcome and Introductions]{.underline}

The Preschool Curriculum Evaluation (PCER) preapplication meeting began
with a welcome and introductions by Dr. Heidi Schweingruber (Program
Officer) and Naomi Karp (Director, National Institute for Early
Childhood Development and Education). Dr. Valerie Reyna (Senior Research
Advisor) provided the purpose and a context for the meeting. She talked
about the need for cognitive development research, OERI's focus on
scientific evidence based research, and a new report from the National
Academy of Sciences on "scientific research in education." She also
spoke about providing documentation on "what works" evidence in
research---the mechanism of why and how the research and methods worked.
In closing, she encouraged the attendees to think about creating the
idea of partnerships for research. Assistant Secretary Grover Whitehurst
was unable to attend the beginning of the meeting, but provided
greetings and closing remarks at the end. A list of all persons who
attended the meeting was recorded and is available upon request.

The attendees received an overview of the agenda, primary goals of the
PCER, and a program timeline. The agenda was organized around the
questions received from potential applicants. There is a broad range of
people who are interested in the competition. There are sites with
extensive field experiences, publishers, and researchers. Therefore, the
agenda operates at several levels and the information is designed to
cover the broad range of applicants represented. Some attendees may be
familiar with certain sections of the agenda, but others in the audience
may not be so familiar with the same information. The attendees also
were directed to a list of Web site resources in one of the handouts.

[1:20--1:50pm Nuts & Bolts of the Application and Funding
Process]{.underline}

Dr. Schweingruber walked through the handouts related to preparing the
application. The overview was intended to highlight aspects of the
Federal Announcement that applicants might overlook. For each section of
the proposal, she provided some general tips that are detailed in the
handouts.

[Key points]{.underline}:

Applicants must be sure to address each of the four review criteria
listed (note the weight that each criterion will receive): National
significance (.2); Quality of the project design (.5); Quality and
potential contributions of personnel (.2); Adequacy of resources (.1).

Pay particular attention to the section of the Federal Register
announcement that is titled PRIORITY (it appears towards the end of the
announcement). This details the features that an application absolutely
MUST have to be considered.

Dr. Schweingruber also provided a quick overview of the E-application
process, by which all were encouraged to submit their applications.

Questions and answers after this segment are included in the separate Q
& A document.

[1:50**−**2:20 Role of the National Contractor]{.underline}

Dr. Schweingruber resumed with a brief presentation of the respective
responsibilities of grantees and the contractor. Again, these details
are included in the hand-outs for this section. In addition, she briefly
reviewed the FACES instruments that will form the core of the evaluation
instruments used. URL's for these instruments are included in the
hand-outs document.

Questions and answers after this segment are included in the separate Q
& A document.

[2:30--3:15pm Technical Issues: Guest Speaker David Arnold,
Ph.D.]{.underline}

After a short break, the attendees were asked to hold their questions
until after the guest speaker, David Arnold, Ph.D., of the University of
Massachusetts at Amherst, finished his presentation. Dr. Arnold offered
to answer program specific questions at <darnold@psych.umass.edu>). Dr.
Arnold's advice was to read the call for proposals thoroughly. In
reference to design issues, he suggested that applicants also take a
look at the Head Start Impact Study Advisory Committee report at:
www2.acf.dhhs.gov/programs/hsb/hsreac/octrep.htm

Dr. Arnold highlighted the following technical points:

Design issues with focus on random assignment

Keeping sites happy

Assessment strategies

Retention of participants

-   Random assignment and how to get schools on board:

```{=html}
<!-- -->
```
-   Inform the sites that a random assignment has to be done. It is a
    grant requirement.

-   Emphasize that it is not forever. It can provide an opportunity to
    try out a curriculum to see if it is worth implementing for the
    whole site.

-   Talk to the sites about the rationale for random assignment. Be
    respectful of directors wanting to know why the procedure is
    necessary. Explain that if a classroom is chosen because one teacher
    needs it the most, or earned the right to use the new curriculum,
    etc., it will not be known whether it would work for other kinds of
    teachers and classrooms. Take the time to talk this through.

-   Random assignment is a great way to avoid hurt feelings and to
    ensure a treatment of fairness. It's a great way to make things less
    personal. If you do not use random assignment, teachers may ask if
    they were chosen because they were the worst or the best, etc.
    Random assignment can be a great equalizer.

```{=html}
<!-- -->
```
-   How to do random assignment

```{=html}
<!-- -->
```
-   Practically speaking, it is easiest to divide by larger units such
    as schools or classrooms. This might work the best in terms of
    avoiding contamination with the control group or demoralization of
    teachers who have or have not been chosen to implement.

-   For a more powerful design, it is the reverse. That is, the more
    powerful design involves random assignment at lower levels of
    analysis. Having one classroom with 20 kids in it is not the same as
    having 20 independent data points.

> [Example]{.underline}: If one classroom has 500 kids in the control
> group and another classroom has 500 kids in the intervention group,
> even if huge differences occur, it would be hard to conclude whether
> it was the intervention or the teacher, because there is really only
> one group.

-   The key to how much power you have is how many groups it's being
    implemented with.

-   The best proposals are probably going to have power analyses to
    justify sizes of groups. Find someone to help figure this out.

```{=html}
<!-- -->
```
-   Consider hierarchical linear modeling (HLM)

```{=html}
<!-- -->
```
-   Allows consideration of multiple levels, nested factors

-   Robust to missing data and maximizing intervals

-   Not finicky

```{=html}
<!-- -->
```
-   Timing of random assignment -- Is random assignment done before kids
    are placed in their classrooms? Is it done before the school year
    starts? Is it done after a pre-test so that children can be matched
    on some variables?

```{=html}
<!-- -->
```
-   Methodologically, matching is a great way to increase power. If
    classrooms are matched before random assignment, the error variance
    can be decreased.

-   In the real world, one may not be able to wait until a pre-test is
    done to tell the teachers what they are going to be doing all year.

-   Think through the pros and cons of various approaches. Find someone
    who can help you with this.

```{=html}
<!-- -->
```
-   Keeping sites happy

    -   The best way to figure out what sites may want is to ask them
        directly. Talk to sites about what will keep them happy.

    -   Communicate that you respect how busy they are, but also that
        you know they want to be a partner. You need to walk the line
        between over burdening them and leaving out information.

    -   Sites have to see you enough to know that they can tell you what
        is NOT going right.

    -   In terms of incentives, think broadly about what incentives can
        be. Sites may differ in terms of what they need.

-   Assessments

```{=html}
<!-- -->
```
-   Look for convergent evidence across multiple kinds of measures.

-   Aggregation. It's hard to measure things exactly, but when you put
    more than one measure together, you tend to get more accurate
    measures. This is especially true for things that are tough to
    measure.

-   A big job of the grantee is documenting the fidelity of the
    intervention. For example, if you don't see strong effects, you will
    want to know whether that was because they didn't implement the
    curriculum correctly, or whether the curriculum "didn't work." You
    really want to know what happened in the classroom.

-   For documenting implementation, you want to look at both near
    outcomes and far outcomes.

> Example: Near things might be: did the teachers come to the meetings?
> Where did you train them? Slightly further might be: did they try
> activities in the classroom? The next level might be: did they do a
> good job implementing the activities? Another level might be: how did
> the kids respond?

-   Think through what would you need to know if things don't go right,
    to figure out where the implementation broke down, or if the results
    are due to ineffectiveness of the curriculum.

```{=html}
<!-- -->
```
-   Good Retention

```{=html}
<!-- -->
```
-   Frequent contact with families. This contact can be of almost any
    kind, even just a phone call to check in.

-   Maintain contact with other people besides the people who will help
    you find a family if they move or relocate.

This ended Dr. Arnold's formal presentation. At the close of the
presentation, Dr. Schweingruber followed up with the audience by stating
that they might want to think about the incentives, think through how
one would use the incentives and how their use might affect the design,
especially the control group. Before the question and answer session,
Dr. Schweingruber asked that any individuals in the audience with
expertise in this area should feel free to offer advice and suggestions.

Questions and answers after this segment are included in the separate Q
& A document.

[2:30--3:15pm Complementary Research Studies]{.underline}

Dr. Schweingruber briefly went over some guidelines for the
complementary research studies. In summary, the studies must

-   Be motivated by a specific conceptual framework and relevant prior
    empirical evidence, both of which must be clearly articulated;

-   Develop hypotheses or research questions that can be tested;

-   Indicate method and why the approach taken optimally addresses the
    research question;

-   Present a clear design;

-   Distinguish independent and dependent, or predictor and criterion,
    variables;

-   Select any additional measures carefully and justify their use; and

-   Include a data analysis plan.

Questions and answers after this segment are included in the separate Q
& A document.

The last question was asked and Assistant Secretary Whitehurst greeted
the audience and offered some encouraging words about the timeliness and
importance of this initiative to President Bush's educational priorities
and the First Lady's testimony before Congress about the need for more
research in early childhood education. As the meeting ended, many of the
participants remained for additional questions and comments.