**A. JUSTIFICATION**

This is a request for OMB clearance to collect data for the study, *An
Evaluation of States' Monitoring and Improvement Practices Under the
Individuals with Disabilities Education Act (IDEA)*. This study is being
conducted for the U.S. Department of Education's National Center for
Special Education Research (NCSER) at the Institute of Education
Sciences (IES), under a contract with Westat and its small business
subcontractors―Chesapeake Research Associates, LLC; Compass Consulting
Group; and Walsh Taylor, Inc.

IES is seeking approval for the second of two primary data collections
for this study. Clearance for the first data collection activity, a mail
survey to obtain information about monitoring and improvement practices
under Parts B and C of IDEA from the 50 states and District of Columbia,
was received 9/30/05 with the terms that ED will brief OMB on the status
of this evaluation following the first round of site visits---which ED
agrees to do. We are now seeking OMB approval for the second data
collection activity, which involves in-depth site visits to a systematic
random sample of 20 states to further define their current monitoring
systems and assess the effectiveness of these systems. The overall study
design and two data collections are described below in item 1, with more
detail provided for the site visit data collection activity, as that is
the focus of this request for OMB clearance.

1.  **Purpose and Authority**

[Overview]{.underline}

Section 664 of IDEA 2004 gives the Director of IES the responsibility to
carry out studies and evaluations to assess progress in the
implementation of IDEA. Congress created NCSER at IES to (1) sponsor
research to expand knowledge and understanding of the needs of infants,
toddlers, and children with disabilities in order to improve the
developmental, educational, and transitional results of such
individuals; (2) sponsor research to improve services provided under,
and support the implementation of, IDEA (20 U.S.C. 1400 et seq.); and
(3) to evaluate the implementation and effectiveness of IDEA in
coordination with the National Center for Education Evaluation and
Regional Assistance.

Since the enactment of the Education of the Handicapped Act (EHA) in
1975 to the most recent passage of the IDEA Improvement Act of 2004,
states have been required to monitor implementation of the law and the
provision of services to children with disabilities and their families.
In addition to ensuring compliance with Parts B and C of IDEA,
monitoring also serves as a means for ensuring that results for infants,
toddlers, children, and youth with disabilities are improving. Under
IDEA, state education agencies are responsible for ensuring compliance
with Part B of IDEA and providing general supervision of all programs
providing Part B services. For Part C, state lead agencies have parallel
responsibilities; that is, lead agencies must ensure that the law's
requirements are met and provide general supervision of early
intervention services provided to infants and toddlers and their
families.

The Department of Education's Office of Special Education Programs
(OSEP) supports states in carrying out IDEA and is responsible for
assessing the effectiveness of state and local efforts to provide
services under the Act and to improve results for children with
disabilities and their families. Together, states and OSEP are required
to ensure that the purposes of IDEA are being realized; as such, state
and OSEP monitoring and accountability efforts are necessarily
intertwined. Until this time, however, there has been no independent and
systematic examination of the differences in the design and
effectiveness of monitoring and improvement activities across the
states.

It should be noted that this evaluation was initiated by OSEP and moved
to IES's NCSER in July 2005. This was the result of Congress' moving the
authority for studies and evaluations of IDEA to IES in IDEA 2004.

[Objectives and Research Questions]{.underline}

This study has three objectives. The first objective is to describe the
nature and scope of monitoring as implemented by the 50 states and the
District of Columbia for Parts B and C of IDEA. This objective includes
assessment of the extent to which states have infused Continuous
Improvement Monitoring Process (CIMP) principles and Continuous
Improvement Focused Monitoring System (CIFMS) design features into their
monitoring practices. The research questions raised under this objective
include the following:

-   How are the monitoring targets (focus areas) selected?

-   Are the methods and/or criteria for selecting monitoring targets
    appropriate?

-   How are the monitoring evaluations designed and implemented?

-   How are data being collected?

-   How are the data analyzed, aggregated, interpreted, and used?

-   Is the monitoring evaluation design sound, implemented properly, and
    useful to stakeholder audiences?

-   Are the data collection methods sound?

-   Are the data relevant and of high quality?

-   Are outcome data being adequately and accurately analyzed,
    aggregated, and interpreted?

-   To what extent are the results of monitoring available to the public
    and especially stakeholders?

The second objective of this evaluation is to assess the effect of the
quality of states' monitoring and related improvement practices on key
outcomes of Parts B and C of IDEA. Westat will examine the impact of
these practices on states' ability to identify and correct noncompliance
with Parts B and C of IDEA and to improve outcomes for infants,
toddlers, children, and youth with disabilities.

The third objective is to identify and develop recommendations for
states relative to potential best practices in monitoring and identify
areas OSEP might pursue for ongoing technical assistance. Using findings
associated with objectives 1 and 2, Westat will draw conclusions and
formulate a series of lessons learned. These lessons learned will be
translated into recommendations for best practices.

[Data Collection Summary]{.underline}

Westat will conduct two primary data collection activities with the
purpose of building a comprehensive understanding of monitoring and
related improvement practices in all 50 states and the District of
Columbia and the effectiveness of these practices. The first activity is
a mail survey of the 50 states and the District of Columbia of current
monitoring practices and changes in those practices since the 1997
amendments to IDEA. The second activity involves in-depth site visits to
a systematic random sample of 20 states to further define their current
monitoring systems and assess the effectiveness of their systems.

1.  *Mail Survey.* Questionnaires were sent to Part B and Part C
    administrators in fall of 2005. The primary purpose of the
    questionnaires is to collect information about states' monitoring
    practices. The questionnaire results will inform the study in
    several ways. Foremost, the questionnaires will supply information
    that will allow us to construct a preliminary picture of current
    monitoring systems in all states, their varied approaches to data
    collection, and the focus of their practices. This information will
    have value in its own right, but it will also contribute to
    important contextual information needed for the subsequent site
    visits that will be completed the following spring. The
    questionnaires address the following areas:

-   The scope of monitoring and related improvement practices;

-   The design of monitoring and evaluation activities;

-   Staffing and training for monitoring;

-   Data collection and analysis activities;

-   Reporting practices (of monitoring findings);

-   Processes for correcting noncompliance and improvement planning;

-   Involvement of local entities in monitoring activities;

-   The role of stakeholders in monitoring and improvement activities;

-   Sources of technical assistance; and

-   Changes to monitoring since the 1997 amendments to IDEA.

2.  *Site Visits.* The second data collection activity will involve site
    visits to 20 states. The primary purposes of the site visits are to
    gather more detailed information that will allow us to construct a
    complete picture of the current monitoring systems that exist
    throughout the country and to assess the quality of those various
    monitoring systems and effects on key outcomes of IDEA. Our sampling
    strategy for selecting the 20 states involves arranging states by
    characteristics that we believe affect variability in monitoring and
    improvement practices, including state size (as measured by combined
    special education and early intervention counts of children served),
    Part C lead agency, and region of the country and then using
    systematic random sampling to select states.

> Each selected state will be visited twice; once in the spring of 2006,
> then again in the spring of 2008. The focus of the site visits will be
> the Part B and Part C monitoring systems in place during the year
> previous to the site visit (2004-05 and 2006-07). Data on both Part B
> and Part C monitoring and improvement activities will be collected in
> each of the 20 states in each of the two data collection years. Westat
> will conduct interviews with state and local monitoring staff and
> review relevant documentation (including memoranda, meeting minutes,
> and reports). Aggregated child/family outcome data will be collected
> to the extent they are available, can be efficiently extracted within
> budget constraints, and will contribute to a better understanding of
> the impact of monitoring.
>
> In preparation for the site visits, Westat has designed
> semi-structured interview protocols to elicit details about monitoring
> and related improvement practices. The interview protocols include
> questions about monitoring resources, activities, and products. The
> questions will be mainly open-ended rather than using predetermined
> response codes, as we expect wide variation in responses and want to
> capture that variation in this data collection. We will interview
> state Part B and C staff involved in monitoring and related
> improvement planning, state management information system personnel,
> members of the state's special education steering committee or other
> stakeholder groups, and other individuals with relevant information
> about the way in which the state's monitoring system was planned,
> implemented, and evaluated in the preceding year. All interviews will
> be audio taped and transcribed.
>
> We will also identify and review materials that will augment our
> understanding of state monitoring and related improvement practices
> and factors that influence states' success. Such materials may
> include, for example, OSEP monitoring reports, General Supervision
> Enhancement Grant reports, state personnel development standards,
> program standards, written descriptions of state dispute resolution
> mechanisms, and written descriptions of technical assistance and
> training activities. The guidelines for reviewing site visit materials
> will be based on the interview protocols. These guidelines will follow
> the same patterns of inquiry, directing reviewers to record
> information on the same topics as the interview data.
>
> Prior to conducting the site visits, Westat staff (as well as our
> subcontractors) will participate in a 2‑day training session at our
> Rockville, MD offices. The training will be designed to enhance the
> reliability and validity of the data collection by describing the
> purposes of the site visits, reviewing the interview protocol,
> discussing 'what if' scenarios, outlining the responsibilities of the
> site visitors, addressing logistical arrangements, and describing the
> standards and criteria that will be used to rate the quality of the
> various monitoring systems.

At this time, OMB clearance is being sought for the in-depth site visit
data collection activity. From this point forward, we focus on the site
visit data collection and mention the mail survey data collection only
as it directly relates to the site visit instruments.

2.  **Use of Information**

The information collected from the site visits will be used by OSEP's
Monitoring and State Improvement Planning (MSIP) division. Along with
the mail survey, the site visits will provide OSEP with an in-depth
understanding of states' monitoring of Parts B and C of IDEA. This
information will allow OSEP to better assist states in their monitoring
efforts, leading to greater accountability and improved results for
children with disabilities and their families.

In addition, it is anticipated that the information from the site visits
will be of great value to state education agencies responsible for
monitoring Part B of IDEA and state agencies responsible for monitoring
Part C of IDEA, as well as to the Council of Chief State School Officers
(CCSSO). Having information about effective monitoring practices will
aid states in identifying potential strengths and weaknesses in their
own efforts, leading to more effective monitoring systems and improved
results.

3.  **Method of Collection**

The site visit data collection will be accomplished through
pencil-and-paper interview protocols that Westat staff will use to guide
discussions with state and local monitoring staff and reviews of
relevant documentation. All interviews will be audio taped and
transcribed. We are not using other electronic or technological
collection techniques (e.g., electronic surveys) however, as the design
and use of these techniques would be inefficient given the overall small
number of respondents and the need to tailor the interview probes for
this data collection.

4.  **Avoidance of Duplication**

No in-depth data currently exist on states' monitoring and improvement
practices under Parts B and C of IDEA. Although individual states may
publicly report on their monitoring efforts, these data are too diverse
in content and quality to be comparable and are an inappropriate base
from which to extrapolate to the nation as a whole.

5.  **Small Business Impact**

No small businesses will be involved as respondents in this data
collection. Therefore, there will be no small business impacts.

6.  **Consequences of Not Collecting Information**

The Department of Education and OSEP, in particular, need information on
state monitoring and improvement practices for several reasons.
Monitoring is crucial to ensuring compliance with IDEA and to improving
results for children with disabilities and their families. Without the
information to be collected by this study, OSEP will not have the data
needed to better understand the nature and scope of state monitoring
strategies and effectiveness. OSEP also will not know the impact of
these practices on states' ability to identify and correct noncompliance
with Parts B and C of IDEA and to improve outcomes for infants,
toddlers, children, and youth with disabilities. Information about
states' monitoring and improvement activities is necessary so that OSEP
can assist states in their monitoring efforts, which will lead to
greater accountability and improved results for children with
disabilities and their families. This is particularly important in light
of the requirement of Section 616 of IDEA 2004 that states implement
focused monitoring.

7.  **Special Circumstances**

The proposed data collection is consistent with 5CFR 1320.6 and
therefore involves no special circumstances.

8.  **Consultation Outside the Agency**

Westat established an Advisory Panel to provide input into the sampling,
data collection, and analysis plans; data collection instruments; and
the content and format of the interim and final evaluation reports.
Panel members have expertise in program and/or education evaluation;
state monitoring practices; parent advocacy; state Part C and B
administration; technical assistance; special education law,
regulations, and policy; and early intervention and preschool special
education. The Panel also includes a representative from the National
Center for Special Education Accountability and Monitoring (NCSEAM).
Members of the Advisory Panel are listed in Exhibit 1.

Advisory Panel members reviewed drafts of the standards and criteria
(from which the site visit instruments are based) and provided written
feedback as well as feedback via conference calls in May of 2005. Panel
members also reviewed the site visit instruments in September of 2005
and provided written feedback.

Exhibit 1. Advisory Panel Members

  ------------------ --------------------------------- -------------------
  **Name**           **Affiliation**                   **Advisory panel
                                                       role**

  Linda Bradford     Independent Consultant            Former 619
                                                       Coordinator

  Barbara Buswell    PEAK Parent Center                Parent/Advocate

  Alan Coulter       National Center for Special       NCSEAM Director
                     Education Accountability          
                     Monitoring (NCSEAM)               

  Maureen Greer      Emerald Consulting                Former Part C State
                                                       Administrator

  Joan Liversidge    Independent Consultant            Former Local Part C
                                                       Program Director

  John McLaughlin    Independent Consultant            Program Evaluation

  Margaret           University of Maryland College    Special Education
  McLaughlin         Park\                             Policy
                     EDUC-Special Education            

  Philip Pickens     Southeast Regional Resource       Former Part B State
                     Center (SERRC)                    Administrator

  Kristin Reedy      Northeast Regional Resource       Technical
                     Center (NERRC); Learning          Assistance Provider
                     Innovations/WestEd                

  Madeleine Will     National Down Syndrome Society    Parent/Advocate
                     (NDSS) National Policy Center     
  ------------------ --------------------------------- -------------------

9.  **Reimbursement of Respondents**

Payment or gifts will not be provided to states for participating in the
site visit data collection activity.

10. **Assurances of Confidentiality**

Letters will be sent to Part B state directors and Part C coordinators
in the 20 selected states to notify them of the site visit data
collection activity. This letter will give assurances that no individual
state, agency, or staff member will be identified. Similar assurances
will also be made during the site visits. All data from the site visits
will be maintained and stored under secure conditions and will be
accessible only to authorized staff members. In reports prepared by
Westat for IES and in all public reports, findings will be presented in
aggregate where appropriate and individual states, and respondents will
not be identified.

11. **Sensitive Items**

There are no questions of a sensitive nature included in the site visit
data collection instruments.

12. **Estimates of Burden**

Estimates of the hour burden are reported below in Exhibit 2. These
estimates are based on two pilot site visits that were conducted in
August 2005. Two Westat staff visited the Part B program in one state,
while two additional staff visited the Part C program in another state.
The pilot site visits were conducted similar to a financial audit. That
is, Westat staff first met with the Part B director of special education
or the Part C coordinator to get a brief overview of the state's
monitoring practices. The remainder of the 2-day site visit was spent
reviewing records, reports, and other supporting documentation, and
talking with other state monitoring staff, as appropriate. We also
conducted telephone interviews with a few local program staff. As
information was collected, Westat staff used the interview protocols to
rate each state on the respective criteria (designed to assess quality).
For both the site visit and the interviews, questions were asked of
fewer than nine individuals.

We anticipate that for the actual site visits, we will conduct a larger
number of telephone interviews with local program staff and that we will
also interview various stakeholders. Furthermore, we expect that we will
need to follow up with states after the site visit to clarify any
remaining issues. The estimated burden hours below include time for
states to gather documentation for site visitors to review; interviews
with the Part B state director or Part C coordinator, monitoring staff,
local program staff, and stakeholders; and follow-up interviews after
the visit.

The estimated burden hours for Part B are slightly larger than for Part
C because we feel that since Part B serves a larger number of children
with disabilities, we will need to conduct a few more interviews during
the Part B site visits than the Part C site visits. It should also be
noted that the hours per response is an average. We expect the length of
the interviews to vary, as some interviews will likely last longer than
others (e.g., the interviews with monitoring staff versus the interviews
with stakeholders, etc.)

Exhibit 2. Estimated Burden

  ----------------- ----------- ------------- ------------ ----------- -----------
  **Respondents**   **Number of **Number of   **Hours per  **Number of **Total
                    states**    respondents   response**   site        burden
                                per state**                visits**    hours**

  Part B            20          12            2.5          2           1,200

  Part C            20          10            2.5          2           1,000

  **Total**                                                            **2,200**
  ----------------- ----------- ------------- ------------ ----------- -----------

The estimated total cost to the respondents for the burden hours are
reported in Exhibit 3 below. The estimated hourly wages are based on
wage reports from the Bureau of Labor Statistics.

Exhibit 3. Estimated Cost to Respondents

  ----------------------- --------------- --------------- -----------------
  **Respondents**         **Total burden  **Estimated     **Cost**
                          hours**         hourly wages**  

  Part B Monitoring Staff 1,200           \$39.00         \$46,800.00

  Part C Monitoring Staff 1,000           \$39.00         \$39,000.00

  **Total**               **2,200**                       **\$85,800.00**
  ----------------------- --------------- --------------- -----------------

13. **Estimated Annual Cost Burden to Respondents**

There are no costs to respondents beyond those shown in item 12 above.

14. **Estimated Annual Burden to Federal Government**

The estimated total cost for the site visit data collection is
\$2,776,728. The estimated cost for Year 1 is \$208,908, including costs
for developing the interview protocol, pilot testing, and preparing the
OMB package. The estimated cost for Year 2 is \$494,098, including costs
for selecting and notifying states of site visits, and conducting the
first wave of site visits. The estimated cost for Year 3 is \$573,517,
including costs for obtaining OMB clearance (if necessary), data
analysis, and reporting. The estimated cost for Year 4 is \$770,549,
including costs for conducting the second wave of site visits, data
analysis, and reporting. The estimated cost for Year 5 is \$729,656,
including costs for data analysis and reporting.

15. **Program Changes in Burden/Cost Estimates**

The program change of burden hours to 896 is an adjustment for the
second of two data collections (this one) together with the first
package that expires 9/30/06 .

16. **Plans/Schedules for Tabulation and Publication**

Data from the site visits will be used to describe the various
monitoring and improvement systems and key monitoring practices, and to
assess the quality of those monitoring systems and key practices. The
information will be analyzed in a three-step process to answer the
evaluation questions.

In the first step, *Construct Monitoring System Designs*, Westat will
analyze each state's effort within a common framework to describe
states' monitoring systems. This step will also facilitate a better
understanding of the components (and relationships between them) of each
state's approach and activities and facilitate the collection of better
information from each state during the second site visit.

The second step, *Assess Quality of the Monitoring System Designs*, will
involve an appraisal of the quality of the various components of the
design. Using evaluation standards and criteria that we have developed,
each component of the monitoring design will be rated on the extent to
which it meets each criterion. Then, individual ratings will be combined
into a comprehensive, and largely quantitative, measure of the quality
of the monitoring design for each state, Part B or C of IDEA, and year.
These data will then be combined in different ways to analyze the
evaluation questions.

The final step, *Explore Monitoring System Effectiveness*, will involve
using available child data and other monitoring results to examine
relationships between monitoring system quality and changes in
compliance and outcomes. Our plans for data analysis and dissemination
of findings are summarized in more detail below. Exhibit 4 presents the
timeline for the site visit data collection activities.

Exhibit 4. Timeline for Site Visit Data Collection Activities

  ------------------------------------------------ ----------------------
  **Data collection activity**                     **Date to be
                                                   completed**

  Pilot interview protocol                         August 15, 2005

  Notify states about first in-depth site visit    March 28, 2006

  Collect site visit data for year 1               June 30, 2006

  Construct monitoring system designs for year 1   August 31, 2006

  Construct quality scores for monitoring systems  August 31, 2006
  for year 1                                       

  Notify states about second in-depth site visit   February 28, 2008

  Collect site visit data for year 2               June 30, 2008

  Revise monitoring system designs based on year 2 August 31, 2008
  data                                             

  Construct quality scores for monitoring systems  August 31, 2008
  for year 2                                       

  Analyze monitoring system effectiveness using    October 31, 2008
  year 1 and 2 data                                
  ------------------------------------------------ ----------------------

[Data Analysis]{.underline}

*Construct Monitoring System Designs.* Using reports, memoranda,
interview data, and other information from the site visits, we will
construct a Monitoring System Design for Part B and another for Part C
for each of the 20 selected states. The designs will be organized around
the general model for monitoring that we developed in consultation with
our Advisory Panel and key OSEP staff. The model will facilitate
sorting, synthesizing, and organizing of the collected information.
Using the general model of monitoring will also assist in identifying
"holes" in the data collected. For example, when the site visit
information sorting and synthesizing are completed, we may find that
little information was obtained on the improvement planning function.
Such findings may require following up with the state through emails,
telephone interviews, etc. to obtain the needed information.

Each Monitoring System Design construction activity will be conducted by
the two staff members who collected the information during the site
visits. They will compile the information into the general model for
monitoring by (1) synthesizing and sorting the information into the
respective monitoring components, and (2) creating supporting paragraphs
that describe each item. In addition to elaborating on the basic
components described in the monitoring model, we are also interested in
external inputs to the system. A key external input is the state context
for the state's monitoring system (e.g., whether a consent decree, court
ruling, funding change, etc. has affected monitoring in that state). The
resulting product will be a report describing each state's Part B and C
Monitoring System Design.

*Assess Quality of Monitoring System Designs*. In order to assess the
quality of Monitoring System Designs, we developed a set of Monitoring
Standards and Criteria that includes sets of quality standards for each
component in our general model for monitoring. In developing the
standards and criteria, we used several sources of information. These
sources generally included the Joint Committee Standards on Educational
Evaluations (which includes sets of standards for utility, feasibility,
propriety, and accuracy); principles from OSEP's Continuous Improvement
Monitoring Process (CIMP); the Continuous Improvement Focused Monitoring
System (CIFMS) checklist developed by NCSEAM, and input from our
Advisory Panel Members. Data from the pilot site visits were used to
refine the list of standards and criteria.

The rating criteria for each standard uses a checklist of binary items
(e.g., yes exists/no does not exist) or a three-point scale (e.g., 0 --
does not exist, 1 -- exists somewhat, 2 -- exists). Through the rating
process that will be completed during the in-depth site visits, we will
generate data for all applicable rating items for all 20 sampled states
(from each of the two site visits). The site visits will be 2 years
apart, with the first site visit occurring in the spring of 2006 and the
second site visit occurring in the spring of 2008. We will convert the
data from the rating process into ordinal assessment scales. The
structure of input data for assessing the quality of state monitoring
systems can be represented with indexes for a sampled state, the year of
data collection, the evaluation component, the standard, and the
criterion, as shown in Exhibit 5. Westat will need, therefore, a total
of 20 vectors of binary components, one per standard, of length 10 or
less, to represent 1 year of data for each state.

Exhibit 5. Binary Inputs for Assessing the Quality of Monitoring
Evaluations

+------+--------+------------------+--------+------------------------+
| Sam  | Year   | Evaluation       | Appl   | Applicable binary      |
| pled | of     |                  | icable | assessment criteria    |
| s    | data   | components (c)   | sta    | (BAC)                  |
| tate | coll   |                  | ndards |                        |
| (st) | ection |                  | (s)    |                        |
|      | (y)    |                  |        |                        |
+------+--------+------------------+--------+------------------------+
| 1-20 | > 1-2  | > Indicator and  | >      | > BAC(st, y, c, s, j,  |
|      |        | > Target         |  A1-A4 | > nj), 1 ≤ j ≤ 4; nj ≤ |
|      |        | > Setting, (1)   |        | > 10                   |
+------+--------+------------------+--------+------------------------+
|      |        | > Indicator Data | >      | > BAC(st, y, c, s, j,  |
|      |        | > Collection and |  B1-B6 | > nj), 1 ≤ j ≤ 6; nj ≤ |
|      |        | > Analysis, (2)  |        | > 10                   |
+------+--------+------------------+--------+------------------------+
|      |        | > Problem        | >      | > BAC(st, y, c, s, j,  |
|      |        | >                |  C1-C3 | > nj), 1 ≤ j ≤ 3; nj ≤ |
|      |        |  Identification, |        | > 10                   |
|      |        | > (3)            |        |                        |
+------+--------+------------------+--------+------------------------+
|      |        | > Problem        | >      | > BAC(st, y, c, s, j,  |
|      |        | > Investigation, |  D1-D8 | > nj), 1 ≤ j ≤ 8; nj ≤ |
|      |        | > (4)            |        | > 10                   |
+------+--------+------------------+--------+------------------------+
|      |        | > Corrective     | >      | > BAC (st, y, c, s,    |
|      |        | > Action and     |  E1-E3 | > j,nj), 1 \< j \< 3;  |
|      |        | > Enforcement,   |        | > nj \< 10             |
|      |        | > (5)            |        |                        |
+------+--------+------------------+--------+------------------------+
|      |        | > Improvement    | >      | > BAC (st, y, c, s, j, |
|      |        | > Planning, (6)  |  F1-F4 | > nj), 1 \< j \< 4; nj |
|      |        |                  |        | > \< 10                |
+------+--------+------------------+--------+------------------------+
|      |        | > Improvement    | >      | > BAC (st, y, c, s, j, |
|      |        | > Planning and   |  G1-G2 | > nj), 1 \< j \< 2; nj |
|      |        | >                |        | > \< 10                |
|      |        |  Implementation, |        |                        |
|      |        | > (7)            |        |                        |
+------+--------+------------------+--------+------------------------+
|      |        | > Reassessment,  | >      | > BAC (st, y, c, s, j, |
|      |        | > (8)            |  H1-H3 | > nj), 1 \< j \< 3; nj |
|      |        |                  |        | > \< 10                |
+------+--------+------------------+--------+------------------------+

To compare state monitoring system designs within and between the two
site visits (hereafter referred to as waves of data collection), Westat
first proposes to synthesize the low-level binary assessment criteria
(BAC) into a more manageable format. To achieve this objective, we will
apply a systematic process comprising three conceptual steps. First, we
will develop a conceptual model for combining all BAC associated with a
standard into numeric scales, [c]{.underline}omposite
[a]{.underline}ssessment [c]{.underline}riteria, indexed by state, year
of data collection, component and standard, CAC~(st, y, c, s)~. Second,
we will develop a conceptual model for combining the composite
assessment criteria (CAC) associated with standards, into
[c]{.underline}omposite [s]{.underline}tandard [s]{.underline}cales,
indexed by state, year of data collection, and component,
CSS~(s,\ y,\ c)~. And third, we will develop a conceptual model for
combining composite standard scores (CSSs) associated with the four
dimensions into a composite scale, by state and year of data collection,
CSS~(s,\ y).~

It would be economical to assume that all composite assessment criteria,
all composite standard scales, and all composite state-year of data
collection scales will align into one single ordering so states ranking
high on one scale also rank high on all other scales. Were this to
happen, states could be declared uniformly low, medium, or high in terms
of the quality of their state monitoring systems. In the absence of such
uniformity, some states might rank high on some scales and low on
others.

While total uniformity across the composite assessment scores is
unlikely, it is probable that the ranking of states using different
scales will be somewhat related, at least for selected groups of scales.
Westat will conduct an exploratory analysis of the scales derived above.
Ideally, this analysis will result in a few largely independent scales
that can be used to mathematically represent the most important ordering
relationships among the binary assessment criteria.

The primary statistical tool Westat proposes is variable clustering. We
will use algorithms appropriate for binary data to: (1) explore and
summarize the association structure among the scales using Wave 1 data
(first site visit); (2) identify and characterize a small number of
largely independent scales that can be used to reconstruct the rest of
the scales, subject to chance variation; (3) redo activities 1 and 2 for
Wave 2 data (second site visit) and for "change" scales obtained by
calculating the difference between Wave 2 and Wave 1 scale values; and
(4) develop a parsimonious interpretation for the scales obtained in the
previous three activities.

Assuming that more than one but still a reasonably small number of
scales are constructed as described above (and this is the likely
scenario in our judgment), Westat will use these scales to identify
clusters of states that are similar to one another in the quality of
their monitoring system. To do so, we propose to use multi-dimensional
scaling. Multi-dimensional scaling is a method for mapping objects into
2- or 3-dimensional planes to easily visualize their differences and
similarities. If successful, the similarity between states in the
quality of their state monitoring systems becomes visible in the form of
spatial proximity. If necessary, alternative clustering techniques will
also be used. To obtain meaningful clusters, we will first use
multi-dimensional scaling and/or clustering techniques to identify two
or more state clusters such that within-cluster variation is small
compared to between-cluster variation. Second, we will perform the
previous step using Wave 1, Wave 2, and between-wave change data and
select clusters for various comparisons. Finally, we will compare state
clusters defined in terms of Wave 1 scales, Wave 2 scales, and scales
for the difference between Wave 1 and Wave 2 and interpret the results.

Westat expects that the results from these analyses will be extremely
useful in identifying the key aspects of state monitoring systems both
in terms of the independent 'drivers' of monitoring and in terms of
between-state and between-wave similarities and differences.

*Explore Monitoring System Effectiveness.* Exploration of monitoring
system design effectiveness will include a mix of qualitative and
quantitative analyses. The qualitative analyses will involve the use of
a diverse set of analytic tools borrowed from case study methodology,
including triangulation, rival explanations, reproducibility of
findings, and the OTTR ("observe, think, test, and revise") strategy.
The quantitative analyses will involve use of analysis of variance
(ANOVA) models to determine similarities and differences between the
characteristics of states' monitoring system designs and to determine
(when data are available) whether meaningful changes occur from Wave 1
to Wave 2 or across states in noncompliance and child results.

The qualitative analyses will involve three processes: data reduction,
data display, and drawing conclusions. In the data reduction process, we
will extract from our interviews, documents, and site visits the
information most relevant to the evaluation. Data displays or models of
state monitoring systems will facilitate the process of monitoring
system design explanations. The third phase involves drawing meaning
from the displays and revisiting the data to verify that theories hold
up.

Our qualitative analysis will involve both within-state and cross-state
analyses. Within-state analyses will involve further data reduction
(following the building of monitoring system designs), data displays,
and drawing conclusions. We will use triangulation (comparing
information from multiple and independent sources) and rival
explanations (testing the hypotheses developed from the mail survey and
during the monitoring system design process).

We will also attempt to identify patterns across states and compare and
contrast states and years. The challenge of cross-case analysis is
balancing the site visit data with the 30 non-visited states. That is,
we must maintain the individuality of each state's design while trying
to understand general processes that drive state monitoring and related
improvement practices. In the cross-case analysis, we will try to
identify types of state monitoring systems. Cases in a set will be
examined to determine if they share certain characteristics. Data
reduction, data display, and drawing conclusions will also be used for
cross-state analyses. As we work through the various state monitoring
designs, we will use the OTTR strategy to draw conclusions about
effective monitoring designs.

Our quantitative analysis will involve exploration of the statistical
association between the quality of state monitoring systems and
compliance and child outcome data, when such data are available. The
more readily available outcome measures for Part B include, for example,
graduation rates and the proportion of students served in various
educational environments; Part C outcomes may include identification
rates and the percentage of infants and toddlers served in various
settings. To the extent data are available, Westat will also obtain
variance estimates for these measures, i.e., for measuring variability
across all local education agencies (LEAs) or Part C programs within
states. We believe that effective monitoring by states will affect the
low end of the distribution for these measures more strongly than it
will affect their averages. All key outcome measures will be estimated
and assessed by state and year of data collection, in relation to state
clusters that are homogeneous on the quality of the state monitoring
system.

Given the uncertainties about the number of state clusters and the
nature of differences among them, it is premature to attempt to fully
specify how best to analyze the effects of between-state differences and
between-year change in the quality of state monitoring systems on key
outcomes and their variances. Our simplest initial approach to
estimating these effects will be formulated in terms of ANOVA models.
States will be the unit of analysis. Outcomes and their variances will
be the dependent variables, and cluster ID will be the main effects. If
the state clusters are not sufficiently homogeneous, ANOVA may be
replaced with nested analysis of co-variance (ANCOVA), with clusters
becoming the nesting factor and state-level characteristics of state
monitoring quality used as covariates. In alternative formulations,
repeated measures models may be used to analyze between-state changes or
the analyses may be recast in terms of between-wave state-level
differences, both in covariates and outcomes. Depending on the
statistical properties of the data, robust methods may be used to
estimate model parameters. Sample design will be accounted for in
generating national estimates as appropriate using our proprietary
software, WESVAR, or alternative packages.

[Dissemination of Findings]{.underline}

We will prepare the following types of reports: Annual Interim Reports
(Years 2-4 of the Westat contract), a Final Report, general-use
presentation materials, brief project summaries, and summaries for the
Annual Report to Congress. We will also present on the findings at
various meetings held by IES and OSEP.

17. **Expiration Date Omission Approval**

Not applicable.

18. **Exceptions**

No exceptions are taken.

**B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS**

Our sampling strategy for selecting the 20 states involves arranging
states by characteristics that we believe affect variability in
monitoring and improvement practices, including state size (as measured
by combined special education and early intervention counts of children
served), Part C lead agency, and region of the country, and then using
systematic random sampling to select states. The sections below
summarize our sampling procedures in more detail.

1.  **Sampling Methods**

The purpose of the sampling plan is to select a sample of states that
will provide an adequate basis for assessing monitoring and related
improvement practices currently used in states across the United States.
This will enable us to obtain an accurate picture of the quality of
monitoring and to assess the effects of monitoring on key outcomes of
Parts B and C of IDEA. As discussed in item 2, a sample of 20 states
should be large enough sample to capture the existing variability in
states' monitoring and related improvement practices.

Our sampling strategy for selecting the 20 states entails arranging the
respondent universe of 50 states and the District of Columbia by
characteristics that affect variability in monitoring practices and then
using systematic random sampling to select states. We conferred with our
Advisory Panel, and together we decided state size, Part C lead agency,
and regional differences among states were the key variables that should
be considered in the sampling process. We are using a combined special
education and early intervention child count as our measure of state
size. For the Part C lead agency variable, we decided to use three
categories: education, health, and other.

State approaches to monitoring might well be affected by the region in
which the state is located and the practices common in that region. For
example, even though Delaware and Montana are similar in terms of the
total number of children served, differences due to their geographic
locations may result in different approaches to monitoring and related
improvement practices. For this reason, we decided to use the four
Census Bureau regions to group states according to whether they are
located in the West, South, North, or Midwest.

We will arrange states according to these variables and first order
states according to the size of their combined special education and
early intervention counts of children served. We will divide them into
quartiles and then, within quartile, sort the states by Part C lead
agency and then by region. The final ordering of states is presented in
Exhibit 6. In the table, combined child count quartiles are noted by
numbers (4 being the largest and 1 the smallest), Part C lead agencies
are noted by letters ([O]{.underline}ther, [H]{.underline}ealth, and
[E]{.underline}ducation), and regions also by letters
([W]{.underline}est, [S]{.underline}outh, [N]{.underline}orth,
[M]{.underline}idwest). Our final step will be to use systematic random
sampling to select our 20 states.

We expect a response rate of 100% from our sample of 20 states (see item
3).

Exhibit 6. States by Combined Special Education and Early Intervention
Child Count, Part C Lead Agency, and Region

+-----------------+-----------------+-----------------+-----------------+
| **State**       | **Child Count   | **Part C Lead   | **Region**      |
|                 | (quartile)**    | Agency**        |                 |
+-----------------+-----------------+-----------------+-----------------+
| California      | 4               | O               | W               |
|                 |                 |                 |                 |
| Texas           | 4               | O               | S               |
|                 |                 |                 |                 |
| Pennsylvania    | 4               | O               | N               |
|                 |                 |                 |                 |
| Illinois        | 4               | O               | M               |
|                 |                 |                 |                 |
| Indiana         | 4               | O               | M               |
|                 |                 |                 |                 |
| Florida         | 4               | H               | S               |
|                 |                 |                 |                 |
| Georgia         | 4               | H               | S               |
|                 |                 |                 |                 |
| North Carolina  | 4               | H               | S               |
|                 |                 |                 |                 |
| New Jersey      | 4               | H               | N               |
|                 |                 |                 |                 |
| New York        | 4               | H               | N               |
|                 |                 |                 |                 |
| Ohio            | 4               | H               | M               |
|                 |                 |                 |                 |
| Michigan        | 4               | E               | M               |
+-----------------+-----------------+-----------------+-----------------+
| Arizona         | 3               | O               | W               |
|                 |                 |                 |                 |
| Alabama         | 3               | O               | S               |
|                 |                 |                 |                 |
| Virginia        | 3               | O               | S               |
|                 |                 |                 |                 |
| Washington      | 3               | H               | W               |
|                 |                 |                 |                 |
| Kentucky        | 3               | H               | S               |
|                 |                 |                 |                 |
| Louisiana       | 3               | H               | S               |
|                 |                 |                 |                 |
| South Carolina  | 3               | H               | S               |
|                 |                 |                 |                 |
| Massachusetts   | 3               | H               | N               |
|                 |                 |                 |                 |
| Wisconsin       | 3               | H               | M               |
|                 |                 |                 |                 |
| Maryland        | 3               | E               | S               |
|                 |                 |                 |                 |
| Tennessee       | 3               | E               | S               |
|                 |                 |                 |                 |
| Minnesota       | 3               | E               | M               |
|                 |                 |                 |                 |
| Missouri        | 3               | E               | M               |
+-----------------+-----------------+-----------------+-----------------+
| Arkansas        | 2               | O               | S               |
|                 |                 |                 |                 |
| Connecticut     | 2               | O               | N               |
|                 |                 |                 |                 |
| Nevada          | 2               | H               | W               |
|                 |                 |                 |                 |
| New Mexico      | 2               | H               | W               |
|                 |                 |                 |                 |
| Utah            | 2               | H               | W               |
|                 |                 |                 |                 |
| Mississippi     | 2               | H               | S               |
|                 |                 |                 |                 |
| West Virginia   | 2               | H               | S               |
|                 |                 |                 |                 |
| Kansas          | 2               | H               | M               |
|                 |                 |                 |                 |
| Colorado        | 2               | E               | W               |
|                 |                 |                 |                 |
| Oregon          | 2               | E               | W               |
|                 |                 |                 |                 |
| Oklahoma        | 2               | E               | S               |
|                 |                 |                 |                 |
| Iowa            | 2               | E               | M               |
|                 |                 |                 |                 |
| Nebraska        | 2               | E               | M               |
+-----------------+-----------------+-----------------+-----------------+
| District of     | 1               | O               | S               |
| Columbia        |                 |                 |                 |
|                 | 1               | O               | M               |
| North Dakota    |                 |                 |                 |
|                 | 1               | H               | W               |
| Alaska          |                 |                 |                 |
|                 | 1               | H               | W               |
| Hawaii          |                 |                 |                 |
|                 | 1               | H               | W               |
| Idaho           |                 |                 |                 |
|                 | 1               | H               | W               |
| Montana         |                 |                 |                 |
|                 | 1               | H               | W               |
| Wyoming         |                 |                 |                 |
|                 | 1               | H               | S               |
| Delaware        |                 |                 |                 |
|                 | 1               | H               | N               |
| New Hampshire   |                 |                 |                 |
|                 | 1               | H               | N               |
| Rhode Island    |                 |                 |                 |
|                 | 1               | E               | N               |
| Maine           |                 |                 |                 |
|                 | 1               | E               | N               |
| Vermont         |                 |                 |                 |
|                 | 1               | E               | M               |
| South Dakota    |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+

2.  **Information Collection Procedures**

As discussed above in item 1, our sampling strategy for selecting the 20
states entails arranging the respondent universe of 50 states and the
District of Columbia by three key variables (state size, Part C lead
agency, and region) that may affect variability in monitoring practices
and then using systematic random sampling to select states. Systematic
random sampling uses a mathematical formula to determine the interval at
which to sample and a random starting point. For example, depending upon
the number of units and the size of the sample, the sampling interval
might be to select every fifth unit. We believe this methodology will
capture the possible variation on our three variables and provide us
with a representative sample of 20 states.

A sample of 20 states represents nearly 40 percent of state-level
jurisdictions (including the District of Columbia). The sampling
algorithm entails selecting every second or third state in the sorted
list of entities. From the point of view of assessing quality of
monitoring, the sample of 20 states will provide a fairly complete
picture of practices occurring across the 51 states. Any procedures that
are used by six or more states have about 96 percent chance of being
represented in our sample; procedures that are used by nine or more
states have about 99 percent chance of being represented in our sample.
For statistical estimates, sampling 40 percent of the population has the
effect of reducing sampling error of reported estimates by nearly 25
percent. While confidence intervals for proportions will still be
somewhat large with 20 states (e.g., ± 0.17 for an estimated proportion
of 0.50), these intervals would only be substantially reduced by taking
a substantially larger sample size, such as 30 states or more, which
would not be feasible in terms of budget or schedule.

3.  **Maximizing Response Rates**

We expect a response rate of 100% from our sample of 20 states. As
specified in EDGAR 76.591, states have a responsibility to cooperate
with evaluations conducted by the Secretary, such as this one, that
examine federally funded, state-administered programs.

4.  **Testing of Instruments**

Westat conducted a pilot test of the site visit interview protocols
(including the rating criteria) in two states -- one for Part B and one
for Part C. Two Westat staff visited one state, while two additional
staff visited another state. The pilot site visits were conducted
similar to a financial audit. That is, Westat staff first met with the
Part B director of special education or Part C coordinator to get a
brief overview of the states' monitoring practices. The remainder of the
2-day site visit was spent reviewing records, reports, and other
supporting documentation, and talking to other state monitoring staff,
as appropriate. We also conducted telephone interviews with a few local
staff. As information was collected, Westat staff rated each state on
the respective criteria.

In general, our data collection and rating strategy worked quite well,
and the pilot visits were extremely informative. Initially, we thought
that we would be able to bring most of the documentation that was
reviewed back to our home office and rate the criteria after the site
visit. However, due to the nature of many of the documents and reports,
we learned that we

would need to review most of the documentation while on-site. Hence,
when conducting the 20 site visits for the main study, Westat staff will
need to be well-trained to ensure that data collection and the rating
process are reliable.

5.  **Individuals Consulted on Statistical Issues**

[Statistical Consultation:]{.underline}

Dr. Paul Zador

Senior Statistician and Study Design Methodologist

Westat

Phone: 301-294-2825

Dr. Jim Bethel

Senior Statistician

Westat

Phone: 301-294-2067

[Site Visit Data Collection and Analysis:]{.underline}

Dr. Marsha Brauen

Project Director

Westat

Phone: 301-738-3668

Dr. William Frey

Principal Investigator

Westat

Phone: 301-610-5198