#### STATEMENT OF WORK FOR A STUDY TO ASSESS FUNDING AND ACCOUNTABILITY SYSTEMS IN VOCATIONAL EDUCATION[^1]

**I. INTRODUCTION**

This work statement describes a 17-month evaluation, consisting of a
survey and case studies, to be conducted as part of a National
Assessment of Vocation Education (NAVE).[^2] The evaluation will assess
program management---funding and accountability---provisions mandated
under the Carl Perkins Vocational and Technical Education Act (Perkins
III), reauthorized in 1998.[^3]

The NAVE will undertake studies in several areas.[^4] The evaluation
described in this work statement is particularly important because new
program management requirements are among the most substantive changes
evident in Perkins III. Certain set aside funding streams were
eliminated to provide greater flexibility to state and local programs,
and a larger share of funds must now go directly to the local level. The
law also greatly increased the significance of program accountability as
a way to achieve federal policy objectives.

Given the NAVE timelines, this evaluation will be able to examine only
the short-term response to these new legislative provisions. However,
the prominence given to program management changes in Perkins suggests
that Congress is likely to be interested even in early evidence about
their consequences. Policymakers will be attentive to shifts in funding
priorities and allocations at both the state and local levels, including
any impacts on special populations historically given particular
consideration in Perkins legislation. Congress will also want to know if
the accountability system is being appropriately implemented, and if it
appears to help states and local education institutions manage their
vocational programs effectively.

The statement of work lays out the basic policy and research issues to
be addressed in this study of Perkins funding and accountability, and a
strategy for addressing these questions. Offerors are encouraged to
suggest alternative approaches that they believe are more effective or
efficient ways to achieve the objectives of this evaluation. Offerors
are also encouraged to form project teams or make use of consultants,
since expertise in both funding and accountability is needed to conduct
this study and the NAVE seeks new perspectives on these issues.

# II. FUNDING AND ACCOUNTABILITY LEGISLATIVE REQUIREMENTS

A. [Funding]{.underline}

In many respects, the basic framework for federal vocational funding
remains the same under Perkins III as in previous vocational
legislation. Each state receives a grant from the U.S. Department of
Education, based on population counts in several age categories. State
agencies then allocate much of their grant to local programs, with the
discretion to determine the share of Perkins funds allocated to
institutions at the secondary versus postsecondary levels. Funds
dedicated to secondary vocational education are distributed to secondary
districts based largely on their proportion of low-income youth; money
for postsecondary vocational education is distributed to community
colleges and other eligible postsecondary institutions based largely on
their share of Pell grant recipients.

However, the Congress that enacted Perkins III (1998) was driven by two
main objectives with respect to the use and allocation of Perkins
funding. First, Congress wanted a larger share of funding allocated to
local programs so that more resources could trickle down to the
classroom, with the potential to affect student outcomes directly.
Second, it wanted to give more flexibility to grantees regarding the use
of funds for program improvement activities. These preferences led
Congress to legislate a host of new funding provisions that have the
potential to affect both grantee allocations and how the money can be
used. Among the most significant changes made were:

-   Elimination of the set-aside funding streams for special populations
    (single parents, displaced homemakers, pregnant teenagers), a state
    gender equity coordinator,[^5] and requirements that local education
    agencies target funds to programs with the greatest concentration of
    special populations.

-   Increase in the share of funds for local programs (from 75 percent
    in Perkins II to 85 percent).

-   New local secondary funding formula, with less emphasis on the
    proportion of students with disabilities than previously[^6].

-   Introduction of a state reserve (up to 10 percent of the allocation
    to local programs) that may be distributed outside of the
    intra-state formula to fund programs in rural areas; areas with high
    percentages or numbers of vocational and technical education
    students; and communities negatively impacted by changes in the
    in-state secondary school formula.

B. [Accountability]{.underline}

On the surface, the general accountability requirements contained in
Perkins III are also similar to those in the earlier Perkins II (1990).
However, the new accountability provisions are a much more significant
instrument of federal policy than in the previous Act. The philosophical
shift in Perkins III toward greater flexibility in use of funds and away
from set-asides for special populations has raised congressional
expectation that states will develop meaningful systems of performance
measurement and accountability. Compared with Perkins II, Perkins III
raises the requirements for state reporting of student outcome data
(including mandating more measures of performance), and the potential
rewards and consequences for states that can and cannot do so.

Perkins III requires each state to develop a system of measurements and
to establish expected levels of performance in four categories[^7]:

1.  Student attainment of academic, vocational, and technical skill
    proficiencies.

2.  Completion of a secondary or postsecondary degree or credential.

3.  Placement and retention in postsecondary education, advanced
    training, employment or the military.

4.  Participation in and completion of programs that lead to
    nontraditional employment.

In addition, each state is required to report on the progress of special
populations with respect to each of these categories.

However, the major changes in Perkins III accountability have less to do
with the types of measures that are required than with how performance
data, once collected, is to be used. Under Perkins III:

-   Performance Goals are Set. States must negotiate quantifiable levels
    of performance[^8] with the Department of Education and report
    yearly to the Secretary on progress made in meeting these standards.
    The previous Act simply required assurance from states that a system
    of measures had been implemented. Target performance levels were not
    required.

-   State Data Must be Submitted. States are required to submit
    performance results to ED; the Secretary of Education is to make
    this information available publicly and to compile state-by state
    comparisons. The previous Act did not require submission of state
    data.

-   Rewards and Penalties are Possible. Perkins III raises the potential
    stakes associated with performance. Under the law, failure to meet
    state-level performance standards could eventually result in loss of
    Perkins funds. As a reward, however, Section 503 of the Workforce
    Investment Act (WIA) provides incentive grants to states that exceed
    performance levels under the Perkins Act, Adult Education and Family
    Literacy Act, and Workforce Investment Act Title I.

III. **STUDY OVERVIEW**

This study of Perkins funding and accountability has several important
dimensions. The study will examine both *how* states and local programs
implement Perkins III program management requirements and the early
*impacts* of doing so. In addition, the study will assess implementation
and impacts at *both the secondary and postsecondary levels*. To
accomplish these objectives, the study will use a variety of data
collection and analysis methods. The general approach to the evaluation
includes:

-   Survey of State Vocational Education Directors. All state directors
    responsible for either secondary or postsecondary vocational
    education (or both) will be surveyed in spring 2001. The
    questionnaire will be designed to collect information from the
    directors, or their designated representatives, about the quality,
    capacity, and uses of state vocational performance measurement
    systems, as well as innovative accountability practices used at the
    state and substate levels. The survey will also include questions
    about Perkins funding priorities and practices, including the use of
    funds at the state level.

```{=html}
<!-- -->
```
-   Analysis of State and Local Grants. As part of the survey, state
    directors will be asked to send financial information (by hardcopy,
    diskette, or e-mail) on all annual local and Tech-Prep grants
    awarded for school years 1998-1999; 1999-2000; 2000-2001; and
    2001-2002. This data will be analyzed and compared to allocations
    under the previous Perkins Act.[^9] The analysis will examine the
    extent to which funding patterns have changed, particularly with
    respect to grantees' urbanicity, poverty, and service of special
    populations.

```{=html}
<!-- -->
```
-   Review of State Plans and Performance Reports. States are required
    to submit plans in 2000 and again in 2001 for how they will meet the
    new Perkins accountability provisions. These plans are expected to
    describe the standards, definitions, and assessments used for
    performance reporting. States also submit Annual Performance
    Reports, and beginning in April 2000 the reports will record actual
    performance data for each of the indicators specified in the
    legislation. Finally, the U.S. Department of Education's Office of
    Vocational and Adult Education (OVAE) will compile an annual list of
    negotiated adjusted levels of performance for each state and
    indicator. These documents represent an important source of
    information on states' reporting capacity and performance progress,
    and can also be used to inform the development of survey and case
    study data collection instruments.

-   Case Studies. Questions about the impacts of new accountability and
    funding provisions, barriers to their implementation, and promising
    practices for performance measurement and grant allocation, can best
    be addressed through in-depth site visits at the state, district,
    and school levels. For this study, five states will be selected. Up
    to two states will be chosen to represent innovative or mature
    accountability systems, one state may be chosen as an example of a
    particularly innovative state vocational funding system, and the
    remaining two will be chosen randomly from among states whose
    program management systems are more typical. Within each state, an
    urban, rural, and suburban community will be selected randomly, but
    ensuring some diversity in size. Interviews will be conducted with
    key staff at the local school district, up to two high schools or a
    combination of one high school and an area vocational center, and
    the nearest community college.

-   Assessment of Information Collected Under Other NAVE Studies. At
    least one other NAVE study will also be collecting some case study
    information on funding and accountability practices during the same
    time period, but on a very limited basis.[^10] To the extent
    possible, states selected for that study and the current study of
    Perkins program management provisions will not overlap. Contractors
    for both studies will be expected to share relevant site visit
    protocols and data collection forms, so that comparable data can be
    collected.

## 

## **IV. KEY POLICY QUESTIONS**

###### Funding

The assessment of funding requirements will rely mostly on comparisons
of grant amounts and expenditures before and after the implementation of
Perkins III. However, the analysis should be designed to address the
following research questions and issues that are relevant for
policymakers:

1.  To what extent have the new provisions affected the distribution of
    funds at the state and local levels?

The allocation of federal education dollars has long been the subject of
debate at the national, state, and local levels. One level of debate has
to do with the share of money earmarked to states versus local programs.
Another concerns the extent to which federal funds should be or how they
are targeted to help educationally or economically disadvantaged
students succeed in school. In Perkins III, Congress has incorporated
its current views on these issues by mandating certain funding formula
changes. On the other hand, Perkins III continues to allow each state to
determine the share of its federal dollars that go to secondary versus
postsecondary vocational education. Given the new funding formulae and
the possibility of shifts in state priorities, it will be important for
the contractor to address the following questions related to the
allocation of Perkins funding:

-   What has been the financial impact of changes in the allocations for
    state leadership and for state efforts to promote vocational
    education in non-traditional fields? Has there been any change in
    the amount spent on state administration?

-   Have states changed the allocation of funds between secondary and
    postsecondary vocational education? If so, why? What factors account
    for differences in priorities across states?

-   How have the new provisions altered the distribution and amount of
    secondary funds at the local level? Has the size of the typical
    district or area vocational center grant changed? How have districts
    or centers with different characteristics\--e.g., urbanicity,
    poverty levels, concentrations of Native Americans and other
    minority populations\--fared under new Perkins funding formulae? To
    what extent is targeting similar to that under Perkins II, and/or
    consistent with current congressional expectations?

```{=html}
<!-- -->
```
-   Which states use the waiver provision -- an alternative funding
    formula -- for the distribution of Perkins secondary grants, what
    are these formulae based on, and how well are the funds targeted
    compared to other states\' funding?

```{=html}
<!-- -->
```
-   What other provisions, if any, have been responsible for changes in
    the targeting of Perkins funds? For example, are states using the
    new reserve fund? If so, what types of eligible districts are
    getting more money through this distribution?

-   To what extent have Tech-Prep grant amounts or targeting changed
    under Perkins III? Do states target Tech-Prep funds differently than
    other Perkins dollars\--i.e., are more affluent communities
    receiving Tech-Prep grants relative to those receiving basic state
    grants?

-   Have there been any changes in postsecondary grant sizes or
    distribution? For example, does the Workforce Investment Act (WIA)
    influence the size and distribution of Perkins III funds to
    postsecondary institutions? How are funds distributed between
    community colleges and other eligible postsecondary institutions?

-   How large are the WIA incentive grants reserved for programs that
    exceed performance expectations and how are they allocated?

2.  How are Perkins funds used and to what extent do they emphasize
    program improvement?

Congress underscored the goal of vocational program improvement in
Perkins III. The Act promotes several strategies to upgrade the content,
delivery, and outcomes of vocational education programs, such as: (1)
building on state efforts to develop challenging academic standards, (2)
integrating academic and vocational education, (3) linking secondary and
postsecondary programs, and (4) broadening instruction beyond narrow
skills training.

To help local programs achieve these objectives, Perkins III provides
increased flexibility in the administration and use of federal
vocational funds. Congress eliminated set-asides funding streams for
certain special populations (e.g., single parents, displaced homemakers,
single pregnant women, etc.) and for particular programs (e.g., those
that promote sex equity or non-traditional employment). By expanding the
basic grant and allowing local entities greater discretion in
determining how Perkins funds would be spent, Congress hoped to improve
vocational education for all students. Whether this flexibility results
in any actual changes in funding priorities or expenditures, or in a
greater emphasis on program improvement activities, will be of interest
to policymakers and to advocacy groups concerned about the progress of
certain historically disadvantaged students. The contractor shall
therefore address the following key questions about the use of Perkins
funds:

-   How do states use State Leadership funds? What are the priorities
    for Perkins funds retained at the state level (e.g., technical
    assistance, evaluation, development of accountability systems)? Have
    these changed over the last several years?

-   Are more states establishing priorities for use of Perkins grants at
    the local level? If so, what are they? What factors affect
    differences in priorities across states (e.g., strong tradition of
    local control, etc.)?

-   What are the local priorities for Perkins funds? How do districts,
    schools, and postsecondary institutions use their grants (e.g.,
    salaries, equipment and technology, outreach or programs for special
    populations, professional development for teachers, support
    services, accountability reporting)? What changes in grant use have
    occurred over the last several years?

-   Is it possible to determine the extent to which Perkins funds
    support the legislated program improvement priorities versus
    maintenance of programs? If so, what is the relative emphasis on
    each of these two broad categories of expenditures?

-   What has been the effect of eliminating set asides and other
    provisions for special populations on state or local strategies for
    serving students in these groups (e.g., key staff or programs added
    or eliminated)?

3.  **To what extent are federal Perkins and state vocational education
    funds targeted and used in consistent ways?**

Perkins provides only a small share of the funds spent on vocational
education. Thus, the way in which state and local vocational funds are
allocated can effect the impact of Perkins on program improvement and
outcomes. In addition, although the allocation of Perkins grants is
driven largely by legislated formulas, in some respects the new Act
gives states greater leeway in how funds are targeted and used. For
example, states can seek a waiver of the required secondary intra-state
formula if they can demonstrate that an alternative approach more
effectively targets funds to recipients in poverty. Finally, some states
are implementing innovative state vocational funding policies. Lessons
learned from these strategies, particularly those linked to reform
efforts and performance, may be of some use for refining federal funding
practices. For these reasons, the contractor shall explore the
relationship between Perkins and state vocational funding approaches.
Key questions include:

-   Have there been changes in the share of Perkins funding relative to
    overall state budgets for vocational education? What is the per
    pupil federal Perkins expenditure by state? How do these figures
    compare to the per pupil state vocational education expenditure by
    state?

```{=html}
<!-- -->
```
-   What criteria do states use for vocational funding? How do these
    differ, if at all, from those criteria used in allocating Perkins
    grants (e.g., to what extent do states target low-income
    communities)?

-   At the district, school, and postsecondary institutional level, to
    what extent are Perkins grants considered \"discretionary\" and used
    to fill in gaps in state funding, versus to augment state funding
    priorities?

```{=html}
<!-- -->
```
-   Do states use state funds for incentives to promote and reward high
    performing programs? To what extent are state sanctions used with
    poor performing programs? How do local districts or postsecondary
    institutions respond to state performance-based funding? Are Perkins
    accountability provisions viewed in the same way?

```{=html}
<!-- -->
```
-   Are there innovative state funding approaches that could be applied
    to Perkins to further improve vocational education programs and
    outcomes?

[B. Accountability]{.underline}

Analysis of the accountability provisions in Perkins III will be based
on a variety of information sources. Together, these data should help
address the following key research issues:

1.  **What is the technical quality of the performance standards and
    measurement procedures developed by states?**

Studies undertaken in the early 1990s of initial efforts to implement
Perkins II accountability requirements raised an important concern that
remains relevant today. While states were, by and large, able to comply
with legal requirements, the overall quality of their performance
measurement systems was weak. States often lacked ways of appropriately
measuring performance indicators or lacked comparable measures or
definitions within the state. Thus, they were unable to aggregate
locally reported results. States had difficulty "tracking" students
after they left high school or college, and because few states had
access to individual student records, were unable to produce reliable
estimates of performance for students considered to be part of special
populations.

In the past several years, states and the federal government have placed
considerably more emphasis on accountability. A recent article suggests
that "states have made dramatic progress toward meaningful
accountability systems," perhaps spurred by the realization that
accountability is not a passing fad.[^11] Anecdotal information
indicates that Perkins III accountability requirements have been taken
much more seriously by federal and state officials than was the case
under the previous Act.

It is appropriate, therefore, to determine whether: (1) the overall
quality of vocational education accountability systems has *improved;*
(2) current practice is *consistent with established standards of
quality*; (3) *expected levels of performance* are based on sound
baseline data, and are realistic but challenging; and (4) states and
communities are developing *greater capacity to measure required
outcomes* in a consistent and reliable manner. The contractor shall
address several critical questions, including:

-   What are the academic standards used by states? Do states have
    different academic standards for vocational than for other students?
    Do the assessment approaches provide measures of the outcomes of
    vocational education (e.g., tests at or near program completion), or
    simply describe which students enroll in vocational education (e.g.,
    tests in 8^th^ or 9^th^ grade or, for college, entry tests)?

-   How many states have statewide technical standards and ways of
    assessing student performance against those standards? If statewide
    measurement procedures are not in place, how do local programs
    assess attainment of technical skills? How reliable are these local
    measures (e.g., are they industry-validated)?

-   Are local entities in a state using measures that are similar to
    state standards? Are they assessing outcomes for the same population
    of students? How inclusive is the vocational student population for
    which data is being reported (e.g., are all vocational students in
    the state included)?

-   How many states have data systems with consistent definitions and
    measures in place to produce reliable information about student
    performance? What capacity do states have to report by different
    special population categories?

-   To what extent have the standards, assessments, and reporting
    systems changed in recent years? What are the barriers to
    improvement?

-   To what extent are states and local entities able to integrate WIA
    and Perkins performance accountability requirements? Or Title I and
    Perkins requirements? What are the barriers?

-   What factors affect the quality and cohesiveness of state
    accountability systems?

2.  How are Perkins performance report results used to manage and
    improve programs?

From their inception, Perkins accountability requirements have been
intended to serve primarily as a way for states and local entities to
improve program management. However, in addition to highlighting data
quality concerns, the earlier accountability studies of Perkins II also
questioned the extent to which data collected was used to "drive"
program management. Most local educators tended to view performance
measurement as a matter of compliance with the law. They reported seeing
little value in the information they were asked to gather and report.
Given this history, it is important for the contractor to examine
several questions about the impact of the new requirements on schools
and postsecondary institutions, including:

-   Are states and communities using the data gathered to manage and
    improve their vocational education programs? If so, how (e.g., by
    publishing results, linking performance measurement to local plans
    or targeted funding, directing technical assistance, providing
    One-Stop Career Center clients with information about local
    occupational program outcomes)? Can states identify local programs
    in need of improvement based on performance data?

-   What factors affect the ease or utility of performance measurement
    as a tool for improvement (e.g., alignment with other programs'
    performance measurement, existence of statewide student data
    systems)? Are certain Perkins indicators more likely to be used to
    improve programs than others?

-   Are the incentives and sanctions credible? To what extent do they
    affect how state and local administrators view performance
    measurement?

-   To what extent have state or local administrators emphasized
    accountability or performance measurement in the professional
    development, technical assistance, or funding provided?

3.  Does state-reported performance data provide an accurate, up-to-date
    national profile of vocational education outcomes?

Perkins III requires the Secretary of Education to conduct an analysis
of performance results and to publish state-by-state comparisons of the
data. In theory, these analyses would allow Congress to compare the
progress of: (1) individual states with each other; and (2) vocational
education with that of other federal education or training programs.
Given the variability among states in performance measurement practices,
it is an open question whether these objectives can be achieved.
Specific issues the contractor shall address include:

-   To what extent is performance measurement comparable across states?
    Can state-reported data be aggregated to produce a national profile
    of vocational education? How does the existence or lack of data
    comparability affect the validity of the national picture?

-   How much change in state performance measurement is expected over
    the next five years (e.g., in standards, assessments, definitions,
    reporting procedures)? How likely are these changes to bring greater
    consistency in the process among states?

4.  Can the Perkins III accountability system be improved?

    The accountability provisions in Perkins III built upon lessons from
    performance data collection and reporting experiences under Perkins
    II[^12]. However, ongoing efforts to implement the new requirements
    may uncover intractable barriers or yield alternative strategies
    with greater potential for program improvement. While it is highly
    desirable to maintain stability in a performance measurement system,
    it is equally important to ensure the system achieves its objectives
    efficiently. Therefore, the contractor shall assess whether
    refinements are warranted by examining the following issues:

-   Why have some states made greater progress in developing
    accountability systems for vocational education than others? What
    are the characteristics of the best performance measurement systems?
    Are these features replicable in other states?

-   Are the current performance indicators in Perkins III the
    appropriate ones? Are there too many indicators?

-   Are Perkins III performance measures targeting the appropriate
    student populations?[^13]

-   What is the burden on reporting entities?

-   Can vocational performance reporting be better aligned with data
    collection systems for other education and workforce development
    programs?

-   What supports or regulatory changes at the federal level could
    improve state and local performance measurement capacity?

-   

-   

-   

-   

-   

**IV. SCOPE OF WORK**

This section specifies the tasks and subtasks the contractor shall
perform. The contractor shall submit all deliverables to ED's
Contracting Officer's Technical Representative (COTR) in draft. *Unless
otherwise specified, the COTR will notify the contractor within two
weeks of receipt of changes required in the deliverable, and the
contractor shall have two weeks to make the necessary changes and submit
a final deliverable to ED.*

**Task 1: Gather Existing Information**

The purpose of this task is for the contractor to become familiar with
existing information on Perkins funding and vocational performance
measurement. Such information can serve as both a source of analytic
data---to be drawn on in a synthesis report\-- and a starting point for
developing the survey instrument and site visit protocols.

The contractor shall:

-   Meet with the COTR, the Contract Specialist (CS), and program staff
    from the Office of Vocational and Adult Education (OVAE). The
    contractor shall allocate one day in Washington, DC for this initial
    meeting. The purpose of this meeting shall be to discuss procedures
    for conducting the study and answer questions that the contractor
    may have.

```{=html}
<!-- -->
```
-   Review available documents to begin compiling data and help guide
    the development of data collection instruments. As an initial step,
    the contractor shall contact OVAE staff to obtain copies of: (1) the
    funding and accountability sections of State Plans submitted under
    Perkins III, (2) Annual Performance Reports submitted under Perkins
    III, and (3) OVAE's list of the adjusted levels of performance
    negotiated with each state[^14]. In addition, the contractor shall
    examine the previous NAVE survey of state vocational education
    directors, to determine which relevant items may be useful to retain
    in the current survey to allow longitudinal comparisons.

```{=html}
<!-- -->
```
-   Review data files and reports from the previous NAVE study of
    funding practices, to maximize the extent to which comparisons can
    be made of grant amounts under Perkins II and Perkins III. These
    data files, programs, and reports will be available from the
    COTR[^15].

```{=html}
<!-- -->
```
-   Examine and critique relevant draft case study protocols and data
    collection forms for another NAVE study, if they have already been
    developed.

The contractor shall use data obtained from these sources to develop a
set of analytic tables, designed to address the research questions
above. These tables may include, for example, numbers or amounts of
Perkins grant by geographic area or poverty levels, dimensions of the
technical quality of states' performance measurement systems, actual
performance levels, evidence of improvement, etc. Some tables may be
empty shells, to which data will be added later when it is collected.
Other tables are likely to include counts or proportions of states or
students meeting key criteria. The contractor shall submit these
analytic tables to ED within two months after the effective date of the
contract.

Deliverables: Analytic Tables

**Task 2: Survey of State Directors**

One important objective of this study is to analyze the way in which
Perkins grants are allocated and used by and within states. Another goal
is to characterize the status of vocational education performance
measurement across the nation, and the likely pace and direction of
these efforts over the next few years. State vocational education
directors are in the best position to report on these issues, given the
new funding formulas and the changes in data collection and reporting
many states are purportedly initiating in response to Perkins III. The
contractor shall conduct a survey of state directors in spring 2001. The
survey will capture information just as states are submitting to OVAE
their final plans for the last several years of the Act.

##### Subtask 2.1: Design Questionnaire

The contractor shall develop a mail survey questionnaire and administer
the questionnaire to all state vocational education directors, which
will include items about both *secondary and postsecondary* vocational
education. Among the funding issues that shall be addressed are: (1)
priorities for and spending of state-level Perkins funds, (2) total
amounts allocated to secondary and postsecondary institutions, (3) use
of alternative formula for secondary distributions, (4) amount of state
vocational funds and allocation criteria, and (5) perceptions of the
impacts of the new funding provisions. In addition, the questionnaire
shall include instructions for state directors to send information, by
hardcopy, diskette, or email, on all annual local secondary and
postsecondary grants for school years 1998-1999, 1999-2000, 2000-01, and
2001-02. The contractor shall request lists with secondary and
postsecondary NCES identification numbers, to match grant amounts to
other data sources, such as the Quality Education Data (QED) or Common
Core Database (for secondary school characteristics) and IPEDS (for
postsecondary institutional characteristics).

The state director survey shall also address accountability issues, such
as: (1) quality of performance standards and measurement, (2) capacity
for data collection and reporting, (3) guidance and assistance provided
to local agencies, (4) use of performance information in program
management, (5) innovations in performance measurement practices, and
(6) perceptions of progress and barriers to improvement in
accountability systems at the federal, state, and local levels. The
contractor shall consider including some funding and accountability
items from the state director survey administered in the previous
National Assessment of Vocational Education.

The survey will take approximately 1 hour to complete. However, much of
that time will be spent by state directors compiling and, in some cases,
computing data relating to the funding questions. The contractor shall
plan for the questionnaire items to take no more than 35 minutes to
complete.

The contractor shall submit a draft of the questionnaire to ED 2 months
after the effective date of the contract, and a final version of the
questionnaire one month later.

Deliverable: State Director Survey Questionnaire (draft and final)

[Subtask 2.2: Conduct Pretest]{.underline}

The contractor shall pretest the survey with no more than nine state
directors of vocational education. The contractor shall debrief each of
the pretest directors to get their reaction to the survey instrument,
and shall elicit comments on the usefulness and likely accuracy of the
information requested as well as the burden associated with providing
it.

Based upon the results of the pretest, the contractor shall prepare a
memorandum discussing recommended changes to the questionnaire. The
contractor shall submit this memo to ED within 3 months after the
effective date of the contract.

Deliverable: Memo on Pretest Results

[Subtask 2.3: Prepare OMB Clearance Package]{.underline}

The contractor shall prepare the necessary forms required for OMB
Clearance. The clearance package must justify the necessity for
collecting the data and comprehensively respond to each required item in
the instructions. The forms clearance package shall include brief,
concise statements of a) the study mandate and objectives, b) types of
information requested, c) steps taken to minimize respondent burden, d)
plans for tabulating data, e) the data collection schedule, and f) steps
taken to review plans by outside persons, g) estimates of burden for
respondents and how such figures were estimated, h) discussion of
sensitive questions, if any, and i) estimates of the cost of the
activity. The contractor shall append copies of the instruments, marked
to show the study's mandate and the voluntary nature of the respondent's
participation, as well as table shells indicating how it will tabulate
and analyze the data. ED will ensure that the funding study contractor
prepares and submits relevant portions of the OMB package.

Forms clearance typically requires approximately 120 days. The
contractor shall not collect data without OMB approval and clearance.
Therefore in planning and scheduling data collection, the contractor
must devote sufficient time and resources to this product to assure
timely completion of the clearance process. The contractor shall submit
the OMB package to ED within 4 months after the effective date of the
contract.

Deliverable: OMB Forms Clearance Package (draft and final)

##### Subtask 2.4: Administer Survey

The contractor shall conduct the state director survey in early spring
2001. The contractor shall administer the survey by mail. The contractor
shall follow-up with nonrespondents by telephone and fax, and with
respondents by telephone to clarify data provided. In addition, the
contractor shall prepare a letter of introduction to the respondents
from ED after receipt of OMB clearance. This letter shall contain the
following:

-   Explanation of the purpose of the survey

```{=html}
<!-- -->
```
-   Indication that the appropriate OMB approval has been received and
    an estimate of the expected burden of the questionnaire

```{=html}
<!-- -->
```
-   Statements stressing the importance of the survey and requesting the
    cooperation of all respondents; and

```{=html}
<!-- -->
```
-   Provision of the names and phone numbers of both ED staff and
    contractor employees to contact with questions and comments

The contractor shall submit a draft of this letter to ED within one week
after receiving OMB clearance and approval and no later than 10 months
after the effective date of the contract. The contractor shall mail the
letter to the universe of state directors one week prior to the
beginning of the survey administration.

The contractor shall track the status of all responses. ED anticipates a
100 percent response to the survey.

Deliverable: Accompanying Letter

[Subtask 2.5: Protection of Data]{.underline}

The contractor shall be familiar with and comply with: The Privacy Act
of 1974, P.L. 93-579, 5 USC 552 a; The "Buckley Amendment," Family
Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of
Information Act, 5 USC 522; and related regulations, including but not
limited to: 41 CFR Part 1-1, 45 CFR Part 5b, and 40 FR 44502 (September
26, 1975); and, as appropriate, the Federal common rule or Department
final regulations on protection of human research subjects.

The contractor shall maintain the strictest of confidentiality of all
individual data collected in this study. The contractor shall under no
circumstance release any personally identifiable information about
sample members or the schools in which they teach unless such a release
is legally required. The contractor shall maintain information which
identifies persons or institutions in files that are physically separate
from other research data and which are accessible only to authorized
agency and contractor personnel. The contractor shall only use
individual identifiers for purposes of data collection, establishing
sample composition, authenticating data collection, or obtaining missing
data. The contractor shall destroy all lists and codes of individual
identifiers in any format within 24 months of completing final data
analysis.

#####  

##### Subtask 2.6: Analyze Data

The contractor shall develop quality assurance and data entry
specifications for the survey data and for grant data received from
state directors. To ensure accuracy, the contractor shall verify all
data entered and conduct edit and consistency checks. The contractor
shall resolve all problems identified through this process through phone
calls to respondents.

The contractor shall prepare an analysis plan that specifies the basic
tabulations of the survey data that will be conducted for the entire
sample and any relevant subgroups. The contractor shall also prepare a
plan for analyzing state and substate grant allocations for the entire
sample and for relevant subgroups defined by variables that can be
obtained from secondary data sources (e.g., the QED, Common Core, or
IPEDS data noted earlier). The contractor shall consider and document
how information on grant amounts will compare with those analyzed under
the previous NAVE.

The analysis will rely primarily on simple descriptive statistics and
cross-tabulations. The contractor shall attempt to determine typical or
average responses and the degree to which there is variation in these
responses. The contractor shall conduct tests for statistically
significant differences across subgroups of grantees and states.

The contractor shall submit a summary of the broad analysis agenda as
part of the OMB package. The contractor shall submit a more detailed
analysis plan to ED by month 7 after the effective date of the contract,
and the completed data analysis tables by month 13.

Deliverables: Analysis Plan

Data Analysis Tables

**Task 3: Case Studies**

The case studies of state and local funding and accountability practices
are central to this evaluation. Observation and structured, but
open-ended, discussion provide the best way to collect information about
how vocational agencies and institutions currently implement the new
requirements, whether they make use of the data reported, and the extent
to which they have or will make changes in response to Perkins III.

The contractor shall conduct site visits as the primary data collection
activity in the case studies. Site visits shall consist of collecting
existing information about funding priorities and allocations (e.g.,
district reports on school funding distributions and uses) and
performance measurement (e.g., technical reports, local data reports,
etc.), observing systems designed for performance measurement, and
interviewing key staff at the state, district, and school or college
levels.

[Subtask 3.1: Selection of States and Sites for Case
Studies]{.underline}

The case studies have two analytic dimensions, which have implications
for how states and local institutions are selected. First, while it will
not be statistically representative, the qualitative data collected
should provide a picture of typical funding and accountability system
implementation, reflecting at least some of the national diversity in
status and strategies. Second, the case studies are intended to document
innovative accountability approaches and to provide some sense of how to
successfully implement performance measurement systems. There is less
emphasis in the case studies on observing innovative funding practices,
because the allocation of Perkins grants are determined largely by
legislated formulae.

Based upon a review of state Perkins plans, existing reports and other
documents, and conversations with ED and state staff, the contractor
shall propose for the case studies a total of: (1) five states; (2)
three communities, each of which will include a district and community
college; and (3) up to two high schools or one high school and one area
vocational center in each district.[^16]

States may be selected purposefully for the case studies. Two states
should represent those with innovative or well-functioning,
comprehensive performance management systems. The remaining three states
should be selected randomly. However, one state with a particularly
innovative state funding approach may be chosen in place of random
selection. The contractor shall include geographic diversity and the
presence of area vocational centers as criteria in sampling states.

The contractor shall select three communities randomly in each state,
divided among rural, suburban, and urban geographic areas.[^17] These
communities should be selected to include some diversity in size and
poverty of the student population. In each community, the contractor
shall select a school district, up to two high schools or a combination
of high school and area vocational center within the district, and a
community college. It is likely that each community will have only one
school district. If that district contains more than two high schools,
the contractor shall select the two schools randomly. The community
college located in the community or which the highest proportion of the
district\'s students attends shall be chosen.

The contractor shall prepare a memorandum listing the proposed sample of
states, communities/school districts, high schools, area vocational
centers, and colleges for inclusion in the case studies, and describing
the rationale for the selection of each site. The contractor shall
submit the memorandum to ED by month 4 after the effective date of the
contract.

Deliverable: Memo on Proposed States, Districts, Schools, and Community
Colleges

[Subtask 3.2: Develop Case Study Protocols]{.underline}

The contractor shall develop data collection protocols that will address
the policy and research questions outlined in the study overview, and
guide the conduct of site visits in selected states, districts, schools,
and colleges. Separate protocols for each level of interviewing seems
most appropriate, although the same general set of issues will be
covered in all of them. At each level or institution, the staff
interviewed must include the individual with primary responsibility for
Perkins (often called the "Perkins coordinator"), staff involved in
grant allocations and monitoring, and any data, evaluation, or systems
staff involved in performance measurement and reporting. The contractor
shall also interview faculty members, curriculum supervisors, or
administrators for more detailed information.

The contractor shall include draft protocols in the OMB package. The
contractor shall submit the revised final protocols to ED for approval 5
months after the effective date of the contract.

Deliverable: Case Study Protocols (final)

[Subtask 3.3: Train Case Study Staff]{.underline}

The contractor shall train the staff conducting the case studies. The
contractor shall submit training materials to ED for review within 5
months of the effective date of the contract.

The contractor shall conduct staff training within 6 months after the
effective date of the contract. This staff, whether in-house staff or
persons hired specifically for the study, shall have experience in
conducting qualitative research and case studies, and be knowledgeable
about vocational education and performance measurement. The training
shall ensure that study staff are thoroughly familiar with the study
objectives, design, and data collection procedures. The contractor shall
ensure that site visit data collection is as consistent as possible
among site visitors, to increase the ability to make generalizations and
comparisons across case study sites.

Deliverables: Training Materials (draft and final)

[Subtask 3.4: Conduct Case Studies]{.underline}

The contractor shall begin conducting the site visits to state agencies
and communities selected to gather information for the case studies
within 7 months of the effective date of the contract. The contractor
shall collect information from a wide range of sources using the
protocols developed under Subtask 3.2, and shall review all extant
documentation and reports.

**Task 4: Report Preparation**

The contractor shall prepare a report summarizing and integrating
findings from the case studies, grant analysis, and state director
survey. This report shall address the policy and research issues
described earlier, in a format that is easy to read and highlights major
points and themes. The contractor shall submit an outline for the report
to ED 12 months after the effective date of the contract. Upon approval
of the outline by ED, the contractor shall submit a draft of the report
to ED 15 months after the effective date of the contract. The final
report, due 17 months after the effective date of the contract, shall be
submitted with 10 hard copies, one electronic version, one PDF version,
and one HTML version.

Deliverable: Synthesis Report (outline, draft, and final)

**Task 5: Monitoring Progress**

The contractor shall submit one copy of each of the following reports to
the Contracting Officer and the COTR on a monthly basis:

-   **Monthly Progress Reports/Exception Reports**. The contractor shall
    prepare monthly progress reports due within ten work days after the
    end of each month. They shall summarize the major activities and
    accomplishments for the reporting period. In addition, they shall
    provide information for each project task regarding significant
    findings and events, problems encountered, and staff use. The
    reports shall also specify the extent to which the project is on
    schedule, briefly describe the activities planned for the next
    month, identify and discuss significant deviations from the
    substantive and time factors in the management plan, and identify
    and discuss any decisions which may be needed from ED. If there are
    no exceptions, the reports shall state that there are no exceptions.
    If there are exceptions to the management plan, the contractor shall
    describe the plan for resolving problems.

```{=html}
<!-- -->
```
-   **Monthly Manpower/Expenditure Reports**. The contractor shall
    prepare monthly expenditure reports due within ten working days
    after the end of each month. These reports, prepared and signed by
    the project director, shall summarize the actual personnel
    assignments for the month just completed, showing for each staff
    member the hours charged by task. The report shall project similar
    assignment information for the upcoming month. The reports shall
    also exhibit expenditures, segregating project costs by individual
    and by task, and specifying for all travel the locations, duration,
    and personnel for each trip.

[^1]: This work statement draws upon advice received from an Independent
    Advisory Panel and experts who prepared commissioned papers on
    various topics. Commissioned papers and a summary of the Advisory
    Panel meeting are available at:
    http://www.ed.gov/rschstat/eval/sectech/nave/index.html

[^2]: The NAVE is a broad-ranging congressionally mandated assessment of
    current practice in vocational education and the implementation of
    federal legislation. Findings and recommendations that result from
    NAVE activities are due to Congress by July 1, 2002. Congress and
    senior Executive Branch officials are the main audience for the NAVE
    results.

[^3]: PL 105-332

[^4]: A description of the NAVE evaluation plan can be downloaded from:
    <http://www.ed.gov/rschstat/eval/sectech/nave/index.html>

[^5]: However, Perkins III does mandate that \$60,000-150,000 of state
    leadership funds be used for sex equity services.

[^6]: Perkins III also allows states to seek a waiver if they have an
    alternative funding formula that more effectively targets resources
    to low-income students. At this time, all states are operating under
    a waiver because the new funding formula is based on Census counts
    and cannot be operationalized until Census 2000 results have been
    released.

[^7]: See Section 113 of PL 105-332

[^8]: States can renegotiate levels of performance prior to the third
    program year covered by the state plan. Timelines for the NAVE
    report will allow it to describe any changes in performance targets
    that may occur, but not the resulting outcomes.

[^9]: See the Final Report to Congress, Volume V, of the previous
    National Assessment of Vocational Education, July 1994.

[^10]: See www.ed.gov/offices/OUS/eval/voced.html for a description of
    the Study to Assess the Quality of Vocational Education in the
    United States.

[^11]: Mikala Rahn and Patricia Holmes, "Accountability Systems:
    Performance Standards and Assessment," *NCRVE Centerpoint* (March
    1999).

[^12]: For example, under Perkins II learning "gains" were to be
    reported. However, the substantial difficulty state and local
    agencies had tracking achievement gains from year to year led to the
    replacement of gain measures with measures of attainment or levels.

[^13]: Some have argued, for example, for a system of performance
    measures that captures outcomes for all students, not just a subset
    of "vocational students."

[^14]: Some of these documents are likely to be made publicly available
    on an OVAE web site.

[^15]: This previous work was conducted by MPR Associates, Berkeley, CA.

[^16]: Some districts have only one high school.

[^17]: In selecting communities, contractors may use school districts as
    a proxy for communities.