Power Point Presentation from the April 8, 2008 Technical Assistance
Workshop

Demonstration Projects To Ensure Students With Disabilities

Receive A Quality Higher Education Program

# Performance Measurement, Program and Project Evaluation

Slide #1

Cover page

DEMONSTRATION PROJECTS TO ENSURE STUDENTS WITH DISABILITIES RECEIVE A
QUALITY HIGHER EDUCATION PROGRAM

Performance Measurement, Program and Project Evaluation

April 8, 2008

Strategic Planning Staff

Slide #2

We need to assess long-term outcomes for all OPE programs:

The Secretary's Academic Competitiveness Council reported in May 2007
that "perhaps the most striking finding in undergraduate education is
the lack of rigorous evaluation of federal investments due to inadequate
mechanisms for the collection of data on long-term student outcomes."

U.S. Department of Education, Report of the Academic Competitiveness
Council, Washington, D.C., 2007

Slide #3

We need to assess long-term outcomes for all OPE programs (continued):

Without access to long-term data on student outcomes, it is impossible
to assess "what works." The measurement of student achievement and
institutional performance is critical for determining effective grantee
projects.

Slide #4

The peer reviewers will be instructed to look closely at the potential
of applicants to successfully reach their individual project goals,
which are measured by the program's performance indicators.

Slide #5

The importance of ensuring that your application includes a strong
evaluation plan (continued):

The evaluation plan should not only include formative and summative
measures, but also, address the use of appropriate controls and
processes that provide for independent evaluation.

Slide #6

The importance of ensuring that your application includes a strong
evaluation plan (continued):

The evaluation plan should help shape the development of the project
from the beginning of the grant period and provide benchmarks for the
monitoring of progress and measurement of that progress throughout the
grant award period.

Slide #7

Criteria for evaluation:

**(**G) QUALITY OF THE PROJECT EVALUATION (Maximum 20 points)

\(1\) The extent to which the methods of evaluation include the use of
objective performance measures that are clearly related to the intended
outcomes of the project and will produce quantitative and qualitative
data to the extent possible. (7 points)

Slide #8

Criteria for evaluation (continued):

> \(2\) The extent to which the methods of evaluation are thorough,
> feasible, and appropriate to the goals, objectives, and outcomes of
> the proposed project. (7 points)
>
> \(3\) The extent to which the methods of evaluation will provide
> performance feedback and permit periodic assessment of progress
> towards achieving intended outcomes. (6 points)

Slide #9

What we expect to see in applications:

Independent Evaluator:

-   The grantee hires an evaluator who is truly \'outside\' the project,
    whether from a different department or a different institution;

-   The grantee includes the evaluator from the planning stage onward;

Slide #10

What we expect to see in applications (continued):

Independent evaluator (continued):

-   The evaluator provides a data collection and analysis plan with
    benchmarks for project assessment over the duration of the grant;

-   The evaluator collects baseline data (for comparison purposes) at
    the start of the project;

Slide #11

What we expect to see in applications (continued):

Independent evaluator (continued):

-   The evaluator reports periodically on progress of the evaluation,
    and the evaluator\'s reports form part of the APR and FPR
    documentation;

-   The grantee considers any reports and recommendations of the
    evaluator as the project continues, and may make changes to the
    project in response to the evaluator\'s recommendations (within the
    allowed parameters of the program).

Slide #12

What we expect to see in applications (continued):

Evaluation plans:

Be sure to describe the evaluation design, indicating:

> \(1\) what types of data will be collected;
>
> \(2\) when various types of data will be collected;
>
> \(3\) what methods will be used;
>
> \(4\) what instruments will be developed and when;

Slide #13

What we expect to see in applications (continued):

Be sure to describe the evaluation design, indicating (continued):

> \(5\) how the data will be analyzed;
>
> \(6\) when reports and outcomes will be available; and
>
> \(7\) how you will use the information collected through your project
> evaluation to monitor the progress of your funded project.

Slide #14

What we expect to see in applications (continued):

Evaluation plans (continued):

-   We are especially interested in seeing applications that include an
    expressed commitment to collect and report outcome data on treated
    (and comparison) groups to the point where there are real graduation
    or post-graduation outcomes, even if that is two or more years
    beyond the end of the grant.

Slide #15

What we expect to see in applications (continued):

Control or comparison group:

We need to understand student success in context. Ideally, applications
would provide a comparison (it could be past history, or a similar (but
untreated) group, or some larger (e.g., national population) so that
results have a reference level.

Slide #16

What we expect to see in annual performance reports (APRs):

-   Baseline data collected

-   Evidence of anything changing

-   Ongoing data on the project

Slide #17

What we expect to see in final performance reports (FPRs):

By the end of the study, we would like to see outcome measure results.
This could potentially mean data collection beyond the grant period.

Slide #18

Possible Evaluation Designs

For more information about possible evaluation designs, please see the
Academic Competitiveness Council's report at:

<http://www.ed.gov/about/inits/ed/competitiveness/acc-mathscience/report.pdf>

Slide #19

GPRA Measures

-   Measure 1.1: The percentage of faculty trained through project
    activities who incorporate elements of their training into their
    classroom teaching.

-   Measure 1.2: The difference between the rate at which students with
    documented disabilities complete courses taught by faculty trained
    through project activities and the rate at which other students
    complete the same courses.

Slide #20

Evaluation Hierarchy

-   Weaknesses and Needs

-   Program and Project Goal(s)

-   Project Objectives: What a project will accomplish to realize
    program goal(s) (approved in application -- cannot be changed)

-   Activities: The steps and action plan to undertake to accomplish the
    program objective(s)

-   Performance Measures: How a project measures progress toward meeting
    its objective (GPRA, Program, Project - Modifiable)

Slide #21

Project Objectives and Performance Measures

-   Objective: what is the project trying to accomplish?

    -   Projects should write a few clear objectives that explain what
        the project is doing to support the overall goal.

-   High-quality project objective criteria: relevance, focus,
    measurability

Slide #22

High-Quality Project Objectives: Relevance and Focus

-   Relevance: How relevant is the project objective to the overall goal
    of the program and/or the goal of the project?

-   Focus: How focused is the project objective?

Slide #23

High-Quality Project Objectives: Measurability

Are there concepts in the project objective that lend themselves to
measurement? If so, is measurement feasible?

-   To enhance student learning. (conceptually hard to measure)

-   To measure the impact of the project on the academic progress of
    students with disabilities. (conceptually hard to measure)

Slide #24

Practice Exercise 1:\
How do these project objectives measure up?

Identify, develop, document, and replicate the implementation of
innovative educational interventions that encourage the retention of
students with disabilities.

> 1\. Relevance
>
> 2\. Focus
>
> 3\. Measurability

Slide #25

Practice Exercise 2:\
How do these project objectives measure up?

Increase the percentage of faculty members using universal design
teaching techniques.

> 1\. Relevance
>
> 2\. Focus
>
> 3\. Measurability

Slide #26

Project Performance Measure Types

-   What is a performance measure?

    -   An indicator to determine how well objectives are being met.

-   What to measure?

    -   A specific achievement/accomplishment

    -   Change in a level of achievement/accomplishment

    -   Achievement/accomplishment (or change) compared with other
        students or institutions

Slide #27

Components of Performance Measures

-   What will change

-   How much change is expected

-   Who will achieve the change

-   When the change will take place

Slide #28

Improving Performance Measures

Original Performance Measure:

To encourage [faculty members]{.underline} to use universal design
teaching techniques. (who)

What: [Increase]{.underline} the number of faculty members using
universal design teaching techniques.

When: [By 2011]{.underline}, increase the number of faculty members
using universal design teaching techniques.

How much: By 2011, increase the number of faculty members using
universal design teaching techniques [by 50 percent]{.underline}.

Slide #29

Improving Performance Measures

Original Performance Measure:

The institution will implement innovative educational interventions that
encourage the retention of students with disabilities.

Institution [faculty]{.underline} (who) will [develop and implement in
six undergraduate courses]{.underline} (what) innovative educational
interventions that encourage successful course completion by students
with disabilities.

Slide #30

Improving Performance Measures (continued)

Institution [faculty]{.underline} (who) will [develop and implement in
six undergraduate courses]{.underline} (what) innovative educational
interventions that encourage successful course completion by students
with disabilities [by the end of the third year of the
grant]{.underline} (when).

[Students with disabilities]{.underline} (who) will have [ten percent
higher]{.underline} (how much) [course completion rates in courses
developed under the grant than in prior existing courses]{.underline}
(what) [by the end of the grant period]{.underline} (when) \[Outcome\].

Slide #31

Remember

Activities are not performance measures!

\###