# Presentation: 914323

## PROJECT EVALUATION 

- IT’S ALL ABOUT STUDENTS

## In partnership, we help America’s students stay in school and graduate by:

- Reducing gaps in college access and completion among differing student populations
- Improving academic attainment
- Strengthening institutions
- Strengthening accountability

## We do this by:

- Expanding and enhancing institutional -
- - Academic quality
- - Management
- - Financial stability
- Using Federal program grant funds
- Implementing specific projects based on established and innovative practices
- Following sound management and oversight processes
- Measuring and reporting the results of the projects

## Success is evaluated by: 

- Students
- Project staff
- Institutions
- Department of Education
- Office of Management and Budget
- Congress
- America’s taxpayers

## Success is measured, in part, by:

**An institution’s evaluation results**
- Project performance data
- Annual Performance Report (APR) data
- Percentage of project goals met or exceeded
**The Department’s program evaluation results**
- Strategic Plan program data
- Achieving a high rating on OMB’s Program Assessment and Rating Tool (PART)
**How well we use this information for -**
- Strategic planning, decision-making, funding, and improving student success nationwide

## Program and Project Processes  for Achieving and Measuring Success:

- _Project_
- Development plan
- Implementation strategy & activity objectives
- Project management plan (key personnel & budget)
**Evaluation Plan**

- _Program _
- Purpose
- Strategic plan with national goals and measures
- Program Management
**Program Results**

## Program and Project Results

**Results are outcomes** based on activities (interventions) and costs
**Outcomes are quantifiable measures of:**
- Annual and long-term performance improvements; and
- Result from the funded activities
**Outcomes affect future policy and funding** **decisions**

## Defining Annual Outcomes

_**Program Measure**_
- Students at grantee institutions **persist at a higher rate** **than other similar students** who previously attended these institutions or currently attend other institutions.
_**Project Measure**_
- Students who receive tutoring **pass courses and improve GPAs at a higher rate** **than similar students** at the school who were not tutored.

## Defining Long -Term Outcomes

- _Program Measure_Students at grantee institutions **graduate at a higher rate** **than similar students** who previously attended these institutions or attend other institutions.
- _Project Measure_ Students who receive tutoring **graduate at a higher rate than similar students** not tutored.

## Outcomes are not:

**Output data**:  Activities necessary to achieve impact and outcome goals -
- Number of tutoring sessions held
- Number students receiving tutoring
- Staff hours and costs per activity
**Descriptive data**: General information about the the project and participants -
- Funding amount
- Length of project
- Socio-economic data
- Race/ethnicity

## Quantitative Outcome Measure

- Consistent measure overtime
- True and accurate measure of the intervention outcome
- Easily interpreted data (e.g., rates and proportions)
- Is not based on qualitative (e.g., surveys, focus groups) or descriptive data

## Planning Before Grant Writing Is Key

**Purpose** – defines the problem
**Goals **– establishes objectives/outcomes that are:
- Measurable and logically related to intervention and anticipated results
- Based on historical data and *benchmarking*
**Evaluation Plan **-
- Shows how the goals/objectives will be met
- Establishes processes for measuring if the goals/objectives were met and at what cost
- Identifies approach for clearly reporting results (outcomes based on activities and costs) in a timely manner

## Benchmark & Benchmarking

**Benchmark -**
- A standard by which something can be measured or judged
**Benchmarking -**
- To measure (rival’s product) according to specified standards in order to compare it with and improve one’s own product

## Data Timing

**Evaluation plan must ensure data are useful for strategic, operational, policy, and budget decision-making**
- _School_
- APR
- _Department _
- Performance accountability report (October)
- Integrated performance and budget (November)
- _OMB_ – President’s integrated performance and budget plan (early in calendar year)

## Evaluation Process for Measuring Outcomes

**Classic**** ****experimental approach**:
- Similar students randomly assigned to intervention and control groups
- Pre- and post- intervention performance measurement
- Multiple interventions and measurements over time

## Difficulties in Implementing Random Assignment

- Students generally self-select
- Identifying a valid comparison group
- _Options to help address these_:
- Draw comparison group from waiting list or those who applied but were not selected.
- Use historical data and benchmarking on similar students performance.

## Difficulties Implementing Pre- and Post-Testing

- Identifying valid baseline and post-activity data measures
- _Options to help address this_:
- Use institutional data measures already collected -
- Institutional records (services offered)
- Course grades
- Graduation rates
- Identify Federal & other data sources already collected, such as the National Center for Educational Statistic’s IPEDS database.

## Difficulties Implementing Multiple Interventions and Data Collections 

- Students may not choose to participate annually or consistently.
- _Options to help address this_:
- Collect student-specific information.
- Measure outcomes each semester.
- Conduct same activity and measurement with similar students over time.
- Conduct activity and measurement at multiple institutions.
- Look for relationships between varying usage of activity and performance of participants.

## Difficulties in Implementing the Evaluation Plan

- Limited resources-staff and money
- Implementation staff also conduct analysis
- _Options to help address these_:
- Request evaluation monies in grant application (5-10%).
- Hire an independent evaluator.
- Use academic staff and students.
- Use measures and student identifiers already collected.

## Difficulties in Collecting and Analyzing Data

- Data are self-reported
- Inconsistent or missing data
- _Options to help address these_:
- Document **clear data definitions**.
- Require **routine, systematic collection** and provide training/guidance.
- Develop **simple data collection tools** (Excel, Access databases).
- Hold **project staff meetings** to regularly review data collection processes, data quality, progress in meeting performance outcomes.

## Analysis of Data and Reporting Your Results

- Flows from evaluation plan and data collection processes in grant application
- Includes descriptive, quantitative, and qualitative data collected
- Results:  outcomes based on activities and costs
- Identification of “best practices”
- Conclusions and recommendations

## Project Evaluation Plan

- Must be fully documented
- Address the three components of an experimental design
- Specify data definitions, and collection and analysis processes.
- Include outcome and cost data needed to measure results.
- Completed for the grant application

## Resources

- ED:  www.ed.gov/about/
- offices/list/ope/fipse/evaluate
- NSF:  www.her.nsf.gov/rec/programs/main.asp
- FY 2004 Annual Plan:
- http://www.ed.gov/about/reports/annual/2004plan/plan2004.doc
- http://www.ed.gov/about/reports/annual/2004plan/edlite-institutionaldev.html