# Presentation: 707399

## Program Evaluation Model and Lessons Learned from 2007 

**PRESENTATION FOR****AIR TRAFFIC COLLEGIATE TRAINING INITIATIVE (AT-CTI) BEST PRACTICES CONFERENCE**

**Program Evaluation Model and Lessons Learned from 2007 **
**AT- CTI School Evaluation Cycle**

**October 16, 2007**

## Presentation Objective and Focus Areas

**Presentation Objective and Focus Areas**

**PRESENTATION OBJECTIVE**
**PRESENTATION FOCUS AREAS**
- AT-CTI School Program Evaluation Model
- Summary of Applications Reviewed for 2007 Evaluation Cycle
- Overview of Evaluation Process
- Report on the Minimum Eligibility Requirements
- FAA Classification of School Acceptance Levels and Best Practices
- Best Practices Areas by School
- Sample of Feedback Reports Prepared for Schools

**To provide information and insight to participants on the AT-CTI Program Evaluation Process, Model, and 2007 Evaluation Outcomes, and Best Practices to encourage full and active participation in the ongoing enhancement and operation of the AT-CTI Program.**

## AT-CTI Program Evaluation Cycle

**AT-CTI Program Evaluation Cycle**

**Program Participant Submit Annual Recertification Application**

**Submit Application for Certification and Participation in AT-CTI  Responding to Program Evaluation Model Components**

**Application is Reviewed for Eligibility by AT-CTI Program Office**

**AT-CTI Schools are Certified and Invited to Share Best Practices at AT-CTI Best Practices Conferences**

**Eligible Applications are Evaluated and Scored by a Team of Trained FAA Evaluators **

**Recommended Applicants Accepted and Ranked by Program Office and Receive Recognition**

**Team of Evaluators Reaches Consensus on Final Score and Recommends for Participation**

**Applicants Achieving a Minimum Qualifying Score are Site Visited by a Team of Trained FAA Evaluators**

**8**

**1**

**2**

**7**

**3**

**6**

**5**

**4**

## AT-CTI School Program Evaluation Model

| Component I | Value |  | Component II | Value | Component III | Value | Component IV | Value | Component V | Value |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  Organizational Foundation and Resources |  |  | II. Organization
Credibility |  | III. Curriculum 
and Facilities |  | IV. Student 
Performance |  | V. Organization 
Performance |  |  |
| 1.1 
Leadership |  | 20 | 2.1 
Accreditation | 15 | 3.1
 Curriculum  and Programs | 100 | 4.1
Graduate Trends | 60 | 5.1
Cooperation and Responsiveness |  | 20 |
| 1.2
Goals, Objectives, 
and Program 
Alignment
 |  | 30 | 2.2
Student Selection Process | 10 | 3.2
Facilities and Equipment  | 60 | 4.2
Stakeholder Input on Performance | 70 | 5.2
Data Submission 
and Reporting Quality |  | 40 |
| 1.3
Scope of Participation and Location |  | 25 | 2.3 
External Relations | 10 | 3.3
Student  
Assessment and Testing | 80 | 4.3
Academic Readiness Trends | 130 | 5.3
Continuous Performance Improvement Initiatives |  | 50 |
| 1.4
Resources, Student Support, and Capacity |  | 25 | 2.4
Outreach and Recruitment | 15 | 3.4
Aviation Program Instructors, Staff, and Management | 80 | 4.4
Academic Achievement Trends | 120 | 5.4
Contributions to 
AT-CTI |  | 40 |
|  |  | 100 |  | 50 |  | 320 |  | 380 |  |  | 150 |

## Summary of Applications Reviewed 

**for the 2007 Evaluation Cycle**

**33 Total Applications Were Received from Schools Nationwide**
**Received Applications from 14 Existing AT-CTI Schools**
**Existing Schools Evaluated on Components I, II, and III **
**No Site Visit Conducted**
**Received Applications from 19 New Schools Interested in AT-CTI**
**New Schools Evaluated on Components I, II, and III**
**Must Meet Minimum Initial Score of 320 to be Site Visited**
**Site Visit Required**
**All 33 school applications were documented in the Application Database**

**Applications Received**

| Applicant Schools | City | State | New / Existing |
| --- | --- | --- | --- |
| 1.        Aims Community College | Greeley | CO | New |
| 2.        Arizona State University | Mesa  | AZ | New |
| 3.        Community College of Baltimore County | Baltimore | MD | New |
| 4.        Community College of Beaver County | Beaver Falls | PA | Existing |
| 5.        Daniel Webster | Nashua | NH | Existing |
| 6.        Dowling College | Shirley | NY | Existing |
| 7.        Eastern New Mexico University - Roswell | Roswell | NM | New |
| 8.        Embry Riddle - Daytona Beach | Daytona Beach | FL | Existing |
| 9.        Embry Riddle - Prescott | Prescott | CA | New |
| 10.     Fairmount State University | Bridgeport | VA | New |
| 11.     Florida Community College at Jacksonville | Jacksonville | FL | New |
| 12.     Folsom Lake College | Folsom | CA | New |
| 13.     Green River Community College | Auburn | WA | New |
| 14.     Hampton University | Hampton | VA | Existing |
| 15.     Hinds Community College | Raymond | MS | New |
| 16.     Inter American University of Puerto Rico | Bayamon | PR | Existing |
| 17.     Kent State University | Kent  | OH | New |
| 18.     Lane Community College | Eugene | OR | New |
| 19.     Lewis University | Romeoville | IL | New |
| 20.     Metropolitan State College of Denver | Denver  | CO | New |
| 21.     Miami Dade County College | Homestead  | FL | Existing |
| 22.     Middle Georgia College | Cochran | GA | New |
| 23.     Middle Tennessee State University  | Murfreesboro | TN | Existing |
| 24.     Minneapolis Community and Technical College | Eden Prairie | MN | Existing |
| 25.     Mount San Antonio | Walnut | CA | Existing |
| 26.     Purdue University | West Lafayette | IN | Existing |
| 27.     Salt Lake Community College | Salt Lake City | UT | New |
| 28.     Texas Southern | Houston | TX | New |
| 29.     Tulsa Community College | Tulsa  | OK | New |
| 30.     University of Alaska | Anchorage | AK | Existing |
| 31.     University of North Dakota | Grand Forks | ND | Existing |
| 32.     University of Oklahoma | Norman | OK | New |
| 33.     Vaughn College of Aeronautics | Flushing | NY | Existing |

## Minimum Eligibility Requirements

**Minimum Eligibility Requirements Established by FAA AT-CTI to be Considered for Participation in AT-CTI**

**ACCREDITATION:**** Be a degree granting, not for profit, two or four year, post-secondary educational institution with regional or equivalent national accreditation recognized by CORPA or equivalent.**
**DEGREE:**** Currently offer a non-engineering aviation degree.**
**PROGRAM OFFERING AND VIABILITY:**** Have a viable aviation program as demonstrated by the number of aviation majors which have graduated over the last five years, which must be in excess of an average of 25 per year.**
**UNDERSTANDING OF AT-CTI PROGRAM PARTICIPATION REQUIREMENTS:**** Be prepared to complete the certification process and adhere to the program participation requirements outlined in this document**.
**MINIMUM INITIAL SCORE:**** Receive a score of >=320 for Components   I, II, and III to a receive site visit and continue evaluation process.**

## FAA Classification of School Acceptance Levels, Rating Scale, and Best Practices

**DESCRIPTION OF LEVLES**

**New schools not meeting minimum eligibility requirements**
**New schools not achieving minimum initial score of 320 and not meeting the requirements specified in Components I, II, and III for site visit to occur**
**New schools not achieving minimum final score of 350 and not meeting the requirements specified in Components I, II, and III **

**LEVEL**

**Not **
**Accepted**

**Probational **
**Acceptance**

**Provisional **
**Acceptance**

**Full **
**Acceptance**

**1**

**2**

**3**

**4**

**SCORE RATING SCALE**

## Best Practices Areas by School

**Based on High Scores in Specific Evaluation Factors**

| Evaluation Components 
and Factor Areas

Schools | Component I: Organizational Foundation and Resources |  |  |  | Component II: Organization Credibility |  |  |  |  | Component III: Curriculum & Facilities |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Leadership | Goals, Objectives, & Program Alignment | Scope of Participation & Location | Resources, Student Support, & Capacity | Accreditation | Student Selection Process | External Relations |  | Outreach & Recruitment | Curriculum & Programs | Facilities & Equipment | Student Assessment & Testing | Aviation Program Instructors, Staff, & Mgmt |
| NEW SCHOOLS SITE VISITED |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  Middle Georgia College |  |  |  |  |  |  |  |  |  |  |  |  |  |
| University of Oklahoma |  |  |  |  |  |  |  |  |  |  |  |  |  |
| The Metropolitan State College of Denver |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Florida Community College-Jacksonville |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Arizona State University |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  Lewis University |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Green River Community College |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Community College of Baltimore County |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Kent State University |  |  |  |  |  |  |  |  |  |  |  |  |  |
| EXISTING SCHOOLS |  |  |  |  |  |  |  |  |  |  |  |  |  |
| University of North Dakota |  |  |  |  |   |   |  |   |   |  |  |   |   |
| Community College of Beaver County |  |  |   |   |   |   |  |   |   |  |  |   |  |
| University of Alaska |   |   |  |   |   |   |  |   |   |  |   |   |   |
| Daniel Webster College |  |   |   |   |   |   |  |   |   |   |  |   |   |
| Vaughn College of Aeronautics |   |   |  |   |   |   |  |   |   |   |   |   |   |
| Purdue University |   |   |  |   |   |   |  |   |   |   |   |   |   |
|  Minneapolis Community and Technical College |   |  |  |   |   |   |  |   |   |   |   |   |   |
| Inter American University of Puerto Rico |   |  |   |   |   |   |  |   |   |   |   |   |   |
| Miami Dade County College |   |   |   |   |   |   |  |   |   |   |  |   |   |

## FEEDBACK REPORT

**Score: XX out of XX**
**Ranking: X of X**
**Status: XXXXXXXX**

**FEEDBACK REPORT**
**SAMPLE**

**SUMMARY OF KEY STRENGTHS**

**SUMMARY OF KEY OPPORTUNITIES**

| Component / Factor | Description | Score | Max | Indicator |
| --- | --- | --- | --- | --- |
| 1 | Organizational Foundation and Resources | 96 | 100 | Best Practice |
| 1.1 | Leadership | 20 | 20 | Best Practice |
| 1.2 | Goals, Objectives, and Program Alignment | 30 | 30 | Best Practice |
| 1.3 | Scope of Participation and Location | 24 | 25 | Best Practice |
| 1.4 | Resources, Student Support, and Capacity | 22 | 25 | Good |
| 2 | Organization Credibility | 31 | 50 | Fair |
| 2.1 | Accreditation | 6 | 15 | Poor |
| 2.2 | Student Selection Process | 6 | 10 | Fair |
| 2.3 | External Relations | 10 | 10 | Best Practice |
| 2.4 | Outreach and Recruitment | 9 | 15 | Fair |
| 3 | Curriculum and Facilities | 308 | 320 | Best Practice |
| 3.1 | Curriculum and Programs | 97 | 100 | Best Practice |
| 3.2 | Facilities and Equipment | 60 | 60 | Best Practice |
| 3.3 | Student Assessment and Testing | 76 | 80 | Excellent |
| 3.4 | Aviation Program Instructors, Staff, and Management | 75 | 80 | Excellent |
| 4 | Not Reviewed at This Time | 0 | 0 | N/A |
| 5 | Not Reviewed at This Time | 0 | 0 | N/A |
| Total |  | 435 | 470 | Excellent |

**School has a 1, 5, 7, and 10-year strategic plan that aligns well with AT-CTI’s goals;  The school’s plans were reviewed by the state Governor**
**School’s leadership exhibited commitment to program**
**Student body is diverse, consisting of 40% female and 10% minority, and the faculty is 1/5 minority female**
**Diverse city of XX is a 2.5 hour drive from the campus; There are also several FAA facilities in the vicinity**
**Substantial financial and non-financial support; Multiple sources **
**School has a huge advisor program that supports the students **
**Excellent affiliations and relationships indicated**
**Targeted hiring and recruitment initiatives nationally and internationally **
**Flight school and other curriculum received multiple awards**
**Outstanding facility; Labs state-of-the-art; Labs open to students**
**Remedial or additional simulation or enrichment sessions available**
**Monthly proficiency tests for faculty and students; The CTO tower checkout serves as the capstone exam**
**Records were well organized and stored in secure cabinets**
**Hired additional faculty; Faculty required to maintained 16 hours of tower currency **

**This is a new program without a history of enrollment or results**
**There were no AT or aviation accreditation or awards mentioned**
**Lack of stringency in student selection criteria and process**
**There were no ATC scholarships available due to the newness of the program **
**Major staff turnover since initial application package mailed**

**SUMMARY OF SCORES**

## Conclusions

**The FAA has enhanced and expanded the    AT-CTI Program for the benefit of the FAA and the Program participants.**
**The FAA encourages qualified schools to apply for participation in the AT-CTI Program.**
**The FAA encourages schools to continue to improve their AT Programs based on best practices.**