THE UNIVERSITY OF 

NOTRE DAME’S ALLIANCE 

FOR CATHOLIC 

EDUCATION TEACHING 

FELLOWS PROGRAM 

Evaluation of Outcomes and Impact, 2015-2017

January 

2018 

Monica Kowalski, Ph.D. 

Stephen Ponisciak, Ph.D.

Gina Svarovsky, Ph.D. 

Program Evaluation and Research

Alliance for Catholic Education 

University of Notre Dame 
 

 

EXECUTIVE SUMMARY 

The ACE Teaching Fellows AmeriCorps program recruits, trains, and places excellent 

teachers in high needs urban and rural Catholic schools across the country, where they serve for 

two years while earning a Master of Education degree from the University of Notre Dame. Now 

in its 24th year, the program recently completed an evaluation guided by the following three 

evaluation questions: 

1. To what extent do ACE Teachers perceive growth in professional competence, 

according to their own self-assessment? 

2. To what extent do ACE Teachers grow in professional competence, according to 

expert observations of their practice? 

3. To what extent do students of ACE Teachers experience academic growth, according 

to performance on achievement tests and compared to students of other first and second year 

teachers? 

Question one was addressed by comparing responses on entry and exit surveys for a 

variety of measures, including preparedness for teaching and teachers’ sense of efficacy. Results 

showed significant growth on perceptions of preparedness for teaching for all items representing 

a variety of instructional tasks. Measures of teachers’ sense of efficacy did not show significant 

changes from entry to exit, which is consistent with prior research on the stability of efficacy 

beliefs from preservice through the first few years of teaching. 

Question two involved analysis of ratings from observations by program-affiliated 

academic supervisors and on-site school principals. ACE Teachers were rated each semester on a 

variety of indicators across four domains of practice: Planning and Preparation, Classroom 

Environment, Instruction, and Professional Responsibilities. The number of teachers scoring in 

the “Proficient” or “Exemplary” ratings for each domain was significantly higher in the fourth 

semester than in the first semester, showing substantial growth over the course of the program. 

Question three focused on the impact of the program for the student-level outcome of 

academic achievement. For a small sample of ACE Teachers, student test scores were obtained 

along with test scores of students of other beginning teachers in the same dioceses. Using this 

comparison group and a value-added modeling procedure, results showed that students of ACE 

Teachers demonstrated more growth in math and reading than students of other beginning 

teachers in one diocese and similar growth to the comparison group in another diocese. 

2 
The findings of this evaluation will be useful for the ACE Teaching Fellows program in 

several ways, including supporting recruitment by highlighting positive findings and informing 

strategy for increasing access to data from ACE schools to enable more comprehensive 

evaluation in the future. ACE Teaching Fellows will continue to engage in rigorous evaluation to 

better understand and improve its program to serve the mission of strengthening and sustaining 

Catholic education through national service. 

3 
 

 

 

 

TABLE OF CONTENTS 

Section Page 

Introduction 5 

Program Theory of Change 6 

Evaluation Questions 7 

Methods and Results 8 

Evaluation Question 1 8 

Evaluation Question 2 14 

Evaluation Question 3 24 

Conclusion 38 

References 39 

4 
 

INTRODUCTION: THE ALLIANCE FOR CATHOLIC EDUCATION 

The University of Notre Dame’s Alliance for Catholic Education has been operating the 

ACE Teaching Fellows AmeriCorps program since 1994. This graduate service teaching 

program recruits, trains, and places high-quality teachers in under-resourced Catholic schools 

throughout the country, where they serve as teachers of record for two years while earning their 

Master of Education (M.Ed.) degree from the University of Notre Dame. 

The ACE Teaching Fellows program was founded to respond to the problem of a lack of 

effective teachers in economically disadvantaged urban and rural Catholic schools across the 

country. Superintendents and principals approached Notre Dame seeking well-prepared 

educators for their schools, and ACE aimed to fulfil that need. Now, 24 years later, ACE 

continues to receive far more requests for teachers than we can fill each year, indicating that the 

need is still great for teachers in challenging placements within Catholic schools. 

The leaders of ACE Teaching Fellows have great confidence in the quality of the 

program, but are still committed to continuous improvement and continuing to demonstrate 

effectiveness through performance measurement and evaluation. Therefore, a rigorous evaluation 

was conducted from 2015-2017 to respond to questions of interest to internal and external 

stakeholders. This report describes the evaluation, with reference to the program theory of 

change, and presents the results and conclusions for informing future practice. 

5 
 

PROGRAM THEORY OF CHANGE 

The ACE Teaching Fellows AmeriCorps program is summarized with the following 

theory of change statement: 

If we recruit and form faith filled, talented, and effective teachers for under-resourced 

Catholic schools, then students will grow academically, spiritually, and socially; ACE 

teachers will experience positive and transformational growth; and school communities 

will experience newfound vitality. 

The program logic model includes inputs and activities including recruiting and enrolling 

applicants, providing an innovate M.Ed. graduate degree program with qualified faculty and 

support staff, placing teachers in high-needs schools, and providing ongoing professional 

development, supervision, and support to foster retention in the field of education. Through their 

service in approximately 100 schools throughout the country, both ACE Teachers (program 

participants) and their students (program beneficiaries) experience short- and long-term changes 

in knowledge, skills, and attitudes. Participants experience growth in professional teaching 

competence as well as personal and spiritual growth through the unique intentional community 

living aspect of the program. Their success and positive experience in the program will foster a 

life-long commitment to education and a high level of retention in the field. For beneficiaries, 

having a well-prepared and dedicated ACE Teacher will enable them to experience academic, 

personal, and spiritual growth and will contribute to the revitalization of the school community. 

While all components of the logic model are important and essential to the program, this 

evaluation focused on outcomes of ACE Teaching Fellows program on the program participants 

(AmeriCorps members) and the impact on their students (beneficiaries). The evaluation 

questions and methods are presented below. 

6 
 

 

 

 

 

 

 

 

 

 

 

 

 

EVALUATION QUESTIONS 

In order to conduct a holistic evaluation of the University of Notre Dame’s Alliance for 

Catholic Education (ACE) program, a multi-tiered set of evaluation questions were articulated at 

the outset of program design and award receipt. These questions were anchored in the theory of 

change that drives the ACE program and linked to the program activities, outputs, and outcomes 

of the program. 

The evaluation team considered both process and outcome-focused evaluation questions 

when initially drafting the evaluation plan, which would then provide feedback on several facets 

of program design, delivery, and impact. However, feedback from CNCS on the evaluation plan 

suggested a narrower focus, o with better alignment between study questions and methods. 

Considering this feedback, the evaluation team revised the evaluation questions and subsequent 

plan design. As a result, the study focused more specifically on questions related to outcomes for 

ACE Teachers (the AmeriCorps members) and the impact on their students (the program 

beneficiaries). 

The key evaluation questions for this report are: 

EQ1. Member Outcomes: To what extent do ACE Teachers perceive growth in their 

professional competence, according to their own self-assessment? 

EQ2. Member Outcomes: To what extent do ACE Teachers grow in their professional 

competence, according to expert observations of their pedagogy and practice? 

EQ3. Beneficiary Impact: To what extent do students of ACE Teachers experience 

academic growth, according to performance on achievement tests and compared to 

students of other first and second year teachers? 

Each question will be considered in turn below, with relevant methods and results presented. 

7 
 

 

METHODS AND RESULTS 

Evaluation Question 1 

To what extent do ACE Teachers perceive growth in their professional competence, according 

to their own self-assessment? 

The ACE Teaching Fellows AmeriCorps program develops effective professional 

educators during their two-year commitment. The extent to which members perceive growth in 

their professional competence is an indication of the overall success of ACE Teaching Fellows. 

Therefore, self-reported assessments of changes in competence and confidence for professional 

teaching skills are important considerations for the program. 

Method 

All admitted participants in the ACE Teaching Fellows program are invited to complete 

an entry survey once they arrive to campus to begin their summer coursework. The survey is 

administered electronically through Qualtrics survey software platform during orientation in the 

first weekend of the ACE summer. Since they are given time to complete the survey during one 

of the orientation sessions, 100% of ACE Teachers typically participate in the entry survey. 

All graduating ACE teachers are invited to complete an exit survey at the conclusion of 

their second year of ACE. The survey is administered electronically by email invitation through 

Qualtrics beginning the first week of June. Multiple reminder emails are sent to teachers to 

encourage their participation before the survey concludes in mid-July. Response rates are 

typically in the 75-95% participation range. 

The surveys were designed in collaboration with ACE faculty and staff and are updated 

annually to better reflect the data needs of the program. Surveys include both quantitative and 

qualitative (open-ended questions) data on aspects of the program including motivation for 

participating in ACE, professional teaching development, community experiences, personal 

growth, and vocational directions. Results are used to inform decision making related to 

recruitment efforts, admission criteria, placement of teachers, and ongoing support of teachers 

8 
 

 

 

 

 

throughout the program as well as for internal and external program evaluation concerns. This 

evaluation will focus on key questions concerning professional teaching development. 

Results 

The following results are based on data from the entry and exit surveys of ACE cohorts 

21 and 22. Table 1 shows the dates when these surveys were administered and the number of 

ACE Teachers completing each survey. 

Table 1. Entry and Exit Surveys of Cohorts 21 and 22 

ACE Cohort Entry Survey n Exit Survey n 

22 June 2015 96 June 2017 84 

21 June 2014 95 June 2016 88 

ACE Teachers provided their university-issued ID numbers on the surveys, which were 

used to match pre- and post- survey responses. All exit surveys from both cohorts were matched 

with pre-surveys, for final sample sizes of 84 and 88, respectively. Entry surveys without 

corresponding exit surveys were not included in the analyses. These either represented ACE 

Teachers who did not complete their 2-year AmeriCorps service or who did not respond to 

repeated requests to complete the exit survey. Paired samples t-tests were then conducted for 

each item. 

As a result of their education coursework, mentoring, and classroom practice during the 

two years in ACE, ACE Teachers experience tremendous growth as professional teachers. From 

the entry surveys to the exit surveys, ACE Teachers report significant increases in their level of 

preparedness for nearly all of the important instructional tasks that were measured (see Tables 2 

and 3, responses based on a rating scale of 1= very unprepared, to 5= very prepared). 

9 
Table 2. ACE Teaching Fellows Cohort 21 Preparedness for Instructional Tasks 

ENTRY EXIT 

Instructional Task 

M SD M SD 

Meet the overall demands of teaching* 2.91 0.98 3.69 0.86 

Develop learning experiences on standards* 2.40 0.96 3.63 0.82 

Vary instruction* 2.85 0.94 3.67 0.88 

Maintain an organized classroom* 2.99 0.48 3.94 0.87 

Integrate culture* 2.77 0.90 3.53 0.85 

Differentiate instruction 2.55 0.92 3.16 0.94 

Support English Language Learners* 2.39 1.05 3.14 0.96 

Integrate technology 2.81 0.96 3.42 0.91 

Communicate with parents of students* 3.11 0.94 3.76 0.90 

Build a classroom community* 3.48 0.91 4.05 0.89 

Effectively teach religion* 3.17 0.96 3.64 0.95 

Note. * indicates that the comparison between entry and exit surveys is statistically significant at 

the .05 level. 

10 
 

 

 

 

Table 3. ACE Teaching Fellows Cohort 22 Preparedness for Instructional Tasks 

ENTRY EXIT 

Instructional Task 

M SD M SD 

Meet the overall demands of teaching* 2.66 0.89 3.75 0.71 

Develop learning experiences on standards* 2.42 0.93 3.59 0.88 

Vary instruction* 2.59 0.84 3.85 0.72 

Maintain an organized classroom* 2.72 0.97 3.94 0.81 

Integrate culture* 2.62 0.98 3.65 0.80 

Differentiate instruction* 2.40 0.88 3.51 0.79 

Support English Language Learners* 2.10 0.92 3.38 0.94 

Integrate technology* 2.79 1.03 3.49 0.81 

Communicate with parents of students* 2.99 0.92 3.72 0.88 

Build a classroom community* 3.08 0.98 4.01 0.82 

Effectively teach religion* 2.84 0.98 3.59 0.85 

Note. * indicates that the comparison between entry and exit surveys is statistically significant at 

the .05 level. 

As Tables 2 and 3 show, ACE Teachers reported statistically significant increases in their 

level of preparedness for nearly all instructional tasks that were measured in the survey. All of 

these are essential skills for effective teachers in Catholic schools. 

To more accurately assess the professional growth of ACE Teachers, an established 

measure of teacher sense of efficacy (Tschannen-Moran & Woolfolk Hoy, 2001) was included 

with demonstrated reliability and validity in the entry and exit surveys. Teacher sense of efficacy 

refers to teachers’ perceptions of their ability to promote student learning and effectively get 

through to all of their students. Research has shown that teacher sense of efficacy is positively 

related to factors such as student achievement and motivation, teacher enthusiasm, persistence, 

and satisfaction with teaching (Midgley, Feldlaufer, &Eccles, 1989; Ross, 1994; TschannenMoran &Woolfolk Hoy, 2001; Woolfolk Hoy, Hoy, & Davis, 2009). Table 4 show the 

11 
 

 

 

 

comparison of pre- and post- survey results for this measure for Cohort 22 (the measure was not 

included in the Cohort 21 surveys). 

Table 4. Teacher Sense of Efficacy for Cohort 22 

ENTRY EXIT 

Efficacy Belief: How much can you do to... 

M SD M SD 

control disruptive behavior in the classroom? 3.88 0.63 4.03 0.61 

motivate students with low interest in school? 3.78 0.79 3.54 0.84 

get students to believe they can do well?* 4.17 0.70 3.87 0.70 

help students value learning?* 4.04 0.72 3.67 0.70 

craft good questions for your students? 4.20 0.74 4.35 0.63 

get children to follow classroom rules?* 3.82 0.53 4.00 0.65 

calm a student who is disruptive or noisy? 3.67 0.60 3.86 0.60 

establish a classroom management system?* 3.75 0.76 4.09 0.66 

use a variety of assessment strategies? 3.84 0.84 4.05 0.71 

provide alternative explanations? 4.16 0.71 4.32 0.68 

assist families in helping their children? 3.51 0.84 3.60 0.85 

implement alternative strategies?* 3.63 0.63 3.88 0.83 

Total Efficacy Scale (alpha=0.90) 3.87 0.52 3.94 0.42 

Table 4 shows that most individual items and the total calculated scale for teacher sense 

of efficacy (mean of all items) did not change significantly from the pre- to the post-survey. That 

is, teachers reported feeling relatively equally capable of promoting learning and reaching all of 

their students before they started teaching as at the end of their second year in the classroom. 

Prior research shows that once teacher sense of efficacy is established in preservice teachers, 

largely based on their experiences as students (known as “apprenticeship of observation”) and 

their general self-confidence, it appears to be resistant to change (Woolfolk Hoy, 2000). 

Nevertheless, five individual items on the scale did result in statistically significant 

differences from the pre- to post- survey, but these results were in mixed directions. Two items 

were rated higher during the pre-survey than the post-survey; getting students to believe they can 

12 
do well and helping students value learning. This suggests that teachers may be overconfident in 

their abilities to shape students’ own perceptions of their competence and value of learning prior 

to teaching. Prior research shows that preservice teachers are generally optimistic and that their 

sense of efficacy often decreases upon beginning teaching as the realities and complexities of 

teaching set in (Woolfolk Hoy, 2000). 

However, three of the items were rated higher at the time of the post-survey; getting 

children to follow rules, establishing a classroom management system, and implementing 

alternative strategies to reach students. These items suggest that ACE Teachers do experience 

real gains in teaching competence during their time in the program, as evidenced by an increase 

in efficacy for these areas of teaching. 

Evaluation Question 1 Summary 

Analysis of entry and exit surveys for ACE Teachers shows that the teachers experience 

self-reported growth in instructional competence during their time in the ACE Teaching Fellows 

Program. For both cohorts included in this study, the ability to “build a classroom community” 

was highest rated in terms of preparedness at the time of the post survey. The ACE program 

greatly emphasizes the importance of working to establish and maintain a positive classroom 

community, so it is a point of pride that our teachers indicate they feel well prepared in this area. 

The lowest rated instructional task for both cohorts was “supporting English language 

learners.” We recognize that this is a difficult skill for any teacher and a true challenge in modern 

American education. We will continue to seek ways to assist our teachers in responding to this 

challenge, such as through additional seminars and courses targeted to teaching English as a new 

language. 

13 
 

 

Evaluation Question 2 

To what extent do ACE Teachers grow in their professional competence, according to expert 

observations of their pedagogy and practice? 

ACE aims to train and place high quality teachers in Catholic schools across the country 

in order to provide the best possible education for students in need. Teacher quality is often cited 

as the most significant factor contributing to student academic performance (Harris & 

McCaffrey, 2010; Sanders & Horn, 1998). One indicator of teacher quality is their actual 

practice and performance in the classroom, which moves beyond what teachers know to what 

they do and how they enact knowledge in practice. Accordingly, teacher performance 

assessments can be viewed as measures of teacher quality (Caughlan & Jiang, 2014). These 

observations of actual teaching practice are invaluable, since ideally “performance assessments 

are regarded as part of the learning process for teacher candidates” (p. 375). Used both 

formatively and summatively, such performance assessments provide one feedback channel for 

measuring teachers’ growth and development. 

Method 

Academic Supervision Evaluations by Faculty 

EDU 65950: Supervised Teaching (Supervision) is a course that centers on the over 3,000 

hours of classroom teaching completed over the two years of ACE. Assigned faculty of 

supervision and instruction visit each school site to monitor ACE Teachers' professional 

development and progress towards addressing and meeting performance indicators. During the 

site visits, additional sources will provide information about ACE teachers’ progress: postobservation meetings with the ACE AmeriCorps teacher, examination of instructional and 

classroom management materials, examination and discussion of content course assignments, 

and meetings with the Mentor Teacher, Principal, and diocesan Superintendent. Documentation 

of site visits occurs via a two-year site visit form that standardizes field notes and aligns a twoyear narrative with performance indicators. Upon reviewing all of the relevant documents and 

sources, the faculty of supervision and instruction rate applicable performance indicators. These 

14 
 

 

ratings correlate to a course grade. The satisfactory meeting of performance indicators provides 

evidence of meeting corresponding State of Indiana Standards for Educators. 

The performance indicators form a framework for feedback and evaluation throughout 

each semester based on Mentor Teacher and Principal instruments (at the local school level), 

faculty site visits and classroom observations, observed teaching artifacts, discussion, reflective 

writing, and content coursework (during the second year). Given this body of evidence, the 

faculty of supervision and instruction rate the appropriate indicators and assign a course grade. 

The number and specific performance indicators are differentiated across the four semesters – 

these increase in number and shift in focus as the ACE Teacher gains in experience. 

The performance indicators are organized in four domains: Planning and Preparation, 

Classroom Environment, Instruction, and Professional Responsibilities. Domain one focuses on 

Planning and Preparation and assesses indicators such as whether the teacher demonstrates 

knowledge of content and pedagogy and is able to select instructional objectives. Domain two 

attends to “Classroom Environment” and measures indicators such as whether the teacher 

manages classroom procedures and manages student behavior. Domain three focuses on 

“Instruction” and tracks indicators such as engagement of students in learning and how well 

teachers provide feedback to students. The fourth instructional domain is referred to as 

“Professional Responsibilities,” and it gauges indicators such as whether a teacher maintains 

accurate records, their effectiveness at communicating with parents and guardians, and whether 

they demonstrate professionalism. 

The performance indicator rating scale ranges from “unsatisfactory” to “exceptional” 

performance in meeting all descriptors pertaining to a performance indicator. This latter, 

exceptional, rating is rare and denotes mastery beyond general beginning teacher level. For 

purposes of course grading, “proficient” is standardized to the A range. For any given semester, a 

minimum threshold number of performance indicators must be scored in the “proficient” range, 

while a maximum number of “basic” ratings and no indicators rated at the “unsatisfactory” level 

contribute to an A grade. Table 5 contains the rating scale aligned with the performance 

indicators. 

15 
 

 

Rating Descriptor 

Unsatisfactory Limited evidence, from site visits, supervision course reflections and 

assignments, principal evaluations, mentor feedback, and content 

coursework that some descriptors relating to the performance indicator are 

being met or addressed. In light of this rating, improvement goals 

continue to be maintained or revised. 

Basic Evidence, from these multiple sources, that descriptors relating to the 

performance indicator are being met and addressed through teaching 

practices-performances at a basic level of proficiency. Professional 

development goals are being maintained or may be revised. 

Proficient Evidence, from these multiple sources, that observed descriptors relating 

to the performance indicator are fully met and/or maintained through 

teaching practices-performances. Professional development goals are 

being maintained or may be revised. 

Exceptional Evidence, from multiple sources, that all descriptors relating to the 

performance indicator are not only maintained at a proficient level, but 

some or all of these areas continue to be improved upon or met through 

new and unique teaching practices-performances. 

Table 5. Performance Indicator Ratings and Descriptions 

For purposes of program evaluation, ACE tracks the percentage of teachers who have 

been rated “proficient” and above in each performance indicator each semester, with the goal of 

each ACE Teaching Fellow scoring in the “proficient” and above range for each performance 

indicator by the end of the fourth semester. 

16 
 

 

 

 

 

 

Academic Supervision Evaluations by Principals 

The school principal (or designee) supports the ACE Teacher and serves as “clinical 

supervisor” with an evaluative role. The administrator provides support for the ACE teacher 

much as would be provided to any new teacher in the building; providing appropriate policies, 

handbooks, curriculum guides and generally orienting them to the school and community. In 

addition, fulfilling the role of evaluator requires that the principal (or designee) observe and meet 

with the ACE teacher. Most principals observe and conference with the ACE teacher two or 

three times each semester. 

Once each semester, principals complete an evaluation of their ACE Teacher via an 

electronic evaluation form. The form requires principals to rate ACE Teachers on the same 

differentiated performance indicators as the Academic Supervision Evaluations. The results are 

recorded on a two-year tracking sheet for each ACE Teacher and summarized each semester for 

each cohort of ACE Teaching Fellows. Similar to the Academic Supervision Evaluations by 

Faculty, the results are used for program and individual teacher evaluation. 

Results 

In the academic years 2015-2016 and 2016-2017, ACE graduated two cohorts of 

teachers, ACE 21 and ACE 22. Given that these two cohorts are the only two who have 

completed the program in these academic years, the results and data analysis will focus only on 

these two cohorts. Cohort ACE 23 taught in academic year 2016-2017 but will not graduate from 

ACE until May 2018. 

Academic Supervision Evaluations for ACE 21 

The summary table of ACE 21’s Academic Supervision Evaluations is included below. 

Reviewing the table, the growth of ACE 21 AmeriCorps teachers from semester 1 to semester 4 

is evident. In Semester 4, even with the increased number and scope of performance indicators, 

far more teachers were rated proficient or above than in semester 1. 

ACE AmeriCorps teachers showed statistically significant growth in all four domains 

from Fall 2014 to Spring 2016. Table 6 summarizes the results of a z-Test for the difference in 

the proportion of students scoring in the “proficient” and “exceptional” categories at two time 

17 
Fall 2014 N 2014 Spring 2016 N 2016 B-A Standard Z Calc p-value Domain 

Error 

63% 72 92% 72 29% 0.066 4.482 0.000* 1 

70% 72 97% 72 27% 0.057 4.695 0.000* 2 

59% 72 87% 72 28% 0.071 3.912 0.000* 3 

76% 72 96% 72 20% 0.055 3.697 0.000* 4 

 

 

points (i.e., 1st and 4th semester) by domain. The growth shown by ACE 21 AmeriCorps 

members is statistically significant for all four domains. This substantial growth across all four 

domains is an important goal for ACE in teacher formation. 

ACE 21 Teaching Fellows Supervisor Evaluation - Results 

E = Exceptional P = Proficient, B = Basic, U = Unsatisfactory 

n = 72 n = 72 n = 72 n = 72 

E P B U E P B U E P B U E P B U 

Pillar I Professional Teaching 

Domain 1: Planning and Preparation 

1. Demonstrates knowledge of content and pedagogy 0% 56% 44% 0% 1% 72% 26% 0% 4% 93% 4% 0% 8% 86% 6% 0% 

2. Demonstrate knowledge of students 0% 89% 11% 0% 3% 89% 8% 0% 

3. Designs coherent unit-based instruction 0% 74% 26% 0% 4% 82% 14% 0% 

4. Selections instructional objectives 0% 69% 31% 0% 0% 83% 17% 0% 1% 94% 4% 0% 4% 90% 6% 0% 

5. Selects instructional outcomes/goals 0% 86% 14% 0% 4% 86% 10% 0% 

6. Designs assessments to provide evidence of learning 1% 72% 26% 0% 0% 88% 13% 0% 

7. Demonstrates knowledge of resources 0% 60% 40% 0% 0% 97% 3% 0% 1% 97% 1% 0% 

Domain 2: The Classroom Environment 

1. Establishes a culture for learning 0% 65% 35% 0% 11% 76% 13% 0% 15% 79% 6% 0% 

2. Manages classroom procedures 0% 74% 26% 0% 0% 88% 13% 0% 3% 93% 4% 0% 6% 93% 1% 0% 

3. Manages student behavior 0% 57% 40% 3% 0% 68% 32% 0% 0% 94% 6% 0% 3% 94% 3% 0% 

4. Organizes physical space 0% 81% 19% 0% 1% 94% 4% 0% 6% 93% 1% 0% 8% 90% 1% 0% 

Domain 3: Instruction 

1. Communicates clearly and accurately 0% 85% 15% 0% 0% 89% 11% 0% 0% 92% 8% 0% 0% 92% 8% 0% 

2. Uses questioning and discuss techniques 0% 47% 53% 0% 0% 57% 43% 0% 1% 79% 19% 0% 1% 83% 15% 0% 

3. Engages students in learning 0% 56% 44% 0% 0% 71% 29% 0% 3% 82% 15% 0% 7% 79% 14% 0% 

4. Provides feedback to students 0% 65% 35% 0% 0% 83% 17% 0% 0% 92% 8% 0% 0% 96% 4% 0% 

5. Modifies instruction according to developmental level, 

language proficiency, and instructional needs of students 0% 13% 88% 0% 0% 64% 36% 0% 1% 89% 10% 0% 

6. Modifies instruction for children with learning 

exceptionalities 0% 17% 83% 0% 1% 68% 31% 0% 

7. Assesses Student Learning 0% 42% 58% 0% 0% 60% 40% 0% 0% 90% 10% 0% 1% 86% 13% 0% 

Domain 4: Professional Responsibilities 

1. Maintains accurate records 0% 88% 13% 0% 0% 93% 7% 0% 0% 94% 6% 0% 0% 99% 1% 0% 

2. Communicate with parents and guardians 0% 78% 22% 0% 0% 89% 11% 0% 3% 93% 4% 0% 3% 97% 0% 0% 

3. Shows professionalism 0% 92% 8% 0% 0% 93% 7% 0% 3% 94% 3% 0% 10% 88% 3% 0% 

Pillar II Community 

1. Contributes to the school community 3% 93% 4% 0% 10% 88% 3% 0% 24% 76% 0% 0% 

2. Contributes to the larger community 7% 75% 18% 0% 

3. Promotes student engagement with community resources 1% 57% 42% 0% 4% 85% 11% 0% 

Pillar III Spirituality 

1. Creates environment of respect and rapport 0% 82% 18% 0% 0% 89% 11% 0% 4% 92% 4% 0% 14% 86% 0% 0% 

2. Fosters character and ethical development 6% 94% 0% 0% 

3. Fosters spiritual development in children 7% 90% 3% 0% 

4. Serves as spiritual and ethical role model 18% 82% 0% 0% 

Semester 1 (Fall 2014) Semester 2 (Spring 2015) Semester 3 (Fall 2015) Semester 4 (Spring 2016) 

Table 6. Cohort ACE 21 Supervision Results z-Test 

Note. * indicates that the change in proportion of students scoring proficient or excellent 

between the first and fourth semesters is statistically significant at the .05 level. 

18 
 

 

 

 

 

 

 

Academic Supervision Evaluations for ACE 22 

The summary table of ACE 22’s Academic Supervision Evaluations is included below. 

Once again, the growth of ACE 22 teachers from Semester 1 to Semester 4 is clear across all 

performance indicators. Table 7 summarizes the results of a z- Test for the proportion of students 

in semester 1 vs. semester 4 scoring in the “proficient” and “exceptional” categories. The growth 

shown by ACE 22 Teachers is again statistically significant in all four domains. ACE 22 

Teachers especially excelled at “organizing physical spaces,” “shows professionalism,” 

“contributes to the school community,” and all indicators of spirituality by Semester 4, with 15% 

or more of ACE 22 teachers rated as “exceptional” in these performance indicators. 

ACE 22 Teaching Fellows Supervisor Evaluation - Results 

E = Exceptional P = Proficient, B = Basic, U = Unsatisfactory 

n = 75 n = 75 n = 75 n = 75 

E P B U E P B U E P B U E P B U 

Pillar I Professional Teaching 

Domain 1: Planning and Preparation 

1. Demonstrates knowledge of content and pedagogy 0% 72% 28% 0% 1% 83% 15% 1% 4% 96% 0% 0% 12% 87% 1% 0% 

2. Demonstrate knowledge of students 5% 81% 13% 0% 11% 87% 3% 0% 

3. Designs coherent unit-based instruction 4% 55% 41% 0% 5% 75% 20% 0% 

4. Selections instructional objectives 0% 64% 36% 0% 0% 87% 13% 0% 0% 89% 11% 0% 4% 95% 1% 0% 

5. Selects instructional outcomes/goals 3% 61% 36% 0% 8% 77% 15% 0% 

6. Designs assessments to provide evidence of learning 3% 75% 23% 0% 8% 85% 7% 0% 

7. Demonstrates knowledge of resources 3% 81% 16% 0% 5% 91% 4% 0% 12% 85% 3% 0% 

Domain 2: The Classroom Environment 

1. Establishes a culture for learning 3% 56% 41% 0% 4% 80% 16% 0% 12% 79% 9% 0% 

2. Manages classroom procedures 0% 71% 29% 0% 1% 80% 19% 0% 3% 96% 3% 0% 8% 92% 0% 0% 

3. Manages student behavior 0% 43% 57% 0% 0% 68% 31% 1% 3% 91% 7% 0% 1% 97% 1% 0% 

4. Organizes physical space 0% 83% 17% 0% 3% 84% 13% 0% 3% 95% 3% 0% 15% 85% 0% 0% 

Domain 3: Instruction 

1. Communicates clearly and accurately 0% 73% 27% 0% 0% 77% 23% 0% 1% 92% 7% 0% 7% 89% 4% 0% 

2. Uses questioning and discuss techniques 0% 59% 41% 0% 0% 75% 25% 0% 1% 88% 11% 0% 0% 91% 9% 0% 

3. Engages students in learning 0% 59% 41% 0% 3% 72% 25% 0% 4% 88% 8% 0% 8% 88% 4% 0% 

4. Provides feedback to students 0% 76% 24% 0% 1% 81% 17% 0% 3% 89% 8% 0% 5% 95% 0% 0% 

5. Modifies instruction according to developmental level, 

language proficiency, and instructional needs of students 0% 37% 63% 0% 3% 68% 29% 0% 1% 91% 8% 0% 

6. Modifies instruction for children with learning 

exceptionalities 0% 37% 63% 0% 0% 55% 45% 0% 

7. Assesses Student Learning 0% 53% 47% 0% 0% 69% 31% 0% 1% 87% 12% 0% 7% 85% 8% 0% 

Domain 4: Professional Responsibilities 

1. Maintains accurate records 0% 85% 15% 0% 3% 81% 16% 0% 4% 89% 7% 0% 12% 88% 0% 0% 

2. Communicate with parents and guardians 0% 79% 21% 0% 0% 84% 16% 0% 1% 96% 3% 0% 4% 96% 0% 0% 

3. Shows professionalism 0% 97% 3% 0% 3% 84% 12% 1% 7% 89% 4% 0% 19% 80% 1% 0% 

Pillar II Community 

1. Contributes to the school community 1% 92% 7% 0% 3% 96% 1% 0% 20% 77% 3% 0% 

2. Contributes to the larger community 8% 67% 25% 0% 

3. Promotes student engagement with community resources 0% 75% 25% 0% 8% 77% 15% 0% 

Pillar III Spirituality 

1. Creates environment of respect and rapport 0% 89% 11% 0% 1% 89% 9% 0% 3% 96% 1% 0% 19% 81% 0% 0% 

2. Fosters character and ethical development 15% 85% 0% 0% 

3. Fosters spiritual development in children 15% 85% 0% 0% 

4. Serves as spiritual and ethical role model 25% 75% 0% 0% 

Semester 1 (Fall 2015) Semester 2 (Spring 2016) Semester 3 (Fall 2016) Semester 4 (Spring 2017) 

19 
 

Table 7. Cohort ACE 22 Supervision Results z-Test 

Fall 2015 N 2015 Spring 2017 N 2017 B-A Standard Z Calc p-value Domain 

Error 

68% 75 93% 75 25% 0.061 4.061 0.000* 1 

65% 75 97% 75 32% 0.058 5.516 0.000* 2 

64% 75 89% 75 25% 0.068 3.732 0.000* 3 

81% 75 96% 75 15% 0.051 2.886 0.004* 4 

Note. * indicates that the change in proportion of Teachers scoring proficient or excellent 

between the first and fourth semesters is statistically significant at the .05 level. 

Principal Evaluations of ACE 21 

Results of principal evaluations for ACE 21 Teachers are included below. As is evident 

in Table 8, 96%, 95%, 91%, and 95% of ACE AmeriCorps teachers are rated by principals as 

proficient or exceptional across the four respective domains by the 4th semester of teaching. The 

table also summarizes the results of the z-Test comparison for the difference in the proportion of 

students scoring in the proficient or exceptional categories. ACE 21 Teachers did show clear 

growth in each of the four domains from semester one to semester four, and that growth was 

statistically significant for three of the four domains. It should be noted that high initial ratings 

by principals makes it less likely for significant changes to be found. 

20 
 

 

ACE 21 Teaching Fellows Principal Evaluation - Results 

A = Advanced P = Proficient, B = Basic, U = Unsatisfactory 

n = 69 n = 66 n = 62 n = 58 

A P B U A P B U A P B U A P B U 

Pillar I Professional Teaching 

Domain 1: Planning and Preparation 

1. Demonstrates knowledge of content and pedagogy 41% 49% 10% 0% 45% 45% 9% 0% 63% 35% 0% 2% 62% 36% 2% 0% 

2. Demonstrate knowledge of students 47% 52% 2% 0% 55% 45% 0% 0% 

3. Designs coherent unit-based instruction 47% 44% 6% 2% 53% 41% 5% 0% 

4. Selections instructional objectives 22% 65% 13% 0% 29% 56% 15% 0% 52% 42% 5% 2% 59% 36% 5% 0% 

5. Selects instructional outcomes/goals 52% 45% 2% 2% 52% 43% 5% 0% 

6. Designs assessments to provide evidence of learning 45% 45% 6% 2% 53% 40% 7% 0% 

7. Demonstrates knowledge of resources 33% 52% 14% 0% 47% 47% 5% 2% 55% 40% 5% 0% 

Domain 2: The Classroom Environment 

1. Establishes a culture for learning 39% 48% 12% 0% 61% 32% 5% 2% 62% 34% 3% 0% 

2. Manages classroom procedures 30% 43% 25% 1% 32% 48% 18% 2% 52% 37% 10% 2% 53% 41% 5% 0% 

3. Manages student behavior 26% 41% 30% 3% 27% 47% 24% 2% 47% 35% 16% 2% 50% 45% 5% 0% 

4. Organizes physical space 25% 49% 25% 1% 29% 55% 17% 0% 47% 42% 11% 0% 53% 40% 7% 0% 

Domain 3: Instruction 

1. Communicates clearly and accurately 30% 51% 19% 0% 39% 52% 9% 0% 48% 45% 5% 2% 57% 40% 3% 0% 

2. Uses questioning and discuss techniques 33% 43% 23% 0% 38% 50% 12% 0% 53% 37% 8% 2% 66% 31% 3% 0% 

3. Engages students in learning 38% 36% 26% 0% 41% 45% 14% 0% 60% 34% 5% 2% 62% 33% 5% 0% 

4. Provides feedback to students 23% 45% 30% 0% 29% 48% 21% 0% 60% 35% 0% 2% 55% 41% 3% 0% 

5. Modifies instruction according to developmental level, 

language proficiency, and instructional needs of students 21% 42% 36% 0% 37% 31% 24% 3% 36% 47% 12% 0% 

6. Modifies instruction for children with learning 

exceptionalities 31% 26% 23% 3% 40% 36% 10% 0% 

7. Assesses Student Learning 19% 51% 30% 0% 26% 53% 20% 2% 45% 47% 6% 2% 50% 45% 5% 0% 

Domain 4: Professional Responsibilities 

1. Maintains accurate records 20% 54% 25% 0% 32% 55% 14% 0% 40% 55% 3% 2% 55% 40% 5% 0% 

2. Communicate with parents and guardians 20% 59% 20% 0% 32% 45% 20% 0% 39% 50% 10% 2% 48% 45% 7% 0% 

3. Shows professionalism 45% 46% 9% 0% 56% 36% 8% 0% 58% 34% 6% 2% 72% 22% 5% 0% 

Pillar II Community 

1. Contributes to the school community 59% 29% 12% 0% 69% 26% 5% 0% 78% 21% 2% 0% 

2. Contributes to the larger community 62% 26% 7% 0% 

3. Promotes student engagement with community resources 48% 32% 13% 0% 43% 47% 7% 0% 

Pillar III Spirituality 

1. Creates environment of respect and rapport 42% 43% 14% 0% 33% 52% 14% 0% 58% 32% 8% 2% 67% 31% 2% 0% 

2. Fosters character and ethical development 74% 24% 2% 0% 

3. Fosters spiritual development in children 62% 34% 3% 0% 

4. Serves as spiritual and ethical role model 71% 29% 0% 0% 

Semester 1 (Fall 2014) Semester 2 (Spring 2015) Semester 3 (Fall 2015) Semester 4 (Spring 2016) 

Table 8. ACE 21Principal Observation Results z-Test 

Fall 2014 N 2014 Spring 2016 N 2016 B-A Standard Z Calc p-value Domain 

Error 

88% 69 96% 58 7% 0.047 1.587 0.112 1 

71% 69 95% 58 23% 0.062 3.785 0.000* 2 

74% 69 91% 58 17% 0.065 2.661 0.008* 3 

83% 69 95% 58 13% 0.054 2.343 0.019* 4 

Note. * indicates that the change in proportion of students scoring proficient or excellent 

between the first and fourth semesters is statistically significant at the .05 level. 

21 
 

 

 

 

 

 

Principal Evaluations of ACE 22 

Principals rated ACE 22 teachers in each semester and results are shown below. Table 9 

summarizes the changes in ratings from semester one to semester four. Principals were 

enthusiastic about their ACE Teachers’ performance, rating them with high levels of 

“Advanced” by semester four. Some indicators had up to 73% of teachers being rated as 

“Advanced.” As the z-score table indicates, ACE Teachers have once again shown statistically 

significant growth over time in professional teaching practice across three of the four domains of 

our performance indicators as rated by their principals. 

ACE 22 Teaching Fellows Principal Evaluation - Results 

A = Advanced, P = Proficient, B = Basic, U = Unsatisfactory 

n = 70 n = 65 n = 69 n = 63 

A P B U A P B U A P B U A P B U Pillar I Professional Teaching 

Domain 1: Planning and Preparation 

1. Demonstrates knowledge of content and pedagogy 27% 57% 16% 0% 31% 60% 9% 0% 57% 42% 1% 0% 57% 43% 0% 0% 

2. Demonstrate knowledge of students 43% 55% 1% 0% 54% 43% 3% 0% 

3. Designs coherent unit-based instruction 39% 58% 3% 0% 48% 48% 5% 0% 

4. Selections instructional objectives 23% 60% 17% 0% 37% 57% 6% 0% 41% 58% 1% 0% 56% 41% 3% 0% 

5. Selects instructional outcomes/goals 38% 61% 1% 0% 49% 48% 3% 0% 

6. Designs assessments to provide evidence of learning 30% 64% 6% 0% 44% 44% 10% 0% 

7. Demonstrates knowledge of resources 28% 55% 17% 0% 45% 49% 6% 0% 52% 41% 6% 0% 

Domain 2: The Classroom Environment 

1. Establishes a culture for learning 42% 45% 14% 0% 54% 45% 1% 0% 63% 33% 3% 0% 

2. Manages classroom procedures 19% 51% 27% 3% 31% 45% 25% 0% 49% 41% 10% 0% 51% 43% 6% 0% 

3. Manages student behavior 13% 44% 40% 3% 23% 45% 32% 0% 36% 51% 13% 0% 40% 51% 10% 0% 

4. Organizes physical space 23% 61% 16% 0% 22% 57% 22% 0% 46% 46% 7% 0% 46% 49% 5% 0% 

Domain 3: Instruction 

1. Communicates clearly and accurately 30% 54% 16% 0% 40% 49% 11% 0% 51% 48% 1% 0% 59% 37% 5% 0% 

2. Uses questioning and discuss techniques 23% 47% 29% 0% 34% 46% 20% 0% 45% 51% 4% 0% 59% 38% 3% 0% 

3. Engages students in learning 33% 41% 26% 0% 43% 45% 12% 0% 45% 54% 1% 0% 60% 37% 3% 0% 

4. Provides feedback to students 27% 53% 19% 0% 29% 52% 18% 0% 48% 45% 7% 0% 49% 46% 5% 0% 

5. Modifies instruction according to developmental level, 

language proficiency, and instructional needs of students 23% 46% 29% 0% 26% 48% 22% 0% 43% 40% 17% 0% 

6. Modifies instruction for children with learning 

exceptionalities 19% 42% 28% 0% 37% 40% 17% 0% 

7. Assesses Student Learning 21% 50% 26% 0% 28% 51% 22% 0% 29% 65% 6% 0% 46% 48% 6% 0% 

Domain 4: Professional Responsibilities 

1. Maintains accurate records 23% 53% 21% 0% 35% 49% 14% 0% 36% 57% 7% 0% 51% 44% 5% 0% 

2. Communicate with parents and guardians 26% 50% 24% 0% 35% 49% 15% 0% 49% 39% 12% 0% 52% 41% 5% 2% 

3. Shows professionalism 54% 39% 7% 0% 52% 38% 9% 0% 68% 26% 6% 0% 67% 25% 8% 0% 

Pillar II Community 

1. Contributes to the school community 57% 38% 5% 0% 68% 29% 3% 0% 71% 29% 0% 0% 

2. Contributes to the larger community 57% 33% 3% 0% 

3. Promotes student engagement with community resources 39% 39% 19% 0% 46% 43% 10% 0% 

Pillar III Spirituality 

1. Creates environment of respect and rapport 44% 44% 11% 0% 51% 40% 8% 2% 61% 36% 3% 0% 62% 33% 5% 0% 

2. Fosters character and ethical development 65% 32% 3% 0% 

3. Fosters spiritual development in children 65% 29% 5% 0% 

4. Serves as spiritual and ethical role model 73% 22% 5% 0% 

Semester 1 (Fall 2015) Semester 2 (Spring 2016) Semester 3 (Fall 2016) Semester 4 (Spring 2017) 

22 
 

 

 

 

Table 9. ACE 22 Principal Observation Results z-Test 

Fall 2015 N 2015 Spring 2017 N 2017 B-A Standard Z Calc p-value Domain 

Error 

84% 70 95% 63 12% 0.051 2.311 0.021* 1 

70% 70 94% 63 24% 0.062 3.793 0.000* 2 

76% 70 91% 63 15% 0.063 2.386 0.017* 3 

84% 70 94% 63 10% 0.053 1.926 0.054 4 

Note. * indicates that the change in proportion of students scoring proficient or excellent 

between the first and fourth semesters is statistically significant at the .05 level. 

Evaluation Question 2 Summary 

While the overall growth and success of ACE 21 and 22 is affirming, there are individual 

areas for potential improvement that are highlighted by the data summary tables. The lowest 

rated performance indicator for both ACE cohorts in ratings from academic supervisors was 

“Modifies instruction for children with learning exceptionalities.” While this is understandably a 

difficult skill to master as a second-year teacher, ACE AmeriCorps teachers do have a threecredit summer course on this exact topic. Looking forward, we will seek to better integrate this 

course with the vast knowledge and resources of ACE’s new Program for Inclusive Education 

(PIE). The PIE faculty and staff have years of practical experience designing modified 

instruction plans in elementary and high school classrooms. This practical focus and expertise 

will help our ACE Teachers better meet the diverse needs of their students. 

Principal evaluations are generally more positive than academic supervisor ratings, 

perhaps because principals spend more time with the ACE Teachers at the school site and 

observe their teaching more frequently than the academic supervisors. We are especially pleased 

to note that 78% of ACE 21 and 71% of ACE 22 Teachers were rated “advanced” at 

“Contributes to the school community,” a hallmark of the ACE Teaching Fellows program. ACE 

Teachers are expected to go above and beyond in their contributions to the school community: 

chairing clubs, coaching sports, tutoring students, and organizing fundraisers. ACE Teachers 

truly impact the school communities in which they serve as AmeriCorps members. 

23 
 

 

 

 

Evaluation Question 3 

To what extent do students of ACE Teachers experience academic growth, according to 

performance on achievement tests and compared to students of other first and second year 

teachers? 

We assess year-to-year changes in academic achievement to evaluate whether students 

who were taught by ACE Teachers demonstrate improved academic performance in literacy 

and/or math, and whether that performance is better than that of students of non-ACE teachers 

with similar levels of experience. For this initial exploration of student achievement, we 

established data sharing agreements with two partner dioceses, who have a total of 16 ACE 

Teachers from cohorts 20, 21, and 22, who completed their service in summer 2015, 2016 and 

2017, respectively. We used student test scores from the 2014-15, 2015-16 and 2016-17 school 

years to estimate student growth models. 

Data 

Test scores come from the annual fall administration of the Iowa Assessments, taken by 

all students in grades 3-8 in diocese 1, and grades 2-8 in diocese 2. Match rates (the proportion of 

students with a posttest who have a matching pretest in the prior grade and year) are shown in 

Table 10 below, and are consistent with those in other school systems in previous studies across 

the nation (Mathematica Policy Research, 2014). Students may be missing a pretest or a posttest 

for many reasons, including absence or moving out of the diocese. 

Table 10: Match Rates for Each Diocese 

Post-test Match Rate (2016- Number of Students Match Rate (2016-17 Number of Students 

Grade 17 Diocese 1) (Diocese 1) Diocese 2) (Diocese 2) 

3 NA NA 0.87 344 

4 0.90 880 0.91 337 

5 0.90 842 0.90 285 

6 0.80 817 0.89 266 

7 0.88 924 0.91 294 

8 0.94 923 0.90 244 

24 
 

 

 

 

 

 

 

 

 

 

 

Measures 

Norm-Referenced Tests and the Normal Curve Equivalent (NCE) 

Norm-referenced tests, such as the Iowa Tests, are used to compare a student’s score at a 

given point in time to the scores of a nationally-representative group of students. The most recent 

Iowa Test norms were set in 2011, and all scores from fall 2012 onward are calibrated to this normreference group.1 

Data can be aggregated, and comparisons can be made, across years if the test scores are 

sufficiently horizontally equated; that is, scores in the same grade have the same meaning each 

year. One analysis of Wisconsin state test data indicates that this condition may not always be 

met (Meyer and Dokumaci, 2010). We use the Normal Curve Equivalent, or NCE, to measure 

results, partly in an effort to avoid this equating issue. NCE scores range from 1 to 99, with an 

average student having an NCE of 50. Scores above 50 indicate students that are performing 

above grade level, while scores below 50 indicate those students scoring below grade level. A 

change of one NCE unit has the same meaning for all students, regardless of where the student 

starts on the achievement distribution. One NCE equals approximately 0.05 standard deviations. 

Measuring growth 

In order to measure annual student growth, we use a value-added model. This model uses 

linear regression to estimate each teacher’s relative contribution to his or her students’ test score 

growth over time. This measure is limited to tested grades and subjects, and is not intended to be 

the only means of evaluating a teacher. 

Students must have both a pretest and a posttest in consecutive grades to be included in 

these analyses. Separate value-added models of mathematics and reading are estimated for each 

grade level and year. Value-added scores are produced for each teacher in each year, grade, and 

subject; these can be aggregated to provide an overall measure for all ACE AmeriCorps teachers 

1 Prior to the development of the 2011 norms, all scores were calibrated to the previous 2005 norms. There are slight 

differences in scoring when using the 2005 and 2011 norms, so scores before and after this transition should be 

interpreted with caution. 

25 

 

 

 

 

 

 

the difference between the “unshrunk” variance and the “noise” variance. The average reliability 

of the estimates is calculated as the ratio of the “true” variance to the “unshrunk” variance, and 

the reliability of each individual teacher effect is calculated similarly, using the individual “true” 

variance. We then “shrink” the effects by multiplying by their respective reliabilities in order to 

achieve the value-added score. 

Because the value-added results are centered, a growth score of zero NCE indicates the 

same growth relative to other teachers in the diocese. A positive growth score suggests higherthan-average growth, while a negative growth score suggests lower-than-average growth, relative 

to other teachers in the diocese. Because value-added results are estimates based on imprecise 

(and finite) data, all value-added results are reported with a standard error. This allows us to 

determine which teachers’ effects are statistically significantly different from the diocesan average 

of 0. 

Results 

We examine growth results graphically by showing them relative to the students’ starting 

points. Sometimes known as a “quadrant” graph, this display shows the average initial 

achievement of a teacher’s students on the horizontal axis, and their growth score on the vertical 

axis. Teachers in the lower left have students whose initial scores are lower than average, and 

they also demonstrate growth that is below the diocesan average. Teachers in the upper left also 

begin with scores that are below average, but show growth that is above average. Teachers in 

the lower right have initial scores that are above the national average, but show growth that is 

below the diocesan average, while those in the upper right are above average in both initial score 

and growth. In figures 1 and 2 (for diocese 1, reading and math) and figures 3 and 4 (for diocese 

2, reading and math), ACE teachers are shown with red “A” letters, while other first- and 

second-year teachers are shown with blue “1” numbers. In figure 1, we see that ACE teachers in 

diocese 1 had students whose incoming reading performance was typically above the national 

average, but not as far above average as several other teachers, while student growth was above 

average for some teachers and below average for others; we see a similar result in figure 2 for 

math. In figure 3, we find that ACE teachers in diocese 2 taught students whose incoming 

reading scores were slightly below the national average, and these students had above-average 

growth; students of ACE teachers in this diocese had below-average incoming scores, and 

27 
demonstrated less-than-average growth for four of the six teachers, as shown in figure 4. 

Note: In figures 1 through 4, value-added scores above 0 indicate growth that was better than the 

diocesan average. Average prior NCE above 50 indicates that students’ incoming scores were 

above the national average. 

28 
29 
30 
 

In figures 5 and 6 (for diocese 1, reading and math) and 7 and 8 (for diocese 2), the 

teachers are shown ordered by their growth scores, with error bars. These displays indicate that 

(as found by other value-added users (i.e., Harris, 2011)) few teachers have growth scores that 

are significantly better or worse than the diocesan average (because most error bars include 0). 

ACE Teachers are shown in blue. This display is for only the first- and second-year teachers. 

31 
32 
33 
34 
When scores are aggregated by teachers’ levels of experience, we find that the students of 

ACE teachers demonstrated growth that was 0.12 NCEs above average in diocese 1 in reading, 

and 1.86 NCEs below average in math, while other first- and second-year teachers had growth 

that was 0.50 NCEs above average in reading, and 0.40 NCEs below average in math. In both 

reading and math, the differences between these groups are statistically significant, indicating 

that the students of ACE teachers demonstrated less growth than the students of other first- and 

second-year teachers 

In diocese 2, students of ACE teachers had growth that was 1.99 NCEs above average in 

reading, and 0.21 NCEs below average in math, while other first- and second-year teachers had 

growth that was 1.31 NCEs below average in reading, and 0.44 NCEs below average in math. In 

both reading and math, the differences between these groups are statistically significant, 

indicating that the students of ACE teachers showed significantly more growth than the students 

of other first- and second-year teachers in this diocese. 

35 
 

 

 

 

 

 

 

 

 

 

 

Limitations 

The sample of only 2 dioceses is not sufficient to be generalizable to all ACE Teaching 

Fellows. We will continue to negotiate access to data from additional diocesan partners to increase 

our ability to make claims about the effectiveness of ACE Teachers. For the data reported here, it 

is important to remember that students are not randomly assigned to schools or teachers. Student 

background characteristics and other factors outside the control of the school or teacher may have 

an influence on attainment and growth. Consequently, any large attainment “gains” or indicators 

of growth when comparing ACE Teaching Fellows to other teachers may not be due purely to 

differences in effectiveness. Any apparent performance differences may also reflect underlying 

differences in their respective student populations, school contexts, or other factors. It should be 

noted that ACE Teachers are placed in the most difficult to fill teaching positions within their 

schools, so they likely are teaching in more challenging situations than other first and second year 

teachers within the diocese, which may also contribute to student achievement. 

Student Engagement 

In addition to measuring student achievement through test scores, we also measured 

growth in student engagement as a result of being in an ACE Teacher’s class as part of our 

performance measurement. ACE developed an instrument, the Students of ACE Teachers 

Engagement Survey (SATES), to measure students’ perceptions of academic engagement and 

belonging. Items were taken from the Chicago Consortium for School Research surveys, which 

have been used extensively and found to be reliable and valid with elementary and middle school 

students. The SATES survey is a retrospective pre-post survey that is taken by students in a 

sample of ACE Teachers’ classes at the end of the year. The survey asks students to think about 

how they feel in their ACE Teachers’ class and then answer the same questions while thinking 

about how they felt in the previous year, prior to being in an ACE Teachers’ class. 

The scale for academic engagement included four items: “I usually look forward to 

coming to this class,” “I work hard to do my best in this class,” “Sometimes I get so interested in 

my work I don’t want to stop,” and “The topics we are studying are interesting and challenging.” 

The belonging scale included five items: “I feel like a real part of my class,” “People here notice 

36 
 

 

when I’m good at something,” “Other students in my class take my opinions seriously,” “People 

in this class are friendly to me,” and “I’m included in lots of activities at this school.” 

The SATES was taken by 364 students of ACE Teachers. The scale scores were 

calculated by averaging the responses to each item within the scale for each individual student. 

When compared, 251 (69%) students reported equal or higher levels of academic engagement 

with an ACE Teacher than in the previous year, and 226 (62%) reported equal or higher levels of 

belonging with an ACE Teacher than in the previous year. In the future, we will continue to use 

the SATES instrument with more students, rolling it out to all ACE Teachers’ classes. 

Evaluation Question 3 Summary 

Over the two-year period under study, students of ACE Teachers showed more growth in 

both math and reading than students of other first- and second-year teachers in one diocese, but 

less growth than students of other first- and second-year teachers in a second diocese. It is not 

possible to determine whether these results are representative of all ACE Teachers in all of the 

dioceses where they are placed, because data is not readily available beyond these two dioceses; 

even within these dioceses, consistent measures of school context are not available. As data 

collection procedures improve in Catholic schools, further exploration of the performance of 

ACE teachers’ students will become feasible, both by expanding to additional dioceses, and by 

making more appropriate adjustments for school context. 

37 
 

 

 

CONCLUSION 

The findings of this evaluation affirm the high quality of teacher preparation offered by 

the ACE Teaching Fellows AmeriCorps program. From their own perspective in self-reported 

survey responses to the expert observations of academic supervisors and school principals, our 

members experience tremendous professional growth in instructional competence during their 

time in the program. 

One way this finding can be helpful for the program is to publicize the evaluation as 

evidence of professional growth for recruitment purposes. Many applicants are drawn to the 

ACE program because they desire not only to do service, but also to gain professional experience 

and training for a career in education. The findings of this evaluation show that in addition to 

earning a graduate degree, participants will develop expertise in all areas of teaching and gain 

confidence in their ability to teach through the program. Excerpts from the evaluation report can 

be used in marketing materials and highlighted on the ACE website to draw attention to the 

evidence of the program effectiveness in the area of professional growth. Improvements in 

evidence-based marketing efforts could lead to increased amount and quality of applications, 

subsequently leading to an even stronger cohort of AmeriCorps members. 

Findings related to student achievement in ACE Teachers’ classrooms were inconclusive 

due to the small sample size. ACE Teaching Fellows should incorporate into its strategy the need 

for more robust data-sharing partnerships with dioceses in which our teachers serve. Obtaining 

more data to further investigate the academic performance of students will help clarify the 

findings of this evaluation and lead to decisions about ways to improve the preparation of 

teachers. Preliminary results of student engagement surveys show that students of ACE Teachers 

report improvement in academic engagement and belonging while having an ACE Teacher. 

38 
 

 

References 

Caughlan, S., & Jiang, H. (2014). Observation and teacher quality: Critical analysis of 

observational instruments in preservice teacher performance assessment. Journal of Teacher 

Education, 65(5), 375-388. 

Harris, D.N. (2011). Value-added measures in education: What every educator needs to know. 

Cambridge, MA: Harvard Education Press. 

Harris, D. N., & McCaffrey, D. F. (2010). Value-added: Assessing teachers’ contributions to 

student achievement. In M. Kennedy (Ed.), Teacher assessment and the quest for teacher quality: 

A handbook (pp. 251–282). San Francisco: Jossey-Bass. 

Isenberg, E., and Walsh, E. (2014). Measuring Teacher Value Added in DC, 2013–2014 School 

Year: Final Report. Washington, DC: Mathematica Policy Research. Available: 

https://dcps.dc.gov/sites/default/files/dc/sites/dcps/publication/attachments/MeasuringValueAdde 

dinDC.pdf. 

Meyer, R. & Dokumaci, E. (2010). Value-added models and the next generation of assessments. 

Austin, TX: Center for K-12 Assessment & Performance Management. Available at 

https://www.ets.org/Media/Research/pdf/MeyerDokumaciPresenterSession4.pdf 

Midgley, C., Feldlaufer, H., & Eccles, J.S. (1989). Change in teacher efficacy and student selfand task-related beliefs in mathematics during the transition to junior high school. Journal of 

Educational Psychology, 81, 247-258. 

Ross, J. A. (1994). The impact of an inservice to promote cooperative learning on the stability of 

teacher efficacy. Teaching and Teacher Education, 10, 381-394. 

Sanders, W. L., & Horn, S. P. (1998). Research findings from the Tennessee Value-Added 

Assessment System (TVAAS) database: Implications for educational evaluation and research. 

Journal of Personnel Evaluation in Education, 12(3), 247-256. 

Tschannen-Moran, M., & Woolfolk Hoy, A. (2001). Teacher efficacy: Capturing an elusive 

construct. Teaching and Teacher Education, 17, 783-805. 

Tschannen-Moran, M., &Woolfolk Hoy, A. (2007). The differential antecedents of self-efficacy 

beliefs of novice and experienced teachers. Teaching and Teacher Education, 23, 944-956. 

Woolfolk Hoy, A. E. (2000). Changes in teacher efficacy during the early years of teaching. 

Paper presented at the annual meeting of the American Educational Research Association, New 

39 
Orleans, LA. Retreived from 

http://wps.ablongman.com/wps/media/objects/290/297451/changes%20in%20efficacy.pdf. 

Woolfolk Hoy, A., Hoy, W. K., & Davis, H. A. (2009). Teachers’ self-efficacy beliefs. In K. 

Wentzel & A. Wigfield (Eds.), Handbook of motivation in school. Mahwah, NJ: Erlbaum. 

40