Evaluation Report 

Notre Dame-AmeriCorps Program 

2014-2015 

Conducted by 

Peter R. Litchka, EdD.

Associate Professor, School of Education 

Loyola University Maryland 

Timonium Graduate Center 

2034 Greenspring Drive 

Timonium, MD 21093 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 2

Table of Contents 

 

Introduction p. 3 

List of Tables p. 4 

Context p. 5 

Methodology p. 9 

Data Presentation p. 14 

Discussion p. 28 

Conclusions p. 30 

Appendices p. 32 

References p. 40 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 3

Introduction 

In the spring of 2014, the national office of Notre Dame Mission Volunteers-AmeriCorps 

(NDMVA) hired Dr. Peter R. Litchka of Loyola University Maryland to conduct an independent 

evaluation of its K-12 education support programs. These programs can be found in more than 

100 schools and community organizations across 23 program sites in cities throughout the 

United States, serving more than 13,000 students. 

The purpose of this particular evaluation was to assess the impact of the services provided by 

NDMVA in Chicago, one of the largest of the program sites, as well as very representative in 

terms of student demographics across all sites. In order to examine the impact of such services, 

student attitudes were measured in terms of their perceived levels of engagement and attitudes in 

school, using a reliable and validated survey, which was administered early in the fall of 2014 

(pre) and again in later in the spring of 2015 (post). Both the treatment groups of students (those 

being provided with services) and comparison group of students (those not being provided with 

services) from three schools within the Chicago site participated in the surveys. In addition, 

randomly selected students from the treatment groups were interviewed as part of the process, 

following the post-survey. 

As was found with evaluations completed in 2012-13 and 2014-15, the impact of the 

NDMVA services provided to students was found to be statistically significant in most every 

area that was analyzed. 

Respectfully Submitted, 

_____________________ July 23, 2015 

Peter R. Litchka, EdD 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 4

List of Tables 

Table 1: Overview of Programs, Sites and Students Served p. 5

Table 2: Individual School Site Student Demographics p. 8

Table 3: Student Survey Participants by School-Fall and Spring, by Groups p. 15

Table 4: Gender of Student Participants by School p. 16

Table 5: Grade Levels of Student Participants by School p. 17

Table 6: Ethnicity of Student Participants, by School p. 18

Table 7: All Schools Fall and Spring Survey Results by Subscale and Groups p. 19

Table 8: All Schools Fall and Spring Survey Differences by Comparison Group for each 

 of the Four Subscales p. 20

Table 9: All Schools Fall and Spring Survey Differences by Treatment Group for each 

 of the Four Subscales p. 21

Table 10: All Schools Mean Difference in Fall/Spring Survey: 

 Levels of Significance by the Four Subscales. p. 22

Table 11: LEARN 8 School Mean Difference in Fall/Spring Survey: 

 Levels of Significance by Subscale. p. 23

Table 12: LEARN Campbell School Mean Difference in Fall/Spring Survey: 

 Levels of Significance by Subscale. p. 24

Table 13: Gender and Grade as Covariant p. 25

Table 14: Student Interview Groups by Grade p. 26

Table 15: Student Interview Comments by Categories p. 27
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 5

Context 

The Notre Dame Mission Volunteers Program was established as a nonprofit volunteer 

organization in 1990 by the Sisters of Notre Dame de Namur (SNDN), a Catholic religious order 

founded in 1804. The Order, which includes approximately 900 sisters across the United States 

and 1,500 around the world, has a mission of providing educational services to the poor. A goal 

of the Notre Dame Mission Volunteers Program was to enhance work of SNDN through fulltime secular volunteers and, in part, to mitigate the impact of the declining number of women 

making lifetime commitments to service in its order. Under the leadership of Sister Katherine 

Corr, the program applied for and received its first AmeriCorps grant in 1995. With the first 

grant, the Notre Dame Mission Volunteers Program was able to place 46 service members in 

four communities: Baltimore, Boston, Cincinnati, and Apopka (FL).1

 

During the 2012-13 academic year, NDMVA had programs in 22 locales across the United 

States. There were more than 400 full-time AmeriCorps service members providing various 

services to more than 9,000 students at 116 sites, as shown in Table 1. 

Table 1 

2 Overview of Programs, Sites and Students Served.

Students 

Program

Partnering 

Sites (n)

Service

Members (n)

Receiving 

Services (n)

Apopka, FL 7 31 515

Atlanta, GA 1 5 160

Baltimore, MD 8 17 695

Bend, OR 1 8 73

Boston, MA 3 15 89

Boulder, CO 9 18 355

 

1

 Corporation for National Community Service (2011). Serving Communities: How four organizations are using 

national service to solve community problems. Washington, D.C.: Author. 

2

 Source: Notre Dame Mission Volunteers AmeriCorps (2013). 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 6

Chicago, IL 7 14 775

Cincinnati, OH 4 18 231

Dayton 8 14 300

Hartford, CT 2 12 169

Los Angeles, CA 4 12 310

Nativity Affiliates3 16 95 1,268

New Orleans, LA 4 14 1,334

New York City, NY 2 9 131

Philadelphia, PA 6 16 397

Phoenix, AZ 3 9 574

Rochester, NY 1 6 52

San Francisco, CA 11 19 2,006

Seattle, WA 2 15 1,986

Thoreau, NM 1 10 153

Washington, DC 6 11 509

Watsonville, CA 8 14 1,158

Wilmington, DE 2 7 99

Total 116 389 13,339

Organizational Structure 

Within the individual program sites, a NDMVA site director oversees the members’ services, 

coordinates with local partnering sites where the members serve, and is the primary liaison 

between the national office (Baltimore) and the members. Partnering sites include public 

schools, charter schools, private religious schools, and other community agencies in 

economically disadvantaged communities. AmeriCorps members provide services to students by 

 

3

 Nativity Affiliates are located in the following cities: Baltimore, Boston, Denver, Hartford, San Diego, San Jose, 

St. Petersburg, and Tampa. In order to avoid double-counting, Nativity Affiliates are listed as a separate program. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 7

serving as teachers’ assistants, providing one-on-one or small group instruction and supporting 

after-school programs, helping to coordinate education and youth development activities. 

Of note are the Nativity Affiliates which are located in 8 cities across the United States 

(Baltimore, Boston, Denver, Hartford, San Diego, San Jose, St. Petersburg, and Tampa). Nativity 

Schools, which began in the early 1970s, are small middle schools found primarily in urban 

settings. These schools provide services to low-income students who may be at a risk of 

eventually dropping out of school, with an emphasis on a small student-to-teacher ratio. In 

addition, extended academic attention, including evening and weekend support, extra tutoring 

after school, and increasing parental involvement, are just some of the services provided. The 

Director of Nativity Affiliates oversees this part of the NDMVA program, reporting directly to 

the NDMVA Executive Director. As shown in Table 1, more than 1,200 students in 16 sites are 

provided with various services from the Nativity Affiliates. 

The foundation for the NDMVA focuses on increasing the academic achievement levels of 

children who are involved in the program activities. Academic and social support are provided 

during the regular school day and after school as well, depending on the needs and available 

resources found at the various partnering sites. According to NDMVA, the following services 

are provided: 

 Providing in-school and after-school academic tutoring and small group instruction 

for children in elementary, middle, and high school, focusing on reading and math. 

 Creating and managing out-of-school enrichment programs in areas such as sports, 

drama, visual arts, creative writing, environmental education, and debating. 

 Providing intern teaching services for small groups of students, typically 6-12 in 

number, under the direction of a master teacher. 

 Providing general classroom assistance to teachers.4

4

 Notre Dame Mission Volunteers AmeriCorps (2012). Application for Federal Assistance. 

Chicago Site 

In Chicago, NDMVA supports four unique educational settings. The KIPP Chicago Charter 

Schools is part of a national network of free, open-enrollment, college preparatory public 

schools. KIPP Chicago serves more than 1000 students in grades K-8 within four schools: KIPP 

 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 8

Ascend Primary, KIPP Ascend Middle, KIPP Bloom and KIPP Create. In these four schools, the 

NDA members serve as Personalized Learning Assistants (PLA) , helping to integrate 

technology into the curriculum as part of the Blended Learning Program (BLP). 

LEARN Charter School is a network of public, college preparatory elementary schools 

serving almost 2,000 underserved students in grades K-8 across five campuses. NDMVA 

partners serve at the LEARN South Chicago campus, LEARN Campbell campus and LEARN 8 

campus. NDA members serve as academic interventionists working one-on-one and in small 

groups with students performing below grade level in reading and writing. Members also assist 

in the classroom and provide support during after-school activities. 

The Marillac Social Center is committed to strengthening and empowering those most in 

need to reach their potential. To fulfill this mission, the Marillac Center offers programs and 

services to children, teens, families and seniors, including an afterschool program that helps 

students with homework, mentoring and other small group activities.

The San Miguel network has campuses in two areas of Chicago, and its mission is to 

transform lives and neighborhoods. NDMVA members serve in classrooms as teaching 

assistants, assists students in the process of applying to high school, and provide graduate 

support to San Miguel alumni..

The three sites that were selected as part of this evaluation were the LEARN 8 site, LEARN 

Campbell site, and LEARN South. As shown in Table 1, the enrollment in these schools ranges 

from 200 to almost 500 students, and each has very high levels of students living in poverty, as 

measured by the participation rates in the federally funded Free and Reduced Lunch (FRL) 

program. 

Table 2

Individual School Site Student Demographics

School Grades Enrollment FRL First Year with 

NDMVA

LEARN 8 6-8 200 97% 2013-14

LEARN Campbell K-6 440 97% 2009-10

LEARN South K-7 439 ??% 2014-15

Note. FRL=Percent of Students Receiving Free/Reduced Lunch
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 9

Methodology 

Students from three of the Chicago sites were selected to be part of this evaluation study. 

Both quantitative and qualitative methods were used to collect the data. 

Quantitative: Survey 

The quantitative portion of the evaluation was conducted through the use of a pre and post 

survey, “The Engagement versus Disaffection with Learning Scale” (Skinner, Kindermann, and 

Furrer, 2009). This instrument was designed to assess student motivation that includes the 

construct of engagement vs. disengagement in terms of the impact that such motivation can have 

on student learning, success and achievement. The conceptual framework of student motivation 

suggests that student engagement in the learning process is enhanced when the social setting 

supports students’ basic needs, both in a behavioral as well as emotional manner. When met, 

such needs provide students with a sense of support, structure, involvement and opportunity. 

Conversely, when such needs are not met, often students feel neglected, not supported or “forced 

to learn”, which can have a serious impact on student learning and achievement. 

The instrument contains 24 items, divided into four subscales: 

 Behavioral Engagement (5 items): makes an effort, pays attention, and is persistent. 

 Emotional Engagement (5 items): motivated, taps emotions, and enjoys learning. 

 Behavioral Disaffection (5 items): lack of effort, withdraws, and pretends to pay 

attention. 

 Emotional Disaffection (5 items): feeling discouraged, dislikes learning, feels 

frustrated during the learning process (see Appendix A). 

For each item, the response scale ranges includes 1 (not at all true), 2 (not very true), 3 (sort 

of true) to 4 (very true). Thus, the higher the ratings for Behavioral Engagement and Emotional 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 10

Engagement, the more positive the student is motivated in school. Conversely, the higher the 

ratings for Behavioral Disaffection and Emotional Disaffection, the less motivated the student 

feels about school. 

Structural analyses conducted by Skinner, Kindermann, and Furrer (2009) suggest that the 

items can be analyzed in various combinations, including separating the four subscales, 

combining both engagement subscales in comparison to both disaffection subscales. 

Below are examples of items for each domain: 

 Behavioral Engagement: “When in class, I listen very carefully.”

 Emotional Engagement: “I enjoy learning new things in class.”

 Behavioral Disaffection: “When in class, I just act like I am learning.”

 Emotional Disaffection: “When we work on something in class, I feel discouraged.”

Skinner, Kindermann, and Furrer (2009) report internal reliabilities from a sample of students 

in grades 3-6 of .61-.85. In addition, when combining behavioral and emotional engagement 

items, levels of consistency were found to be from .79 to .86 for student reporting. In addition, 

Skinner, Kindermann, and Furrer report that factory analyses find that the four-factor model was 

the best fit and correlated as expected: the behavioral and emotional subscales correlated 

positively and the engagement and disaffection subscales correlated negatively. 

The pre-survey (fall) was administered in early October, while the post-survey (spring) was 

administered in May. Students who had access to computers and the internet were able to take 

both surveys during school time via Survey Monkey, a web-based, on-line survey tool. Once the 

on-line surveys were completed, the results were immediately made available only to the 

evaluator. For those students who did not have access to computers and/or the internet, a paperpencil version of the survey was provided under the supervision of a teacher or NDA member. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 11

The site director transformed these into the web-based format, and additionally, made copies of 

the paper results and sent them onto the evaluator. A two-week window was provided to each 

site for completion of the survey both in the fall and spring. 

In order to analyze differences in the perceptions of both engagement and disaffection as well 

as to provide anonymity of students, demographic items were added to the survey. Specifically, 

the following information was requested: 

1. Unique identification number 

2. Gender 

3. Name of school 

4. Grade 

5. Ethnicity 

6. Comparison or Treatment group 

Once both the pre-and post-surveys were completed, data was entered in to SPSS, a software 

package used for statistical analyses. Descriptive data was completed, and then a comparison 

between pre-and post-survey means was completed, using a combination of independent sample 

t-tests, ANOVA, and regression analysis. 

Qualitative: Focus Groups 

The purpose of the qualitative portion of the evaluation was to develop a better understanding 

of the meanings constructed students who participate in the NDMVA program in the three 

Chicago sites. Unlike the quantitative portion, this part of the evaluation sought to interpret, 

translate, and assess student experiences, particularly as it relates to the unique features of the 

context found within. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 12

The evaluator contacted the site director to set up focus group interviews with the evaluator. 

The focus group interviews were conducted via Skype, an online telecommunications application 

software produce that provides video chatting and conference calling from computers, tablets 

and mobile devices via the internet. Students from the treatment groups were selected to 

participate in these focus group discussions. The discussions were semi-structured in their 

format, in that the interviews included a mix of predetermined quested and less structured 

questions, which allowed for the evaluator to probe deeper, and for the interviewees to provide 

examples and share experiences. The structured portion focused around the following questions 

for students: 

1. Please describe the services you receive as a student from the NDMVA staff? 

2. What do you see as the benefits of receiving such services? 

3. What concerns do you have about the program? 

4. What would your life as a student be like if you were not part of the program? 

Interviews were recorded and then transcribed for analysis purposes. 

Evidence of Success 

Two of the subscales-Behavioral Engagement (BE) and Emotional Engagement (EE)-are 

considered to be positive in nature (the higher the rating, the more positive the student feels 

about her/himself in the school environment), while the other two subscales-Behavioral 

Disaffection (BD) and Emotional Disaffection (ED)-are negative in nature (the higher the score, 

the less positive the student feels about her/himself in the school environment). Thus, if both 

Behavioral Engagement and Emotional Engagement increase, as well as Behavioral Disaffection 

and Emotional Disaffection decrease over time, it can be concluded that the student’s perception 

of behavioral and emotional engagement in school over this period has become more positive, 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 13

and conversely, if the Behavioral Engagement and Emotional Engagement decrease, and/or the 

Behavioral Disaffection and Emotional Disaffection increase, it can be concluded that the 

student’s perception of their behavioral and emotional engagement has become more negative.

Since there was a Fall (pre) survey and Spring (post) survey given to students who were 

receiving services from NDA (treatment group) and those students not receiving services 

(comparison group), evidence of success would be determined by comparing the results of the 

Fall and Spring surveys for both the Treatment groups and the comparison groups in the 

following ways:

 Did the ratings of students from the treatment group change in the Behavioral 

Engagement and Emotional Engagement? If so, to what extent and was the change 

significant?

 Did the ratings of students from the comparison group change in the Behavioral 

Engagement and Emotional Engagement? If so, to what extent and was the change 

significant?

 In comparing the changes for both the treatment group and comparison, were the 

changes similar or different? To what extent? Were the differences statistically 

significant?

 Did the ratings of students from the treatment group change in the Behavioral 

Disaffection and Emotional Disaffection? If so, to what extent and was the change 

significant?

 Did the ratings of students from the comparison group change in the Behavioral 

Disaffection and Emotional Disaffection? If so, to what extent and was the change 

significant?
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 14

 In comparing the changes for both the treatment group and comparison, were the 

changes similar or different? To what extent? Were the differences statistically 

significant? 

Data Presentation 

Descriptive Data 

Three schools from Chicago were involved in this evaluation: LEARN 8, LEARN Campbell, 

and LEARN South. The “Engagement versus Disaffection with Learning Scale” (Skinner, 

Kindermann, and Furrer, 2009) was the survey instrument used, and it was given to students in 

each of the three schools during early October 2014 (fall) and May 2015 (spring). Students 

selected to participate in both surveys were grouped according to those who receive services 

from AmeriCorps (treatment group) and those students who do not receive services (comparison 

group). Each student had a unique identification number so that survey results from the spring 

could be paired up with results from the fall. This would allow for statistical analysis, including 

but not limited to paired-sample t-tests. 

As shown in Table 3, more than 300 students from the three schools took both the fall and 

spring survey. There was a decrease in the number of students taking the survey in the spring as 

compared to the fall, with the treatment group decreasing from 154 to 128. One of the reasons 

for this decrease is that some of the students have become proficient in reading and have reached 

their grade-level, thus no longer requiring academic support from NDA members.. Overall, the 

percentage of students in the treatment group as compared to the comparison group from fall to 

spring did not change significantly. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 15

Table 3 

Student Survey Participants by School-Fall and Spring, by Groups (%) 

 Fall Spring

School Comparison Treatment Total Comparison Treatment Total

LEARN 

8

53 

(56.4%)

41 

(43.6%)

94 

(100.0%)

41 

(53.9%)

35 

(46.1%)

76 

(100.0%)

LEARN 

Campbell

120 

(53.8%)

107 

(47.2%)

227 

(100%)

136 

(61.0%)

87 

(39.0%)

223 

(100.0%)

LEARN 

South

6 

(50.0%)

6 

(50.0%)

12 

(100.0%)

6 

(50.0%)

6 

(50.0%)

12 

(100.0%)

All 

Schools

179 

(53.8%)

154 

(46.2%)

333 

(100.0%)

183

(58.8%)

128 

(41.2%)

311 

(100.0%

 

 

Demographic data was also collected from the survey, including gender, grade level, and 

ethnicity of the students who participated in the study. As shown in Table 4, overall 

participation according to gender was relatively even, with some slight disparities among the 

individual schools. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 16

Table 4

Gender of Student Participants by School (%)

Gender

School Female Male Total

LEARN 8 43 

(56.5%)

33 

(43.5%)

 76 

(100.0%)

LEARN Campbell 106 

(47.5%)

117 

(52.5%)

223 

(100.0%)

LEARN South 5 

(41.7%)

7 

(58.3%)

 12 

(100.0%)

All Schools 154 

(49.5%) 

157 

(50.5%)

311 

(100.0%)

As shown in Table 5, the range of grades for students participating in the surveys was from 

grade 3 through grade 6, with an overall proportionate number of students from each grade. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 17

Table 5

Grade Levels of Student Participants by School

School Grade 3 Grade 4 Grade 5 Grade 6 Total

LEARN 8 0 0 0 76 76

LEARN

Campbell

74 83 66 0 223

LEARN South 5 4 3 0 12

All Schools 79 87 69 76 311

The final demographic data that was collected from the students was ethnicity. As shown in 

Table 6, more than 90 percent of the students who participated in the survey were AfricanAmerican. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 18

Table 6

Ethnicity of Student Participants, by School (%)

School AfricanAmericanAsianAmericanCaucasian Hispanic Other Total

LEARN 8 71 

(93.4%)

2 

(2.6%)

0 

(0.0%)

1 

(1.3%)

2 

(2.6%)

76 

(100.0%)

LEARN

Campbell

206 

(92.3%)

3 

(1.3%)

2 

(0.89%)

3 

(1.3%)

9 

(3.9%)

223 

(100%)

LEARN

South

7 

(58.3%)

0 

(0.0%)

0 

(0.0%)

4 

(33.3%)

1 

(8.3%)

12 

(100.0%)

All Schools 284 

(91.3%)

5 

(1.6%)

2 

(0.6%)

8 

(2.6%)

12 

(3.8%)

311 

(100.0%)

Survey Data and Analysis 

In order to examine the differences in student perceptions between the fall and spring 

surveys, mean scores and standard deviations were computed according to each of the subscales 

and grouping of students (comparison and treatment). As shown in Table 7, the treatment 

group’s difference in the mean ratings between the fall and spring was more positive in three of 

the four subscales (BE, EE, BD and ED). 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 19

Table 7

All Schools Fall/Spring Survey Results by Subscale and Groups

Comparison

Group 

(n=134)

Treatment

Group 

(n=177)

Subscale Fall mean

(SD)

Spring mean

(SD)

Difference Fall mean 

(SD)

Spring mean

(SD)

Difference

Behavioral 

Engagement

3.02 (.54) 3.05 (.54) .03 2.95 (.57) 3.24 (.63) .29

Emotional 

Engagement

2.52 (.53) 2.59 (.63) .07 2.40 (.58) 2.97 (.71) .57

Behavioral 

Disaffection

2.09 (.60) 1.99 (.58) (.10) 2.15 (.54) 1.90 (.60) (.25)

Emotional 

Disaffection

2.35 (.64) 2.27 (.66) (.14) 2.43 (.57) 2.13 (.60) (.30)

In order to determine if such differences in the mean ratings between fall and spring were 

statistically significant or not, two statistical tests were applied. First, a comparison of the change 

in means for the comparison group alone, and then with the treatment group alone. For each 

group, a paired-sample t-test was used to determine if changes between the fall and spring 

surveys were significant or not. As shown in Table 8, the comparison group’s changes were all 

positive but the changes were not statistically significant. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 20

Table 8 

All Schools Fall/Spring Survey Differences by Comparison Groups for Each Subscale 

Subscale Fall Mean 

(Standard 

Deviation)

Spring Mean 

(Standard 

Deviation)

t Sig.*

Behavioral 

Engagement

3.02 (.54) 3.05 (.43) .933 .352

Emotional 

Engagement

2.51 (.53) 2.59 (.52) 1.384 .168

Behavioral 

Disaffection

2.05 (.60) 1.99 (.43) 1.365 .174

Emotional 

Disaffection

2.35 (.64) 2.27 (.51) 1.403 .162

Note. *p<.05

The statistical test for the treatment groups found that, for each of the four subscales, the 

difference between the fall and spring survey were found to be positive and statistically 

significant. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 21

Table 9 

All Schools Fall/Spring Survey Differences by Treatment Groups for Each Subscale

Subscale

Fall Mean 

(Standard 

Deviation)

Spring Mean 

(Standard 

Deviation)

t Sig.

Behavioral 

Engagement

2.96 (.57) 3.25 (.63) 9.28 .001*

Emotional 

Engagement

2.39 (.58) 2.96 (.71) 11.61 <.001*

Behavioral 

Disaffection

2.28 (.54) 1.88 (.60) 7.69 .002*

Emotional 

Disaffection

2.42 (.57)

2.12 (.70) 7.64 .003*

Note. *p<.05

The next test was the General Linear Model (GLM) was used. As shown in Table 10, the 

difference in the means was found to be statistically significant in each of the four subscales 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 22

Table 10

All Schools Mean Difference in Fall/Spring Survey: Levels of Significance by Subscale.

Subscale Group Fall/Spring 

Difference

F Sig.

Behavioral Engagement Comparison

Treatment

.03

.29 10.3 .001*

Emotional Engagement Comparison

Treatment

.07

.57 18.2 <.001*

Behavioral Disaffection Comparison

Treatment

(.10)

(.25) 9.78 .003

Emotional Disaffection Comparison

Treatment

(.14)

(.30) 4.93 .027*

Note. *p<.05

The General Linear Model was used in a similar manner with two of the three schools: 

LEARN 8 and LEARN Campbell. However, it was not used for LEARN South, since the 

number of participants in the survey was too low for statistical analysis (n=12). 

As shown in Table 11, the results of the statistical analysis for LEARN 8 was that the 

Treatment Group’s change in their perceptions were both more positive and statistically 

significant in three of the four subscales: Behavioral Engagement, Emotional Engagement, and 

Emotional Disaffection. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 23

Table 11

LEARN 8 Mean Difference in Fall/Spring Survey: Levels of Significance by Subscale.

Subscale Group Fall/Spring 

Difference

F Sig.

Behavioral Engagement Comparison

Treatment

.09

.27 4.97 .045*

Emotional Engagement Comparison

Treatment

.07

.34 5.62 .049*

Behavioral Disaffection Comparison

Treatment

(.18)

(.22) 1.94 .168

Emotional Disaffection Comparison

Treatment

.02

(.31) 6.62 .027*

Note. *p<.05

At LEARN Campbell, an analysis of the GLM found that the treatment group’s change in the 

mean score from fall to spring was greater than the comparison group’s in each of the four 

subscales, and as shown in Table 12, the differences was statistically significant in each subscale 

as well. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 24

Table 12

LEARN Campbell Mean Difference in Fall/Spring Survey: Levels of Significance by Subscale.

Subscale Group Fall/Spring 

Difference

F Sig.

Behavioral Engagement Comparison

Treatment

.06

.29 20.41 <.001*

Emotional Engagement Comparison

Treatment

.05

.30 9.95 .002*

Behavioral Disaffection Comparison

Treatment

.06

(.43) 47.86 <.001*

Emotional Disaffection Comparison

Treatment

(.05)

(.30) 14.43 <.001*

Note. *p<.05

In addition, further statistical analyses were completed to determine whether ethnicity, 

gender or grade of the student had a significant impact on the change in perceptions from fall to 

spring. Ethnicity as a variable was not used in this analysis since more than 90 percent of the 

student participants were of one ethnic group-African American (91.3%), while no other ethnic 

group of students made up more than 4 percent of the sample. 

Furthermore, gender and grade of student participants were used as covariants to determine 

their impact on student perceptions for each of the four subscales. As shown in Table 13, neither 

the gender of the students or the grades of the students had a significant impact on their 

perceptions for each of the four subscales. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 25

Table 13 

Gender and Grade as Covariant 

Gender as Covariant Grade as Covariant

Subscale F Sig. F Sig.

Behavioral Engagement .099 .753 .048 .826

Emotional Engagement .687 .408 1.293 .256

Behavioral Disaffection .111 .739 1.723 .178

Emotional Disaffection .122 .727 .006 .939

Interviews and Observations 

A number of students receiving the services from AmeriCorps were interviewed by the 

evaluator as part of the qualitative portion of the evaluation. Students were interviewed in small 

focus groups at the school which they attended. 

The semi-structured nature of the interviews focused on the following questions: 

1. What types of services do you receive from AmeriCorps? 

2. How often do you receive such services? 

3. What benefits do you receive from such services? 

4. Having received these services for most of the school year, has it changed how you 

feel about school and about yourself? If so, how and why? 

As shown in Table 14, twenty-one students were interviewed by the evaluator. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 26

Table 14

Student Interview Groups by Grade

Grade Students Interviewed

3 6

4 5

5 3

6 7

Total 21

Student comments were transcribed and then were categorized into general trends of how the 

students perceived the support they were receiving, as shown in Table 15. A total of 109 

comments were categorized into services, frequency of services and benefits of services. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 27

Table 15

Student Interview Comments by Categories

Grade Services How Often Benefits

3 Help with reading and 

sometimes other 

subjects. (3)

Individual help. (3)

Small groups. (2)

Pull-out. (2)

Every day. 20-30 

minutes per day.

(4)

Better grades. (5)

Working harder. (5)

Having adults who 

care. (4)

Enjoying school. (5)

4 Help with reading. (4)

Fun activities with other 

students. (3)

Group work. (2)

Pull-out. (3)

Usually 4 days 

per week; 20-30 

minutes per day.

(4)

Better grades. (5)

Getting along with 

others. (3)

Learning how to 

study better. (4)

Feel better about 

myself. (4)

5 One-on-one

Reading and math help.

(2)

Small reading groups 

and help with 

homework. (3)

Pull-out. (2)

4-5 days per 

week, 30-40 

minutes each day.

(3)

Being in a smaller 

class for part of the 

day makes school 

better. (3)

Help with problems.

(4)

More positive about 

school (3).

6 Small groups and 

individual help with 

reading and homework. 

(4)

Pull-out. (4)

Everyday. 20-30 

minutes per day; 

sometimes longer.

(6)

Adult support. (3)

People who care 

about us. (3)

More friends (3).

Feel better about 

school. (3)

Helping us be better 

to ourselves and 

others. (4)
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 28

Discussion 

Results of various statistical analyses demonstrate the significant impact the NDMVA 

program is having on students in three selected sites in Chicago. In comparing the change in 

student perceptions over time between the comparison group (those students not receiving 

services) and the treatment group (those receiving services) among all three schools together, the 

treatment group’s change from fall to spring was more positive in three of the four subscales, and 

in each of these, the difference was found to be statistically significant. The three subscales in 

which this occurred were Behavioral Engagement, Emotional Engagement, and Emotional 

Disaffection. In the one subscale in which the comparison group’s improvement was greater 

than the treatment group (Behavioral Disaffection), the difference was not found to be 

statistically significant. 

Similar results were found at LEARN 8, with the same three subscales showing changes that 

were statistically significant for three subscales (Behavioral Engagement, Emotional 

Engagement, and Emotional Disaffection) for the treatment group, but the difference was not 

found to be statistically significant for the comparison group, which had a greater change than 

the treatment group. 

At LEARN Campbell, the change in student perceptions for the treatment group was greater 

in each of the four subscales than that of the comparison group, and in each case, the difference 

was statistically significant. 

In addition, neither the gender or the grade level of the students, regardless of whether they 

were in the treatment or comparison group, had impact on the changes in the perceptions over 

time. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 29

Finally, data collected from LEARN South was not disaggregated due to the small number of 

participants. 

Small focus groups were held in order to provide students who receive services with the 

opportunity to discuss their perceptions. At total of 21 students participated in the focus group 

discussions. 

Students, in small groups, were asked to respond to the following structured questions: 

1. Please describe the services you receive as a student from the NDMVA staff? 

2. What do you see as the benefits of receiving such services? 

3. What concerns do you have about the program? 

4. What would your life as a student be like if you were not part of the program? 

The interviewer then asked students to elaborate and provided examples and/or reasons for 

such responses. 

As shown in Table 15, students provided evidence of the services received, how often the 

services occurred and the benefits from receiving services from AmeriCorps. Below is a sample 

of quotes from students regarding the benefits of their participation in this program: 

Without the help of Miss ___ [NDMVA member], I would not be passing onto the next 

grade. She understood me, helped me with my reading and some math, and helped me 

deal with some of my problems I had with my teacher and other students. (Male student 

R, 5

th grader) 

I was pulled-out of my class every day. In the beginning, I didn’t want to go but then I 

began to like it. Miss ___ [NDMVA member] gave me lots of help that many of my 

friends in other classes were not able to get. I didn’t like school, but I like it better now. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 30

My grades are better and I am getting along better with others. (Female student R, 5th

grader) 

We really trust Miss ___ [NDMVA member]. We have been with her for the whole year 

and she really believes in us-not all our teachers do. We all think school is very good 

place to be and we want to be good students. (Female student T, Male student L, Male 

student K, all 6th graders. 

I was not doing well in school, but Miss ___ [NDMVA member] made reading and math 

fun for me. I like school much better now and I really want to do well. My reading is 

getting better, and math is too...but I like reading with Miss ___ [NDMVA member] the 

most. (Female student Z, 6th grader). 

Conclusions 

In this report, two approaches were used to examine the effectiveness of the NDMVA 

program for the three selected schools in Chicago. The first approach was the use of a survey 

administered to students-both treatment and comparison groups-in the fall (pre) and the spring 

(post). The survey instrument used was “The Engagement versus Disaffection with Learning 

Scale” (Skinner, Kindermann, & Furrer, 2009), which contained 24 items within four subscales: 

Behavioral Engagement, Emotional Engagement, Behavioral Disaffection, and Emotional 

Disaffection). The theoretical framework supporting this instrument suggests that student 

engagement in behavioral and emotional terms has a direct impact on student learning and 

achievement, as well as the ability of students to interact in their school social setting. In 

addition, a sample of students were interviewed in small focus groups in order to elaborate on 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 31

their perceptions of the support they receive, particularly in the areas of academic and social 

needs for success. 

The data collected from this examination provide overwhelming and compelling evidence 

about the significant impact that the NDA Chicago program has on students who receive such 

services and support.. As shown in Tables 8 through 12, students receiving services increased 

their behavioral and emotional engagement between the fall and spring surveys in a higher and 

more positive in a manner that was found to be statistically significant in every example except 

one (LEARN 8 Behavioral Disaffection). Of note, the gender and grade level of students did not 

impact their perceptions of their behavioral and emotional engagement in schools. 

The evaluator finds that the interviews of students (n=21) confirm the positive changes of 

student attitudes found in the surveys. Students felt supported, attached to the NDMVA 

members and appreciative of the services. They also felt that they would not have had such 

success during this current school year without the support of the NDMVA members. 

All of these students come from very challenging backgrounds, in which poverty is abundant 

and a lack of opportunities for success can often be limiting. Yet these students feel successful 

and demonstrate their positive outlook in terms of both personal and school growth. In particular, 

this success in Chicago is through in-school support provided by NDMVA members in small 

group and individual support sessions, usually on a daily basis. 

The evaluator concludes that NDMVA and its members are providing a significant 

contribution and supplement to the success of these students through various instructional and 

social support. Of critical importance, the results of this evaluation support the conclusions 

found in previous evaluations (Fenzel, 2009; Litchka, 2013) of the statistically significant impact 

the NDMVA program has on students who receive services. 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 32

APPENDICES 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 33

Appendix A 

Survey Instrument (Survey Monkey) 

NDMVA Chicago 

1. In the space below, write down the initial of your first name, middle name, and last name. Then, 

wait for your teacher to give you a special number to write down as well. 

2. What is the name of your school? 

Learn Campbell 

Learn Middle 

Learn South 

3. What grade are you in? 

1 

2 

3 

4 

5 

6 

7 

8 

9 

4. What is your gender? 

Female 

Male 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 34

5. What is your ethnicity? 

African American 

Asian American 

Caucasian 

Hispanic 

Other 

 

Directions: 

Read each of the following items and select the answer that best describes how true the statement is about you. 

6. I try hard to do well in school. 

Never 

Sometimes 

Most of the time 

All of the time 

7. I enjoy learning new things in school. 

Never 

Sometimes 

Most of the time 

All of the time 

8. I worry that I won't be able to do well in school. 

Never 

Sometimes 

Most of the time 

All of the time 

9. When I am in school, I don't feel like doing much. 

Never 

Sometimes 

Most of the time 

All of the time 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 35

10. I enjoy school. 

Never 

Sometimes 

Most of the time 

All of the time 

11. I listen very carefully to my teacher. 

Never 

Sometimes 

Most of the time 

All of the time 

12. I get upset when I make a mistake in school. 

Never 

Sometimes 

Most of the time 

All of the time 

13. I am interested in what we are learning in school. 

Never 

Sometimes 

Most of the time 

All of the time 

14. It bothers me when I can't answer a question in class. 

Never 

Sometimes 

Most of the time

All of the time 

15. I like to work with other students in class. 

Never 

Sometimes 

Most of the time 

All of the time 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 36

16. When in class, I like to think about other things than what we are learning. 

Never 

Sometimes 

Most of the time 

All of the time 

17. I enjoy learning new things in school. 

Never 

Sometimes 

Most of the time 

All of the time 

18. I get upset when other students do well in class and I don't. 

Never 

Sometimes 

Most of the time 

All of the time 

19. In class, I don't try very hard. 

Never 

Sometimes 

Most of the time 

All of the time 

20. I feel safe when I am at school. 

Never 

Sometimes 

Most of the time 

All of the time 

21. In class, I pretend to be working hard. 

Never 

Sometimes 

Most of the time 

All of the time 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 37

22. I want to be a good student. 

Never 

Sometimes 

Most of the time 

All of the time 

23. School is boring to me. 

Never 

Sometimes 

Most of the time 

All of the time 

24. Working hard in school is a waste of time. 

Never 

Sometimes 

Most of the time 

All of the time 

25. I pay attention when I am in class. 

Never 

Sometimes 

Most of the time 

All of the time 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 38

Appendix B: 

Institutional Review Board (IRB) Project Approval, Loyola University Maryland 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 39
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 40

General References

Fredericks, J., Blumenfeld, P., and Paris, A. (2004). School engagement: potential of the 

concept: state of the evidence. Review of Educational Research, 74, 59-119. 

Fredericks, J., McColskey, W., Meli, J., Montrosse, B., and Mooney, K. (2011). Measuring 

student engagement in upper elementary through high school: a description of 21 

instruments. (Issues and Answers Report, REL 2011-No.098). Washington, DC: U.S. 

Department of Education, Institute of Education Sciences, National Center for Education 

Evaluation and Regional Assistance, Regional Educational Laboratory Southeast. Retrieved 

from http://ies.ed.gov/ncee/edlabs on November 17, 2014. 

Jenner, E. and Jenner, L. (2007). Results from a first-year evaluation of academic impacts of an 

after-school program for at-risk students. Journal of Education for Students Placed at Risk, 

(12)2, 213-237.

Klem, A. & Connell, J. (2004). Relationships matter: Linking teacher support to student 

engagement and achievement. Journal of School Health (74)7, 262-273. 

Leary, M. (2004). Introduction to behavioral research methods (4th Ed.). Boston: Pearson 

Education, Inc. 

Lehr, C., Sinclair, M. and Christenson, S. (2009). Addressing student engagement and truancy 

during the elementary school years: a replication study of the check & connect model. 

Journal of Education for Students Placed at Risk, (9)3, 279-301. 

Marks, H. (2000). Student engagement in instructional activity: patterns in the elementary, 

middle, and high school years. American Education Research Journal, 37, 153-184.

Skinner, E., Kindermann, T. and Furrer, C. (2008). A motivational perspective on engagement 

and disaffection: conceptualization and assessment of children’s behavioral and emotional 
Evaluation of the Notre Dame Mission Volunteers-AmeriCorps 41

participation in academic activities in the classroom. Educational and Psychological 

Measurement, 20(10), 1-33.

Yazzie-Mintz, E. (2007). Voices of students on engagement: a report on the 2006 high school 

survey of student engagement. Bloomington, IN: Indiana University, Center for Evaluation 

and Education Policy. Retrieved February 2, 2015, from http://www.indiana.edu

/~ceep/hssse/images/HSSE%20Overview%20Report%20-%202006.pdf