Attachment 2

Memorandum--‐WINGS Evaluation Study 

Implementation Study Report 

WINGS Program Evaluation 

College of Charleston 

Summer 2015 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Contents 

Executive Summary and Conclusions-­­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­--­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­­-­­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­--­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­--­-­ ­-­ ­-­­--­ i

Technical Report-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­--­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­--­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­--­-­ ­-­ ­-­­-­­-­ ­-­­-­­-­­-­ ­--­ 1

Method-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­--­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­--­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­--­-­­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­--­ 1

Emotional Climate-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­--­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­--­-­ ­--­-­ ­-­ ­-­­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­­-­­-­ ­-­ ­--­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­--­ 2

Instructional Strategies-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­--­-­ ­-­­-­ ­-­ ­--­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­--­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­--­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­--­ 10

Learning Environment Organization-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­--­--­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­­-­­--­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­--­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­--­ 15

References-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­--­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­--­-­ ­-­­--­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­ ­-­­-­­-­ ­-­ ­--­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­--­ 24

Appendix-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­--­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­--­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­--­-­­-­ ­-­­-­­-­ ­--­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­--­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­ ­-­­-­ ­-­ ­-­­-­­-­ ­-­­-­­-­ ­--­ 26
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Executive Summary and Conclusions 

The WINGS for Kids Program Evaluation addresses the quality of the implementation of the WINGS 

program at the Nest level between WINGSLeaders and their students, in terms of (1) emotional and 

relationship quality, (2) instructional quality, and (3) organizational quality. In the following technical 

report, we cover these sections in more detail within corresponding sections: (1) emotional climate, (2) 

instructional strategies, and (3) learning environment organization. Multiple raters and measures 

converged to create an overall picture of the quality of interactions in Nests.

The findings presented in this report provide evidence that quality varies greatly among Nests. Across 

multiple measures, scores ranged from the minimum to the maximum. High scores indicate that Nests 

are capable of being friendly, supportive, and nurturing places that are conducive to learning and 

growth. On the other hand, low scores reflect a presence of negativity and hostility where management 

is lacking and the potential for student growth and learning is weakened. The range of experiences 

across Nests suggests that implementation fidelity is inconsistent. Opportunities to improve consistency 

may include attention to initial and sustained training and mentoring. Below, we discuss findings in 

more detail.

Emotional and Relationship Quality. Children thrive in emotionally-­­supportive learning environments 

that encourage student participation and positive relationships; this is particularly true for children at risk

for school failure (Hamre & Pianta, 2005; Ladd, Birch, & Buhs, 1999). However, at-­­risk children who are in

less-­­supportive environments often have more conflicts with teachers and do not fare as well 

academically (Hamre & Pianta, 2005). Observational data collected by the Classroom Assessment Scoring

System (CLASS) and the Out-­ ­of-­­School-­ ­Time (OST) measures provide information regarding the 

emotional and relationship quality within Nests. On the CLASS, WINGS Nests received a mid-­­range 

average (4.46 on a 1 to 7 scale) for the positive climate dimension, which is comparable to elementary

school classrooms in other studies (4.44 (4Rs1); 4.91 (Responsive Classroom 2)). The frequency of codes

indicate that a majority of observations (69.3%) were coded in the mid-­­range, with 24.8% within the high-

­­range, and a small percentage (5.8%) in the low-­­range. High-­ ­range codes illustrate the presence of 

consistently warm, supportive relationships, positive communication, and respect; low codes indicate 

the absence of these positive environmental indicators.

Furthermore, the CLASS negative climate dimension measures the presence of negativity in the WINGS 

Nests. By looking at code frequencies, we can see that WINGS Nests were coded the vast majority of 

times (82.4%) in the low-­ ­range, which is preferable for this dimension and indicates the absence of or very 

rare instances of negativity within the environment. However, it should be noted that some (16%) 

observations were coded in the mid-­ ­range, indicating mild displays of irritability, occasional yelling, 

sarcasm, and disrespect; and two observations were coded as having instances of severe negativity 

(score of 6) within the high-­­range, indicating physical aggression among children.

1 Study of third grade classrooms (n=82) in New York city with 45.6% Hispanic/Latino and 41.1% African American 

children; 61.8% of children were living at or below 100% of the poverty line at the time of the study (Jones, Brown, 

& Aber, 2011).

2 Study of first through fifth grade classrooms (n=88) in an urban district in the northeast. 53.63% were ethnic

minorities and 35.32% were eligible for free or reduced lunch (Rimm-­­Kaufman, as cited in Pianta, La Paro, & 

Hamre, 2011).

i 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

The OST averages of youth relationships (3.27 on a scale of 1 to 7) and staff-­ ­youth relationships (3.46 on a

scale of 1 to 7) fall within the mid-­­range; when comparing WINGS relationship averages to other 

afterschool programs , staff-­ ­youth relationship averages as measured by the OST were more than 1-­­ point 

lower (3.46 (WINGS); 4.5 (New Jersey After 33)) and the youth relationship average was more than

2--­points lower (3.27 (WINGS); 5.86 (New Jersey After 3)).

Instructional Quality. Instructional quality can have a profound influence on student engagement and 

learning. Relatedly, students who are attending to academic tasks are less likely to be disruptive and/or off-

­­task. Instructional quality focuses on the processes that effectively support cognitive and language 

development. WINGS delivers its social-­ ­emotional curriculum through instruction which is guided by thirty 

weekly social and emotional learning objectives and lessons. Thus, the quality of instruction is important, if 

not key, to effectively supporting the social and emotional development of WINGS kids.

On observational measures (OST and CLASS), Nest averages fall in the low-­ ­range for instructional quality. 

On the OST dimension of instructional strategies the WINGS average was 2.57 (in the low-­­range), while 

averages of two comparison studies were within the mid-­­range (3.6 (New Jersey After 3); 3.77 (The After 

School Corporation (TASC4)); the same is true when comparing WINGS averages on the CLASS

dimensions of concept development (1.46 (WINGS); 3.82 (Responsive Classroom); 3.84 (4Rs)) and quality 

of feedback (2.13 (WINGS); 4.77 (Responsive Classroom); 3.54 (4Rs)). Within the low range of quality, 

learning environments are characterized by drill--­oriented activities, teacher or WINGSLeader--­controlled 

conversations, and close--­ended questions that focus on rote information and recall. This stands in 

comparison to what is possible with a high--­level of instruction; for example, an environment that 

incorporates problem solving, creativity, brainstorming, teachable moments (a key provision of the WINGS

curriculum), back--­and--­forth exchanges, and the use of advanced language and open--­ended questions.

Improving the quality of instructional processes is particularly important for schools and out--­of--­school-­­ 

time programs, like WINGS, that serve students who are at risk for school failure. In one study, students 

from families with low--­parental education who were placed in first--­grade classrooms offering mid-­ ­ to-­­ 

high instructional quality displayed academic achievement at similar levels as peers from families with 

higher parental education; whereas those students in low instructional quality classrooms fell further 

behind (Hamre & Pianta, 2005). Moreover, we must remember that differences in just over 1 point on the

CLASS 7--­point scale translate into improved achievement and social skill development for students 

(CASTL, n.d.). In essence, this suggests that if WINGS demonstrated even small changes in effective 

interactions, it would have practical implications for the success of its students.

Organizational Quality. According to the WINGS training manual, “WINGS works with the toughest kids 

in the toughest schools,” and the program goal is to transform each child through improved behavior and

self-­­regulation. Common sense tells us that learning environments with well-­ ­behaved students function

best; however, the relationship is two-­­fold. The development and expression of self-­­control and self-

­­regulatory skills is highly dependent on the nature of the learning environment (Anderson, Evertson, & 

Emmer, 1980; Emmer & Stough, 2001; Kounin, 1970; Pintrich, 2000; Sanford & Evertson, 1981; Soar &

3 Study of programs (n=10) for children in grades kindergarten through eight in New Jersey; 87% of children were 

African American or Hispanic and 57% were eligible for free or reduced lunch.

4 Study of programs (n=10) for children in grades four through eight in New York City (Birmingham, et al., 2005).

ii
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

III

Soar, 1979). In other words, students exhibit better self-­­regulation in organized, structured, and 

engaging learning environments.

Observational data collected by the CLASS shows that WINGS Nests received a behavior management 

average in the mid--­range (4.16 on a 1 to 7 scale). Classrooms in the 4Rs and Responsive Classroom 

studies had averages in the mid-­­ to mid--­high range (4.98 (4Rs); 5.14 (Responsive Classroom)). Beyond 

averages, a look at the frequency of scores reveals variability in behavior management quality: 19% of 

Nest observations received a code of 6 (high level), indicating clearly stated rules and expectations, 

consistent use of proactive techniques, individualized praise, and well--­behaved students. Conversely, 

28% of observations received a score of 3 (at the base of mid-­ ­ level codes), indicating inconsistently 

enforced expectations, periodic episodes of misbehavior with time taken away from learning to manage 

behavior, and more reactive than proactive responses. Thus, observations during the school year reveal 

variability in practice among Nests. Notably, learning behavior management skills is one of the most 

frequently identified problems by pre--­service teachers (Briton & Holten, 1989; Greenlee & Ogletree, 

1993; Meister & Melnick, 2003; Avalos & Aylwin, 2007; Ozturk, 2008; Ulvik et al. 2009). Consequently, 

behavior management skills are difficult to teach except through experience, when teachers have the 

opportunity to work with a diverse classroom of students (Debreli, 2013; Johnson, 1996).

Given that classrooms run the smoothest when children have interesting things to do, the organization 

of the learning environment also takes into consideration student engagement and the ways in which the

teacher/WINGSLeader facilitates that engagement. WINGS describes itself as “an instruction--­based 

learning curriculum weaved into an active and engaging after--­school program,” thus student engagement 

is one of its key provisions. Instructional learning formats, a CLASS dimension, focuses on the teacher and 

the extent to which he/she facilitates student engagement. The WINGS average for this dimension falls in

the mid--­range (3.5 on a 1 to 7 scale), within the same range but slightly lower than the Responsive 

Classroom and 4Rs studies (4.23 and 4.21 respectively). Here we see that the difference between WINGS 

and these other studies is slight (0.73 and 0.71), however it is important to know that differences in just

over 1 point on the CLASS 7--­point scale translate into improved achievement and social skill development 

for students (CASTL, n.d.). A look at the frequency of scores reveals variability in quality that is not 

evident when considering the overall average. While most of the observations (63.5%) were coded in the

mid--­range, 27.7% of observations were coded in the low--­range, indicating that the teacher/WINGSLeader 

made few attempts at guiding students to learning objectives, did not provide or was inconsistent in

providing interesting, creative materials and hands--­on opportunities, and did little to expand students’ 

involvement or ask questions. A small percentage of observations were coded in the high--­range (8.75%).

The OST assesses student engagement in its youth participation component. The WINGS average within 

this component falls at the upper end of the low-­­range (2.76 on a 1 to 7 scale). This average indicates that 

youth were not at all or rarely on task, actively listening, contributing to discussions, or taking 

leadership roles. The lack of participation captured on OST is meaningful for WINGS because the 

cornerstone of its social and emotional learning curriculum is Discussion, a twenty-­­minute lesson in which 

WINGSLeaders introduce and discuss with students the weekly learning objective. The WINGS average 

for youth participation as assessed during this specific component (2.40 on a 1 to 7 scale) indicates that 

students are not effectively or actively participating in one of the most important instructional 

components of the program.
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

IV

Additionally, poorly managed routines contribute to less instructional time, student disengagement, and 

often lead to student misbehavior (Gettinger & Seibert, 2002; Sprick, Garrison, & Howard, 1998).

Observational time sampling data that was collected on 6 Nests during WINGS program hours revealed that

students spent 20-­­46% of their time in transition.

Stakeholder Perceptions. Observations, while perhaps the most important type of information, 

provide only one lens with which to view WINGS. Stakeholders, such as principals, teachers, and 

program staff, lend a unique and valuable perspective. Principals and program staff were 

interviewed and teachers were surveyed in order to capture a fuller and more nuanced 

description of the significance of WINGS as well as how it functions. When given a list of 

adjectives, teachers described WINGS as loud (59%), but also engaging (52%) and positive (52%). 

They described WINGS staff as caring (55%), friendly (46%), and well-­­intentioned (41%).

Additionally, WINGSLeaders often described their work environment as having “a family vibe” 

and discussed relationships with their students as the best part of their day.

Nevertheless, the qualitative interviews captured a broad range of experiences that also reflect 

the variability in scores on the observational measures. Many WINGSLeaders reported behavior 

management as their biggest challenge, while only a few reported it as one of their strengths. 

Again, this finding is common among pre--­service teachers. Most striking was the variability 

within individual WINGSLeader reports. Specifically, many WINGSLeaders affirmed that they had 

a positive relationship with students and were “most of the time” able to meet instructional 

goals, but would then go on to further explain that they felt overwhelmed when they 

encountered behavior management challenges and when struggling to keep kids engaged with 

lessons that were “too easy,” “over their heads,” or “boring.” This is to say that implementation 

quality appears to be variable not only depending on the WINGSLeader, but on the day and the 

activity. Accordingly, in interviews, principals mentioned that program implementation was less 

than ideal. One principal provided, “I think that the program has good intentions, [but] it is not 

being implemented well.”

Conclusions. Taken together, all data collection efforts uncovered areas of growth for the 

program that can have a great impact on program functioning and, consequently, student 

learning and growth. Here, we offer conclusions and suggestions to guide improvement efforts.

(1) Emotional and relationship quality is moderate. While WINGS averages are roughly similar

to other elementary school classrooms and out--­of--­school--­time programs, this should be

a strength of a social--­emotional intervention program targeting at--­risk youth. 

WINGSLeaders would benefit from constant, ready support on the job; this can be 

accomplished by having extra support staff ready when challenges occur. Additionally, 

Nest sizes could be smaller to aid the bond between WINGSLeader and children, and in 

turn help with behavior management. Finally, staff selection should be a priority; 

understanding characteristics of successful WINGSLeaders is an area for further study.

(2) Instructional quality is poor. Enhancing the instruction of the WINGS curriculum will 

increase student learning and growth. This can be ameliorated through training and 

ongoing support for WINGSLeaders to focus on tailoring instruction for varying age
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

groups so that instruction allows for higher--­order thinking and real--­world application of 

social--­emotional concepts.

(3) Organizational quality is moderate. Structured, organized, and engaging learning 

environments allow for student learning and growth. Ongoing training and support can 

be provided for WINGSLeaders to develop behavior management skills and focus on 

increasing student engagement during routines and instruction. Additionally, fine--­tuning 

the program schedule to eliminate unnecessary lag--­time (i.e. time spent on transitions 

and managerial tasks) can increase quality instructional and skill development time.

v 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Technical Report 

Introduction 

This report synthesizes data collected from Fall 2013 to Spring 2015 for an implementation study chiefly 

carried out by the College of Charleston WINGS for Kids program evaluation team members in 

Charleston, SC. This implementation study has been carried out in conjunction with an impact study 

conducted on the WINGS for Kids program led by the University of Virginia and Portland State 

University. Therefore, this study complements efforts related to the impact study by seeking to 

contextualize and illustrate program-­ ­level characteristics at play within the program, as well as within the 

larger school context.

Structure of Report 

This report is organized into three sections: (1) emotional climate, (2) instructional strategies, and (3) 

learning environment organization. Hence, data from each of the methods employed (observations, 

interviews, surveys) is presented within these overarching sections. The first two sections include 

information that was briefly covered at the 2015 WEAC meeting within the fidelity two-­­pager. The third 

section (Learning Environment Organization) includes new information that was not within the scope of 

the two-­­pager.

Method 

This report draws from four data sources: (1) video-­­recorded observations, (2) live observations, (3) 

surveys, and (4) qualitative interviews.

Observations 

Observations were conducted during program hours at the four study schools (Chicora Elementary 

School, Memminger Elementary School, and North Charleston Elementary School, and James Simons 

Elementary School) during the 2013--­2014 school year using three observational tools: the (1) Classroom 

Assessment Scoring System (CLASS; Pianta, La Paro, & Hamre, 2008), the (2) Out of School Time observation

instrument (OST; Pechman, Mielke, Russell, White, & Cooc, 2008) and the (3) Hunter--­Bailin WINGS 

Checklist.

Observers video--­recorded program activities at all four sites. Those recorded segments were later 

observed and coded using the CLASS. CLASS coding only focused on Discussion, Choice Time, and WINGS 

Works; Community Unity was excluded due to the inability of video cameras to accurately capture the 

nature of interactions within specific Nests while all staff and students were in one room. Recorded 

segments (n=137) were approximately 15 minutes long and were coded by evaluation team members at the

College of Charleston and Portland State University; approximately 15% were double--­coded with an inter-

-­rater reliability of 88.57%.

Live observations were conducted during Community Unity, Discussion, Choice Time, and WINGS Works 

activities by College of Charleston team members using the OST and the Hunter-­­Bailin Checklist. These 

activities were selected for observation because of their importance in delivering and facilitating the 

WINGS curriculum and objectives and also because of their representativeness of the week’s activities. 

OST observations consisted of approximately 6 cycles per day, with 10 minutes of activity observed per

1 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

cycle. 20% of the observations were dual –coded with an inter-­­rater reliability of 92.4% at the item-­­level 

and at least 89.7% at the indicator-­ ­level. The Hunter-­­Bailin observations consisted of approximately 3 

cycles per day, 1 cycle per activity, lasting the length of the activity. Within Community Unity, Choice Time,

and Academic Center, two segments per activity were dual-­­coded in each observation time period (fall,

spring); overall inter-­­rater reliability was 88.46%.

Time sampling data was collected at the three study schools during the 2014-­ ­2015 school year and 

consisted of observations on two Nests per school (n=6 Nests). Students were in the first, third, and fifth 

grades; one girl Nest and one boy Nest per grade. Nests were selected by random sampling. Time 

sampling observations spanned 2.5 hours per day (program day minus meal time); coded observational 

time was approximately 2 hours per day. Observers coded students’ activities in 1-­ ­minute intervals across 

the entire observation and tracked time spent in transitions.

Interviews 

In--­depth audio--­recorded interviews were conducted with WINGS staff representing the four programs in 

Charleston. Participants included WINGS Leaders (WLs), Peace Managers (PMs), Program Assistants 

(PAs), and Program Directors (PDs). In total, 26 interviews were conducted with program staff (14 WLs, 4 

PMs, 4 Pas, and 4 PDs). WLs were sampled through a combination of random and purposeful sampling 

to ensure a fair representation of those that work with younger grades, middle grades, and older grades, 

as well as a variety of levels of experience and demographics.

Interviews were also conducted with school principals in order to get an understanding of the school-­­ 

program partnership. At the time of these interviews (Spring 2015), one WINGS program was 

discontinued due to changing school demographics, leaving three study schools. Therefore, the three 

remaining study school principals in the Charleston area were interviewed (Chicora Elementary School, 

Memminger Elementary School, and North Charleston Elementary School).

Surveys 

All teachers in each of the three remaining study schools in Charleston were surveyed using paper 

surveys (n=54; 37% overall response rate). The paper survey consisted of 13 items which gauged the 

teachers’ opinions regarding the program.

Emotional Climate 

Positive, emotionally--­supportive environments help children feel connected with their peers and 

teachers; these types of environments encourage student participation in the learning environment, 

which is necessary for achievement in the early years of schooling (Ladd, Birch, & Buhs, 1999). Research 

provides that children who may be at risk for school failure benefit the most from emotionally-­­ 

supportive environments (Hamre & Pianta, 2005). Additionally, at--­risk children who are in less-­­ 

supportive environments do not fare as well academically and also tend to have more conflicts with 

teachers (2005).

2 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

o Positive Climate: (4.46)

o Negative Climate: (1.68)

o 

o 

o 

o 

o 

o

o Teacher Sensitivity: (4.01)

o Regard for Student Perspectives: (2.99)

CLASS domain: Emotional support 

The CLASS domain that focuses on aspects of emotional climate is called Emotional Support. Within this 

domain, there are four dimensions that provide observational direction: (1) positive climate, (2) negative 

climate, (3) teacher sensitivity, and (4) regard for student perspectives (see appendix, table A).

Positive climate captures the relationships between teachers and students as well as among students; 

the authors of the observational tool define this dimension as “[a reflection of] the emotional 

connection between the teacher and students and among students and the warmth, respect, and 

enjoyment communicated by verbal and nonverbal interactions” (Pianta, La Paro, & Hamre, 2008, p.22). 

Observable indicators that show positive climate include: relationships (peer-­ ­peer assistance, matched 

affect), positive affect (smiling, laughter), positive communication (positive expectations, verbal 

affection), and respect (eye contact, warm voice, respectful language) (2008).

Graph 1 shows WINGS CLASS dimension averages for the domain emotional support. The data shows the

positive climate dimension average was 4.46, within the mid-­­range of codes. This average indicates that 

in observed WINGS segments, the indicators of positive climate (for instance, matched emotion, smiling, 

laughter, positive expectations, respectful language) were sometimes observed across all segments. This

should be compared to a high-­­range code, which indicates these behavioral markers are consistently 

observed. Graph 2 shows WINGS positive climate frequencies; these frequencies show that 69.3% of

observations were coded in the mid-­ ­range. The graph also shows that only 24.8% of observations were

coded in the high-­­range for positive climate; as noted earlier, a code in the high-­­ range indicates an 

emotional connection between teacher and students as well as among students is consistently observed 

throughout the segment.

Graph 1. WINGS CLASS dimension averages for the emotional support domain. 

WINGS CLASS Emoponal Support 

Dimension Averages

7

6

5

4

3

2

1

4.46

4.01

2.99

1.68

Posipve Climate Negapve Climate Teacher Sensipvity Regard for Student

Perspecpves

High Low

Mid

Codes

3 

*A low code for the negative climate dimension is preferable (indicating low instances of negative climate indicators).
WINGS for KIDS | IMPLEMENTATION STUDY REPORT 

Graph 2. CLASS: Frequency of codes for the positive climate dimension. 

3 4 5 

40

30

20

10

0 

Frequency of Codes: 

PosiTve Climate 

37

31 27

26

7 8 

1 

1 2 6 7 

Low 

Codes 

High 

No. of Observapons 

The negative climate dimension takes into consideration the harshness of the environment. This 

dimension is defined as, “[reflecting] the overall level of expressed negativity in the classroom; the 

frequency, quality, and intensity of teacher and peer negativity are key to this scale” (2008, p. 22). This 

dimension is unique from all of the other dimensions within CLASS in that it is preferable to have a low 

code, indicating the absence of a negative climate. Observable indicators of negative climate include (all 

take into consideration interactions between teacher-­­student as well as student-­ ­student): negative affect 

(anger, harsh voice, irritability), punitive control (yelling, threats), sarcasm/disrespect (teasing, 

humiliation), and severe negativity (bullying, physical punishment) (2008).

The negative climate dimension average was 1.68, within the low-­­range of codes (see graph 1). As noted 

earlier, a code of 1 is the lowest that can be given and indicates the absence of negative indicators. The 

WINGS average indicates that at times, the program environment was coded as having instances of 

negativity. Graph 3 shows the negative climate frequencies. While 56.9% of observations were coded 

with a 1 (indicating the absence of negative climate), it is concerning that 43% of observations were 

coded higher than 1. While codes in the low-­­range (1,2) indicate no instances or very rare instances of 

negativity, codes within the mid-­ ­range (3,4,5) indicate occasional negativity (sarcasm, disrespect), and any

code within the high-­ ­range (6,7) indicates the presence of consistent negativity and instances of physical 

altercations.

Graph 3. CLASS: Frequency of codes for the negative climate dimension. 

4 

78

 

35

 

15 5 2 2

 

0

Frequency of Codes: 

NegaTve Climate 

100

80

60

40

20

0 

3 

Low 

4 

Codes 

5 6 7 

High 

No. of Observ

1 2 

apons 

*A low code for the negative climate dimension is preferable (indicating low instances of negative 

climate indicators). 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

The teacher sensitivity dimension incorporates the teacher’s awareness of his or her students’ needs. 

Here, the dimension is defined as, “[encompassing] the teacher’s awareness of and responsivity to 

students’ academic and emotional needs; high levels of sensitivity facilitate students’ ability to actively 

explore and learn because the teacher consistently provides comfort, reassurance, and encouragement” 

(2008, p.22). Observable indicators include: awareness (of students who need extra support, assistance, 

or attention), responsiveness (acknowledges emotions, provides comfort and assistance), addresses 

problems (is effective and timely at resolving problems), and student comfort (seeks support and 

guidance from teacher) (2008).

The teacher sensitivity dimension average was 4.01; this average falls within the mid--­range of codes (see 

Graph 1). This mid--­range average shows that the teacher/WINGSLeader was sometimes aware of student 

needs and was both responsive and unresponsive to students throughout an observation. An average in

the high--­range would indicate teachers/WINGSLeaders who were consistently aware of students’ needs, 

who were responsive at all times to students, and consistently effective at helping students. Graph 4

shows that 61.3% of teacher sensitivity observations were coded in the mid--­range, indicating the 

teachers/WINGS Leaders across segments were sometimes responsive and aware of students’ needs. 

The remaining frequencies show the variability in observational segments: 21.1% were coded in the low-

-­range and 17.5% were coded in the high--­range.

3 4 5 

Frequency of Codes: 

Teacher SensiTvity 

40

30

20

10

0 

26 26

35

23

15

9 

3 

1 2 6 7 

Low Codes High 

No. of Observapons 

Graph 4. CLASS: Frequency of codes for the teacher sensitivity dimension. 

5 

The regard for student perspectives dimension takes into account the support and encouragement the 

teacher provides in understanding and valuing student perspectives. In addition, this dimension takes 

into consideration the extent to which students are independent in thoughts and actions. Here, the 

dimension is defined as: “[capturing] the degree to which the teacher’s interactions with students and 

classroom activities place an emphasis on students’ interests, motivations, and points of view and 

encourage student responsibility and autonomy” (2008, p. 22). Observable indicators include: flexibility 

and student focus (incorporates students’ ideas and follows students’ lead), support for autonomy and 

leadership (gives students choices, gives students responsibility), student expression (encourages 

student talk, elicits ideas and/or perspectives), and restriction of movement (allows independence of 

movement, is not rigid with student placement) (2008).
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

6

The regard for student perspectives dimension average for WINGS was 2.99, in the low-­ ­range of codes 

(see Graph 1). In the low-­­range, teachers/WINGSLeaders are rigid with their lesson plans, often do not 

incorporate students’ ideas in the activities, fail to support student independence and leadership, and 

offer little opportunity for student talk and expression. When looking at the frequencies for the regard 

for student perspectives dimension (see graph 5), we can see that 52.5% of observations were coded in 

the mid-­­range, indicating that WINGSLeaders periodically supported students’ autonomy and only 

sometimes allowed for student talk and expression. The graph additionally shows that 42.3% of 

observations were coded in the low-­­range, showing that WINGSLeaders did not support student 

autonomy and provided few opportunities for student talk and expression.

Graph 5. CLASS: Frequency of codes for the regard for student perspectives dimension. 

50

40

30

20

10

0 

Frequency of Scores: 

Regard for Student PerspecTves 

40

1 2 6 7 

Low 

Codes High 

3 4 5 

 

31 26 

18 15 

 

7

 

0

No. of Observapons 

When compared to other studies (see table 1), WINGS dimension averages of positive climate and 

negative climate are similar (see Graph 6). The WINGS teacher sensitivity average, however, is slightly 

lower than the other studies, but still within the same mid-­ ­range. The biggest difference in averages is 

seen in the regard for student perspectives dimension; here, the 4Rs average was 1.27 points higher.

Table 1. 4Rs and Responsive Classroom study background information.

4Rs (Jones, Brown, & Aber, 2011) Responsive Classroom (Rimm-­ ­Kaufman, as 

cited in Pianta, La Paro, & Hamre, 2011)

Grade 3 (n=82) Grades 1--­5 (n=88)

45.6% Hispanic/Latino

41.1% Black/African American

61.8% living at or below 100% of the poverty 

line

53.63% ethnic minorities

35.32% eligible for free or reduced lunch

New York city Urban district in the Northeast
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

7

Graph 6. Study comparison of CLASS emotional support dimension averages.

CLASS Emoponal Support Dimension Comparisons 

Regard for Student Perspecpves 2.99 4.28 

Teacher Sensipvity 4.74 

4.01 4.6 

Negapve Climate 1.35 

1.68 2.22 

Posipve Climate 4.44 4.91 

1 2 3 4 

Codes 

4.46 

5 6 7 

Low High 

Responsive Classroom 4Rs WINGS 

OST: Youth and staff relationships 

Two components within the OST measure also provide information on the emotional climate of the 

program environment: (1) youth relationship building and (2) staff relationship building. The youth 

relationship component focuses on the students and their relationships. Observable indicators 

include: [students are] friendly and relaxed with one another, respect one another, show positive 

affect to staff, assist one another, and are collaborative (Pechman, Meilke, Russell, White, & Cooc, 

2008). Staff relationship building focuses on staff and their relationships with the students. Here, 

observable markers are: [staff] use positive behavior management, encourage participation of all, 

show positive affect toward all, attentively listen to and/or observe youth, encourage sharing of 

ideas, opinions, concerns, engage personally with youth, and guide positive peer interactions (2008; 

see appendix, table D).

WINGS OST relationship data 

Graph 7 shows WINGS youth relationship and staff relationship averages. The WINGS youth relationship

component average was 3.27; this falls at the low end of the mid-­­range of codes. A code of 3 indicates 

that the exemplar is rarely evident and a code of 5 indicates that the exemplar is evident or implicit. The

WINGS staff-­­youth relationship component average was 3.46. Here again, this average falls in the lower

end of the mid-­­range.
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

8

Graph 7. WINGS OST youth and staff--­youth relationship averages.

Gr

WINGS OST Relaponship Components Averages 

7 

6 

5 

4 

3 

2 

1 

3.27 3.46 

Youth Relaponships Staff-­­Youth Relaponships 

High Low Mid 

Codes 

When comparing WINGS relationship averages to other afterschool programs (see table 2), both 

WINGS youth relationship and staff relationship averages were lower (see Graph 8). In the staff 

relationships component, all three studies had averages in the mid-­­range, but the WINGS average was 

slightly lower (roughly 1 point) than the comparison studies. The youth relationships component 

comparison shows a much greater difference; the WINGS average was at the lower end of the mid-­­ 

range, while the New Jersey After 3 average was well within the high-­ ­range of codes, indicating 

observable evidence of consistent positive youth relationships.

Table 2. New Jersey After 3 and TASC study background information.

New Jersey After 3 (Walking Eagle, et al., 2009) The After School Corporation (TASC) 

Follow-­­Up Study (Birmingham, et al., 2005)

Grades K-­ ­8 (n=78 afterschool classes at 10 

programs)

Grades 4-­­8 (n=31 activities at 10 afterschool 

programs)

87% African American or Hispanic 

57% eligible for free or reduced lunch

No demographic data available

New Jersey New York City
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

9

Graph 8. Study comparison of OST relationship averages.

OST Relaponship Components Comparison 

3.27 

Youth Relaponships 

5.86 

3.46 

Staff-­­Youth Relaponships 4.58 

4.5 

1 2 3 

WINGS TASC 

4 

Codes 

NJ Aser 3 

5 6 7 

Low High 

*Youth relationships average included the following indicators: (1) are friendly and relaxed with one another, (2) respect one 

another, and (3) show positive affect with staff.

+Staff-­­youth relationships average included the same indicators as WINGS observations except: (1) guide positive peer interactions.

Perspectives of program staff and school personnel 

WINGSLeaders and Peace Managers

In interviews, WLs and PMs mention the importance of relationships in the WINGS program. A majority

of WLs discussed their relationships with children as the best part of their job. Here, two reiterate this:

“[The best part is] growing bonds with the kids on an individual basis. The kids are a lot of fun.” 

“[The best part is] the kids coming in and swarming you with love.”

However, the inconsistency of emotional support within the program is highlighted as a PM provides 

that when WLs are “too hard on the kids,” children’s support to learn and grow is impaired. Here, 

he/she provides:

“We have WLs even here that are too hard on the kids. When that happens the kids aren't 

getting what they need so there is no change. They are not really learning.”

Teachers 

From surveys, teachers’ perceptions regarding the program atmosphere and staff were positive. When 

given a list of adjectives to describe the atmosphere, most teachers felt it was (1) loud (59%), but also

(2) engaging (52%), and (3) positive (52%). When asked about their perceptions of the program staff, 

most teachers felt they were (1) caring (55%), (2) friendly (46%), and (3) well-­­intentioned (41%).
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Instructional Strategies 

Instructional practices that afford students the opportunity to learn content in usable and meaningful 

ways has been shown to affect cognitive development (National Research Council, 1999). This is 

observed when teachers interact with students in a way that delivers instruction, yet places priority on 

students’ comprehensive understanding, instead of on rote memorization of isolated facts. Interactions 

in the learning environment act as a mechanism to forward students’ thinking; through classroom 

discussions, students’ explanations of their thinking, and back--­and--­forth exchanges.

CLASS domain: Instructional support 

The CLASS domain that targets these aspects of the learning environment is termed Instructional 

Support. This domain is divided into three dimensions: (1) concept development, (2) quality of feedback, 

and (3) language modeling (see appendix, table B).

Concept development focuses on the ways the teacher allows his/her students to interact with the 

content of the lesson/activity in meaningful ways. The authors define this dimension as, “[measuring] 

the teacher’s use of instructional discussions and activities to promote students’ higher-­­order thinking skills 

and cognition and the teacher’s focus on understanding rather than on rote instruction” (Pianta, La 

Paro, & Hamre, 2008, p. 64). Within this dimension, there are four observable indicators that guide 

observations: analysis and reasoning (the teacher asks why and how questions, students take part in 

problem solving, prediction/experimentation), creating (students brainstorm, plan, and produce their 

own ideas and products), integration (the teacher helps connect concepts and integrates the new 

content with previous knowledge), and connections to the real world (new concepts are related to 

students’ lives).

Graph 9 shows the WINGS concept development average was 1.46, in the low-­­range of codes. This 

average indicates that the teacher/WINGSLeader rarely encouraged analysis and reasoning in 

discussions, rarely provided opportunities for the students to create their own products or ideas, 

concepts were not tied to previous knowledge, and concepts were not related to students’ lives. The 

frequencies for this dimension (see graph 10) show that 92.7% of the observations were coded in the 

low-­­range; 62% of observations were coded as 1, the lowest possible code.

Low Mid 

Codes 

WINGS CLASS Instrucponal Support 

Dimension Averages 

7 

6 

5 

4 

3 

2 

1 

1.46 2.13 1.89 

Concept Development Quality of Feedback Language Modeling 

High 

Graph 9. WINGS CLASS dimension averages for the instructional support domain.

10
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Graph 10. CLASS: Frequency of codes for the concept development dimension. 

CLASS Frequency of Codes: Concept Development 

100

80

60

40

20

0 

85

42

Low 

Codes 

High 

6

 4 0 0 0

1 2 3 4 5 6 7

No. of Observapons 

The quality of feedback dimension targets the way the teacher facilitates learning through the feedback 

he/she gives. This dimensions is defined as, “[assessing] the degree to which the teacher provides 

feedback that expands learning and understanding and encourages continued participation” (2008, p. 

72). This dimension includes the following observable indicators: scaffolding (the teacher’s use of hints 

and assistance to develop student thought), feedback loops (follow-­ ­up questions to expand student 

thinking), prompting thought processes (students explain their thinking), providing information 

(teachers expand on student thinking with additional information), and encouragement and affirmation 

(the teacher encourages and supports student persistence).

The quality of feedback dimension average was 2.13, also within the low-­­range of codes (see Graph 9). This 

indicates that the teacher/WINGSLeader rarely provided scaffolding to students, often times did not 

provide meaningful feedback to students (only perfunctory), rarely asked students to explain their 

thinking, rarely offered additional information to students to expand understanding, and rarely offered 

encouragement of students’ efforts. Frequencies show that 65.6% of observations were coded in the 

low-­­range, with 33.5% coded in the mid-­­range (see graph 11). This indicates that while most of the 

observations showed low-­­levels of quality feedback, there were some instances of mid-­­level quality 

feedback present.

Graph 11. CLASS: Frequency of codes for the quality of feedback dimension.

1 2 

CLASS Frequency of Codes: Quality of Feedback 

46

50

40

30

20

10

0 

44

32

12

2 1 

3 4 

Codes 

5 

 

6 

0 

7 

Low High 

No. of Observapons 

11
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Finally, the language modeling dimension focuses on how well the teacher and the learning 

environment supports students’ language development. Here, the dimension is defined as, “[capturing] the

quality and amount of the teacher’s use of language-­­stimulation and language-­ ­facilitation techniques” 

(2008, p. 79). This dimension is assessed through the following indicators: frequent conversation

(prevalence of teacher-­­student and student-­­student discussions), open-­ ­ended questions (questions illicit

more than one-­­word responses), repetition and extensions (the teacher repeats and extends student

responses), self-­­ and parallel talk (the teacher describes his/her actions or student actions), and advanced 

language (a variety of words are used in the learning environment).

The language modeling average of 1.89 is also in the low-­­range of codes (see graph 9). In this range, there 

are few or no conversations in the learning environment, the majority of the teacher’s/WINGSLeader’s

questions are close-­­ended, the teacher/WINGSLeader rarely incorporates techniques to develop

students’ language skills (repetition and extension, self-­­ and parallel talk), and does not use advanced 

language with students. The frequencies provide that 76.6% of observations were coded in the low-­ ­range,

with 23.3% of observations coded in the mid-­­range (see graph 12). These frequencies illustrate that while 

most of the observations had poor language stimulation, some observations had occasional instances of

mid-­ ­level language modeling.

1 2 

CLASS Frequency of Codes: Language Modeling 

60

50

40

30

20

10

0 

51 54

25

6 

1 0 0 

3 4 

Codes 

5 6 7 

Low High 

No. of Observapons 

Graph 12. CLASS: Frequency of codes for the language modeling dimension.

12

When comparing WINGS instructional support averages with others (4Rs and Responsive Classroom; table

1), we see that while WINGS averages are in the low--­range for all dimensions, both the Responsive 

Classroom and 4Rs studies have quality of feedback and concept development averages in the mid-­­ 

range; with the Responsive Classroom quality of feedback dimension average at the high--­end of the

mid-­ ­range (see Graph 13). It should be noted that at the time of the 4Rs and Responsive Classroom 

studies, the CLASS measure did not contain a language modeling dimension; hence, there was no data 

collected for that specific dimension.
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

CLASS Instrucponal Support Dimension Comparisons 

Language Modeling 1.89 

QoF 

2.13 

3.54 

4.77 

CD

1.46 3.82 3.84 

1 2 3 4 

Codes 

5 6 7 

Low High 

WINGS Responsive Classroom 4Rs 

Graph 13. Study comparison of CLASS instructional support dimension averages.

OST: Staff instructional strategies 

The OST component that targets similar indicators to the CLASS Instructional Support dimensions is 

called instructional strategies. Here, the staff are assessed on the presence and quality of how they: 

communicate goals, the purpose, and expectations, verbally recognize efforts and accomplishments, 

assist youth without taking control, ask youth to expand upon their answers and ideas, challenge youth 

to move beyond current levels, employ a variety of teaching strategies, and plan for/ask youth to work 

together (Pechman, Mielke, Russell, Whiite, & Cooc; see appendix, table E).

Graph 14 shows the WINGS OST overall average of instructional strategies was 2.57, in the low--­range of 

codes. As mentioned previously a code of 3 indicates the exemplar is rarely evident and a code of 1 

indicates the exemplar is not evident. When looking at the OST instructional strategies average per 

activity (see graph 15), we see that the lowest instructional strategies codes were observed in the 

Discussion activity, a component when WINGS deliberately delivers its social--­emotional content.

Graph 14. OST: WINGS instructional strategies average.

WINGS OST: Instrucponal Strategies 

Overall Average 

7 

6 

5 

4 

3

2.57 

2 

1 

Instrucponal Strategies 

Low Mid High 

Codes 

Graph 15. OST: WINGS instructional strategies averages by activity. Codes 

WINGS OST: Instrucponal Strategies 

Averages by Acpvity 

7 

6 

5 

4 

3 

2 

1 

2.50 2.56 2.18 2.81 

Comm. Unity Academic 

Center 

Discussion Choice Time 

WINGS Component 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

14

When comparing the WINGS OST instructional strategies dimension average to two other studies (TASC &

New Jersey After 3; table 2), we see that while both comparison studies had averages in the mid-­­ range of

codes (3.6 & 3.77), the averages were within the lower--­end of this mid--­range (see Graph 16). As stated 

previously, the WINGS average indicates that the behavioral markers (indicators) of instructional strategies 

(i.e. staff assist youth without taking control, ask youth to expand on their ideas, verbally recognize youth

efforts) were rarely evident; a mid--­range average, however, indicates the exemplars are more 

consistently evident.

Graph 16. Comparison of OST instructional strategies dimension averages.

OST Instrucponal Strategies Dimension Comparison 

WINGS 2.57 

NJ Aser 3 3.6 

TASC 3.77 

1 2 3 4 

Codes 

5 6 7 

Low High 

Hunter--‐Bailin: Teachable moments 

An additional mechanism that WINGS uses to deliver content, teachable moments, is assessed on the 

Hunter-­­Bailin measure. Teachable moments are thought of as an informal teaching method that 

provides the SEL curriculum to students throughout the day; for instance, a WL discussing delayed 

gratification with his/her Nest as the students complete homework before an ice cream party.

Teachable moments will either include references to a specific SEL objective (for instance, “Give me your 3Es-

­­eyes, ears, and energy”) or the WINGS creed (“I listen to you, and you listen to me”).

Graph 17 shows that the majority of teachable moments present within observations were of low to 

moderate quality. There were many instances of low quality teachable moments involving both SEL 

objectives (42.7%) and the creed (43.3%). This indicates that WLs either did not use teachable moments 

or used them in a counterproductive way (for example yelling “I listen to you, you listen to me” while 

disregarding the student’s needs).
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

15

Graph 17. Hunter--­Bailin: Frequency and quality of teachable moments.

Hunter-­­Bailin: Teachable Moments 

50%

45%

40%

35%

30%

25%

20%

15%

10%

5%

0%

 [VALUE] 42.7% 

31.1% 

Creed 

SEL 

25.0% 

18.3% 18.3% 

14.0% 

7.3% 

1-­ ­Low Quality 2-­ ­Moderate Quality 3-­ ­Moderate-­ ­High 4-­ ­High Quality 

Perceptions of program staff 

WINGSLeaders and Program Directors

When interviewed, WLs felt they could accomplish the goals of the activities during the program, but 

they also mentioned feelings of frustration when planning and facilitating a discussion or brainstorming 

ideas with children. One PD speaks to this and also discussed the value in offering support to WLs when 

she said:

“Discussion is very hard...WINGS just gives [the WLs] text, so it’s up to the WLs to make it fun and 

engaging. Sometimes with the Discussion, the WLs don’t understand the objective. The PDs who 

go over it on Monday and what the Discussion will look like for the week definitely have more 

successful WLs, because they can deliver it more confidently.”

Learning Environment Organization 

Research has provided that learning environments are most effective when they are well-­­managed; this 

includes the behavior of the students, the organization of activities and routines, and the use of 

interesting materials and activities coupled with teacher facilitation (Emmer & Stough, 2001).

Specifically, learning environments that incorporate positive behavior management practices and are 

productive are inclined to have the most engaged students (Emmer & Stough, 2001; Evertson, Emmer, 

Sanford, & Clements, 1983; Evertson & Harris, 1999; Brophy & Evertson, 1976; Coker, Medley, & Soar, 

1980; Good & Grouws, 1977; Stallings, 1975; Stallings, Cory, Fairweather, & Needels, 1978). Research 

also provides that in order for learning to occur, students must be interested in the activities, supporting 

the need for engaging activities and materials for active student participation (Yair, 2000). However, 

poorly managed routines, including transitions, often result in student misbehavior and ultimately take
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

16

away instructional time needed for engaged learning (Gettinger & Seibert, 2002; Sprick, Garrison, & 

Howard, 1998).

CLASS domain: Classroom organization

The CLASS domain that focuses on these aspects is called Classroom Organization. Within this domain, 

there are three dimensions that provide observational direction: (1) behavior management, (2) 

productivity, and (3) instructional learning formats (see appendix, table C).

CLASS: Behavior Management

Behavior management captures the teacher/WINGSLeader’s ability to set behavior expectations and 

deal with misbehavior; the authors of the CLASS define this dimension as “[encompassing] the teacher’s 

ability to provide clear behavioral expectations and use effective methods to prevent and redirect 

misbehavior” (Pianta, La Paro, & Hamre, 2008, p.44). Observable indicators include: clear behavior 

expectations, proactive (low escalation and reactivity), redirection of misbehavior, and student behavior 

(compliance with rules and the presence of defiance) (2008).

Graph 18 shows the CLASS behavior management dimension average falls within the mid-­­range of codes.

This mid-­­range average indicates that during observations, rules and expectations were unclear at times, 

the teacher/WINGSLeader used a mix of proactive and reactive behavior management techniques, the 

teacher/WINGSLeader was somewhat effective at redirecting behavior, and there were occasional 

episodes of misbehavior in the learning environment.

Graph 18. WINGS CLASS dimension averages for the classroom organization domain.

WINGS CLASS 

Classroom Organizapon Dimension Averages 

7 

6 

5 

4 

3 

2 

1 

4.16 4.21 

Behavior Management Producpvity Inst. Learning Formats 

 

3.5

 

 

High Mid Low

Codes 

When looking at the code frequencies for the CLASS behavior management dimension (see graph 19), 

we see that 68.6% of codes were within the mid-­­range, followed by 21.8% of codes in the high-­­range. 

These frequencies indicate that high-­ ­quality behavior management did occur in the WINGS program 

within 21.8% of the observations; where rules and expectations were clear and consistently enforced, 

the teacher/WINGSLeader used proactive techniques throughout the observation, the 

teacher/WINGSLeader was effective at redirecting misbehavior, and there was little student
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

17

misbehavior in the learning environment. However, this high-­ ­level of quality was not observed for the 

majority of observations.

Graph 19. CLASS: Frequency of codes for the behavior management dimension.

3 4 5 

CLASS Frequency of Codes: Behavior Management 

50

40

30

20

10

0 

39

31 26

24

11

2 4 

1 2 6 7 

Low Codes High 

No. of Observapons 

Hunter--‐Bailin: Behavior management techniques 

The Hunter-­ ­Bailin captures the quality of behavior techniques specific to the WINGS program in regards to 

frequency and effectiveness of implementation. These are proactive (MESS), corrective (GET PAST), 

and instructional (DDADA) techniques WINGS trains and encourages WINGSLeaders to employ in order 

to shape the behavior of their students.

The Hunter-­ ­Bailin data reveals that the quality of both proactive and corrective techniques was wide-­­ 

ranging (see graph 20). The majority of observations for both proactive (MESS; 59.1%) and corrective (GET

PAST; 59.8%) techniques are scored as moderate and moderate-­ ­high. More instances of high-­­ quality 

implementation are observed for proactive (MESS; 24.4%) in comparison to corrective (GET PAST; 8.5%) 

techniques. In contrast, the overwhelming majority of time instructional techniques (DDADA) are scored 

at a low level of quality, indicating that the technique was never used or was used in a 

counterproductive way. The range in scores indicates that students’ exposure to the program’s trademark 

techniques varied greatly in regards to frequency and effectiveness.
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

18

Graph 20. Hunter--­Bailin: Frequency and quality of behavioral techniques.

Hunter-­­Bailin: Behavioral Techniques 

80%

70%

60%

50%

40%

30%

20%

10%

0%

73.8% 

MESS 

GET PAST 

DDADA 

26.2% 

15.9% 

35.4% 

31.7% 27.4% 

24.4% 

24.4% 

12.8% 

8.5% 5.5% 

4.3% 

1-­ ­Low Quality 2-­ ­Moderate Quality 3-­ ­Moderate-­ ­High 4-­ ­High Quality 

Perceptions among staff and school partners

WINGSLeaders

Some WLs discussed behavior management as their biggest strength. One WL provides:

“I’m good at proactive prevention of potential incidents.”

However, other WLs explicitly mentioned student behavior or behavior management techniques as a 

challenge. One WL adds:

“Most difficult days, a lot of things are going wrong at once, I get a little bit 

overwhelmed. I can deal with one kid misbehaving, but when the majority are 

misbehaving, I usually have to call someone and just be like, ‘I need a reset.’ Once one is 

quiet, another one is talking; managing the big group is really my biggest difficulty.”

Still others discussed the difficulty in staying positive and managing their own behavior when faced with 

negative and difficult behavior from students. Two WLs provided:

“You can be happy all day, then kids do things that are frustrating, then you become 

upset and want to quit sometimes.”

“Lately I’ve noticed that I’m very quick to lose my cool.” 

School Principals 

Principals are uneasy that school rules and expectations for the children during the school day are not 

upheld afterschool. To this end, two principals described it as ‘loud.’ Here, they added:
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

19

“... it’s just always loud. And the one thing that I said is if the kids are loud, then the 

WINGSLeader can’t be louder, cause now you’re all loud, and [it looks like] you can’t 

manage discipline. And that’s the part that I’m not in love with with WINGS, to be honest 

with you. I’m just not in love with it.”

“The biggest concern is that when the bell rings at [X], the kids ... are running and 

shouting and yelling and so one of the concerns was, we need them to have a cleaner, 

quieter dismissal cause that’s not what the expectation is here. No one runs here, yells 

here.”

CLASS: Productivity 

Productivity encompasses the teacher’s/WINGSLeader’s ability to manage the flow of the learning 

environment; its activities and routines. This dimension is defined as, “[considering] how well the 

teacher manages instructional time and routines and provides activities for students so that they have 

the opportunity to be involved in learning activities” (2008, p. 44). Observable indicators include: 

maximizing learning time (there is something for the students to do, steady pacing), routines (students 

have a clear understanding of what to do), transitions (are brief and efficient), and preparation (the 

teacher is prepared and knows the lessons) (2008).

The WINGS CLASS data in graph 18 also shows the productivity average (4.21) was in the mid--­range. Mid-

­­range codes indicate that students had an activity to take part in most of the time, there was some 

evidence of an understanding of classroom routines, the teacher/WINGSLeader was mostly prepared to 

conduct the activity/lesson, and transitions were sometimes too frequent and too inefficient. The CLASS 

frequencies for the productivity dimension (see graph 21) show that 58% of observations were coded in 

the mid-­­range, with 25.5% in the high-­­range, and 16% in the low-­­range. This, again, shows that there were 

instances of high productivity within observations of WINGS settings, but the majority of observations

showed mid-­­level productivity. No. of Observapons

Graph 21. CLASS: Frequency of codes for the productivity dimension.

3 4 5

CLASS Frequency of Codes: Producpvity

40

30

20

10

0

24 24

32 29

18

4 6

1 2 6 7

Low Codes High
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

20

Time Sampling 

By looking at the time sampling data (see table 3), we see that students within Nests at the three study 

schools spent between 20--­46% of their time transitioning in and out of activities. Table 4 is an example 

of an observation detailing the time spent in transition for the third grade girls’ Nest in WINGS. This 

shows the Nest was in transition a total of 67 minutes during the observation period, accounting for 

46.5% of the time (out of 144 minutes).

Table 3. Percentages of time spent in transition.

Grade Gender Time in 

Transition

1 M 19.61%

1 F 21.57%

3 M 30.77%

3 F 46.53%

5 M 35.86%

5 F 31.16%

Table 4. Example time sampling observation for the third grade girls’ Nest.

Amount of Time Activity

1 Countdown

1 From watching demo to starting SEL

1 From starting SEL game to snack

5 Cleaning up snack, countdown, starting creed

6 Waiting to be dismissed from community unity to academic center

10 Lining up in community unity, walking in the hall to academic center

5 Coming into academic center room, finding a seat, waiting for pencils, 

worksheets passed out

4 Cleaning up and packing after academic center

13 Lining up in academic center, walking to cafeteria, waiting to be dismissed to 

choice time

9 Lining up, getting hula hoops for choice time, walking down hallway to choice 

time

2 Getting kids to find their line--­up position for rehearsing their hoop routine

2 Getting kids to listen

8 Lining up, walking down the hallway from choice time to dinner

67 minutes Total Time in Transition

CLASS: Instructional Learning Formats 

Instructional learning formats brings into consideration the teacher’s/WINGSLeader’s ability to facilitate 

lessons and activities in a way that engages students. This dimension is defined as, “[focusing] on the 

ways in which the teacher maximizes students’ interest, engagement, and ability to learn from lessons 

and activities” (2008, p. 57). Observable indicators of this dimension include: effective facilitation (the 

teacher is involved in the activity and uses questions to engage), variety of modalities and materials 

(considers the types of activities and materials used in engaging students), student interest (how
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

21

focused and active the students are), and clarity of learning objectives (considers the teacher’s attempts 

at focusing the students towards the objective) (2008).

Graph 18 shows the instructional learning format average was 3.5. While this average is within the mid-­­ 

range of codes, it is on the lower-­ ­end of the mid-­­range. A mid-­ ­range average indicates the 

teacher/WINGSLeader was sometimes active in facilitating student engagement within the activity, the 

students were sometimes engaged with the lesson/activity, and the teacher/WINGSLeader sometimes 

oriented students to the learning objectives. Frequencies support the prevalence of mid-­­range codes across

observations; 63.5% of codes are within the mid-­­range, while 27.7% are in the low-­­range (see graph 22). A

code in the low-­ ­range indicates no teacher/WINGSLeader facilitation towards sparking student interest, 

rare instances of student engagement, and the teacher/WINGSLeader is unsuccessful or makes no 

attempt at orienting the students towards the lesson objective during WINGS observations.

Graph 22. CLASS: Frequency of codes for the instructional learning formats dimension.

3 4 5 

40

30

CLASS Frequency of Codes: 

Instrucponal Learning Formats 

33

25

27 27

20

10

0 

13

11

1 2 6 

1 

7 

Low Codes High 

No. of Observapons 

When compared to two other studies (Responsive Classroom and 4Rs; table 1), the mid--­range WINGS 

CLASS averages were similar to the other studies, although WINGS averages were slightly lower (see 

graph 23). This graph also shows that the Responsive Classroom average of behavior management was 

within the high--­range of codes.

Graph 23. Study comparison of classroom organization dimension averages.

CLASS Classroom Organizapon Dimension Comparisons 

Instrucponal Learning Formats 3.5 

Producpvity 

4

4

.

.

2

2

1

3 

4.21 4.694.98 

Behavior Management 4.16 4.5

9

.

8

14

1 2 3 4 

Codes 

5 6 7 

Low 

Responsive Classroom 

High 

4Rs WINGS 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

OST: Youth participation 

The youth participation component of the OST measure allows the observer to focus on the students 

and the extent of their participation in the activities. This component addresses the following

observable indicators: [students are] on task, listen actively and attentively to peers and staff, contribute 

opinions, ideas, and/or concerns to discussions, have opportunities to make meaningful choices, and 

take leadership responsibility/roles (Pechman, Mielke, Russell, Whiite, & Cooc; see appendix, table F).

Graph 24 shows the WINGS youth participation average. The WINGS overall average of 2.76 falls within 

the low-­ ­range of codes, indicating the observable indicators were not evident or were rarely evident.

When we look at the youth participation within each activity (see graph 25), averages show that most 

student participation occurred during Community Unity, with an average code of 3 (at the base of the 

mid-­ ­level of codes). However, for all other activities observed, youth participation was in the low-­ ­range.

Graph 24. WINGS OST youth participation average. Graph 25. WINGS OST youth participation averages by activity.

WINGS OST: Youth Parpcipapon 

7 

6 

5 

4 

3 

2 

1 

Comm. Unity Academic 

Center 

Discussion Choice Time 

WINGS Component 

 3.00 2.41 2.40 2.89

 

Codes 

WINGS OST: Youth Parpcipapon 

7 

6 

5 

4 

3 

2 

1 

2.76 

Youth Parpcipapon 

High 

Codes 

Low Mid 

When comparing the WINGS youth participation average to another study (TASC, table 2), we see that 

the TASC study had an average of 4.49; at the higher-­ ­end of the mid-­ ­range (see Graph 26). The TASC 

average, however, combines the two dimensions of youth participation and youth relationships to result 

in this composite score.

 

22

Graph 26. Study comparison of OST youth participation averages.

*TASC youth participation average included youth relationship component indicators as well.

WINGS OST Youth Parpcipapon Comparison

WINGS 2.76

TASC 4.49

1 2 3

Low

4

Codes

5 6 7

High
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Program Staff Perception 

WINGSLeaders

During interviews, some WLs discussed the difficulty in preparing and facilitating activities to engage 

students. Here, two WLs illustrate this:

“I can interact with a couple of kids, but managing a group of twelve or ten is my, kind of...that’s 

what I’ve had to struggle with. And I think I’ve gotten better, you know. But definitely it’s like 

very teacher-­ ­oriented. Keeping kids engaged, you know.”

“And especially with things that aren’t fun, like WINGS Works: it’s difficult, it feels like we’re 

forcing it. None of the WINGSLeaders really like it, it’s hard. Every Wednesday is the most 

difficult day.”

23
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

References 

Birmingham, J., Pechman, E. M., Russell, C. A., & Mielke, M. (2005). Shared features of high performing 

after-­ ­school programs: A follow-­­up to the TASC evaluation. Washington, DC: Policy Studies 

Associates.

Brophy, J., & Evertson, C. (1976). Learning from teaching: A developmental perspective. Boston: Allyn 

and Bacon.

Center for Advanced Study of Teaching and Learning. (n.d.). Measuring and improving teacher student 

interactions in pk-­­12 settings to enhance students’ learning”(brief). 

http://curry.virginia.edu/uploads/resourceLibrary/CLASS-­­MTP_PK-­­12_brief.pdf

Coker, H., Medley, D.M., & Soar, R.S. (1980). How valid are expert opinions about effective teaching? Phi 

Delta Kappan, 62 (2), pp. 131-­­134, 149.

Emmer, E.T., & Stough, L. (2001). Classroom management: A critical part of educational psychology, with 

implications for teacher education. Educational Psychologist, 36 (2), pp. 103-­­112.

Evertson, C., Emmer, E., Sanford, J., & Clements, B. (1983). Improving classroom management: An 

experiment in elementary classrooms. Elementary School Journal, 84, pp. 173-­ ­188.

Evertson, C. & Harris, A. (1999). Support for managing learning-­­centered classrooms: The classroom 

organization and management program. In H.J. Freiberg (Ed.), Beyond behaviorism: Changing the

classroom management paradigm, pp. 59-­­74. Boston: Allyn & Bacon.

Gettinger, M. & Seibert, J.K. (2002). Best practices in increasing academic learning time. In A. Thomas 

(Ed.), Best practices in school psychology IV, pp. 773-­­787. Bethesda, MD: National Association of 

School Psychologists.

Good, T., & Grouws, D. (1977). Teaching effects: A process--­product study of fourth grade mathematics 

classrooms. Journal of Teacher Education, 28, pp. 49--­54.

Hamre, B.K. & Pianta, R.C. (2007). Can instructional and emotional support in the first grade classroom 

make a difference for children at risk of school failure? Child Development, 76(5), pp. 949-­­967.

Jones, S. M., Brown, J.L., & Aber, J.L. (2011). Two-­ ­year impacts of a universal school-­ ­based social-­­ 

emotional and literacy intervention: An experiment in translational developmental research. 

Child Development, 82 (2), pp. 533-­ ­554.

Ladd, G.W., Birch, S.H., & Buhs, E.S. (1999). Children’s social and scholastic lives in kindergarten: Related 

spheres of influence? Child Development, 70 (6), pp. 1373-­­1400.

National Research Council. (1999). How people learn: Brian, mind, experience, and school. Washington, 

DC: National Academies Press.

Pechman, E.M., Mielke, M. B., Russell, C.A., White, R.N., & Cooc, N. (2008). Out-­ ­of-­­school time (OST) 

observation instrument: Report of the validation study. Policy Studies Associates, Inc.

24
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

25

Pianta, R.C., La Paro, K.M., & Hamre, B.K. (2008). Classroom Assessment Scoring System (CLASS) Manual, K-­­3. 

Baltimore, Maryland: Brookes Publishing.

Sprick, R., Garrison, M., & Howard, L. (1998). CHAMPS: A proactive and positive approach to classroom 

management. Longmont, CO: Sopris West.

Stallings, J. (1975). Implementation and child effects of teaching practices in follow through classrooms.

Monographs of the Society for Research in Child Development, 40 (7--­8, Serial No. 163).

Stallings, J., Cory, R., Fairweather, J., & Needels, M. (1978). Early childhood education classroom 

evaluation. Menlo Park, CA: SRI International.

Walking Eagle, K.P., Miller, T.D., Cooc, N., LaFleur, J., & Reisner, E.R. (2009). Evaluation of New Jersey after

3: Reaching and engaging New Jersey’s youth through after-­ ­school programs, 2005-­ ­2008. New 

Brunswick, NJ: Policy Studies Associates, Inc.
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

26

Table A. CLASS emotional support domain, dimensions, indicators, and behavioral markers.

Dimension Observable Indicators Behavioral Markers

Positive Climate

Relationships Physical proximity 

Shared activities 

Peer assistance 

Matched affect 

Social conversation

Positive affect Smiling 

Laughter 

Enthusiasm

Positive communication Verbal affection 

Physical affection 

Positive expectations

Respect Eye contact 

Warm, calm voice

Respectful language 

Cooperation and/or sharing

Negative Climate

Negative affect Irritability 

Anger 

Harsh voice

Peer aggression

Disconnected or escalating negativity

Punitive control Yelling 

Threats

Physical control 

Harsh punishment

Sarcasm/disrespect Sarcastic voice/statement 

Teasing

Humiliation

Severe negativity Victimization 

Bullying

Physical punishment

Teacher Sensitivity

Awareness Anticipates problems and plans appropriately 

Notices lack of understanding and/or difficulties

Responsiveness Acknowledges emotions 

Provides comfort and assistance 

Provides individualized support

Addresses problems Helps in an effective and timely manner 

Helps resolve problems

Student comfort Seeks support and guidance 

Freely participates

Takes risks

Regard for Student 

Perspectives

Flexibility and student focus Shows flexibility 

Incorporates students’ ideas 

Follows students’ lead

Support for autonomy and 

leadership

Allows choice

Allows students to lead lessons 

Gives students responsibility

Student expression Encourages student talk

Elicits ideas and/or perspectives

Restriction of movement Allows movement 

Is not rigid
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

27

Table B. CLASS instructional support domain, dimensions, indicators, and behavioral markers.

Dimension Observable Indicators Behavioral Markers

Concept Development

Analysis and reasoning Why and/or how questions 

Problem solving 

Prediction/experimentation 

Classification/comparison 

Evaluation

Creating Brainstorming 

Planning 

Producing

Integration Connects concepts

Integrates with previous knowledge

Connections to the real world Real-­­world applications 

Related to students’ lives

Quality of Feedback

Scaffolding Hints 

Assistance

Feedback loops Back and forth exchanges 

Persistence by teacher 

Follow-­­up questions

Prompting thought processes Asks students to explain thinking 

Queries responses and actions

Providing information Expansion 

Clarification 

Specific feedback

Encouragement and affirmation Recognition 

Reinforcement 

Student persistence

Language Modeling

Frequent conversation Back and forth exchanges 

Contingent responding 

Peer conversations

Open--­ended questions Questions require more than a one-­­word response 

Students respond

Repetition and extension Repeats 

Extends/elaborates

Self-­ ­ and parallel talk Maps own actions with language 

Maps student action with language

Advanced language Variety of words

Connected to familiar words and/or ideas
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

28

Table C. CLASS program organization domain, dimensions, indicators, and behavioral markers.

Dimension Observable Indicators Behavioral Markers

Behavior Management

Clear behavior expectations Clear expectations 

Consistency 

Clarity of rules

Proactive Anticipates problem behavior or escalation 

Low reactivity

Monitors

Redirection of misbehavior Effective reduction of misbehavior 

Attention to the positive

Uses subtle cues to redirect 

Efficient redirection

Student behavior Frequent compliance

Little aggression and defiance

Productivity

Maximizing learning time Provision of activities 

Choice when finished 

Few disruptions

Effective completion of managerial tasks 

Pacing

Routines Students know what to do 

Clear instructions

Little wandering

Transitions Brief

Explicit follow-­­through 

Learning opportunities within

Preparation Materials ready and accessible 

Knows lessons

Instructional Learning 

Formats

Effective facilitations Teacher involvement 

Effective questioning

Expanding children’s involvement

Variety of modalities and materials Range of auditory, visual, and movement opportunities 

Interesting and creative materials

Hands-­­on opportunities

Student interest Active participation 

Listening

Focused attention

Clarity of learning objectives Advanced organizers 

Summaries

Reorientation statements
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

29

Table D. OST relationship components and indicators.

Youth Relationship 

Building

Youth are friendly and relaxed with one another 

Youth respect each other

Youth show positive affect with staff 

Youth assist one another

Youth are collaborative

Staff Relationship 

Building

Staff use positive behavior management 

Staff encourage participation of all

Staff show positive affect toward all

Staff attentively listen to and/or observe youth 

Staff encourage sharing of ideas, opinions,concerns 

Staff engage personally withyouth

Staff guide positive peer interactions

Table E. OST instructional strategies component and indicators.

Instructional Strategies Staff communicate goals, purpose, expectations Staff

verbally recognize efforts and accomplishments Staff 

assist youth without taking control

Staff ask youth to expand upon their answers and ideas 

Staff challenge youth to move beyond current levels 

Staff employ a variety of teaching strategies

Staff plan for/ask youth to work together

Table F. OST youth participation component and indicators.

Youth Participation Youth are on task

Youth listen actively and attentively to peers and staff

Youth contribute opinions, ideas, and/or concerns to discussions 

Youth have opportunities to make meaningful choices

Youth take leadership responsibility/roles