APPENDIX A- WINGS Learning Objectives 

The WINGS program specifies specific skills and behaviors as its learning objectives. These learning 

objectives seek to both build increased positive skills and behaviors and avoid negative, distracting and 

disruptive behaviors. Each week, WINGS leaders focus on one of the 30 learning objectives listed below. 

 

Self-awareness 

A. Kids will be able to identify their feelings. 

1. Kids will understand that there is a wide range of emotion within the human experience. They 

will develop a vocabulary that demonstrates their understanding of this range. 

2. Kids will be able to identify the ways in which a wide range of emotions manifests itself within 

one’s body.

B. Kids will increase their introspection skills, noting patterns in their emotional lives, recognize their 

wants in relationship to their needs, and recognize their blessings. 

3. Kids will learn to observe and identify patterns in both their feelings and their responses to 

those feelings. 

4. Kids will recognize that their wants may sometimes be complementary to their needs and at 

other times conflict with them. They will identify their blessings. 

C. Kids will demonstrate increased self-acceptance. They will learn the process of knowing oneself 

and monitoring internal self-talk. They will show an ability to identify and value their strengths and 

understand their weaknesses. 

5. Kids will claim and value personal strengths, weaknesses and individuality. 

6. Kids will view themselves as having emerging personalities, recognizing which traits they are 

proud of and which traits might not be the best. 

7. Kids will recognize the need to consciously encourage themselves. Kids will monitor negative 

self-talk and replace critical comments with supportive ones. 

Self-Management 

A. Kids know how to regulate and apply appropriate emotional responses to life situations. 

8. Kids will understand the dynamic between thoughts, feelings, and actions, and that this 

understanding assists in managing impulses. 

9. Kids will understand the factors that can cause an emotional hijacking. 

B. Kids will learn techniques for handling their emotions. 

10. Kids will brainstorm alternative emotional responses to negative situations and predict their 

consequences to choose the best response. Kids will practice evaluating the level of success in 

their response. 

11. Kids will brainstorm alternative emotional responses to positive situations and predict their 

consequences to choose the best response. Kids will practice evaluating the level of success in 

their response. 

12. Kids will learn how to focus their attention inward to limit distractions. 

13. Kids will identify circumstances in which they experience worry, fear, anger, or other 

stressful emotions. 

14. Kids will learn alternative responses to stress including: deep breathing, physical exercise, 

and creative expression. 

Responsible Decision Making 

A. Kids will demonstrate personal decision-making skills. They will know how to identify alternative 

actions and their consequences. 
15. Kids will learn how to make an agreement with themselves or others and understand the 

importance of taking responsibility for upholding agreements. Kids will learn what to do if 

agreements are broken. 

16. Kids will understand their responsibility for positive and negative outcomes. 

17. Kids will learn how to brainstorm alternative actions and predict consequences before 

choosing the action they will take. Kids will practice evaluating the level of success in the action 

chosen. 

18. Kids will learn what peer pressure looks and feels like. Kids will learn how to say “no.”

B. Kids will set goals and understand the need for delaying gratification and accepting personal 

responsibility. 

19. Kids will develop goal-setting strategies. Kids will heighten motivation by recognizing that 

small successes build on one another. Kids will develop an ability to be patient and delay 

gratification. 

20. Kids will recognize the importance of attitude and its influence on behavior. 

Kids will apply perseverance in the face of disappointment, creating resiliency. Kids will 

understand how competition acts as a motivating force. 

Social Awareness 

A. Kids will develop empathy: the ability to understand other people’s nonverbal and verbal

emotional messages. 

21. Kids will identify the emotions manifested in others through an awareness of facial 

expression, tone of voice, pitch, and gesture. Kids will learn skills to understand the emotional 

state of others. 

B. Kids will demonstrate an understanding that differences amongst people make life rich and 

interesting, and that those differences are to be valued. Kids will understand the importance of nonprejudicial attitudes. 

22. Kids will increase their sensitivity to individual preferences and differences of others. Kids 

will acknowledge and appreciate the points of view of others. Kids will learn methods to express 

their understanding and support. 

23. Kids will learn the many ways in which people can differ and how spending time with 

different people can have a positive effect. They will demonstrate an understanding that all 

people are given gifts and have challenges in life. Prejudicial attitudes will diminish as a result of 

these interactions. 

Relationship Skills 

A. Kids will be able to identify components of a trusting relationship, communicate honestly and 

work cooperatively with others. 

24. Kids will identify the components of a trusting relationship. They will experience sharing 

confidences and maintaining confidentiality in building a safe and trusting environment. 

25. Kids will work in teams to recognize the value of each individual’s contribution as well as the 

benefits of teamwork. 

B. Kids will demonstrate problem-solving and conflict resolution skills. 

26. Kids will understand how conflicts escalate and learn skills to keep conflicts from escalating. 

They will learn and practice techniques for problem solving and conflict resolution. 

27. Kids will learn ways to make amends. 

28. Kids will recognize the difference between win/win and win/lose outcomes. 

They will recognize that conflict and change are natural and valuable components of life and are 

not to be feared or avoided. 
C. Kids will demonstrate communication skills through expressing their feelings and thoughts 

effectively, actively listening to others, offering supportive feedback, and assertively expressing their 

own needs.

29. Kids will recognize the underlying intent and/or motivation in a comment.

They will understand the importance of positive feedback and learn how to give it 

appropriately. They will learn to make supportive, rather than critical, suggestions.

30. Kids will learn how to express their feelings and thoughts and will practice doing so 

assertively, not passively or aggressively. They will learn active and reflective listening 

techniques by learning how to ask clarifying questions and understand their importance.
APPENDIX B- WINGS Program Components 

The WINGS program has a weekly structure and curriculum that is described below. 

A. COMMUNITY UNITY (30 Minutes daily) 

The WINGS program begins each day with Community Unity, which consists of four parts: 

1. The Welcome, when WINGS Leaders greet incoming students; 

2. Eat and Meet, providing nutritious snacks with group interaction; 

3. Circle Time, when leaders connect with kids and engage them in brief activities; 

4. Good News and Announcements, for sharing student information with everyone. 

During Community Unity, the kids gather in the cafeteria. This is the time they all recite Our Creed. The 

Nests of 10-12 kids sit in circles playing a game. Their WINGS Leader, a college student, teaches the life 

skills curriculum through team-building and group activities. 

Then the Program Director (PD) introduces the learning objective for the week through storytelling. For 

example, the PD tells the kids about the time when she was in a play and forgot her lines because she 

was looking out into the audience. She asks for other staff members and kids to come up and give 

examples of when they got distracted and it ended badly. 

B. DISCUSSION (40 Minutes Monday) 

Children begin each week with the Creed: 

The WINGS Creed

I soar with WINGS. Let me tell you why. 

I learn lots of skills that help me reach the sky.

I love and accept who I am on the inside 

and know my emotions are nothing to hide

Life’s full of surprises that make me feel different ways. 

If I can control myself, I will have much better days.

I understand the choices I make should be what’s best for me to do, 

and what happens is on me and not any of you.

I understand others are unique. I want to learn more about everyone I meet. 

I want to step into their shoes and see what they are going through.

I am a friend. I support and trust. Working together is a must.

Kind and caring I will be. I listen to you. You listen to me.
I soar with WINGS. I just told you why. 

All of these things are why I fly high.

Following the Creed, the learning objective for the week is introduced. Children learn one new objective 

each week in a way that is developmentally appropriate. After introducing the objective, WINGS leaders 

check for understanding either through role-playing, sharing personal stories, or telling a fictional story 

and asking children to solve social problems together. 

C. CHOICE TIME (40 minutes Tues/Thurs)

As a regular part of our life skills curriculum, our kids have one Choice Time twice a week over the 

course of a semester. Choice Time offers a group of electives divided into appropriate developmental 

levels and utilizes collaborations with community partners. Once kids make their choices, they pursue 

their electives for one semester. Many electives will be suggested by the students which increases their 

involvement.

To build decision-making skills, students will think through their interests and strengths and discuss 

them with teachers when selecting their Choice Time Activity. Two days a week the kids participate in 

activities like dance, exercise and nutrition, history, music, computer capabilities, reading and writing, 

science, chess or visual and verbal puzzle-solving, art, sports, and photography. To strengthen and 

develop affinities, students are encouraged to select activities that build upon their strengths or allow 

them to explore their curiosities.

D. ACADEMIC CENTER (40 minutes Mon-Thurs)

Academic Center comes at the end of the day. All students spend 40 minutes in quiet Academic Centers 

where staff and volunteers provide assistance and encouragement in a productive atmosphere and the 

adult-student ratio is 1 to 12. Keeping with the learning objective example, a WINGS Leader reinforces 

skills which can be applied to a classroom setting.

E. WINGSWORKS (40 minutes Weds)

The life skills curriculum includes WINGSWorks: student-led community service around the school and 

community. Research demonstrates that when students contribute through activities that serve others, 

it improves attachment to school and increases their own pride and self-confidence.

F. WILDWINGS (90 minutes Friday)

WildWINGS is a school-wide event to end the weekly life skills curriculum. WildWINGS includes games, 

discussions and role-playing that help students comprehend the relationship between thoughts, 

emotions, and actions. It’s always something new – and the emphasis is on fun and team building. 
APPENDIX C- WINGS Organizational Chart 

Chief Executive 

Officer 

Chief Strategy 

Officer 

Chief Financial 

Officer 

Chief Program 

Officer 

Executive Directors 

(city level) 

Regional 

Operations 

Directors 

Program Directors 

 (school level) 

Wings Leaders 

Peace Managers 

Program Assistant 

Development 

Director 

Director of 

Community and 

Engagement 
APPENDIX D

Implementation Study Report 

WINGS Program Evaluation 

College of Charleston 

Summer 2015
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Contents 

Executive Summary and Conclusions-------------------------------------------------------------- i

Technical Report----------------------------------------------------------------------------------------- 1

Method----------------------------------------------------------------------------------------------------- 1

Emotional Climate--------------------------------------------------------------------------------------- 2

Instructional Strategies-------------------------------------------------------------------------------- 10

Learning Environment Organization---------------------------------------------------------------- 15

References------------------------------------------------------------------------------------------------ 24

Appendix-------------------------------------------------------------------------------------------------- 26

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Executive Summary and Conclusions 

The WINGS for Kids Program Evaluation addresses the quality of the implementation of the WINGS 

program at the Nest level between WINGSLeaders and their students, in terms of (1) emotional and 

relationship quality, (2) instructional quality, and (3) organizational quality. In the following technical 

report, we cover these sections in more detail within corresponding sections: (1) emotional climate, (2) 

instructional strategies, and (3) learning environment organization. Multiple raters and measures 

converged to create an overall picture of the quality of interactions in Nests. 

The findings presented in this report provide evidence that quality varies greatly among Nests. Across 

multiple measures, scores ranged from the minimum to the maximum. High scores indicate that Nests 

are capable of being friendly, supportive, and nurturing places that are conducive to learning and 

growth. On the other hand, low scores reflect a presence of negativity and hostility where management 

is lacking and the potential for student growth and learning is weakened. The range of experiences 

across Nests suggests that implementation fidelity is inconsistent. Opportunities to improve consistency 

may include attention to initial and sustained training and mentoring. Below, we discuss findings in 

more detail. 

Emotional and Relationship Quality. Children thrive in emotionally-supportive learning environments 

that encourage student participation and positive relationships; this is particularly true for children at 

risk for school failure (Hamre & Pianta, 2005; Ladd, Birch, & Buhs, 1999). However, at-risk children who 

are in less-supportive environments often have more conflicts with teachers and do not fare as well 

academically (Hamre & Pianta, 2005). Observational data collected by the Classroom Assessment 

Scoring System (CLASS) and the Out-of-School-Time (OST) measures provide information regarding the 

emotional and relationship quality within Nests. On the CLASS, WINGS Nests received a mid-range 

average (4.46 on a 1 to 7 scale) for the positive climate dimension, which is comparable to elementary 

school classrooms in other studies (4.44 (4Rs1

); 4.91 (Responsive Classroom 2)). The frequency of codes 

indicate that a majority of observations (69.3%) were coded in the mid-range, with 24.8% within the 

high-range, and a small percentage (5.8%) in the low-range. High-range codes illustrate the presence of 

consistently warm, supportive relationships, positive communication, and respect; low codes indicate 

the absence of these positive environmental indicators. 

Furthermore, the CLASS negative climate dimension measures the presence of negativity in the WINGS 

Nests. By looking at code frequencies, we can see that WINGS Nests were coded the vast majority of 

times (82.4%) in the low-range, which is preferable for this dimension and indicates the absence of or 

very rare instances of negativity within the environment. However, it should be noted that some (16%) 

observations were coded in the mid-range, indicating mild displays of irritability, occasional yelling, 

sarcasm, and disrespect; and two observations were coded as having instances of severe negativity 

(score of 6) within the high-range, indicating physical aggression among children. 

 

1

 Study of third grade classrooms (n=82) in New York city with 45.6% Hispanic/Latino and 41.1% African American 

children; 61.8% of children were living at or below 100% of the poverty line at the time of the study (Jones, Brown, 

& Aber, 2011). 

2

 Study of first through fifth grade classrooms (n=88) in an urban district in the northeast. 53.63% were ethnic 

minorities and 35.32% were eligible for free or reduced lunch (Rimm-Kaufman, as cited in Pianta, La Paro, & 

Hamre, 2011). 

i

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

The OST averages of youth relationships (3.27 on a scale of 1 to 7) and staff-youth relationships (3.46 on 

a scale of 1 to 7) fall within the mid-range; when comparing WINGS relationship averages to other 

afterschool programs , staff-youth relationship averages as measured by the OST were more than 1-

point lower (3.46 (WINGS); 4.5 (New Jersey After 33

)) and the youth relationship average was more than 

2-points lower (3.27 (WINGS); 5.86 (New Jersey After 3)). 

Instructional Quality. Instructional quality can have a profound influence on student engagement and 

learning. Relatedly, students who are attending to academic tasks are less likely to be disruptive and/or 

off-task. Instructional quality focuses on the processes that effectively support cognitive and language 

development. WINGS delivers its social-emotional curriculum through instruction which is guided by 

thirty weekly social and emotional learning objectives and lessons. Thus, the quality of instruction is 

important, if not key, to effectively supporting the social and emotional development of WINGS kids. 

On observational measures (OST and CLASS), Nest averages fall in the low-range for instructional quality. 

On the OST dimension of instructional strategies the WINGS average was 2.57 (in the low-range), while 

averages of two comparison studies were within the mid-range (3.6 (New Jersey After 3); 3.77 (The After 

School Corporation (TASC4)); the same is true when comparing WINGS averages on the CLASS 

dimensions of concept development (1.46 (WINGS); 3.82 (Responsive Classroom); 3.84 (4Rs)) and quality 

of feedback (2.13 (WINGS); 4.77 (Responsive Classroom); 3.54 (4Rs)). Within the low range of quality, 

learning environments are characterized by drill-oriented activities, teacher or WINGSLeader-controlled 

conversations, and close-ended questions that focus on rote information and recall. This stands in 

comparison to what is possible with a high-level of instruction; for example, an environment that 

incorporates problem solving, creativity, brainstorming, teachable moments (a key provision of the 

WINGS curriculum), back-and-forth exchanges, and the use of advanced language and open-ended 

questions. 

Improving the quality of instructional processes is particularly important for schools and out-of-schooltime programs, like WINGS, that serve students who are at risk for school failure. In one study, students 

from families with low-parental education who were placed in first-grade classrooms offering mid- tohigh instructional quality displayed academic achievement at similar levels as peers from families with 

higher parental education; whereas those students in low instructional quality classrooms fell further 

behind (Hamre & Pianta, 2005). Moreover, we must remember that differences in just over 1 point on 

the CLASS 7-point scale translate into improved achievement and social skill development for students 

(CASTL, n.d.). In essence, this suggests that if WINGS demonstrated even small changes in effective 

interactions, it would have practical implications for the success of its students. 

Organizational Quality. According to the WINGS training manual, “WINGS works with the toughest kids 

in the toughest schools,” and the program goal is to transform each child through improved behavior 

and self-regulation. Common sense tells us that learning environments with well-behaved students 

function best; however, the relationship is two-fold. The development and expression of self-control and 

self-regulatory skills is highly dependent on the nature of the learning environment (Anderson, Evertson, 

& Emmer, 1980; Emmer & Stough, 2001; Kounin, 1970; Pintrich, 2000; Sanford & Evertson, 1981; Soar & 

 

3

 Study of programs (n=10) for children in grades kindergarten through eight in New Jersey; 87% of children were 

African American or Hispanic and 57% were eligible for free or reduced lunch. 

4

 Study of programs (n=10) for children in grades four through eight in New York City (Birmingham, et al., 2005). 

ii

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Soar, 1979). In other words, students exhibit better self-regulation in organized, structured, and 

engaging learning environments.

Observational data collected by the CLASS shows that WINGS Nests received a behavior management

average in the mid-range (4.16 on a 1 to 7 scale). Classrooms in the 4Rs and Responsive Classroom 

studies had averages in the mid- to mid-high range (4.98 (4Rs); 5.14 (Responsive Classroom)). Beyond 

averages, a look at the frequency of scores reveals variability in behavior management quality: 19% of 

Nest observations received a code of 6 (high level), indicating clearly stated rules and expectations, 

consistent use of proactive techniques, individualized praise, and well-behaved students. Conversely, 

28% of observations received a score of 3 (at the base of mid- level codes), indicating inconsistently 

enforced expectations, periodic episodes of misbehavior with time taken away from learning to manage 

behavior, and more reactive than proactive responses. Thus, observations during the school year reveal 

variability in practice among Nests. Notably, learning behavior management skills is one of the most 

frequently identified problems by pre-service teachers (Briton & Holten, 1989; Greenlee & Ogletree, 

1993; Meister & Melnick, 2003; Avalos & Aylwin, 2007; Ozturk, 2008; Ulvik et al. 2009). Consequently, 

behavior management skills are difficult to teach except through experience, when teachers have the 

opportunity to work with a diverse classroom of students (Debreli, 2013; Johnson, 1996). 

Given that classrooms run the smoothest when children have interesting things to do, the organization 

of the learning environment also takes into consideration student engagement and the ways in which 

the teacher/WINGSLeader facilitates that engagement. WINGS describes itself as “an instruction-based 

learning curriculum weaved into an active and engaging after-school program,” thus student 

engagement is one of its key provisions. Instructional learning formats, a CLASS dimension, focuses on 

the teacher and the extent to which he/she facilitates student engagement. The WINGS average for this 

dimension falls in the mid-range (3.5 on a 1 to 7 scale), within the same range but slightly lower than the 

Responsive Classroom and 4Rs studies (4.23 and 4.21 respectively). Here we see that the difference 

between WINGS and these other studies is slight (0.73 and 0.71), however it is important to know that 

differences in just over 1 point on the CLASS 7-point scale translate into improved achievement and 

social skill development for students (CASTL, n.d.). A look at the frequency of scores reveals variability 

in quality that is not evident when considering the overall average. While most of the observations 

(63.5%) were coded in the mid-range, 27.7% of observations were coded in the low-range, indicating 

that the teacher/WINGSLeader made few attempts at guiding students to learning objectives, did not 

provide or was inconsistent in providing interesting, creative materials and hands-on opportunities, and 

did little to expand students’ involvement or ask questions. A small percentage of observations were 

coded in the high-range (8.75%). 

The OST assesses student engagement in its youth participation component. The WINGS average within 

this component falls at the upper end of the low-range (2.76 on a 1 to 7 scale). This average indicates 

that youth were not at all or rarely on task, actively listening, contributing to discussions, or taking 

leadership roles. The lack of participation captured on OST is meaningful for WINGS because the 

cornerstone of its social and emotional learning curriculum is Discussion, a twenty-minute lesson in 

which WINGSLeaders introduce and discuss with students the weekly learning objective. The WINGS 

average for youth participation as assessed during this specific component (2.40 on a 1 to 7 scale) 

indicates that students are not effectively or actively participating in one of the most important 

instructional components of the program. 

iii

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Additionally, poorly managed routines contribute to less instructional time, student disengagement, and 

often lead to student misbehavior (Gettinger & Seibert, 2002; Sprick, Garrison, & Howard, 1998). 

Observational time sampling data that was collected on 6 Nests during WINGS program hours revealed 

that students spent 20-46% of their time in transition. 

Stakeholder Perceptions. Observations, while perhaps the most important type of information, 

provide only one lens with which to view WINGS. Stakeholders, such as principals, teachers, and 

program staff, lend a unique and valuable perspective. Principals and program staff were 

interviewed and teachers were surveyed in order to capture a fuller and more nuanced 

description of the significance of WINGS as well as how it functions. When given a list of 

adjectives, teachers described WINGS as loud (59%), but also engaging (52%) and positive (52%). 

They described WINGS staff as caring (55%), friendly (46%), and well-intentioned (41%). 

Additionally, WINGSLeaders often described their work environment as having “a family vibe” 

and discussed relationships with their students as the best part of their day. 

Nevertheless, the qualitative interviews captured a broad range of experiences that also reflect 

the variability in scores on the observational measures. Many WINGSLeaders reported behavior 

management as their biggest challenge, while only a few reported it as one of their strengths. 

Again, this finding is common among pre-service teachers. Most striking was the variability 

within individual WINGSLeader reports. Specifically, many WINGSLeaders affirmed that they had

a positive relationship with students and were “most of the time” able to meet instructional 

goals, but would then go on to further explain that they felt overwhelmed when they 

encountered behavior management challenges and when struggling to keep kids engaged with 

lessons that were “too easy,” “over their heads,” or “boring.” This is to say that implementation 

quality appears to be variable not only depending on the WINGSLeader, but on the day and the 

activity. Accordingly, in interviews, principals mentioned that program implementation was less 

than ideal. One principal provided, “I think that the program has good intentions, [but] it is not 

being implemented well.”

Conclusions. Taken together, all data collection efforts uncovered areas of growth for the 

program that can have a great impact on program functioning and, consequently, student 

learning and growth. Here, we offer conclusions and suggestions to guide improvement efforts. 

(1) Emotional and relationship quality is moderate. While WINGS averages are roughly 

similar to other elementary school classrooms and out-of-school-time programs, this 

should be a strength of a social-emotional intervention program targeting at-risk youth. 

WINGSLeaders would benefit from constant, ready support on the job; this can be 

accomplished by having extra support staff ready when challenges occur. Additionally, 

Nest sizes could be smaller to aid the bond between WINGSLeader and children, and in 

turn help with behavior management. Finally, staff selection should be a priority; 

understanding characteristics of successful WINGSLeaders is an area for further study. 

(2) Instructional quality is poor. Enhancing the instruction of the WINGS curriculum will 

increase student learning and growth. This can be ameliorated through training and 

ongoing support for WINGSLeaders to focus on tailoring instruction for varying age 

iv

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

groups so that instruction allows for higher-order thinking and real-world application of 

social-emotional concepts. 

(3) Organizational quality is moderate. Structured, organized, and engaging learning 

environments allow for student learning and growth. Ongoing training and support can 

be provided for WINGSLeaders to develop behavior management skills and focus on 

increasing student engagement during routines and instruction. Additionally, fine-tuning 

the program schedule to eliminate unnecessary lag-time (i.e. time spent on transitions 

and managerial tasks) can increase quality instructional and skill development time. 

v 

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Technical Report 

Introduction 

This report synthesizes data collected from Fall 2013 to Spring 2015 for an implementation study chiefly 

carried out by the College of Charleston WINGS for Kids program evaluation team members in 

Charleston, SC. This implementation study has been carried out in conjunction with an impact study 

conducted on the WINGS for Kids program led by the University of Virginia and Portland State 

University. Therefore, this study complements efforts related to the impact study by seeking to 

contextualize and illustrate program-level characteristics at play within the program, as well as within 

the larger school context. 

Structure of Report 

This report is organized into three sections: (1) emotional climate, (2) instructional strategies, and (3) 

learning environment organization. Hence, data from each of the methods employed (observations, 

interviews, surveys) is presented within these overarching sections. The first two sections include 

information that was briefly covered at the 2015 WEAC meeting within the fidelity two-pager. The third 

section (Learning Environment Organization) includes new information that was not within the scope of 

the two-pager. 

Method 

This report draws from four data sources: (1) video-recorded observations, (2) live observations, (3) 

surveys, and (4) qualitative interviews. 

Observations 

Observations were conducted during program hours at the four study schools (Chicora Elementary 

School, Memminger Elementary School, and North Charleston Elementary School, and James Simons 

Elementary School) during the 2013-2014 school year using three observational tools: the (1) Classroom 

Assessment Scoring System (CLASS; Pianta, La Paro, & Hamre, 2008), the (2) Out of School Time 

observation instrument (OST; Pechman, Mielke, Russell, White, & Cooc, 2008) and the (3) Hunter-Bailin 

WINGS Checklist. 

Observers video-recorded program activities at all four sites. Those recorded segments were later 

observed and coded using the CLASS. CLASS coding only focused on Discussion, Choice Time, and WINGS 

Works; Community Unity was excluded due to the inability of video cameras to accurately capture the 

nature of interactions within specific Nests while all staff and students were in one room. Recorded 

segments (n=137) were approximately 15 minutes long and were coded by evaluation team members at 

the College of Charleston and Portland State University; approximately 15% were double-coded with an 

inter-rater reliability of 88.57%.

Live observations were conducted during Community Unity, Discussion, Choice Time, and WINGS Works 

activities by College of Charleston team members using the OST and the Hunter-Bailin Checklist. These 

activities were selected for observation because of their importance in delivering and facilitating the 

WINGS curriculum and objectives and also because of their representativeness of the week’s activities. 

OST observations consisted of approximately 6 cycles per day, with 10 minutes of activity observed per 

1 

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

cycle. 20% of the observations were dual –coded with an inter-rater reliability of 92.4% at the item-level 

and at least 89.7% at the indicator-level. The Hunter-Bailin observations consisted of approximately 3 

cycles per day, 1 cycle per activity, lasting the length of the activity. Within Community Unity, Choice 

Time, and Academic Center, two segments per activity were dual-coded in each observation time period 

(fall, spring); overall inter-rater reliability was 88.46%. 

Time sampling data was collected at the three study schools during the 2014-2015 school year and 

consisted of observations on two Nests per school (n=6 Nests). Students were in the first, third, and fifth 

grades; one girl Nest and one boy Nest per grade. Nests were selected by random sampling. Time 

sampling observations spanned 2.5 hours per day (program day minus meal time); coded observational 

time was approximately 2 hours per day. Observers coded students’ activities in 1-minute intervals 

across the entire observation and tracked time spent in transitions. 

Interviews 

In-depth audio-recorded interviews were conducted with WINGS staff representing the four programs in 

Charleston. Participants included WINGS Leaders (WLs), Peace Managers (PMs), Program Assistants 

(PAs), and Program Directors (PDs). In total, 26 interviews were conducted with program staff (14 WLs, 4 

PMs, 4 Pas, and 4 PDs). WLs were sampled through a combination of random and purposeful sampling 

to ensure a fair representation of those that work with younger grades, middle grades, and older grades, 

as well as a variety of levels of experience and demographics. 

Interviews were also conducted with school principals in order to get an understanding of the schoolprogram partnership. At the time of these interviews (Spring 2015), one WINGS program was 

discontinued due to changing school demographics, leaving three study schools. Therefore, the three 

remaining study school principals in the Charleston area were interviewed (Chicora Elementary School, 

Memminger Elementary School, and North Charleston Elementary School).

Surveys 

All teachers in each of the three remaining study schools in Charleston were surveyed using paper 

surveys (n=54; 37% overall response rate). The paper survey consisted of 13 items which gauged the 

teachers’ opinions regarding the program.

Emotional Climate

Positive, emotionally-supportive environments help children feel connected with their peers and 

teachers; these types of environments encourage student participation in the learning environment, 

which is necessary for achievement in the early years of schooling (Ladd, Birch, & Buhs, 1999). Research 

provides that children who may be at risk for school failure benefit the most from emotionallysupportive environments (Hamre & Pianta, 2005). Additionally, at-risk children who are in lesssupportive environments do not fare as well academically and also tend to have more conflicts with 

teachers (2005). 

2 

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

CLASS domain: Emotional support 

The CLASS domain that focuses on aspects of emotional climate is called Emotional Support. Within this 

domain, there are four dimensions that provide observational direction: (1) positive climate, (2) negative 

climate, (3) teacher sensitivity, and (4) regard for student perspectives (see appendix, table A).

Positive climate captures the relationships between teachers and students as well as among students; 

the authors of the observational tool define this dimension as “[a reflection of] the emotional 

connection between the teacher and students and among students and the warmth, respect, and 

enjoyment communicated by verbal and nonverbal interactions” (Pianta, La Paro, & Hamre, 2008, p.22). 

Observable indicators that show positive climate include: relationships (peer-peer assistance, matched 

affect), positive affect (smiling, laughter), positive communication (positive expectations, verbal 

affection), and respect (eye contact, warm voice, respectful language) (2008). 

Graph 1 shows WINGS CLASS dimension averages for the domain emotional support. The data shows 

the positive climate dimension average was 4.46, within the mid-range of codes. This average indicates 

that in observed WINGS segments, the indicators of positive climate (for instance, matched emotion, 

smiling, laughter, positive expectations, respectful language) were sometimes observed across all 

segments. This should be compared to a high-range code, which indicates these behavioral markers are 

consistently observed. Graph 2 shows WINGS positive climate frequencies; these frequencies show that 

69.3% of observations were coded in the mid-range. The graph also shows that only 24.8% of 

observations were coded in the high-range for positive climate; as noted earlier, a code in the highrange indicates an emotional connection between teacher and students as well as among students is 

consistently observed throughout the segment. 

o Positive Climate: (4.46)

o Negative Climate: (1.68) 

o

o

o

o

o

o

o Teacher Sensitivity: (4.01) 

o Regard for Student Perspectives: (2.99) 

4.46

1.68

4.01

2.99

1

2

3

4

5

6

7

Positive Climate Negative Climate Teacher Sensitivity Regard for Student

Perspectives

Codes

WINGS CLASS Emotional Support 

Dimension Averages

Mid

Graph 1. WINGS CLASS dimension averages for the emotional support domain. 

Low High 

*A low code for the negative climate dimension is preferable (indicating low instances of negative climate indicators). 

3 

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

1

7

27

31

37

26

8

0

10

20

30

40

1 2 3 4 5 6 7

No. of Observations

Codes

Frequency of Codes: 

Positive Climate

Graph 2. CLASS: Frequency of codes for the positive climate dimension. 

Low High 

The negative climate dimension takes into consideration the harshness of the environment. This 

dimension is defined as, “[reflecting] the overall level of expressed negativity in the classroom; the 

frequency, quality, and intensity of teacher and peer negativity are key to this scale” (2008, p. 22). This 

dimension is unique from all of the other dimensions within CLASS in that it is preferable to have a low

code, indicating the absence of a negative climate. Observable indicators of negative climate include (all 

take into consideration interactions between teacher-student as well as student-student): negative 

affect (anger, harsh voice, irritability), punitive control (yelling, threats), sarcasm/disrespect (teasing, 

humiliation), and severe negativity (bullying, physical punishment) (2008). 

The negative climate dimension average was 1.68, within the low-range of codes (see graph 1). As noted 

earlier, a code of 1 is the lowest that can be given and indicates the absence of negative indicators. The 

WINGS average indicates that at times, the program environment was coded as having instances of 

negativity. Graph 3 shows the negative climate frequencies. While 56.9% of observations were coded 

with a 1 (indicating the absence of negative climate), it is concerning that 43% of observations were 

coded higher than 1. While codes in the low-range (1,2) indicate no instances or very rare instances of 

negativity, codes within the mid-range (3,4,5) indicate occasional negativity (sarcasm, disrespect), and 

any code within the high-range (6,7) indicates the presence of consistent negativity and instances of 

physical altercations. 

Graph 3. CLASS: Frequency of codes for the negative climate dimension. 

78

35

15 5 2 2

0

0

20

40

60

80

100

No. of Observations

1 2 3 4 5 6 7

Codes

Frequency of Codes: 

Negative Climate

Low High 

*A low code for the negative climate dimension is preferable (indicating low instances of negative 

climate indicators). 

4 

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

The teacher sensitivity dimension incorporates the teacher’s awareness of his or her students’ needs. 

Here, the dimension is defined as, “[encompassing] the teacher’s awareness of and responsivity to 

students’ academic and emotional needs; high levels of sensitivity facilitate students’ ability to actively 

explore and learn because the teacher consistently provides comfort, reassurance, and encouragement” 

(2008, p.22). Observable indicators include: awareness (of students who need extra support, assistance, 

or attention), responsiveness (acknowledges emotions, provides comfort and assistance), addresses 

problems (is effective and timely at resolving problems), and student comfort (seeks support and 

guidance from teacher) (2008). 

The teacher sensitivity dimension average was 4.01; this average falls within the mid-range of codes (see 

Graph 1). This mid-range average shows that the teacher/WINGSLeader was sometimes aware of 

student needs and was both responsive and unresponsive to students throughout an observation. An 

average in the high-range would indicate teachers/WINGSLeaders who were consistently aware of 

students’ needs, who were responsive at all times to students, and consistently effective at helping 

students. Graph 4 shows that 61.3% of teacher sensitivity observations were coded in the mid-range, 

indicating the teachers/WINGS Leaders across segments were sometimes responsive and aware of 

students’ needs. The remaining frequencies show the variability in observational segments: 21.1% were 

coded in the low-range and 17.5% were coded in the high-range. 

The regard for student perspectives dimension takes into account the support and encouragement the 

teacher provides in understanding and valuing student perspectives. In addition, this dimension takes 

into consideration the extent to which students are independent in thoughts and actions. Here, the 

dimension is defined as: “[capturing] the degree to which the teacher’s interactions with students and 

classroom activities place an emphasis on students’ interests, motivations, and points of view and 

encourage student responsibility and autonomy” (2008, p. 22). Observable indicators include: flexibility 

and student focus (incorporates students’ ideas and follows students’ lead), support for autonomy and 

leadership (gives students choices, gives students responsibility), student expression (encourages 

student talk, elicits ideas and/or perspectives), and restriction of movement (allows independence of 

movement, is not rigid with student placement) (2008). 

5 

Graph 4. CLASS: Frequency of codes for the teacher sensitivity dimension.

3

26

23

26

35

15

9

0

10

20

30

40

1 2 3 4 5 6 7

No. of Observations

Codes

Frequency of Codes: 

Teacher Sensitivity

Low High 

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

The regard for student perspectives dimension average for WINGS was 2.99, in the low-range of codes 

(see Graph 1). In the low-range, teachers/WINGSLeaders are rigid with their lesson plans, often do not 

incorporate students’ ideas in the activities, fail to support student independence and leadership, and 

offer little opportunity for student talk and expression. When looking at the frequencies for the regard 

for student perspectives dimension (see graph 5), we can see that 52.5% of observations were coded in 

the mid-range, indicating that WINGSLeaders periodically supported students’ autonomy and only 

sometimes allowed for student talk and expression. The graph additionally shows that 42.3% of 

observations were coded in the low-range, showing that WINGSLeaders did not support student 

autonomy and provided few opportunities for student talk and expression. 

When compared to other studies (see table 1), WINGS dimension averages of positive climate and 

negative climate are similar (see Graph 6). The WINGS teacher sensitivity average, however, is slightly 

lower than the other studies, but still within the same mid-range. The biggest difference in averages is 

seen in the regard for student perspectives dimension; here, the 4Rs average was 1.27 points higher. 

6 

Graph 5. CLASS: Frequency of codes for the regard for student perspectives dimension.

APPENDIX D

18

40

31

26

15

7

0

0

10

20

30

40

50

1 2 3 4 5 6 7

No. of Observations

Codes

Frequency of Scores: 

Regard for Student Perspectives

Low High 

Table 1. 4Rs and Responsive Classroom study background information.

4Rs (Jones, Brown, & Aber, 2011) Responsive Classroom (Rimm-Kaufman, as 

cited in Pianta, La Paro, & Hamre, 2011)

Grade 3 (n=82) Grades 1-5 (n=88)

45.6% Hispanic/Latino

41.1% Black/African American

61.8% living at or below 100% of the poverty 

line

53.63% ethnic minorities

35.32% eligible for free or reduced lunch

New York city Urban district in the Northeast
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

OST: Youth and staff relationships 

Two components within the OST measure also provide information on the emotional climate of the 

program environment: (1) youth relationship building and (2) staff relationship building. The youth 

relationship component focuses on the students and their relationships. Observable indicators 

include: [students are] friendly and relaxed with one another, respect one another, show positive 

affect to staff, assist one another, and are collaborative (Pechman, Meilke, Russell, White, & Cooc, 

2008). Staff relationship building focuses on staff and their relationships with the students. Here, 

observable markers are: [staff] use positive behavior management, encourage participation of all, 

show positive affect toward all, attentively listen to and/or observe youth, encourage sharing of 

ideas, opinions, concerns, engage personally with youth, and guide positive peer interactions (2008; 

see appendix, table D). 

WINGS OST relationship data 

Graph 7 shows WINGS youth relationship and staff relationship averages. The WINGS youth 

relationship component average was 3.27; this falls at the low end of the mid-range of codes. A code 

of 3 indicates that the exemplar is rarely evident and a code of 5 indicates that the exemplar is 

evident or implicit. The WINGS staff-youth relationship component average was 3.46. Here again, 

this average falls in the lower end of the mid-range. 

7 

4.46

1.68

4.01

2.99

4.44

2.22

4.6

4.28

4.91

1.35

4.74

1 2 3 4 5 6 7

Positive Climate

Negative Climate

Teacher Sensitivity

Regard for Student Perspectives

Codes

CLASS Emotional Support Dimension Comparisons

Responsive Classroom 4Rs WINGS

Low High 

Graph 6. Study comparison of CLASS emotional support dimension averages.

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

8 

Graph 7. WINGS OST youth and staff-youth relationship averages.

APPENDIX D

Gr 3.27 3.46

1

2

3

4

5

6

7

Youth Relationships Staff-Youth Relationships

Codes

WINGS OST Relationship Components Averages

High Mid Low 

When comparing WINGS relationship averages to other afterschool programs (see table 2), both 

WINGS youth relationship and staff relationship averages were lower (see Graph 8). In the staff 

relationships component, all three studies had averages in the mid-range, but the WINGS average 

was slightly lower (roughly 1 point) than the comparison studies. The youth relationships component 

comparison shows a much greater difference; the WINGS average was at the lower end of the midrange, while the New Jersey After 3 average was well within the high-range of codes, indicating 

observable evidence of consistent positive youth relationships. 

Table 2. New Jersey After 3 and TASC study background information.

New Jersey After 3 (Walking Eagle, et al., 2009) The After School Corporation (TASC) 

Follow-Up Study (Birmingham, et al., 2005)

Grades K-8 (n=78 afterschool classes at 10 

programs)

Grades 4-8 (n=31 activities at 10 afterschool 

programs)

87% African American or Hispanic

57% eligible for free or reduced lunch

No demographic data available

New Jersey New York City
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Graph 8. Study comparison of OST relationship averages.

9 

APPENDIX D

4.5

5.86

4.58

3.46

3.27

1 2 3 4 5 6 7

Staff-Youth Relationships

Youth Relationships

Codes

OST Relationship Components Comparison

WINGS TASC NJ After 3

+ 

Low High 

*Youth relationships average included the following indicators: (1) are friendly and relaxed with one another, (2) respect one 

another, and (3) show positive affect with staff. 

+Staff-youth relationships average included the same indicators as WINGS observations except: (1) guide positive peer interactions. 

Perspectives of program staff and school personnel 

WINGSLeaders and Peace Managers 

In interviews, WLs and PMs mention the importance of relationships in the WINGS program. A majority 

of WLs discussed their relationships with children as the best part of their job. Here, two reiterate this: 

“[The best part is] growing bonds with the kids on an individual basis. The kids are a lot of fun.”

“[The best part is] the kids coming in and swarming you with love.”

However, the inconsistency of emotional support within the program is highlighted as a PM provides 

that when WLs are “too hard on the kids,” children’s support to learn and grow is impaired. Here, 

he/she provides: 

 

“We have WLs even here that are too hard on the kids. When that happens the kids aren't 

getting what they need so there is no change. They are not really learning.”

Teachers 

From surveys, teachers’ perceptions regarding the program atmosphere and staff were positive. When 

given a list of adjectives to describe the atmosphere, most teachers felt it was (1) loud (59%), but also 

(2) engaging (52%), and (3) positive (52%). When asked about their perceptions of the program staff, 

most teachers felt they were (1) caring (55%), (2) friendly (46%), and (3) well-intentioned (41%). 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Instructional Strategies

Instructional practices that afford students the opportunity to learn content in usable and meaningful 

ways has been shown to affect cognitive development (National Research Council, 1999). This is 

observed when teachers interact with students in a way that delivers instruction, yet places priority on 

students’ comprehensive understanding, instead of on rote memorization of isolated facts. Interactions 

in the learning environment act as a mechanism to forward students’ thinking; through classroom 

discussions, students’ explanations of their thinking, and back-and-forth exchanges. 

CLASS domain: Instructional support 

The CLASS domain that targets these aspects of the learning environment is termed Instructional 

Support. This domain is divided into three dimensions: (1) concept development, (2) quality of feedback, 

and (3) language modeling (see appendix, table B). 

Concept development focuses on the ways the teacher allows his/her students to interact with the 

content of the lesson/activity in meaningful ways. The authors define this dimension as, “[measuring] 

the teacher’s use of instructional discussions and activities to promote students’ higher-order thinking 

skills and cognition and the teacher’s focus on understanding rather than on rote instruction” (Pianta, La 

Paro, & Hamre, 2008, p. 64). Within this dimension, there are four observable indicators that guide 

observations: analysis and reasoning (the teacher asks why and how questions, students take part in 

problem solving, prediction/experimentation), creating (students brainstorm, plan, and produce their 

own ideas and products), integration (the teacher helps connect concepts and integrates the new 

content with previous knowledge), and connections to the real world (new concepts are related to 

students’ lives).

Graph 9 shows the WINGS concept development average was 1.46, in the low-range of codes. This 

average indicates that the teacher/WINGSLeader rarely encouraged analysis and reasoning in 

discussions, rarely provided opportunities for the students to create their own products or ideas, 

concepts were not tied to previous knowledge, and concepts were not related to students’ lives. The 

frequencies for this dimension (see graph 10) show that 92.7% of the observations were coded in the 

low-range; 62% of observations were coded as 1, the lowest possible code. 

Graph 9. WINGS CLASS dimension averages for the instructional support domain.

10

APPENDIX D

1.46 2.13 1.89

1

2

3

4

5

6

7

Concept Development Quality of Feedback Language Modeling

Codes

WINGS CLASS Instructional Support

Dimension Averages

High Mid Low 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Graph 10. CLASS: Frequency of codes for the concept development dimension.

85

42

6

4 0 0 0

0

20

40

60

80

100

1 2 3 4 5 6 7

No. of Observations

Codes

CLASS Frequency of Codes: Concept Development

Low High 

The quality of feedback dimension targets the way the teacher facilitates learning through the feedback 

he/she gives. This dimensions is defined as, “[assessing] the degree to which the teacher provides 

feedback that expands learning and understanding and encourages continued participation” (2008, p. 

72). This dimension includes the following observable indicators: scaffolding (the teacher’s use of hints 

and assistance to develop student thought), feedback loops (follow-up questions to expand student 

thinking), prompting thought processes (students explain their thinking), providing information 

(teachers expand on student thinking with additional information), and encouragement and affirmation 

(the teacher encourages and supports student persistence). 

The quality of feedback dimension average was 2.13, also within the low-range of codes (see Graph 9). 

This indicates that the teacher/WINGSLeader rarely provided scaffolding to students, often times did not 

provide meaningful feedback to students (only perfunctory), rarely asked students to explain their 

thinking, rarely offered additional information to students to expand understanding, and rarely offered 

encouragement of students’ efforts. Frequencies show that 65.6% of observations were coded in the 

low-range, with 33.5% coded in the mid-range (see graph 11). This indicates that while most of the 

observations showed low-levels of quality feedback, there were some instances of mid-level quality 

feedback present. 

Graph 11. CLASS: Frequency of codes for the quality of feedback dimension.

11

APPENDIX D

46

44

32

12

2 1

0

0

10

20

30

40

50

1 2 3 4 5 6 7

No. of Observations

Codes

CLASS Frequency of Codes: Quality of Feedback

Low High 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Finally, the language modeling dimension focuses on how well the teacher and the learning 

environment supports students’ language development. Here, the dimension is defined as, “[capturing] 

the quality and amount of the teacher’s use of language-stimulation and language-facilitation 

techniques” (2008, p. 79). This dimension is assessed through the following indicators: frequent 

conversation (prevalence of teacher-student and student-student discussions), open-ended questions 

(questions illicit more than one-word responses), repetition and extensions (the teacher repeats and 

extends student responses), self- and parallel talk (the teacher describes his/her actions or student 

actions), and advanced language (a variety of words are used in the learning environment). 

The language modeling average of 1.89 is also in the low-range of codes (see graph 9). In this range, 

there are few or no conversations in the learning environment, the majority of the 

teacher’s/WINGSLeader’s questions are close-ended, the teacher/WINGSLeader rarely incorporates 

techniques to develop students’ language skills (repetition and extension, self- and parallel talk), and 

does not use advanced language with students. The frequencies provide that 76.6% of observations 

were coded in the low-range, with 23.3% of observations coded in the mid-range (see graph 12). These 

frequencies illustrate that while most of the observations had poor language stimulation, some 

observations had occasional instances of mid-level language modeling. 

12

Graph 12. CLASS: Frequency of codes for the language modeling dimension.

APPENDIX D

51 54

25

6

1 0 0

0

10

20

30

40

50

60

1 2 3 4 5 6 7

No. of Observations

Codes

CLASS Frequency of Codes: Language Modeling

Low High 

When comparing WINGS instructional support averages with others (4Rs and Responsive Classroom; 

table 1), we see that while WINGS averages are in the low-range for all dimensions, both the Responsive 

Classroom and 4Rs studies have quality of feedback and concept development averages in the midrange; with the Responsive Classroom quality of feedback dimension average at the high-end of the 

mid-range (see Graph 13). It should be noted that at the time of the 4Rs and Responsive Classroom 

studies, the CLASS measure did not contain a language modeling dimension; hence, there was no data 

collected for that specific dimension. 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Graph 13. Study comparison of CLASS instructional support dimension averages.

APPENDIX D

3.84

3.54

3.82

4.77

1.46

2.13

1.89

1 2 3 4 5 6 7

CD

QoF

Language Modeling

Codes

CLASS Instructional Support Dimension Comparisons

WINGS Responsive Classroom 4Rs

Low High 

OST: Staff instructional strategies 

The OST component that targets similar indicators to the CLASS Instructional Support dimensions is 

called instructional strategies. Here, the staff are assessed on the presence and quality of how they: 

communicate goals, the purpose, and expectations, verbally recognize efforts and accomplishments, 

assist youth without taking control, ask youth to expand upon their answers and ideas, challenge youth 

to move beyond current levels, employ a variety of teaching strategies, and plan for/ask youth to work 

together (Pechman, Mielke, Russell, Whiite, & Cooc; see appendix, table E).

Graph 14 shows the WINGS OST overall average of instructional strategies was 2.57, in the low-range of 

codes. As mentioned previously a code of 3 indicates the exemplar is rarely evident and a code of 1 

indicates the exemplar is not evident. When looking at the OST instructional strategies average per 

activity (see graph 15), we see that the lowest instructional strategies codes were observed in the 

Discussion activity, a component when WINGS deliberately delivers its social-emotional content. 

Graph 14. OST: WINGS instructional strategies average. 

2.57

1

2

3

4

5

6

7

Instructional Strategies

Codes

WINGS OST: Instructional Strategies 

Overall Average

High Mid Low 

2.50 2.56 2.18

2.81

1

2

3

4

5

6

7

Comm. Unity Academic

Center

Discussion Choice Time

Codes

WINGS Component

WINGS OST: Instructional Strategies 

Averages by Activity

Graph 15. OST: WINGS instructional strategies averages by activity. 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

When comparing the WINGS OST instructional strategies dimension average to two other studies (TASC 

& New Jersey After 3; table 2), we see that while both comparison studies had averages in the midrange of codes (3.6 & 3.77), the averages were within the lower-end of this mid-range (see Graph 16). As 

stated previously, the WINGS average indicates that the behavioral markers (indicators) of instructional

strategies (i.e. staff assist youth without taking control, ask youth to expand on their ideas, verbally 

recognize youth efforts) were rarely evident; a mid-range average, however, indicates the exemplars are 

more consistently evident. 

14

Graph 16. Comparison of OST instructional strategies dimension averages. 

APPENDIX D

3.77

3.6

2.57

1 2 3 4 5 6 7

TASC

NJ After 3

WINGS

Codes

OST Instructional Strategies Dimension Comparison

Low High 

Hunter-Bailin: Teachable moments 

An additional mechanism that WINGS uses to deliver content, teachable moments, is assessed on the 

Hunter-Bailin measure. Teachable moments are thought of as an informal teaching method that 

provides the SEL curriculum to students throughout the day; for instance, a WL discussing delayed 

gratification with his/her Nest as the students complete homework before an ice cream party. 

Teachable moments will either include references to a specific SEL objective (for instance, “Give me your 

3Es-eyes, ears, and energy”) or the WINGS creed (“I listen to you, and you listen to me”). 

Graph 17 shows that the majority of teachable moments present within observations were of low to 

moderate quality. There were many instances of low quality teachable moments involving both SEL 

objectives (42.7%) and the creed (43.3%). This indicates that WLs either did not use teachable moments 

or used them in a counterproductive way (for example yelling “I listen to you, you listen to me” while 

disregarding the student’s needs).
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

15

APPENDIX D

Graph 17. Hunter-Bailin: Frequency and quality of teachable moments.

43.3%

31.1%

18.3%

7.3%

42.7%

25.0%

18.3%

14.0%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Hunter-Bailin: Teachable Moments

Creed

SEL

1-Low Quality 2-Moderate Quality 3-Moderate-High 4-High Quality

Perceptions of program staff 

WINGSLeaders and Program Directors 

When interviewed, WLs felt they could accomplish the goals of the activities during the program, but 

they also mentioned feelings of frustration when planning and facilitating a discussion or brainstorming 

ideas with children. One PD speaks to this and also discussed the value in offering support to WLs when 

she said:

“Discussion is very hard...WINGS just gives [the WLs] text, so it’s up to the WLs to make it fun and 

engaging. Sometimes with the Discussion, the WLs don’t understand the objective. The PDs who 

go over it on Monday and what the Discussion will look like for the week definitely have more 

successful WLs, because they can deliver it more confidently.”

Learning Environment Organization 

Research has provided that learning environments are most effective when they are well-managed; this 

includes the behavior of the students, the organization of activities and routines, and the use of 

interesting materials and activities coupled with teacher facilitation (Emmer & Stough, 2001). 

Specifically, learning environments that incorporate positive behavior management practices and are 

productive are inclined to have the most engaged students (Emmer & Stough, 2001; Evertson, Emmer, 

Sanford, & Clements, 1983; Evertson & Harris, 1999; Brophy & Evertson, 1976; Coker, Medley, & Soar, 

1980; Good & Grouws, 1977; Stallings, 1975; Stallings, Cory, Fairweather, & Needels, 1978). Research 

also provides that in order for learning to occur, students must be interested in the activities, supporting 

the need for engaging activities and materials for active student participation (Yair, 2000). However, 

poorly managed routines, including transitions, often result in student misbehavior and ultimately take 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

away instructional time needed for engaged learning (Gettinger & Seibert, 2002; Sprick, Garrison, & 

Howard, 1998). 

CLASS domain: Classroom organization 

The CLASS domain that focuses on these aspects is called Classroom Organization. Within this domain, 

there are three dimensions that provide observational direction: (1) behavior management, (2) 

productivity, and (3) instructional learning formats (see appendix, table C). 

CLASS: Behavior Management 

Behavior management captures the teacher/WINGSLeader’s ability to set behavior expectations and 

deal with misbehavior; the authors of the CLASS define this dimension as “[encompassing] the teacher’s 

ability to provide clear behavioral expectations and use effective methods to prevent and redirect 

misbehavior” (Pianta, La Paro, & Hamre, 2008, p.44). Observable indicators include: clear behavior 

expectations, proactive (low escalation and reactivity), redirection of misbehavior, and student behavior 

(compliance with rules and the presence of defiance) (2008). 

Graph 18 shows the CLASS behavior management dimension average falls within the mid-range of 

codes. This mid-range average indicates that during observations, rules and expectations were unclear 

at times, the teacher/WINGSLeader used a mix of proactive and reactive behavior management 

techniques, the teacher/WINGSLeader was somewhat effective at redirecting behavior, and there were 

occasional episodes of misbehavior in the learning environment. 

Graph 18. WINGS CLASS dimension averages for the classroom organization domain. 

16

APPENDIX D

4.16 4.21

3.5

1

2

3

4

5

6

7

Behavior Management Productivity Inst. Learning Formats

Codes

WINGS CLASS 

Classroom Organization Dimension Averages

Mid Low High 

When looking at the code frequencies for the CLASS behavior management dimension (see graph 19), 

we see that 68.6% of codes were within the mid-range, followed by 21.8% of codes in the high-range. 

These frequencies indicate that high-quality behavior management did occur in the WINGS program 

within 21.8% of the observations; where rules and expectations were clear and consistently enforced, 

the teacher/WINGSLeader used proactive techniques throughout the observation, the 

teacher/WINGSLeader was effective at redirecting misbehavior, and there was little student 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

misbehavior in the learning environment. However, this high-level of quality was not observed for the 

majority of observations. 

17

Graph 19. CLASS: Frequency of codes for the behavior management dimension. 

APPENDIX D

2

11

39

24

31 26

4

0

10

20

30

40

50

1 2 3 4 5 6 7

No. of Observations

Codes

CLASS Frequency of Codes: Behavior Management

Low High 

Hunter-Bailin: Behavior management techniques 

The Hunter-Bailin captures the quality of behavior techniques specific to the WINGS program in regards 

to frequency and effectiveness of implementation. These are proactive (MESS), corrective (GET PAST), 

and instructional (DDADA) techniques WINGS trains and encourages WINGSLeaders to employ in order 

to shape the behavior of their students. 

The Hunter-Bailin data reveals that the quality of both proactive and corrective techniques was wideranging (see graph 20). The majority of observations for both proactive (MESS; 59.1%) and corrective 

(GET PAST; 59.8%) techniques are scored as moderate and moderate-high. More instances of highquality implementation are observed for proactive (MESS; 24.4%) in comparison to corrective (GET 

PAST; 8.5%) techniques. In contrast, the overwhelming majority of time instructional techniques 

(DDADA) are scored at a low level of quality, indicating that the technique was never used or was used 

in a counterproductive way. The range in scores indicates that students’ exposure to the program’s

trademark techniques varied greatly in regards to frequency and effectiveness. 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

18

Graph 20. Hunter-Bailin: Frequency and quality of behavioral techniques.

APPENDIX D

15.9%

31.7% 27.4% 24.4% 26.2%

35.4%

24.4%

8.5%

73.8%

12.8%

4.3%

5.5%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Hunter-Bailin: Behavioral Techniques

MESS

GET PAST

DDADA

1-Low Quality 2-Moderate Quality 3-Moderate-High 4-High Quality

Perceptions among staff and school partners 

WINGSLeaders 

Some WLs discussed behavior management as their biggest strength. One WL provides: 

“I’m good at proactive prevention of potential incidents.”

However, other WLs explicitly mentioned student behavior or behavior management techniques as a 

challenge. One WL adds: 

“Most difficult days, a lot of things are going wrong at once, I get a little bit 

overwhelmed. I can deal with one kid misbehaving, but when the majority are 

misbehaving, I usually have to call someone and just be like, ‘I need a reset.’ Once one is 

quiet, another one is talking; managing the big group is really my biggest difficulty.”

Still others discussed the difficulty in staying positive and managing their own behavior when faced with 

negative and difficult behavior from students. Two WLs provided: 

“You can be happy all day, then kids do things that are frustrating, then you become 

upset and want to quit sometimes.”

“Lately I’ve noticed that I’m very quick to lose my cool.”

School Principals 

Principals are uneasy that school rules and expectations for the children during the school day are not 

upheld afterschool. To this end, two principals described it as ‘loud.’ Here, they added: 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

“... it’s just always loud. And the one thing that I said is if the kids are loud, then the 

WINGSLeader can’t be louder, cause now you’re all loud, and [it looks like] you can’t 

manage discipline. And that’s the part that I’m not in love with with WINGS, to be honest 

with you. I’m just not in love with it.”

“The biggest concern is that when the bell rings at [X], the kids ... are running and 

shouting and yelling and so one of the concerns was, we need them to have a cleaner, 

quieter dismissal cause that’s not what the expectation is here. No one runs here, yells 

here.” 

CLASS: Productivity 

Productivity encompasses the teacher’s/WINGSLeader’s ability to manage the flow of the learning 

environment; its activities and routines. This dimension is defined as, “[considering] how well the 

teacher manages instructional time and routines and provides activities for students so that they have 

the opportunity to be involved in learning activities” (2008, p. 44). Observable indicators include: 

maximizing learning time (there is something for the students to do, steady pacing), routines (students 

have a clear understanding of what to do), transitions (are brief and efficient), and preparation (the 

teacher is prepared and knows the lessons) (2008). 

The WINGS CLASS data in graph 18 also shows the productivity average (4.21) was in the mid-range. 

Mid-range codes indicate that students had an activity to take part in most of the time, there was some

evidence of an understanding of classroom routines, the teacher/WINGSLeader was mostly prepared to 

conduct the activity/lesson, and transitions were sometimes too frequent and too inefficient. The CLASS 

frequencies for the productivity dimension (see graph 21) show that 58% of observations were coded in 

the mid-range, with 25.5% in the high-range, and 16% in the low-range. This, again, shows that there 

were instances of high productivity within observations of WINGS settings, but the majority of 

observations showed mid-level productivity. 

Graph 21. CLASS: Frequency of codes for the productivity dimension.

19

APPENDIX D

4

18

24 24

32 29

6

0

10

20

30

40

1 2 3 4 5 6 7

No. of Observations

Codes

CLASS Frequency of Codes: Productivity

Low High 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Time Sampling 

By looking at the time sampling data (see table 3), we see that students within Nests at the three study 

schools spent between 20-46% of their time transitioning in and out of activities. Table 4 is an example 

of an observation detailing the time spent in transition for the third grade girls’ Nest in WINGS. This 

shows the Nest was in transition a total of 67 minutes during the observation period, accounting for 

46.5% of the time (out of 144 minutes). 

Table 3. Percentages of time spent in transition.

Grade Gender Time in 

Transition

1 M 19.61%

1 F 21.57%

3 M 30.77%

3 F 46.53%

5 M 35.86%

5 F 31.16%

Table 4. Example time sampling observation for the third grade girls’ Nest.

Amount of Time Activity 

1 Countdown

1 From watching demo to starting SEL

1 From starting SEL game to snack

5 Cleaning up snack, countdown, starting creed

6 Waiting to be dismissed from community unity to academic center

10 Lining up in community unity, walking in the hall to academic center

5 Coming into academic center room, finding a seat, waiting for pencils, 

worksheets passed out

4 Cleaning up and packing after academic center

13 Lining up in academic center, walking to cafeteria, waiting to be dismissed to 

choice time

9 Lining up, getting hula hoops for choice time, walking down hallway to choice 

time

2 Getting kids to find their line-up position for rehearsing their hoop routine

2 Getting kids to listen

8 Lining up, walking down the hallway from choice time to dinner

67 minutes Total Time in Transition 

CLASS: Instructional Learning Formats 

Instructional learning formats brings into consideration the teacher’s/WINGSLeader’s ability to facilitate 

lessons and activities in a way that engages students. This dimension is defined as, “[focusing] on the 

ways in which the teacher maximizes students’ interest, engagement, and ability to learn from lessons 

and activities” (2008, p. 57). Observable indicators of this dimension include: effective facilitation (the 

teacher is involved in the activity and uses questions to engage), variety of modalities and materials 

(considers the types of activities and materials used in engaging students), student interest (how 

20

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

focused and active the students are), and clarity of learning objectives (considers the teacher’s attempts 

at focusing the students towards the objective) (2008). 

Graph 18 shows the instructional learning format average was 3.5. While this average is within the midrange of codes, it is on the lower-end of the mid-range. A mid-range average indicates the 

teacher/WINGSLeader was sometimes active in facilitating student engagement within the activity, the 

students were sometimes engaged with the lesson/activity, and the teacher/WINGSLeader sometimes

oriented students to the learning objectives. Frequencies support the prevalence of mid-range codes 

across observations; 63.5% of codes are within the mid-range, while 27.7% are in the low-range (see 

graph 22). A code in the low-range indicates no teacher/WINGSLeader facilitation towards sparking 

student interest, rare instances of student engagement, and the teacher/WINGSLeader is unsuccessful 

or makes no attempt at orienting the students towards the lesson objective during WINGS observations. 

Graph 22. CLASS: Frequency of codes for the instructional learning formats dimension.

13

25

33

27 27

11

1

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7

No. of Observations

Codes

CLASS Frequency of Codes: 

Instructional Learning Formats

Low High 

When compared to two other studies (Responsive Classroom and 4Rs; table 1), the mid-range WINGS 

CLASS averages were similar to the other studies, although WINGS averages were slightly lower (see 

graph 23). This graph also shows that the Responsive Classroom average of behavior management was 

within the high-range of codes. 

Graph 23. Study comparison of classroom organization dimension averages. 

21

APPENDIX D

4.16

4.21

3.5

4.98

4.69

4.21

5.14

4.98

4.23

1 2 3 4 5 6 7

Behavior Management

Productivity

Instructional Learning Formats

Codes

CLASS Classroom Organization Dimension Comparisons

Responsive Classroom 4Rs WINGS

High Low 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

OST: Youth participation 

The youth participation component of the OST measure allows the observer to focus on the students 

and the extent of their participation in the activities. This component addresses the following 

observable indicators: [students are] on task, listen actively and attentively to peers and staff, contribute 

opinions, ideas, and/or concerns to discussions, have opportunities to make meaningful choices, and 

take leadership responsibility/roles (Pechman, Mielke, Russell, Whiite, & Cooc; see appendix, table F).

Graph 24 shows the WINGS youth participation average. The WINGS overall average of 2.76 falls within 

the low-range of codes, indicating the observable indicators were not evident or were rarely evident. 

When we look at the youth participation within each activity (see graph 25), averages show that most 

student participation occurred during Community Unity, with an average code of 3 (at the base of the 

mid-level of codes). However, for all other activities observed, youth participation was in the low-range. 

Low High 

Graph 24. WINGS OST youth participation average.

2.76

1

2

3

4

5

6

7

Youth Participation

Codes

WINGS OST: Youth Participation

High Mid Low 

Graph 25. WINGS OST youth participation averages by activity.

3.00

2.41 2.40

2.89

1

2

3

4

5

6

7

Comm. Unity Academic Center Discussion Choice Time

Codes

WINGS Component

WINGS OST: Youth Participation

When comparing the WINGS youth participation average to another study (TASC, table 2), we see that 

the TASC study had an average of 4.49; at the higher-end of the mid-range (see Graph 26). The TASC 

average, however, combines the two dimensions of youth participation and youth relationships to result 

in this composite score. 

Graph 26. Study comparison of OST youth participation averages.

22

APPENDIX D

4.49

2.76

1 2 3 4 5 6 7

TASC

WINGS

Codes

WINGS OST Youth Participation Comparison

*TASC youth participation average included youth relationship component indicators as well. 
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Program Staff Perception 

WINGSLeaders 

During interviews, some WLs discussed the difficulty in preparing and facilitating activities to engage 

students. Here, two WLs illustrate this: 

“I can interact with a couple of kids, but managing a group of twelve or ten is my, kind of...that’s 

what I’ve had to struggle with. And I think I’ve gotten better, you know. But definitely it’s like 

very teacher-oriented. Keeping kids engaged, you know.”

“And especially with things that aren’t fun, like WINGS Works: it’s difficult, it feels like we’re 

forcing it. None of the WINGSLeaders really like it, it’s hard. Every Wednesday is the most 

difficult day.”

23

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

 References 

Birmingham, J., Pechman, E. M., Russell, C. A., & Mielke, M. (2005). Shared features of high performing 

 after-school programs: A follow-up to the TASC evaluation. Washington, DC: Policy Studies 

 Associates. 

Brophy, J., & Evertson, C. (1976). Learning from teaching: A developmental perspective. Boston: Allyn 

 and Bacon. 

Center for Advanced Study of Teaching and Learning. (n.d.). Measuring and improving teacher student 

 interactions in pk-12 settings to enhance students’ learning”(brief). 

http://curry.virginia.edu/uploads/resourceLibrary/CLASS-MTP_PK-12_brief.pdf

Coker, H., Medley, D.M., & Soar, R.S. (1980). How valid are expert opinions about effective teaching? Phi

Delta Kappan, 62 (2), pp. 131-134, 149. 

Emmer, E.T., & Stough, L. (2001). Classroom management: A critical part of educational psychology, with 

 implications for teacher education. Educational Psychologist, 36 (2), pp. 103-112. 

Evertson, C., Emmer, E., Sanford, J., & Clements, B. (1983). Improving classroom management: An 

 experiment in elementary classrooms. Elementary School Journal, 84, pp. 173-188. 

Evertson, C. & Harris, A. (1999). Support for managing learning-centered classrooms: The classroom 

 organization and management program. In H.J. Freiberg (Ed.), Beyond behaviorism: Changing

the classroom management paradigm, pp. 59-74. Boston: Allyn & Bacon. 

Gettinger, M. & Seibert, J.K. (2002). Best practices in increasing academic learning time. In A. Thomas 

 (Ed.), Best practices in school psychology IV, pp. 773-787. Bethesda, MD: National Association of 

 School Psychologists. 

Good, T., & Grouws, D. (1977). Teaching effects: A process-product study of fourth grade mathematics 

 classrooms. Journal of Teacher Education, 28, pp. 49-54. 

Hamre, B.K. & Pianta, R.C. (2007). Can instructional and emotional support in the first grade classroom 

 make a difference for children at risk of school failure? Child Development, 76(5), pp. 949-967. 

Jones, S. M., Brown, J.L., & Aber, J.L. (2011). Two-year impacts of a universal school-based social-

 emotional and literacy intervention: An experiment in translational developmental research. 

Child Development, 82 (2), pp. 533-554. 

Ladd, G.W., Birch, S.H., & Buhs, E.S. (1999). Children’s social and scholastic lives in kindergarten: Related 

 spheres of influence? Child Development, 70 (6), pp. 1373-1400. 

National Research Council. (1999). How people learn: Brian, mind, experience, and school. Washington, 

 DC: National Academies Press. 

Pechman, E.M., Mielke, M. B., Russell, C.A., White, R.N., & Cooc, N. (2008). Out-of-school time (OST) 

 observation instrument: Report of the validation study. Policy Studies Associates, Inc. 

24

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Pianta, R.C., La Paro, K.M., & Hamre, B.K. (2008). Classroom Assessment Scoring System (CLASS) Manual, 

K-3. Baltimore, Maryland: Brookes Publishing. 

Sprick, R., Garrison, M., & Howard, L. (1998). CHAMPS: A proactive and positive approach to classroom 

 management. Longmont, CO: Sopris West. 

Stallings, J. (1975). Implementation and child effects of teaching practices in follow through classrooms. 

Monographs of the Society for Research in Child Development, 40 (7-8, Serial No. 163). 

Stallings, J., Cory, R., Fairweather, J., & Needels, M. (1978). Early childhood education classroom

evaluation. Menlo Park, CA: SRI International. 

Walking Eagle, K.P., Miller, T.D., Cooc, N., LaFleur, J., & Reisner, E.R. (2009). Evaluation of New Jersey 

after 3: Reaching and engaging New Jersey’s youth through after-school programs, 2005-2008. 

 New Brunswick, NJ: Policy Studies Associates, Inc. 

25

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Table A. CLASS emotional support domain, dimensions, indicators, and behavioral markers.

Dimension Observable Indicators Behavioral Markers

Positive Climate

Relationships Physical proximity

Shared activities

Peer assistance

Matched affect

Social conversation

Positive affect Smiling

Laughter

Enthusiasm

Positive communication Verbal affection

Physical affection

Positive expectations

Respect Eye contact

Warm, calm voice

Respectful language

Cooperation and/or sharing

Negative Climate

Negative affect Irritability

Anger

Harsh voice

Peer aggression

Disconnected or escalating negativity

Punitive control Yelling

Threats

Physical control

Harsh punishment

Sarcasm/disrespect Sarcastic voice/statement

Teasing

Humiliation

Severe negativity Victimization

Bullying

Physical punishment

Teacher Sensitivity

Awareness Anticipates problems and plans appropriately

Notices lack of understanding and/or difficulties

Responsiveness Acknowledges emotions

Provides comfort and assistance

Provides individualized support

Addresses problems Helps in an effective and timely manner

Helps resolve problems

Student comfort Seeks support and guidance

Freely participates

Takes risks

Regard for Student 

Perspectives

Flexibility and student focus Shows flexibility

Incorporates students’ ideas

Follows students’ lead

Support for autonomy and 

leadership

Allows choice

Allows students to lead lessons

Gives students responsibility

Student expression Encourages student talk

Elicits ideas and/or perspectives

Restriction of movement Allows movement

Is not rigid

26

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Table B. CLASS instructional support domain, dimensions, indicators, and behavioral markers.

Dimension Observable Indicators Behavioral Markers

Concept Development

Analysis and reasoning Why and/or how questions

Problem solving

Prediction/experimentation

Classification/comparison

Evaluation

Creating Brainstorming

Planning

Producing

Integration Connects concepts

Integrates with previous knowledge

Connections to the real world Real-world applications

Related to students’ lives

Quality of Feedback

Scaffolding Hints

Assistance

Feedback loops Back and forth exchanges

Persistence by teacher

Follow-up questions

Prompting thought processes Asks students to explain thinking

Queries responses and actions

Providing information Expansion

Clarification

Specific feedback

Encouragement and affirmation Recognition

Reinforcement

Student persistence

Language Modeling

Frequent conversation Back and forth exchanges

Contingent responding

Peer conversations

Open-ended questions Questions require more than a one-word response

Students respond

Repetition and extension Repeats

Extends/elaborates

Self- and parallel talk Maps own actions with language

Maps student action with language

Advanced language Variety of words

Connected to familiar words and/or ideas

27

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Table C. CLASS program organization domain, dimensions, indicators, and behavioral markers.

Dimension Observable Indicators Behavioral Markers

Behavior Management

Clear behavior expectations Clear expectations

Consistency

Clarity of rules

Proactive Anticipates problem behavior or escalation

Low reactivity

Monitors

Redirection of misbehavior Effective reduction of misbehavior

Attention to the positive

Uses subtle cues to redirect

Efficient redirection

Student behavior Frequent compliance

Little aggression and defiance

Productivity

Maximizing learning time Provision of activities

Choice when finished

Few disruptions

Effective completion of managerial tasks

Pacing

Routines Students know what to do

Clear instructions

Little wandering

Transitions Brief

Explicit follow-through

Learning opportunities within

Preparation Materials ready and accessible

Knows lessons

Instructional Learning 

Formats

Effective facilitations Teacher involvement

Effective questioning

Expanding children’s involvement

Variety of modalities and materials Range of auditory, visual, and movement opportunities

Interesting and creative materials

Hands-on opportunities

Student interest Active participation

Listening

Focused attention

Clarity of learning objectives Advanced organizers

Summaries

Reorientation statements

28

APPENDIX D
WINGS for KIDS | IMPLEMENTATION STUDY REPORT

Table D. OST relationship components and indicators.

Youth Relationship 

Building 

Youth are friendly and relaxed with one another

Youth respect each other

Youth show positive affect with staff

Youth assist one another

Youth are collaborative

Staff Relationship 

Building

Staff use positive behavior management

Staff encourage participation of all

Staff show positive affect toward all

Staff attentively listen to and/or observe youth

Staff encourage sharing of ideas, opinions, concerns

Staff engage personally with youth

Staff guide positive peer interactions

Table E. OST instructional strategies component and indicators.

Instructional Strategies Staff communicate goals, purpose, expectations

Staff verbally recognize efforts and accomplishments

Staff assist youth without taking control

Staff ask youth to expand upon their answers and ideas

Staff challenge youth to move beyond current levels

Staff employ a variety of teaching strategies

Staff plan for/ask youth to work together

Table F. OST youth participation component and indicators.

Youth Participation Youth are on task

Youth listen actively and attentively to peers and staff

Youth contribute opinions, ideas, and/or concerns to discussions

Youth have opportunities to make meaningful choices

Youth take leadership responsibility/roles

29

APPENDIX D
APPENDIX	E-	Characteristics	of	Outcome	Measures	

Table	E.1	Relationships	between	Measures	and	Outcomes	

ASSESSMENT TOOLS SHORT-TERM	OUTCOMES INTERMEDIATE	OUTCOMES LONG-TERM	OUTCOMES

FIVE	SEL	COMPETENCIES RELATIONSHIPS	&	BEHAVIOR ACADEMIC	ACHIEVEMENT

SelfawarenessSocial	

awareness

Responsible

decisionmaking

SelfmanagementRelationship

skills

Relationships Problem	

Behaviors

Language/

Literacy

Math Academic	

KnowledgeEF,

Other

Emotion Knowledge Task (EMT) X X

Assessment of Emotion	Skills (ACES) X X

Delay of gratification task X X

Head-Toes-Knees-Shoulders (HTKS) X X

NEPSY II theory of mind X

DAS working memory X

DAS naming vocabulary X

DAS verbal comprehension X

WJ III	Letter-word ID X

WJ III Applied problems X

WJ III Academic knowledge X

TEACHER-REPORT

Student-Teacher Relationship Scale X

Social Skills Improvement System X X X X X X

Devereaux Student Strengths Assess X X X X X

Child	Behavior Rating Scale X X X

PARENT-REP

Child	Behavior Rating Scale X X X

Child-Parent Relationship Scale X

Social Skills Improvement System X X X X X X

Devereaux Student Strengths Assess X X X X X
Appendix	E.2-	Description	of	Measures	and	Psychometrics	

Short-term Outcomes:	SEL Competencies

Measure	Name	and	Author Measure	Description	and	

Age	Range

Sub-Scales Psychometric	Properties Data	Collection	Method	

and	Timeline

Wally	Child	Social	ProblemSolving	Detective	Game	

(WALLY,	Webster-Stratton	&	

Hammond,	1997);	developed	

from	Spivak	and	Shure’s	

(1985)	Preschool	ProblemSolving	Test	and	Rubin	&	

Krasnor’s	Social	Problems	

Solving	Test	(1986)

The	WALLY	game	evaluates	

dimensions	of	Social	

Problem	Solving. Children	

are	shown	12	brightly	

colored	pictures	of	

hypothetical	problem	

situations	(e.g.,	about	

teasing,	conflict)	and	asked	

to	be	“a	problem-solving	

detective”	to	solve	them.	

Appropriate	through	the	

early	years	of	school

Two	summary	scores:	1)	Number	

of	positive	strategies	proposed	

and	2)	ratio	of	different	positive	

strategies	to	total	negative	

strategies.	

Acceptable	internal	

consistency	(.50-.64).	Interrater	reliabilities	ICC	above	

.95.	Construct	validity	with	

the	Rubin	positive/negative	

strategies	(r=.60	and	.50,	

respectively;	WebsterStratton	&	Hammond,	1997).	

Direct	assessment	at	

baseline	and	post-test	each	

year	of	participation	during	

the	summer	months.

Emotion	Matching	Task	

(EMT;	Morgan,	Izard,	&	King,	

2009)

The	Emotion	Matching	Task

(EMT:	Izard	et	al.,	2003)	

was	designed	for	preschool	

age	children,	including	

those	who	are	

economically	

disadvantaged.	The	EMT	

features	brightly	colored	

photographs	of	ethnically	

diverse	children	making	

facial	expressions	of	

happiness,	sadness,	anger,	

fear/surprise,	and	“neutral”

(no	visible	facial	muscle	

movement	that	signals	

The	EMT	consists	of	four	parts	

which	measure	the	components	

of	receptive	emotion	knowledge,

expressive	emotion	knowledge,	

emotion	situation	knowledge,	

and	emotion	expression	

matching.

Regression	analyses	revealed	

moderate	to	strong	

predictive	validity	for	EMT.	

Compared	to	KEI	and	AKT,	

the	EMT	was	a	more	robust	

predictor	of	teacher	rated	

emotion	regulation	and	

parent	reported	effortful	

control.	Compared	to	KEI	and

AKT,	the	EMT	correlated	

similarly	with	verbal	ability	

and	age	(Morgan,	Izard,	&	

King,	2009)

Direct	assessment	at	

baseline	and	post-test	each	

year	of	participation	during	

the	summer	months.
emotion).

Assessment	of	Children’s	

Knowledge	Task	(ACES)

Assesses	children's	emotion	

attribution	accuracy	and	

emotion	biases.

Age	range:	elementary	

school	years

Three	sections	that	cover	social	

behaviors	(presented	with	15	

brief	social	situations,	children	

are	asked	to	label	appropriate	

emotions),	social	situations	(three

vignettes),	and	facial	expressions	

(labeling	emotions	from	26	

photographs).

Reliability:	Internal	

consistency	is	adequate	

across	studies

(e.g.,	Mavroveli	et	al.,

2009;	Schultz	et	al.,	2004).	

This	scale	had	moderate	

internal	reliability	(a	=	.71).	

These	items	cohered	

moderately	well	(Cronbach’s	

alpha	=	.68).		Validity:	

Mavroeli	et	al.(2009)	found	

that	the	ACES	correlated	well

with	trait	emotional	

intelligence.

Direct	assessment	at	

baseline	and	post-test	each	

year	of	participation	during	

the	summer	months.

Head-Toes-Knees-Shoulders	

(HTKS,	Ponitz,	McClelland,	et	

al.,	2008)

Children	must	pay	

attention,	remember	the	

instructions,	and	

demonstrate	selfmanagement	in	a	

behavioral	motor	response	

by	touching	their	head	

when	asked	to	“touch	your	

toes”	and	by	touching	their	

toes	when	asked	to	“touch	

your	head.”	The	task	

increases	in	complexity	and

is	appropriate	for	pre-K	

through	early	elementary	

years.

The	HTKS	task	has	been	

conceptualized	by	Ponitz,	et	al.,	

(2008)	as	a	measure	of	inhibitory	

control	(a	child	must	inhibit	the	

dominant	response	of	imitating	

the	examiner),	working	memory	

(a	child	must	remember	the	rules	

of	the	task)	and	attention	

focusing	(must	focus	attention	to	

the	directions	being	presented	by

the	examiner).

Scores	on	the	HTKS	and	its	

two-rule	predecessor,	the	

Head-to-Toes	Task	(HTT)	

have	shown	strong	reliability	

and	construct	and	predictive	

validity	in	several	studies	

with	diverse	samples	

(McClelland,	Cameron	et	al.,	

2007;	Ponitz,	McClelland,	et	

al.,	2008;	Matthews	et	al.,	in	

press;	Ponitz	et	al.,	in	press).

Direct	assessment	at	

baseline	and	post-test	each	

year	of	participation	during	

the	summer	months.

Theory	of	Mind	sub-test	of	

the	Developmental	

NEuroPSYchological	

Assessment	(NEPSY-II;	

Korkman,	Kirk	&	Kemp,	

2007a,	2007b)

Two	tasks	designed	to	

assess	ability	to	understand	

mental	functions	and	

another’s	point	of	view.	1)	

Verbal	task	assessement		

Understanding	of	another’s	

Theory	of	Mind	construct,	or	the	

ability	to	recognize	emotions,	to	

guess	what	another	person	is	

thinking	and	feeling,	empathy.

Up-to-date	psychometric	

norms	are	based	on	the	

standardization	of	over	1,000	

U.S.	children	which	enables	

the	comparison	of	a	child's	

performance	to	others	in	the	

Direct	assessment	at	

baseline	and	post-test	each	

year	of	participation	during	

the	summer	months.
thoughts,	ideas,	feelings.	2)	

Contextual	task	assesses	

ability	to	relate	emotion	to	

social	context	Appropriate	

for	ages	3-16.

appropriate	age	group.	

Validity	Studies	were	carried	

out	with	NEPSY,	WISC-IV,	

DAS—IIII,	WNV,	WIAT—IIII,	

CMS,	DKEFS,	BBCS:3R,	DSMD,	

ABAS—II,	Brown	ADD	Scales	

and	CCC-2.

Sticker	and	candy	delay	of	

gratification	task	(Brock	

2008,	unpublished	

dissertation	thesis;	

Hongwanishkul	et	al,	2005).

Each	child	asked	to	choose	

either	a	single	desirable	

item	(sticker,	candy)	

immediately	or	a	greater	

number	(2,	4,	or	6)	of	the	

same	item	after	a	delay.	

Dichotomous	outcomes	of	single	

item	now	versus	greater	number	

later;	6	summary	scores	

calculated.	

Inter-rater	reliability	is	high	

(1.00,	Brock,	2008).	

Performance	on	a	task	

involving	prolonged	delay	has

been	found	to	be	predictive	

of	academic	performance	

into	high	school	(Mischel,	

Shoda,	and	Rodriguez,	M.L,	

1989).

Direct	assessment	at	

baseline	and	post-test	each	

year	of	participation	during	

the	summer	months.

Devereux	Student	Strengths	

Assessment (DESSA;	Lebuffe,

Shapiro,	&	Naglieri,	2008).

72	items	rated	on	a	5-point	

scale	varying	from	“never”	

to	“very	frequently.”

Appropriate	for	K-8th	

grade.

Five	constructs	include:	SelfAwareness,	Self-Management,	

Social	Awareness,	Relationship

Skills,	Responsible	decisionmaking

Reliability:	Alpha	coefficients	

for	scales	ranged	from	.82	to	

.98.

Test-retest	reliability	.79-.94	

for	parent	and	teacher	rating.

Validity: Mean	scores	

between	regular	education	

and	students	classified	as	

seriously	emotionally	

disturbed	were	different.	

Scores	on	the	DESSA	

correlated	with	scores	on	the	

Behavioral	and	Emotional	

Rating	Scale	(BERS)	and	the	

Behavioral	Assessment	

System	for	Children	(BASC-2).	

(Nickerson	&	Fishman,	2009).

Teacher-report	

questionnaire	completed	

each	fall	and	spring.

Parent-report	through	

interview	each	summer.

Intermediate	Outcomes:	Relationships	and	Behaviors	

Measure	Name	and	Author Measure	Description	and	 Sub-Scales Psychometric	Properties Data	Collection	Method	
Age	Range and	Timeline

Individualized	Classroom	

Assessment	Scoring	System -	

Child	Version (inCLASS;	

Downer,	Booren,	et	al.,	

2009)

The	inCLASS is	an	

observational	measure	of

the	quality	of	children’s	

interactions	in	the	

classroom.		

Includes	10	dimensions	rated	on	a

7-point	scale	based	on	the	child’s	

display	of	specific	behavioral	

markers,	e.g.,				Interactions	with	

Teachers (positive	engagement

with	teachers,	teacher	

communication,	teacher	conflict)	

and	Interactions	with	Peers (peer	

sociability,	peer	communication,	

peer	assertiveness,	peer	conflict).	

ICCs	from	double-coded	

observations	above	.84.	

inCLASS	scores	are	associated	

with	relevant	teacher	ratings,	

ranging	.20-.50

Shows	predictive	validity	with	

direct	assessments	of	selfregulation	(Downer,	Booren,	

Luckner,	&	Pianta,	2009)	and	

expressive	language	and	

letter	knowledge	(Vitiello,	

Downer,	Williford,	&	Booren,	

2010).	

Classroom	observation	by	a	

certified	assessor	in	the	

spring	of	each	year.	

The	Student-Teacher	

Relationship	Scale (STRS;	

Pianta,	2001)

Teachers	assess	the	quality	

of	their	relationship	with	an	

individual	child	in	their	

classroom	using	a	15-item,	

5-point	scale	(Pianta,	

2001).

Scales	include	Teacher-Child

Conflict and	Closeness.

Internal	consistencies	range	

.86-.89	(Pianta,	1992);	

predictive	relations	with		

children’s	classroom	

behavior,	school	retention,	

and	academic	outcomes	

(Hamre	&	Pianta,	2001;	

Pianta,	Steinberg,	&	Rollins,	

1995).		

Teacher-report	

questionnaire	completed	

each	fall	and	spring.

The	teacher	version	of	the	

Social	Skills	Improvement	

System	(SSIS;	Gresham	&	

Elliott,	2008).

The	SSIS	is	a	widely	used	

teacher-reported	measure	

of	an	individual	child’s	

relationships	and	social	

behaviors.	The	SSIS	uses	

norms	generated	from	a	

sample	of	more	than	4,500

children	from	across	the	

United	States.

The	child’s	Cooperation,	

Responsibility,	Empathy,	

Engagement,	and	Self-Control.	

Problem	behaviors	assessed	

include	Externalizing,	Bullying,	

Hyperactive	/Inattention, and	

Internalizing.	

High	levels	of	internal	

consistency	reliability	(alphas	

range	from	.75-.97)	and	

interrater	reliability	(alphas	

range	from	.74-.86).

Correlated	above	.50	with	

other	established	measures	

of	social	skills,	such	as	the	

previous	version	of	this	

measure	(SSRS;	and	the	

BASC-2.

Teacher-report	

questionnaire	completed	

each	fall	and	spring.

The	Child-Parent	

Relationship	Scale (CPRS;	

Parents	assess	the	quality	

of	relationship	with	WINGS

Conflict,	Positive	Aspects of	the	 Reliability	of	the	CPRS	was	

demonstrated	on	a	sample

Parent-report	through	
Pianta,	1992). study	child	with	a	30-item	

Likert-type	scale	(1-5).

relationship,	and	Dependence (n=714)	of	children	aged	4.5	

to	5.5	years	of	age	(alphas	

range	from	.50	for	the	

dependence	scale	to	.83	for	

the	conflicts	scale;	Pianta,	

1992).

interview	each	summer.

The	parent	version	of	the	

Social	Skills	Improvement	

System	(SSIS;	Gresham	&	

Elliott,	2008).

The	SSIS is	a	widely-used	

parent-reported	measure	

of	an	individual	child’s	

relationships	and	social	

behaviors.	

Subscales	include	Cooperation,	

Responsibility,	Empathy,	

Engagement,	and	Self-Control.	

Problem	behaviors	assessed	

include	Externalizing,	Bullying,	

Hyperactive/Inattention,	and	

Internalizing.		

Internal	consistency	

reliability	above	.80;	

interrater	reliability	above	

.70.	Correlated	above	.47	

with	other	established	

measures	of	social	skills,	such

as	the	previous	version	of	

this	measure	(SSRS;	and	the	

BASC-2.

Parent-report	through	

interview	each	summer.

Long-Term	School	Outcomes:	Academic	Achievement	and	Social	Adjustment

Measure	Name	and	Author Measure	Description	and	

Age	Range

Sub-Scales Psychometric	Properties Data	Collection	Method	

and	Timeline

Administrative	records	from	

the	child’s	school.	

Records	documenting	the	

child’s	school	experiences	

and	documented	problems.

Discipline	Problems;	Attendance;	

Special	Education	Referral; and	

Repeated	a	Grade.

Coordination	with	school	

district	the	year	following

participation.

Differential	Ability	Scales	II	

(Elliott,	2007)		

The	WJ-III	is	a	widely	used,	

individually	administered	

assessment	battery	that	

measures	general	cognitive

abilities	and	achievement.

Appropriate	for	ages	2	1⁄2	

through	17	years.

Working	Memory:	Recall	of	

Sequential	Order	subtest	

Receptive	Vocabulary:	Verbal	

Comprehension	subtest

Expressive	Vocabulary:	Naming

Vocabulary	subtest

Demonstrates	convergent	

validity	with

WISC–IV;	WPPSI–	III;	WIAT–II:	

Kaufman	Test	of	Educational	

Achievement	II; Woodcock	

Johnson-	III

Direct	assessment	at	

baseline	and	post-test	each	

year	of	participation	during	

the	summer	months.

The Woodcock-Johnson-III	

Tests	of	Achievement (WJ-III;	

Woodcock;	McGrew,	&	

Mather,	2001).	

The	WJ-III	is	a	widely	used,	

individual	assessment	

battery	that	measures	

general	cognitive	abilities	

and	achievement	from	age	

two	through	adulthood,	

providing	standardized	

performance	relative	to	the

Reading	skills:	Letter-Word	

Identification	(name	letters	and	

read	real	words).	Mathematics	

skills:	Applied	Problems	(simple	

calculations	and	word	problems).

Broad	Academic	Performance:	

Subtests	demonstrate	high	

internal	reliability	and	

acceptable	validity.

Direct	assessment	at	

baseline	and	post-test	each	

year	of	participation	during	

the	summer	months.
same-age	population. (Academic	Knowledge)
Appendix F- Teacher and Parent Survey Instruments 

This Appendix contains three sections: teacher, parent and family instruments. The teacher 

measures are administered at various times; please see the following table which contains 

descriptions of each, their subscales, psychometric properties, collection method and timeline. 

The Parent Quantitative and Qualitative Interview gather information at the beginning of each 

school year. 







Teacher Measures 

o

o

o

o

o

o

o

o

o

Devereux Student Strengths Assessment (DESSA; Lebuffe, Shapiro, & Naglieri, 

2008)

Student-Teacher Relationship Scale (STRS; Pianta, 2001) 

Social Skills Improvement System (SSIS; Gresham & Elliott, 2008) 

Woodcock-Johnson-III Tests of Achievement (WJ-III; Woodcock; McGrew, & 

Mather, 2001) 

Differential Ability Scales (DAS)







DAS II- Verbal Comprehension 

DASII- Recall of Sequential Order 

DAS II- Naming Vocabulary 

Head-Toes-Knees-Shoulders Task (HTKS, Ponitz, McClelland, et al., 2008) 

Emotion Matching Task (EMT) (EMT; Morgan, Izard, & King, 2009) and 

Assessment of Children’s Knowledge Task (ACES; Mavroveli et al., 2009) 

Theory of Mind (NEPSY II; Korkman, Kirk & Kemp, 2007a, 2007b) 

Delay of Gratification Task, (Mischel, Shoda, & Rodriguez, 1989) 

Parent Quantitative Interview 

Family Qualitative Interview 
Appendix F - Description of Teacher Measures and Psychometrics

Measure Name 

and Author

Measure Description and Age 

Range

Sub-Scales Psychometric Properties Data Collection 

Method and Timeline

Devereux Student 

Strengths 

Assessment

(DESSA; Lebuffe, 

Shapiro, & Naglieri, 

2008).

72 items rated on a 5‐point 

scale varying from “never” to 

“very frequently.”

Appropriate for K-8th grade.

Five constructs include: Self‐

Awareness, Self‐Management, 

Social Awareness, Relationship 

Skills, Responsible decision‐

making

Reliability: Alpha coefficients for scales 

ranged from .82 to .98.

Test-retest reliability .79‐.94 for parent and 

teacher rating.

Validity: Mean scores between regular 

education and students classified as 

seriously emotionally disturbed were 

different. Scores on the DESSA correlated 

with scores on the Behavioral and Emotional

Rating Scale (BERS) and the Behavioral 

Assessment System for Children (BASC‐2).

(Nickerson & Fishman, 2009).

Teacher-report 

questionnaire 

completed each fall 

and spring.

Parent-report 

through interview 

each summer.

The StudentTeacher 

Relationship Scale

(STRS; Pianta, 

2001)

Teachers assess the quality of 

their relationship with an 

individual child in their 

classroom using a 15-item, 5-

point scale (Pianta, 2001).

Scales include Teacher-Child

Conflict and Closeness.

Internal consistencies range .86-.89 (Pianta, 

1992); predictive relations with children’s 

classroom behavior, school retention, and 

academic outcomes (Hamre & Pianta, 2001; 

Pianta, Steinberg, & Rollins, 1995). 

Teacher-report 

questionnaire 

completed each fall 

and spring.

The teacher version 

of the Social Skills 

Improvement 

System (SSIS; 

Gresham & Elliott, 

2008).

The SSIS is a widely used 

teacher-reported measure of 

an individual child’s 

relationships and social 

behaviors. The SSIS uses 

norms generated from a 

sample of more than 4,500 

children from across the 

United States.

The child’s Cooperation, 

Responsibility, Empathy, 

Engagement, and Self-Control. 

Problem behaviors assessed 

include Externalizing, Bullying, 

Hyperactive /Inattention, and 

Internalizing.

High levels of internal consistency reliability 

(alphas range from .75-.97) and interrater 

reliability (alphas range from .74-.86).

Correlated above .50 with other established 

measures of social skills, such as the 

previous version of this measure (SSRS; and 

the BASC-2.

Teacher-report 

questionnaire 

completed each fall 

and spring.

The WoodcockJohnson-III Tests of 

Achievement (WJIII; Woodcock; 

McGrew, & 

Mather, 2001). 

The WJ-III is a widely used, 

individual assessment battery 

that measures general 

cognitive abilities and 

achievement from age two 

through adulthood, providing 

standardized performance 

relative to the same-age 

population.

Reading skills: Letter-Word 

Identification (name letters and 

read real words). Mathematics 

skills: Applied Problems 

(simple calculations and word 

problems). 

Broad Academic Performance: 

(Academic Knowledge)

Subtests demonstrate high internal 

reliability and acceptable validity.

Direct assessment at 

baseline and posttest each year of 

participation during 

the summer months.
Measure Name 

and Author

Measure Description and Age 

Range

Sub-Scales Psychometric Properties Data Collection 

Method and Timeline

Differential Ability 

Scales II (Elliott, 

2007)

The WJ-III is a widely used, 

individually administered 

assessment battery that 

measures general cognitive 

abilities and achievement.

Appropriate for ages 2 1⁄2 

through 17 years.

Working Memory: Recall of 

Sequential Order subtest 

Receptive Vocabulary: Verbal 

Comprehension subtest

Expressive Vocabulary: Naming 

Vocabulary subtest

Demonstrates convergent validity with

WISC–IV; WPPSI– III; WIAT–II: Kaufman Test

of Educational Achievement II; Woodcock 

Johnson- III

Direct assessment at 

baseline and posttest each year of 

participation during 

the summer months.

Head-Toes-KneesShoulders (HTKS, 

Ponitz, McClelland, 

et al., 2008)

Children must pay attention, 

remember the instructions, 

and demonstrate selfmanagement in a behavioral 

motor response by touching 

their head when asked to 

“touch your toes” and by 

touching their toes when 

asked to “touch your head.”

The task increases in 

complexity and is appropriate 

for pre-K through early 

elementary years.

The HTKS task has been

conceptualized by Ponitz, et al., 

(2008) as a measure of 

inhibitory control (a child must 

inhibit the dominant response 

of imitating the examiner), 

working memory (a child must 

remember the rules of the 

task) and attention focusing 

(must focus attention to the 

directions being presented by 

the examiner).

Scores on the HTKS and its two-rule 

predecessor, the Head-to-Toes Task (HTT) 

have shown strong reliability and construct 

and predictive validity in several studies with 

diverse samples (McClelland, Cameron et al., 

2007; Ponitz, McClelland, et al., 2008; 

Matthews et al., in press; Ponitz et al., in 

press).

Direct assessment at 

baseline and posttest each year of 

participation during 

the summer months.

Emotion Matching 

Task (EMT; 

Morgan, Izard, & 

King, 2009)

The Emotion Matching Task 

(EMT: Izard et al., 2003) was 

designed for preschool age 

children, including those who 

are economically 

disadvantaged. The EMT 

features brightly colored 

photographs of ethnically 

diverse children making facial 

expressions of happiness, 

sadness, anger, fear/surprise, 

and “neutral” (no visible facial 

muscle movement that 

signals emotion).

The EMT consists of four parts 

which measure the 

components of receptive 

emotion knowledge, expressive 

emotion knowledge, emotion 

situation knowledge, and 

emotion expression matching.

Regression analyses revealed moderate to 

strong predictive validity for EMT. Compared 

to KEI and AKT, the EMT was a more robust 

predictor of teacher rated emotion 

regulation and parent reported effortful 

control. Compared to KEI and AKT, the EMT 

correlated similarly with verbal ability and 

age (Morgan, Izard, & King, 2009)

Direct assessment at 

baseline and posttest each year of 

participation during 

the summer months.
Measure Name 

and Author

Measure Description and Age 

Range

Sub-Scales Psychometric Properties Data Collection 

Method and Timeline

Assessment of 

Children’s 

Knowledge Task 

(ACES)

Assesses children's emotion 

attribution accuracy and 

emotion biases.

Age range: elementary school 

years

Three sections that cover social 

behaviors (presented with 15 

brief social situations, children 

are asked to label appropriate 

emotions), social situations 

(three vignettes), and facial 

expressions (labeling emotions 

from 26 photographs).

Reliability: Internal consistency is adequate 

across studies

(e.g., Mavroveli et al., 2009; Schultz et al., 

2004). This scale had moderate internal

reliability (a = .71). These items cohered

moderately well (Cronbach’s alpha = .68). 

Validity: Mavroeli et al.(2009) found that the 

ACES correlated well with trait emotional 

intelligence.

Direct assessment at 

baseline and posttest each year of 

participation during 

the summer months.

Theory of Mind 

sub-test of the 

Developmental 

NEuroPSYchological 

Assessment 

(NEPSY-II; 

Korkman, Kirk & 

Kemp, 2007a, 

2007b)

Two tasks designed to assess 

ability to understand mental 

functions and another’s point 

of view. 1) Verbal task 

assessment 

Understanding of another’s 

thoughts, ideas, feelings. 2) 

Contextual task assesses 

ability to relate emotion to 

social context. Appropriate 

for ages 3-16.

Theory of Mind construct, or 

the ability to recognize 

emotions, to guess what 

another person is thinking and 

feeling, empathy.

Up-to-date psychometric norms are based 

on the standardization of over 1,000 U.S. 

children which enables the comparison of a 

child's performance to others in the 

appropriate age group. Validity Studies were 

carried out with NEPSY, WISC-IV, DAS—IIII, 

WNV, WIAT—IIII, CMS, DKEFS, BBCS:3R, 

DSMD, ABAS—II, Brown ADD Scales and 

CCC-2.

Direct assessment at 

baseline and posttest each year of 

participation during 

the summer months.

Sticker and candy 

delay of 

gratification task 

(Brock 2008, 

unpublished 

dissertation thesis; 

Hongwanishkul et 

al, 2005). 

Each child asked to choose 

either a single desirable item 

(sticker, candy) immediately 

or a greater number (2, 4, or 

6) of the same item after a 

delay. 

Dichotomous outcomes of 

single item now versus greater 

number later; 6 summary 

scores calculated. 

Inter-rater reliability is high (1.00, Brock, 

2008). Performance on a task involving 

prolonged delay has been found to be 

predictive of academic performance into 

high school (Mischel, Shoda, and Rodriguez, 

M.L, 1989).

Direct assessment at 

baseline and posttest each year of 

participation during 

the summer months.
CHILD ID___________________

DESSA Items Scale

During the past 4 weeks, how often did the child...

1. Never 2. Rarely 3. Occasionally

4. Frequently5. Very 

Frequently

Don't 

know

Prefer 

not to 

answer 

(refuse)

1. cope well with insults and mean comments? 1 2 3 4 5 DK RF

2. get along with different types of people? 1 2 3 4 5 DK RF

3. act respectfully in a game or competition? 1 2 3 4 5 DK RF

4. respect another person's opinion? 1 2 3 4 5 DK RF

5. contribute to group efforts? 1 2 3 4 5 DK RF

6. resolve a disagreement? 1 2 3 4 5 DK RF

7. share with others? 1 2 3 4 5 DK RF

8. cooperate with peers or siblings? 1 2 3 4 5 DK RF

9. forgive somebody who hurt or upset her/him? 1 2 3 4 5 DK RF

10. follow the example of a positive role model? 1 2 3 4 5 DK RF

11. compliment or congratulate somebody? 1 2 3 4 5 DK RF

12. accept responsibility for what she/he did? 1 2 3 4 5 DK RF

13. do something nice for somebody? 1 2 3 4 5 DK RF

14. make accurate statements about events in her/his life? 1 2 3 4 5 DK RF

15. show good judgment? 1 2 3 4 5 DK RF

16. pay attention? 1 2 3 4 5 DK RF

17. wait for her/his turn? 1 2 3 4 5 DK RF

18. show appreciation of others? 1 2 3 4 5 DK RF

19. focus on a task despite a problem or distraction? 1 2 3 4 5 DK RF

20. greet a person in a polite way? 1 2 3 4 5 DK RF

21. act comfortable in a new situation? 1 2 3 4 5 DK RF

22. teach another person to do something? 1 2 3 4 5 DK RF

23. attract positive attention from peers? 1 2 3 4 5 DK RF

24. perform the steps of a task in order? 1 2 3 4 5 DK RF
CHILD ID___________________

Scale

DESSA (continued) Items

1. Never 2. Rarely 3. Occasionally

4. Frequently5. Very 

Frequently

Don't 

know

Prefer 

not to 

answer 

(refuse)

25. seek advice? 1 2 3 4 5 DK RF

26. think before he/she acted? 1 2 3 4 5 DK RF

1 2 3 4 5 DK RF

28. express concern for another person? 1 2 3 4 5 DK RF

29. accept another choice when his/her first choice was unavailable? 1 2 3 4 5 DK RF

30. ask questions to clarify what he/she did not understand? 1 2 3 4 5 DK RF

31. show an awareness of her/his personal strengths? 1 2 3 4 5 DK RF

32. ask somebody for feedback? 1 2 3 4 5 DK RF

33. stay calm when faced with a challenge? 1 2 3 4 5 DK RF

34. attract positive attention from adults? 1 2 3 4 5 DK RF

35. describe how he/she was feeling? 1 2 3 4 5 DK RF

36. give an opinion when asked? 1 2 3 4 5 DK RF

37. make a suggestion or request in a polite way? 1 2 3 4 5 DK RF

38. learn from experience? 1 2 3 4 5 DK RF

39. follow the advice of a trusted adult? 1 2 3 4 5 DK RF

40. adjust well to changes in plans? 1 2 3 4 5 DK RF

41. show the ability to decide between right and wrong? 1 2 3 4 5 DK RF

42. use available resources (people or objects) to solve a problem? 1 2 3 4 5 DK RF

43. offer to help somebody? 1 2 3 4 5 DK RF

44. respond to another person's feelings? 1 2 3 4 5 DK RF

45. adjust well when going from one setting to another? 1 2 3 4 5 DK RF

27. pass up something he/she wanted, or do something he/she did 

not like, to get something better in the future? 
Child ID

Student-Teacher Relationships

Please reflect on the degree to which each of the following statements currently applies to your 

relationship with this student. Circle the appropriate number for each item. Definitely does not apply Not really

Neutral, not 

sure

Applies 

somewhat 

Definitely 

applies

a. I share an affectionate, warm relationship with this child. 1 2 3 4 5

b. This child and I always seem to be struggling with each 

other. 

1 2 3 4 5

c. If upset, this child will seek comfort from me. 1 2 3 4 5

d. This child is uncomfortable with physical affection or 

touch from me.

1 2 3 4 5

e. This child values his/her relationship with me. 1 2 3 4 5

f. When I praise this child, he/she beams with pride. 1 2 3 4 5

g. This child spontaneously shares information about 

himself/herself.

1 2 3 4 5

h. This child easily becomes angry at me. 1 2 3 4 5

i. It is easy to be in tune with what this student is feeling. 1 2 3 4 5

j. This child remains angry or is resistant after being 

disciplined.

1 2 3 4 5

k. Dealing with this child drains my energy. 1 2 3 4 5

l. When this child arrives in a bad mood, I know we’re in for 

a long and difficult day. 

1 2 3 4 5

m. This child’s feelings toward me can be unpredictable or 

can change suddenly. 

1 2 3 4 5

n. This child is sneaky or manipulative with me. 1 2 3 4 5

o. This child openly shares his/her feelings and experience 

with me.

1 2 3 4 5

© Pianta, 1992
Child ID

Teacher ID: ______________(for office use only)

Social Skills Rating Scale 

We are interested in learning about children’s social competence and problem 

behaviors in classroom settings. Please read the following statements with the above 

child in mind. Decide how frequently the statement applies to him or her (1=never, 

2=seldom, 3=often, 4=almost always). Circle just ONE number for each statement. Be 

sure to respond to each question. Almost Never Seldom Often always SOCIAL COMPETENCE

1. Observes rules and follows directions without 

requiring repeated reminders.

1 2 3 4 

2. Interacts well with other children. 1 2 3 4 

3. Joins activities that have already started. 1 2 3 4 

4. Uses appropriate language when upset. 1 2 3 4 

5. Responds appropriately when pushed or hit. 1 2 3 4 

6. Participates in games or group activities. 1 2 3 4 

7. Returns to unfinished tasks after interruption. 1 2 3 4 

8. Responds to instructions and then begins an 

appropriate task without being reminded.

1 2 3 4 

9. Takes time to do his/her best on a task. 1 2 3 4 

10. Invites others to join in activities. 1 2 3 4 

11. Starts conversations with peers. 1 2 3 4 

12. Makes a compromise during a conflict. 1 2 3 4 

13. Attempts new challenging tasks. 1 2 3 4 

14. Stays calm when disagreeing with others. 1 2 3 4 

15. Makes friends easily. 1 2 3 4 

16. Completes learning tasks involving two or more 

steps (e.g. cutting and pasting) in organized way. 1 2 3 4 

17. Completes tasks successfully. 1 2 3 4 

18. Introduces herself/himself to others. 1 2 3 4 

19. Concentrates when working on a task; is not easily 

distracted by surrounding activities.

1 2 3 4 

20. Stays calm when teased. 1 2 3 4 

21. Takes criticism without getting upset. 1 2 3 4 

22. Finds and organizes materials and works in an 

appropriate place when activities are initiated. 1 2 3 4 

23. Sees own errors in a task and corrects them. 1 2 3 4 

24. Resolves disagreements with you calmly.

 

 (Continue on Back) 

1 2 3 4 
Child ID

Teacher ID: ______________(for office use only)

 Almost

 Never Seldom Often always

PROBLEM BEHAVIORS

25. Acts without thinking. 1 2 3 4 

26. Bullies others. 1 2 3 4 

27. Has difficulty waiting for turn. 1 2 3 4 

28. Does things to make others feel scared. 1 2 3 4 

29. Fidgets or moves around too much. 1 2 3 4 

30. Forces others to act against their will. 1 2 3 4 

31. Withdraws from others. 1 2 3 4 

32. Has temper tantrums. 1 2 3 4 

33. Keeps others out of social circles. 1 2 3 4 

34. Breaks into or stops group activities. 1 2 3 4 

35. Is aggressive toward people or objects. 1 2 3 4 

36. Gets embarrassed easily. 1 2 3 4 

37. Cheats in games or activities. 1 2 3 4 

38. Acts lonely. 1 2 3 4 

39. Is inattentive. 1 2 3 4 

40. Fights with others. 1 2 3 4 

41. Says bad things about self. 1 2 3 4 

42. Disobeys rules or requests. 1 2 3 4 

43. Has low energy or is lethargic. 1 2 3 4 

44. Gets distracted easily. 1 2 3 4 

45. Talks back to adults. 1 2 3 4 

46. Acts sad or depressed. 1 2 3 4 

47. Lies or does not tell the truth. 1 2 3 4 

48. Acts anxious with others. 1 2 3 4 

Thank you for providing this important information!




















Child ID: _____________ 

HTKS Form B Extended © – July 2011 1 

HEAD-TOES-KNEES-SHOULDERS (HTKS)

©2011 Cameron & McClelland 

Parts I, II, and III

FORM B – Extended

REFERENCES: 

McClelland, M. M., Cameron, C. E., Connor, C. M., Farris, C. L., Jewkes, A. M., & Morrison, F. J. (2007). 

Links between behavioral regulation and preschoolers' literacy, vocabulary, and math skills. 

Developmental Psychology, 43, 947-959.

Cameron Ponitz, C. E., McClelland, M. M., Matthews, J. S., & Morrison, F. J. (2009). A structured 

observation of behavioral self-regulation and its contribution to kindergarten outcomes. 

Developmental Psychology, 45, 605-619.

Cameron Ponitz, C. E., McClelland, M. M., Jewkes, A. M., Connor, C. M., Farris, C. L., & Morrison, F. J. 

(2008). Touch your toes! Developing a direct measure of behavioral regulation in early childhood. 

Early Childhood Research Quarterly, 23

Directions: After establishing positive rapport with the child, say or read the directions in bold 

type aloud. Words in CAPITAL LETTERS should be emphasized. Administer the task seated or 

standing; the child should stand, about 3 feet from you, during the task. Administer Part II if the 

number of points in the testing section totals to 4 or more. Administer Part III if the number of 

points in the testing section totals to 4 or more. 

The person symbol indicates that you should perform the motion to demonstrate the correct 

movement to the child. If the child produces the correct (opposite) response immediately, score 

the item “2”. If they self-correct to the correct response, score the item “1”. If they do not touch 

the correct part of their body at all or touch the named part, score the item “0”. 

A self-correct occurs if the child makes any discernible motion toward an incorrect response, but 

then changes his/her mind and makes the correct response. Pausing to think, not moving, and 

then responding correctly does not count as a self-correction – it would be scored as correct.

COPYRIGHT STATEMENT

Copyright and other intellectual property laws protect these materials. Reproduction or retransmission of the materials, in whole or in part, in any 

manner, without the prior written consent of the copyright holder, is a violation of copyright law. Users may not distribute such copies to others, 

whether or not in electronic form, whether or not for a charge or other consideration, without prior written consent of the copyright holder of the 

materials. Users may not post the task online. Contact Megan McClelland (megan.mcclelland@oregonstate.edu) or Claire Cameron 

(cecamero@buffalo.edu) for requests for permission to reproduce the Head-Toes-Knees-Shoulders task. 
Child ID: _____________ 

HTKS Form B Extended © – July 2011 2 

PART I: INTRODUCTION

 Incorrect self-correct correct

 

Now we’re going to play a game. The game has two parts. First, copy what I do. Touch your 

knees.

Touch your knees; wait for the child to touch his/her knees. 

 

Touch your shoulders; wait for the child to touch his/her shoulders. 

Repeat the two commands with motions again, or until the child imitates you correctly.

Good! Now touch your shoulders.

PART I: PRACTICE

Now we’re going to be a little silly and do the OPPOSITE of what I say. When I say to touch 

your KNEES, INSTEAD of touching your knees, you touch your SHOULDERS. When I say to 

touch your SHOULDERS, you touch your KNEES. So you’re doing something DIFFERENT 

from what I say. 

If the child responds correctly: Provide positive feedback on each practice item where 

the child responds correctly. 

**If the child responds incorrectly at any point during the practice portion, provide 

additional explanations up to 3 times before beginning the test portion: 

Remember, when I say to touch your ____, instead of touching your ____, 

you touch your ____. Do the OPPOSITE of what I say.

Number of additional explanations given: 0 1 2 3 

A1. What do you do if I say “touch your knees”? 0 (other than shoulders) 1 2 (shoulders)

A2. What do you do if I say “touch your shoulders”? 0 (other than knees) 1 2 (knees)

If the child responds verbally: “can you show me? “

Ok, let’s practice a few more. 

Incorrect self-correct correct

B1. Touch your knees 0 (other than shoulders) 1 2 (shoulders)

B2. Touch your shoulders 0 (other than knees) 1 2 (knees)

B3. Touch your knees 0 (other than shoulders) 1 2 (shoulders)

B4. Touch your shoulders 0 (other than knees) 1 2 (knees)

Proceed to Part I test section. Do not explain any parts of the task again. Do not provide feedback during 

the test portion.
Child ID: _____________ 

HTKS Form B Extended © – July 2011 3 

PART I: TESTING

We will keep playing this game, and you keep doing the OPPOSITE of what I say.

Incorrect Self-Correct Correct

1. Touch your knees 0 (other than shoulders) 1 2 (shoulders)

2. Touch your shoulders 0 (other than knees) 1 2 (knees)

3. Touch your shoulders 0 (other than knees) 1 2 (knees)

4. Touch your knees 0 (other than shoulders) 1 2 (shoulders)

5. Touch your shoulders 0 (other than knees) 1 2 (knees) 

6. Touch your knees 0 (other than shoulders) 1 2 (shoulders)

7. Touch your knees 0 (other than shoulders) 1 2 (shoulders) 

8. Touch your shoulders 0 (other than knees) 1 2 (knees) 

9. Touch your knees 0 (other than shoulders) 1 2 (shoulders) 

10. Touch your shoulders 0 (other than knees) 1 2 (knees) 

TOTAL POINTS IF THE CHILD SCORED 4 OR MORE POINTS, CONTINUE TO PART II 

IF THE CHILD SCORED LESS THAN 4 POINTS: Thank you for playing this game with me today!
Child ID: _____________ 

HTKS Form B Extended © – July 2011 4 

PART II: INTRODUCTION

 

Ok, now that you’ve got that part, we’re going to add a part. Now, you’re going to touch 

your HEAD and your TOES. First, touch your HEAD. 

Touch your head; wait for the child to touch his/her head. 

Touch your toes; wait for the child to touch his/her toes.

Repeat the two commands with motions again, or until the child imitates you correctly.

Now, touch your toes. 

PART II PRACTICE:

Ok, now we’re going to be silly again. You keep doing the opposite of what I say like 

before. But this time, touch your HEAD and TOES. When I say to touch your HEAD, you 

touch your TOES, and when I say to touch your TOES, you touch your HEAD. 

If the child responds correctly: Provide positive feedback on each practice item where 

the child responds correctly. 

**If the child responds incorrectly at any point during the practice portion, provide 

additional explanations up to 2 times before beginning the test portion: 

Remember, when I say to touch your ____, you touch your ____, so you 

are doing something DIFFERENT from what I say. Let’s try another one.

Number of additional explanations given: 0 1 2 

 Incorrect Self-correct Correct

C1. What do you do if I say “touch your head”? 0 (other than toes) 1 2 (toes)

If the child responds verbally: “can you show me? “

 Incorrect Correct

D1. Touch your head 0 (other than toes) 1 2 (toes)

D2. Touch your toes 0 (other than head) 1 2 (head)

D3. Touch your head 0 (other than toes) 1 2 (toes)

D4. Touch your toes 0 (other than head) 1 2 (head)

Self-correct

Proceed to Part II test section. Do not explain any parts of the task again. Do not provide feedback during 

the test portion.
Child ID: _____________ 

HTKS Form B Extended © – July 2011 5 

Now that you know all the parts, we’re going to put them together. You’re going to keep 

doing the opposite of what I say to do, but you won’t know what I’m going to say. 

There are four things I could say:

If I say touch your HEAD, you touch your TOES. 

If I say touch your TOES, you touch your HEAD. 

If I say touch your KNEES, you touch your SHOULDERS. 

If I say touch your SHOULDERS, you touch your KNEES.

Are you ready? Let’s try it.

PART II TESTING:

Incorrect Self-Correct Correct

11. Touch your head 0 (other than toes) 1 2 (toes) 

12. Touch your toes 0 (other than head) 1 2 (head)

13. Touch your knees 0 (other than shoulders) 1 2 (shoulders)

14. Touch your toes 0 (other than head) 1 2 (head

15. Touch your shoulders 0 (other than knees) 1 2 (knees)

16. Touch your head 0 (other than toes) 1 2 (toes)

17. Touch your knees 0 (other than shoulders) 1 2 (shoulders)

18. Touch your knees 0 (other than shoulders) 1 2 (shoulders)

19. Touch your shoulders 0 (other than knees) 1 2 (knees) 

20. Touch your toes 0 (other than head) 1 2 (head)

TOTAL POINTS IF THE CHILD SCORED 4 OR MORE POINTS, CONTINUE TO PART III 

IF THE CHILD SCORED LESS THAN 4 POINTS: Thank you for playing this game with me today!
Child ID: _____________ 

HTKS Form B Extended © – July 2011 6 

PART III INTRODUCTION

You are doing so well we just have one more part! Now we are going to change the rules 

of the game. 

When I say to touch your HEAD, you touch your KNEES. 

When I say touch your KNEES, you touch your HEAD. 

When I say touch your SHOULDERS, you touch your TOES. 

And when I say touch your TOES, you touch your SHOULDERS. 

Ok? Let’s practice!

If the child responds correctly: Provide positive feedback on each practice item where 

the child responds correctly. 

**If the child responds incorrectly at any point during the practice portion, provide 

additional explanations up to 2 times before beginning the test portion: 

 

 

Remember, we changed the rules. “Touch your head” means touch your 

KNEES – head goes with knees now. “Touch your shoulders” means touch 

your TOES – shoulders goes with toes. 

Number of additional explanations given: 0 1 2 

PART III PRACTICE:

 Incorrect self-correct correct

F1. Touch your head 0 (other than knees) 1 2 (knees)

F2. Touch your shoulders 0 (other than toes) 1 2 (toes)

F3. Touch your toes 0 (other than shoulders) 1 2 (shoulders)

F4. Touch your knees 0 (other than head) 1 2 (head)

If the child responds verbally: “can you show me? “

 Incorrect self-correct correct

E1. What do you do if I say “touch your head”? 0 (other than knees) 1 2 knees)

E2. What do you do if I say “touch your shoulders”? 0 (other than toes) 1 2 (toes) 

You’re doing great! Let’s do a few more. 

Proceed to Part III test section. Do not explain any parts of the task again. Do not provide feedback 

during the test portion.
Child ID: _____________ 

HTKS Form B Extended © – July 2011 7 

PART III TESTING:

Incorrect Self-Correct Correct

21. Touch your shoulders 0 (other than toes) 1 2 (toes) 

22. Touch your head 0 (other than knees) 1 2 (knees) 

23. Touch your knees 0 (other than head) 1 2 (head) 

24. Touch your toes 0 (other than shoulders) 1 2 (shoulders) 

25. Touch your toes 0 (other than shoulders) 1 2 (shoulders)

26. Touch your knees 0 (other than head) 1 2 (head)

27. Touch your shoulders 0 (other than toes) 1 2 (toes)

28. Touch your head 0 (other than knees) 1 2 (knees)

29. Touch your head 0 (other than knees) 1 2 (knees)

30. Touch your shoulders 0 (other than toes) 1 2 (toes)

Thank you for playing this game with me today!
EMT/ACES Combined Measure

Test Administration:

1. Sit beside the child, not across from them. This will help you match up the test 

booklet photographs with the recording sheet. 

2. For each of the 48 items of EMT, make sure you have the child’s attention 

focused on the task. If giving the instructions exactly as they are written for 

each Part does not elicit a response to the item, say in a neutral tone “Ok, let’s 

do the next one.” Then write “DK” for that item on the answer form. 

3. Administer Parts, 1,2,3, and 4 to all children. Only administer Parts 5 and 6 

to children who answered at least 6 questions correctly on Part 2. 

4. When children are expected to respond by indicating one of four pictures, 

mark the quadrant they choose with a circle (an X might give a child the 

impression that they’ve answered the question incorrectly). For example: 
Part 1: Matching two Expressions of the Same Emotion

Note: Always keep the Quad of expressions on the right page covered until you are ready to ask 

the child to point to the Key expression.

Child Instructions: 

Show child Key expression from 1st warm-up item:

“Now we will play the game with pictures of children’s faces. The faces show how the 

children feel. Let’s play the game. OK?”

1st Warm-up:

 “Look at her face. [Point to Key expression]. Her face shows how she feels. Now, I want to find 

the face over here [uncover and point to the photos in the quad] that shows the same feeling as 

this one [point back to Key expression]. [Pause 2s, then point to the matching photo and say]: It’s 

this one, isn’t it! Her face shows that she [still pointing to the matching photo] feels the same 

way as this one.” [Point to the Key expression].

2nd Warm-up: 

Turn to 2nd warm up item:

“Look at her face. [Point to the Key expression] Her face shows how she feels.”

[Interviewer waits about 3s.] “Now look at the faces over here. [Uncover and point to the 

adjacent Quad of expressions]. Now I want you to point to the one over here [point to the Quad] 

whose face shows the same feeling as this one [pointing to the Key expression].”

If necessary, repeat: “Look at these faces. Show me the one [point to the pictures in the Quad] 

that feels the same way as this one” [Point to the Key expression].

 Child’s Answer: 

Test: 

For each of the following 12 Key Expressions and associated Quads, ask the child to:

“Look at her face. [Point to Key expression] Her face shows how she feels. [Pause 4s] Now look 

at the faces over here [point to the Quad for item 1]. Show me which one of these [point to the 

Quad] feels the same as this one [point to Key expression].” 

1. 2. 3. 4. 5. 6.

7. 8. 9. 10. 11. 12.
Part 2: Matching a Spoken Description of an Emotion Activator With One of 

Four Expressions

Note: If the child does not give an answer in 4 seconds, repeat the question. If he or she still does 

not give an answer after 4 seconds, say in a neutral tone: “Ok, let’s do the next one.” Record “DK” 

for that item and continue with the next item.

Child Instructions: 

Show child Quad from the warm-up item: 

“Now, we’ll play another game with the pictures of children’s faces. It’s about faces children 

make when something happens to them. I’m going to say what just happened to one of 

these children [pointing to the first Warm-Up Quad] and I want you to show me which child it 

happened to.”

Warm-up:

Point again to the Quad for the first Warm-Up item and say:

“OK. Show me the one who just got a nice new toy, just what they 

wanted.” 

Test: Total Correct (score immediately):______________

1. Show me the one whose nice 

drawing just got torn up by a 

mean kid.

7. Show me the one who is all 

alone and has no one to play 

with.

2. Show me the one who just 

got invited to a party to play 

games with friends.

8. Show me the one who woke 

up one night and thought a 

monster was in the room.

3. Show me the one whose 

puppy just got lost and did not 

come home.

9. Show me the one who just 

got pushed away from the 

table by another kid.

4. Show me the one whose nice 

block tower was just kicked 

over by a mean kid.

10. Show me the one who is in 

the doctor’s office about to get 

a shot.

5. Show me the one who just 

saw a large dog running up 

and barking loud.

11. Show me the one whose 

mom is sick and has to go to 

the hospital.

6. Show me the one who got a 

pretty puppy for a birthday 

present.

12. Show me the one who did a 

puzzle faster than anyone else.

Child’s Answer:
Part 3: Emotion Expression Labeling

Note: If the child says a general emotion word or a non-emotion word (e.g., bad, good, big eyes), 

say “Can you say another word that tells me how she feels?” If the child gives a second 

response, place a “Q” before it.

Note: If the child does not give a response in 4 or 5s, repeat the initial question. If the child still 

does not respond after an additional 4s, say in a neutral tone: “Let’s do another one”. Write “DK” 

for any non-response or if the child says they do not know an answer.

Child Instructions: 

See warm-up.

Warm-up:

Show child warm up picture.

“Now, I’d like you to say a word that tells us how this person feels” [pointing to the warm-up 

item]. “Look at his face. His face shows how he feels. Tell me: How does he feel?” [If no 

response after 4s, say:] “Can you say a word that tells me how he feels?”

When the child responds, say “OK” in a neutral tone of voice. Then proceed to test items.

Child’s Answer: ___________________________________________________________

Test:

For each of the 12 test items, repeat: 

“Look at her face. How does she feel?”

1. 7.

2. 8.

3. 9.

4. 10.

5. 11.

6. 12. 
Part 4: Matching a Spoken Emotion Label with One of Four Emotion 

Expressions

Note: If the child does not give an answer in 4 seconds, repeat the question. If he or she still does 

not give an answer after 4 seconds, say in a neutral tone: “Ok, let’s do the next one.” Record “DK” 

for that item and continue with the next item.

Child Instructions: 

Show child the Quad for the warm-up item:

“Now we’ll play a different game. I’m going to tell you how one of these children feels [point 

to the Quad for the warm-up item, where Key expression is high intensity Happy]. Look at each of 

the pictures [point to the Quad again] and show me the one who feels what I say.”

Warm-up:

Show child warm-up Quad:

OK. Now, show me the one who feels happy. 

Test:

Child’s Answer:

1. Now, show me the one who 

feels happy.

7. Show me the one who feels 

scared or surprised.

2. Show me the one who feels 

scared or surprised.

8. Show me the one who feels 

mad. 

3. Show me the one who feels 

sad

9. Show me the one who feels 

scared or surprised.

4. Show me the one who feels 

mad. 

10. Show me the one who feels 

mad. 

5. Show me the one who feels 

sad.

11. Show me the one who feels 

happy.

6. Show me the one who feels 

happy.

12. Show me the one who feels 

sad.
Part 5: Emotional Situations

Child Instructions: 

“I’m going to tell you about some kids your age, and I want you to tell me if you think they 

would feel happy, sad, mad, or scared. Sometimes you might think they feel two emotions, 

like both mad and sad. If so, I want you to pick the feeling you think they would have more 

strongly. Sometimes they may not feel any emotion strongly, and you can tell me that by 

saying, "no feeling." Don't say "no feeling" just because you're not sure how they would 

feel, though. If you think they would feel anything, I want you to take a guess at what it is, 

okay?”

Test:

HappySadMadScaredNo Feeling 

1. Tim’s parents told him that they would take him to the fair. When it is time to go, his parents say that none 

of them can go. Do you think Tim feels happy, sad, mad, scared, or no feeling? H S M Sc N 

2. Kelly just finished coloring a picture. You tell her that it looks nice. Do you think Kelly feels happy, sad, mad, 

scared, or no feeling? H S M Sc N 

3. Jasmine took care of her kitten, which she loved very much. One day the kitten disappeared and never came 

back. Do you think Jasmine feels happy, sad, mad, scared, or no feeling? H S M Sc N 

4. Juan walks down the hall. A big kid walks right at Juan and tells him to get out of the way. Do you think Juan 

feels happy, sad, mad, scared, or no feeling? H S M Sc N 

5. Melissa is building a big tower of blocks. Another kid comes over and knocks it over and laughs. Do you 

think Melissa feels happy, sad, mad, scared, or no feeling? H S M Sc N 

6. Scott lets Ryan play with his favorite toy. Ryan plays with the toy, and it breaks. Do you think Scott feels 

happy, sad, mad, scared, or no feeling? H S M Sc N 

7. Lonnie is in line for lunch. Darren steps in front of him without asking. Do you think Lonnie feels happy, sad, 

mad, scared, or no feeling? H S M Sc N 

8. Sarah was riding her bike. She went down a big hill and started going faster than she wanted. Do you think 

Sarah feels happy, sad, mad, scared, or no feeling? H S M Sc N 

9. Alex made a nice card for his friend Josh. Josh likes the card a lot. Do you think Alex feels happy, sad, mad, 

scared, or no feeling? H S M Sc N 

10. Mary’s grandfather died. Do you think Mary feels happy, sad, mad, scared, or no feeling? H S M Sc N 

11. Adrian’s parents are having a fight in the bedroom. He can hear them yelling. Do you think Adrian feels 

happy, sad, mad, scared, or no feeling? H S M Sc N 

12. Brian was at the park, and his mother bought him an ice cream cone. Brian took one lick and then 

accidentally dropped the ice cream cone. Do you think Brian feels happy, sad, mad, scared, or no feeling? H S M Sc N 

13. James brings his favorite candy bar to school in his book bag. A boy sees the candy bar, takes it, and eats it. 

Do you think James feels happy, sad, mad, scared, or no feeling? H S M Sc N 

14. Michael is playing in the woods with Andy. Andy runs away and leaves Michael alone in the woods. It’s 

getting dark. Do you think Michael feels happy, sad, mad, scared, or no feeling? H S M Sc N 

15. It is the first day of school. Your friend Maria hasn’t seen you all summer. She sees you in class. Do you 

think Maria feels happy, sad, mad, scared, or no feeling? H S M Sc N 
Part 6: Emotional Behaviors

Child Instructions: 

I’m going to tell you about some kids your age, and I want you to tell me if they would feel 

happy, sad, mad, or scared. Sometimes you might think they would feel two emotions, like 

both mad and sad. If so, I want you to pick the feeling you think they would have more 

strongly. Sometimes they may not feel any emotion strongly, and you can tell me that by 

saying, "no feeling." Don't say "no feeling" just because you're not sure how they would 

feel, though. If you think they would feel something, I want you to take a guess at what it is, 

okay?

Test:

HappySadMadScaredNo Feeling 

1. Jack doesn’t feel like playing ball at recess. Instead, he just sits alone. Do you think Jack feels happy, sad, 

mad, scared, or no feeling? H S M Sc N 

2. You see Shelley hit Yvonne. When Shelley hit Yvonne, do you think Shelley felt happy, sad, mad, scared, or no 

feeling? H S M Sc N 

3. Instead of playing with his new toy, Marquis just sits there. Do you think Marquis feels happy, sad, mad, 

scared, or no feeling? H S M Sc N 

4. At recess you watch Mark play with some other kids. Mark gets the ball, and his body seems to freeze. Do 

you think Mark feels happy, sad, mad, scared, or no feeling? H S M Sc N 

5. You see Terry running to join you and the other kids in the game. Do you think Terry feels happy, sad, mad, 

scared, or no feeling? H S M Sc N 

6. Mark doesn’t want anyone to talk to him. Do you think Mark feels happy, sad, mad, scared, or no feeling? H S M Sc N 

7. When the teacher asks Laurie a question, you see Laurie look down. Do you think Laurie feels happy, sad, 

mad, scared, or no feeling? H S M Sc N 

8. Jeff is being nice to everybody. Do you think Jeff feels happy, sad, mad, scared, or no feeling? H S M Sc N 

9. David calls Kevin a bad name. When David called Kevin a bad name, do you think David felt happy, sad, mad, 

scared, or no feeling? H S M Sc N 

10. Jill talks softly, and her eyes seem watery. Do you think Jill feels happy, sad, mad, scared, or no feeling? H S M Sc N 

11. A group of kids are called into the principal’s office. You see Patrice walking at the back of the group slowly. 

Do you think Patrice feels happy, sad, mad, scared, or no feeling? H S M Sc N 

12. Rosa has her arms crossed. Do you think Rosa feels happy, sad, mad, scared, or no feeling? H S M Sc N 

13. Janell doesn’t feel like playing ball at recess. Instead, she sits alone. Do you think Janell feels happy, sad, 

mad, scared, or no feeling? H S M Sc N 

14. You see Joanne skipping down the hallway and whistling. Do you think Joanne feels happy, sad, mad, 

scared, or no feeling? H S M Sc N 

15. Jenn walks slowly with her head down. Do you think Jenn feels happy, sad, mad, scared, or no feeling? H S M Sc N 
NEPSY Theory of Mind Items with Scoring Guidelines

1. Picture of box on table; box has picture of cookies on it.

When Andre opened the cookie box, he saw that Mom had put some spaghetti in there. He was sad and put the box 

back. His brother came in and saw the cookie box. What did his brother think was in the box?

2. Picture of a mom and a girl looking through a department store window. There’s a couch on sale, and on the couch is 

a doll. 

Mom and Maya are looking in the store window. Mom is thinking about buying a new sofa. What does Mom (point) 

think that Maya (point) is thinking about buying? 

3. Picture of a man with a pensive look (kind of like a modern take on the Rodin Le Penseur sculpture from the neck up).

What is the man doing?

(Full credit for something like “thinking”; partial credit for “touching his lip” or other literal description; and zero points 

for “picking his nose”). 

4. Shows a photo of a boy walking on a wall.

Now I am going to act out a rhyme. Watch and do what I do. 

Walking on a wall is fun (examiner “walks” index fingers 3-4 steps across the table).

When we walk it in the sun. We pretend we’re giants tall (examiner holds arms up over head)

When we’re walking on the wall (examiner resumes finger walking).

1 point for 2 or more “steps” and 1 point for both hands over head.

5. Three pictured scenes: a girl swimming on a dolphin, a girl dreaming of hugging a dolphin, and a girl reading a book 

about dolphins.

Ming lives by the ocean (point). Her daddy lets her swim with the dolphins. Sheryl had a dream last night (point). In 

her dream, she was hugging a dolphin. Luz loves to read about dolphins (point). Who can hug a dolphin in real life: 

Ming (point), Sheryl (point), or Luz (point). 

6. Place 2 boxes on table, both of which have a picture of blocks on the lid. One of the boxes, however, has pencils in 

it.

What is in the box? (Typically, child guesses blocks). Let’s see (examiner opens box). Yes, this box is full of blocks. 

What is in this box? (Typically, child guesses blocks). OK, let’s see. (Opens box to reveal pencils). Whoops! It is not 

blocks at all! Let’s put the pencils back.

If your friend came in now, what would your friend think was in this box (i.e., the one with the pencils).

7. No visual on this one.

Brandon has a hard time with spelling. He didn’t do well on his spelling test in school. That afternoon, Mom said, 

“You’ll feel better if you go play with Cameron.” Brandon went to Cameron’s house. Cameron wanted to play Word 

Spell. Brandon decided to go home. Why?
Full credit involves identifying that Brandon didn’t like to spell or that he is poor at it. Partial is more vague (e.g., “He 

couldn’t do it.”)

8. Photo of merry-go-round with Ferris wheel in background.

Reggie and Patrick wanted to ride the Ferris wheel. Audrey and Hannah didn’t want to ride it so they decided to go on 

the merry-go-round. When the boys got to the Ferris wheel, the line was too long, so they went to the fun house instead. 

When Audrey and Hannah were finished at the merry-go-ground, where would they look for the boys, at the fun house 

or the Ferris wheel? 

9. Photo of boy dressed in a suit with a briefcase next to him.

Look at the picture. What is Eric pretending?

Full credit for something that gets at the idea of someone working, partial credit for him being a man or grown-up.

10. No visual on this one. (This is one of the egregious ones in terms of listening comprehension)

Laurie Lamb asked Mother Sheep if she could go play. Mother said, “Yes, dear, but don’t go near the forest. Mr. Wolf 

is hiding there.” Laurie skipped off with her friends to play hide and seek. Laurie ran into the woods to hide behind a 

tree. Just then she saw a big sheep with a white wooly coat, a long nose, and big teeth smiling as it came torward her. 

Mother Sheep was watching. Suddenly she grabbed Laurie and they ran from the woods! Why did Mother Sheep run 

with Laurie?

Credit for identifying the big sheep as the wolf or conveying fear/need for help. 

11. Picture of a teapot made to look like an apple (it looks more like a cross between the two)

What is this?

Credit for recognizing that it’s a teapot. No points for identifying it as an apple.

12. Picture of a boy working on a craft.

Mrs. Russell’s class was making presents for the people at the nursing home. It was almost time for recess. Mrs. 

Russell said, “Class, we’d better wrap it up now.” What did she mean?

Credit for getting that that’s an expression to hurry up or finish up.

13. Photo of two very similar looking girls.

Denise and Emily are sisters. Mama says they are like two peas in a pod. What does that mean?

Credit for them being alike or twins or very close. No credit for identifying them as best friends or sisters or liking to 

do things together.

14. No visual. 

This story is a lot like the story I read before, but listen for something different. Laurie Lamb asked Mother Sheep if she 

could go play. Mother said, “Yes, dear, but don’t near the forest. Mr. Wolf is hiding there.” Laurie skipped off to play 

hide and seek with her friends. Laurie ran into the woods to hide behind a tree. Just then she saw a big sheep with a 

white wooly coat, skinny gray legs, a long nose, and big teeth, smiling as it came toward her. Suddenly, a funnylooking wooly brown bear came roaring into the woods and chased off the big sheep. Laure was so scared that she ran 

all the way home. Grandma Sheep hugged Laure and said, “Mama will be home soon,” and smiled to herself. Just 
then Mother Sheep came up the path out of breath. Her wool was all wet, and it was torn in places. Laurie ran to her 

and said, “Mama. I won’t ever go into the woods again. There is a funny-looking, roaring bear in there.” Mama 

hugged Laurie and laughed. 

Who was the funny-looking wooly bear? Credit for identifying Mama.

Why did Grandmother smile to herself when she hugged Laurie? Full credit for GM knowing that Mama was the bear, 

or that Laurie had been tricked, or that Laurie was safe. Partial credit for vaguer instances of above.

15. Photo of mother and young boy.

Oscar said, “Mom, Uncle Carlos is going to take me for ice cream!” Mother smiled and said, “Oscar, you have Uncle 

Carlos wrapped around your little finger.” What does Oscar’s mother mean?

Full credit for Carlos spoiling Oscar or getting him whatever he wants; partial for them caring for each other or them 

spending a lot of time with each other. 

The remainder of the items are the contextual ones, with the sketched pictures of scenarios and four photos of different 

facial expressions. 

The next items are about Julia. I will show you some drawings of things that happen to Julia. Next to the drawings are 

four photos of Julia’s face. Look carefully at the drawing and point to the picture of Julia’s face that most closely 

shows how she feels in the drawing. Let’s try some.
PROTOCOL FOR DELAY TASK

MATERIALS FOR DELAY TASK:

•

•

•

•

•

•

•

•

•

Parent letter describing task (explaining candy & stickers)

Envelope for letter and delayed rewards 

Paper bag for candy and stickers

Stapler to attach rewards

Tape to seal envelope

Small containers for showing skittles

Large container of skittles

Paper (to stick stickers)

Coding sheet 

IMPLEMENTING DELAY TASK:

•

•

•

•

•

•

This task requires some pre-planning. It is helpful to have materials arranged in 

advance (i.e. stickers pre-cut). It is also helpful to have the desk clear of any 

materials so that the child can focus on the task at hand. 

The tester will first demonstrate an immediate reward situation and a delay 

situation (these situations will remain constant)

Next the tester will offer various delay situations to the child. These situations 

will be presented in random order (study children will have coding sheets with 

varying delay sequences)

If a child chooses a delay condition, the candy/stickers should be removed from 

the table and placed in a location out of the child’s view. If a child chooses not to 

delay, the candy/sticker should be immediately presented and the child should be 

given an opportunity to eat/stick. 

If a child chooses an immediate reward condition, allow a few seconds for the 

child to enjoy and savor the reward. 

Upon completion of this task, all of the delay items should be placed in a small 

paper bag, stapled to the parent letter and placed in a sealed envelope for the 

child’s backpack. 

CODING THE DELAY TASK:

•

•

Circle the number of items that the child selects.

Give examples of the verbal comments made by the child.
SCRIPT FOR DELAY TASK

Demonstration phase

To introduce the new task: 

“Now we are going to play a new game. I brought some fun things with me today. I 

have scratch and sniff stickers and candy for us and you can choose to have some now 

or when you get home”

Now model a delay situation:

“Let me show you how to play. Here is a cup with one candy in it, and here is a cup

with two candies in it. I can have one candy now, or two candies later. I am going to 

choose to have two later.” 

The tester puts the skittle out of sight for “later”. 

Here, model a non-delay situation:

Here are some scratch and sniff stickers. I can choose to have one now or six later. I 

think I will have one now.” 

Take the sticker, scratch it and put it on your shirt. Savor the smell (e.g., “mmm...smells 

like peaches and I love peaches) and smile.

Test Phase

Here, the script will vary according to the random order of the conditions presented. 

“Now it’s your turn. Do you want to have one________ now, or do you want to wait 

until you get home to have (x amount of x items) ?” 

Repeat for all six conditions. 

Pay attention to any motoric or verbal behaviors and document on the coding sheet. Try 

to capture verbal behaviors in writing (under strategies heading) to the extent that it does 

not interrupt the natural flow of the task. 

 

* It is important to remain neutral about the choices children make. Avoid any 

comments that could be perceived as value judgments (e.g. “Good job”). 

Hypotheticals

If the child wants both options, say, “You can only choose one of these.” and gesture 

with hand motions showing both options. An alternate prompt can be “You can only 

make one choice” or “You can have this or this”. 
Demographic Questionnaire- First I will ask you some factual questions about you and your child

Items Responses 

1. What is your relationship with the child? 1. Mother 2. Father 3. Grandmother 4. Aunt 5. Other__________________

Gender: 

2. Are you the primary caregiver? 1. Yes 2. No

If yes, how long have you been? _____________ If no, who is? _____________

3. What is your date of birth? __ __/ __ __/ __ __

Month Day Year

4. What is your ethnicity or race?

1. Black/ Afri. 

Amer

2. Native Amer/ 

Indian 3. White/ Caucasian

4. Hispanic/ 

Spanish/Latino 5. Asian 6. Other 7. Multi-racial

5. What is the child's mother's age, if you are not the mother? ___________ years __________________ months

6. What was the estimated birth weight of this child? ______Pounds ______Ounces

7a. Last year, did the child attend preschool or head start? 1. Yes 2. No

7b. If yes, how many hours per week did he/she attend? Hr/wks:________

8. How many times has this child moved in the last two years? _______________________

9. Has this child been referred for special education services? 1. Yes 2. No

9b. If answer is yes, what was diagnosis/identified disability:__________________________

10. What language does this child speak at home? 1. English 2. Spanish 3. Other

If no, what did the child do in the year before 

K'garten?________________________________
11. What is the highest level of education of this child's mother? 

12. What is the highest level of education of this child's primary caregiver (if mother is not primary caregiver)? 

13. Including the study child, who are the other children who live in the child's home?

Name Relationship to Child Age Gender Education Level (Grade 

Only)
14. Including yourself, who are all of the adults who live in the child's home?

Gender Employment 

Pay Rate

Education level (Primary 

Caregiver Only)

Employment status (Primary 

Caregiver Only)

Employment Schedule (Hours 

Per Week

Relationship to 

Child

Name Age
15. (COHORT 1 Only) What time did your child go to bed on a school night during the school year?______________________

16. (COHORT 1 Only) Last year during the school year, where did your child go after the end of the school day until bedtime?

(if situation changed by day or mid-year, describe below:)

6. Other: ____________________________

5. Babysitter

After School Location

1. At home with mother/primary caregiver

Hours per day

4. After school program 

(Name:__________________)

2. With other family member or friend 

3. Day care 
17. (COHORT 1 Only) Last school year, how often did your child participate in any of these activities after the end of the school day?

Homework alone

Homework with adult

Homework with older sibling or older peer

watch tv

play videogames

play outside

play inside

reading

nap

Other:

Don't know

(if situation changed by day or mid-year, describe below:)

1 hour 2 hours 3 or more 

hours

Afterschool program

After School Activities Never Half-hour
Family Interview – Qualitative FALL 2014 

WINGS Introduction: Thank you for taking the time to talk with me today. [Introduce yourself, a 

little about you and what you’re doing with the project.] We appreciate your willingness to spend 

some time with us. We are interested in learning more about your family and your and your 

child’s experiences with WINGS. We are conducting these interviews as part of the study of the 

WINGS program and the lives of children in the program. This interview should take about an 

hour. You have the right to stop the interview at any time and you can also feel free to skip any 

question that makes you uncomfortable, just tell me you would like to move on. I will be asking 

you some general questions about your family and its routines, as well as some questions about 

your child, your neighborhood, your child’s school, and your child’s out-of-school experiences. 

Do you have any questions for me before you start? 

CONTROL Introduction: Thank you for taking the time to talk with me today. [Introduce 

yourself, a little about you and what you’re doing with the project.] We appreciate your 

willingness to spend some time with us. We are interested in learning more about your family 

and your child’s transition to kindergarten and what s/he does after-school. We are conducting 

these interviews as part of the program evaluation that we have been talking to you about. This 

interview should take about an hour. You have the right to stop the interview at any time and you 

can also feel free to skip any question that makes you uncomfortable, just tell me you would like 

to move on. I will be asking you some general questions about your family and its routines, as 

well as some questions about your child, your neighborhood, your child’s school, and your 

child’s out-of-school experiences. Do you have any questions for me before you start? 

EVERYONE: We would like to record the interview. We will delete the recording after we 

transcribe it and your name will be replaced with a code number to protect your privacy. Is it ok 

for me to turn on the recorder now? 

Ok – I want to first just get some updated contact information. 

1. Has your address changed? If yes, get new address

a. If yes, ask the following questions about the move: 

i. When did you move? 

ii. What led you to move?

iii. What factors did you consider in deciding where to move? 

iv. Has this move affected your child’s school or after-school activities at all? 

If yes, how? 

v. IF CHILD IS NO LONGER IN WINGS BECAUSE OF MOVE: How did 

you feel about moving to a school that did not offer WINGS?

vi. How satisfied are you with your new home? 

b. Can we get contact information for 2-3 other people who would know how to get 

in touch with you if you changed your phone number or your address? We want 

this information so that we can follow-up with you later in the study in case you 

move or change your number. 

Thanks, now I’d like to ask you some questions about your family’s routines. 
1. Who does the child spend time with every week when they are not in school? [Note to 

Interviewer, if they start naming a lot of the kids’ friends you can have them just 

summarize with the total number of friends their age they see on a weekly basis and how 

many are boys vs. girls. We are not asking specific ages, just whether it is an adult or a 

child – over or under age 18]

[USE SEPARATE TABLE OF PEOPLE IN CHILD’S LIFE TO FILL IN WITH 

PARTICIPANT]

Name Relationship 

to child

Over or 

Under 18? 

Gender How often 

interacts 

with child

2. Are there other people in the child’s life who they don’t spend time with on a weekly 

basis but are still important who are not on the previous table? 

[USE SEPARATE TABLE OF PEOPLE IN CHILD’S LIFE TO FILL IN WITH 

PARTICIPANT]

Name Relationship 

to child

Over or 

under 18?

Gender How often 

interacts with 

child

3. Are there any other adults who contribute to the household either financially or through 

material or childcare support [reword if needed to: are there any other adults who provide 

the people in your house with money, childcare, or items to help out the family?]

a. If yes, in what ways do they contribute? 

4. Before we do the timeline about your family routine, which you might remember from 

last year, I wanted to just get your thoughts on how, if at all, your family life or routines 

have changed in the last year.

a. If they describe any changes – why do you think those things have changed? 

b. Thinking back on when your child transitioned to Kindergarten, how would you 

compare your family life and routines now to during that transition time? 

Now I want to ask you some more detailed questions about your family’s typical activities. [USE 

SEPARATE TIMELINE DOCUMENT TO FILL IN WITH PARTICIPANT]

5. Tell me about your day yesterday [or Friday if interview is on a Monday] from when 

your child got home from school (or after-school) to when he or she went to bed. Using 

these different colors for different categories, show me what [study child] is doing during 
that time, who they are with during each activity, and where they are for each activity 

(provide timeline to fill in).

a. Activities

i. Homework 

ii. Household chores 

iii. Meals 

iv. Personal hygiene (e.g., bathing, brushing teeth, doing hair) 

v. Playing video games

vi. Playing board games 

vii. Playing recreational games (physical activity) inside

viii. Playing recreational games (physical activity) outside

ix. Reading 

x. Taking a trip/going to _____[fill in the blank] 

xi. Talking (what about?) 

xii. Talking on the phone 

xiii. Television watching 

xiv. Using the computer 

b. People 

c. Places

a. Rooms in the home 

b. Places outside the home 

d. Was yesterday a typical day? If no – what would be different on a typical day in 

your home? 

6. Now we are going to do the same thing from when your child got up this morning to 

when he or she went to school. Using these different colors for different categories, show 

me what [study child] is doing during that time and who they are with during each 

activity (provide timelines to fill in)

a. Activities

i. Homework 

ii. Household chores 

iii. Meals 

iv. Personal hygiene (e.g., bathing, brushing teeth, doing hair) 

v. Playing video games

vi. Playing board games 

vii. Playing recreational games (physical activity) inside

viii. Playing recreational games (physical activity) outside

ix. Reading 

x. Taking a trip/going to _____[fill in the blank] 

xi. Talking (what about?) 

xii. Talking on the phone 

xiii. Television watching 

xiv. Using the computer 

b. People 
c. Places

a. Rooms in the home 

b. Places outside the home 

c. Places outside the home 

d. Was today a typical day? If no – what would be different on a typical day in your 

home? 

7. What kinds of things does your family do on the weekend? 

8. In general, what time does [study child] get up on the weekends? 

9. In general, what time does s/he go to bed? 

10. Are there any regular activities that [study child] does every weekend?

Are there any activities that you do with your child every now and then that are not included on 

any of these timelines? [e.g., library, museum, zoo, family reunion, visiting family or friends, 

etc.]

Thank you. Now I want to ask you a few general questions about your family. [NOTE: if the 

person being interviewed is not the child’s family, questions should be re-worded to ask about 

the study child’s family and home] 

11. Tell me a bit about your family. 

12. What are the biggest joys in your family? 

13. All families have challenges, what are the biggest challenges faced by your family? [you 

can use the word “struggle” instead of challenges if interviewee doesn’t respond to 

challenges] 

a. How do you encourage [study child] when s/he or the family are struggling with 

something? 

i. Prompt for follow-up if they can’t think of a challenge: Doesn’t need to be 

a big challenge, could be something small. 

ii. If haven’t had challenges – if you had a situation where child/family was 

struggling, how do you think you’d talk about it with your child? 

14. Has your family had any major events happen within the last year (e.g., death or birth, 

moving, loss of a job, gain of a job, marriage, divorce, etc.)? 

a. How did [study child] respond to that? 

b. Was that something that you had to talk to [study child] about? If yes, how did 

you talk about it? 

c. Sometimes things happen outside our family that are difficult for us to talk about 

with children but that kids hear about from other kids, on tv or the internet, or see 

in the neighborhood. Some families talk about these kinds of things with their 

kids and other families don’t. Have you ever had a conversation about something 

like that with your child? 

i. If yes, how did you approach that?

ii. If no, do you think you ever would? How would you approach that? 

15. What is your relationship with [the study child]? 

a. Do you live with child? 

b. Does child stay with you after school? 
c. How would you describe your relationship with [the study child]? [possible rewording or prompt: What is your relationship like with [study child]?]

d. How would you describe [study child]? [possible re-wording or prompt: Tell me 

a bit about [the study child].] 

i. What are [the study child]’s strengths?

ii. What are [the study child]’s challenges?

e. In some families, people talk to each other a lot and in other families people don’t 

talk that much. What’s the norm for your family? 

i. If you were spending time with your child, say riding the bus, doing 

errands, eating a meal, or taking a walk, what might you talk about, if 

anything?

Now we are going to talk a little bit about your neighborhood and [the study child’s] school. 

16. Tell me about your neighborhood. 

f. What things do you like about your neighborhood? 

g. What things do you dislike about your neighborhood, if anything? 

h. What kinds of things are there for kids to do in your neighborhood (for example, 

parks, after-school centers, programs run by churches, etc)? 

i. As we talked about earlier, all families sometimes face challenges, what kinds of 

challenges/struggles are faced by families in your neighborhood? 

17. Tell me about your child’s school. 

a. What things do you like about your child’s school? 

b. What things do you dislike about your child’s school? 

c. What does your child like about the school? 

d. What does your child dislike about the school? 

e. IF CHILD HAS CHANGED SCHOOLS: How was the transition to the new 

school for your child? 

f. How satisfied are you with the education your child gets at his/her school? 

j. How would you describe the overall feeling (vibe or atmosphere) at the school to 

a new family? 

k. How would you describe your child’s friends at school? 

i. At some schools parents tend to know each other and at other schools 

parents don’t really know each other very much, how well do you know 

the parents of the other kids at school, if at all? 

l. What is your relationship with the school like?

i. How often are you at the school? [try to get specific, e.g. weekly daily, 

monthly, versus “a lot”] 

ii. What are you usually doing when you are at the school? 

iii. Who do you interact with or talk to when you come to the school? 

m. Who is your child’s main teacher? 

i. Some teachers talk to parents a lot and others don’t, how often do you talk 

to your kid’s main teacher(s)?

ii. How would you describe your child’s main teacher?

iii. Some parents feel like they have good relationships with their kids’ 

teachers and other parents don’t feel like they have very good 
relationships with their kids’ teachers. How would you describe your 

relationship with your kid’s teacher(s)? 

iv. How would you describe your child’s relationship with his/her teacher?

n. Who is your child’s other teacher? 

i. How often do you talk to your kid’s other teacher(s)? 

ii. How would you describe your child’s other teacher? 

iii. How would you describe your relationship with your kid’s other 

teacher(s)?

iv. How would you describe your child’s relationship with his/her other 

teacher?

o. Some principals have active relationships with and talk to parents a lot and other 

principals don’t talk to parents very much, how often do you talk to your child’s 

principal? 

i. If they report talking to principal at least sometimes: how would you 

describe your relationship with your kid’s principal? 

p. [If they named anyone else they interact with or talk to in iii ask the following 

two questions about that person/those people]:

i. How often do you talk to X? 

ii. How would you describe your relationship with X? 

18. People have very different feelings about and experiences with school. How did you feel 

about school when you were a kid? 

a. How interested were you in school back then? 

b. What things do you wish had been different about your own schooling? 

c. How would you compare your experiences in school as a kid with your child’s 

experiences in school now? 

d. How do you feel about education now? 

i. If that is different from how you used to feel, what changed? 

e. What was your last/most recent experience with school?

i. If parent talks about being in school now or starting school soon or 

having gone back to school earlier, ask about that timeline: It sounds like 

you have had the chance to start and stop school at different points, can 

you tell me a bit about when in your life you have stopped and started 

school? 

19. How important is school to you? 

20. How does your child feel about school? 

a. How interested is your child in school? 

b. Has that changed at all over time [for later interviews, not Time 1] 

c. What concerns do you have about your child in school? 

21. How far do you hope your child will go in school? 

22. How far do expect him/her to go in school? 

a. If s/he doesn’t get to [their response] would you be disappointed? Why/why not? 

23. In what ways do you influence your child’s academic achievement? 

a. Looking ahead, what influence do you think you will have on your child’s 

academic achievement as s/he gets older?

b. Who else do you think influences your child’s academic achievement? How? 

24. How much do you think school is related to being successful in life? 
Now I’d like to ask you some questions about the after-school arrangements that you have for 

[study child] and your other children [if applicable]. [Ask the following questions first about the 

study child and then about every other child listed on demographic sheet as living in the same 

house as the study child. If a child has more than one after school arrangement, ask the series of 

questions about each place that the child spends time after school.]

FOR WINGS FAMILIES ONLY [control group skip to Question 37]

25. What would [study child] do after-school if s/he did not go to WINGS?

iii. About how much would that cost you?

26. On a scale of 1 to 4, with 1 being very unhappy and 4 being very happy, how happy are 

you with the WINGS program?

1 – very unhappy 2 – unhappy 3 – happy 4 – very happy

27. What do you like best about WINGS?

28. Since part of the goal of this research is to help improve WINGS, we are also interested 

in what, if anything, you don’t like about WINGS.

29. Does your child talk to you about what they do in WINGS? If yes, what do they tell you?

a. What do you think your child likes best about WINGS?

b. What do you think your child likes least about WINGS?

c. Does your child use any WINGS language at home (e.g., The Creed)?

d. What, if anything, do you think your child gets out of WINGS?

e. Does your child behave the same or differently at home and at WINGS? If 

differently, how so? 

f. How does your child feel about his or her friends in WINGS?

30. Have you noticed any changes/differences in your child since being in WINGS? 

a. If yes: How?

b. If yes: Do you think these changes are because of WINGS? Why or Why not?

31. What do you think your child learns in WINGS?

a. What do you think the goals of WINGS are?

b. What do you want your child to get out of WINGS?

c. Does your child do his/her homework in WINGS?

i. How happy are you with that arrangement?

32. How often do you interact with or talk to WINGS staff, if at all? 

a. If interact or talk to them at all: How would you describe your relationship with 

WINGS staff? What do you talk to them about?

b. Do you feel that’s too little, just the right amount, or too much?

c. How happy are you with your relationship with the WINGS staff?

d. How would you describe your child’s relationship with their WINGS leader?

e. How would you describe your child’s relationship with other WINGS staff?

33. The transition to Kindergarten can be difficult for children and families, thinking back, 

how was that transition for you and your child?

a. How did WINGS affect your child’s or your adjustment to Kindergarten, if at all?

34. Looking back on your child’s experience in WINGS last year (and the year before if 

relevant), how has their experiences changed over the years, if at all?
35. IF RELEVANT: How would you compare your older child’s experience to your younger 

child’s experience in WINGS?

36. How do you think your and your family’s life would be different without WINGS, if at 

all?

FOR FAMILIES OF KIDS NOT IN WINGS [including non-study kids who don’t attend 

WINGS but are in WINGS families]

37. What does [study child] do after school?

a. [if not at home] Where is that located? 

b. [if more than one location] How much time does [study child] spend at each 

location? [if they can’t say then ask “where does [study child] spend the most 

amount of time after school?]

38. On a scale of 1 to 4, with 1 being very unhappy and 4 being very happy, how happy are 

you with the after-school care that [study child] receives??

1 – very unhappy 2 – unhappy 3 – happy 4 – very happy

39. Does your child talk to you about what they do after school? If yes, what do they talk to 

you about?

a. What do you think your child likes best about what s/he does after school?

b. What do you think your child likes least about what s/he does after school?

c. Does your child do his/her homework in this after school setting?

i. How happy are you with this arrangement?

40. How much does this cost you each week? [possible re-wording: how much do you pay 

for [study child’s] after school care each week?]

a. If they don’t know weekly give option to answer for whole year

IF CHILD IS CARED FOR AT HOME SKIP TO QUESTION 42

IF CHILD IS CARED FOR AT MULTIPLE LOCATIONS OVER A WEEK ASK 43 AND THEN 

ASK QUESTIONS 41-48 ONLY FOR THE PLACE THAT THE CHILD SPENDS A MAJORITY 

OF TIME.

41. How does [study child] get to that location? 

42. How does [study child] get home from that location?

43. How much time does [study child] spend there each week?

44. How would you describe the place?

45. [If the child is in home care at someone else’s home] Can you tell me about the 

neighborhood where the home is?

46. Who are the adults who are with [study child] after school? [Make sure to get the total 

number of adults even if they don’t know all the names] 

a. Total # of adults: ______

b. [if it is not the interviewee] How well, if at all, do you know that person?

c. What is your relationship like with [that person/people]?

d. How would you describe your child’s relationship with [that person]?
47. Who are the other kids who are with [study child] after school? [Make sure to get the 

total number of kids even if they don’t know all the names] 

e. Total # of kids: ______

f. How well, if at all, do you know the parents of the other kids?

g. How does your child feel about his/her friends in the after school setting?

48. What does [study child] usually do there (e.g., types of activities), if you know? It’s ok if 

you don’t know.

a. What kinds of things are available for your child to use there (if they ask what we 

mean or for examples can say: e.g., toys, games, outdoors equipment, art supplies, 

books, videos, etc)

49. The transition to Kindergarten can be difficult for children and families, thinking back, 

how was that transition for you and your child?

a. How did your after-school arrangements for your child affect your child’s or your 

adjustment to Kindergarten, if at all?

50. Looking back on your child’s after-school experiences last year (and the year before if 

relevant), how has their experiences changed over the years, if at all?

51. What other activities and programs are available in your community for kids to do afterschool? 

a. School-based options

b. Community-based options, for example, music or dance lessons, sports, 

cheerleading, church groups, Boys & Girls Clubs, Boy Scouts or Girl Scouts, 4-H, 

etc. 

c. There are lots of reasons why parents don’t choose to use certain programs, 

sometimes they don’t like the program, sometimes the program is too expensive, 

sometimes the location isn’t convenient, and lots of other reasons. What made you 

decide not to use those programs or activities? 

FOR KIDS WHO USED TO BE IN WINGS BUT ARE NOT ANYMORE – ASK Q52-61 

AND THEN ASK THE FOLLOWING: 

I know that [study child] used to attend WINGS but does not any more. I’d like to ask you a few 

questions about your experiences with WINGS when s/he was in it and your decision to leave.

52. Why did [study child] stop attending WINGS?

a. What made you choose [current after school arrangements] over WINGS?

53. On a scale of 1 to 4, with 1 being very unhappy and 4 being very happy, how happy were 

you with the WINGS program when s/he was in it?

1 – very unhappy 2 – unhappy 3 – happy 4 – very happy

54. What did you like best about WINGS?

55. What did you like least about WINGS?

56. Did your child talk to you about what they did in WINGS? If yes, what did they tell you?

a. What do you think your child liked best about WINGS?

b. What do you think your child liked least about WINGS?

c. What, if anything, do you think your child got out of WINGS?
d. How did your child feel about the other kids in WINGS?

57. What do you think your child learned in WINGS?

a. What do you think the goals of WINGS are?

b. What did you want your child to get out of WINGS?

58. How often did you interact with or talk to WINGS staff, if at all? 

a. If interact or talk to them at all: How would you describe your relationship with 

WINGS staff? What did you talk to them about?

b. Do you feel that’s too little, just the right amount, or too much?

c. How would you describe your child’s relationship with their WINGS leader?

d. How would you describe your child’s relationship with other WINGS staff?

59. The transition to Kindergarten can be difficult for children and families, thinking back, 

how was that transition for you and your child?

a. How did WINGS affect your child’s or your adjustment to Kindergarten, if at all?

60. How would you compare your child’s experiences in their current after-school 

arrangements to their experience in WINGS?

61. Would you ever go back to WINGS?

a. Why/why not?

FOR EVERYONE

62. What does [study child] do during the summer? 

63. How satisfied are you with what [study child] does over the summer?

64. About how much does this cost you?

Repeat questions 62-64 for every child who lives in home with study child

65. What other activities, programs, and summer camps are available in your community for 

kids to do during the summer? 

a. School-based options

b. Community-based options

c. There are lots of reasons why parents don’t choose to use certain programs, 

sometimes they don’t like the program, sometimes the program is too expensive, 

sometimes the location isn’t convenient, and lots of other reasons. What made you 

decide not to use those programs or activities? 

66. Is there anything that you feel that you spend a lot of time on with your child or thinking 

about in regards to your child that we haven’t asked about?
Contact Sheet

Address Change (if applicable): 

Secondary Contact Information 

Name Phone # Relationship to family
Appendix B

Tables and Figures for Two-Year Results
Table 7.1 Teacher Assessment of Positive Behavior (ITT)

Teacher Positive ITT Listwise Data Estimates Imputed Data Estimates 

Bucket Measure Null Pretest Covariates Null Pretest Covariates 

SEL 

Competence 

- Teacher 

DESSA

Self 

Awareness

beta 0.13 0.12 0.15 0.17 0.14 0.14

se 0.16 0.15 0.16 0.15 0.14 0.14

p 0.42 0.42 0.33 0.24 0.31 0.3

Social 

Awareness

beta 0.09 0.04 0.06 0.15 0.1 0.11

se 0.16 0.15 0.16 0.15 0.14 0.14

p 0.59 0.79 0.71 0.32 0.5 0.44

Self 

Management

beta 0.21+ 0.13 0.11 0.25++ 0.19+ 0.2+ 

se 0.16 0.15 0.15 0.15 0.14 0.14

p 0.19 0.39 0.46 0.09 0.19 0.16

Decision 

Making

beta 0.15 0.14 0.14 0.19+ 0.15 0.17+ 

se 0.16 0.15 0.15 0.14 0.14 0.13

p 0.34 0.37 0.35 0.19 0.28 0.19

Relationship 

Skills

beta 0.11 0.1 0.06 0.14 0.1 0.1

se 0.16 0.15 0.15 0.15 0.14 0.13

p 0.49 0.51 0.7 0.34 0.46 0.46

Classroom 

Relationships 

and 

Behaviors

STRS - 

Closeness

beta 0.18 0.26+ 0.28++ 0.19 0.2 0.17

se 0.16 0.16 0.17 0.16 0.16 0.16

p 0.27 0.11 0.1 0.24 0.22 0.29

T SSIS - 

Self Control

beta 0.08 0.1 0.07 0.08

se 0.16 0.17 0.15 0.15

p 0.61 0.55 0.64 0.59

T SSIS - 

Engagement

beta 0.02 0.02 0 -0.03

se 0.16 0.16 0.16 0.15

p 0.92 0.88 0.98 0.86

T CBRS - 

Self 

Regulation

beta 0.2 0.2 0.19 0.17

se 0.16 0.16 0.15 0.15

p 0.21 0.23 0.22 0.24

Figure 7.1 Teacher Assessment of Positive Behavior (ITT)
Table 7.2 Teacher Assessment of Positive Behavior (TOT)

Teacher Positive TOT Listwise Data Estimates Imputed Data Estimates 

Bucket Measure Null Pretest Covariates Null Pretest Covariates 

SEL 

Competence 

- Teacher 

DESSA

Self 

Awareness

beta 0.31 0.29 0.35 0.46 0.37 0.37

se 0.38 0.36 0.35 0.39 0.37 0.35

p 0.41 0.41 0.32 0.24 0.31 0.29

Social 

Awareness

beta 0.21 0.1 0.14 0.4 0.26 0.29

se 0.39 0.36 0.36 0.4 0.38 0.37

p 0.59 0.79 0.71 0.32 0.5 0.44

Self 

Management

beta 0.5+ 0.3 0.26 0.67++ 0.5+ 0.52+ 

se 0.39 0.35 0.35 0.4 0.38 0.37

p 0.2 0.39 0.46 0.1 0.19 0.17

Decision 

Making

beta 0.37 0.32 0.32 0.5+ 0.39 0.44+ 

se 0.39 0.36 0.34 0.38 0.36 0.34

p 0.34 0.37 0.35 0.19 0.28 0.2

Relationship 

Skills

beta 0.27 0.24 0.13 0.37 0.27 0.25

se 0.38 0.36 0.34 0.38 0.36 0.34

p 0.49 0.5 0.7 0.34 0.46 0.46

Classroom

Relationships 

and 

Behaviors

STRS - 

Closeness

beta 0.43 0.61+ 0.65+ 0.5 0.52 0.44

se 0.39 0.39 0.4 0.43 0.43 0.42

p 0.28 0.12 0.1 0.24 0.23 0.29

T SSIS - 

Self Control

beta 0.2 0.24 0.19 0.21

se 0.39 0.4 0.4 0.4

p 0.61 0.55 0.64 0.59

T SSIS - 

Engagement

beta 0.04 0.06 -0.01 -0.07

se 0.39 0.38 0.41 0.4

p 0.92 0.88 0.98 0.86

T CBRS - 

Self 

Regulation

beta 0.48 0.46 0.49 0.45

se 0.39 0.38 0.41 0.39

p 0.22 0.23 0.23 0.25

Figure 7.2 Teacher Assessment of Positive Behavior (TOT)
Table 7.3 Teacher Assessment of Negative Behavior (ITT)

Teacher Negative ITT Listwise Data Estimates Imputed Data Estimates 

Bucket Measure Null Pretest Covariates Null Pretest Covariates 

Classroom 

Relationships 

and 

Behaviors

STRS - 

Conflict

beta -0.13 -0.16 -0.22+ -0.17 -0.21+ -0.24++ 

se 0.16 0.15 0.16 0.15 0.14 0.14

p 0.4 0.29 0.17 0.25 0.14 0.09

T SSIS - 

Internalizing

beta -0.1 -0.14 -0.11 -0.12

se 0.16 0.17 0.16 0.15

p 0.52 0.4 0.46 0.42

T SSIS - 

Externalizing

beta -0.23+ -0.27+ -0.21+ -0.23+ 

se 0.16 0.17 0.16 0.16

p 0.15 0.11 0.19 0.15

T SSIS - 

Bullying

beta -0.3++ -0.32++ -

0.26++ -0.26++ 

se 0.16 0.17 0.15 0.16

p 0.06 0.06 0.09 0.1

T SSIS - 

Hyperactivity

beta -

0.28++ -0.33++ -0.27++ -0.3++ 

se 0.16 0.17 0.15 0.15

p 0.08 0.05 0.08 0.05

T SSIS - 

Problem 

Behaviors

beta -

0.27++ -0.31++ -0.25+ -0.27++ 

se 0.16 0.17 0.15 0.16

p 0.1 0.07 0.11 0.09

Figure 7.3 Teacher Assessment of Negative Behavior (ITT)
Table 7.4 Teacher Assessment of Negative Behavior (TOT)

Teacher Negative TOT Listwise Data Estimates Imputed Data Estimates 

Bucket Measure Null Pretest Covariates Null Pretest Covariates 

Classroom 

Relationships 

and 

Behaviors

STRS - 

Conflict

beta -0.32 -0.37 -0.49+ -0.46 -0.55+ -0.63+ 

se 0.39 0.35 0.37 0.4 0.38 0.38

p 0.41 0.3 0.18 0.26 0.15 0.1

T SSIS - 

Internalizing

beta -0.25 -0.33 -0.3 -0.32

se 0.39 0.4 0.41 0.4

p 0.52 0.41 0.47 0.43

T SSIS - 

Externalizing

beta -0.56+ -0.63+ -0.54+ -0.59+ 

se 0.39 0.41 0.42 0.41

p 0.16 0.13 0.2 0.16

T SSIS - 

Bullying

beta -

0.74++ -0.74++ -0.68+ -0.68+ 

se 0.4 0.41 0.41 0.42

p 0.07 0.07 0.1 0.1

T SSIS - 

Hyperactivity

beta -

0.67++ -0.76++ -0.71++ -0.77++ 

se 0.4 0.41 0.41 0.41

p 0.09 0.07 0.09 0.06

T SSIS - 

Problem 

Behaviors

beta -0.65+ -0.71++ -0.65+ -0.69+ 

se 0.4 0.41 0.42 0.42

p 0.11 0.09 0.12 0.1

Figure 7.4 Teacher Assessment of Negative Behavior (TOT)
Table 7.5 Parent Assessment of Positive Behavior (ITT)

Parent Positive ITT Listwise Data Estimates Imputed Data Estimates 

Bucket Measure Null Pretest Covariates Null Pretest Covariates 

SEL 

Competence 

- Parent 

DESSA

Self 

Awareness

beta -0.05 -0.11 -0.23+ -0.13 -0.15 -0.19

se 0.16 0.15 0.16 0.15 0.15 0.17

p 0.75 0.48 0.14 0.4 0.3 0.27

Social 

Awareness

beta 0.14 -0.03 -0.13 0.07 -0.01 -0.09

se 0.16 0.15 0.15 0.16 0.16 0.18

p 0.37 0.84 0.39 0.69 0.93 0.62

Self 

Management

beta 0.09 0.01 -0.09 0.01 0.01 0.03

se 0.16 0.14 0.15 0.16 0.15 0.18

p 0.56 0.93 0.55 0.97 0.97 0.86

Decision 

Making

beta -0.03 -0.09 -0.13 -0.1 -0.1 -0.09

se 0.16 0.14 0.15 0.16 0.16 0.17

p 0.84 0.55 0.38 0.53 0.54 0.58

Relationship 

Skills

beta -0.01 -0.15 -0.23+ -0.06 -0.11 -0.13

se 0.16 0.15 0.15 0.16 0.15 0.18

p 0.95 0.32 0.13 0.69 0.5 0.47

Home 

Relationships 

and 

Behaviors

CPRS - 

Closeness

beta 0.21+ 0.18 0.25+ 0.15 0.17 0.18

se 0.16 0.15 0.15 0.15 0.14 0.15

p 0.19 0.25 0.11 0.3 0.22 0.24

Figure 7.5 Parent Assessment of Positive Behavior (ITT)
Table 7.6 Parent Assessment of Positive Behavior (TOT)

Parent Positive TOT Listwise Data Estimates Imputed Data Estimates 

Bucket Measure Null Pretest Covariates Null Pretest Covariates 

SEL 

Competence 

- Parent 

DESSA

Self 

Awareness

beta -0.12 -0.24 -0.47+ -0.34 -0.41 -0.51

se 0.37 0.34 0.33 0.41 0.4 0.47

p 0.75 0.48 0.16 0.41 0.31 0.28

Social 

Awareness

beta 0.32 -0.07 -0.27 0.17 -0.03 -0.24

se 0.37 0.32 0.31 0.43 0.41 0.48

p 0.38 0.84 0.4 0.69 0.94 0.62

Self 

Management

beta 0.21 0.03 -0.18 0.02 0.01 0.08

se 0.37 0.32 0.3 0.41 0.39 0.47

p 0.57 0.93 0.56 0.97 0.97 0.86

Decision 

Making

beta -0.07 -0.19 -0.27 -0.27 -0.25 -0.25

se 0.36 0.32 0.31 0.44 0.41 0.45

p 0.84 0.55 0.38 0.53 0.54 0.59

Relationship 

Skills

beta -0.02 -0.32 -0.46+ -0.17 -0.28 -0.34

se 0.37 0.32 0.31 0.43 0.41 0.47

p 0.95 0.32 0.14 0.69 0.5 0.47

Home 

Relationships 

and 

Behaviors

CPRS - 

Closeness

beta 0.48+ 0.4 0.52+ 0.4 0.45 0.47

se 0.37 0.35 0.33 0.39 0.37 0.4

p 0.2 0.25 0.11 0.3 0.22 0.25

Figure 7.6 Parent Assessment of Positive Behavior (TOT)
Table 7.7 Parent Assessment of Negative Behavior (ITT)

Parent Negative ITT Listwise Data Estimates Imputed Data Estimates 

Bucket Measure Null Pretest Covariates Null Pretest Covariates 

Home 

Relationships 

and 

Behaviors

CPRS - 

Conflict

beta -0.01 -0.02 -0.01 0.01 0.01 -0.01

se 0.16 0.14 0.14 0.16 0.14 0.17

p 0.94 0.87 0.93 0.97 0.97 0.94

P SSIS - 

Internalizing

beta -0.09 -0.1 0 -0.07 -0.15 -0.06

se 0.16 0.14 0.14 0.15 0.14 0.18

p 0.57 0.45 0.98 0.67 0.27 0.72

P SSIS - 

Externalizing

beta -0.04 -0.07 -0.05 -0.09 -0.12 -0.14

se 0.16 0.13 0.13 0.14 0.12 0.17

p 0.79 0.6 0.68 0.51 0.32 0.4

P SSIS - 

Bullying

beta 0.06 0.02 -0.03 -0.02 -0.04 0 

se 0.16 0.16 0.16 0.15 0.15 0.19

p 0.71 0.9 0.83 0.89 0.8 0.99

P SSIS - 

Hyperactivity

beta -0.17 -0.15 -0.14 -0.18 -0.16 -0.2

se 0.16 0.13 0.13 0.14 0.13 0.17

p 0.29 0.27 0.32 0.21 0.22 0.24

P SSIS - 

Problem 

Behaviors

beta -0.07 -0.08 -0.07 -0.11 -0.14 -0.14

se 0.16 0.13 0.13 0.14 0.13 0.17

p 0.66 0.53 0.59 0.45 0.25 0.43

Figure 7.7 Parent Assessment of Negative Behavior (ITT)
Table 7.8 Parent Assessment of Negative Behavior (TOT)

Parent Negative TOT Listwise Data Estimates Imputed Data Estimates 

Bucket Measure Null Pretest Covariates Null Pretest Covariates 

Home 

Relationships 

and 

Behaviors

CPRS - 

Conflict

beta -0.03 -0.05 -0.03 0.02 0.01 -0.03

se 0.37 0.32 0.28 0.42 0.37 0.44

p 0.94 0.87 0.93 0.97 0.97 0.94

P SSIS - 

Internalizing

beta -0.23 -0.23 -0.01 -0.4 -0.4 -0.17

se 0.31 0.31 0.29 0.37 0.37 0.47

p 0.45 0.45 0.98 0.27 0.27 0.72

P SSIS - 

Externalizing

beta -0.15 -0.15 -0.11 -0.32 -0.32 -0.37

se 0.29 0.29 0.27 0.33 0.33 0.44

p 0.6 0.6 0.68 0.32 0.32 0.41

P SSIS - 

Bullying

beta 0.05 0.05 -0.07 -0.1 -0.1 0.01

se 0.37 0.37 0.35 0.4 0.4 0.5

p 0.9 0.9 0.83 0.8 0.8 0.99

P SSIS - 

Hyperactivity

beta -0.33 -0.33 -0.28 -0.41 -0.41 -0.53

se 0.3 0.3 0.28 0.34 0.34 0.46

p 0.27 0.27 0.31 0.22 0.22 0.25

P SSIS - 

Problem 

Behaviors

beta -0.16 -0.19 -0.15 -0.28 -0.38 -0.36

se 0.37 0.3 0.28 0.38 0.33 0.45

p 0.66 0.53 0.59 0.45 0.26 0.43

Figure 7.8 Parent Assessment of Negative Behavior (TOT)
Table 7.9 Building Block Skills from Direct Child Measures (ITT)

Building Blocks ITT Listwise Data Estimates Imputed Data Estimates 

Bucket Measure Null Pretest Covariates Null Pretest Covariates 

Building 

Blocks

Choice 

Delay

beta -0.03 -0.03 -0.13 -0.1 -0.1 -0.11

se 0.16 0.16 0.17 0.16 0.16 0.15

p 0.84 0.84 0.42 0.53 0.53 0.46

DAS 

Naming 

Vocab

beta 0.25+ 0.23++ 0.32* 0.25+ 0.26++ 0.26* 

se 0.15 0.13 0.14 0.15 0.14 0.12

p 0.11 0.08 0.02 0.1 0.06 0.04

DAS Recall 

of Seq 

Order

beta 0.05 0.04 -0.1 0.03 0.06 0 

se 0.16 0.15 0.16 0.15 0.15 0.15

p 0.77 0.79 0.53 0.85 0.67 1 

DAS Verbal 

Comp

beta -0.09 -0.04 0.01 -0.09 -0.03 -0.05

se 0.16 0.15 0.17 0.15 0.15 0.15

p 0.57 0.81 0.96 0.56 0.82 0.76

EMT-ACES

beta -0.04 0.04 0 -0.03 0.03 0.07

se 0.16 0.14 0.15 0.16 0.15 0.15

p 0.78 0.76 0.99 0.87 0.84 0.67

HTKS

beta 0.21+ 0.26++ 0.3++ 0.21+ 0.27++ 0.26++ 

se 0.15 0.15 0.16 0.16 0.15 0.15

p 0.19 0.07 0.07 0.19 0.08 0.09

NEPSY TOM

beta -0.16 -0.06 -0.08 -0.13 -0.05 -0.07

se 0.16 0.14 0.16 0.16 0.15 0.14

p 0.3 0.66 0.63 0.41 0.73 0.6

VMI

beta 0 -0.04 -0.04 -0.09 -0.04 -0.07

se 0.16 0.13 0.14 0.15 0.14 0.13

p 1 0.78 0.8 0.57 0.77 0.58

Figure 7.9 Building Block Skills from Direct Child Measures (ITT)
Table 7.10 Building Block Skills from Direct Child Measures (TOT)

 Building Blocks TOT Listwise Data Estimates Imputed Data Estimates 

Bucket Measure Null Pretest Covariates Null Pretest Covariates 

Building 

Blocks

Choice 

Delay

beta -0.07 -0.07 -0.28 -0.27 -0.27 -0.24

se 0.35 0.35 0.35 0.43 0.43 0.33

p 0.84 0.84 0.43 0.53 0.53 0.46

DAS 

Naming 

Vocab

beta 0.53+ 0.49++ 0.66* 0.66+ 0.68++ 0.57* 

se 0.34 0.28 0.3 0.4 0.37 0.28

p 0.11 0.08 0.03 0.11 0.07 0.04

DAS Recall 

of Seq 

Order

beta 0.1 0.09 -0.21 0.08 0.17 0 

se 0.33 0.33 0.33 0.4 0.39 0.39

p 0.77 0.79 0.53 0.85 0.67 1 

DAS Verbal 

Comp

beta -0.19 -0.08 0.02 -0.24 -0.09 -0.1

se 0.33 0.32 0.35 0.41 0.39 0.34

p 0.57 0.82 0.96 0.56 0.82 0.76

EMT-ACES

beta -0.1 0.09 0.01 -0.07 0.08 0.17

se 0.35 0.31 0.32 0.41 0.39 0.4

p 0.78 0.76 0.99 0.87 0.84 0.67

HTKS

beta 0.46+ 0.58++ 0.63++ 0.54+ 0.7++ 0.67++ 

se 0.35 0.33 0.35 0.41 0.4 0.4

p 0.19 0.08 0.07 0.19 0.08 0.1

NEPSY TOM

beta -0.35 -0.13 -0.15 -0.34 -0.13 -0.16

se 0.34 0.31 0.32 0.41 0.38 0.31

p 0.31 0.66 0.63 0.42 0.73 0.6

VMI

beta 0 -0.08 -0.08 -0.23 -0.11 -0.16

se 0.34 0.3 0.3 0.41 0.38 0.29

p 1 0.78 0.8 0.57 0.77 0.58

Figure 7.10 Building Block Skills from Direct Child Measures (TOT)
Table 7.11 Academic Skills from Direct Child Measures (ITT)

Academics ITT Listwise Data Estimates Imputed Data Estimates 

Bucket Measure Null Pretest Covariates Null Pretest Covariates 

Academics WJ - App 

Prob

beta -0.22+ -0.2+ -0.15 -0.22+ -0.16 -0.21+ 

se 0.15 0.13 0.14 0.15 0.13 0.13

p 0.16 0.13 0.27 0.14 0.24 0.11

WJ - Ac 

Know

beta 0.06 0.02 0.09 0.01 0.01 0.08

se 0.16 0.12 0.13 0.15 0.12 0.12

p 0.71 0.87 0.5 0.94 0.94 0.53

WJ - 

LetterWord ID

beta 0.22+ 0.18+ 0.18 0.16 0.13 0.15

se 0.15 0.13 0.14 0.15 0.14 0.13

p 0.16 0.17 0.2 0.31 0.33 0.25

Figure 7.11 Academic Skills from Direct Child Measures (ITT)
Table 7.12 Academic Skills from Direct Child Measures (TOT)

Academics TOT Listwise Data Estimates Imputed Data Estimates 

Bucket Measure Null Pretest Covariates Null Pretest Covariates 

Academics WJ - App 

Prob beta

-

0.48+ -0.44+ -0.33

-

0.58+ -0.42 -0.48+ 

se 0.35 0.3 0.3 0.4 0.36 0.31

p 0.17 0.14 0.27 0.15 0.25 0.12

WJ - Ac 

Know

beta 0.13 0.04 0.19 0.03 0.02 0.18

se 0.34 0.27 0.28 0.39 0.33 0.28

p 0.71 0.87 0.5 0.94 0.94 0.53

WJ - LetterWord ID

beta 0.48+ 0.4+ 0.39 0.41 0.35 0.34

se 0.35 0.29 0.3 0.4 0.36 0.3

p 0.17 0.17 0.21 0.31 0.33 0.25

Figure 7.12 Academic Skills from Direct Child Measures (TOT)
Appendix H – Fidelity Measures 

This Appendix contains three items: 

 OST (Out of School Time) cover sheet and Observation Instrument. Three live 

observations per nest are made each year: fall, winter and spring. 

 WINGS Leader Questionnaire. This questionnaire is completed monthly by the 

program director. 

 Hunter Bailin Modified Tool. Three live observations per nest are made each year: 

fall, winter and spring. 
ata*

one TOTAL PARTICIPANTS #

Classroom Total number of boys

Academic Center Gym Total number of girls

Cafeteria

Auditorium

Hallway TOTAL STAFF #

ata* Computer Lab WINGS Leaders

Program Director

Story reading/listening Peace Manager

Other:__________________ Program Assistant

Volunteer/Partner

Outside Playground

TYPE OF SPACE

Visual Arts

GRADE LEVELS 

(circle all that apply)

OUT OF SCHOOL TIME (OST) 

OBSERVATION INSTRUMENT

COVER SHEET

ACTIVITY TYPE

Homework Help

ACTIVITY

Community Unity

Discussion

Choice Time

Free Play

Date: (mm/dd/yy)
Observer Initials:

__________________

OUT OF SCHOOL TIME (OST)

OBSERVATION INSTRUMENT

COVER SHEET

COHORT: 

Program: WINGS Leader: Observation Date: Start Time: End Time:

Number:

Activity Overview (1-2 

sentence description):

Activity Type  

*ata Activity Name  

one Total Participants #

Homework Community Unity Total number of girls

Academic Activities 

(not homework) Academic Center Total number of boys

Story reading/listening Discussion Grade Levels

Visual arts Choice Time K 1 2

3 4 5

Dance Freeplay Total Staff #

Music WINGS Works WINGS Leader

Drama Type of Space  Program Director

Crafts Classroom Peace Manager

Sports Gym Program Assistant

Open, unstructured 

time (e.g., free play) Cafeteria Partner/Volunteer

Staff-assigned learning 

games 

Auditorium Other:______________

Community service Hallway Co-Observed?  Yes 

 No

Cultural awareness 

clubs/projects Outside Playground

Other:_____________ Other:___________ *ata = all that apply
12 

OST INDICATOR ITEM RATINGS 

Directions to Observers: After 15 minutes of observation, assign a rating of 1 (not evident) to 7 (highly evident and

consistent) to each item below. To select a rating, identify the ODD NUMBER that most closely reflects how evident and 

pervasive an indicator is. If that number does not precisely reflect the level of evidence observed, then move down or up 

to the adjacent even number that more accurately reflects the item’s presence within an activity. 

Note that each indicator may not be present or applicable in each observation; therefore, a rating of “1” may be 

accurately descriptive and not necessarily negative. The “5” rating is also used in cases where the indicator’s presence 

is implicit within the activity. For instance, if youth are generally friendly to each other throughout the observation, but 

most do not go beyond a casual, friendly interaction, the rating would be a “5.” If the congeniality is active, pervasive, 

and continuous, the rating would be a “7.” 

RATINGS: 

-------1------- -------2------- -------3------- -------4------- -------5------- -------6------- -------7------- 

Exemplar is 

not evident 

Exemplar is 

rarely evident

Exemplar is 

evident or 

implicit 

Exemplar is 

highly evident 

and consistent

RELATIONSHIP BUILDING: YOUTH... PARTICIPATION: YOUTH...

Are friendly and relaxed with one another. Youth 

socialize informally. They are relaxed in their interactions 

with each other. They appear to enjoy one another’s 

company. 

Are on-task. Youth are focused, attentive, and not easily 

distracted from the task/project. They follow along with the 

staff and/or follow directions to carry on an individual or group 

task. Noise level and youth interactions can be high if youth 

are engaged in the expected task(s). 

Respect one another. Youth refrain from causing 

disruptions that interfere with others accomplishing their 

own tasks. When working together, they consider one 

another’s viewpoints. They refrain from derogatory 

comments or actions about an individual person and the 

work s/he is doing; if disagreements occur, they are 

handled constructively. 

Listen actively and attentively to peers and staff. Youth 

listen and respond to each other and staff. They appear 

interested in what others have to say. They look at peers 

and/or staff when they speak, and they provide concrete and 

constructive feedback about ideas or actions. 

Show positive affect to staff. Youth interact with the 

staff, and these interactions are generally friendly 

interactions. For example, they may smile at staff, laugh 

with them, and/or share good-natured jokes. 

Contribute opinions, ideas, and/or concerns to 

discussions. Youth discuss/express their ideas and respond 

to staff questions and/or spontaneously share connections 

they’ve made. This item goes beyond basic Q&A and refers to 

sharing as part of the activity and within the class norms. 

Calling out – or disruptively talking out of turn – is not part of 

this item. 

Assist one another. One or more youth formally or 

informally reach out to help/mentor peers and help them 

think about and figure out how to complete a task. This 

item refers to assistance that is intentional and prolonged, 

going beyond answering an incidental question. May 

include assisting one another with drama, dance, step, or 

sports techniques/moves.

Have opportunities to make meaningful choices. Within 

this activity, youth choose what they do, how they do it, and/or 

with whom they collaborate, and they experience the 

consequences of their choices. This item refers to genuine 

options about how to accomplish the task, not simple choices 

such as choosing between two types of games, or two sets of 

homework pages. 

Are collaborative. Youth work together/share materials 

to accomplish tasks. Youth are equal partners in the 

work. This item includes strategizing how to complete a 

product and includes planning a cohesive product or 

performance (e.g., a dance, a play, or a musical event) or 

winning a game. This item is different from item D 

(above) in that it involves a joint intellectual effort. 

Take leadership responsibility/roles. Youth have 

meaningful responsibility for directing, mentoring or assisting 

one another to achieve an outcome; they lead some part of the 

activity by organizing a task or a whole activity, or by leading a 

group of youth within the activity. 
13 

RELATIONSHIP BUILDING: STAFF... INSTRUCTIONAL STRATEGIES: STAFF...

Use positive behavior management techniques that 

allow youth to accomplish the activity’s objectives. 

Staff set consistent limits and clear behavioral 

standards, and these are appropriate to the age of the 

youth and the activity type. If it is necessary to 

discipline, staff do so in a firm manner, without 

unnecessary accusations, threats, or anger and there 

is no evidence of disciplinary problems. 

Communicate goals, purpose, expectations. Staff make 

clear the purpose of what youth are doing and/or what they 

expect them to accomplish. Activity goals/expectations may 

also be implicit if students are clearly on task without staff 

direction. This item goes beyond how youth are expected to 

behave (which would be captured in item K). 

Encourage the participation of all. Regardless of 

gender, race, language ability, or other evident 

differences among students, staff try to engage 

students who appear isolated; they do not favor (or 

ignore) a particular student or small cluster of 

students. 

Verbally recognize youth’s efforts and accomplishments. 

Staff explicitly acknowledge youth’s participation and progress 

to motivate them using praise, encouragement, and/or 

constructive guidance/modeling. (Must involve verbal 

statements not just implied affirmation.) 

Show positive affect toward youth. Staff tone is 

caring and friendly; they use positive language, smile, 

laugh, or share good-natured jokes. They refrain from 

threats, cutting sarcasm, or harsh criticism. If no 

verbal interaction is necessary, staff demonstrate a 

positive and caring affect toward youth. 

Assist youth without taking control. Staff refrain from taking 

over a task or doing the work for the youth. They coach, 

demonstrate, or employ scaffolding techniques that help youth 

to gain a better understanding of a concept or complete an 

action on their own. This assistance goes beyond checking that 

work is completed. 

Attentively listen to and/or observe youth. Staff 

look at youth when they speak and acknowledge what 

they have said by responding and/or reacting verbally 

or nonverbally. They pay attention to youth as they 

complete tasks and are interested in what youth are 

saying/doing. 

Ask youth to expand upon their answers and ideas. Staff 

encourage youth to explain their answers, to give evidence, or 

suggest conclusions. They ask youth “why,” “how,” and “if” 

questions to get youth to expand, explore, better clarify, 

articulate, or concretize their thoughts/ideas. This item goes 

beyond basic Q&A. 

Encourage youth to share their ideas, opinions, 

and concerns about the content of the activity. 

Staff actively elicit youth ideas, opinions, and concerns 

on the activity content through discussion and/or 

writing. This item goes beyond basic Q&A to fully 

engage with youth’s ideas and thinking. 

Challenge youth to move beyond their current level of 

competency. Staff give constructive feedback that is designed 

to motivate youth, to set a higher standard, and meant to help 

youth gauge their progress. Staff help youth determine ways to 

push themselves intellectually, creatively, and/or physically. 

Engage personally with youth. Staff show a 

personal interest in youth as individuals, ask about 

their interests, and engage in discussion about events 

in their lives. This goes beyond content-based 

discussions to include personal interest and 

demonstrate caring by the adults. 

Employ varied teaching strategies. To engage students 

and/or reach those with different learning styles, staff use 

diverse instructional strategies, which may include: direct 

instruction, coaching, modeling, demonstrating, or others. 

Varied instructional strategies can occur simultaneously and/or

sequentially within the observation period. 

Guide positive peer interactions. The lesson 

structure/content explicitly encourages positive 

relationships/interactions and/or teaches interpersonal 

skills. May involve staff explaining or through planned 

activity content why negative behavior (e.g., bullying, 

teasing, etc.) is unacceptable and offering constructive 

behavior alternatives. However, this item does not 

refer to behavior management, as described above 

(see item K). 

Plan for/ask youth to work together. Staff structure activities 

so that youth work cooperatively to solve problems, and/or 

accomplish tasks. The focus of the activity is youth-to-youth, 

rather than youth-to-staff. This item goes beyond staff-assigned 

teams for competitive games and sports. In the case of staffassigned teams, staff actively encourage youth to collaborate, 

plan, devise strategies, etc. 

CONTENT AND STRUCTURE: ACTIVITY

(Note: When homework is the observed activity, do not score these indicators.) 

Is well organized. Activity has clear (implicitly or 

explicitly stated) goals/objectives; there is evidence of 

a clear lesson plan and process(es), and tasks can be 

conducted in the timeframe available. If special 

materials are needed, they are prepared and available. 

Involves the practice/a progression of skills. Activity 

involves practicing skills needed to complete tasks. If a longterm project, youth’s activity on the project provides the 

opportunity to apply or expand skills or techniques previously 

learned. 

Challenges students intellectually, creatively, 

developmentally, and/or physically. Activity’s level 

of challenge is not so difficult that youth have trouble 

participating successfully and not so easy that youth 

complete tasks routinely, without thought, and become 

restless/disengaged. 

Requires analytic thinking. Activity calls on students to think 

about and solve meaningful problems and/or juggle multiple 

activities or strategies/dimensions to accomplish a task. For 

example, the activity requires youth to think about two or more 

ideas, and/or understand and apply sequencing or patterns. 

This can apply to complex dance, arts, theater, or sports 

moves, routines, or strategies. 
15 

ENVIRONMENTAL CONTEXT 

1. Is the level of adult supervision appropriate to activity and age group? Yes No 

 If no: Why not? 

2. Is the work space conducive to the activity? Yes No 

 If no: Why not?

3. Are necessary materials available and in sufficient supply? Yes No 

If no: Why not? 

OBSERVER’S SYNTHESIS AND RATING 

On a 1-7 scale, rate the extent to which this activity demonstrates these features:

Element OBSERVER’S SYNTHESIS RATING 

SEQUENCED: Activity builds

progressively more sequenced and 

advanced skills and knowledge and 

challenges youth to achieve clear 

goals.

ACTIVE: Youth engage actively in

learning. They lead/participate in 

discussions, develop or research a 

product, contribute original ideas, 

collaborate, take on leadership roles, 

and/or are oriented toward 

completing tasks. 

PERSONALLY FOCUSED:

Actively strengthens relationships

among youth and between youth and 

staff.

EXPLICIT: The activity explicitly

targets specific learning and/or 

developmental goals




1 

Hunter-Bailin Observation Tool 

Child Trends modifications 11/5/15

Activity-Based Implementation Standards

The Welcome/Eat and Greet N/A 0 1 2 3 Notes

01. The WL is present and ready to interact 

when the children begin to enter the room X 

02. The PD, PA and/or PM are present and 

ready to interact when the children begin to 

enter the room X 

03. As needed, the PD, PA and/or PM interact 

with children—greeting them by name, 

encouraging them to join their nests, and 

managing any disruptions that occur outside 

the nests

X 

04. WL actively engages their nest X 

05. All kids sit down in their nest, when 

expected to. x 

06. All kids sit in their nests and eat snack

07. Community Unity starts on time or in a 

timely manner YES NO

WINGS for kids - Implementation Assessment October 2015

 Community Unity Observation Protocol

Site (school):_______________________________________

Date (include day of week):___________________________

WL (first name only):_________________________________ 

Observer:__________________________________________

Total # Participants (include # girls/ # boys):_____________

Grade/Cohort:_______________________________________
2 

Group time N/A

0123 Notes

08. The PD or PA explains the SEL game 

using the microphone and if the kids do not 

already know the game, the D-D-A-D-A steps 

are used to explain it (Describe it, 

Demonstrate, Ask questions, Do it, Adapt)

(Note: sometimes Kids Praise and Soaring 

Stars Awards are given out in place of SEL 

games)

09. WINGS staff make the game fun and 

exciting (Note: Also applies to other activities 

such as Heys and Praise, and Awards) X 

10. WINGS staff ensure that all kids are 

demonstrating Creed-like behavior (Note: in 

addition to the SEL game, this applies to other 

activities such as Heys and Praise, and 

Awards)

X 

11. Staff member on the mic encourages the 

kids to participate by asking questions or 

calling on kids to share comments X 

12. Staff member on the mic makes his/her 

announcements to the entire community X 

Creed recital N/A

0123 Notes

13. The entire community says the WINGS 

Creed together X 

14. WL ensures that everybody in their nest

says the Creed in a respectful manner (this is at 

the nest level, not the program level) x 

15. Community Unity ends on time YES NO
3 

Social and Emotional Learning Implementation Standards

N/A 0 1 2 3 Notes

16. WINGS staff use teachable moments to 

teach kids about the Creed X 

17. WINGS staff use teachable moments to 

teach kids about the weekly SEL objectives x 
4 

Behavior Management Implementation Standards

Shaping/Modifying/Reinforcing behavior N/A 0 1 2 3 Notes

18. WINGS staff apply the M-E-S-S method to 

shape behavior (Make it fun, Expectations (set 

them clearly), Stay busy, Show kids how to 

help)

Note: expectation-setting may not be observed if 

expectations were set at a prior time; use notes 

column to record specific components of M-ES-S observed/ not observed

X 

19. WINGS staff apply the G-E-T P-A-S-T 

method to shape behavior (Give choices, 

Experience consequences, Take it away, 

Problem (state it), Amends, State feelings 

strongly, Tell it in a word)

20. Were there any instance(s) of a kid(s) 

needing assistance with behavior issues, school 

problems, or personal situations?

YES NO

21. If yes to #20, WLs make use of the Peace 

Manager to provide needed assistance

22. WL use Behavior Boosters (aka The Great 

Trait Raffle) to recognize kids for displaying 

positive behaviors and SEL skills

Note: this includes the use of school-specific 

reward systems such as ‘NCES Cash’

X 
5 

WINGS for kids - Implementation Assessment October 2015

 Academic Centers Observation Protocol

Site (school):_______________________________________

Date (include day of week):___________________________

WL (first name only):_________________________________ 

Observer:__________________________________________

Total # Participants (include # girls/ # boys):_____________

Grade/Cohort:_______________________________________

Academic-Based Implementation Standards

Getting organized N/A 0 1 2 3 Notes

01. Academic Centers starts on time or in a timely manner Yes No

02. Kids get organized and ready to start their work, following 

clearly established routines X 

03. The WL at no time allows kids back into their classrooms to 

get anything they forgot (PD/PA does this) Yes No

04. The WL sets up the Academic Center Box and ensures that it 

is complete with all supplies X 

05. The WL sets up the WINGSReads box and ensures that it 

contains sufficient books based on the reading level of the kids X 
6 

Doing homework N/A 0 1 2 3 Notes

06. WL fills out HMWK sheet and signs off Yes No

07. The WL walks around the room answering questions and 

helping kids with difficult assignments. If there is no homework 

WL is engaged in helping with student activities (reading, 

worksheets, puzzles, etc.)

X 

08. Kids are sitting at desk/assigned space and not getting up 

without permission X 

09. WL provides feedback that encourages kids to be persistent 

and complete their homework or other academic activity. 

10. WL offers support for kids who are having difficulty 

understanding or completing their homework or academic activity.

11. The WL checks the assignment and gives it back to the child 

to make any corrections

12. Kids have clear activities to do when waiting or finished with 

their work, also in the case they do not have homework. x 

13. Kids are quietly working and only speaking when they have 

raised their hand or have permission. X 
7

Wrap-up Notes

14. The WL puts all WINGS supplies away Yes No

15. The kids pack up their supplies Yes No

16. The classroom is left exactly as it is found Yes No

17. Academic Center ends on time Yes No
8

Social and Emotional Learning Implementation Standards

N/A 0 1 2 3 Notes

18. WLs use teachable moments to teach kids about the Creed X

19. WLs use teachable moments to teach kids about the weekly 

SEL objectives X
9 

Behavior Management Implementation Standards

Shaping/Modifying/Reinforcing behavior N/A 0 1 2 3 Notes

20. WLs apply the M-E-S-S method to shape behavior (Make it 

fun, Expectations (set them clearly), Stay busy, Show kids how to 

help)

Note: expectation-setting may not be observed if expectations 

were set at a prior time; use notes column to record specific 

components of M-E-S-S observed/ not observed

X 

21. WLs apply the G-E-T P-A-S-T method to shape behavior 

(Give choices, Experience consequences, Take it away, Problem 

(state it), Amends, State feelings strongly, Tell it in a word)

22. Were there any instance(s) of a kid(s) needing assistance with 

behavior issues, school problems, or personal situations? YES NO

23. If yes to #22, WLs make use of the Peace Manager to provide 

needed assistance

24. WL use Behavior Boosters (aka The Great Trait Raffle) to 

recognize kids for displaying positive behaviors and SEL skills

Note: this includes the use of school-specific reward systems such 

as ‘NCES Cash’

X 
10

WINGS for kids - Implementation Assessment October 2015

 Discussion Observation Protocol

Site (school):_______________________________________

Date (include day of week):___________________________

WL (first name only):_________________________________ 

Observer:__________________________________________

Total # Participants (include # girls/ # boys):_____________

Grade/Cohort:_______________________________________

Activity-Based Implementation Standards

Discussion N/A 0 1 2 3 Notes

01. WL engages kids in the discussion of the day X 

02. The Talking Stick is used throughout every discussion. Only 

the person with the stick may speak. X 
11

Social and Emotional Implementation Standards

N/A 0 1 2 3 Notes

03. WLs use teachable moments to teach kids about the Creed x 

04. WLs use teachable moments to teach kids about the weekly 

SEL objectives X 
12

Behavior Management Implementation Standards

Shaping/Modifying/Reinforcing behavior N/A 0 1 2 3 Notes

05. WLs apply the M-E-S-S method to shape behavior (Make it 

fun, Expectations (set them clearly), Stay busy, Show kids how to 

help)

Note: expectation-setting may not be observed if expectations 

were set at a prior time; use notes column to record specific 

components of M-E-S-S observed/ not observed

X 

06. WLs apply the G-E-T P-A-S-T method to shape behavior 

(Give choices, Experience consequences, Take it away, Problem 

(state it), Amends, State feelings strongly, Tell it in a word)

07. Were there any instance(s) of a kid(s) needing assistance with 

behavior issues, school problems, or personal situations?

YES NO

08. If yes to #07, WLs make use of the Peace Manager to provide 

needed assistance

09. WL use Behavior Boosters (aka The Great Trait Raffle) to 

recognize kids for displaying positive behaviors and SEL skills

Note: this includes the use of school-specific reward systems such 

as ‘NCES Cash’

X 
13

WINGS for kids - Implementation Assessment October 2015

 Choice Time Observation Protocol

Site (school):_______________________________________

Date (include day of week):___________________________

WL (first name only):_________________________________ 

Observer:__________________________________________

Total # Participants (include # girls/ # boys):_____________

Grade/Cohort:_______________________________________

Activity-Based Implementation Standards

Choice time N/A 0 1 2 3 Notes

01. Choice Time starts on time (according to weekly schedule) YES NO

02. WL and/or Choice Time Partner engage the kids throughout 

Choice Time X 

PLEASE DESCRIBE THE SPECIFIC CHOICE TIME 

ACTIVITY THAT TOOK PLACE

03. The Choice Time room is left in the same condition as it was 

found

YES NO
14

Social and Emotional Implementation Standards

N/A 0 1 2 3 Notes

04. WLs use teachable moments to teach kids about the Creed X 

05. WLs use teachable moments to teach kids about the weekly 

SEL objectives X 
15

Behavior Management Implementation Standards

Shaping/Modifying/Reinforcing behavior N/A 0 1 2 3 Notes

06. WLs apply the M-E-S-S method to shape behavior (Make it 

fun, Expectations (set them clearly), Stay busy, Show kids how to 

help)

Note: expectation-setting may not be observed if expectations 

were set at a prior time; use notes column to record specific 

components of M-E-S-S observed/ not observed

X 

07. WLs apply the G-E-T P-A-S-T method to shape behavior 

(Give choices, Experience consequences, Take it away, Problem 

(state it), Amends, State feelings strongly, Tell it in a word)

08. Were there any instance(s) of a kid(s) needing assistance with 

behavior issues, school problems, or personal situations?

YES NO

09. If yes to #08, WLs make use of the Peace Manager to provide 

needed assistance

10. WL use Behavior Boosters (aka The Great Trait Raffle) to 

recognize kids for displaying positive behaviors and SEL skills

Note: this includes the use of school-specific reward systems such 

as ‘NCES Cash’

X 
16

WINGS for kids - Implementation Assessment October 2015

 WINGS Works Observation Protocol

Site (school):_______________________________________

Date (include day of week):___________________________

WL (first name only):_________________________________ 

Observer:__________________________________________

Total # Participants (include # girls/ # boys):_____________

Grade/Cohort:_______________________________________

Activity-Based Implementation Standards

WINGS Works N/A 0 1 2 3 Notes

01. Wings Works starts on time YES NO

02.WL is prepared for the day's activity X 

03. WLs leads their kids with a good attitude X 

04. The theme of service is woven into the activity and geared 

towards kids’ interest X 

PLEASE DESCRIBE THE SPECIFIC ACTIVITY THAT 

TOOK PLACE

05. WINGS Works ends on time YES NO
17

Social and Emotional Implementation Standards

N/A 0 1 2 3 Notes

06. WLs use teachable moments to teach kids about the Creed X 

07. WLs use teachable moments to teach kids about the weekly 

SEL objectives x 
18

Behavior Management Implementation Standards

Shaping/Modifying/Reinforcing behavior N/A 0 1 2 3 Notes

08. WL apply the M-E-S-S method to shape behavior (Make it 

fun, Expectations (set them clearly), Stay busy, Show kids how to 

help)

Note: expectation-setting may not be observed if expectations 

were set at a prior time; use notes column to record specific 

components of M-E-S-S observed/ not observed

X 

09. WL apply the G-E-T P-A-S-T method to shape behavior 

(Give choices, Experience consequences, Take it away, Problem 

(state it), Amends, State feelings strongly, Tell it in a word)

10. Were there any instance(s) of a kid(s) needing assistance with 

behavior issues, school problems, or personal situations?

YES NO

11. If yes to #10, WLs make use of the Peace Manager to provide 

needed assistance

12. WL use Behavior Boosters (aka The Great Trait Raffle) to 

recognize kids for displaying positive behaviors and SEL skills

Note: this includes the use of school-specific reward systems such 

as ‘NCES Cash’

X 
19

Scoring Instructions 

0 = The behavior/activity was expected to occur but it never occurred; or the behavior/activity occurred with extremely low quality

1 = The behavior/activity occasionally occurred, or was demonstrated in relation to some but not all children, or it occurred but with moderate quality 

2 = The behavior/activity frequently occurred, or was demonstrated in relation to most but not all children, or it occurred but with moderately high 

quality 

3 = The behavior/activity always occurred, or was demonstrated in relation to all children, or it occurred but with extremely high quality 

N/A = Only use N/A when the item describes something that did not occur and did not need to occur or was not expected to occur. If something did 

not occur, but it should have occurred, that would be scored a 0. For example, for Item 9, “All kids sit in their nests and eat snack,” you can check 

N/A if snack time does not occur as part of Community Unity at that school, or you can score it 0 if the kids are supposed to be sitting by nests to eat 

snack but they are up wandering around. 
IES	WINGS	Evaluation	| CONTEXT CHANGES OVER	TIME | May	2016	 Page |	1	

Appendix	I	Changing	School	and	WINGS	Factors	During	the	3	Year	Study	Period	(2012-2015)

Table	I.1	summarizes	the	important	changes	occurring	by	year	and	school	that	might	both	affect	

impact	results	and/or	provide	a	context	to	better	interpret	results.	These	changes	are	described	in	

more	detail	below.				

Table	I.1	Important	Changes	in	the	WINGS	Program	and	Schools	over	the	2012-2105	Period.

Change School and Year of Change

North Charleston Chicora Memminger 1 James Simmons

School Changes

School schedule changed 2012-2013

School location changed 2013-2014 2013-2014

Teaching 

changed

pedagogy 2014-2015 2013-2014

Principal changed 2014-2015 2014-2015 2014-2015

WINGS Program 

Changes

Expansion to Atlanta 2012-2013 2012-2013 2012-2013 2012-2013

Program 

changed

schedule 2012-2013

Program director changed 2014-2015 2013-2014 2014-2015

Access to 

changed

school space 2014-2015

WINGS 

changed

attendance policy 2012-2013

2013-2014

2012-2013

2013-2014

2012-2013

2013-2014

2012-2013

2013-2014

1

James Simmons’ WINGS program closed at end of 2013-2014 school year because the student population had changed so 

significantly.

Overall	Major	Changes	

1. Two	School	Moves:	

a. James	Simons	and	Memminger	Elementary- YEAR	2 (2013-14):		

b. North	Charleston	to	Downtown	

c. Transportation	was	an	issue	for	families

d. Student	population	changed	

2. School	Pedagogical	Change	at	JSE:	YEAR	2	(2013-14):		

a. JSE	moved	to	a		Montessori	model	for	PK-	3rd grade

b. Student	demographics	changes-	higher	SES	

c. Large	teacher	shift	due	to	required	training

d. WINGS	was	affected-	no	kindergarten,	then	no	program	

3. School	Administrative	Changes	at	Memminger	and	Chicora:	YEARS	2	and	3	(2013-15)

a. Principal	changes	impacted	WINGS

b. Relationships	with	WINGS	Program	Directors	changed	(Memminger	only)

4. WINGS	Schedule	Changes	at	JSE	due	to	Shared	Building:	YEAR	1	(2012-2013)
I ES	WINGS	Evaluation	| CONTEXT CHANGES OVER	TIME | May	2016	 Page |	2	

a. WINGS	at	JSE	started	later	due	to	staggered	start	of	school	day	(bussing	issue)

b. WINGS	shortened	program	to	avoid	ending	too	late	in	the	day

c. Families	still	complained	that	children	were	getting	home	too	late

5. WINGS	Administrative	Changes:	Expanding	to	Atlanta:	YEAR	1	(2012-2013)

a. Some	staff	moved	to	Atlanta

b. Trainings	were	bigger	and	involved	the	new	staff	in	Atlanta

6. WINGS	Policy	Change:	Children	asked	to	leave	program-	YEAR	2	(2013-14):

a. In	Year	1-	WINGS	kept	children	in

b. In	Year	2-	returned	to	their	old	model	of	having	students	leave	for	behavioral	and	attendance	

reasons

7. WINGS Space	Restrictions	at	Memminger-	YEAR	3 (2014-2015):	

a. WINGS	was	banned	from	using	classrooms

b. Access	only	to	gym	and	cafeteria	

c. Schedule	of	WINGS	changed

Major	Changes	by	School

NORTH	CHARLESTON	ELEMENTARY	SCHOOL	(NCES)

School/	Administrative	Factors:	During	the	three year study period,	NCES	has been	stably	located at	the	same	

site in	North	Charleston. NCES	hired	a	new	principal	(Ms.	Townsend),	and	she	remained	principal	until	the	

spring	of	Year	3	(2014-2015)	of	the	study,	at	which	time,	the	assistant	principals	took	over.	

WINGS Factors:	 The	principal	was	very	open	and	welcoming	to	WINGS,	and	the	WINGS program	remained	

very	solid during	the	three year	study	period,	with	Nicole	directing	the	WINGs	program	for	the	first	2	years,	

and	Cara	becoming	the	Program	Director	in	Year	3.

Major	Impressions	of	NCES

•

•

Very stable	school	with	respect	to	location,	administration	and	WINGS	

WINGS	had	a	solid program	during	all	three	years

CHICORA	ELEMENTARY	SCHOOL

School/	Administrative	Factors: During	the	three year	study	period,	Chicora	was	housed	in	the	Robert	McNair	

Building	(3795	Spruill	Ave).	Chicora	hired	a	new	principal	(Mr.	Agnew),	who	remained	in	this	role	for	the	first	2	

years	of	the	study.	During	the	third	year,	Ms.	Coakley,	previously	the	assistant	principal,	moved	into the	role	of	

Principal.	She	overhauled	the	teaching	staff,	bringing	in	many	new	teachers.	

WINGS	Factors:	During	all	three	study	years the	WINGS	program	had	full	support	from	both	principals.	Even	

though	there	was	some	turnover	of	Program	Directors	(from	Cheryl	to	Ashley in	Year	2),	the	program	was very

solid,	extremely	well	run,	and	had	a	good	reputation	in	the	community	(families	wanted	to	be	part	of	it).	The	

principal	and	PD	had	a	great	working	relationship	the	entire	time.

Major	Impressions	of	Chicora
IES	WINGS	Evaluation	| CONTEXT CHANGES OVER	TIME | May	2016	 Page |	3	

•

•

Very	stable	school	with	respect	to	location,	administration	and	WINGS	

WINGS	had	a	very	strong	program	during	all	three	years

MEMMINGER		ELEMENTARY

School/	Administrative	Factors: Several	years	prior	to	the	study	Memminger,	originally	located	in	downtown	

Charleston	on	Beaufain	St.,	had	moved to	the	Brentwood	campus	in	North	Charleston (2685	Leeds	Ave.).	At	

the	Brentwood	campus,	Memminger	shared	a	building with	James	Simons	Elementary.	This	move	was	hard on	

families	who	lived	downtown because	the	children	now	had	to	be	bussed to	a	new	school.	Additionally,	

Memminger	enrolled all	students	living	within	its North	Charleston	vicinity,	and	thus	mixed	children	from	two	

different	communities.	

Prior	to	the	start	of	the	second	year	of	the	study	(July,	2013)	,	Memminger	moved from	the	Brentwood	

campus	in	North	Charleston	back	to	its	original	space	in	downtown Charleston	(20	Beaufain	St),	where	it	

remained	throughout	the	course	of	the	study.	This	move	was	not	completed	by	the	time	school	began,	with	

boxes	still	being	unpacked	on	the	first	day	of	school.	The	school	now	did	not	accept	children	if	they	were	no	

longer	districted	for	the	downtown	location.	As	a	result	all	North	Charleston	families	previously	attending were	

redistricted.

Memminger’s	principal,	Ms.	Taylor	remained	for	the	first	and	second	year	of	the	study.	During	the	

third year	of	the	study,	Dr.	Woods	became	the	principal. She	began	school	procedures	for	International	

Baccalaureate certification.	Some	related	standards	made	their	way	into	school	programs	that	year.

WINGS	Factors:	 During	the	three year	study	period,	there	was	a	shift in Program	Directors.	Li	was	in	the	role	

of	director	during	the	first	2	years	of	the	study	with Mallory	stepping	in	during	Year	3. The	principal	initially	

was	fairly	uninvolved	with	WINGS.	However,	she	did	express	concerns during	the	end	of	Year	2	that	property	

being	defaced in	the	classrooms.	By the	third	year,	she	banned	WINGS from	using	classrooms and	only	

allowed	use	of	the gym	and	the	cafeteria.	This	forced	WINGS	to change its	regular	schedule in	order	to	serve	

its many	students	within	such	rigid	space	constraints.	

Major	Impressions	of	Memminger

•

•

External	and	internal	disruptions	affected	school	staff	and	families

o

o

o

o

moving	

administrative	changes

transportation	issues	for	families

beginnings	of	pedagogical	changes

WINGS	had	to	work	around	space	restrictions	and	a	schedule	change	(shortening	of	program)

JAMES	SIMONS	ELEMENTARY (JSE)

School/	Administrative	Factors: Like	Memminger (though	smaller),	JSE	was	originally	a	downtown	school that	

moved	from	741	King	Street	to	the	Brentwood	Campus	in	North	Charleston several	years	prior to	the	study.	

This	move was	hard	on	families	who	lived	downtown	(those	originally	districted	to	JSE),	not	only	because	the	
IES	WINGS	Evaluation	| CONTEXT CHANGES OVER	TIME | May	2016	 Page |	4	

children	now	had	to	be	bussed	to school,	but	the	school	had	to	stagger	its	start	and	end	time	with	that	of	

Memminger’s.	JSE	families	(who	all	lived	downtown)	complained	about	significant	transportation	issues,	and	

their	children	getting	home	late	and	exhausted.

Midway	through	the	second	year,	the	school	moved back downtown to	its	original	address on King	

Street.	The	demographics of	the	school	changed,	with	a	greater	percentage	of	the	newly	districted	children	

being	of	a	higher	socioeconomic	group.	In	addition,	at	the	start	of	this	school	year,	JSE	switched	to	a	

Montessori	model.	All	grades	below	fourth	followed	this	model.

A	new	principal,	Ms.	White,	was	hired	during	the	summer	before	our	study,	and	remained	in	her	role	

for	all	three	years	of	the	study.	In	response	to	the	shift	to	a	Montessori	model	during	Year	2,	she

made	changes	to	her	teaching	staff,	hiring new	Montessori	trained	teachers,	and	moving other	untrained	

teachers	around,	either	within	or	outside	of	the	school.	

WINGS	Factors: At	the	start	of	the	study,	the	principal expressed interested	in	the	details	of	the	study,	and	

wanted	to	support	families	in	any	way	she	could.	During	the	first	year,	JSE	implemented	a	staggered	start	and	

end to	the school	day,	due	to	sharing	space	with	Memminger.		The	long	school	day that resulted forced	

WINGS	to	shorten its	program to	2	hours.	Families	still	complained	about	getting	home	late.		

In	Year	2,	as	a	result	of	JSE’s change	to	Montessori,	WINGS did	not	hold	a	kindergarten group	but	still	

had	first	grade	students.		By	the	third	year	of	the	study,	the	WINGS	program	was	no	longer	implemented at	

JSE.	They	had	finished	out	the	previous	year	and	did	not	even	start	this	year	because	the	demographics	had	

changed	so	much.

Major	Impressions	of	JSE

•

•

Schedule	changes	due	to	sharing	schools	in		Years	1-2	

o

o

Shortened	WINGS

Families	stills	struggled	with	children	coming	home	late

School	Pedagogical	Change:	Shift	to	Montessori	Model	in	Year	2

o

o

o

Student	demographic	changed

WINGS	affected-	no	Kindergarten	in	Year	2

WINGS	affected-	no	WINGS	program	in	Year	3

GENERAL	OBSERVATIONS	ABOUT	WINGS	PROGRAM	AND	SCHOOL	ADMINISTRATION

Within	the context	of	each	study	school,	and	across	each	of	the	three	years,	the	backdrop	of	the	WINGS	

program	inevitably	did	not stay	the	same.	Significant	stressors	to	schools	and	study	families	change	

components	of	both	school	and	WINGS	programming.	These stressors	also	ultimately	affected which	children	

flow	in	and	out	of	the	program.

How	much	WINGS	was	or	was	not	supported	by	schools	varied	from	school	to	school,	principal	to	principal,	

and	year	to	year.	However,	it	is	important	to	recognize	that	the	quality	of	the	relationship	between	school	

administrators,	teachers	and	WINGS	staff	is	a	critical	ingredient	to	WINGS’	success.		
IES	WINGS	Evaluation	| CONTEXT CHANGES OVER	TIME | May	2016	 Page |	5	

Two	important	WINGS-specific	factors	that	occurred	during	the	study	period	were	1)	WINGS	expansion,	and	2)	

changes	in	policy	about	children	leaving	the	program.	During	the	course	of	the	three year	study,	WINGS	was	

expanding	its	program	across	the	east	coast.	During	the	first	year	of	the	study,	WINGS	expanded	to	Atlanta.	In	

addition	to	staff	movement,	the	summer	training	of	WINGS	leaders	in	Charleston	was	bigger,	given that	WINGS	

bussed the	trainees	from	Atlanta	to	Charleston	to	join	the	Charleston	cohort	in	the	training.	Regarding	the	

second	factor,	in	the	first	year	of	the	study,	WINGS	changed	its	policy	about	asking children	to	leave the	

program	for	behavioral	or	attendance	reasons.	They	kept	children	in	WINGS	to	help	minimize	program	

attrition. In Year	2,	WINGS	returned	to	the old	model	of	asking	students	to	leave	for	behavioral	and	attendance	

reasons.
Appendix J- Publications

Ph.D. Thesis 

Anderson, R. E. (under review). And Still WE Rise: Parent-child relationships 

and child school readiness in urban Black families. 

Kim, H., (in preparation). Patterns of behavioral self-regulation in lowincome kindergarten children: Integrating Variable- and PersonCentered Approaches. AERA Division E Outstanding Dissertation Award 

in Human Development. 

Duran, C., (anticipated May 2016). The Family Stress Model and Children’s 

Self-Regulation. 

Publications 

Anderson, R. E. (2015). Focusing on family: Parent–child relationships and 

school readiness among economically impoverished Black children. The 

Journal of Negro Education, 84, 442-456.

Brock, L. L., Adams, C. L., Kim, H., Mashburn, A., & Grissmer, D. W. (revise 

& resubmit). Theory of mind as an indicator of school readiness: The 

role of perspective-taking in a sample of low-income kindergartners, 

Early Education & Development. 

Cameron, C. E., Kim, H., Duncan, R. J., Decker, D. R., & McClelland, M. M. 

(in preparation). Cognitive and Academic Skills in Kindergarten: 

Evidence for Cross-Domain Gains, with Specificity. 

Doromal, J. B., Cottone, E. A., & Kim, H. (in preparation). Validation of the 

DESSA in low-income kindergarten children. 

Duran, C. A. K., Cottone, E. A., Mashburn, A., & Grissmer, D. W. (revise & 

resubmit). The Family Stress Model and Children’s Self-Regulation. 

Child Development

Kim, H., Cameron, C. E., Adams, C., West, H., Mashburn, A., & Grissmer, D. 

(revise & resubmit). Validating an individualized child observational 

measure in low-income kindergarten classrooms. Early Childhood 

Research Quarterly. 

Kim, H., Cameron, C. E., Doromal, J. B., & Grissmer, D. (in preparation). 

Patterns of behavioral self-regulation in low-income kindergarten 

children: Integrating Variable- and Person-Centered Approaches. 

Mace, A., Cameron, C. E., & Cottone, E. A. (manuscript in preparation). Are 

low-income children’s social problem-solving skills associated with the 

relationship they develop with their kindergarten teacher? 
Appendix J- Publications

Presentations 

Adams, C., Brock, L. L., (2014, March). Theory of mind as an indicator of 

school readiness. Paper presented at the Southeastern Psychological 

Association meeting, Nashville, TN. 

Anderson, R. (May, 2014). And Still WE Rise: Poverty Risk, Parent-Child 

Relationships, and Child School Readiness Indicators in Urban Black 

Families. Paper presented at the Black Graduate Conference in 

Psychology, Washington, DC. 

Brock, L. L., Kim, H., Adams, C., Mashburn, A., & Grissmer, D. (2015, 

March). Theory of mind as an indicator of school readiness: The role of 

perspective-taking in a sample of low-income kindergartners. Poster 

presented at 2015 Society for Research in Child Development 

Conference, Philadelphia, PA. 

Cameron, C. E., Kim, H., Mashburn, A., Adams, C., West, H., & Grissmer, D. 

(2015, March). Classroom Conditions to Consider When Observing 

Children for an RCT Evaluation. Paper presented in R. Bulotsky Shearer 

(chair), Helping young children to positively engage in early childhood 

classrooms: Findings from three social-emotional learning programs, 

2015 Society for Research in Child Development Conference, 

Philadelphia, PA. 

Cameron, C. E., & Kim, H. (2015, March). Beyond initial level, does

improvement in EF and visuomotor skills predict kindergarten 

achievement gains?. Poster presented at a Special Evening Poster 

Session on the Strategic Plan at 2015 Society for Research in Child 

Development Conference, Philadelphia, PA. 

Duran, C.A.K., Cottone, E. A., Mashburn, A., & Grissmer, D. W. (2016, 

February). Extending the Family Stress Model: Stressful Life Events, 

Family Stress Processes, and Development of Self-Regulation. Paper 

presented at 2016 Curry Research Conference. Charlottesville, VA. 

Cameron, C. E., Kim, H., Duncan, R. J., Decker, D. R., & McClelland, M. M. 

(2016, April). Improving in EF and visuo-motor integration predicts 

kindergarten achievement: Evidence from Two U.S. States. Poster 

presented at 2016 American Educational Research Association Annual 

Conference, Washington, DC. 
Appendix J- Publications

AERA Classroom Observation Special Interest Group Exemplary 

Paper Nomination: 

Kim, H., Cameron, C. E., Adams, C., & West, H. (2016, April). Validating an 

individualized child observational measure in low-income kindergarten 

classrooms. Paper presented in B. L. Alford (chair), Classroom 

observations: Instruments, application, and findings, 2016 American 

Educational Research Association Annual Conference, Washington, DC. 

Mace, A., Cameron, C. E. (2016, April). Are low-income children’s social 

problem-solving skills associated with the relationship they develop 

with their kindergarten teacher? Poster presented at GSE Student 

Research Symposium, Buffalo, NY, USA. 

Duran, C.A.K., Cottone, E. A., Mashburn, A., & Grissmer, D. W. (2016, May). 

The Family Stress Model and Children’s Cognitive Self-Regulation. 

Poster presented at Education and Inequality in 21st Century America 

Conference. Stanford, CA. 

Duran, C.A.K., Cottone, E. A., Mashburn, A., & Grissmer, D. W. (2016, July). 

The Family Stress Model and Children’s Cognitive Self-Regulation. 

Poster presented at 2016 National Research Conference for Early 

Childhood, Washington, DC. 

Kim, H., Doromal, J. B., & Cameron, C. E. (2016, July). Patterns of 

behavioral self-regulation in low-income kindergarten children: A 

person-centered approach. Poster presented at 2016 National 

Research Conference for Early Childhood, Washington, DC. 

Mace, A., & Cameron, C. E. (2016, October). Kindergarteners' initial social 

problem-solving skills predict the relationship they develop with their 

teacher. Poster presentation accepted for the Northeastern Educational 

Research Association, Trumbull, CT, USA. 

Brock, L. L., Kim, H., & Adams, C. (2017, April). Self-regulation, 

perspective-taking, and student-teacher relationships in kindergarten. 

Paper accepted to C. Mulcahy (chair), Self-regulation and studentteacher relationships: Variable and person-centered approaches 

examining connections to development, 2017 Society for Research in 

Child Development Conference, Austin, TX. 

Brock, L. L., Kim, H., & Adams, C. (2017, April). The development of theory 

of mind: Predictors and moderators of improvement in kindergarten.

Paper accepted for 2017 American Educational Research Association 

Annual Conference, San Antonio, TX. 
Appendix J- Publications

Brock, L. L., Kim, H., & Adams, C. (2017, April). Longitudinal associations 

among executive function, visuomotor integration, and achievement: 

Timing is everything. Paper accepted for 2017 American Educational 

Research Association Annual Conference, San Antonio, TX. 

Brock, L. L., Kim, H., & Adams, C. (2017, April). Mental representation, 

theory of mind, and social skills: Cross-lagged associations across 

kindergarten and first grade. Poster accepted to 2017 Society for 

Research in Child Development Conference, Austin, TX. 

Kim, H., Cameron, C. E., Doromal, J. B., & Grissmer, D. W. (2017, April). 

Patterns of behavioral self-regulation in low-income kindergarten 

children: Integrating variable- and person-centered approaches. Paper 

accepted to C. Mulcahy (chair), Self-regulation and student-teacher 

relationships: Variable and person-centered approaches examining 

connections to development, 2017 Society for Research in Child 

Development Conference, Austin, TX. 

West, H., Adams, C., & Kim, H. (2017, April). Classroom quality as a 

predictor of classroom behavior in a sample of socio-demographically 

at-risk students. Poster accepted to 2017 Society for Research in Child 

Development Conference, Austin, TX.