Efficacy of Schoolwide Programs to 

Promote Social and Character 

Development and Reduce Problem 

Behavior in Elementary School Children

Report From the Social and Character Development 

Research Program

OCTOBER 2010

Social and Character Development Research Consortium

NCER 2011-2001

U.S. DEPARTMENT OF EDUCATION
This report was prepared for the National Center for Education Research, Institute of Education Sciences, under 

Contract No. ED-01-CO-0039/0006 with Mathematica Policy Research, Inc. Synergy Enterprises, Inc., under 

Contract No. ED-04-CO-0076/0007, provided editorial and graphic support for the report.

Disclaimer

The findings and conclusions expressed in this report are the authors’ and do not necessarily represent the opinions 

and positions of the Institute of Education Sciences, the U.S. Department of Education, or the Centers for Disease 

Control and Prevention. 

U.S. Department of Education

Arne Duncan 

Secretary

Institute of Education Sciences

John Q. Easton

Director

National Center for Education Research

Lynn Okagaki

Commissioner

October 2010

This is a report by the Social and Character Development Research Consortium. It was written by Allen Ruby and 

Emily Doolittle of the Institute of Education Sciences. 

The report is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission 

to reprint this publication is not necessary, the citation should be: Social and Character Development Research 

Consortium (2010). Efficacy of Schoolwide Programs to Promote Social and Character Development and Reduce Problem Behavior in 

Elementary School Children (NCER 2011-2001). Washington, DC: National Center for Education Research, Institute of 

Education Sciences, U.S. Department of Education. 

To order copies of this report, 

• Write to ED Pubs, Education Publications Center, U.S. Department of Education, P.O. Box 22207, Alexandria, 

VA 22304.

• Call in your request toll free to 1-877-4ED-Pubs. If 877 service is not available in your area, call 800-872-5327. 

Those who use a telecommunications device for the deaf (TDD) or a teletypewriter (TTY) should call 

800-437-0833. 

• Fax your request to 703-605-6794 or order online at www.edpubs.gov. 

The report is also available on the IES website at http://ncer.ed.gov. 

On request, this publication is available in alternate formats, such as Braille, large print, or computer diskette. For 

more information, contact the Department’s Alternate Format Center at 202-260-0852 or 202-260-0818.
iii

Social and Character Development Research Consortium 

Institute of Education Sciences (IES)

Caroline Ebanks 

Edward C. Metz

Amy Silverman1

Elizabeth R. Albro 

Division of Violence Prevention, National Center 

for Injury Prevention and Control, Centers for 

Disease Control and Prevention (CDC) 

Tamara M. Haegerich2

Le’Roy Reese3

Corinne David-Ferdon 

Mathematica Policy Research, Inc. (MPR)

Karen Needels

John A. Burghardt

Laura M. Kalb

Heather Koball

Peter Z. Schochet

Victor Battistich (University of Missouri-St. Louis)

Children’s Institute

Deborah B. Johnson

Hugh F. Crean

New York University4

J. Lawrence Aber

Stephanie M. Jones5(Fordham University)

Joshua L. Brown6

Oregon State University7

Brian R. Flay

Carol G. Allred (Positive Action)

David L. DuBois (University of Illinois at Chicago)

Michael L. Berbaum (University of Illinois at Chicago)

Peter Ji (University of Illinois at Chicago)

Vanessa Brechling (University of Illinois at Chicago)

University at Buffalo, The State University 

of New York

William E. Pelham, Jr.

Greta M. Massetti8

Daniel A. Waschbusch

University of Maryland

Gary D. Gottfredson

University of North Carolina at Chapel Hill

Mark W. Fraser

Thomas W. Farmer9

Maeda J. Galinsky

Kim Dadisman

Vanderbilt University

Leonard Bickman

Ana Maria Brannan

Catherine Smith

 

1

 Amy Silverman is currently a clinical psychologist at the Jewish Family & Career Services of Atlanta, Inc. 

2

 Tamara M. Haegerich served as IES Project Officer before joining the CDC. 

3 Le’Roy Reese is currently on the faculty at Morehouse School of Medicine.

4

 This award was originally made to Columbia University and was transferred to New York University (NYU) when the 

principal investigator, J. Lawrence Aber, accepted a position at NYU. 

5

 Stephanie M. Jones is now on the faculty at Harvard University. 

6

 Joshua L. Brown is now on the faculty at Fordham University. 

7

 This award was originally made to the University of Illinois at Chicago, but was transferred to Oregon State University 

(OSU) when the principal investigator, Brian R. Flay, accepted a position at OSU. 

8

 Greta M. Massetti is now at the Division of Violence Prevention, National Center for Injury Prevention and Control, 

CDC. 

9

 Thomas W. Farmer was the principal investigator when the award was made to the University of North Carolina at 

Chapel Hill. During the project, he accepted a position at Pennsylvania State University, and Mark Fraser became the 

principal investigator. 
This page is intentionally blank. 
v 

Acknowledgments 

This report resulted from a collaboration among the scientists who are members of the Social and Character 

Development Research Consortium (SACD Research Consortium), school district personnel, and curriculum 

developers. Many school districts, principals, teachers, and school staff members assisted the evaluation by 

providing records and coordinating site visits for data collection. We appreciate their help and willingness to 

share critical information with the evaluation team. 

The authors listed under the SACD Research Consortium represent only part of the research team involved 

in the project. We would like to thank the research staff at each research team’s site, especially each team’s 

site coordinator. These people worked closely with the local schools’ staff and the contractors’ data collection 

teams to facilitate the successful collection of the data. 

We would also like to acknowledge the contribution of the many contract staff members who worked on the 

data collection and analysis. The authors wish to thank Mathematica Policy Research, Inc. (MPR) staff 

members Audrey McDonald and Carolyn Miller for their tireless efforts in managing data collection, and 

Shawn Marsh and Larry Snell for their development and maintenance of the tracking system. We give special 

thanks to Decision Information Resources, Inc., in particular Douglas Hermond and Carla Prince, for 

assisting in data collection and the development of coding protocols for analysis. We appreciate the 

contributions of Marvin Berkowitz and Mindy Bier at the University of Missouri-St. Louis to the evaluation. 

Finally, we wish to provide thanks to Clotilde Benitez and Ken Carlisle from Friday Systems, which provided 

support for the SACD working meetings. 

The SACD Research Consortium would also like to remember Dr. Victor Battistich from the College of 

Education at the University of Missouri-St. Louis. Dr. Battistich, who passed away on June 20, 2008, served 

as a consultant to MPR during this project and was a leading researcher in children’s social development and 

school-based programming. 

The mention of trade names, commercial products, or organizations in the description of the projects, or the 

reporting of study findings, does not imply endorsement by the U.S. government. 
This page is intentionally blank.
vii

Disclosure of Potential Conflicts of Interest 

The Social and Character Development (SACD) Research Consortium consists of research teams (principal 

investigators and co-principal investigators from each grantee site); Institute of Education Sciences (IES) 

staff; Centers for Disease Control and Prevention (CDC) staff; and the evaluation contractor, Mathematica 

Policy Research, Inc. (MPR). Except for the three exceptions described in the paragraph below, the grantee 

research teams, IES staff, CDC staff, and contractor staff from MPR have no interests that could be affected 

by findings from the evaluation of the intervention programs that are described in this report. 

Three of the SACD program research teams were led by and/or included personnel who had developed the 

intervention programs implemented at their respective research sites. Dr. William E. Pelham, Jr. and his 

colleagues developed the Academic and Behavioral Competencies Program, which was implemented in elementary 

schools in New York. The Competence Support Program, implemented in North Carolina schools, is made up of 

three distinct interventions (Making Choices, Competence Enhancement Behavior Management, and Classroom Social 

Dynamics Training). Dr. Mark W. Fraser and his colleagues developed the Making Choices program. Dr. Thomas 

W. Farmer and his colleagues developed the Competence Enhancement Behavior Management and the Social Dynamics 

Training programs. The Positive Action program was developed by Carol G. Allred, who is a member of the 

research team that implemented the program in schools in Illinois and is married to Dr. Brian R. Flay, the 

leader of that team. 

All seven teams of researchers were selected to receive funding for their SACD research projects in a 

competitive grant application process. Each research team implemented its intervention and conducted sitespecific analysis examining the effects of these interventions on student outcomes. MPR, the evaluation study 

contractor, conducted independent evaluations of all the intervention programs that were included in the 

SACD study. A data collection team from MPR independently collected the data for the evaluation presented 

in this report with the exception of the fidelity of implementation data, which were collected by the seven 

teams. The MPR data analysis team completed all descriptive and impact analyses. The 

developers/implementers of these interventions did not conduct the impact analyses that are summarized in 

this report. 
This page is intentionally blank. 
ix

Glossary 

4Rs—The 4Rs Program (Reading, Writing, Respect, and Resolution) 

ABC—Academic and Behavioral Competencies Program

ADHD—attention deficit hyperactivity disorder 

CDC—Centers for Disease Control and Prevention 

combined-program—all seven programs together 

CR—Child Report 

CSP—Competence Support Program

DIR—Decision Information Resources, Inc., a subcontractor to Mathematica Policy Research, Inc. (MPR), 

the contractor selected to do the independent evaluation of the SACD programs 

ES—effect size 

HHS—U.S. Department of Health and Human Services 

HLM—hierarchical linear model 

ICC—intraclass correlation 

IES—Institute of Education Sciences, U.S. Department of Education 

LBW—Love In a Big World 

leavers—students who left schools that were participating in the study before the study was completed

MANOVA—multivariate analysis of variance 

MDES—minimum detectable effect size 

middle childhood—the developmental period from 6 to 12 years of age 

MPR—Mathematica Policy Research, Inc., the contractor selected to do the independent evaluation of the 

SACD programs 

multiprogram evaluation—the evaluation of the impact of the seven SACD programs on students’ social and 

character development. The evaluation looked at the average impact of all seven SACD programs combined 

and the average impact of each SACD program individually. 

named activities—activities carried out in the context of a named SACD program to support students’ social 

and character development 

named program—a specific SACD program 

new entrants—students who entered schools participating in the study after the study had begun 

outcome domain—a set of outcomes grouped together because they measure a similar school activity, teacher 

instructional behavior, or student behavior 
Glossary 

x 

outcomes—what the SACD programs were expected to change regarding school activities, teacher 

instructional behavior, or student behavior 

PA—Positive Action

PATHS—Promoting Alternative Thinking Strategies 

PCR—Primary Caregiver Report 

SACD—social and character development 

SACD activities—activities to support students’ social and character development 

SACD goals—six specific goals defined by the SACD Research Program and promoted through the use of 

SACD activities to support students’ social and character development. The goals are (1) violence prevention 

and peace promotion, (2) social and emotional development, (3) character education, (4) tolerance and 

diversity, (5) risk prevention and health promotion, and (6) civic responsibility and community service. 

SACD program—a specific program that has the purpose of improving the social and character development 

of students 

SACD Research Consortium—the researchers responsible for the SACD Research Program. These include 

staff at the Institute of Education Sciences (IES) in the U.S. Department of Education; the Division of 

Violence Prevention in the National Center for Injury Prevention and Control, Centers for Disease Control 

and Prevention (CDC); and Mathematica Policy Research, Inc. (MPR), the contractor, selected to do the 

independent evaluation of the SACD programs; as well as the seven research teams that received funding 

under cooperative agreements to evaluate one SACD program of their choosing under an experimental 

design. 

SACD Research Program—a collaboration between the Institute of Education Sciences (IES) in the U.S. 

Department of Education and the Division of Violence Prevention in the National Center for Injury 

Prevention and Control, Centers for Disease Control and Prevention (CDC). Its purpose is to evaluate 

multiple universal, elementary school-based programs that target student social development and behavior 

outcomes. 

SACD training—professional development for teachers on instructional methods to promote student social 

and character development 

school-based program—a program delivered in a school setting 

site—the set of schools, both treatment and control, specific to each individual SACD program evaluation 

SS—Second Step 

stayers—students who were in schools participating in the study for the entire study 

targeted school-based programs—programs that are designed to address the needs of a subset of the children 

in a school 

TRCS—Teacher Report on Classroom and School 

TRS—Teacher Report on Student 

universal programs—programs that are designed to address the needs of all children in a school 

WWC—the What Works Clearinghouse
xi

Contents 

Page 

Social and Character Development Research Consortium ..................................................... iii

Acknowledgments ..................................................................................................................... v

Disclosure of Potential Conflicts of Interest ...........................................................................vii

Glossary ................................................................................................................................. ix

List of Tables .......................................................................................................................... xvi

List of Figures ....................................................................................................................... xxiv

Executive Summary ............................................................................................................... xxv

Study Design ..............................................................................................................................................................xxvi

Research Questions ..................................................................................................................................................xxxii

Data Collection .........................................................................................................................................................xxxii

Initial Characteristics ................................................................................................................................................xxxv

Analysis and Results ................................................................................................................................................xxxvi

Annual Impacts on the Use of Social and Character Development Activities .........................................xxxvi

Annual Impacts on Student Outcomes and Perceptions of School Climate ............................................xxxix

Impacts on Growth of Student Outcomes .......................................................................................................xlvi

Discussion ........................................................................................................................................................................ li

Chapter 1. The Social and Character Development Multiprogram Evaluation ........................ 1

Social Development and Risk Prevention .................................................................................................................. 3 

Universal School-Based Social and Character Development Programs ................................................................ 4 

Research Questions ........................................................................................................................................................ 5 

The Social and Character Development Research Program ................................................................................... 8 

Study Design and Methodology ................................................................................................................................. 20

Sample and Random Assignment .........................................................................................................................21

Measures ....................................................................................................................................................................28

Construction of Outcome Variables ....................................................................................................................37

Construction of Moderator Variables ..................................................................................................................41

Data Collection ........................................................................................................................................................45

Consent Rates, Completion Rates, and Percentage of Sample With Data .....................................................49

Initial Characteristics ....................................................................................................................................................55

Characteristics of Children, Their Families, and Communities .......................................................................55

Characteristics of Teachers and Schools ..............................................................................................................62

The Initial Level of Social and Character Development Activities in the Schools .......................................65

Year-by-Year Impacts on Use of Social and Character Development Activities ..............................................70

Analysis of Teacher Reports ..................................................................................................................................70

Analysis of Principal Interviews ............................................................................................................................72

Results From Teacher Reports ..............................................................................................................................72
Contents 

xii

Page 

Results From Principal Interviews ........................................................................................................................81

Findings Regarding Use of SACD Activities ......................................................................................................84

Year-by-Year Impacts on Students and Perceptions of School Climate .............................................................85

Analysis ......................................................................................................................................................................85

Results .......................................................................................................................................................................93

Year-by-Year Analysis of Impacts on Student Subgroups ..................................................................................100

Analysis ....................................................................................................................................................................101

Results .....................................................................................................................................................................103

Summary of Results Regarding Year-by-Year Impacts ........................................................................................127

Impacts on Growth of Student Outcomes ............................................................................................................127

Sample .....................................................................................................................................................................128

Analysis ....................................................................................................................................................................133

Results .....................................................................................................................................................................135

Sensitivity Analyses ................................................................................................................................................157

Discussion ....................................................................................................................................................................158

Chapter 2. Academic and Behavioral Competencies Program ............................................. 163 

Intervention ................................................................................................................................................................. 163

Sample and Random Assignment ............................................................................................................................167

Implementation...........................................................................................................................................................169

Training ...................................................................................................................................................................169

Data Collection ......................................................................................................................................................169

Consent Rates, Completion Rates, and Percentage of Sample With Data ...................................................171

Fidelity of Implementation ..................................................................................................................................175

Initial Characteristics ..................................................................................................................................................175

Characteristics of Children, Their Families, and Communities .....................................................................175

Characteristics of Teachers and Schools ............................................................................................................179

The Level of SACD in the Schools Near the Beginning of the Study ..........................................................181

Impacts on Use of SACD Activities .......................................................................................................................186

Use of Activities .....................................................................................................................................................186

Use of Materials and Strategies ............................................................................................................................192

Participation in Professional Development .......................................................................................................195

Attitudes and Practices .........................................................................................................................................197

Year-by-Year Impacts on Students and Perceptions of School Climate ...........................................................197

Impacts on Child Outcomes Over Time ................................................................................................................205

Summary ......................................................................................................................................................................208

Chapter 3. Competence Support Program ............................................................................. 211 

Intervention ................................................................................................................................................................. 211

Sample and Random Assignment ............................................................................................................................214

Implementation...........................................................................................................................................................216

Training ...................................................................................................................................................................216
Contents 

xiii

Page 

Data Collection ......................................................................................................................................................216

Consent Rates, Completion Rates, and Percentage of Sample With Data ...................................................218 

Fidelity of Implementation ..................................................................................................................................222

Initial Characteristics ..................................................................................................................................................222

Characteristics of Children, Their Families, and Communities .....................................................................222

Characteristics of Teachers and Schools ............................................................................................................226

The Level of SACD in the Schools Near the Beginning of the Study ..........................................................227

Impacts on Use of SACD Activities .......................................................................................................................231

Use of Activities .....................................................................................................................................................231

Use of Materials and Strategies ............................................................................................................................237

Participation in Professional Development .......................................................................................................240

Attitudes and Practices .........................................................................................................................................242

Year-by-Year Impacts on Students and Perceptions of School Climate ...........................................................242

Impacts on Child Outcomes Over Time ................................................................................................................250

Summary ......................................................................................................................................................................253

Chapter 4. Love In a Big World ............................................................................................. 255

Intervention ................................................................................................................................................................. 255

Sample and Random Assignment ............................................................................................................................258

Implementation...........................................................................................................................................................260

Training ...................................................................................................................................................................260

Data Collection ......................................................................................................................................................260

Consent Rates, Completion Rates, and Percentage of Sample With Data ...................................................262

Fidelity of Implementation ..................................................................................................................................266

Initial Characteristics ..................................................................................................................................................266

Characteristics of Children, Their Families, and Communities .....................................................................266

Characteristics of Teachers and Schools ............................................................................................................270

The Level of SACD in the Schools Near the Beginning of the Study ..........................................................271

Impacts on Use of SACD Activities .......................................................................................................................276 

Use of Activities .....................................................................................................................................................276 

Use of Materials and Strategies ............................................................................................................................282 

Participation in Professional Development .......................................................................................................285 

Attitudes and Practices .........................................................................................................................................287 

Year-by-Year Impacts on Students and Perceptions of School Climate ...........................................................287 

Impacts on Child Outcomes Over Time ................................................................................................................295 

Summary ......................................................................................................................................................................298 

Chapter 5. Positive Action ..................................................................................................... 301

Intervention ................................................................................................................................................................. 301

Sample and Random Assignment ............................................................................................................................304 

Implementation...........................................................................................................................................................306 

Training ...................................................................................................................................................................306 

Data Collection ......................................................................................................................................................306 
Contents 

xiv

Page 

Consent Rates, Completion Rates, and Percentage of Sample With Data ...................................................308 

Fidelity of Implementation ..................................................................................................................................312 

Initial Characteristics ..................................................................................................................................................312 

Characteristics of Children, Their Families, and Communities .....................................................................312 

Characteristics of Teachers and Schools ............................................................................................................ 316

The Level of SACD in the Schools Near the Beginning of the Study ..........................................................317 

Impacts on Use of SACD Activities .......................................................................................................................322 

Use of Activities .....................................................................................................................................................322 

Use of Materials and Strategies ............................................................................................................................328 

Participation in Professional Development ....................................................................................................... 331

Attitudes and Practices .........................................................................................................................................333 

Year-by-Year Impacts on Students and Perceptions of School Climate ...........................................................333 

Impacts on Child Outcomes Over Time ................................................................................................................ 341

Summary ......................................................................................................................................................................344 

Chapter 6. Promoting Alternative Thinking Strategies (PATHS) ........................................ 347

Intervention ................................................................................................................................................................. 347 

Sample and Random Assignment ............................................................................................................................ 350

Implementation...........................................................................................................................................................352

Training ...................................................................................................................................................................352

Data Collection ......................................................................................................................................................352

Consent Rates, Completion Rates, and Percentage of Sample With Data ...................................................353

Fidelity of Implementation ..................................................................................................................................358

Initial Characteristics ..................................................................................................................................................358

Characteristics of Children, Their Families, and Communities .....................................................................358

Characteristics of Teachers and Schools ............................................................................................................ 361

The Level of SACD in the Schools Near the Beginning of the Study ..........................................................363

Impacts on Use of SACD Activities .......................................................................................................................368

Use of Activities .....................................................................................................................................................368

Use of Materials and Strategies ............................................................................................................................374

Participation in Professional Development .......................................................................................................377

Attitudes and Practices .........................................................................................................................................379

Year-by-Year Impacts on Students and Perceptions of School Climate ...........................................................379

Impacts on Child Outcomes Over Time ................................................................................................................387

Summary ...................................................................................................................................................................... 390

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) ........................ 393

Intervention ................................................................................................................................................................. 393

Sample and Random Assignment ............................................................................................................................396

Implementation...........................................................................................................................................................398

Training ...................................................................................................................................................................398

Data Collection ......................................................................................................................................................398

Consent Rates, Completion Rates, and Percentage of Sample With Data ...................................................400
Contents 

xv

Page 

Fidelity of Implementation ..................................................................................................................................404

Initial Characteristics ..................................................................................................................................................404

Characteristics of Children, Their Families, and Communities .....................................................................404

Characteristics of Teachers and Schools ............................................................................................................408

The Level of SACD in the Schools Near the Beginning of the Study ..........................................................409

Impacts on Use of SACD Activities .......................................................................................................................414

Use of Activities .....................................................................................................................................................414

Use of Materials and Strategies ............................................................................................................................ 420

Participation in Professional Development .......................................................................................................423

Attitudes and Practices .........................................................................................................................................425

Year-by-Year Impacts on Students and Perceptions of School Climate ...........................................................425

Impacts on Child Outcomes Over Time ................................................................................................................433

Summary ......................................................................................................................................................................436

Chapter 8. Second Step .......................................................................................................... 439

Intervention ................................................................................................................................................................. 439

Sample and Random Assignment ............................................................................................................................442

Implementation...........................................................................................................................................................444

Training ...................................................................................................................................................................444

Data Collection ......................................................................................................................................................444

Consent Rates, Completion Rates, and Percentage of Sample With Data ...................................................445

Fidelity of Implementation .................................................................................................................................. 450

Initial Characteristics .................................................................................................................................................. 450

Characteristics of Children, Their Families, and Communities ..................................................................... 450

Characteristics of Teachers and Schools ............................................................................................................454

The Level of SACD in the Schools Near the Beginning of the Study ..........................................................455

Impacts on Use of SACD Activities ....................................................................................................................... 460

Use of Activities ..................................................................................................................................................... 460

Use of Materials and Strategies ............................................................................................................................466

Participation in Professional Development .......................................................................................................469

Attitudes and Practices ......................................................................................................................................... 471

Year-by-Year Impacts on Students and Perceptions of School Climate ........................................................... 471

Impacts on Child Outcomes Over Time ................................................................................................................ 479

Summary ......................................................................................................................................................................482

References .............................................................................................................................. 485

Appendix A: Analysis of Cohorts 1 and 2 .............................................................................. A-3 

Sample .......................................................................................................................................................................... A-3 

Initial Data ................................................................................................................................................................... A-6 

Analysis ..................................................................................................................................................................... A-18

Impacts on Use of SACD Activities ................................................................................................................ A-18
List of Tables 

xvi

Page 

Impacts on Student Outcomes: Year-by-Year and Growth Analyses Results ......................................... A-23

Growth Analysis ................................................................................................................................................. A-39

Summary ................................................................................................................................................................... A-45

Appendix B: Technical Notes on the Development of Outcome Measures, Selection of 

Covariates, Construction of Sample Weights, and Sensitivity Analyses ............................... B-1

Development of the Outcome Measures ............................................................................................................... B-3

Confirmatory Factor Analyses of Student-Level Outcome Measures .......................................................... B-4

Robustness Checks ................................................................................................................................................ B-6

Construct and Reporter-Based Variance ........................................................................................................... B-8

Psychometric Properties of School-Level Outcome Measures ...................................................................... B-8

Selection of Covariates .............................................................................................................................................. B-8

Objectives in Selecting Covariates ...................................................................................................................... B-9

Tailoring Covariates to Specific Outcomes and Sites ...................................................................................... B-9

Implementation of the Covariate Selection Procedure ................................................................................. B-10

Construction of Sample Weights ........................................................................................................................... B-37

Base Weights ........................................................................................................................................................ B-37

Adjusted Weights ................................................................................................................................................. B-37

Weights for the Growth Curve Analysis .......................................................................................................... B-36

Sensitivity Analysis ................................................................................................................................................... B-38

Excluding Covariates .......................................................................................................................................... B-40

Employing Alternative Weighting Schemes .................................................................................................... B-42

Including Classroom-Level Random Effects in the Error Structure .......................................................... B-45

Accounting for the Pairwise Matching of Schools ......................................................................................... B-47

Treating Pretest Measures as Dependent Variables ....................................................................................... B-49

Using Alternative Software Packages ............................................................................................................... B-52

Imputing Missing Outcomes ............................................................................................................................. B-55

Averaging Program-Level Impact Estimates .................................................................................................. B-57

Excluding New Entrants .................................................................................................................................... B-60

Including Restricted Sets of Covariates in the Regression Models ............................................................. B-61

List of Tables 

Table Page 

Executive Summary 

A. Research teams, SACD programs, and number of schools ....................................................................xxvii

B. Student sample, overall and by program, for all students and for the treatment and control 

groups .............................................................................................................................................................xxviii

C. Outcomes, by domain and data instrument ...............................................................................................xxxi

D. Consent rates, completion rates, and percentage of sample with data from each report .................xxxiv 

E. Significant impacts on use of SACD activity domains, overall and by program .............................. xxxviii
List of Tables 

xvii

Table Page 

F. Significant impacts on student outcomes, overall and by program, for all 3 years ................................ xlii

G. Individual program statistically significant impacts and nonsignificant but substantively important 

impacts ...............................................................................................................................................................xliii

H. Impacts on growth of child outcomes from combined-program analysis ...........................................xlviii

I. Significant impacts from the growth curve analyses of the individual programs ....................................... l 

Chapter 1. The Social and Character Development Multiprogram Evaluation 

1.1. Social and Character Development programs ................................................................................................. 9 

1.2. Multiprogram evaluation school sample ........................................................................................................ 22 

1.3. Characteristics of the student sample in the multiprogram evaluation ..................................................... 24 

1.4. Student sample, overall and by program, for all students and for the treatment and control groups . 26 

1.5. Outcome variables .............................................................................................................................................39

1.6. Outcomes, by domain and data instrument ..................................................................................................41

1.7. Fidelity ratings of program implementation at treatment schools ............................................................. 44 

1.8. Period between implementation and data collection and between data collections, by program ........ 47 

1.9. Consent rates, completion rates, and percentage of sample with data from each report ......................51

1.10. Consent rates, completion rates, and percentage of sample with data: Stayers versus new entrants ... 52 

1.11. Percentage of sample with data, by report and program ............................................................................. 53 

1.12. Initial characteristics of children, their families, and communities ............................................................ 57 

1.13. Mean scores and standard deviations for initial outcome measures .......................................................... 60 

1.14. Mean scores and standard deviations for initial risk factors ....................................................................... 61 

1.15. Initial characteristics of teachers ...................................................................................................................... 63 

1.16. Initial characteristics of schools ....................................................................................................................... 64 

1.17. Principal and teacher initial reports on use of SACD programs or activities .......................................... 66 

1.18. Teacher initial reports on use of SACD materials and classroom strategies ............................................68

1.19. Principal and teacher initial reports on SACD professional development ...............................................69

1.20. Impacts on teacher-reported SACD classroom activities ........................................................................... 74 

1.21. Impacts on use of SACD classroom materials and teaching strategies ..................................................... 77 

1.22. Impacts on teacher-reported SACD professional development ................................................................ 80 

1.23. Significant impacts on use of SACD activity domains, overall and by program ..................................... 82 

1.24. Covariates used with outcomes from each report for combined-program analysis ...............................88

1.25. Adjusted minimum detectable effect sizes for combined-program impact evaluation .......................... 92 

1.26. Combined-program impacts on outcomes .................................................................................................... 94 

1.27. Individual program statistically significant impacts and nonsignificant but substantively important 

impacts ................................................................................................................................................................. 97 

1.28. Combined-program impacts on child outcomes, by gender .....................................................................105 

1.29. Combined-program impacts on child outcomes, by stayer versus new entrant .................................... 108

1.30. Combined-program impacts on child outcomes, by initial socioeconomic risk ....................................111 

1.31. Combined-program impacts on child outcomes, by initial family risk ...................................................114 

1.32. Combined-program impacts on child outcomes, by initial perceptions of community risk ...............117
List of Tables 

xviii

Table Page 

1.33. Combined-program impacts on child outcomes, by initial child behavior risk as reported by 

 teacher ................................................................................................................................................................120

1.34. Combined-program impacts on child outcomes, by child behavior risk as reported by primary 

caregiver .............................................................................................................................................................122 

1.35. Combined-program results for child outcomes, by fidelity of implementation ....................................125 

1.36. Growth curve analysis sample size, percentage of sample universe, and percentage of initial 

sample ................................................................................................................................................................ 129

1.37. Child-level outcomes, by survey period and treatment group status ....................................................... 131

1.38. Impacts on growth of child outcomes from combined-program analysis .............................................136 

1.39. Significant impacts from the growth curve analyses of the individual programs .................................. 138

1.40. Impacts on growth of child outcomes, by gender, from combined-program analysis ......................... 139

1.41. Impacts on growth of child outcomes for new entrants versus original members, from combinedprogram analysis ............................................................................................................................................... 141

1.42. Impacts on growth of child outcomes, by initial socioeconomic risk .....................................................144 

1.43. Impacts on growth of child outcomes, by initial family risk .....................................................................146 

1.44. Impacts on growth of child outcomes, by initial perceptions of community risk................................. 148

1.45. Impacts on growth of child outcomes, by child behavior risk as reported by teacher .........................150 

1.46. Impacts on growth of child outcomes, by child behavior risk as reported by primary caregiver .......152 

1.47. Sample size ranges for outcome analyses ....................................................................................................154 

Chapter 2. Academic and Behavior Competencies Program 

2.1. Academic and Behavior Competencies Program.....................................................................................................165

2.2 Sample––ABC ..................................................................................................................................................168

2.3. Data collection dates—ABC ..........................................................................................................................170

2.4. Consent rates, completion rates, and percentage of sample with data from each report—ABC .......172

2.5. Consent rates, completion rates, and percentage of sample with data: Stayers versus new 

 entrants—ABC .................................................................................................................................................174

2.6. Initial characteristics of children, their families, and communities—ABC ............................................176

2.7. Mean scores and standard deviations for initial outcome measures of sample—ABC ........................178

2.8. Initial characteristics of teachers in sample—ABC ....................................................................................180

2.9. Initial characteristics of schools in sample––ABC .....................................................................................181

2.10. Principal and teacher initial reports on use of SACD programs or activities in sample—ABC .........182

2.11. Teacher initial reports on use of SACD materials and classroom strategies in sample––ABC ..........184

2.12. Principal and teacher initial reports on SACD professional development in sample––ABC .............185

2.13. Impacts on teacher-reported SACD classroom activities—ABC ............................................................188

2.14. Impacts on use of SACD classroom materials and teaching strategies—ABC .....................................193

2.15. Impacts on teacher-reported SACD professional development––ABC .................................................196

2.16. Covariates used with outcomes from each report for analysis––ABC ...................................................198

2.17. Adjusted minimum detectable effect sizes for impact evaluation—ABC ..............................................201

2.18. Impacts on child and school outcomes—ABC ..........................................................................................203

2.19. Impacts on growth of child outcomes––ABC ............................................................................................206
List of Tables 

xix

Table Page 

Chapter 3. Competence Support Program 

3.1. Competence Support Program ...............................................................................................................................212

3.2. Sample––CSP ...................................................................................................................................................215

3.3. Data collection dates––CSP ...........................................................................................................................217

3.4. Consent rates, completion rates, and percentage of sample with data from each report—CSP ........219

3.5. Consent rates, completion rates, and percentage of sample with data: Stayers versus new 

entrants––CSP ..................................................................................................................................................221

3.6. Initial characteristics of children, their families, and communities––CSP ..............................................223

3.7. Mean scores and standard deviations for initial outcome measures of sample––CSP .........................225

3.8. Initial characteristics of teachers in sample—CSP .....................................................................................226

3.9. Initial characteristics of schools in sample––CSP ......................................................................................227

3.10. Principal and teacher initial reports on use of SACD programs or activities in sample––CSP ..........228

3.11. Teacher initial reports on use of SACD materials and classroom strategies in sample––CSP............229

3.12. Principal and teacher initial reports on SACD professional development in sample––CSP ...............230

3.13. Impacts on teacher-reported SACD classroom activities—CSP .............................................................233

3.14. Impacts on use of SACD classroom materials and teaching strategies—CSP ......................................238

3.15. Impacts on teacher-reported SACD professional development––CSP ..................................................241

3.16. Covariates used with outcomes from each report for analysis––CSP .....................................................243

3.17. Adjusted minimum detectable effect sizes for impact evaluation—CSP ...............................................246

3.18. Impacts on child and school outcomes—CSP............................................................................................248

3.19. Impacts on growth of child outcomes––CSP .............................................................................................251

Chapter 4. Love In a Big World 

4.1. Love In a Big World ............................................................................................................................................256

4.2. Sample––LBW .................................................................................................................................................. 259

4.3. Data collection dates—LBW .........................................................................................................................261

4.4. Consent rates, completion rates, and percentage of sample with data from each repor—LBW ........263

4.5. Consent rates, completion rates, and percentage of sample with data: Stayers versus new 

entrants—LBW ................................................................................................................................................265

4.6. Initial characteristics of children, their families, and communities—LBW ............................................267

4.7. Mean scores and standard deviations for initial outcome measures of sample—LBW ....................... 269

4.8. Initial characteristics of teachers in sample—LBW ...................................................................................270

4.9. Initial characteristics of schools in sample—LBW .....................................................................................271

4.10. Principal and teacher initial reports on use of SACD programs or activities in sample––LBW ........272

4.11. Teacher initial reports on use of SACD materials and classroom strategies in sample—LBW ..........274

4.12. Principal and teacher initial reports on SACD professional development in sample––LBW .............275

4.13. Impacts on teacher-reported SACD classroom activities—LBW ...........................................................278

4.14. Impacts on use of SACD classroom materials and teaching strategies—LBW ....................................283

4.15. Impacts on teacher-reported SACD professional development—LBW ................................................286

4.16. Covariates used with outcomes from each report for analysis––LBW ...................................................288

4.17. Adjusted minimum detectable effect sizes for impact evaluation—LBW ..............................................291

4.18. Impacts on child and school outcomes—LBW ..........................................................................................293

4.19. Impacts on growth of child outcomes––LBW ...........................................................................................296
List of Tables 

xx

Table Page 

Chapter 5. Positive Action 

5.1. Positive Action .....................................................................................................................................................302

5.2. Sample––PA .....................................................................................................................................................305

5.3. Data collection dates––PA.............................................................................................................................307

5.4. Consent rates, completion rates, and percentage of sample with data from each 

report––PA .......................................................................................................................................................309

5.5. Consent rates, completion rates, and percentage of sample with data: Stayers versus new 

entrants––PA.................................................................................................................................................... 311

5.6. Initial characteristics of children, their families, and communities––PA................................................313

5.7. Mean scores and standard deviations for initial outcome measures of sample––PA...........................315

5.8. Initial characteristics of teachers in sample––PA .......................................................................................316

5.9. Initial characteristics of schools in sample––PA ........................................................................................317

5.10. Principal and teacher initial reports on use of SACD programs or activities in sample––PA............318

5.11. Teacher initial reports on use of SACD materials and classroom strategies in sample––PA.............. 320

5.12. Principal and teacher initial reports on SACD professional development in sample––PA................. 321

5.13. Impacts on teacher-reported SACD classroom activities––PA...............................................................324

5.14. Impacts on use of SACD classroom materials and teaching strategies—PA ........................................329

5.15. Impacts on teacher-reported SACD professional development––PA....................................................332

5.16. Covariates used with outcomes from each report for analysis––PA.......................................................334

5.17. Adjusted minimum detectable effect sizes for impact evaluation—PA .................................................337

5.18. Impacts on child and school outcomes––PA..............................................................................................339

5.19. Impacts on growth of child outcomes––PA...............................................................................................342 

Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

6.1 Promoting Alternative Thinking Strategies ...........................................................................................................348

6.2. Sample––PATHS ............................................................................................................................................. 351

6.3. Data collection dates—PATHS .....................................................................................................................353

6.4. Consent rates, completion rates, and percentage of sample with data from each 

report––PATHS ...............................................................................................................................................355

6.5. Consent rates, completion rates, and percentage of sample with data: Stayers versus new 

entrants—PATHS ...........................................................................................................................................357

6.6. Initial characteristics of children, their families, and communities––PATHS .......................................359

6.7. Mean scores and standard deviations for initial outcome measures of sample—PATHS .................. 361

6.8. Initial characteristics of teachers in sample—PATHS ...............................................................................362

6.9. Initial characteristics of schools in sample––PATHS ................................................................................363

6.10. Principal and teacher initial reports on use of SACD programs or activities in sample––PATHS ....364

6.11. Teacher initial reports on use of SACD materials and classroom strategies in sample––PATHS .....366

6.12. Principal and teacher initial reports on SACD professional development in sample––PATHS ........367

6.13. Impacts on teacher-reported SACD classroom activities––PATHS ....................................................... 370

6.14. Impacts on use of SACD classroom materials and teaching strategies––PATHS ................................375

6.15. Impacts on teacher-reported SACD professional development––PATHS ...........................................378

6.16. Covariates used with outcomes from each report for analysis––PATHS .............................................. 380

6.17. Adjusted minimum detectable effect sizes for impact evaluation—PATHS .........................................383

618. Impacts on child and school outcomes––PATHS ..................................................................................... 385

6.19. Impacts on growth of child outcomes––PATHS ....................................................................................... 388 
List of Tables 

xxi

Table Page 

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

7.1 The 4Rs Program (Reading, Writing, Respect, and Resolution) ............................................................................ 394

7.2. Sample––4Rs .................................................................................................................................................... 397

7.3. Data collection dates––4Rs ............................................................................................................................ 399

7.4. Consent rates, completion rates, and percentage of sample with data from each report—4Rs .........401

7.5. Consent rates, completion rates, and percentage of sample with data: Stayers versus new 

entrants––4Rs ...................................................................................................................................................403

7.6. Initial characteristics of children, their families, and communities––4Rs ...............................................405

7.7. Mean scores and standard deviations for initial outcome measures of sample––4Rs ..........................407

7.8. Initial characteristics of teachers in sample––4Rs ......................................................................................408

7.9. Initial characteristics of schools in sample––4Rs .......................................................................................409

7.10. Principal and teacher initial reports on use of SACD programs or activities in sample––4Rs ........... 410

7.11. Teacher initial reports on use of SACD materials and classroom strategies in sample––4Rs.............412

7.12. Principal and teacher initial reports on SACD professional development in sample––4Rs ................413

7.13. Impacts on teacher-reported SACD classroom activities––4Rs .............................................................. 416

7.14. Impacts on use of SACD classroom materials and teaching strategies—4Rs ....................................... 421

7.15. Impacts on teacher-reported SACD professional development––4Rs ...................................................424

7.16. Covariates used with outcomes from each report for analysis––4Rs ......................................................426

7.17. Adjusted minimum detectable effect sizes for impact evaluation—4Rs ................................................429

7.18. Impacts on child and school outcomes––4Rs............................................................................................. 431

7.19. Impacts on growth of child outcomes––4Rs ..............................................................................................434 

Chapter 8. Second Step 

8.1 Second Step .......................................................................................................................................................... 440

8.2. Sample––SS.......................................................................................................................................................443

8.3. Data collection dates––SS ..............................................................................................................................445

8.4. Consent rates, completion rates, and percentage of sample with data from each report—SS ...........447

8.5. Consent rates, completion rates, and percentage of sample with data: Stayers versus new 

entrants––SS .....................................................................................................................................................449

8.6. Initial characteristics of children, their families, and communities––SS................................................. 451

8.7. Mean scores and standard deviations for initial outcome measures of sample––SS ............................453

8.8. Initial characteristics of teachers in sample––SS.........................................................................................454

8.9. Initial characteristics of schools in sample––SS..........................................................................................455

8.10. Principal and teacher initial reports on use of SACD programs or activities in sample––SS .............456

8.11. Teacher initial reports on use of SACD materials and classroom strategies in sample––SS...............458

8.12. Principal and teacher initial reports on SACD professional development in sample––SS..................459

8.13. Impacts on teacher-reported SACD classroom activities––SS ................................................................462

8.14. Impacts on use of SACD classroom materials and teaching strategies—SS..........................................467

8.15. Impacts on teacher-reported SACD professional development––SS..................................................... 470

8.16. Covariates used with outcomes from each report for analysis––SS........................................................472

8.17. Adjusted minimum detectable effect sizes for impact evaluation—SS...................................................475

8.18. Impacts on child and school outcomes––SS...............................................................................................477

8.19. Impacts on growth of child outcomes––SS ................................................................................................ 480
List of Tables 

xxii

Table Page 

Appendix A. Analysis of Cohorts 1 and 2 

A.1. Combined Cohorts 1 and 2: Sample size and percentage of sample universe for fourth-graders, 

by program and by treatment group status .................................................................................................. A-5

A.2. Combined Cohorts 1 and 2: Initial characteristics of children, their families, and communities ....... A-7

A.3. Combined Cohorts 1 and 2: Mean scores and standard deviations for initial outcome measures ..... A-9

A.4. Combined Cohorts 1 and 2: Initial characteristics of teachers .............................................................. A-10

A.5. Combined Cohorts 1 and 2: Initial characteristics of schools ............................................................... A-11

A.6. Combined Cohorts 1 and 2: Teacher initial reports on use of SACD activities and training in SACD 

activities .......................................................................................................................................................... A-13

A.7. Cohort 1 versus combined Cohorts 1 and 2: Comparison of consent rates, completion rates, 

and percentage of sample with data ........................................................................................................... A-15

A.8. Comparison of Cohort 1 and Cohorts 1 and 2: Adjusted minimum detectable effect sizes 

for fourth-grade outcomes for combined-program evaluation ............................................................. A-17

A.9. Combined Cohorts 1 and 2: Impacts on classroom activities ............................................................... A-19

A.10. Combining Cohorts 1 and 2: Impacts on use of SACD classroom materials and teaching 

 strategies ......................................................................................................................................................... A-20

A.11. Combined Cohorts 1 and 2: Impacts on teacher-reported SACD professional development ........ A-21

A.12. Combined Cohorts 1 and 2: Significant impacts on domains of use of SACD activity overall 

and by program ............................................................................................................................................. A-22

A.13. Combined Cohorts 1 and 2: Combined-program impacts on outcomes for fourth-graders ........... A-24

A.14. Comparison of individual program’s statistically significant impacts and nonsignificant substantive 

impacts between Cohort 1 and combined Cohort 1 and Cohort 2 fourth-graders ........................... A-26

A.15. Cohort 1 and combined Cohorts 1 and 2: Comparison of program effects ....................................... A-28

A.16. Cohort 1 versus Cohorts 1 and 2: Comparison of significant impacts by subgroup ......................... A-34

A.17. Combined Cohorts 1 and 2: Impacts on growth of child outcomes from combined-program 

analysis ............................................................................................................................................................ A-40

A.18. Combined Cohorts 1 and 2: Statistically significant impacts from the growth curve analyses of 

the indvidual programs ................................................................................................................................ A-42

A.19. Combined Cohorts 1 and 2: Statistically significant impacts from the growth curve analyses of 

the subgroups ................................................................................................................................................ A-44

A.20 Sample size ranges for outcome analyses .................................................................................................. A-46

Appendix B. Technical Notes on the Development of Outcome Measures, Selection of 

Covariates, Construction of Sample Weights, and Sensitivity Analyses 

B.1. Scales and internal consistency for child outcomes, by reporter type ..................................................... B-4

B.2. Fit indexes from confirmatory factor analyses five-factor measurement model: First four waves 

of data collection, by respondent type .......................................................................................................... B-6

B.3. Child and school outcomes used for covariate selection and the percentage missing at spring 

2005 (Year 1) analysis.................................................................................................................................... B-11

B.4. Potential covariates and the percentage missing at spring 2005 (Year 1) analysis ............................... B-12

B.5. Child and school outcomes used for covariate selection and the percentage missing at spring 

2006 (Year 2) analysis .................................................................................................................................... B-16

B.6. Potential covariates and the percentage missing at spring 2006 (Year 2) analysis ............................... B-17

B.7. Child and school outcomes used for covariate selection and the percentage missing at spring 

2007 (Year 3) analysis .................................................................................................................................... B-19

B.8. Potential covariates and the percentage missing at spring 2007 (Year 3) analysis ............................... B-20
List of Tables 

xxiii

Table Page 

B.9. Covariates selected for child-reported outcomes, for the combined-program sample and programspecific samples .............................................................................................................................................. B-24

B.10. Covariates selected for primary caregiver-reported child outcomes, for the combined-program 

 and program-specific samples ...................................................................................................................... B-26

B.11. Covariates selected for teacher-reported child outcomes, for the combined-program and 

 program-specific samples ............................................................................................................................. B-28

B.12. Covariates selected for teacher-reported class and school outcomes, for the combined-program 

 and program-specific samples ...................................................................................................................... B-30

B.13. Adjusted R2 values from models using spring 2005 (Year 1) outcomes, with pretest only and 

 with the full covariate set, for the combined-program sample ............................................................... B-31

B.14. Adjusted R2 values from models using spring 2005 (Year 1) outcomes, with pretest only and 

 with the full covariate set, by site ................................................................................................................ B-32

B.15. Adjusted R2 values from models using spring 2006 (Year 2) outcomes, with pretest only and 

 with the full covariate set, for the combined-program sample ............................................................... B-33

B.16. Adjusted R2 values from models using spring 2006 (Year 2) outcomes, with pretest only and 

 with the full covariate set selected for each program, by program ........................................................ B-34

B.17. Adjusted R2 values from models using spring 2007 (Year 3) outcomes, with pretest only and 

 with the full covariate set, for the combined-program sample ............................................................... B-35

B.18. Adjusted R2 values from models using spring 2007 (Year 3) outcomes, with pretest only and 

 with the full covariate set selected for each program, by program ........................................................ B-36

B.19. Combined-program impacts in effect size units when baseline covariates are excluded from the 

models .................................................................................................................................................. B-41

B.20. Combined-program impacts in effect size units when weights are not adjusted for study 

 nonconsent or survey nonresponse ............................................................................................................ B-43

B.21. Combined-program impacts in effect size units when weights not used ............................................. B-44

B.22. Combined-program impacts in effect size units when accounting for classroom-level clustering ... B-46

B.23. Combined-program impacts in effect size units when accounting for pairwise matching ................ B-48

B.24. Combined-program impacts in effect size units when dependent variable is the pretest-posttest 

difference (gain score) ................................................................................................................................... B-50

B.25. Combined-program impacts in effect size units when posttest and pretest are treated as separate 

dependent variables ....................................................................................................................................... B-51

B.26. Combined-program impacts in effect size units when using different statistical software 

 packages ........................................................................................................................................................... B-53 

B.27. Combined-program impacts in effect size units when using imputation procedures for missing 

outcome data .................................................................................................................................................. B-56

B.28. Combined-program impacts in effect size units using averages of program-specific impacts .......... B-58

B.29. Combined-program impacts in effect size units when excluding new entrants .................................. B-61

B.30. Combined-program impacts in effect size units using restricted sets of covariates (pretest of 

outcomes only and child and primary caregiver demographic measures only) .................................... B-62
List of Figures 

xxiv

List of Figures 

Figure Page 

1.1 Conceptual model for social and character development interventions .................................................... 6 
xxv

Executive Summary 

A variety of universal school-based programs designed to help elementary schools foster positive student 

behaviors, reduce negative behaviors, and, ultimately, improve academic performance are available; however, 

more evidence from rigorous evaluations is needed to better understand their effects. Such information is 

important because the development of social competencies during middle childhood has been linked to 

adjustment to schooling and academic success, while the failure to develop such competencies can lead to 

problem behavior that interferes with success in school (Bennett et al. 2003; Carlson et al. 1999; Farrington 

1989; Fors, Crepaz, and Hayes 1999; Malecki and Elliot 2002; McCord et al. 2000; Najaka, Gottfredson, and 

Wilson 2001; O’Donnell, Hawkins, and Abbott 1995; Trzesniewski et al. 2006; Wentzel 1993).1

 

The Institute of Education Sciences (IES) and the Division of Violence Prevention in the National Center for 

Injury Prevention and Control, Centers for Disease Control and Prevention (CDC) collaborated to conduct a 

rigorous impact evaluation of programs aimed at improving students’ behavior. For this evaluation, such 

programs were termed Social and Character Development (SACD) programs.2

 Seven programs were 

evaluated, and all were coherent in that their activities were integrated and logically organized based on a 

theory of action (that differed among the programs), school-based in that they were implemented in the 

schools by school personnel, and universal in that they were to be implemented for all students in all 

elementary classrooms in a school. 

This report provides the results from the evaluation of the seven SACD programs on one cohort of students 

as they moved from third through fifth grades starting in fall 2004 and ending in spring 2007.3 The evaluation 

examined the effects on these students of the seven programs, together and separately, after 1, 2, and 3 

school years and also estimated the impact on students’ growth in social and character development over the 

3 years. Chapter 1 discusses the evaluation of the programs when considered together and provides summary 

results for each program. Chapters 2 through 8 detail the findings for each of the programs individually. 

There are two appendixes: appendix A examines whether the addition of the smaller second cohort of 

students to the study affected the results, and appendix B contains additional technical information 

concerning the analyses. 

 

1

 More information on the value of, evidence for, and theories behind these programs can be found in the second and 

third sections of chapter 1. 

2 Activities carried out by these programs in support of students’ social and character development are termed SACD 

activities. The SACD evaluation examined SACD activities intended to promote six goals (termed SACD goals) and 

behavior management. The six SACD goals included (1) violence prevention and peace promotion, (2) social and 

emotional development, (3) character education, (4) tolerance and diversity, (5) risk prevention and health promotion, 

and (6) civic responsibility and community service. 

3

 From 2005 to 2007, a smaller, second cohort of students was followed from third through fourth grades in a separate 

set of schools. This cohort is discussed in appendix A. The Executive Summary and chapters 1 through 8 describe the 

analysis of Cohort 1 only. 
Executive Summary 

xxvi

Study Design 

Through a competitive application process that included a scientific peer review, seven research teams 

received funding under cooperative agreements to evaluate one SACD program of their choosing under an 

experimental design. Each research team recruited 10 to 14 schools4 (for a total of 84), with half of the 

schools implementing one of the seven SACD programs for the 3 years of the study (the treatment group) 

and the other half continuing with their traditional SACD activities (the control group). Each team’s schools 

were randomly assigned to the treatment or control group through a stratified sampling process. All but one 

school (a control school) participated in the study for the full 3 years. Table A lists the research teams, the 

SACD program each evaluated, key features of each program, and the number of treatment and control 

schools.5

 

Under a separate peer-reviewed competition, Mathematica Policy Research, Inc. (MPR) received a contract to 

carry out a multiprogram evaluation of the seven SACD programs using (1) standardized data collection for 

all sites, (2) a common set of descriptive measures on the types and level of school-based activities (i.e., 

SACD activities) that targeted social and behavioral outcomes at both the treatment and control schools, (3) a 

common set of outcome measures, and (4) a uniform statistical analysis.6

 The evaluation examined the 

impacts of all seven programs together, assessed the effect of each program separately to identify any 

contrasts with the overall findings across programs, and tested differences in effects on student subgroups. 

Data were first collected from students, their primary caregivers, teachers, and principals in the fall of 2004 

when students were starting third grade. Follow-up data were collected at four time points: (1) end of third 

grade (spring 2005), (2) beginning of fourth grade (fall 2005), (3) end of fourth grade (spring 2006), and (4) 

end of fifth grade (spring 2007). Some students stayed at the schools in the study for the full 3 years (stayers), 

others left (leavers) and were not followed, and new students entered the schools in the first (after the initial 

data collection), second, or third years of the study (new entrants). All students in each grade of the cohort 

were included in the sample. Table B describes the student sample overall and by program for all students 

and for the treatment and control groups. The study began with about 6,600 students in third grade and 

ended with about 6,200 in fifth grade. By the end of the fifth grade, 31 percent of the original sample had left 

and new entrants made up 28 percent of the fifth-graders. There were no statistically significant differences in 

the percentages of leavers and new entrants between the treatment and control groups overall, though there 

were some such differences within the individual programs.7

 

 

4

 New York University recruited 18 schools but only 14 were included in this evaluation. 

5

 A longer version of table A with more detail on the programs appears as table 1 in chapter 1. Chapters 2 through 8 

provide greater discussion of each program. 

6

 Research teams carried out program-specific evaluations using their own procedures and measures. These are to be 

published separately in the literature. 

7

 Greater detail on the experimental design and on the sample is provided in chapter 1 under Study Design and 

Methodology. 
Executive Summary 

xxvii

Table A. Research teams, SACD programs, and number of schools 

Research team Program Program features

Number of 

Treatment 

schools

Number of 

Control 

schools

University at Buffalo, 

 State University of 

 New York

Academic and 

Behavioral 

Competencies

Program

Social skills training and behavior 

management

6 6

University of North 

 Carolina at Chapel 

 Hill

Competence Support 

Program 

Social and emotional learning, 

social dynamics training, and 

behavior management: social 

information processing, social 

problem solving, peer networks

5 5

Vanderbilt University Love In a Big World Character education: courage, 

honesty, kindness, caring

6 6

1

Oregon State 

 University

Positive Action Social and emotional learning: 

values, empathy, self-control,

social skills, social bonding, selfefficacy, honesty, goal setting

7 7

The Children’s 

 Institute

Promoting Alternative 

Thinking Strategies 

Social and emotional learning: 

emotional literacy, self-control, 

social competence, peer 

relations, interpersonal problem 

solving

5 5

New York University The 4Rs Program 

(Reading, Writing, 

Respect, and 

Resolution)

Conflict resolution and literacy: 

social problem solving, anger 

management, mediation

7 7

University of 

 Maryland, College 

 Park

Second Step Violence prevention and social 

and emotional learning: empathy, 

anger management, impulse 

control, and problem solving

6 6

1 Dropped to five after one control school became a magnet school and dropped out of the study prior to Year 2. 

SOURCE: The Social and Character Development (SACD) Research Program.
Table B. Student sample, overall and by program, for all students and for the treatment and control groups 

Year 1

(Fall 3rd grade)

Year 1

(Spring 3rd grade)

Year 2

(Spring 4th grade)

Year 3

(Spring 5th grade)

Intervention program

All

students

All 

students

Leavers

New 

entrants All 

students

Leavers

New 

entrants All 

students

Leavers

New 

entrants

# %

1

# %

2

# %

1

# %

2

# %

1

# %

2

All programs 6,567 6,597 364 6 394 6 6,415 1,457 22 1,305 20 6,249 2,067 31 1,749 28

 Treatment group 3,367 3,388 179 5 200 6 3,327 742 22 702 21 3,172 1,078 32 883 28

 Control group 3,200 3,209 185 6 194 6 3,088 715 22 603 20 3,077 989 31 866 28

ABC 879 875 43 5 39 4 877 160 18 158 18 871 289 33 281 32

 Treatment group 380 373 17 4 10 3** 367 72 19 59 16 353 135 36 108 31

 Control group 499 502 26 5 29 6 510 88 18 99 19 518 154 31 173 33

CSP 959 975 36 4 52 5 969 230 24 240 25 947 238 25 226 24

 Treatment group 476 485 20 4 29 6 474 135 28** 133 28* 458 139 29** 121 26

 Control group 483 490 16 3 23 5 495 95 20 107 22 489 99 20 105 21

LBW 986 1,007 60 6 81 8 959 228 23 201 21 944 308 31 266 28

 Treatment group 548 565 25 5** 42 7 556 110 20 118 21 567 145 26** 164 29

 Control group 438 442 35 8 39 9 403 118 27 83 21 377 163 37 102 27

PA 811 812 74 9 75 9 764 251 31 204 27 655 408 50 252 38

 Treatment group 410 416 33 8 39 9 425 108 26** 123 29 327 209 51 126 39

 Control group 401 396 41 10 36 9 339 143 36 81 24 328 199 50 126 38

PATHS 786 783 39 5 36 5 778 150 19 142 18 778 243 31 235 30

 Treatment group 377 374 21 6 18 5 373 66 18 62 17 378 114 30 115 30

 Control group 409 409 18 4 18 4 405 84 21 80 20 400 129 32 120 30

4Rs 1,202 1,194 86 7 78 7 1,109 320 27 227 20 1,065 492 41 355 33

 Treatment group 652 647 49 8 44 7 599 183 28 130 22 556 279 43 183 33

 Control group 550 547 37 7 34 6 510 137 25 97 19 509 213 39 172 34

See notes at end of table.

xxviii

Executive Summary
Table B. Student sample, overall and by program, for all students and for the treatment and control groups—Continued

Year 1

(Fall 3rd grade)

Year 1 

(Spring 3rd grade)

Year 2

(Spring 4th grade)

Year 3

(Spring 5th grade)

Intervention program

All 

students

All 

students

Leavers

New 

entrants All 

students

Leavers

New 

entrants All 

students

Leavers

New 

entrants

# %

1

# %

2

# %

1

# %

2

# %

1

# %

2

SS 944 951 26 3 33 3 959 118 13 133 14 989 89 9 134 14

 Treatment group 524 528 14 3 18 3 533 68 13 77 14 533 57 11* 66 12

 Control group 420 423 12 3 15 4 426 50 12 56 13 456 32 8 68 15

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

1

Leavers as a percentage of fall 2004 enrollment (these values are cumulative over the years).

2 New entrants as a percentage of spring enrollment.

NOTE: Abbreviations are 

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

SOURCE: The Social and Character Development (SACD) Research Program.

xxix

Executive Summary
Executive Summary 

xxx

Twenty student and school outcomes related to social and character development were used to evaluate the 

impact of the SACD programs on student outcomes and perceptions of school climate.8

 These were grouped 

into four domains: (1) Social and Emotional Competence, which contained three outcomes; (2) Behavior, 

which contained nine outcomes; (3) Academics, which contained two outcomes;9 and (4) Perceptions of 

School Climate, which contained six outcomes. Four major data collection instruments were used to collect 

the scales on which the outcome measures were based. For the Child Report, students individually completed 

a set of 10 scales during school, and these contributed to 10 outcomes that fell across all four domains. For 

the Primary Caregiver Report, primary caregivers (usually parents) filled in a written survey (or completed a 

phone interview if they failed to complete the written version) that contained 12 scales. Six of these 

contributed to three outcomes in the Behavior domain (the other 6 were used to develop measures of 

characteristics associated with greater prevalence of child behavior problems). In the Teacher Report on 

Student, each student’s teacher reported on 10 scales regarding a student’s typical behavior in the past 30 

days, and these were converted into five outcomes in the Behavior and Academics domains. In the Teacher 

Report on Classroom and School, the third-, fourth-, and fifth-grade teachers in a school reported on two 

scales that were converted into two outcomes that described their Perceptions of School Climate. In addition, 

they completed items that described the level of SACD activity in the classroom and school. Table C lists the 

20 outcomes by domain and the reports from which they were obtained. Three of the outcomes in the 

Behavior domain were measured in more than one report. Altruistic Behavior and Problem Behavior were 

separately measured three times using responses from children, primary caregivers, and teachers; Positive 

Social Behavior was separately measured twice using responses from primary caregivers and teachers. 

 

8

 The original scales and the outcomes are described in chapter 1 under Measures. 

9

 The Academics domain was planned to but did not include student grades and standardized test scores; not all research 

teams were able to collect these data from their schools and districts, and the data that were collected varied in quality. 
Executive Summary 

xxxi

Table C. Outcomes, by domain and data instrument 

Instrument

Domain/Outcome Child Report

Primary 

Caregiver 

Report

Teacher Report 

on Student

Teacher Report 

on Classroom 

and School

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction (+) 

 Normative Beliefs About Aggression (-) 

 Empathy (+) 

Behavior Domain

 Altruistic Behavior (+)   

 Positive Social Behavior (+)  

 Problem Behavior (-)   

 ADHD-Related Behavior (-) 

Academics Domain

 Engagement with Learning (+) 

 Academic Competence and Motivation (+) 

Perceptions of School Climate Domain

 Positive School Orientation (+) 

 Negative School Orientation (-) 

 Student Afraid at School (-) 

 Victimization at School (-) 

 Feelings of Safety (+) 

 Student Support for Teachers (+) 

NOTE: Abbreviations are

 ADHD: Attention deficit hyperactivity disorder

 : Outcome addressed

 Blank cell: Outcome not addressed

The +/- signs in parentheses indicate the direction of a beneficial outcome. 

SOURCE: The Social and Character Development (SACD) Research Program.
Executive Summary 

xxxii

Research Questions 

The purpose of the SACD evaluation was to determine whether seven coherent, universal, school-based 

programs improved student social and emotional competence; improved behavior, including reducing 

negative behavior; improved student achievement; and improved student and teacher perceptions of school 

climate. The evaluation considered the programs together and individually. In addition, the evaluation 

considered the programs’ impacts on students with different backgrounds that have been found to increase 

the risk of poor outcomes and possibly change student responses to the SACD programs. Also, the 

evaluation took into account findings from previous work showing that the level of implementation of a 

program affects its impact. In addition, the expectation of a positive impact on student outcomes raised the 

issue of whether and in what ways the programs increased the prevalence of SACD instruction. These issues 

led to the development of five research questions: 

1. What is the average effect of the seven universal, school-based, social and character development 

programs on social and character development instruction in the schools? 

2. What is the average effect of the seven universal, school-based, social and character development 

programs on students’ social and emotional competence, behavior, and academics, and on 

perceptions of school climate? 

3. What is the average effect of each specific social and character development program on students’ 

social and emotional competence, behavior, and academics, and on perceptions of school climate? 

4. Do the average effects of the seven universal, school-based social and character development 

programs differ by (a) students’ gender and (b) students’ initial risk factors (socioeconomic, family, 

community, and earlier child behavior)? 

5. In the treatment schools, is there an association between the level of implementation of the social 

and character development programs and student outcomes? 

Data Collection 

Three issues regarding the conduct of the data collection have implications for the analysis: (1) timing of data 

collection, (2) percentages of the sample for which data were available for analysis, and (3) students who left 

the study (leavers). 

Ideally, the first fall data collection would have started at the very beginning of the year to reduce the 

possibility that program implementation could have affected responses to the student, primary caregiver, and 

teacher surveys. For practical reasons, fall data collection was often delayed for several weeks to allow school 

populations to settle, to obtain primary caregiver consent, and to avoid disrupting planned school activities. 

As a result, program implementation began before initial data collection for six of the research teams 

(Vanderbilt University was the exception). This interval ranged from 2 to 6 weeks. In addition, at all schools 

teachers and principals received training on the intervention before the fall 2004 data collection. As a result, 

the fall 2004 reports from teachers and principals, and possibly students, are unlikely to reflect the true preintervention condition but instead capture what was being done at the beginning of the evaluation.10

 

Data were not successfully collected from all students, primary caregivers, and teachers. Data were not 

collected when written consent was not provided by primary caregivers or teachers, or when respondents 

refused to take part (even after consent had been given) or were unavailable at the time of data collection. 

Table D presents the overall consent and completion rates for each report by year and by treatment versus 

control group. This table also presents the percentages of the sample for which there are data for each report. 

These are calculated by multiplying the consent rate by the completion rate. Table D shows that 60 percent to 

 

10 Additional details on timing issues can be found under Data Collection in chapter 1. 
Executive Summary 

xxxiii

65 percent of students had data supplied by themselves over the 3 years, 46 percent to 59 percent had data 

provided by their primary caregivers, and 61 percent to 67 percent had data provided by teachers. In Year 1 

(third grade), a statistically significant larger percentage of the treatment group had data than the control 

group; there were no significant differences in Years 2 and 3. Table D also shows that data on classrooms and 

schools were obtained from 86 percent to 90 percent of the teachers.11

 

The evaluation did not follow all students originally assigned to the treatment or control groups (what is 

known as an “intent to treat” study) because data were not collected from students who left the schools. If 

the SACD programs caused differential student mobility in the treatment schools versus the control schools, 

then the impact of this mobility would be combined with the impact of the programs and the two could not 

be disentangled. Descriptive analyses, presented in chapter 1, did not identify statistically significant 

differential mobility in the treatment and control schools, but this is not definitive evidence that it does not 

exist. 

 

 

11 For additional comparisons see Consent Rates in chapter 1 and chapters 2 through 8 regarding each program. 
Table D. Consent rates, completion rates, and percentage of sample with data from each report 

Year 1

(Fall 3rd grade)

Year 1

(Spring 3rd grade)

Year 2

(Spring 4th grade)

Year 3

(Spring 5th grade)

Report Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 Student sample size 6,567 3,367 3,200 6,597 3,388 3,209 6,415 3,327 3,088 6,249 3,172 3,077

Child Report (percent)

 Primary caregiver consent rate 65 67** 63 66 68** 64 67 67 66 66 67 66

 Student completion rate 94 93* 94 96 96 96 95 96 95 96 97 96

 Students with data161 62* 60 63 65** 61 63 65 62 64 65 63

Primary Caregiver Report (percent)

 Primary caregiver consent rate 63 64** 61 64 66** 62 64 65 63 64 65 64

 Primary caregiver completion rate 92 92 92 80 80 81 78 78 77 72 71 72

 Primary caregivers with data157 59* 56 51 52 50 50 51 49 46 46 46

Teacher Report on Student (percent)

 Primary caregiver consent rate265 67** 63 66 68** 64 67 67 66 66 67 66

 Teacher completion rate 96 96 96 99 99 99 100 100** 99 98 98 99

 Students with data162 64** 61 65 67** 63 66 67 65 65 66 65

Teacher Report on Classroom and 

 School (3rd- to 5th-grade teachers)

 (percent)

 Teacher consent rate 96 98*** 92 95 97* 94 95 97 94 96 97 95

 Teacher completion rate 91 90 93 91 90 91 94 94 94 92 91 93

 Teachers with data187 88 86 87 88 86 90 90 89 89 88 89

* Treatment group significantly different than control group at the .05 level.

** Treatment group significantly different than control group at the .01 level.

*** Treatment group significantly different than control group at the .001 level.

1 Calculated as consent rate x completion rate.

2

The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

xxxiv

Executive Summary
Executive Summary 

xxxv

Initial Characteristics 

An examination of the initial characteristics of the students, families, teachers, and schools found that the 

treatment and control groups were similar on a set of observed characteristics (with the exception of the use 

of SACD activities in the schools), providing evidence that the random assignment of schools within 

programs created similar groups.12 The data for this examination were collected in fall 2004 from enrolled 

third-grade students, their primary caregivers, and their third-grade teachers. In addition, third-, fourth-, and 

fifth-grade teachers and principals in the study schools provided information about SACD activities being 

used in the classroom and school. The initial data were collected after the staff at the treatment schools began 

receiving training in their programs and, for six of the programs, after implementation had begun. 

The sample’s treatment and control groups were similar along the observed student, primary caregiver, and 

community characteristics. These included (1) student gender, race/ethnicity, and age; (2) primary caregiver 

race/ethnicity, age, employment, marital status, education, and household income; and (3) community risks 

and resources. In addition, there were no significant differences between the treatment and control groups for 

the 20 outcome measures and the five measures of initial risk, showing that the two groups of students 

started, on average, at the same place in third grade. 

The teachers in the treatment and control schools were similar in gender, race/ethnicity, years of teaching, 

and certification. The only statistically significant difference concerned degree attainment, as a larger 

percentage of treatment teachers (60%) had a master’s or doctoral degree than did control teachers (52%). 

There was no significant difference between the treatment and control schools with regard to student 

composition (race/ethnicity and school lunch eligibility), number of students, number of full-time teachers, 

Title I status, highest and lowest grades, urbanicity, and number of years the principal had been at the school. 

There was also no significant difference in teacher reports on nine dimensions concerning their school 

environment: feelings of safety, resource adequacy, student support, staff freedom to teach as desired, 

affiliation with and ties to colleagues, innovation regarding trying new approaches to teaching, professional 

interest, participatory decisionmaking, and work pressure. 

An examination of the initial level of activities to promote social and character development in the classroom 

and schoolwide, and of the materials and methods used in these activities, revealed that the control condition 

for the evaluation was not a “no treatment” control. Rather, it was a “standard practice” control condition, in 

which more than half of teachers and 80 percent to 90 percent of principals reported schoolwide and 

classroom activities designed to promote social and character development. Standard practice at the control 

schools included the use of specific materials and practices to promote social and character development, as 

well as professional development related to social and character development of students for staff. Many 

kinds of activities and strategies were provided at rates and in types and amounts similar to those reported in 

treatment schools. 

Treatment teachers reported greater use of and training in SACD activities than control teachers more often 

than would be expected by chance. This may reflect actual differences in the use of SACD activities prior to 

implementation of the programs, or it may reflect that program implementation (for six programs) and 

program training for staff had started before initial data collection.13

 

 

 

12 For greater detail, see Initial Characteristics in chapter 1. 

13 For details on these comparisons, see The Initial Level of Social and Character Development in the Schools in 

chapter 1. 
Executive Summary 

xxxvi

Analysis and Results 

Four sets of analyses were done to evaluate the impacts of the SACD programs.14 First, the effects of the 

adoption of the programs on the use of SACD activities in the classroom and school were examined on an 

annual basis for Years 1 to 3. Second, the programs’ impacts on student outcomes and perceptions of school 

climate were analyzed at the end of each of the 3 years of the study (i.e., from the start of third grade to the 

end of third grade, to the end of fourth grade, and to the end of fifth grade). Third, subgroup analyses were 

done on the same annual basis to examine whether the programs’ impacts on student outcomes, when 

combined, differed by four subgroups: (1) girls versus boys, (2) stayers versus new entrants, (3) students with 

different levels of baseline risk, and (4) students in treatment schools with high versus low fidelity of 

implementation. Fourth, a growth curve analysis was done to determine impacts on annual student growth in 

the outcomes over the 3 years. All but the third analysis were done for the combination of all seven programs 

and for each program separately. 

Annual Impacts on the Use of Social and Character Development Activities 

The SACD programs were expected to increase the use of activities to promote social and character 

development in the classroom and school. The analysis of activity use was based on data from the Teacher 

Report on Classroom and School (TRCS), which was filled out by all the third-, fourth-, and fifth-grade 

teachers. Data from the TRCS were used to create 83 SACD activity outcome measures, and these were 

grouped into six domains: (1) teacher use of any activities to promote social and character development, 

which had 16 variables; (2) teacher use of any activities associated with a specific SACD program (known as a 

“named” program), which had 14 variables; (3) teacher use of materials and classroom strategies for SACD 

activities, which had 29 variables; (4) use of schoolwide strategies, which had 6 variables; (5) teacher 

involvement in related professional development, which had 9 variables; and (6) teacher support for SACD 

efforts in the school and the prevalence of an environment conducive to the social and character 

development of students, which had 9 variables. 

To estimate the impact of the SACD programs on use of SACD activities, the statistical significance of the 

differences in means between the treatment and control groups was tested for each of the 83 SACD activity 

outcome measures.15 For the overall analysis, the data were first weighted to give equal weight to each 

program and to each school within a site. For the program-by-program analyses, each school within a site 

received equal weight (school weights differed between program analyses, as the number of schools was not 

constant among programs). 

In addition to estimating the impacts of the SACD programs on the individual outcome measures, the 

impacts on the six domains were also examined. Testing the impact on the domains was done to adjust for 

the multiple comparisons made within each domain in order to address the increased chances of finding a 

spurious outcome when more than one test was done. As a result, two sets of results were obtained: (1) the 

 

14 The analytical techniques used for each of the four sets of analyses are summarized in the following sections and 

discussed in detail in chapter 1. 

15 Three factors contributed to the decision to use differences in means. First, because of random assignment, simple 

treatment-control contrasts provided unbiased estimates of program impacts. Second, only initial values (rather than true 

baseline values) for these outcomes were available to use in a model because training (at all treatment schools) and 

program implementation (at 36 treatment schools) began before data collection. The decision to use initial values in an 

analysis partly depends on whether the initial training and implementation occurring before data collection would be 

expected to have immediate and large impacts on the outcomes (Schochet 2008b). For this analysis, the outcomes are 

based on teacher actions and so would likely be upwardly influenced by the teacher training and short period of teacher 

implementation before pretesting (in contrast to student outcomes, which would be less likely to be so influenced). For 

this reason, a model-based analysis using the initial values as covariates was not chosen. Third, preliminary analyses 

indicated no gain in precision from the inclusion of other covariates. 
Executive Summary 

xxxvii

impacts on the individual outcomes unadjusted for multiple comparisons, and (2) the impacts on the domains 

that serve as the multiple comparison adjustment. To test whether the SACD programs had a statistically 

significant impact on each of the six domains, a set of three heuristics was used and a significant effect on the 

domain was found if any one of the heuristics was met.16

 

The results provide evidence that the SACD programs increased the reported implementation of SACD 

activities in the classroom. Over the 3 years, 249 comparisons (83 outcome variables times 3 years) of 

treatment and control teachers were tested, with 12 to 13 expected to be statistically significant by chance. 

The analysis found 127 comparisons were statistically significant, with all showing greater reported use of 

SACD activities by treatment teachers. When examining the individual outcomes within each domain, the 

analysis found the following: (1) treatment teachers reported significantly greater use of SACD activities in 

the classroom for 31 of the 48 comparisons, (2) treatment teachers reported significantly greater use of SACD 

activities linked to a named SACD program for 39 of the 42 comparisons, (3) treatment teachers reported 

significantly greater use of materials and instructional methods to promote social and character development 

for 40 of the 87 comparisons, (4) there were no significant differences for the 18 comparisons made regarding 

the use of schoolwide strategies, (5) treatment teachers reported significantly greater receipt of training to 

promote social and character development for 13 of the 27 comparisons, and (6) treatment teachers reported 

significantly greater use of practices conducive to social and character development but similar attitudes 

toward it for 4 of the 27 comparisons.17

 

The results from the analysis of the individual outcomes also provide further evidence that the control group 

was a “standard practice” rather than a “no treatment” control. Over the 3 years, control teachers continued 

to report use of SACD activities. For example, over the 3 years, 86 percent to 90 percent of control teachers 

reported using a SACD activity to address any one of the six SACD goals versus 95 percent to 96 percent of 

the treatment teachers. Similarly, over the 3 years, 20 percent to 36 percent of control teachers reported using 

a SACD activity linked to a named SACD program to address any one of the six SACD goals versus 68 

percent to 72 percent of the treatment teachers.18

The findings from the analysis of the six domains are consistent with the results from the individual outcome 

analysis. For all 3 years, treatment teachers reported statistically significant greater implementation in four of 

the six SACD activity domains: (1) use of any SACD activities in the classroom, (2) use of SACD activities 

from named programs in the classroom, (3) use of materials and teaching strategies for SACD activities, and 

(4) participation in relevant professional development. No evidence was found that the programs affected the 

other two domains—the use of schoolwide strategies and attitudes and practices that create an environment 

conducive to students’ social and character development. Table E shows where statistically significant impacts 

on the use of SACD activities occurred by domain for all seven programs and for each program by year. In 

table E, a plus sign indicates a significant positive impact on the domain, and superscript numerals show 

which heuristics identified the domain as significant. 

 

16 The three heuristics included (1) determining if more than half of the individual outcomes within a domain had a 

similar and statistically significant impact, (2) doing one overall test of impact on all outcomes within a domain, and 

(3) checking for any statistically significant results among the outcomes within a domain after applying a specific 

statistical adjustment for multiple comparisons (Benjamini-Hochberg 1995) to each one. See Year-by-Year Impacts on 

Use of Social and Character Development Activities in chapter 1 for longer descriptions of these heuristics and how they 

were used. 

17 For more details on the impacts on the individual outcomes making up the domains see Year-by-Year Impacts on Use 

of Social and Character Development Activities in chapter 1. 

18 For more details on the percent of treatment and control teachers reporting on the use of SACD activities see Yearby-Year Impacts on Use of Social and Character Development Activities in chapter 1. 
Executive Summary 

xxxviii

Table E. Significant impacts on use of SACD activity domains, overall and by program 

SACD activity domain

Program

SACD 

activities

SACD activities 

linked to named 

programs

Classroom 

materials and 

strategies

Schoolwide 

strategies

Professional 

development

Attitudes 

and 

practices

Overall

 Year 1 +

1,2,3 +1,2,3 +2,3 +1,2,3

 Year 2 +

1,2,3 +1,2,3 +1,2,3 +2,3

 Year 3 +

1,2,3 +1,2,3 +1,2,3 +2,3

ABC

 Year 1 +

2

+

1

 Year 2

 Year 3 +

3

CSP

 Year 1 +

2

+

1,3 +3

 Year 2 +

3

 Year 3 +

3

LBW

 Year 1 +

2,3 +3

+

3

+

3

 Year 2 +

3

+

3

+

2

 Year 3

PA

 Year 1 +

2,3 +1,3 +3

+

3

 Year 2 +

1,3

 Year 3 +

3

+

3

PATHS

 Year 1 +

2,3 +1,3 +1,3

 Year 2 +

3

+

1,3

 Year 3 +

1,3

4Rs

 Year 1 +

1,2,3 +1,2,3 +3

+

1,3

 Year 2 +

1,3 +1,3

 Year 3 +

1,3 +1,2,3 +3

+

3

See notes at end of table.
Executive Summary 

xxxix

Table E. Significant impacts on use of SACD activity domains, overall and by program—

Continued

SACD activity domain

Program

SACD 

activities

SACD activities 

linked to named 

programs

Classroom 

materials and 

strategies

Schoolwide 

strategies

Professional 

development

Attitudes 

and 

practices

SS

 Year 1 +

3

+

1,3 +3

 Year 2 +

1,3 +1,3 +3

 Year 3 +

3

+

1,3 +3

1

Based on univariate statistical tests, at least half of the impacts were positive and statistically significant and no impact was 

negative and statistically significant. 

2

The omnibus impact for all the outcomes measured together was positive and statistically significant on the basis of a multivariate 

statistical test. 

3

At least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after 

applying the Benjamini-Hochberg (1995) procedure to adjust significance levels downward to account for the multiple testing of 

impacts. 

NOTE: Abbreviations are 

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

Abbreviations of the findings are

 +: Statistically significant beneficial impact on domain

 Blank cell: Finding of no statistically significant impact

Significance is based on p ≤ .05. No detrimental impact was found statistically significant at or below the .05 level. Description of 

SACD Activity Domains and the heuristics used to determine the statistically significant beneficial impact on the domain (for more 

detail, see the Measures section in chapter 1):

 SACD activities: based on 16 teacher-reported measures on the use of SACD activities in the classroom. 

 SACD activities linked to named programs: based on 14 teacher-reported measures on the use of SACD activities associated 

 with a named program in the classroom. 

 Classroom materials and strategies: based on 29 teacher-reported measures, 7 concerning materials used in the classroom and

 22 concerning classroom strategies. 

 Schoolwide strategies: based on six teacher-reported measures concerning strategies to promote SACD schoolwide. 

 Professional development: based on nine teacher-reported measures concerning their participation in SACD-related training. 

 Attitudes and practices: based on nine teacher-reported measures, three concerning teacher attitudes toward SACD efforts in 

 the school and six concerning school practices conducive to the social and character development of students. 

SOURCE: The Social and Character Development (SACD) Research Program.

Annual Impacts on Student Outcomes and Perceptions of School Climate 

The SACD programs were expected to improve children’s social and emotional competence, behavior, 

academics, and perceptions of school climate as measured by 20 outcome variables. One test of these 

hypotheses was to examine the year-by-year impacts of the SACD programs on these outcomes over the 3 

years as the students progressed from third through fifth grades. The examination of year-by-year impacts 

entailed three sets of analyses resulting in three sets of impacts. The first set of analyses compared the 

outcomes of treatment and control students from the fall of third grade to the spring of third grade. The 

second set compared the outcomes from the fall of third grade to the spring of fourth grade, and the third set 

compared outcomes from the fall of third grade to the spring of fifth grade. Within each set of year-by-year 

analyses, an analysis of all the programs together provided impact results for the set of seven coherent, 

universal, school-based programs, and separate analyses of each individual program provided results specific 

to each program. The combined analysis was able to detect smaller statistically significant impacts, because of 
Executive Summary 

xl

its larger sample size and the associated greater power, than the analyses of each SACD program, which were 

based on smaller samples.19

The random assignment of schools ensured that unbiased estimates of the average impacts of the SACD 

programs (relative to the social and character development activities offered in the control schools) could be 

computed as the differences in the average outcomes of students and teachers in the treatment and control 

schools. However, regression procedures were used rather than simple differences-in-means procedures to 

estimate impacts to improve the statistical precision of the estimates; to address the clustering of students 

within schools; and to adjust for differences between treatment and control group observable characteristics 

due to random selection, study nonconsent, and interview nonresponse. A hierarchical linear model (HLM) 

was used to estimate regression-adjusted impacts (Bryk and Raudenbush 1992). The basic model consisted of 

two levels that were indexed by students or teachers and by schools. The model included covariates that 

adjusted for statistically significant treatment and control differences at initial data collection. These covariates 

were chosen because they had predictive power across a broad range of outcomes. Sample weights were used 

in the analyses in order to (1) give each site equal weight in the calculation of pooled impact estimates, (2) give 

each school equal weight in each site, and (3) adjust for missing outcome data due to nonconsent and 

nonresponse. The model was estimated using data from all seven programs together (using all three types of 

weights) and individually for each program (using the second and third types of weights). A set of analyses to 

examine differences by subgroup for the combined data was done by examining the significance of a 

coefficient on an interaction term between the treatment status indicator variable and subgroup indicator 

variable (or multiple coefficients and multiple interaction terms when there were more than two subgroups) 

when the subgroup variables were included. The association of fidelity of implementation with the outcomes 

was examined in a similar way, using a fidelity indicator variable and an interaction term between the fidelity 

variable and the treatment status variable.20

 

Results from estimating the model were provided in effect sizes.21 A standard two-tailed test was used to 

determine the p-value for the coefficient of each outcome measure. Coefficients with p-values of .05 or below 

were considered statistically significant and identified as such. Impacts that were not statistically significant 

but were .25 standard deviation units or more in magnitude were identified as “substantively important,” 

following the practice used by the What Works Clearinghouse.22 Substantively important impacts identify 

effects that may be large enough to have practical importance but are not found to be statistically significant, 

potentially because of sample size constraints. 

The 20 outcome variables were grouped under a set of four domains: (1) Social and Emotional Competence, 

which contained 3 outcomes; (2) Behavior, which contained 9 outcomes; (3) Academics, which contained 2 

outcomes; and (4) Perceptions of School Climate, which contained 6 outcomes. As in the case of the SACD 

activities domains, a set of heuristics was used to test the significance of the impacts on the four outcome 

 

19 The combined analysis provided a sample size sufficient to detect student-level impacts at minimum detectable effect 

sizes ranging from 0.03 to 0.24 standard deviations (see table 1.25 in chapter 1 for details). 

20 For a discussion of the regression model, see Analysis under the section Year-by-Year Impacts on Use of Social and 

Character Development Activities in chapter 1. 

21 Effect sizes were calculated by dividing the estimated impact (the coefficient estimated by the regression model) by the 

standard deviation of the outcome measure for the control group. The standard deviation was calculated using data for 

the weighted control group. It was calculated at the time of data collection for which the effect size (impact) was 

estimated. 

22 The What Works Clearinghouse was established in 2002 by the U.S. Department of Education’s Institute of 

Education Sciences to provide educators, policymakers, researchers, and the public with a central and trusted source of 

scientific evidence of what works in education. 
Executive Summary 

xli

domains to adjust for the multiple comparisons made within each domain.23 The analysis of the year-by-year 

impacts then produced two sets of results: (1) the impacts on the individual outcomes unadjusted for multiple 

comparisons, and (2) the impacts on the domains that served as the multiple comparison adjustment.

Column 2 of table F reports the number of statistically significant and substantively important impacts found 

for the year-by-year analysis of all seven programs together. Specifically, 2 of 60 estimated impacts on the 

outcomes for the seven SACD programs combined for the 3 years were found statistically significant (versus 

3 expected by chance). The 2 statistically significant impacts were that the combined SACD programs had a 

beneficial impact on the teacher-reported measure for Student Support for Teachers in Years 1 and 2, with 

effect sizes of 0.12 and 0.16, respectively. None of the remaining 58 estimated impacts were found to be 

substantively important.24 In addition, 12 impacts on the domains were estimated (4 domains times 3 years), 

with 1 expected to be statistically significant by chance. The analysis found 2 significant negative impacts on 

the domain of Social and Emotional Competence in Years 2 and 3 (these data are not shown in a table). The 

results for the individual outcomes and the domains provide no evidence that the SACD programs improved 

student outcomes and perceptions of school climate. 

The lack of statistically significant impacts found in the combined-program analysis was not due to offsetting 

beneficial and detrimental impacts among the individual programs. For the individual SACD programs, 16 

significant impacts were found over the 3 years (9 beneficial and 7 detrimental) versus 21 expected by chance 

from the 420 statistical tests done (see column 3 of table F). In only one program did a significant impact 

occur on an outcome in more than one year, and in this case the impact was beneficial in Year 1 and 

detrimental in Year 2. In addition, 19 substantively important impacts were found (10 beneficial and 9 

detrimental). There was little replication of the substantively important impacts: one program had a 

substantively important beneficial impact on one outcome for all 3 years. Table G identifies the statistically 

significant and nonsignificant substantively important results by program, outcome, and year. It provides a 

visual view of the balance between beneficial and detrimental impacts and of the lack of persistence in 

impacts by program across the years. 

 

23 The heuristics were used to determine whether the results for the multiple outcomes within each domain showed a 

statistically significant impact on the domain as a whole. A significant effect on the domain was found if any one of the 

heuristics was met. In addition to the three heuristics mentioned in footnote 16, a fourth heuristic (not applicable to the 

analysis of teacher-reported SACD activities) was used. For this heuristic, the statistical model used to estimate impacts 

on the individual outcomes was re-estimated using a composite of all the outcome variables within a domain. The 

composite was formed by standardizing each outcome variable using its standard deviation, combining the values of the 

outcome variables, and taking the average of the final value. 

24 For the actual effect size and significance level of each outcome, see table 1.26 in chapter 1. 
Executive Summary 

xlii

Table F. Significant impacts on student outcomes, overall and by program, for all 3 years 

Domain (number of outcomes)

Combined-program analysis

(all seven programs together)

(60 impact estimates)1

Individual program analyses

(each program separately)

(420 impact estimates)2

Total

 Statistically significant outcomes 2

(Beneficial)

16

(9 Beneficial, 7 Detrimental)

 Substantively important outcomes 0 19

(10 Beneficial, 9 Detrimental)

Social and Emotional Competence (3)

 Statistically significant outcomes 0 0

 Substantively important outcomes 0 0

Behavior (9)

 Statistically significant outcomes 0 8

(6 Beneficial, 2 Detrimental)

 Substantively important outcomes 0 6

(2 Beneficial, 4 Detrimental)

Academics (2)

 Statistically significant outcomes 0 4

(1 Beneficial, 3 Detrimental)

 Substantively important outcomes 0 0

Perceptions of School Climate (6)

 Statistically significant outcomes 2

(Beneficial)

4

(2 Beneficial, 2 Detrimental)

 Substantively important outcomes 0 13

(8 Beneficial, 5 Detrimental)

1

For each year, 20 impacts (on 20 outcomes) were estimated and 1 statistically significant impact would be expected each year by 

chance (for a total of 3).

2 For each year, 140 impacts were estimated (7 programs by 20 outcomes) and 7 statistically significant impacts would be expected 

each year by chance (for a total of 21). 

NOTE: For each outcome, a finding of “beneficial” indicates the program(s) had a beneficial impact on that particular outcome; a 

finding of “detrimental” indicates a detrimental impact on that outcome. Significance is based on p ≤ .05. The number of results 

found significant was no more than expected by chance.

SOURCE: The Social and Character Development (SACD) Research Program.
Table G. Individual program statistically significant impacts and nonsignificant but substantively important impacts 

Statistically significant1 Nonstatistically significant but substantive2

Beneficial impacts Detrimental impacts Beneficial impacts Detrimental impacts

Program (Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value)

Total 9 7 10 9

 Year 1 3 2 2 0

 Year 2 5 2 6 1

 Year 3 1 3 2 8

ABC

 Year 1 Altruistic Behavior 

(TRS) (.39) (.026)

 Year 2 Academic Competence 

(CR) (.31) (.011) 

Feelings of Safety 

(TRCS) (.75) (.003)

Altruistic Behavior 

(CR) (-.20) (0.029)

Student Support

for Teachers 

(TRCS) (.27) (.276)

 Year 3 Positive Social Behavior 

(PCR) (.21) (.041)

Feelings of Safety 

(TRCS) (.31) (.235)

CSP

 Year 1

 Year 2 Problem Behavior 

(PCR) (-.21) (.042)

Altruistic Behavior 

(TRS) (.47) (.132) 

Student Afraid at School 

(CR) (-.26) (.090)

 Year 3 Altruistic Behavior 

(TRS) (-.41) (.132) 

Feelings of Safety

(TRCS) (-.36) (.246)

LBW

 Year 1 Altruistic Behavior 

(PCR) (.31) (.005) 

Student Support

for Teachers 

(TRCS) (.52) (.022)

 Year 2 Student Support

for Teachers

(TRCS) (.28) (.428)

Altruistic Behavior 

(TRS) (-.34) (.270)

See notes at end of table.

xliii

Executive Summary
Table G. Individual program statistically significant impacts and nonsignificant but substantively important impacts—Continued

Statistically significant1 Nonstatistically significant but substantive2

Beneficial impacts Detrimental impacts Beneficial impacts Detrimental impacts

Program (Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value)

 Year 3 Engagement with Learning 

(CR) (-.35) (.030) 

Positive School Orientation 

(CR) (-.33) (.047) 

Feelings of Safety

(TRCS) (-.70) (.046)

Problem Behavior 

(CR) (.31) (.223) 

Student Support for Teachers 

(TRCS) (-.26) (.543)

PA

 Year 1 Engagement with Learning 

(CR) (-.25) (.017)

Altruistic Behavior 

(TRS) (.27) (.480)

 Year 2 Positive Social Behavior 

(PCR) (.24) (.039) 

Problem Behavior 

(TRS) (-.24) (.048)

Student Support for Teachers 

(TRCS) (.28) (.113)

 Year 3

PATHS

 Year 1

 Year 2

 Year 3 Altruistic Behavior 

(TRS) (-.31) (.485) 

Feelings of Safety 

(TRS) (-.29) (.582)

4Rs

 Year 1 Academic Competence 

(CR) (-.17) (.032)

 Year 2

 Year 3 Feelings of Safety 

(TRS) (-.42) (.146) 

Student Support for Teachers 

(TRCS) (-.35) (.109)

See notes at end of table.

xliv

Executive Summary
Table G. Individual program statistically significant impacts and nonsignificant but substantively important impacts—Continued

Statistically significant1 Nonstatistically significant but substantive2

Beneficial impacts Detrimental impacts Beneficial impacts Detrimental impacts

Program (Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value)

SS

 Year 1 Feelings of Safety 

(TRCS) (.37) (.216)

 Year 2 Positive Social Behavior 

(PCR) (-.14) (.050)

Feelings of Safety 

(TRCS) (.39) (.197)

 Year 3 Feelings of Safety 

(TRCS) (.52) (.062)

1 Out of the 140 comparisons made for each year, 7 would be expected to be statistically significant at the .05 level by chance (for a total of 21).

2 Defined as impacts that were not statistically significant but were .25 standard deviation units (absolute value) or more in magnitude.

NOTE: Abbreviations are 

ABC: Academic and Behavioral Competencies Program

CSP: Competence Support Program

LBW: Love In a Big World

PA: Positive Action

PATHS: Promoting Alternative Thinking Strategies

4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

SS: Second Step

CR: Child Report

PCR: Primary Caregiver Report

TRS: Teacher Report on Student

TRCS: Teacher Report on Classroom and School

Blank cell: Finding of no impact

All impact estimates were calculated using regression models in which each school within a program was weighted equally. The standard errors of all estimates account for design 

effects due to unequal weighting and the clustering of students within schools. Significance is based on p ≤ .05. The number of results found significant was no more than expected by 

chance.

SOURCE: The Social and Character Development (SACD) Research Program.

xlv

Executive Summary
Executive Summary 

xlvi

The lack of statistically significant impacts found in the combined-program analysis was not due to offsetting 

impacts among subgroups defined by gender, stayer versus new entrant status, and different levels of initial 

student risk. A greater number of significant differences were found than would be expected by chance in the 

gender and initial student risk analyses (but not in the stayer analysis); however, the differences did not favor 

any one subgroup, and only in seven cases were they replicated across the years (these data are not shown in a 

table). The analysis by gender found 8 statistically significant differences between the genders out of 54 

possible, where 3 would be expected by chance; half of the significant impacts were beneficial impacts and 

half were detrimental impacts (one of the detrimental impacts was replicated in Years 2 and 3). The analysis 

of stayers versus new entrants found no statistically significant differences in the 36 comparisons. For the five 

types of different initial risk, the analysis found 41 statistically significant differences among three levels of 

student risk (low, average, and high) out of the 270 possible (13 or 14 would have been expected to be 

significant by chance); of these, 26 showed more beneficial impacts for higher risk students versus lower risk 

students (4 of these were replications across years), and 15 showed more detrimental impacts for higher risk 

students (2 of these were replications across years).25

The analysis of the fidelity data found little evidence of a relationship between high fidelity and more 

beneficial outcomes (these data are not shown in a table). The number of significant associations found 

between fidelity and beneficial outcomes was higher than expected by chance (5 associations found compared 

to 3 that might be expected by chance out of 54 estimated impacts) but 4 of the 5 significant results were due 

to detrimental associations between low fidelity and outcomes (rather than beneficial associations between 

high fidelity and outcomes). 

In conclusion, the analysis of the year-by-year impacts did not yield evidence that the seven SACD programs, 

combined and individually, improved student social and character development. A small number of findings 

were statistically significant (but no more than would be expected by chance, except for several of the 

subgroups) or substantively important. These results were split into similar numbers of beneficial and 

detrimental impacts; that is, the SACD programs improved some outcomes but worsened others. In the 

majority of cases, the results (both beneficial and detrimental) occurred in only 1 year and were not replicated 

across the 3 years of the study. 

Impacts on Growth of Student Outcomes 

A growth curve analysis was done to examine the change over time in the impacts on the outcomes between 

fall 2004 and spring 2007.26 The growth curve analysis used the same covariates and compared results across 

the same subgroups as the cross-sectional analyses to ensure the comparability of the results. However, it 

differed from the cross-sectional analyses by examining the estimated impacts on the trajectories of student 

outcomes over time, rather than at a point in time. The sample of students for the growth curve analysis 

included all students who were enrolled in one of the study schools during the study period and who 

completed a survey during the initial data collection or at any of the four follow-up survey points. The 

percentages of the sample with responses were similar for treatment and control schools in most survey 

waves; however, the percentages were statistically significantly higher for the treatment group on the Child 

Report and Primary Caregiver Report in fall 2004 and on the Child Report in spring 2005. There was 

considerable turnover within the sample by spring 2007. Across the three survey instruments that reported on 

students, about two-thirds of the sample had taken part in the original fall 2004 survey (66% for the Child 

Report and the Primary Caregiver Report, and 68% for the Teacher Report on Student). There were no 

statistically significant differences in the level of turnover by treatment status.27 The 18 child-level outcome 

measures collected from the Child Report, Primary Caregiver Report, and Teacher Report on Student were 

used in the growth curve analysis. The 2 other outcomes, Feelings of Safety and Student Support for Teacher, 

 

25 The section Year-By-Year Subgroup Analysis of Impacts on Students in chapter 1 provides the details on these results. 

26 See Impacts on Growth of Student Outcomes in chapter 1 for more detail on the growth curve analysis and results. 

27 For details see table 1.36 in chapter 1. 
Executive Summary 

xlvii

collected by the Teacher Report on Classroom and School were not used because they were measured at the 

teacher level. 

The impacts over time were estimated using growth curve models (with time since implementation of the 

program as the time metric) by examining treatment and control group differences in the trajectories of 

student outcomes during the study while accounting for clustering at the school level. The growth curve 

models were estimated using a three-level hierarchical linear model, where Level 1 corresponded to time since 

implementation of the program, Level 2 to students, and Level 3 to schools. The models included the same 

set of covariates that were used for the cross-sectional analysis except they excluded the initial outcome 

measure as a covariate because it was used as the outcome measure for the growth curve analysis at time 1 

(fall 2004). Similarly, sample weights were used in all analyses to (1) give each program equal weight within 

each time period; (2) give each school equal weight in each program (within each time period); and (3) give 

each time period equal weight in the analysis. However, the weights were not adjusted for consent and 

response differences across classrooms or schools because the population of students within the schools 

changed over time as students entered and left the schools. Similar to the year-by-year analyses, the growth 

model was estimated for the seven programs combined and for each program individually. In addition, the 

effects of the combined programs by subgroup were estimated by including interaction terms between 

treatment status, time since implementation, and indicators of membership in subgroups. 

Growth curve effect sizes were calculated by dividing the estimated impact of the treatment on the outcome 

growth trajectory by the standard deviation of that outcome.28 The growth curve estimates the change in 

outcomes over 1 year, so the estimated impact of the treatment on the growth trajectory equals the difference 

between the treatment group’s outcome and the control group’s outcome, on average, after 1 year of the 

study.29 The effect size measures the number of standard deviations the treatment group differs from the 

control group after 1 average year of the study, making it analogous to the effect size calculations for the 

cross-sectional analysis. 

The growth curve analysis found no significant effects of the seven SACD programs when combined (table 

H). None of the 18 estimated impacts on the trajectories of child outcomes from the average of the seven 

SACD programs were statistically significant. The estimated effect sizes all fell below .07 (absolute value). 

The lack of significant effects reflected in the analysis of the seven programs together was not found to be 

due to differences among the individual programs. The results from the analysis of individual programs 

indicate that the lack of significant impacts in the overall evaluation reflected the lack of significant impacts at 

the program level. Six statistically significant impacts were found in the program analyses, the same number 

expected by chance given that 126 impacts were estimated. Two were beneficial impacts and four were 

detrimental impacts (table I). 

 

28 The standard deviation was calculated using data for the weighted control group. It was calculated at the time of data 

collection for which the effect size (impact) was estimated. 

29 This impact estimate takes into account differences between the initial levels of the outcome for the treatment and 

control groups, differences in their covariates, and the effects of clustering at the school level. 
Table H. Impacts on growth of child outcomes from combined-program analysis 

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2

Treatment 

group

Control 

group

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 2.95 0.13 0.13 0.00 0.00 0.01 0.942

 Normative Beliefs About Aggression–CR (-) 1.23 0.10 0.07 0.02 0.04 0.01 0.115

 Empathy–CR (+) 2.41 -0.14^ -0.12 -0.02 -0.05 0.01 0.070

Behavior Domain

 Altruistic Behavior–CR (+) 1.41 -0.17 -0.16 -0.01 -0.01 0.02 0.681

 Altruistic Behavior–PCR (+) 2.32 -0.03 -0.03 0.01 0.01 0.01 0.616

 Altruistic Behavior–TRS (+) 1.40 -0.04 0.00 -0.04 -0.07 0.03 0.224

 Positive Social Behavior–PCR (+) 2.99 0.04 0.03 0.01 0.02 0.01 0.217

 Positive Social Behavior–TRS (+) 3.00 0.00 0.01 -0.01 -0.01 0.03 0.671

 Problem Behavior–CR (-) 0.24 0.10 0.08 0.01 0.02 0.01 0.351

 Problem Behavior–PCR (-) 1.58 -0.01 -0.01 0.00 0.00 0.01 0.843

 Problem Behavior–TRS (-) 1.38 0.05 0.05 0.00 0.00 0.01 0.909

 ADHD-Related Behavior–TRS (-) 1.75 -0.02 -0.01 -0.01 -0.01 0.02 0.707

Academics Domain

 Engagement with Learning–CR (+) 3.69 -0.03 -0.03 0.00 -0.01 0.01 0.707

 Academic Competence and Motivation–TRS (+) 2.87 0.02 0.03 -0.01 -0.01 0.02 0.590

See notes at end of table.

xlviii

Executive Summary
Table H. Impacts on growth of child outcomes from combined-program analysis—Continued

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2

Treatment 

group

Control 

group

Impact on 

growth3

Effect

size4

Standard 

error of 

impact

p-value of 

impact

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 3.09 -0.24 -0.21 -0.03 -0.03 0.02 0.163

 Negative School Orientation–CR (-) 1.84 0.10 0.09 0.01 0.01 0.02 0.696

 Student Afraid at School–CR (-) 2.38 -0.08 -0.08 0.00 0.00 0.02 0.956

 Victimization at School–CR (-) 0.76 -0.03 -0.02 -0.01 -0.01 0.02 0.368

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 Pertains to the estimated slope of the outcome for the treatment or control groups.

2 The average score at implementation is calculated across treatment and control groups, using regression models for adjustment on covariates. 

3 Estimated difference between the slope of the treatment and control groups.

4 The slope of the treatment group minus the slope of the control group divided by the standard deviation of the outcome for the combined-program control group (the standard 

deviation is calculated without accounting for school-level clustering or regression adjustments).

NOTE: Abbreviations are 

CR: Child Report

PCR: Primary Caregiver Report

TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. No findings were found statistically significant at or below the .05 level. All impact estimates were calculated 

using HLM 6.06.

SOURCE: The Social and Character Development (SACD) Research Program.

xlix

Executive Summary
Executive Summary 

l 

Table I. Significant impacts from the growth curve analyses of the individual programs 

Program

Significant beneficial impacts 

(Report) (Effect size) (p-value)

Significant detrimental impacts 

(Report) (Effect size) (p-value)

Total significant*1

impacts

Total 2 4 6

ABC 0

CSP Victimization at School 

(CR) (-.09) (.050)

1

LBW Positive School Orientation 

(CR) (-.13) (.016)

1

PA 0

PATHS Academic Competence 

(TRS) (.08) (.048)

1

4Rs 0

SS Engagement with Learning 

(CR) (-.09) (.021) 

Positive Social Behavior 

(TRS) (-.19) (.019) 

Empathy 

(CR) (-0.13) (.028)

3

* Significantly different from zero at the .05 level.

1 Out of the 126 comparisons made (7 programs times 18 outcomes), 6 would be expected to be statistically significant at the .05 

level by chance.

NOTE: Abbreviations are 

ABC: Academic and Behavioral Competencies Program

CSP: Competence Support Program

LBW: Love In a Big World

PA: Positive Action

PATHS: Promoting Alternative Thinking Strategies

4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

SS: Second Step

CR: Child Report

TRS: Teacher Report on Student

Blank cell: Finding of no impact

The number of results found significant was no more than expected by chance.

SOURCE: The Social and Character Development (SACD) Research Program.
Executive Summary 

li

The lack of findings from the combined-program analysis was not found to be due to differences among 

subgroups (these data are not shown in a table). The subgroup analyses did not find evidence regarding 

differential impacts of the seven SACD programs, combined, on the subgroups. For the gender subgroup 

analysis, none of the 36 estimated impacts were statistically significant. For the analysis of new entrants, 1 

impact was found to differ significantly and detrimentally from the impact on members of the original 

sample, which was no more than would be expected by chance. There were 6 outcomes out of 90 estimated 

for which growth trajectories differed significantly across initial risk levels (4 to 5 would have been expected 

by chance); for 4 of the 6 outcomes, the programs had more beneficial impacts for children with higher initial 

risk levels, while for 2 outcomes they had more detrimental impacts for children with higher initial risk levels.

Discussion 

The year-by-year analysis and the growth curve analysis did not find that the seven SACD programs 

improved student outcomes when considered together, individually by program, or for specific subgroups. 

For the combined-program analyses, the year-by-year analysis found fewer significant impacts than expected 

by chance (2 out of 60 estimated impacts) and the growth curve analysis found no significant impacts. For the 

individual program analyses, the year-by-year analysis found fewer significant impacts than expected by 

chance (16 out of 420 estimated impacts), with 9 having beneficial impacts and 7 having detrimental impacts. 

The growth curve analyses of the individual programs found the same number of impacts as expected by 

chance (6 out of 126 estimated impacts), with 2 having beneficial impacts and 4 having detrimental impacts. 

For the subgroup analyses, the year-by-year analyses found more significant impacts than expected by chance 

for gender (8 out of 54 estimated impacts) and initial risk levels (41 out of 270 estimated impacts). For 

gender, half of the significant impacts showed a beneficial impact of the intervention and half showed a 

detrimental impact. For initial risk levels, 26 showed a beneficial impact of the intervention on high-risk 

students and 15 showed a detrimental impact on high-risk students. The growth analysis found fewer 

significant impacts than expected by chance for all the subgroups except those based on initial risk levels (6 

out of 90 estimated impacts). Four of the 6 significant impacts were beneficial and 2 were detrimental. In 

sum, the SACD combined-program evaluation provides no evidence that the seven universal, schoolwide 

programs improved students’ social and character development. 

Several explanations for this finding can be considered: (1) failure of the conceptualization and design of the 

intervention, (2) weak implementation of the intervention, (3) nonsubstantial differences in the level of 

SACD activities in the treatment and control schools, and (4) methodological limitations of the evaluation. 

Failure of the conceptualization and design of the intervention refers to the possibility that the seven 

programs tested might not have altered students’ social and character development in the expected ways 

because the theories underlying them or the combinations of activities chosen to bring about the desired 

changes in students’ attitudes and behaviors were inadequate for the purpose. For example, one alternative 

view to that adopted in the SACD evaluation is that only a subset of elementary-aged children has deficits in 

social behavior and character, and these deficits require a more targeted, more intensive intervention than 

schoolwide programs can provide. Therefore, for a school-based program to be effective, a combination of 

schoolwide and targeted activities might be required for the intervention to make a significant difference in 

student outcomes (e.g., see Conduct Problems Prevention Research Group 1999). Such explanations, if 

confirmed by other studies, would lead toward focusing more effort on understanding how social and 

character development occurs among elementary-aged children, how this development can be affected, and 

what types of practices in classrooms and schools can be used to bring about desired effects. 

Weak implementation of the intervention refers to the possibility that, although the intervention might have 

been well conceived and well designed, the treatment schools did not implement the practices effectively on 

average. Weakly implemented programs may not have positive impacts on students. The SACD evaluation 

examined seven different SACD programs, each with unique features, and the fidelity rating that was used 

measured fidelity of implementation relative to the targets established for each specific intervention. In this 
Executive Summary 

lii

way, the ratings were standardized relative to each site’s program-specific benchmarks, allowing them to be 

compared across programs and years. The analysis of the fidelity data found little evidence of a relationship 

between high fidelity and more beneficial outcomes. The number of significant associations found between 

fidelity and beneficial outcomes was higher than expected by chance (5 found, with 3 expected by chance out 

of 54 estimated impacts), but 4 of the 5 significant results were due to detrimental associations between low 

fidelity and outcomes rather than beneficial associations between high fidelity and outcomes. The approach 

used to obtain comparable ratings required two compromises. First, it could not account for differences 

among the programs’ implementation standards—for example, whether programs differed in how difficult 

they were to implement. Second, it provided little information about why implementation was of a certain 

quality. As a result, the SACD evaluation fidelity measure may not provide adequate information about 

whether low fidelity might have been the reason behind the lack of significant findings. The fidelity measures 

used by each research team, which used team-chosen criteria, may provide additional information on how 

well each program was implemented in each treatment school. 

Nonsubstantial differences in the level of SACD activities refers to the possibility that the implementation 

differences between the treatment and control schools were not great enough to generate statistically 

significant differences in student outcomes. Like the treatment schools, the control schools joined the study 

with a willingness to implement a SACD program showing a willingness to promote social and character 

development. In addition, some of the sites were located in states where legislation required or promoted 

such activities. The control group represented “standard practice,” which included the reported use of SACD 

activities in the classroom. For example, 86 percent to 90 percent of control teachers reported using activities 

to promote any one of the six SACD goals. While a statistically significant larger percentage of teachers in the 

treatment schools (95% to 96%) reported conducting such activities, the 5- to 10-percentage-point 

differences may not have been large enough to lead to improved student outcomes. At the same time, the 

significant differences between treatment and control teacher reports were larger than 5 to 10 percentage 

points for other responses regarding the use of SACD activities. For example, the differences between 

treatment and control teachers regarding the use of activities from specifically named programs were 29 to 34 

percentage points across the 3 years. These results, plus the finding that treatment teachers reported greater 

use of some instructional materials and methods to promote SACD goals, provide evidence that the 

treatment teachers were making a more intensive effort to promote social and character development. 

There are three methodological limitations of the evaluation that may have contributed to the finding of no 

impacts on student outcomes. 

First, the evaluation relied on self-reported data by teachers and principals regarding the use of SACD 

activities. Observational studies were not done to validate these reports. If treatment teachers over-reported 

their use of SACD activities (possibly because they felt an expectation to report high use given that a SACD 

program was being implemented in their school), the impacts of the treatment could be misestimated. That is, 

if there really were no differences in the levels of SACD activities between the treatment and control groups, 

then a lack of effects might be expected. However, treatment teachers did differ in their reported use of any 

activities to promote one of the six SACD goals (95% to 96% reported such use) versus their reported use of 

any activities from named programs (68% to 72%) This, while not ruling out the possibility of systematic 

over-reporting, might suggest that some teachers were candid in their reporting on their use of the treatment 

programs. The research teams used observations or product measures to check implementation of their 

specific programs, and the results from this work may provide additional evidence about the potential 

importance of over-reporting of implementation by treatment teachers. 

A second methodological limitation was that student-provided data (used for 12 of the 20 outcomes) were 

not available for 36 percent to 39 percent of students, depending on the year, because primary caregivers did 

not provide written consent or students did not assent to take part in the study (primary caregiver data, used 

for three outcomes, were not available for 49% to 54% of students). It is possible that students included in 

the study differed from those not included due to an absence of data. As the study did not collect descriptive 
Executive Summary 

liii

data on the nonobserved students, the existence of such differences could not be determined, and how the 

inclusion of these students in the study would have affected the findings is unknown. Given the few 

statistically significant and substantively important impacts found with the existing sample, there would need 

to be a large and consistent impact on the nonobserved students (had they been included) to change the 

findings. For example, because two-thirds of the population who were observed received an average impact 

of zero, the nonobserved one-third would need to have received an average impact of nearly one-third of a 

standard deviation to bring the overall mean to one-tenth of a standard deviation unit. In addition, because 

the subgroup analyses did not find systematically significant impacts, there is no evidence that should these 

missing students come from one of the subgroups (e.g., higher initial risk) their inclusion would change the 

findings. 

A third methodological limitation was the sample size for the individual program evaluations. The combined 

analysis of all seven programs provided a sample size sufficient to detect student-level impacts at minimum 

detectable effect sizes (MDES) ranging from 0.03 to 0.23 standard deviations (with more than 75% of them 

below 0.10 standard deviations). The power to detect impacts at the level of the individual programs was 

more limited (the MDES ranged from 0.09 to 1.04 over the 3 years), and individual program-level effects 

might have been missed. To address this limitation at the program level, nonsignificant impacts of at least 

0.25 standard deviations were identified as substantively important results. Only a small number of these were 

identified, with an almost equal division into beneficial and detrimental effects. 

The seven programs tested in the SACD evaluation were similar in being coherent, universal, school-based 

programs to promote social and character development of third- to fifth-grade students. They were diverse 

both in their specific goals and in their approaches to promoting social and character development for all 

students. In addition, they were evaluated in diverse types of locales in which schools served communities 

with very different ethnic and socioeconomic compositions. On average, the seven programs did not improve 

students’ social and emotional competence, behavior, academic achievement, and student and teacher 

perceptions of school climate. In addition, although the numbers of schools and students in each program 

were not always sufficient to support firm conclusions at the program level, the patterns of estimated impacts 

for each program were largely similar: students’ outcomes were not affected. 
1 

Chapter 1. The Social and Character Development 

Multiprogram Evaluation 

The Social and Character Development (SACD) Research Program was created in response to the need for 

rigorous evaluations of universal school-based programs designed to help schools foster positive behaviors 

among students (e.g., behaviors illustrating good character and social-emotional competence), reduce negative 

behaviors (e.g., aggression and conduct problems), and ultimately improve students’ academic performance. 

For this evaluation, such programs were termed SACD programs, the goals they intended to promote were 

termed SACD goals, and the activities they employed to promote those goals were termed SACD activities.1

 

The SACD Research Program built on advances in understanding the development of social competence and 

problem behaviors occurring in the fields of developmental, social, community, behavioral, and cognitive 

psychology (e.g., Damon 1999; Eisenberg 2000; Larson 2000; Seligman and Csikszentmihalyi 2000), and from 

intervention and evaluation research in the areas of prevention science, public health, youth development, 

and character education (e.g., Catalano et al. 2004; Dahlberg and Simon 2006; Emler 1996; Flay 2002). Both 

federal legislation (such as the No Child Left Behind Act of 2001) and state legislation (Limber and Small 

2003) have supported prevention and intervention programs, and the SACD Research Program sought to add 

to the research base underlying such efforts. 

In 2003, the Institute of Education Sciences (IES) in collaboration with the Division of Violence Prevention 

(DVP) in the National Center for Injury Prevention and Control, Centers for Disease Control and Prevention 

(CDC) began the SACD Research Program to evaluate multiple school-based programs in a consistent 

manner. Under a competitive application process, applicants proposed programs to be evaluated as long as 

the programs (a) had either preliminary evidence of success or a history of previous implementation in 

schools, (b) aimed to influence social development and behavior outcomes, and (c) utilized a universal 

approach to be implemented in all elementary school classrooms. Applications submitted to IES were peer 

reviewed, and seven research institutions were funded under cooperative agreements for a 3-year evaluation 

of seven universal school-based SACD programs. At each of seven sites, one research team recruited and 

randomized 10 to 18 schools to either continue their current practice or implement a coherent program 

targeting social and behavioral outcomes. The programs employed activities to promote six SACD goals 

(character education, violence prevention and peace promotion, social and emotional development, tolerance 

and diversity, risk prevention and health promotion, and civic responsibility and community service) as well 

as behavior management. The programs were coherent in that their activities were integrated and logically 

organized based on a theory of action (that differed among the programs), school-based in that they were 

implemented in the schools by school personnel, and universal in that they were to be implemented for all 

students in all elementary classrooms.2

 The research institutions participating in the study and the programs 

evaluated included the following: 

 

1

 The SACD evaluation examined seven SACD programs that employed SACD activities to promote behavior 

management and six SACD goals: (1) violence prevention and peace promotion, (2) social and emotional development, 

(3) character education, (4) tolerance and diversity, (5) risk prevention and health promotion, and (6) civic responsibility 

and community service. 

2 As used here, ―elementary classrooms‖ refers to grades K-5. During the study, there were five exceptions to 

implementation in all elementary classrooms in the treatment schools. In Year 1 (the 2004-05 school year), all treatment 

schools implemented the intervention in their K-5 classrooms. In Year 2 (2005-06), one school implemented the full 

intervention in fourth grade only (the cohort grade that year), and one school could not implement in fifth grade because 

that grade had been transferred to a middle school. In Year 3 (2006-07), one school implemented the intervention in 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

2 

Research Institution Program Evaluated

University at Buffalo, The State University of 

New York

Academic and Behavioral Competencies Program 

(ABC)

University of North Carolina at Chapel Hill Competence Support Program (CSP)

Vanderbilt University Love In a Big World (LBW)

Oregon State University Positive Action (PA)

The Children’s Institute Promoting Alternative Thinking Strategies (PATHS)

New York University The 4Rs Program (Reading, Writing, Respect & 

Resolution) (4Rs)

University of Maryland Second Step (SS)

In addition to conducting their own program evaluations, applicants were expected to participate in the 

collection of data, facilitated by an independent evaluation contractor. Under another competitive peer-review 

process, Mathematica Policy Research, Inc. (MPR) was selected to independently evaluate the seven 

programs. 

The evaluation of the SACD programs thus occurred at both the program-specific and multiprogram level. In 

the program-specific studies, each research team collected data to answer questions that were specific to the 

program under evaluation. For example, in the program-specific studies, investigators sought to answer 

questions about the effects of the program on targeted proximal outcomes and about the importance of 

program implementation quality. Research teams designed their own studies and were expected to report 

their findings in the relevant literature. 

For the multiprogram evaluation, an independent team from MPR assessed the effect of the seven SACD

programs through an evaluation of all programs together using (1) standardized data collection for all sites; 

(2) a common set of descriptive measures on the types and level of SACD activities taking place at both the 

treatment and control schools; (3) a common set of outcome measures grouped under four outcome 

domains: Social and Emotional Competence, Behavior, Academics, and Perceptions of School Climate; and 

(4) a uniform statistical analysis. The effects of each program were also assessed separately to identify any 

contrasts with the overall findings across programs. The evaluation followed one cohort of third-grade 

students at 84 schools for 3 years through fifth grade. Initial data were collected in fall 2004, and follow-up 

data were collected in spring 2005, fall 2005, spring 2006, and spring 2007. In addition, four research teams 

added a second cohort of third-graders at 12 schools, where initial data were collected in fall 2005 and followup data collected in spring 2006 and spring 2007 (at the end of third and fourth grades) in order to increase 

sample sizes and the associated power of the program-specific analyses. 

This report contains the results for the 3-year multiprogram evaluation and focuses on the first cohort of 

students. It details the impacts of the seven SACD programs after 1, 2, and 3 school years of implementation 

in the treatment schools and also estimates the effect of the programs on students’ growth in social and 

character development over the 3 years. Chapter 1 discusses the evaluation of the seven programs when 

considered together as a subset of universal school-based programs that aim to affect student social 

development and behavior. Chapter 1 also provides summary results for each program. The seven chapters 

that follow detail the findings for each individual program. There are two appendixes: appendix A examines 

whether the addition of the smaller second cohort of students for four of the seven programs affected the 

results and appendix B contains additional technical information concerning the analyses. 

 

third through fifth grades, one school implemented in fifth grade only (the cohort grade that year), and one school’s fifth 

grade attended a middle school so it was not able to implement the intervention with its cohort grade. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

3 

Social Development and Risk Prevention 

By 8 to 12 years of age, children typically have well-developed skills of self-regulation, perspective taking, 

emotional understanding, and caring for others. Middle childhood is also a time when children’s beliefs about 

aggression and conflict resolution skills are developing (e.g., Samples and Aber 1998). Because these social 

competencies underlie many primary developmental tasks of middle childhood (e.g., development of mutual 

friendships), researchers predict that deficits in these social skills will likely lead to the emergence of problem 

behaviors. 

Research points to several key intrapersonal factors that increase a child’s risk for behavior problems, 

including children’s beliefs about the acceptability of aggression (Guerra et al. 1995), lack of problem-solving 

skills (Slaby and Guerra 1988), attention problems and hyperactivity (Farrington 1989; Rey, Sawyer, and Prior 

2005), and attribution of aggression to others’ innocuous behavior (Dodge and Coie 1987). Family, peer, and 

community risk factors also increase the likelihood of behavior problems. These risk factors include poor 

parental supervision and monitoring (Gorman-Smith et al. 1996), association with aggressive peers 

(Farrington and Hawkins 1991; Henry, Tolan, and Gorman-Smith 2001), community disorganization 

(Sampson 2000), and exposure to community violence (Schwartz and Proctor 2000). 

Researchers have hypothesized that programs targeting the risk factors described above will be effective in 

reducing problem behaviors among children. Over time, however, there has been increasing recognition that 

prevention efforts should not focus on risk alone but take an integrative approach to reducing risk and 

simultaneously fostering protective factors by developing social competence in young people (Catalano et al. 

2004; Weissberg, Kumpfer, and Seligman 2003). Many protective factors that are hypothesized to promote 

social competence are also thought to inoculate children from developing externalizing behavior problems. 

Although the research literature on protective factors that relate to risk has been slow in developing, evidence 

suggests that socially adaptive interpersonal problem-solving (Wentzel 1991), empathy (Schultz, Izard, and 

Bear 2004), and parental use of positive rewards and encouragement to support appropriate behavior 

(Gorman-Smith et al. 2000) are related to a lower occurrence of problem behaviors, such as aggression and 

delinquency. 

From an interventionist and public health perspective, schools are an obvious site for addressing positive 

youth development and prevention efforts because of universal access to children over time that, in turn, 

allows for efficient distribution of these efforts to a comprehensive population of youth. Elementary school is 

thought to be a critical time for prevention; 7 is the average age at which students start down the path of 

problem behavior (Thornberry, Huizinga, and Loeber 2004). Most types of antisocial behavior are already 

evident by third grade (Loeber and Stouthamer-Loeber 1998), and problem behaviors in childhood are 

predictive of violence and other antisocial behavior later in adolescence and adulthood (Moffitt 1993). 

Programs that address social development and behavior are also attractive to school administrators because 

of the prevalence of problem behaviors that teachers perceive as interfering with their ability to teach and 

students’ ability to learn (Mansfield et al. 1991). For example, in the 2005-06 school year, 21 percent of 

primary schools reported occurrences of student bullying at least once per week, 12 percent of schools 

reported student acts of disrespect for teachers once per week, and 28 percent of 12- to 18-year-old students 

reported that they had been bullied at school during the 6 months prior to the survey (Dinkes, Cataldi, and 

Lin-Kelly 2007). A substantial body of literature has shown that disruptive classroom behavior, conduct 

problems, aggression, delinquency, and substance use are associated with poor academic achievement and 

with a lack of student feelings of school connectedness and involvement (Bennett et al. 2003; Farrington 

1989; Fors, Crepaz, and Hayes 1999; McCord et al. 2000; Najaka, Gottfredson, and Wilson 2001; O’Donnell, 

Hawkins, and Abbott 1995; Trzesniewski et al. 2006). On the positive side, social competencies have been 

linked with higher levels of achievement and school adjustment (Carlson et al. 1999; Malecki and Elliot 2002; 

Wentzel 1993). 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

4 

One school element that has gained increasing attention in recent years is the creation of a positive school 

climate. Researchers have theorized that the development of a warm, caring community within a school might 

reduce student problem behaviors, such as aggression and bullying (Battistich et al. 1997; McEvoy and 

Welker 2000; Orpinas and Horne 2006); however, research directly testing this hypothesis is limited. Overall, 

the abundance of evidence supporting relations among social competence, problem behavior, and 

achievement, and the development of theories surrounding the potential effects of a positive school climate 

on student outcomes, has motivated the development and, in turn, evaluation of school-based programs. 

Universal School-Based Social and Character Development Programs 

Schools commonly use support services, intervention curricula, and discipline management strategies to 

promote social and character development and prevent problem behavior (Crosse et al. 2001; Gottfredson 

and Gottfredson 2001). Many of these programs are targeted programs; that is, they focus on addressing 

behavior problems for children who are at risk or for children who are already exhibiting adjustment 

difficulties (see Mytton et al. 2006 for a review of the effectiveness of such approaches). Alternatively, a 

universal approach can be taken to influence the attitudes and behaviors of all children in the general student 

population. Universal approaches focus on preventing problems before they occur by addressing factors that 

place youth at risk for problem behavior and promoting factors that foster positive youth development 

(Walker and Shinn 2002). 

Universal school-based prevention programs have followed different theoretical traditions. Character 

education programs teach moral values through the curriculum and attempt to create a climate of caring and 

moral discipline (Lickona 1993). Social and emotional learning programs stress goal-setting, emotion 

identification, responsible decisionmaking, perspective-taking, and effective interpersonal skills, within a 

caring and engaging school climate (Greenberg et al. 2003; Stage and Quiroz 1997). Through systematic 

assessment and evaluation of behavior, Behavior management approaches utilize learning theory to apply 

strategies such as positive reinforcement, consistent schoolwide discipline, and antecedent control to 

minimize disruptive and aggressive behaviors and promote prosocial behaviors in all settings in the school 

(Kazdin 2001). Primary youth violence prevention approaches identify the individual, relationship, and 

environmental factors that place youth at risk for engaging in violence-related behaviors. They implement 

strategies that modify those risk factors, such as by changing attitudes, beliefs, behaviors, and environments, 

to disrupt developmental pathways to violence (e.g., through cognitive-behavioral and social skill training, 

changing peer group norms, and modifying school settings via teacher training and setting school 

administrative policies and rules; Dahlberg 1998). What each of these universal school-based approaches has 

in common is the desired goal of promoting students’ social development and reducing engagement in 

problem behavior. As a group, these approaches aim to improve social and character development. 

Evidence of the effectiveness of universal school-based programs to promote social and character 

development is mixed. Some recent meta-analyses suggest that universal school-based approaches to 

preventing aggressive behavior can be effective (Hahn et al. 2007; Wilson and Lipsey 2007). Other rigorous 

studies and meta-analyses have shown significantly positive effects on some outcomes of interest but 

indeterminate or non-statistically significant effects on other outcomes, no significant effects at all, and even 

potentially negative effects (e.g., CPPRG 1999; Grossman et al. 1997; Flannery et al. 2003, Park-Higgerson et 

al. 2008, Merrell et al. 2008). 

In addition, some universal, school-based SACD programs have yet to be evaluated, while others have been 

evaluated using nonrigorous methodology, design, and analysis (e.g., small sample sizes and low statistical 

power, severe attrition, lack of randomization to condition, and inappropriate level of analysis). For example, 

at the time this report was written, the What Works Clearinghouse had reviewed 55 evaluation studies of 13 

character education programs and found that two-thirds of these studies either did not meet Clearinghouse 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

5 

standards or met its standards with reservations (What Works Clearinghouse, n.d.-a).3 The What Works 

Clearinghouse also identified 14 additional character education programs for which no evaluations were 

found. 

Given the methodological limitations of many previously conducted studies, the mixed findings from 

previous rigorous evaluations, and the theoretical rationales for conducting school-based programs, 

researchers have called for further evaluation of prevention programs in the elementary school years to assist 

in our understanding of whether middle childhood is an optimal time for prevention (Tolan, Guerra, and 

Kendall 1995). In particular, there has been a call for more rigorous evaluation of school-based programs in 

which a universal approach to development and prevention is used, including random assignment to 

condition; documentation of control group practice; reliable, valid, and age-appropriate measures from 

multiple data sources; short- and long-term follow-up of students; sufficient statistical power; appropriate 

multilevel analyses; and exploration of mediator and moderator variables (Farrell and Camou 2006; 

Greenberg 2004; Weissberg, Kumpfer, and Seligman 2003). 

One difficulty with interpreting the results of the existing evaluation literature has been the use of different 

measurement strategies to assess similar social and behavioral outcomes. That is, similar outcomes have been 

assessed using different measures across evaluations. Meta-analyses have been used as one approach to 

summarizing the effectiveness of universal programs on behavior by clustering outcomes by conceptual 

domains (e.g., aggression, disruptive behavior; Wilson and Lipsey 2007). Another approach, one taken by the 

SACD Research Program, is to use a common set of measures to evaluate multiple programs on the same 

outcomes of interest. 

Universal SACD programs both share and differ in their structures, approaches, and theories of change. To 

assist in developing hypotheses and selecting measures for the multiprogram evaluation, a global conceptual 

model was developed within which all seven programs could be included, although none would strictly follow 

the entire model. The model was used to help in the design of the multiprogram evaluation but was not to be 

tested itself. Figure 1.1 shows the set of common structures and approaches on which the seven SACD 

programs draw. It identifies a set of moderating factors expected to influence the success of all programs; 

these include characteristics of the student, the student’s family, the school, and the community, as well as the 

program. The model shows the four outcome domains (Social and Emotional Competence, Behavior, 

Academics, and Perceptions of School Climate) expected to be affected by the SACD programs and the 

specific outcomes that were to be examined within each domain. The pathways in the model show that, in 

general, the programs were expected to increase the proximal domains of improving social competencies and 

perceptions of school climate and, directly and indirectly through the proximal domains, to affect the distal 

domains of improving students’ behavior and academics. 

Research Questions 

All seven of the programs in the SACD multiprogram evaluation were universal, school-based interventions 

that targeted social and behavioral outcomes. The evaluation examined both the average effect of all seven 

programs combined and the average effect of each specific program on student outcomes and on student and 

teacher perceptions of school climate. 

The primary research questions for the evaluation were as follows: 

1. What is the average effect of the seven universal, school-based, social and character development 

programs on social and character development instruction in the schools? 

 

3

 The What Works Clearinghouse was established in 2002 by the U.S. Department of Education’s Institute of Education 

Sciences to assess the rigor of research evidence on the effectiveness of educational interventions. 
Figure 1

.1. Conceptual model for social and character development interventions

6

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

7 

2. What is the average effect of the seven universal, school-based, social and character development 

programs on students’ social and emotional competence, behavior, and academics, and on 

perceptions of school climate? 

3. What is the average effect of each specific social and character development program on 

students’ social and emotional competence, behavior, and academics, and on perceptions of 

school climate? 

As noted in figure 1.1, a set of moderating factors was expected to affect the success of any SACD program. 

The potential impact of these factors raised secondary research questions concerning the differential impact 

of a SACD program on student subgroups and by the quality of implementation of the program. 

Subgroups of students may differ enough in their behaviors that the impact of a SACD program may have 

differential impacts on those subgroups. Significant gender differences exist in the frequency of cooperative, 

prosocial behaviors and problem behaviors among young children. Aggression and hyperactive behaviors 

tend to be more common among boys. Among violence prevention programs that have examined whether 

programs are differentially effective for boys and girls, four have indicated significant effects for boys but not 

for girls (Farrell and Meyer 1997; Flay et al. 2004; Kellam et al. 1998). Given these observed differences, the 

average effects of the seven programs were expected to differ by student gender. 

Previous research has also documented that certain family and community characteristics are associated with 

greater prevalence of behavioral problems for children. Regarding the family, children from poor families are 

at greater risk for behavioral or emotional problems (Brooks-Gunn and Duncan 1997) and show difficulties 

with aspects of social competence such as self-regulation and impulsivity (Takeuchi, Williams, and Adair 

1991). Poor parental supervision and monitoring are associated with the development of problem behaviors 

in children (Gorman-Smith et al. 1996), and parental involvement has been linked to children’s overall 

behavior in school, motivation to learn, grades and test scores, and long-term achievement (Griffith 1996; 

Grolnick, Ryan and Deci 1991; Shaver and Walls 1998; Zellman and Waterman 1998). At the community 

level, disorganization (Sampson 2000) and exposure to community violence (Schwartz and Proctor 2000) are 

associated with greater risk for behavior problems. Neighborhood disadvantage has also been consistently 

linked with increased levels of child aggression, emotional problems, inattention problems, and conduct 

problems (Caspi et al. 2000; Gershoff and Aber 2006). As a consequence, the average effect of the seven 

programs was expected to differ for children experiencing different levels of family and community risk. 

Early childhood conduct problems and aggressive behaviors are another risk factor for later problem 

behaviors such as fighting, delinquency, and drug involvement (Dodge and Pettit 2003). Meta-analytic studies 

that have examined the effect of school-based programs on problem behaviors indicate larger effect sizes for 

programs targeting ―high-risk‖ populations than general school populations (Wilson, Gottfredson, and 

Najaka 2001; Wilson, Lipsey, and Derzon 2003). 

The differential behavior found among certain student subgroups and some evidence of differential responses 

by these subgroups to programs led to the fourth research question: 

4. Do the average effects of the seven universal, school-based social and character development 

programs differ by (a) students’ gender and (b) students’ initial risk factors (socioeconomic, family, 

community, and earlier child behavior)? 

Fidelity of program implementation has been found to strongly affect the impacts of interventions of all 

types. A number of studies show that programs delivered with high fidelity lead to better outcomes than 

programs implemented with lower fidelity (e.g., Crandall et al. 1982; Stringfield et al. 1997). This raised a fifth 

research question: 

5. In the treatment schools, is there an association between the level of implementation of the social 

and character development programs and impacts on student outcomes? 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

8 

The Social and Character Development Research Program 

In 2003, IES, in collaboration with DVP, conducted a peer-reviewed research grant competition in which 

applicants proposed to (1) implement a universal, school-based intervention for elementary school children 

intended to promote positive behaviors and attitudes and/or reduce negative or antisocial behaviors and 

attitudes and (2) evaluate the intervention using a cluster randomized design in which schools would be the 

unit of assignment. Applicants were encouraged to both identify measures that assessed the specific attitudes 

and behaviors that their proposed intervention was designed to improve and to take into account the fact that 

the implementation and evaluation of the intervention would take place within the context of a larger 

multiprogram evaluation in which a third party would collect comparable outcomes from all schools 

participating in the evaluation. That is, in addition to whatever measures the applicants proposed to use, all of 

the interventions in the SACD Research Program would be evaluated on a set of measures that would be 

collected by an evaluation team that would be separate from the research teams involved in implementing the 

interventions. 

Seven research teams were selected through a peer-review process, and they entered into cooperative 

agreements with IES to randomize schools to either implement an elementary school SACD program or 

continue with their existing educational practice. Each research team evaluated one program in its chosen site 

by following one cohort of students from third through fifth grades, with the treatment students receiving the 

program in all 3 years (students in the other elementary grades in the treatment schools also received the 

program but were not part of the evaluation). Six of the programs had separate curriculum units for each 

grade, and one program (ABC) had a single standard unit with modifications for each grade. Programs were 

not randomized across teams or their sites; schools were randomized to treatment or control conditions 

within a team and its site. The six panels of table 1.1 identify each program and its research team4

 and sites; 

describe general program characteristics, strategies, and type and amount of professional development 

provided; outline the SACD goals addressed by program activities; and note the outcomes addressed by the 

evaluation. The key point here is that the panels demonstrate how the seven programs drew from a common 

set of characteristics, techniques, goals, and outcomes, while differing in which of these each program used or 

addressed. For example, Panel 2 shows that four programs included behavior management while three did 

not, and that all provided either a manual or scripted lessons but that programs differed in the degree of 

teacher adaptation. Panel 3 shows that programs used many of the same activities and often addressed similar 

content (e.g., problem solving), but each program had specific content that it alone covered. Panel 4 shows 

that all programs used the combination of pre-implementation training plus ongoing consultation but that 

training time, nonteacher personnel included, and frequency and method of consultation varied among them. 

Panels 5 and 6 note that some programs addressed all the SACD goals and outcomes (described later in this 

chapter) while others focused on a selection of these. More detailed descriptions of the programs are included 

in the program-specific chapters of this report. 

 

4

 Members of two of the seven research teams had developed the interventions they were evaluating (University at 

Buffalo, The State University of New York, and the University of North Carolina at Chapel Hill), and one team included 

a developer of its intervention (Oregon State University). 
Table 1.1. Social and Character Development programs 

Panel 1: The seven programs 

Program/Source Features

Curriculum 

structure Research team Site

Academic and Behavioral Competencies Program Social skills training and behavior 

management

Classroom 

curriculum and 

targeted 

component

University at 

Buffalo, State 

University of 

New York

Buffalo, New York, 

and two charter 

schools

 Center for Children and Families

 University at Buffalo, State University of New York

Competence Support Program Social and emotional learning, social 

dynamics training, and behavior 

management: social information 

processing, social problem solving, 

peer networks

Classroom 

curriculum and 

intensive teacher 

training

University of 

North Carolina

at Chapel Hill

Hoke and Wayne 

Counties, North 

Carolina

 School of Social Work

 University of North Carolina-Chapel Hill

Love In a Big World Character education: courage, honesty, 

kindness, caring

Classroom 

curriculum and 

whole-school 

approach

Vanderbilt 

University

Maury and 

Murfreesboro 

Counties, 

Tennessee

 Love In a Big World

 Nashville, Tennessee

Positive Action Social and emotional learning: values, 

empathy, self-control, social skills, 

social bonding, self-efficacy, honesty, 

goal setting

Classroom 

curriculum and 

whole-school 

approach

Oregon State 

University

Chicago, Illinois

 Positive Action, Inc.

 Twin Falls, Idaho

Promoting Alternative Thinking Strategies Social and emotional learning: 

emotional literacy, self-control, social 

competence, peer relations, 

interpersonal problem solving

Classroom 

curriculum

The Children’s 

Institute

Robbinsdale, 

Minnesota, and 

Rochester and 

Rush-Henrietta, 

New York

 Channing Bete Company

 South Deerfield, Massachusetts

The 4Rs Program (Reading, Writing, Respect, and 

 Resolution)

Conflict resolution and literacy: social 

problem solving, anger management, 

mediation

Classroom 

curriculum 

New York 

University

New York City, 

New York

 Morningside Center for Teaching Social Responsibility 

 New York, New York

Second Step Violence prevention and social and 

emotional learning: empathy, anger 

management, impulse control, and 

problem solving

Classroom 

curriculum 

University of 

Maryland, 

College Park

Anne Arundel 

 Committee for Children County, Maryland

 Seattle, Washington

See notes at end of table.

9

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.1. Social and Character Development programs—Continued

Panel 2: General characteristics

Program

Target 

population

Program components Level of 

integration

Flexibility

Peer Parent Classroom Schoolwide Community Training Manualized Adaptability

ABC Universal 

and 

targeted

In class

and out 

of class

Contact Lessons and 

behavior 

management

Planned 

events, reward 

programs, 

schoolwide 

rules, 

discipline 

policies

None or 

not major 

focus

Pretraining 

and 

ongoing

Add-on 

curriculum 

and 

schoolwide 

activities

Manual 

includes 

modules for all 

program 

components

Program may 

be 

individualized 

by school, 

classroom, and 

student

CSP Universal In class No contact Lessons, 

behavior 

management, 

and social 

dynamics

None or not 

major focus

None or 

not major 

focus

Pretraining 

and 

ongoing for 

teachers 

and 

counselors

Add-on 

curriculum

Scripted 

lessons for 

teachers

Teachers may 

adapt program 

to setting

LBW Universal In class

and out 

of class

Contact and 

involvement

Lessons and 

behavior 

management

Planned 

events, 

modeling, 

program 

artifacts

None or 

not major 

focus

Pretraining 

and 

ongoing

Add-on 

curriculum 

and 

schoolwide 

activities

Curriculum 

guidebook

Less adaptable

PATHS Universal In class Contact and 

involvement

Lessons Planned 

events,

program 

artifacts

None or 

not major 

focus

Pretraining 

and 

ongoing

Add-on 

curriculum 

and 

schoolwide 

activities

Scripted 

curriculum 

guidebook

Less adaptable

See notes at end of table.

10

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.1. Social and Character Development programs—Continued

Panel 2: General characteristics—Continued

Program

Target 

population

Program components Level of 

integration

Flexibility

Peer Parent Classroom Schoolwide Community Training Manualized Adaptability

PA Universal In class

and out 

of class

Training, 

contact, 

and 

involvement

Lessons and 

behavior 

management

Planned events,

includes 

program 

artifacts PA 

coordinator and 

PA committee

None or not 

major focus

Pretraining 

and 

ongoing

Add-on 

curriculum 

and 

schoolwide 

activities

Curriculum 

guidebook with 

scripted lessons 

and materials, 

schoolwide 

guidebooks, 

parent manual, 

and support 

staff manual

Program staff 

and principal 

adapt activities 

to needs and 

setting

4Rs Universal In class Contact 

and 

involvement

Lessons None or not 

major focus

None or not 

major focus

Pretraining 

and 

ongoing

Core 

curriculum 

(language 

arts)

Curriculum 

guidebook

Less adaptable

SS Universal In class Contact Lessons Program 

artifacts and 

modeling

None or not 

major focus

Pretraining 

and 

ongoing

Add-on 

curriculum

Curriculum 

guidebook

Character 

development 

planning teams

See notes at end of table.

11

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.1. Social and Character Development programs—Continued

Panel 3: Description of strategies 

Program

Classroom

Supplement to 

classroom

Schoolwide 

activities

Lessons Strategies

Who 

delivers

Activities 

and tools Content Frequency

Who 

delivers

Activities 

and tools Frequency

ABC Teacher Social skill of 

the day 

description and 

classroom or 

schoolwide 

role-play

Social skills, 

problem solving, 

classroom 

management, 

peer 

relationships

Daily 5-10 

minutes plus 

reinforcement; 

twice a week 

45-minute peer 

program 

Teacher Recognition 

and 

reinforcement 

of good 

behavior 

(e.g., positive 

notes, 

behavior 

reports, time 

out, fun 

activities)

Daily Homework, 

peer 

mediation, and 

parenting 

program

Schoolwide 

rules, recording 

of rule violations, 

positive 

affirmation, 

afterschool 

program

CSP Teacher Story reading, 

discussion, 

role-playing, 

worksheets, 

games, artifacts 

(e.g., turtle 

puppet), art 

projects

Problem 

solving, emotion 

identification 

and regulation, 

identifying 

social cues, 

goal formation, 

choosing 

options, and 

behavior 

regulation

45 minutes, 

once per 

week; 28 

lessons

Teacher Prevention of 

social 

hierarchies 

that promote 

conflict; social 

reinforcement 

for positive 

behavior

Daily None None

LBW Teacher Story reading, 

writing, 

interdisciplinary 

activities, 

rewards, singing

Character traits 

and moral 

virtues

Daily 10- to

15-minute 

lessons, 30 

weeks

Teacher Recognition 

and 

reinforcement 

of good 

behavior; 

Modeling of 

character 

traits

Daily Parent 

newsletters

Weekly 

announcements; 

occasional 

assemblies; two 

service projects; 

program artifacts

See notes at end of table.

12

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.1. Social and Character Development programs—Continued

Panel 3: Description of strategies—Continued

Program

Classroom

Supplement to 

classroom

Schoolwide 

activities

Lessons Strategies

Who 

delivers

Activities 

and tools Content Frequency

Who 

delivers

Activities 

and tools Frequency

PATHS Teacher Direct 

instruction, 

storytelling, 

discussion, 

role-playing, 

utilization of 

artifacts (e.g., 

posters, turtle 

puppet), 

worksheets

Emotion 

understanding 

and control, 

behavior 

regulation, 

problem solving, 

making friends

20-30 

minutes per 

day, 3-5 days 

per week

Teacher Modeling of 

skills

Daily Parent 

newsletter 

and 

engagement 

in homework

End-of-year 

PATHS party 

and program 

artifacts

PA Teacher Direct 

instruction, 

story reading, 

writing, roleplaying, 

discussion, 

singing, games, 

worksheets, 

puppets, plays 

and poems

Self-concept, 

physical health, 

intellectual 

growth, 

emotional and 

behavioral selfregulation, 

getting along with 

others (social 

skills), selfhonesty, selfimprovement, 

goal setting, 

character traits

15- to 20-

minute 

lessons, 4 

days per 

week

Teacher Teaches, 

practices, 

recognizes, 

and models 

positive 

actions with 

curriculum 

and climate 

activities and 

materials

Daily Parent 

newsletters; 

parent manual 

with PA 

activities, 

letters, and 

strategies; 

and parent 

night

Occasional 

assemblies; 

service projects; 

PA days and 

year-end event; 

Principal climate 

program with 

reinforcement 

recognition 

activities

See notes at end of table.

13

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.1. Social and Character Development programs—Continued

Panel 3: Description of strategies—Continued

Program

Classroom

Schoolwide 

activities

Lessons Strategies

Who 

delivers

Activities 

and tools Content Frequency

Who 

delivers

Activities 

and tools Frequency

Supplement to

classroom

4Rs Teacher Story reading, 

role-playing, 

discussions, 

reflections, 

sharing 

exercises, 

brainstorming,

songs, 

worksheets

Literacy focus; 

building 

community, 

feelings, 

listening, 

assertiveness, 

problem solving, 

diversity, making 

a difference

1-hour 

lessons at 

least once 

per week; 

31 lessons

Teacher Modeling of 

skills

Daily Parent-child 

connections 

homework and 

parent 

workshops

None

SS Teacher Storytelling, 

discussion, 

videos, 

role-playing, 

anticipation, 

recall, 

learning 

points

Empathy, 

impulse control, 

problem solving, 

anger 

management

30-minute 

lessons, 

1-2 days 

per week

Teacher Modeling of 

and 

reinforcement 

of skills

Weekly 

lessons: 

daily 

generalization

Take-home 

letters, 

homework, 

family guide 

and video

Program artifacts, 

training of school 

staff

See notes at end of table.

14

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.1. Social and Character Development programs—Continued

Panel 4: Professional development

Program

Pre-implementation Ongoing consultation

Teachers Other Teachers Other

Content Duration Content Duration Content Duration Content Duration

ABC Behavioral 

management 

techniques, 

schoolwide rules, 

discipline 

policies, 

classroom 

management 

policies

9 hours Staff receive 

teacher training

9 hours Teacher 

consultation, 

and coaching 

on use of 

classroom 

management

Monthly or 

more 

frequently per 

teacher or 

school 

request

School 

consultation on 

policies and 

procedures

As needed, with 

regular followup

CSP Training on 

curriculum 

implementation, 

behavior 

management, 

and social 

dynamics

4 hours Principal 

consultation on 

development of 

schoolwide 

management 

policy

As requested Consultations 

on use of 

program 

materials

2 times 

monthly

None None

LBW Workshop on 

lesson plans and 

logistics of 

program 

implementation

1 day; 3 hours Principal and 

staff; Same as 

teacher training

1 day; 3 hours Faculty 

boosters with 

reminders 

about program 

implementation 

and discussion 

of challenges

24 weekly 

sessions; 

biannual 

meeting

Program 

coordinator; 

calls with 

program 

developer to 

improve 

schoolwide 

implementation

Once every 6 

months

See notes at end of table.

15

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.1. Social and Character Development programs—Continued

Panel 4: Professional development—Continued

Program

Pre-implementation Ongoing consultation

Teachers Other Teachers Other

Content Duration Content Duration Content Duration Content Duration

PATHS Training on 

concepts, 

curriculum 

implementation, 

how to integrate 

activities with 

traditional 

instruction

2 days Principal and 

school mental 

health staff; 

same as 

teacher training

2 days Technical 

assistance and 

implementation 

consultation, 

individual 

meetings with 

each teacher, 

attendance at 

grade-level 

meetings, 

planning for 

following year

Weekly 

consultations; 

2-day summer 

meeting

Principal and 

school staff; 

model lessons, 

team teaching, 

general 

feedback; 

planning for 

following year

Biweekly calls; 

2-day summer 

meeting

PA Training on 

concepts and 

delivery of 

curriculum, 

school climate 

activities to 

reinforce positive 

behaviors and 

parent 

involvement

1/2 day Principal 

training on 

appointing 

committees 

and 

coordinating 

school climate 

activities with 

family groups

1/2 day Group session 

to provide 

technical 

assistance and 

share 

experiences 

and challenges; 

visits for 

technical 

support

Up to 1/2 day 

once per year, 

and up to 1/2

day visits to 

schools 

monthly

Meeting of 

principals and 

coordinators to 

provide 

technical 

assistance and 

share 

successes and 

challenges

One session of 

3 hours per 

year

See notes at end of table.

16

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.1. Social and Character Development programs—Continued

Panel 4: Professional development—Continued

Program

Pre-implementation Ongoing consultation

Teachers Other Teachers Other

Content Duration Content Duration Content Duration Content Duration

4Rs Course to learn 

curriculum, 

improve on own 

skills, create a 

vision of 

community, set 

consistent rules, 

behavior 

management

25 hours None None Training and 

support in 

mediation, 

negotiation, 

role-playing, 

class meetings, 

behavior 

management

Individualized 

coaching 12 

times per year; 

3-day summer 

institute

None None

SS Training on child 

development, 

social skills, and 

curriculum 

delivery

2 days Character 

development 

planning team, 

school 

counselor, and 

other staff; 

training on child 

development, 

social skills, and 

curriculum 

content

2.5 hours Year-end 

workshop to 

review student 

outcome data 

and plan for 

following year

1 day School 

counselor 

guidance on 

strategies and 

challenges; 

principal and 

staff review of 

student 

outcome data 

and plan for 

following year

1/2 day 

counselor 

meetings about 

monthly; 1-day 

year-end 

workshop

See notes at end of table.

17

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.1. Social and Character Development programs—Continued 

Panel 5: Activities to promote the six SACD goals and behavior management 

Program

Violence prevention 

and peace 

promotion

Social and emotional 

development Character education

Tolerance and 

diversity

Risk prevention and 

health promotion

Civic responsibility 

and community 

service

Behavior 

management

ABC   

CSP   

LBW     

PATHS       

PA       

4Rs       

SS   

See notes at end of table.

18

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.1. Social and Character Development programs—Continued 

Panel 6: SACD outcomes addressed 

Program

Self-Efficacy 

for Peer 

Interactions

Normative 

Beliefs About 

Aggression Empathy

Altruistic 

Behavior

Positive 

Social 

Behavior

Problem 

Behavior

Engagement with 

Learning

Academic 

Competence 

and Motivation

Positive 

School 

Orientation

Negative

School 

Orientation

Student 

Afraid at 

School

Victimization 

at School

Feelings 

of Safety

Student 

Support for 

Teachers

ABC     

CSP        

LBW         

PA              

PATHS           

4Rs         

SS     

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

 : Outcome addressed

 Blank cell: Outcome not addressed

SOURCE: The Social and Character Development (SACD) Research Program.

19

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

20

The commonalities and differences among the seven SACD programs are important for several reasons. They 

make clear that there is no single SACD universal schoolwide program, but a set of them that are in many 

ways similar but also different enough to distinguish them. The commonalities justify this study’s evaluation 

approach of both combining results from all seven programs and examining the results from each program 

separately for a common set of outcome measures for measuring students’ social and character development. 

At the same time, it is important to keep in mind the differences when considering the results for each of the 

seven programs. 

To standardize the evaluation, a competitive peer-review process was used to select a third party to collect 

and analyze the common set of data from all the schools. MPR was selected for this role. 

As in most evaluations of education interventions, the SACD programs were not compared to a no-treatment 

control. The control schools continued to implement the social and character development activities that 

constituted their standard practice. A number of researchers have observed that schools commonly use 

support services, intervention curricula, and discipline management strategies to promote social and character 

development and prevent problem behavior (e.g., Crosse et al. 2001; Gottfredson and Gottfredson 2001), and 

in many states, schools are required by law to implement such programs and practices. For example, under 

the Student Citizen Act of 2001, character education was made part of the standard curriculum for North 

Carolina schools starting in the 2002-03 school year; in New York, the Safe Schools Against Violence in 

Education Act of 2001 encouraged districts to promote instruction in civility, citizenship, and character 

education; and Illinois’ Children’s Mental Health Act required all school districts to develop a policy for 

incorporating social and emotional competence into the district’s educational program by August 31, 2004. 

For this reason, data were collected on the implementation of social and character development activities 

from all schools. These data provide contextual information on the prevalence of social and character 

development activities occurring in both the treatment and control schools. 

Study Design and Methodology 

The SACD multiprogram evaluation was a 3-year longitudinal study following one cohort of students from 

third through fifth grades.5 It was designed to determine the effects of seven SACD programs on the social 

and character development of students attending the schools in which the interventions were implemented. 

This design stems in part from the programs being chosen by the quality of the grant applications, so they are 

not necessarily representative of the full set of universal school-based SACD interventions. The results, then, 

are not generalizable beyond the seven programs. 

The evaluation examined the average effect of the programs on a common set of student outcomes including 

social and emotional competence, behavior, academics, and perceptions of school climate. Initial data were 

collected on consenting students in the third grade during fall 2004. Follow-up data were collected at 

subsequent time points: (1) in spring 2005 from third-grade students, (2) in fall 2005 when the original thirdgrade cohort began fourth grade, (3) in spring 2006 when the original third-grade cohort completed fourth 

grade, and (4) in spring 2007 when the original third-grade cohort completed fifth grade. Data for the study 

included survey data from students, students’ primary caregivers, teachers, and interview responses from 

principals. 

 

5

 The multiprogram evaluation also included 12 additional schools recruited in the second year of the study. This second 

cohort of third-grade students was followed for 2 years through the fourth grade. Initial data were collected in fall 2005, 

and follow-up data in spring 2006 and spring 2007. These schools were recruited by the Children’s Institute (n = 4), 

University at Buffalo, The State University of New York (n = 2), University of North Carolina at Chapel Hill (n = 4), 

and Vanderbilt University (n = 2). Appendix A describes the combined analysis of Cohorts 1 and 2. The Executive 

Summary and chapters 1-8 describe the analysis of Cohort 1 only. 
 Chapter 1. The Social and Character Development Multiprogram Evaluation 

21

Sample and Random Assignment 

Schools were the unit of assignment, in part because the programs were applied at the school level (or at least 

in multiple grades in a school). As a result, contamination from the treatment group to the control group was 

unlikely because of the physical separation of schools. From each school, data were collected from students in 

grades 3, 4, and then 5 in consecutive years. Data were also collected from students’ primary caregivers, from 

the principals of the schools, and from the teachers in all three grades. Parents or primary caregivers, 

principals, and teachers all had to give their consent before taking part in the study. Students gave assent after 

their primary caregivers had provided consent. Beginning in the spring and early fall of 2004, consent forms 

were sent home with students for primary caregivers to review and sign, if desired, indicating their consent 

for their own participation in the study and for the student’s participation in the study. The consent process 

continued at each follow-up for students who were new entrants to the schools, or who did not return a 

consent form in fall 2004. During each data collection, students themselves were asked whether they would 

assent to participate in the study. The research teams carried out the consent process and other procedures 

concerning human subjects after their review and approval by institutional review boards at each participating 

institution and by the Public/Private Ventures Institutional Review Board. 

Each research team set the eligibility criteria for schools within its site and recruited 10 to 14 schools for the 

study.6 In general, schools were purposively selected based on (1) their lack of schoolwide institutionalized 

programs addressing social and character development, (2) their willingness to participate in random 

assignment and engage in data collection activities, and (3) their ability to implement the program schoolwide 

(all elementary grades) if assigned to the treatment condition. Additional eligibility criteria employed by 

individual teams included relatively low performance on standardized tests, a relatively high percentage of 

students eligible for free and reduced-price lunch, low student mobility rates, and a sufficient number of 

students per school to meet sample size requirements. 

The research teams recruited a total of 84 schools into the study beginning with the initial data collection in 

fall 2004 (table 1.2). Eighty-three schools completed the evaluation, with one (a control school) dropping out 

before the start of the second year after converting to a magnet school. Some schools changed their grade 

structures over time but in only one school did this lead to the students in the study not receiving the 

intervention (the loss of students in fifth grade at one CSP school). In Year 1, the highest grade was fourth or 

fifth grade at 46 schools, sixth or seventh grade at 18 schools, and eighth grade at 20 schools. In Years 2 and 

3, these figures were 47 and 49 schools having fourth or fifth grade as the highest grade; 15 and 12 schools 

with sixth or seventh grade; and 21 and 22 schools with eighth grade, respectively. 

After recruiting the schools, each research team created five to seven matched pairs of schools. Within each pair, 

one school was randomly assigned to the treatment group and the other to the control group. Stratified sampling 

was used to guard against the possibility that random assignment of a small number of schools could produce sets 

of treatment and control schools that had markedly different characteristics. An algorithm was developed to select 

the best pairs by minimizing the distance between several measurable characteristics for schools within each pair. A 

set of candidate pairs was selected such that the overall quality of matches across all the schools remained as high 

as possible, without creating any serious mismatches for any subset of the individual pairs. Each team executed the 

matching algorithm with somewhat different school characteristic variables, depending on what data were 

available, and exercised its best judgment, based on knowledge of the schools involved, as to which candidate 

pairing was the best one. After the best pairing was established, one member of each pair was randomly selected to 

be in the treatment group and the other was assigned to the control group. 

 

 

6

 The New York University research team recruited 18 schools but only 14 were included in the multiprogram 

evaluation. 
Table 1.2. Multiprogram evaluation school sample 

Year 1

(Spring/Fall 3rd grade)

Year 2

(Spring/Fall 4th grade)

Year 3

(Spring/Fall 5th grade)

Program Total Treatment Control Total Treatment Control Total Treatment Control

 Total number of schools 84 42 42 83 42 41 83 42 41

ABC 12 6 6 12 6 6 12 6 6

CSP 10 5 5 10 5 5 10 5 5

LBW112 6 6 11 6 5 11 6 5

PA 14 7 7 14 7 7 14 7 7

PATHS 10 5 5 10 5 5 10 5 5

4Rs 14 7 7 14 7 7 14 7 7

SS 12 6 6 12 6 6 12 6 6

1 One control school became a magnet school and dropped out of the evaluation.

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

SOURCE: The Social and Character Development (SACD) Research Program.

22

Chapter 1. The Social and Character Development Multiprogram Evaluation
 Chapter 1. The Social and Character Development Multiprogram Evaluation 

23

The teacher sample for the evaluation included the teachers in the grade that the student cohort attended in order 

to obtain teacher reports on students and all the third-, fourth-, and fifth-grade teachers in the participating schools 

during all 3 school years in order to measure the level of SACD activities in the schools. The principal sample 

included the principals at each of the 84 schools. 

The student sample for the multiprogram evaluation included students enrolled in the third grade during the 2004-

05 school year, students enrolled in the fourth grade during the 2005-06 school year, and students enrolled in the 

fifth grade during the 2006-07 school year, with ―enrollment‖ defined as being present in the study schools at the 

time of student data collection. Thus, the sample included a cohort of third-grade students as they progressed 

through their fifth-grade year. In addition, it included students who entered the study schools over the course of 

the study (i.e., students who entered study schools as third graders during the 2004-05 academic year, or as fourth 

graders in the 2005-06 academic year, or as fifth graders in the 2006-07 academic year). Thus, the specific students 

in the sample changed between the beginning of data collection during fall 2004 and subsequent rounds of data 

collection. Up to the time of a specific data collection follow-up, students exposed to an intervention were from 

one of three groups: 

1. Children enrolled in third grade at a study school in fall 2004 and during the subsequent rounds of data 

collection—original sample who remained at the schools (stayers); 

2. Children enrolled in third grade at a study school in fall 2004 but not enrolled during a subsequent data 

collection period—original sample who left between fall 2004 data collection and a subsequent data 

collection (leavers); and

3. Children who enrolled in the grade of the student cohort at a study school after the fall 2004 data 

collection but before the subsequent data collection initiative (new entrants). 

A total of 6,567 third-graders were initially enrolled in the study schools in fall 2004 (table 1.3). Due to the 

combination of leavers and new entrants, the total was 6,597 in spring 2005. It fell to 6,415 in spring 2006 and to 

6,249 in spring 2007. Over this period the treatment group made up 51 percent to 52 percent of the sample. The 

average number of students per school ranged from 75 to 81; however, the actual number per school varied across 

the programs from 23 to 164. Over time, new entrants grew as a percentage of the sample from 6 percent to 20 

percent to 28 percent, and there were no statistically significant differences each year between the percentages of 

new entrants in the treatment and control groups. The percentage of the original sample that left also grew from 

follow-up to follow-up, going from 6 percent to 22 percent to 32 percent, again with no statistically significant 

differences each year between the treatment and control groups. Leavers were not tracked once they left the study 

schools.7

 That is, if a third-grade student was enrolled in a study school during fall 2004 but then left before spring 

2005, that student was not tracked into the new school and new data from that student were not collected in spring 

2005. As a result of this turnover, 69 percent of the original students (stayers) remained in the study and they made 

up 72 percent of the Year 3 student sample. 

 

7

 A small percentage of leavers moved to another school in the study and so remained in the study. For descriptive 

purposes, they took on the assignment of their school to treatment or control. For analysis purposes, they retained their 

initial assignment. The percentages of Cohort 1 students who switched status at each round grew at each follow-up point 

but remained relatively small (growing from about 0.2% to 1.4%). 
Table 1.3. Characteristics of the student sample in the multiprogram evaluation 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Characteristic Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 School sample size 84 42 42 84 42 42 83 42 41 83 42 41

 Student sample size 6,567 3,367 3,200 6,597 3,388 3,209 6,415 3,327 3,088 6,249 3,172 3,077

Stayers † † † 6,203 3,188 3,015 5,110 2,625 2,485 4,500 2,289 2,211

New entrants † † † 394 200 194 1,305 702 603 1,749 883 866

 New entrants as percent of spring 

 enrollment1† † † 6.0 5.9 6.0 20.3 21.1 19.5 27.9 27.8 28.1

Total leavers (from original cohort) † † † 364 179 185 1,457 742 715 2,067 1,078 989

 Leavers as percent of fall 2004 

 enrollment1† † † 5.5 5.3 5.8 22.2 22.0 22.3 31.5 32.0 30.9

Number of students per school (mean) 78 80 76 79 81 76 77 80 76 75 76 75

Range of number of students per school 23-149 38-149 23-148 26-155 35-147 26-155 25-151 36-151 25-144 25-164 28-164 25-139

† Not applicable.

1 No statistically significant differences were found between values for treatment and control groups.

SOURCE: The Social and Character Development (SACD) Research Program.

24

Chapter 1. The Social and Character Development Multiprogram Evaluation
 Chapter 1. The Social and Character Development Multiprogram Evaluation 

25

Because leavers were not followed, the SACD evaluation is not an intent-to-treat impact analysis (an analysis 

that follows all students originally assigned to the treatment or control groups). Therefore, there is the 

possibility that the SACD programs caused differential student and teacher mobility in the treatment schools 

versus mobility in the control schools. If this occurred, the estimates of the impact of the programs would 

include both the extent to which the programs improved the outcomes of the average student and the 

mobility effects resulting from any differences in the average outcomes of treatment and control students 

who entered or left the schools after random assignment was completed in third grade. Descriptive analyses, 

presented later, do not identify differential mobility in the treatment and control schools, although this cannot 

be definitively established. 

Table 1.4 provides information on the student sample by program. The same trends in the growth in the 

percentages of leavers and new entrants were apparent but there were differences among programs in this 

growth. For example, by Year 3 the percentages of leavers ranged from 9 percent to 50 percent, and the 

percentages of new entrants ranged from 14 percent to 38 percent among programs. The table also shows 

that over all 3 years there was a statistically significant difference between the treatment and control groups 

within a program for the percentage of leavers in six cases and the percentage of new entrants in one case. 

Student turnover could have affected the results of the evaluation. First, if the SACD programs affected 

turnover—and thus altered the composition of the student population—then the difference between the 

treatment and control groups would reflect (1) the changes in student behavior that were due to the 

intervention, and (2) the changes that arose indirectly because the student population changed. Second, 

students had differing amounts of exposure to the program, and this may have affected the benefits to 

students. In response to these issues, sensitivity analyses were conducted to address whether there were 

differences between the estimated impacts of the SACD programs for the stayers and for the new entrant 

subgroups. 
Table 1.4. Student sample overall and by program for all students and for the treatment and control groups 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Intervention program

All 

students

All 

students

Leavers

New

entrants All 

students

Leavers

New 

entrants All 

students

Leavers

New

entrants

# %

1

# %

2

# %

1

# %

2

# %

1

# %

2

All programs 6,567 6,597 364 6 394 6 6,415 1,457 22 1,305 20 6,249 2,067 31 1,749 28

 Treatment group 3,367 3,388 179 5 200 6 3,327 742 22 702 21 3,172 1,078 32 883 28

 Control group 3,200 3,209 185 6 194 6 3,088 715 22 603 20 3,077 989 31 866 28

ABC 879 875 43 5 39 4 877 160 18 158 18 871 289 33 281 32

 Treatment group 380 373 17 4 10 3** 367 72 19 59 16 353 135 36 108 31

 Control group 499 502 26 5 29 6 510 88 18 99 19 518 154 31 173 33

CSP 959 975 36 4 52 5 969 230 24 240 25 947 238 25 226 24

 Treatment group 476 485 20 4 29 6 474 135 28** 133 28* 458 139 29** 121 26

 Control group 483 490 16 3 23 5 495 95 20 107 22 489 99 20 105 21

LBW 986 1,007 60 6 81 8 959 228 23 201 21 944 308 31 266 28

 Treatment group 548 565 25 5** 42 7 556 110 20 118 21 567 145 26** 164 29

 Control group 438 442 35 8 39 9 403 118 27 83 21 377 163 37 102 27

PA 811 812 74 9 75 9 764 251 31 204 27 655 408 50 252 38

 Treatment group 410 416 33 8 39 9 425 108 26** 123 29 327 209 51 126 39

 Control group 401 396 41 10 36 9 339 143 36 81 24 328 199 50 126 38

PATHS 786 783 39 5 36 5 778 150 19 142 18 778 243 31 235 30

 Treatment group 377 374 21 6 18 5 373 66 18 62 17 378 114 30 115 30

 Control group 409 409 18 4 18 4 405 84 21 80 20 400 129 32 120 30

4Rs 1,202 1,194 86 7 78 7 1,109 320 27 227 20 1,065 492 41 355 33

 Treatment group 652 647 49 8 44 7 599 183 28 130 22 556 279 43 183 33

 Control group 550 547 37 7 34 6 510 137 25 97 19 509 213 39 172 34

See notes at end of table.

26

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.4. Student sample overall and by program for all students and for the treatment and control groups—Continued

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Intervention program

All

students

All 

students

Leavers

New 

entrants All 

students

Leavers

New 

entrants All 

students

Leavers

New 

entrants

# %

1

# %

2

# %

1

# %

2

# %

1

# %

2

SS 944 951 26 3 33 3 959 118 13 133 14 989 89 9 134 14

 Treatment group 524 528 14 3 18 3 533 68 13 77 14 533 57 11* 66 12

 Control group 420 423 12 3 15 4 426 50 12 56 13 456 32 8 68 15

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

1

Leavers as a percentage of fall 2004 enrollment (these values are cumulative over the years).

2 New entrants as a percentage of spring enrollment.

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

SOURCE: The Social and Character Development (SACD) Research Program.

27

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation

28

Measures 

The SACD multiprogram evaluation included six data collection reports: (a) Child Report; (b) Primary 

Caregiver Report; (c) Teacher Report on Student; (d) Teacher Report on Classroom and School; (e) Principal 

Interview; and (f) Implementation Fidelity Rating.8 The measures included in these reports were largely taken 

from instruments used in previous research. Four measures (child responsibility, child and teacher 

perceptions of school safety, and primary caregiver perceptions of community resources) were developed for 

the SACD evaluation due to a lack of adequate existing measures. Modifications were made to some of the 

measures to enhance readability, reduce the number of items, or reduce the number of possible responses to 

each item. Children were asked to respond based on their experiences in the past couple of weeks; primary 

caregivers and teachers responded based on their experiences in the past 30 days. The surveys were pilottested in December 2003, and revisions were made prior to the initial administration in fall 2004. A series of 

exploratory and confirmatory factor analyses using the initial data collapsed the scales into a smaller number 

of more reliable measures that were used as outcome or moderator variables in the impact analyses. In 

addition to developing the outcome variables, these analyses provided a check of how well the measures fit 

under the outcome domains as set out in the conceptual model (Social and Emotional Competence, Behavior, 

Academics, and Perceptions of School Climate). The discussion of the measures is broken into two parts. 

First, the scales collected are described by report and domain. Their published reliabilities, if available, are 

reported.9

 Second, the outcome and moderator measures developed through the factor analyses and then 

used in the analyses for the SACD evaluation are described. 

Child Report 

The Child Report was a group-administered questionnaire consisting of a battery of measures that are 

described below. The instrument was delivered to groups averaging 15 to 20 students in classrooms at each 

school during a 1- to 2-day visit (or during a make-up session) by the MPR data collection team. A proctor 

read the directions aloud as children followed along in their survey booklets during a 50-minute classroom 

session. Children reported on their own thoughts, feelings, and behaviors. There is some evidence in the 

literature that self-reports of behavior problems are not valid in elementary-age aggressive and ADHD 

children (McMahon and Frick 2005; Pelham, Fabiano, and Massetti 2005), which may limit the value of such 

measures for this type of student (though similar measures were used with primary caregivers and teachers). 

The following measures (grouped by domain) were included in the battery: 

I. Social and Emotional Competence 

A. Beliefs about aggression. This measure is an 8-item modified version of the Normative Beliefs About 

Aggression Scale (Huesmann and Guerra 1997). Children responded on a 4-point scale, 

indicating their beliefs about the acceptability of verbal and physical aggression, either proactive 

or in retaliation for others’ behavior (e.g., how wrong or okay it is to hit, shove, fight with, or 

verbally assault others). Its reliability has been evaluated through internal consistency (0.90). 

B. Self-efficacy for peer interaction. This measure is a 12-item modified version of the Self-Efficacy for 

Peer Interaction Scale (Wheeler and Ladd 1982). Children responded on a 4-point scale, 

 

8

 In addition, the SACD evaluation intended to include two additional reports for each program: a cost report and a 

report on the presence of artifacts indicating the use of SACD activities. For the cost report, a common training on 

collection of cost data was provided for the research teams; however, all teams did not collect such data and the data 

collected were not consistent. As a result, cost reports were not generated and the SACD evaluation did not include a 

cost analysis. For the artifact study, school and classroom observations were done in Year 1. However inter-rater 

reliability was considered too low, so the observations were not continued in later years and the Year 1 results are not 

reported. 

9

 Five of the measures have no published reliabilities, and no reliabilities were computed for these measures based on the 

study data. Items from these measures were used to create the outcome measures. The outcome measures they 

contributed to and the reliabilities of those outcome measures calculated from the study data are shown in table 5. 
 Chapter 1. The Social and Character Development Multiprogram Evaluation 

29

indicating their perceived ability to perform verbal or persuasive prosocial skills in conflict and 

nonconflict peer interactions (e.g., how hard or easy it is to engage in play with other children or 

resist negative social pressures). Its reliability has been evaluated through internal consistency 

(0.85) and test-retest (0.90 for boys and 0.80 for girls). 

C. Empathy. This measure is a 16-item modified version of the Children’s Empathy Questionnaire

(Funk et al. 2003). Children responded on a 3-point scale about the degree to which they would 

respond empathically to hypothetical and actual anticipated events (e.g., whether other people’s 

problems bother them or if they would feel happy when a friend gets a good grade). Its reliability 

has been evaluated through internal consistency (0.72). 

II. Behavior 

A. Altruistic behavior. The Altruism Scale, Child Version (Soloman et al. 2000) is an 8-item measure to 

which children responded on a 4-point scale about how often they engage in helping behavior 

(e.g., cheer someone up who is feeling sad or help someone who is being picked on). Its 

reliability has been evaluated through internal consistency (0.82). 

B. Aggression. The Aggression Scale (Orpinas and Frankowski 2001) is a 6-item measure to which 

children responded on a 4-point scale about how often they engage in verbal or physical 

aggression (e.g., tease others, shove others). Its reliability has been evaluated through internal 

consistency (0.87). 

C. Minor delinquency. The modified Frequency of Delinquent Behavior (Dahlberg, Toal, and Behrens 

1998; Dunford and Elliott 1984) Scale was modified for the SACD multiprogram evaluation. 

Children responded to 6 items on a 4-point scale about how often they engage in minor 

delinquency in school (e.g., taking something that belongs to others, skipping class). There is no 

published or study-based reliability for this scale. 

III. Academics 

A. School engagement. The Engagement vs. Disaffection with Learning Scale, (Furrer and Skinner 

2003) is a 10-item measure to which children responded on a 4-point scale, indicating how much 

they agree with statements indicative of their behavioral and emotional engagement, effort, 

attention, and persistence in the classroom (e.g., paying attention in class, listening carefully, 

joining in class discussion). Its reliability has been evaluated through internal consistency (0.75 to 

0.86). 

B. Standardized test scores and grades. The research teams were to collect student standardized test 

scores and grades. However, not all teams were able to collect these data from their schools and 

districts, and the data that were collected varied in quality. As a result, neither standardized test 

scores nor grades could be used as outcome measures in the evaluation, although they had been 

proposed for such use in the conceptual model. 

IV. Perceptions of School Climate 

A. School connectedness. The Sense of School as a Community Scale (Roberts, Horn, and Battistich 

1995) is a 14-item measure to which children responded on a 4-point scale about how much they 

agree with statements indicative of the social relationships among students, teachers, and staff 

within a school (e.g., students care about each other, students and teachers treat each other with 

respect). Its reliability has been evaluated through internal consistency (0.91). 

B. Safety. The Feelings of Safety at School Scale was developed for the SACD multiprogram 

evaluation and is a 5-item measure to which children responded on a 4-point scale about how 

much they agree with statements that are indicative of students’ feelings of safety at school (e.g., 

students feel safe, students are afraid someone will hurt them, students are afraid someone will 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

30

tease them). The scale was constructed by IES and DVP staff for the SACD multiprogram 

evaluation. There is no published or study-based reliability for this scale. 

C. Victimization. The Victimization Scale (Orpinas and Kelder 1995) is a 6-item measure to which 

children responded on a 4-point scale about how often they are victimized at school (e.g., teased, 

pushed, or threatened). Its reliability has been evaluated through internal consistency (0.90). 

Primary Caregiver Report 

The Primary Caregiver Report was a self-administered survey that took primary caregivers approximately 15 

minutes to complete. Primary caregivers, who were usually but not always parents, reported on the target 

child’s behavior, demographic factors, and home and community environment. Primary caregivers completed 

the questionnaire at home and returned it to school in a sealed envelope. If a completed Primary Caregiver 

Report was not returned within 3 or 4 weeks, MPR attempted to complete a computer-assisted telephone 

interview with the primary caregiver. Primary caregivers were offered a small financial incentive as a thankyou. The following measures (grouped by domain or type of moderator) were used: 

I. Behavior 

A. Responsibility. The Responsibility Scale is an 8-item measure developed for the SACD 

multiprogram evaluation to which primary caregivers responded on a 4-point scale about the 

degree to which a child takes responsibility for his or her own actions (e.g., how often a child 

asks permission, apologizes, takes care of property, or denies responsibility for wrongdoing). The 

scale was constructed by IES and DVP staff for the SACD multiprogram evaluation. There is no 

published or study-based reliability for this scale. 

B. Self-regulation, cooperation, and prosocial behavior. The Social Competence Scale (CPPRG 1999) is a 19-

item measure to which primary caregivers responded on a 4-point scale about a child’s emotional 

and behavioral regulation, cooperative behavior, and prosocial behavior (e.g., how often a child 

calms down when wound up, works well in groups, and acts friendly toward others). Its 

reliability has been evaluated through internal consistency (0.87). 

C. Altruistic behavior. The Altruism Scale, Primary Caregiver Version (Soloman et al. 2000) is an 8-

item measure to which primary caregivers responded on a 4-point scale about how often their 

child engages in helping behavior (e.g., cheers someone up who is feeling sad or helps someone 

who is being picked on). Its reliability has been evaluated through internal consistency (0.82). 

D. Aggression. The Behavior Assessment System for Children (BASC) Aggression Subscale, Parent 

Version (Reynolds and Kamphaus 1998) is a 13-item assessment to which primary caregivers 

responded on a 4-point scale about how often their child engages in verbal or physical aggression 

(e.g., teases others, hits other children). Its reliability has been evaluated through internal 

consistency (0.83) and test-retest (0.84). 

E. Conduct problems. The BASC Conduct Problems Subscale, Parent Version (Reynolds and 

Kamphaus 1998) is an 11-item assessment to which primary caregivers responded on a 4-point 

scale about how often their child engages in socially deviant and disruptive behaviors (e.g., lies, 

gets into trouble). Its reliability has been evaluated through internal consistency (0.82) and testretest (0.92). 

II. Perceptions of School Climate 

A. Parent involvement. The modified Parent and Teacher Involvement Questionnaire, Parent Version 

(CPPRG 1991) is a 9-item measure to which primary caregivers responded on a 4-point scale 

about how often they interact with teachers and school staff, participate in school activities, and 

are involved in their children’s academic lives (e.g., call the child’s teacher, attend parent-teacher 
 Chapter 1. The Social and Character Development Multiprogram Evaluation 

31

conferences, take the child to the library). Its reliability has been evaluated through internal 

consistency (0.77). 

III. Family Moderators 

A. Parenting practices. The Alabama Parenting Questionnaire, Positive Parenting, and Poor 

Monitoring and Supervision Subscales (Shelton, Frick, and Wootton 1996) is a 16-item measure 

to which primary caregivers responded on a 4-point scale about how often they support and 

reward their child, and monitor and supervise their child (e.g., praise the child for good behavior, 

check when the child comes home). Its reliability has been evaluated through internal 

consistency (0.75 for monitoring and 0.85 for positive parenting). 

B. Home atmosphere. The modified Confusion, Hubbub, and Order Scale (Matheny et al. 1995) is a 

14-item measure to which primary caregivers responded on a 5-point scale about how much they 

agree or disagree with statements that reflect disorganization in the home (e.g., can find things 

when you need them, there is always a fuss going on, there is a regular routine). Its reliability has 

been evaluated through internal consistency (0.79) and through test-retest (0.74). 

C. Socioeconomic risk. Primary caregivers completed a series of questions indicating ethnicity, 

education, employment status, household composition, and income. 

IV. Community Moderators 

A. Perceptions of community risks and protective factors. The Community Risks and Protective Factors Scale 

is a 12-item measure developed for the multiprogram evaluation based on research on 

community risks by Forehand, Brody, and Armistead (2000) and supplemented with items to 

assess community resources constructed by researchers involved in the SACD evaluation. 

Primary caregivers indicated on a 4-point scale how much statements about community 

characteristics described the neighborhood the primary caregiver and child live in (e.g., trash 

pickup is a problem in the neighborhood, there are safe outdoor parks for children to play in). 

Seven of the items were taken from the original Community Risks Scale whose reliability has 

been evaluated through internal consistency (0.89). The other five items were from the 

Community Resources Scale constructed by IES and DVP staff for the SACD multiprogram 

evaluation and for which there is no published or study-based reliability. 

B. Intergenerational closure. The Intergenerational Closure Scale (Sampson, Morenoff, and Earls 1999) 

is a 5-item measure to which primary caregivers responded on a 4-point scale to statements 

about adults’ connections with children in the neighborhood (e.g., parents know their children’s 

friends, there are adults for kids to look up to). Its reliability has been evaluated through internal 

consistency (0.72). 

C. Child-centered social control. The Child-Centered Social Control Scale (Sampson, Morenoff, and 

Earls 1999) is a 5-item measure on which primary caregivers indicated on a 5-point scale how 

likely it is that neighbors could be counted on to ―do something‖ if something occurred in the 

neighborhood (e.g., youth sprayed graffiti, children were hurt, neighbors were in need). Its 

reliability has been evaluated through internal consistency (0.74). 

Teacher Report on Student 

The Teacher Report on Student was a self-administered survey completed by teachers of the student cohort 

in the study (i.e., third-grade teachers in 2004-05, fourth-grade teachers in 2005-06, and fifth-grade teachers in 

2006-07). Teachers reported on each consented student’s typical behavior in the past 30 days. Teachers 

completed the questionnaires independently and returned the surveys to a central point in the school for 

collection by the research team. The survey took approximately 15 minutes for teachers to complete for each 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

32

consented student in their classroom. Teachers were offered a small financial incentive as a thank-you. The 

following measures (grouped by domain) were used: 

I. Behavior 

A. Responsibility. The Responsibility Scale is an 8-item measure developed for the SACD 

multiprogram evaluation. It uses teachers’ responses on a 4-point scale to measure the degree to 

which a child takes responsibility for his or her own actions (e.g., how often a child asks 

permission, apologizes, takes care of property, or denies responsibility for wrongdoing) for the 

SACD multiprogram evaluation. The scale was constructed by IES and DVP staff for the SACD 

multiprogram evaluation. There is no published or study-based reliability for this scale. 

B. Self-regulation, cooperation, and prosocial behavior. The Social Competence Scale (CPPRG 1999) is an 

19-item measure to which teachers responded on a 4-point scale about a child’s emotional and 

behavioral regulation, cooperative behavior, and prosocial behavior (e.g., how often a child calms 

down when wound up, works well in groups, and acts friendly toward others). Its reliability has 

been evaluated through internal consistency (0.87). 

C. Altruistic behavior. The Altruism Scale (Soloman et al. 2000) is an 8-item measure to which 

teachers responded on a 4-point scale about how often a student engages in helping behavior 

(e.g., cheers someone up who is feeling sad or helps someone who is being picked on). Its 

reliability has been evaluated through internal consistency (0.82). 

D. Aggression. The BASC Aggression Subscale, Teacher Version (Reynolds and Kamphaus 1998) is a 

14-item assessment to which teachers responded on a 4-point scale about how often a student 

engages in verbal or physical aggression (e.g., bullies others, calls other children names). Its 

reliability has been evaluated through internal consistency (0.95) and through test-retest (0.91). 

E. Conduct problems. The BASC Conduct Problems Subscale, Teacher Version (Reynolds and 

Kamphaus 1998) is a 10-item assessment to which teachers responded on a 4-point scale about 

how often a student engages in socially deviant or disruptive behavior in school (e.g., skips class, 

cheats, steals). Its reliability has been evaluated through internal consistency (0.77) and through 

test-retest (0.80). 

F. Attention deficit hyperactivity disorder (ADHD)-related behavior. This measure is a set of 10 items that 

assess a range of symptoms of attention deficits and hyperactivity such as inattention, 

distractibility, verbal and physical impulsivity, losing things, and difficulty organizing activities. It 

draws on two sources. Five inattention/overactivity items were taken from the IOWA Conners 

Teacher Rating Scale (Loney and Milich 1982). The reliability of the original Conners scale has 

been evaluated through internal consistency (0.89 to 0.92). Five items were based on diagnostic 

criteria for ADHD from the Diagnostic and Statistical Manual for Mental Disorders, fourth edition 

(American Psychiatric Association 2000) that have been shown to be among the most powerful 

for predicting ADHD diagnoses in school settings (Pelham et al. 1992). 

II. Academics 

A. Academic competence. The Academic Competence and Motivation Scale was developed for the 

SACD multiprogram evaluation and is based on the academic competency subscales of the 

Social Skills Rating System (Gresham and Elliott 1990) and items from the Teacher’s Report 

Form (Achenbach 1991). On this 5-item measure, teachers indicated how a student performed 

overall, performed in reading, performed in math, and functioned intellectually compared to 

grade-level expectations; and indicated how motivated a student was compared to the average 

student at grade level. The reliability of the Social Skills Rating System has been evaluated 

through internal consistency (0.96) and through test-retest (0.93). The reliability of the Teacher 
 Chapter 1. The Social and Character Development Multiprogram Evaluation 

33

Report Form has been evaluated through internal consistency (0.72 to 0.95) and through testretest (0.62 to 0.96). 

III. Perceptions of School Climate 

A. Parent and teacher involvement. This measure is a 7-item modified version of the Parent and Teacher 

Involvement Questionnaire, Teacher Version (CPPRG 1991). Teachers responded on a 4-point 

scale indicating how often a student’s primary caregiver interacts with the teachers at school and 

attends school events (e.g., calls on the phone, attends parent-teacher conferences). Teachers 

also responded to three questions about communication with a student’s parents, the degree of 

involvement a parent has in a student’s life, and the academic encouragement provided to a 

student. Its reliability has been evaluated through internal consistency (0.77). 

Teacher Report on Classroom and School 

The Teacher Report on Classroom and School was a self-administered questionnaire that took a teacher 

approximately 30 minutes to complete. Teachers reported on the organizational climates of their schools, the 

social and character development activities they had implemented in their classrooms, and their own 

professional backgrounds. The questionnaire was distributed to third-, fourth-, and fifth-grade teachers at 

each school. The following measures (grouped by domain or type of moderator) were used: 

I. Perceptions of School Climate 

A. Organizational climate. The School-Level Environment Questionnaire (Rentoul and Fraser 1983) is 

a 56-item measure to which teachers responded on a 5-point scale about the degree to which 

teachers are supported and valued by administration officials, are provided adequate resources, 

are autonomous in managing their own activities, are collegial with one another, and have a voice 

in school policy decisions. Its reliability has been evaluated through internal consistency (0.82). 

B. Safety. The Feelings of Safety at School Scale was developed for the SACD multiprogram 

evaluation and is a 4-item measure. Teachers responded on a 4-point scale about how much they 

agree with statements that are indicative of students’ feelings of safety at school (e.g., students 

feel safe at this school, students are afraid someone will hurt them at school, students are afraid 

someone will tease them at school). This scale was constructed by IES and DVP staff for the 

SACD multiprogram evaluation. There is no published or study-based reliability for this scale. 

II. Level of SACD in the School 

The SACD Activities Teacher Survey is a self-report questionnaire constructed by SACD Research 

Consortium members. Teachers indicated the SACD-related programs and activities occurring in the 

classroom and school, and the degree to which teachers and staff incorporated SACD-like qualities in 

their professional relationships and styles. The questionnaire consisted of 11 major questions, each 

with multiple parts. A set of 83 SACD activities variables was created from teacher responses to 

individual questions. These variables were used to describe the level of social and character 

development formally occurring in every school. As such, they were also used as outcome variables 

(no additional construction was done for their use in the analysis) to evaluate the impact of the 

SACD programs on the use of SACD activities in the schools. The individual outcomes were 

grouped under six domains:10 (1) use of SACD activities, (2) use of SACD activities associated with a 

named SACD program, (3) use of classroom materials and strategies to promote social and character 

development, (4) use of schoolwide strategies to promote social and character development, (5) 

professional development of teachers to promote social and character development, and (6) teachers’ 

attitudes toward the promotion of social and character development and school practices conducive 

 

10 Outcomes were grouped into the six domains based on face validity alone. No psychometric procedures were used, 

and no measures of reliability or validity were obtained. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

34

to such promotion. Below is a description of each of the SACD Activities’ variables developed from 

the survey and grouped by domain. 

A. Use of Social and Character Development Activities. Teachers reported whether they engaged in 

activities to promote six SACD goals and behavior management. The six SACD goals were 

defined as follows: 

1. Violence Prevention and Peace Promotion: to make students and schools safer and more 

peaceful (e.g., reducing verbal and physical aggression, weapons, and bullying while 

promoting conflict resolution and peacemaking); 

2. Social and Emotional Competence: to foster positive emotional, behavioral, and 

interpersonal skill (e.g., anger management, recognizing emotions, empathy, respectful 

communication, and building and maintaining friendships); 

3. Character Education: to encourage the development of desirable traits, values, and ethics 

(e.g., respect, responsibility, honesty, fairness, and caring); 

4. Tolerance and Diversity: to promote understanding of and respect for the ways in which 

people differ (e.g., promote acceptance of others, celebrate cultural and ethnic differences, 

and reduce prejudice); 

5. Risk Prevention and Health Promotion: to reduce unhealthy behaviors and promote healthy 

ones (e.g., prevent alcohol, tobacco, and drug use and promote healthy life choices); and 

6. Civic Responsibility and Community Service: to promote participation in and commitment 

to the common good including service to the classroom, school, and community (e.g., raising 

resources for charity, recycling, and promoting understanding of one’s role in creating 

community). 

For each goal, teachers were asked to provide a brief description of any activity that addressed the 

goal in their class since the beginning of the school year, to indicate whether they did the activity with 

all or some students, and the frequency and amount of time devoted to these activities. Two sets of 

seven variables were developed from these responses. The first set of seven variables was composed 

of binary variables (yes or no) as to whether teachers reported that they had engaged in a SACD 

activity for each SACD goal separately (six variables) and whether they reported that they had 

engaged in a SACD activity for any of the SACD goals (one variable). The second set of seven 

variables was composed of binary variables as to whether teachers reported that they had engaged in 

a SACD activity for at least 1 hour per week for each SACD goal separately (six variables) and 

whether they reported that they had engaged in a SACD activity for at least 1 hour per week for any 

of the SACD goals (one variable). One hour was chosen as the cutoff point because most of the 

programs expected teachers to conduct weekly lessons that would require approximately this much 

time. 

In addition, respondents also reported separately on the use and amount of time devoted to behavior 

management activities. These reports were used to create two binary variables: (1) whether teachers 

reported that they had engaged in activities to support behavior management, and (2) whether 

teachers reported that they had engaged in activities to support behavior management for at least 1 

hour per week. 

In total, this domain contains 16 outcome variables: (1) Six variables regarding teachers’ reported use 

of SACD activities to promote each of the six SACD goals; (2) six variables regarding teachers’ 

reported use of SACD activities for more than an hour per week to promote each of the six SACD 

goals, (3) two variables regarding teachers’ reported promotion of any of the six SACD goals using 

SACD activities or using SACD activities for at least an hour per week, and (4) two variables 
 Chapter 1. The Social and Character Development Multiprogram Evaluation 

35

regarding teachers’ reported promotion of behavior management using activities or using activities 

for at least an hour per week 

B. Use of Social and Character Development Activities Associated With a Named Program. When teachers 

provided the data described above, regarding their engagement in activities to promote the six 

SACD goals and behavior management, they also could note whether the activity was associated 

with a specific SACD program. Two sets of 7 variables were developed from these responses for 

a total of 14 variables within this domain. The first set of seven variables was composed of 

binary variables (yes or no) as to whether teachers reported that they had engaged in a SACD 

activity linked to a named SACD program for each SACD goal separately (6 variables) and 

whether they reported that they had engaged in a SACD activity linked to a named SACD 

program for any of the SACD goals (1 variable). The second set of 7 variables was composed of 

binary variables as to whether teachers reported that they had engaged in a SACD activity linked 

to a named SACD program for at least 1 hour per week for each SACD goal separately (6 

variables) and whether they reported that they had engaged in a SACD activity linked to named 

SACD program for at least 1 hour per week for any of the SACD goals (1 variable). Few 

respondents mentioned a specific behavior management program by name, so no measures were 

created for use of behavior management associated with a named program. 

C. Classroom Materials and Strategies Used to Promote Social and Character Development. Teachers were 

asked about their use of six specific types of materials and 20 specific classroom strategies to 

promote social and character development in the classroom. Responses to these questions were 

used to create variables indicating the teacher-reported use of each type of material and strategy. 

Twenty-nine outcome variables are contained under this domain, including 28 binary variables 

(yes or no) and 1 count variable: (1) 7 binary variables regarding reported use of materials, with 1 

for each of the six types of materials plus a binary variable for no report of using any of the six 

materials; (2) 21 binary variables regarding reported use of the strategies, with 1 for each of the 

20 types of strategies plus a binary variable for reported use of any of the 20 strategies; and (3) 1 

count variable for the average number of strategies reported used. 

D. Schoolwide Strategies Used to Promote Social and Character Development. Teachers were asked whether 

their school used six specific types of schoolwide strategies to promote social and character 

development. These included morning announcements or videos, school assemblies, school 

newspapers or bulletins, special school days, special events, or other activities. Responses to 

these questions were used to create six binary variables for whether or not the teacher reported 

that the school used each specific schoolwide strategy. 

E. Professional Development of Teachers to Promote Social and Character Development. Teachers were asked 

about the training they received in instructional methods related to social and character 

development during the prior 12 months. Information was collected on whether any training was 

received, the total number of hours of training, whether training was received in each of the six 

SACD goal areas, and whether teachers participated in behavior management training. Nine 

outcome variables (eight binary and one count) were developed from these responses: (1) one 

binary variable as to whether teachers reported any such training within the past 12 months, 

(2) seven binary variables as to whether teachers reported any such training within the past 12 

months linked to each of the six SACD goals separately or to behavior management, and (3) one 

count variable as to the number of reported hours spent in such training during the past 12 

months. 

F. Attitudes Toward SACD Efforts and School Practices Conducive to SACD. Teachers were asked to 

characterize their own attitudes about efforts to promote social and character development 

(enthusiastic, cooperative, or in open dislike) and to characterize whether the culture of their 

school reflected the use of six practices (most of the time) conducive to the social and character 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

36

development of students. These practices included teacher modeling of positive character and 

behavior with students and with other staff, involvement of students in discussions, students’ 

voice in governance, school encouragement of parent involvement, and discipline practices that 

included promoting development rather than only punishing misbehavior. Nine binary outcome 

variables were developed from these responses: (1) three regarding whether teachers reported 

being enthusiastic, cooperative, and in open dislike to promoting social and character 

development, and (2) six regarding the reported use of each school practice. 

III. Teacher Experience 

The Teacher Survey on Professional Development and Training was developed for the SACD 

multiprogram evaluation as a 10-item questionnaire to which teachers responded about their 

demographic background, education, teaching experience, and engagement in professional 

development activities. Items for this survey were drawn from those used in an earlier study on 

teacher quality carried out by the National Center for Education Statistics (Lewis et al. 1999). 

Principal Interview 

Principals were individually interviewed in a 45-minute period about the prevalence and characteristics of the 

activities used to promote social and character development and about the percentage of staff trained in them 

and staff support of their use. Interviewers used both open-ended prompts and questions with defined 

answers. In fall 2004 and spring 2005, interviews were conducted in person. However, the recording of the 

open-ended responses was not of acceptable quality, including differences in the level of detail provided by 

interviewers. For the second and third years of the project, principal interviews were done by a smaller group 

of more experienced staff through phone interviews. Because of the quality concerns, the spring 2005 openended data are not reported. The principal interviews were used to develop six measures of the SACD 

implementation in all the schools. 

I. Percentages of Schools Using Activities to Address Each SACD Goal 

The percentages of treatment and control schools for which the principal noted that social and 

character development activities were directed toward each of the following six SACD goals: 

(1) Violence Prevention and Peace Promotion, (2) Social and Emotional Development, (3) Character 

Education, (4) Tolerance and Diversity, (5) Risk Prevention and Health Promotion, and (6) Civic 

Responsibility and Community Service. In addition, principals reported on the number of activities 

per goal. Both open- and closed-ended questions were used. 

II. Delivery Method 

Both open- and closed-ended questions were used to code data into yes or no variables on 13 methods 

and seven actors used to deliver activities to promote social and character development. Delivery 

methods included the use of assemblies, structured lessons, targeted reading or writing, and words of 

the day; and the actors included the classroom teacher, other teachers, and guidance counselors or 

social workers. 

III. Targeting 

Principals were asked to specify whether the SACD activities were targeted to the whole school, specific 

grades, or other. If the principal selected other, he or she was asked to describe which students the 

activity targeted. Both open- and closed-ended questions were used. 

IV. Teacher Professional Development 

Data about teacher training were collected in both open- and closed-ended questions. In the openended responses, principals described the training teachers received for each activity designed to 

promote social and character development. These responses were coded to indicate whether a 

teacher received guided or self-guided training. In the closed-ended responses, principals reported, at 
 Chapter 1. The Social and Character Development Multiprogram Evaluation 

37

the school level, the percentages of teachers and staff who received training in activities to promote 

social and character development and how many hours of training teachers received. 

V. Staff Response 

Data about the staff response and support of SACD were collected in the closed-ended questions. 

Principals reported, at the school level, on the level of staff enthusiasm for activities to promote 

social and character development. Principals also reported on whether teachers and staff modeled 

positive behaviors, whether students were involved in decisionmaking at the school, and whether 

discipline strategies focused on promoting development rather than only punishing misbehavior. 

VI. Parental Involvement 

Data on whether parents were actively or passively involved in activities to promote social and 

character development were coded from responses to open-ended questions that asked principals to 

describe how parents were involved in these activities. 

Implementation Fidelity Rating 

In order to develop a common implementation measure for the multiprogram evaluation, the SACD 

Research Program Global Fidelity Ratings form was provided to each grantee for each of the 3 study years. 

The form contained a request that each grantee rate the fidelity with which the SACD program was 

implemented in each treatment school during each school year. The instructions explained that, for the 

fidelity ratings, the standard of comparison should be program standards or benchmarks. Furthermore, the 

instructions explained that the basis of all fidelity ratings should be the site-specific fidelity data collected as 

part of the grantee’s own evaluation work,11 the program benchmarks, and the rater’s own judgment. In 

addition to rating each school, each grantee was asked to provide a justification for each rating. Two 

individuals with the most experience in monitoring program implementation in the SACD treatment schools 

and interpreting the site-specific fidelity data were to complete the ratings independently, discuss each rating, 

and then decide on a final consensus rating for each treatment school. 

For each school, each rater was to rate the amount of implementation and the quality of implementation. 

After making individual ratings, each pair was to discuss their ratings and arrive at a consensus rating. The 

rating values ranged from 1 to 5. For example, a rating of 1 for the amount of implementation at a school 

indicated that ―On average, the program was delivered at a level substantially below program benchmarks.‖ A 

rating of 2 indicated ―On average, the program was delivered at a level somewhat below program benchmarks.‖ 

A rating of 3 indicated the program met benchmarks; while ratings of 4 and 5 were for delivery at levels 

somewhat above and substantially above program benchmarks, respectively. The ratings for the quality of 

implementation at a school were analogous. For example, a rating of 1 for the quality of implementation at a 

school indicated that ―On average, the program was delivered with quality substantially below program 

benchmarks.‖

Construction of Outcome Variables 

To develop a set of student and school climate outcome measures with better psychometric characteristics 

and to determine whether a smaller number of scales could represent them,12 a set of exploratory factor 

analyses using the fall 2004 survey data for each group of respondents (children, primary caregivers, and 

 

11 For their independent evaluations, the research teams developed their own approaches and used a variety of methods 

(including training logs, teacher reports, and classroom observations) to measure the implementation fidelity of their 

separate programs. 

12 A preliminary factor analysis of the fall 2004 data indicated that certain scales had poor internal consistency while 

others were highly intercorrelated. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

38

teachers) was done on a randomly selected half of the sample.13 Exploratory analysis of data for each reporter 

found that the item responses of children, primary caregivers, and teachers were optimally represented by 10, 

3, and 5 underlying factors, respectively, for a total of 18 constructs. Factors representing 2 constructs 

(Altruistic Behavior and Problem Behavior) were identified for all three respondent groups. A factor 

representing a third construct (Positive Social Behavior) was identified for both primary caregivers and 

teachers, and a factor representing 2 highly related constructs was identified for teachers and children 

(Academic Competence and Motivation, and Engagement with Learning). The remaining identified factors 

were specific to each responding group. These 18 factors were subjected to a confirmatory factor analysis 

with the remaining half of the fall 2004 sample using structural equation modeling procedures. Two 

additional constructs (Feelings of Safety, and Student Support for Teachers) were developed through a 

principal component analysis of the Teacher Report on Classroom and School. Further analysis determined 

that the results were robust across gender and race/ethnic subgroups, individually for each of the seven 

SACD programs, when missing item responses were imputed, and when the full fall 2004 and spring 2005 

data were used. As a result of these analyses, 20 constructs were identified and these were used as the 

outcome measures for the SACD evaluation. Table 1.5 lists the outcome measures by report and details the 

number of items in each, their reliability, and the original measures that provided the items for them. Changes 

in these measures may be beneficial or detrimental depending on the specific outcome, so a positive change 

does not automatically reflect an improvement or a negative change a decline. The +/- sign in parentheses 

next to each outcome in the first column of table 1.6 notes whether a gain or loss can be interpreted as 

beneficial for each outcome (as do the first columns of all tables reporting impacts on the outcomes). The 20 

outcome measures were grouped under four domains based on the conceptual model: Social and Emotional 

Competence, Behavior, Academics, and Perceptions of School Climate (table 1.6).14

 

13 See appendix B for details on the development of the outcome measures. 

14 One further analysis was done to determine whether the multiple reports for each of three constructs within the 

Behavior domain could be combined: Altruistic Behavior (child, caregiver, and teacher), Positive Social Behavior 

(caregiver and teacher), and Negative Behavior (child, caregiver, and teacher). A series of exploratory multitrait, 

multirespondent analyses found that these outcomes were affected by both the behavior construct being assessed and by 

the reporter. As a result, separate outcome measures for each reporter (child, caregiver, and teacher) were retained. 
Table 1.5. Outcome variables 

Report/Outcome variable Items Range Reliability1 Original measure from which items were taken

Child Report

 Altruistic Behavior 8 0-3 0.88 Altruism Scale, Child Version

 Engagement with Learning 4 1-4 0.84 4 items from Engagement vs. Disaffection with Learning Scale

 Normative Beliefs About 

 Aggression

8 1-4 0.83 Normative Beliefs About Aggression Scale

 Self-Efficacy for Peer Interaction 12 1-4 0.83 Self-Efficacy for Peer Interaction Scale

 Empathy 11 1-3 0.78 11 items from the Children’s Empathy Questionnaire

 Problem Behavior 12 0-3 0.86 6 items from Frequency of Delinquent Behavior Scale and 6 items from 

Aggression Scale

 Positive School Orientation 10 1-4 0.86 9 items from Sense of School as a Community Scale and 1 item from Feelings of 

Safety at School Scale

 Negative School Orientation 8 1-4 0.78 4 items from Engagement vs. Disaffection with Learning Scale and 4 items from 

Sense of School as a Community Scale

 Student Afraid at School 4 1-4 0.79 4 items from Feelings of Safety at School Scale

 Victimization at School 6 0-3 0.86 Victimization Scale

Primary Caregiver Report

 Altruistic Behavior 8 1-4 0.88 Altruism Scale, Primary Caregiver Version

 Positive Social Behavior 25 1-4 0.93 6 items from Responsibility Scale and 19 items from Social Competence Scale

 Problem Behavior 20 1-4 0.86 12 items from BASC Aggression Subscale, 6 items from BASC Conduct 

Problems Subscale, and 2 items from the Responsibility Scale

See notes at end of table.

39

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.5. Outcome variables—Continued

Report/Outcome variable Items Range Reliability1 Original measure from which items taken

Teacher Report on Student

 Altruistic Behavior 8 1-4 0.89 Altruism Scale, Teacher Version

 Academic Competence and 

 Motivation

5 1-5 0.95 Academic Competence and Motivation Scale

 Positive Social Behavior 25 1-4 0.97 6 items from Responsibility Scale and 19 items from the Social Competence

Scale

 Problem Behavior 23 1-4 0.95 14 items from BASC Aggression Subscale, Teacher Version, 7 items from BASC 

Conduct Problems Subscale, Teacher Version and 2 items from Responsibility 

Scale

 ADHD-Related Behavior 10 1-4 0.91 5 items from DSM-IV Criteria for ADHD and 5 items from IOWA Conners 

Teacher Rating Scale

Teacher Report on Classroom 

 and School

 Feelings of Safety 4 1-5 0.89 Feelings of Safety at School Scale

 Student Support for Teachers 7 1-5 0.89 7 items from the School-Level Environment Questionnaire

1

Internal consistency as measured by Cronbach’s alpha.

SOURCE: The Social and Character Development (SACD) Research Program.

40

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

41

Table 1.6. Outcomes, by domain and data instrument 

Instrument

Domain/Outcome Child Report

Primary 

Caregiver 

Report

Teacher 

Report on 

Student

Teacher 

Report on 

Classroom 

and School

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction (+) 

 Normative Beliefs About Aggression (-) 

 Empathy (+) 

Behavior Domain

 Altruistic Behavior (+)   

 Positive Social Behavior (+)  

 Problem Behavior (-)   

 ADHD-Related Behavior (-) 

Academics Domain

 Engagement with Learning (+) 

 Academic Competence and Motivation (+) 

Perceptions of School Climate Domain

 Positive School Orientation (+) 

 Negative School Orientation (-) 

 Student Afraid at School (-) 

 Victimization at School (-) 

 Feelings of Safety (+) 

 Student Support for Teachers (+) 

NOTE: Abbreviations are

 ADHD: Attention deficit hyperactivity disorder

 : Outcome addressed

 Blank cell: Outcome not addressed

The +/- signs in parentheses indicate the direction of a beneficial outcome.

SOURCE: The Social and Character Development (SACD) Research Program.

Construction of Moderator Variables 

The impact of the SACD programs was hypothesized to vary by student, family, and community 

characteristics and by the level of implementation. Collected student demographic characteristics (such as 

gender) could be directly used in the analysis. Other characteristics required the use of scales and the 

construction of moderator variables for use in the analysis. 

Initial Risk Factors 

Certain individual, family, and community characteristics are associated with greater prevalence of child 

behavior problems. Students with these characteristics began the SACD study with initial risks that were 

hypothesized to lead to more negative outcomes. To help determine the impact of these initial risks on the 

effects of the SACD programs, four risk factors were constructed to represent (1) socioeconomic risk, 

(2) family risk, (3) perceived community risk, and (4) child behavior risk. To select the measures to be 

included in the first three risk domains, associations were examined between the risk measures and a set of 

baseline outcome measures. 

Socioeconomic risk was defined as the number of the following risk factors present in the child’s life at the 

initial data collection: (1) the child was in a single-parent household, (2) the child was in a low-income 

household (below 135% of the poverty level), and (3) the child’s primary caregiver had failed to graduate 

from high school. A cumulative risk index across the three risk factors was calculated by adding the number 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

42

of these factors each child had: 0 (no risk factors), 1 (one risk factor), and 2 (two or three risk factors). Scores 

ranged from 0 to 2, with a mean of 0.73 and a standard deviation of 0.74. The fourth measure of 

socioeconomic risk collected, employment status of the child’s caregiver, was not included in the 

socioeconomic risk measure because it was not found predictive of baseline outcomes when the other three 

measures were included. 

Family risk was measured using the Poor Monitoring and Supervision Subscale from the Alabama Parenting 

Questionnaire (Shelton et al. 1996) collected in the Primary Caregiver Report. Scores ranged from 1 to 3.5, 

with a mean of 1.1 and standard deviation of 0.20. The other two measures of family risk collected, the 

Positive Parenting Subscale and the Confusion, Hubbub, and Order Scale, appeared to reflect reporter 

variance rather than construct variance and so were not included in family risk measure. 

Perceived community risk was measured using items on the Community Risk Scale (Forehand et al. 2000), 

collected in the Primary Caregiver Report. Scores ranged from 1 to 4, with a mean of 1.5 and a standard 

deviation of 0.64. The other two measures of community risk collected, the Community Resources Scale and 

the Child-Centered Social Control Scale were less consistently related to the outcome variables and so were 

not included in the community risk measure. 

Child behavior risk was measured using the Conduct Problem Subscales from the BASC (Reynolds and 

Kamphaus 1998) collected in the Teacher Report on Student and the Primary Caregiver Report. For the 

teacher-reported child behavior risk variable, scores ranged from 42 to 120, with a mean of 50.82 and a 

standard deviation of 9.78. For the parent-reported child behavior risk variable, scores ranged from 36 to 113, 

with a mean of 50.33 and a standard deviation of 9.02. 

Fidelity of Implementation 

How well an intervention was implemented was hypothesized to be associated with how the SACD programs 

affected the outcomes. The constructs used for the fidelity data analysis were developed so that the SACD 

evaluation would be able to examine whether impacts on student outcomes were estimated to be larger in 

treatment group schools that achieved high-fidelity implementation of their SACD program (relative to their 

matched control group counterparts) than were impacts in treatment group schools that achieved lower 

fidelity implementation (relative to their control group counterparts). Data collection was designed in such a 

way that each program was to be assessed relative to the benchmarks or expectations established for the 

specific program. Consensus ratings on a quality-of-implementation 5-point scale and on a quantity-ofimplementation 5-point scale were to be used to categorize the treatment schools as high in implementation 

fidelity (high-fidelity subgroup) or low in implementation fidelity (low-fidelity subgroup) (see Measures 

section). The ratings on quality and quantity of implementation were found to be highly correlated for each 

school (0.78 in Year 1, 0.82 in Year 2, and 0.83 in Year 3). As a result, they were combined into a single 10-

point composite measure of fidelity of implementation, which was used to create the high- and low-fidelity 

subgroups. 

The cutoff point on the consensus score used to place the schools into the two fidelity subgroups was 

established using two criteria. First, every year a pair of raters independently rated each school within a 

specific program and then agreed on a consensus score for each school.15 The cutoff point was to be set so 

that the individual ratings were to agree on the placement for the largest possible percentage of schools. The 

agreement between the two raters’ individual scores on their schools’ placement was measured using Cohen’s 

kappa, which adjusts the empirical match rate by the expected number of exact matches that would occur by 

chance. Second, a sufficient number of schools were to be placed in each subgroup so that the size of each 

 

15 Over the 3 years, the ratings were not always consistently provided by the research teams. In Year 1, four teams 

provided both the ratings from the independent raters and the consensus ratings, two teams provided only the 

independent ratings, and one team provided a single rating. In Year 2, all teams provided the requested ratings. 

In Year 3, six teams provided the requested ratings and one team provided only the independent ratings. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

43

was sufficient to generate precise estimates. When applying these two criteria to the spring 2005 data, 

placement into the high- versus low-fidelity subgroup was defined using a cutoff of 7 for the consensus score: 

high implementers had consensus scores that were 7 or greater; and low implementers had consensus scores 

less than 7. The cutoff value of 7 led to the highest agreement rate between the raters and the most balanced 

division of the sample. 16

 

The seven rater pairs placed 30 of the 36 schools in the same fidelity group, and there was an overall kappa 

value of 0.67.17 Twenty schools were categorized as high implementers and 22 were categorized as low 

implementers. Replication of this procedure with the 2006 and 2007 data yielded similar results, including 

kappas of 0.71 and 0.76, respectively, and 20 and 19 high-implementing schools and 22 and 23 lowimplementing schools, respectively. Table 1.7 shows these results by program and provides the inter-rater 

reliability for the two raters in each program, the agreement rate between raters for the categorization of 

schools as high implementers, and Cohen’s kappa for this rate. The overall inter-rater reliability coefficients 

(Cronbach’s alpha) were 0.71 in Year 1 (0.83 if the LBW’s alpha of -0.08 is excluded), 0.90 in Year 2, and 0.94 

in Year 3. The program-specific alphas were 0.70 or higher in Year 1 (except for LBW and SS, for which only 

a single rating was provided), 0.74 or higher in Year 2, and 0.72 or higher in Year 3. These findings suggest 

that, in most programs, the variance of ratings across observers was small relative to the variance of mean 

ratings across schools. 

 

16 Alternative cutoff values led to lower kappa values (almost half the size) and less balanced groups. 

17 While there were 42 treatment schools, one research team did not provide rater-specific information for the spring 

2005 fidelity data so there were only 36 schools for which rater-specific information was available. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

44 

Table 1.7. Fidelity ratings of program implementation at treatment schools 

Program

Number of 

treatment schools

Inter-rater reliability Rater fidelity 

agreement rate (%) Cohen’s kappa1(Cronbach’s alpha)

Year 1

 Total 42 0.71 83 0.67

 ABC 6 1.00 100 1.00

 CSP 5 0.70 80 0.62

 LBW 6 -0.08 67 —

 PA 7 0.85 86 0.72

 PATHS 5 0.78 80 0.55

 4Rs 7 0.84 86 0.70

 SS26 — — —

Year 2

 Total 42 0.90 86 0.71

 ABC 6 0.98 100 1.00

 CSP 5 0.80 80 0.55

 LBW 6 0.83 100 1.00

 PA 7 0.74 71 0.46

 PATHS 5 0.93 60 0.17

 4Rs 7 0.78 86 0.59

 SS 6 1.00 100 1.00

Year 3

 Total 42 0.94 88 0.76

 ABC 6 0.99 83 0.67

 CSP 5 0.99 100 1.00

 LBW 6 0.97 100 1.00

 PA 7 0.98 100 1.00

 PATHS 5 0.86 60 0.29

 4Rs 7 0.72 71 0.30

 SS 6 0.99 100 1.00

— Not available.

1 Cohen’s kappa adjusts the agreement rate for the expected number of Rater 1 and Rater 2 exact matches that would occur 

because of chance.

2

In Year 1, Second Step provided a single consensus rating.

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

45

In Year 1, these high- or low-fidelity subgroups were used directly in the analysis. In Years 2 and 3, the actual 

subgroups used in the analysis were based on a school’s assignment to the high- or low-fidelity subgroup in 

each year. This was done to address the possibility that a program’s fidelity over time might have cumulative 

effects on students. As a result, three subgroups were created for Year 2 and Year 3 by combining the annual 

results: (1) high: placement in the high-fidelity subgroup in all years; (2) low: placement in the low-fidelity 

subgroup in all years; and (3) mixed: placement in the high- and low-fidelity subgroups in different years.

The multiprogram evaluation examined seven different SACD programs, each with unique features, and the 

fidelity rating measured fidelity of implementation relative to the targets established for each specific 

intervention.18 In this way, the ratings were standardized relative to each site’s program-specific benchmarks 

and could be compared across programs and years. Thus, for example, a rating of 6 at a school using one 

program shows higher fidelity to implementation of that program than a rating of 4 at another school using a 

different program (relative to that program’s benchmarks). The compromise in developing this fidelity 

measure was that basing the ratings on implementation relative to each program’s standards allows for crossprogram comparisons but it cannot account for differences among the programs’ implementation standards. 

For example, one SACD program might be simple to implement and so require a less intensive 

implementation, which would make high fidelity easier to reach. Another program might be very difficult to 

fully implement and require intensive implementation, which would make high fidelity difficult. As a result, 

schools using the first program might score higher on the fidelity rating scale than schools using the second 

program because high fidelity was easier for them to obtain. However, if the second program was more 

effective than the first, even when not fully implemented, then an analysis comparing the fidelity subgroups 

might erroneously lead to the conclusion that it is better to implement programs with a low level of fidelity. 

In addition, high and low fidelity were not randomly assigned, therefore programs and schools were not 

necessarily represented in the fidelity subgroups by chance. Instead, they may have ended up in their 

subgroup because of pre-existing characteristics. Comparisons of the estimated impacts at high-fidelity 

schools to those of low-fidelity schools would lead to an erroneous conclusion regarding the effect of fidelity 

of implementation if the impact of fidelity was due to other differences between the subgroups. In other 

words, if high and low fidelity occurred because of differences among schools and these differences also had 

an impact on the student outcomes, then fidelity would be an outcome of these differences between the 

schools rather than a cause of student outcomes. These differences might be more observable (e.g., schools 

with high fidelity had students who, on average, had a different socioeconomic status than schools with low 

fidelity) or less observable (e.g., differences in the talent and dedication of the staff or strength of school 

leadership). 

In sum, the fidelity ratings provided an indicator of the extent to which the SACD programs were 

implemented according to program benchmarks. Because of how the ratings were constructed and the 

possibility that program and school characteristics may have contributed to each school’s rating, differences 

in estimated impacts by implementation group could reflect differences in the level of implementation and/or 

underlying differences between the students, the schools, and the SACD programs. As these effects cannot 

be fully disentangled, the analysis making use of the fidelity data was considered exploratory rather than 

definitive. 

Data Collection 

The SACD study had five data collection points, beginning with the initial collection of data in fall 2004 (start 

of third grade) and followed by four follow-up points: spring 2005 (end of third grade), fall 2005 (start of 

 

18 One drawback of this approach is that the fidelity data provide little information about why implementation was of a 

certain quality. The fidelity data collected by each research team using team-chosen criteria were expected to provide this 

type of information rather than facilitate the multiprogram analysis. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

46

fourth grade), spring 2006 (end of fourth grade) and spring 2007 (end of fifth grade).19 Ideally, fall data 

collections would have occurred at the very beginning of the school year to reduce the possibility that the 

implementation of the program could affect the data (for fall 2004) and to maximize the length of time that 

the intervention would have to affect the annually collected outcomes. Spring data collections would ideally 

have occurred near the end of the school year in order to maximize the length of time the program had to 

affect annually collected outcomes. 

For practical reasons, fall data collections were often delayed for several weeks to allow school populations to 

settle, to obtain primary caregiver consent, and to avoid disrupting planned school activities. Spring data 

collections occurred earlier than the end of school to fit into school schedules that included spring district 

testing and vacations. 

Across the seven SACD programs, the fall 2004 data collection encompassed a 3-month period, although onsite data collection in each program typically took 2 to 4 weeks (table 1.8). The earliest fall 2004 data 

collection started on August 23, 2004 (LBW), and the last ended on November 12, 2004 (ABC). In all 

programs except LBW, implementation of the program started before the fall 2004 data collection visit. In 

the other six programs, the interval between program implementation and fall 2004 data collection was 2 to 6 

weeks, which may have had some effect on student reports. Teachers and principals received training on the 

intervention before the fall 2004 data collection, and this could have affected their reports on the use of 

activities to promote social and character development in their classrooms and schools. As a result, the fall 

2004 reports, especially those from teachers and principals, are unlikely to reflect the true pre-intervention 

conditions but instead capture what was being done near the beginning of the evaluation. For this reason, the 

fall 2004 data are termed ―initial data‖ rather than ―baseline data.‖

The spring 2005 data follow-up occurred between March 21 and June 3, with most of it concentrated in April 

and May. On-site data collection in each program typically took 2 to 4 weeks. Because of differences in the 

start dates of implementation and the dates of spring data collection, there was variation among the programs 

in terms of how long they had been implemented by the time of the spring data collection. This interval 

ranged from a low of 26 weeks for 4Rs to a high of 36 weeks for CSP. These differences should be kept in 

mind when viewing the results of the analysis comparing the outcomes of students at spring 2005 in 

treatment schools with those of students at spring 2005 in control schools (while controlling for the fall value 

of the relevant outcome measure), although they did not affect individual program evaluations as each 

program’s treatment and control group was compared over the same period. 

The fall 2005 data collection encompassed a 3-month period from August to November, with on-site data 

collection in each program typically taking 3 to 4 weeks. The gap between the start of school and the fall data 

collection was 5 weeks for six of the programs and 8 weeks for the seventh, SS. The 2006 spring data 

collection lasted for approximately a 2-month period, occurring between March 20, 2006, and May 26, 2006. 

The length of the data collection ranged from less than 2 weeks to about 6 weeks. The time between the start 

of the school year and the spring data collection varied from 29 to 40 weeks across the programs. 

The spring 2007 data collection lasted approximately 2 months and occurred between March 26 and June 1. 

On-site data collection in each program took 2 to 5 weeks. The time between the start of the school year and 

spring data collection ranged from 29 to 40 weeks, similar to Year 2. 

 

19 Data collection for the second cohort (discussed in appendix A) began in fall 2005 (start of third grade) with followups in spring 2006 (end of third grade), and spring 2007 (end of fourth grade). 
Table 1.8. Period between implementation and data collection and between data collections, by program 

Data collection schedule ABC CSP LBW PA PATHS 4Rs SS

Fall 2004 data collection

 Start date 10/18/0418/30/04 8/23/04 10/12/04 10/18/04 10/18/04 10/11/04

 End date 11/12/04 9/16/04 9/9/04 10/28/04 10/28/04 11/10/04 10/29/04

 Calendar weeks from program implementation 6 4 †

2

2 6 4 5

Spring 2005 data collection

 Start date 4/11/05 4/11/05 5/2/05 5/16/05 3/28/05 3/21/05 4/25/05

 End date 5/6/05 4/29/05 5/20/05 6/3/05 4/28/05 5/5/05 5/13/05

 Calendar weeks from end of fall data collection 21 30 34 29 22 17 25

 Calendar weeks from program implementation 31 36 33 33 29 26 33

Fall 2005 data collection

 Start date 10/10/0519/26/0538/29/05 10/11/05 10/11/05 10/11/05 10/24/05

 End date 10/27/05 10/20/05 9/23/05 11/2/05 10/27/05 11/16/05 11/11/05

 Calendar weeks from start of school 5 5 5

4

5 5 5 8

Spring 2006 data collection

 Start date 3/27/06 4/24/06 5/1/06 5/1/06 3/20/06 3/27/06 4/24/06

 End date 5/1/06 5/5/06 5/19/06 5/26/06 4/11/06 5/8/06 5/12/06

 Calendar weeks from end of fall data collection 21 27 31 26 20 20 23

 Calendar weeks from start of school 29 35 40 34 29 29 34

See notes at end of table.

47

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.8. Period between implementation and data collection and between data collections, by program—Continued

Data collection schedule ABC CSP LBW PA PATHS 4Rs SS

Spring 2007 data collection

 Start date 4/18/07 4/26/07 5/1/07 5/7/07 3/26/07 3/26/07 4/23/07

 End date 5/8/07 5/10/07 5/16/07 6/1/07 4/27/07 4/26/07 5/10/07

 Calendar weeks from start of school 32535 40635 29 29 34

† Not applicable.

1

At two schools data collection began the week of 9/20 in 2004 and 2005.

2 Data collection occurred before program implementation.

3 Data collection at one school occurred 3 weeks earlier.

4 One-third of the schools started 2 weeks later.

5

Two schools started several weeks earlier.

6

Two schools started 2 weeks later.

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

There may be slight discrepancies in the end dates for field data collection due to scheduling of Child Report make-up sessions and principal interviews.

SOURCE: The Social and Character Development (SACD) Research Program.

48

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

49

Training of Data Collectors 

Data collection was done by MPR (along with its subcontractor Decision Information Resources, Inc. [DIR]) 

and the research teams. MPR established on-site data collection teams to administer and collect the Child 

Report from the study schools. These teams were trained by senior MPR and DIR data collection managers 

at one of two 4-day training sessions held August 7-10, 2004, in Tennessee and September 18-21, 2004, in 

New Jersey. At these sessions, data collectors were introduced to the study objectives, reviewed data 

collection procedures, discussed working in schools, and reviewed the distribution and collection of all survey 

materials. To monitor quality and consistency of data collection across all sites, senior MPR and DIR data 

collection managers observed the first day or two of on-site data collection at each site in fall 2004 and spring 

2005. Because most of the data collectors from fall 2004 returned for spring 2005 data collection, MPR held a 

refresher training session in spring 2005. This refresher training included several activities: (1) distribution of a 

revised training manual, (2) a videotape and detailed memorandum on changes in procedures and/or 

instruments since fall data collection for home study, (3) a telephone conference call to review home study 

materials, and (4) an in-person half-day meeting a few days before the beginning of data collection. Ninetythree percent of the collectors returned for the fall 2005 data collection; at one site it proved difficult to hire 

new collectors so a team was composed of collectors from other sites. MPR provided training through a 

review memo mailed to collectors, conference calls, and a 4-hour refresher course with each team prior to 

data collection. Again an MPR or DIR representative worked with each team for the first day or two of data 

collection. By spring 2006, turnover was high enough that new collectors were provided a 2-day training in 

March or April. Each team also went through the 4-hour refresher course prior to data collection. A similar 

approach was taken for spring 2007, with a 2-day training for new data collectors and a 4-hour refresher 

course before data collection. 

The two teacher reports and the primary caregiver report were self-administered questionnaires prepared by 

MPR. They were distributed either by MPR or the research team during on-site data collection (in some sites 

teachers received their questionnaires before spring on-site data collection). Teachers received their surveys 

directly and sent the primary caregiver surveys home with students. If a completed Primary Caregiver Report 

was not returned within 2 to 4 weeks, MPR attempted to contact the child’s primary caregiver by telephone, 

and if the primary caregiver was willing, the Primary Caregiver Report was administered as a computerassisted telephone interview.

Consent Rates, Completion Rates, and Percentage of Sample With Data 

Data collection required a two-part process: consent and completion of data reports. First, written consent 

had to be obtained from primary caregivers before their children could fill out the Child Report, their 

children’s teachers could fill out the Teacher Report on Student, and they themselves could fill out the 

Primary Caregiver Report. Written consent from teachers had to be obtained before they could fill out the 

Teacher Report on Classroom and School. Second, even when consent was received, reports might not be 

completed for other reasons. Students may have decided not to complete the Child Report or were absent on 

the days of the data collection and make-up sessions. Primary caregivers may have failed to complete the 

Primary Caregiver Report or did not complete the follow-up phone interview. Teachers may have failed to 

complete the Teacher Report on Student or the Teacher Report on Classroom and School. 

The percentage of the sample for which there are data is calculated by multiplying the consent rate and the 

completion rate for each report. This section discusses the percentage of the sample for which there are data 

as follows: (1) the overall sample, which includes treatment and control groups; (2) students from the original 

cohort (stayers) versus new entrants; and (3) by individual SACD program. The consent and completion rates 

are also provided for the first two and are available in chapters 2 to 8 by program. 

Table 1.9 presents the percentage of sample for which there are data for each report by year and by treatment 

versus control group. For the Child Report and the Teacher Report on Student, between 61 percent and 65 

percent of students have data over the 3 years. In Year 1 (third grade), the treatment group had statistically 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

50

significantly larger percentages of data but this was not the case in Years 2 and 3. For the Primary Caregiver 

Report, the percentage of primary caregivers with data ranged from 57 percent in Year 1 to 46 percent in 

Year 3, due to declines in completion rates. The treatment group had a significantly higher percentage of 

primary caregivers with data in the fall of Year 1 but not thereafter. For the Teacher Report on Classroom 

and Schools, the percentage of teachers with data ranged from 87 percent to 90 percent over the 3 years, with 

no significant difference between the treatment and control schools. 

Table 1.10 examines differences for stayers versus new entrants. For all three reports on students (student, 

primary caregiver, and teacher) stayers had statistically significantly higher percentages of data than new 

entrants due to higher consent rates. 

Table 1.11 presents the percentage of sample with data for treatment versus control group by SACD 

program. Out of 112 comparisons, 18 showed statistically significant differences (6 such differences would be 

expected by chance) with all but 2 favoring the treatment group. These break out as 6 differences in the Child 

Report, 3 differences in the Primary Caregiver Report, 7 differences in the Teacher Report on Student, and 2 

differences in the Teacher Report on Classroom and School. 

In sum, over the 3 years of the SACD evaluation, 61 percent to 65 percent of the student sample had data 

(with 46% to 57% having primary caregiver data). 20

 

 

20 The majority of the students without data (67% to 97%, depending on report) did not have consent to take part in the 

study. As data could not be collected on these students, no comparisons could be made to determine whether they 

significantly differed from those taking part in the study. 
Table 1.9. Consent rates, completion rates, and percentage of sample with data from each report 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 Student sample size 6,567 3,367 3,200 6,597 3,388 3,209 6,415 3,327 3,088 6,249 3,172 3,077

Child Report (percent) 

 Primary caregiver consent rate 65 67** 63 66 68** 64 67 67 66 66 67 66

 Student completion rate 94 93* 94 96 96 96 95 96 95 96 97 96

 Students with data161 62* 60 63 65** 61 63 65 62 64 65 63

Primary Caregiver Report (percent)

 Primary caregiver consent rate 63 64** 61 64 66** 62 64 65 63 64 65 64

 Primary caregiver completion rate 92 92 92 80 80 81 78 78 77 72 71 72

 Primary caregiver with data157 59* 56 51 52 50 50 51 49 46 46 46

Teacher Report on Student (percent)

 Primary caregiver consent rate265 67** 63 66 68** 64 67 67 66 66 67 66

 Teacher completion rate 96 96 96 99 99 99 100 100** 99 98 98 99

 Students with data162 64** 61 65 67** 63 66 67 65 65 66 65

Teacher Report on Classroom and School

 (3rd- to 5th-grade teachers) (percent)

 Teacher consent rate 96 98*** 92 95 97* 94 95 97 94 96 97 95

 Teacher completion rate 91 90 93 91 90 91 94 94 94 92 91 93

 Teachers with data187 88 86 87 88 86 90 90 89 89 88 89

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

*** Treatment group significantly different from control group at the .001 level.

1 Calculated as consent rate x completion rate.

2

The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

51

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.10. Consent rates, completion rates, and percentage of sample with data: Stayers versus new entrants 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total Stayers

New 

entrants Total Stayers

New 

entrants Total Stayers

New 

entrants

 Student sample size 6,597 6,203 394 6,415 5,110 1,305 6,249 4,500 1,749

Child Report (percent)

 Primary caregiver consent rate 65.9 66.5*** 56.1 66.5 69.4*** 55.2 66.4 69.1*** 59.7

 Student completion rate 95.9 96.0 94.6 95.4 95.2 96.7 96.2 96.1 96.4

 Students with data163.2 63.8*** 53.0 63.5 66.0*** 53.4 63.9 66.4*** 57.5

Primary Caregiver Report (percent)

 Primary caregiver consent rate 63.8 64.5*** 53.8 64.2 66.7*** 54.1 64.2 66.3*** 58.8

 Primary caregiver completion rate 80.3 80.2 81.6 77.7 77.5 78.6 71.7 70.7* 74.5

 Primary caregivers with data151.3 51.7** 43.9 49.9 51.7*** 42.5 46.0 46.9* 43.8

Teacher Report on Student (percent)

 Primary caregiver consent rate265.9 66.5*** 56.1 66.5 69.4*** 55.2 66.4 69.1*** 59.7

 Teacher completion rate 98.8 98.9* 97.3 99.7 99.7 99.4 98.5 98.6 98.0

 Students with data165.1 65.7*** 54.6 66.3 69.2*** 54.9 65.4 68.1*** 58.5

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

*** Treatment group significantly different from control group at the .001 level.

1 Calculated as consent rate x completion rate.

2

The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

52

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.11. Percentage of sample with data by report and program1

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

Child Report

 ABC 59.4 62.7* 56.7 62.5 66.0 60.0 62.3 62.7 62.0 63.4 69.1** 59.5

 CSP 64.8 62.8 66.6 67.8 67.2 68.4 66.5 66.2 66.7 65.4 59.2*** 71.2

 LBW 60.1 61.5 58.4 60.8 61.4 60.0 60.4 60.4 60.3 58.6 59.1 57.8

 PA 74.4 73.9 74.8 71.2 71.6 70.7 66.6 66.8 66.4 76.5 78.3 74.7

 PATHS 52.2 54.4 50.1 56.1 58.6 53.8 55.8 59.0 52.8 55.4 59.5* 51.5

 4Rs 54.9 56.0 53.6 59.1 62.3* 55.4 61.2 62.8 59.4 62.4 63.5 61.3

 SS 62.3 65.7* 58.0 65.6 68.4 62.2 71.0 72.4 69.2 68.0 70.4 65.4

Primary Caregiver Report

 ABC 56.6 60.0 53.9 48.1 52.5 44.8 45.8 45.8 45.9 44.3 47.3 42.3

 CSP 59.6 56.3* 62.9 53.9 50.9 56.9 52.1 50.2 53.9 46.7 43.4 49.7

 LBW 62.6 64.8 59.8 53.1 54.0 52.0 51.2 52.9 48.9 46.9 45.7 48.8

 PA 72.3 72.7 71.8 58.9 58.9 58.8 52.2 50.6 54.3 50.5 51.7 49.4

 PATHS 53.6 56.0 51.3 48.3 52.9 44.0 45.5 49.6** 41.7 43.4 45.2 41.8

 4Rs 41.0 42.2 39.6 41.7 41.9 41.5 41.0 40.9 41.2 33.1 32.0 34.4

 SS 62.3 65.7* 57.8 57.3 58.0 56.5 61.8 63.2 60.1 56.7 58.3 54.8

Teacher Report on Student

 ABC 59.7 65.3** 55.5 63.7 66.8 61.4 64.7 65.4 64.1 67.2 73.7* 62.7

 CSP 66.4 64.1 68.7 68.7 67.8 69.6 68.3 67.3 69.3 62.8 55.0*** 70.1

 LBW 64.3 66.3 61.9 66.1 67.8 63.8 64.4 65.5 63.0 63.7 63.8 63.4

 PA 74.6 72.9 76.3 74.8 73.3 76.3 72.4 71.1 74.0 78.3 79.5 77.1

 PATHS 56.4 58.6 54.3 58.4 62.0* 55.0 57.6 60.9 54.6 56.3 60.3* 52.5

 4Rs 53.8 55.5 51.8 59.6 62.4* 56.3 65.5 67.3 63.3 64.1 64.4 63.9

 SS 64.8 68.5** 60.2 65.6 68.4 62.2 70.8 72.0 69.2 68.1 70.4 65.6

See notes at end of table.

53

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.11. Percentage of sample with data by report and program1—Continued

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

Teacher Report on Classroom and 

 School (3rd- to 5th-grade teachers)

 ABC 85.6 91.8 80.8 84.2 83.6 84.6 86.0 84.7 87.0 81.6 86.7 77.8

 CSP 84.1 83.1 85.2 78.5 77.9 79.0 88.0 85.1 90.9 90.6 86.3 95.4

 LBW 88.4 84.0 93.8 95.9 95.2 96.9 87.9 89.7 85.2 91.9 90.4 94.3

 PA 89.5 90.0 88.9 82.6 82.0 83.3 86.3 86.0 86.7 83.7 78.0 90.5

 PATHS 78.7 88.9** 69.9 80.1 92.1** 69.9 83.6 90.0 78.4 83.5 81.7 84.9

 4Rs 92.7 92.0 93.4 90.6 89.0 92.4 97.2 96.8 97.6 96.4 97.8 94.9

 SS 92.2 90.3 94.4 93.1 93.5 92.6 96.6 96.8 96.2 89.0 90.8 86.8

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

*** Treatment group significantly different from control group at the .001 level.

1 Calculated as consent rate x completion rate.

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

SOURCE: The Social and Character Development (SACD) Research Program.

54

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

55

Initial Characteristics 

This section examines the initial characteristics of the students, primary caregivers, teachers, and schools 

participating in the SACD evaluation. These characteristics were collected from students who were enrolled 

in the third grade at the study schools in fall 2004, as well as from their primary caregivers and third-grade 

teachers. In addition, third-, fourth-, and fifth-grade teachers and principals in the study schools provided 

information about activities related to social and character development in these schools. Documenting the 

characteristics of students, primary caregivers and the students’ households, teachers, and schools—and initial 

measures of key outcomes at a point before the programs had been operating for an extended period—helps 

to determine whether the random assignment of schools to treatment and control status produced treatment 

and control groups with similar distributions of observed characteristics. 

For the treatment schools that were implementing six of the seven programs (36 of the 42 treatment schools), 

implementation began 2 to 6 weeks before the fall 2004 data were collected (as shown in table 1.8). The 

implementation of the six programs, and the training of teachers that preceded the implementation of all 

seven programs, could have affected the student, primary caregiver, and teacher reports from the treatment 

schools. If such effects occurred, they could not be disentangled from pre-existing differences between the 

treatment and control schools. The likelihood of an implementation effect differed by the type of 

characteristic reported on. Some characteristics could not be affected (e.g., gender and race/ethnicity). Some 

were unlikely to be affected (e.g., primary caregiver’s highest education level) and others could have been 

affected (e.g., student outcomes and risk factors) but the likelihood was low given how short the time the 

students were exposed to the programs before the data collection. Other characteristics were likely to have 

been affected (e.g., teacher reports of SACD activities and professional development over the past year) given 

that teachers prepared for implementation and implemented the programs before data collection. How likely 

a characteristic was to have been affected should be kept in mind when considering how similar the treatment 

and control groups appeared from the fall 2004 data collection. For example, the lack of significant 

differences in family characteristics discussed below appears to provide valid evidence that the treatment and 

control groups were similar, while the lack of differences in student outcomes and risk factors gives similar 

but perhaps less valid evidence. On the other hand, the findings that treatment teachers used more SACD 

activities in the classroom and received more SACD training provide evidence of a difference between the 

treatment and control schools before the study began, but the training in and implementation of the 

programs before data collection lead to less confidence in this finding. 

Characteristics of Children, Their Families, and Communities 

The sample’s treatment and control groups were similar along the observed student, primary caregiver, and 

community characteristics (table 1.12). For the comparison of 21 characteristics, the expectation would be 

that 1 would be significant by chance. For all seven programs together, there was only 1 statistically significant 

difference (treatment group students were more likely to come from smaller households than control group 

students: 4.5 versus 4.7 people per household). At the program level, there was only 1 statistically significant 

difference (control group primary caregivers had significantly higher levels of education than treatment group 

primary caregivers for the SS program) versus the 7 that would be expected by chance.21 For students, the 

mean age was 8 years. The sample contained more girls than boys (53% versus 48%, respectively) but the 

difference was not statistically significant. This pattern prevailed in five of the seven programs. The sample 

was ethnically diverse. White non-Hispanic students made up 42 percent of the sample, Black non-Hispanic 

students made up 31 percent, and Hispanic students made up 19 percent. In four programs, a single ethnic 

group comprised the majority of all students (White non-Hispanics made up 83% of SS students, 65% of 

LBW students, and 56% of PATHS students, while 51% of PA students were Black non-Hispanic). 

 

21 Program-level results are detailed in the program-specific chapters. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

56

The sample was also diverse in levels of family income, education levels of primary caregivers, and family 

situation. Thirty-nine percent of children lived in a household in which the income was 135 percent of the 

federal poverty level or lower, which is the income threshold for eligibility for free school meals. About 15 

percent of primary caregivers had not completed high school. Sixty percent of the children lived with both 

their father and their mother. Programs varied in their economic diversity. Three of the programs operated in 

schools in which the majority of children were in families that had incomes below 135 percent of the federal 

poverty level (66% for PA, 58% for 4Rs, and 50% for ABC). These three programs also had between 47 

percent and 50 percent of children living with both parents; 27 percent to 28 percent of primary caregivers 

lacked a high school diploma in 4Rs and PA. On the other hand, for SS, only 5 percent of children were from 

families with incomes below 135 percent of the federal poverty level, only 4 percent of primary caregivers 

reported not having a high school diploma, and 80 percent of children lived with both parents. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

57

Table 1.12. Initial characteristics of children, their families, and communities 

Characteristic Total Treatment Control

 Student sample size 3,774 1,980 1,794

Student demographics

 Gender (percent)

 Male 47.5 46.9 48.1

 Female 52.5 53.1 51.9

 Race/ethnicity (percent)

 White (non-Hispanic) 42.1 39.7 44.5

 Black (non-Hispanic) 31.0 34.2 27.9

 Hispanic 19.2 18.5 20.0

 Other 7.7 7.7 7.6

 Age (in years) (mean) 8.1 8.1 8.1

Primary caregiver and family characteristics

 Primary caregiver’s age (in years) (mean) 36.0 36.0 36.0

 Primary caregiver’s race/ethnicity (percent)

 White (non-Hispanic) 46.1 43.8 48.4

 Black (non-Hispanic) 30.7 33.8 27.6

 Hispanic 17.6 16.9 18.3

 Other 5.6 5.5 5.6

 Primary caregiver’s education (percent)

 Did not complete high school 14.7 13.5 15.9

 Completed high school or equivalent 24.3 23.3 25.4

 Some college 39.7 41.3 38.1

 Bachelor’s or higher degree 21.2 21.9 20.5

 Primary caregiver’s employment (percent)

 Full-time 48.3 48.6 48.0

 Part-time 14.7 14.5 14.9

 Student and employed 4.6 5.1 4.1

 Not employed 31.3 30.8 31.8

 Other 1.1 1.0 1.2

 Primary caregiver’s marital status (percent)

 Single 23.9 24.5 23.2

 Married 57.0 56.4 57.7

 Separated or divorced 12.7 12.6 12.9

 Widowed 1.2 1.2 1.1

 Other15.2 5.2 5.2

 Students who live in one household (percent) 93.6 93.6 93.6

 Number of individuals in household (mean) 4.6 4.5* 4.7

See notes at end of table.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

58

Table 1.12. Initial characteristics of children, their families, and communities—Continued

Characteristic Total Treatment Control

 Primary caregiver’s relationship to child (percent)

 Mother (stepmother) 86.1 86.3 85.9

 Father (stepfather) 9.0 8.1 9.8

 Grandparent 2.9 3.2 2.6

 Other relative 1.3 1.4 1.1

 Nonrelative 0.8 1.0 0.6

 Student lives with (percent)

 Mother (stepmother) and father (stepfather) 60.0 58.5 61.6

 Mother (stepmother) only; father (stepfather) not present 33.6 34.9 32.3

 Father (stepfather) only; mother (stepmother) not present 2.5 2.3 2.8

 Other relative/nonrelative, parents not present 3.9 4.4 3.3

 Highest education of anyone in household (percent) 

 Did not complete high school 10.0 8.7 11.4

 Completed high school or equivalent 21.9 20.8 23.0

 Some college 40.3 42.4 38.2

 Bachelor’s or higher degree 27.8 28.1 27.5

 Total household income (percent)

 Less than $20,000 33.2 31.7 34.7

 $20,000 to $39,999 24.4 25.3 23.4

 $40,000 to $59,999 15.1 15.9 14.4

 $60,000 or more 27.3 27.1 27.5

 Household income-to-poverty threshold ratio (percent)

 Below 135 percent 39.0 37.0 41.0

 135 to 185 percent 18.2 18.9 17.5

 Above 185 percent 42.8 44.1 41.5

 Alabama Parenting Questionnaire—Poor Monitoring and 

 Supervision Subscale (mean) 1.2 1.1 1.2

 Alabama Parenting Questionnaire—Positive Parenting

 Subscale (mean) 3.5 3.5 3.5

 Confusion, Hubbub, and Order Scale (mean) 2.2 2.2 2.2

Community characteristics (mean)

 Community Risks Scale 1.5 1.5 1.5

 Community Resources Scale 2.7 2.7 2.7

 Child-Centered Social Control Scale 3.1 3.1 3.1

* Treatment group significantly different from control group at the .05 level.

1 Categories combined to protect confidentiality.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included

program indicators to account for the sample design and adjusted for clustering at the school level.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

59

The mean values of the outcome measures for children’s behavior and attitudes as reported by the primary 

caregiver, child, and teacher in fall 2004, along with each measure’s range, are shown in table 1.13. The values 

were similar for the treatment and control groups. There were three statistically significant differences at the 

program level (these data are not shown in a table) but their number and pattern suggest that they may have 

been due to chance; at the 5 percent significance level, 6 of the 126 comparisons made would have been 

expected to show statistical significance by chance. Table 1.13 also addresses the issue of whether students 

had room to improve from their initial scores on the outcome measures. Five of the mean outcomes were 

within 0.4 units of the minimum or maximum outcome range (Normative Beliefs About Aggression; 

Altruistic Behavior, Teacher Report on Student; Problem Behavior, Child Report and Teacher Report on 

Student; and Engagement with Learning). However, these five were more than half a standard deviation from 

their minimum or maximum value, which is larger than the minimal detectable impacts for the study. In 

addition, the other mean outcomes were one or more standard deviations from their boundaries. In sum, the 

evaluation had enough power to identify a minimum detectable effect for all the outcomes before potential 

ceiling and floor effects could occur. 

The mean values of the five risk factors as reported in fall 2004 showed no statistically significant difference 

between the treatment and control groups (table 1.14). In addition, there were no statistically significant 

differences at the program level (these data are not shown in a table). 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

60

Table 1.13. Mean scores and standard deviations for initial outcome measures 

Total Treatment Control

Outcome measure–Report Range Mean SD Mean SD Mean SD

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 1-4 2.9 0.6 2.9 0.6 2.9 0.6

 Normative Beliefs About Aggression–CR 1-4 1.2 0.4 1.2 0.4 1.2 0.4

 Empathy–CR 1-3 2.4 0.4 2.4 0.4 2.4 0.4

Behavior Domain

 Altruistic Behavior–CR 0-3 1.5 0.8 1.5 0.9 1.4 0.8

 Altruistic Behavior–TRS 1-4 1.4 0.5 1.4 0.5 1.4 0.4

 Altruistic Behavior–PCR 1-4 2.3 0.7 2.3 0.7 2.3 0.7

 Positive Social Behavior–TRS 1-4 3.0 0.7 3.0 0.7 3.0 0.7

 Positive Social Behavior–PCR 1-4 3.0 0.5 3.0 0.5 3.0 0.5

 Problem Behavior–CR 0-3 0.2 0.4 0.2 0.4 0.2 0.4

 Problem Behavior–TRS 1-4 1.4 0.4 1.4 0.4 1.3 0.4

 Problem Behavior–PCR 1-4 1.6 0.3 1.6 0.3 1.6 0.3

 ADHD-Related Behavior–TRS 1-4 1.7 0.6 1.7 0.7 1.7 0.6

Academics Domain

 Academic Competence and Motivation–TRS 1-5 2.9 0.9 2.9 0.9 2.9 0.9

 Engagement with Learning–CR 1-4 3.7 0.6 3.7 0.6 3.7 0.6

Perceptions of School Climate Domain

 Positive School Orientation–CR 1-4 3.2 0.7 3.2 0.7 3.2 0.6

 Negative School Orientation–CR 1-4 1.8 0.7 1.9 0.7 1.8 0.7

 Student Afraid at School–CR 1-4 2.4 0.9 2.4 0.9 2.4 0.9

 Victimization at School–CR 0-3 0.8 0.8 0.8 0.8 0.8 0.8

 Sample size—PCR1 3,774 1,980 1,794

 Sample size—CR1 3,997 2,092 1,905

 Sample size—TRS

1

 4,104 2,158 1,946

1 Sample size may differ for some outcomes due to nonresponse.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 SD: Standard deviation

 ADHD: Attention deficit hyperactivity disorder

No findings were found statistically significant at or below the .05 level. Weights, which assign equal weight to each school within 

each of the programs and to each program across programs, were used in producing the treatment, control, overall means, and 

standard deviations. Statistical tests were conducted using regressions that included program indicators to account for the sample 

design and adjusted for clustering at the school level.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

61

Table 1.14. Mean scores and standard deviations for initial risk factors 

Total Treatment Control

Risk factor Mean SD Mean SD Mean SD

Family risk 1.2 0.2 1.1 0.2 1.2 0.2

Community risk 1.5 0.7 1.5 0.7 1.5 0.7

Child behavior risk–TRS 50.9 10.3 51.2 10.4 50.5 10.2

Child behavior risk–PCR 50.5 10.1 50.5 9.8 50.6 10.3

Socioeconomic risk2

 No risk factors 58.8 49.2 60.9 48.8 56.8 49.5

 One risk factor 29.8 45.7 29.0 45.4 30.6 46.1

 Two or three risk factors 11.4 31.7 10.1 30.1 12.6 33.2

 Sample size—PCR1 3,774 1,980 1,794

 Sample size—CR1 3,997 2,092 1,905

 Sample size—TRS

1

 4,103 2,157 1,946

1

Sample size may differ for some outcomes due to nonresponse.

2

As a sensitivity test, the index was also treated as a continuous variable and no significant differences were found.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 SD: Standard deviation

No findings were found statistically significant at or below the .05 level. Weights, which assign equal weight to each school within 

each of the programs and to each program across programs, were used in producing the treatment, control, overall means, and

standard deviations. Statistical tests were conducted using regressions that included program indicators to account for the sample 

design and adjusted for clustering at the school level.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

62

Characteristics of Teachers and Schools 

The third-, fourth-, and fifth-grade teachers at the study schools were predominantly White non-Hispanic 

(75%), female (88%), and had an average of nearly 13 years of total teaching experience. About 60 percent 

held an advanced or specialist degree (table 1.15). Six characteristics were compared, with less than one 

expected to be significant by chance. The only statistically significant difference between the treatment and 

control group of teachers concerned degree attainment: a test of the distribution of teachers’ education levels 

across five categories yielded a p-value of 0.043. Among the programs there was one significant difference 

versus six expected by chance: for PATHS, male teachers made up a smaller percentage of the treatment 

teachers than the control teachers (17% versus 35%). 

Data regarding school characteristics were drawn from the U.S. Department of Education’s Common Core 

of Data (2003-04) in order to compare treatment versus control schools (table 1.16). Comparisons were made 

on nine characteristics, with less than one expected to be found significant by chance. These characteristics 

included student composition (race/ethnicity and school lunch eligibility), number of students, number of 

full-time teachers, Title I status, highest and lowest grades, school location, and years the principal had been 

at the school. Overall, one significant difference was found: a smaller percentage of treatment schools than 

control schools (50% versus 67%) had prekindergarten. No significant differences were found among the 

programs. 

In the Teacher Report on Classroom and School, teachers reported on nine dimensions concerning their 

school environment, including feelings of safety, resource adequacy, student support, staff freedom to teach 

as desired, affiliation with and ties to colleagues, innovation regarding trying new approaches to teaching, 

professional interest, participatory decisionmaking, and work pressure. There were no statistically significant 

differences (these data are not shown in a table) in these reports between treatment and control schools, 

overall or at the program level. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

63

Table 1.15. Initial characteristics of teachers 

Characteristic Total Treatment Control

 Teacher sample size 847 436 411

Gender (percent)

 Male 12.3 11.9 12.7

 Female 87.7 88.1 87.3

Race/ethnicity (percent)

 White (non-Hispanic) 75.2 73.1 77.4

 Black (non-Hispanic) 16.8 18.6 15.1

 Hispanic 4.9 5.9 3.8

 Other 3.1 2.4 3.7

Number of years teaching (mean) 12.8 12.5 13.0

Number of years teaching in this school (mean) 7.3 6.8 7.7

Type of teaching certificate (percent)

 Regular state certificate or advanced professional certificate 84.7 83.4 86.0

 Provisional certificate (for those in alternative certification programs) 6.2 6.8 5.5

 Probationary certificate (for those who have satisfied all 

 requirements except for completing the probationary period) 4.9 5.7 4.2

 Emergency certificate or waiver (for those who must 

 complete a certification program to continue teaching) 0.0 0.0 0.0

 Other14.1 4.1 4.2

Education (percent) *

2

 Less than a bachelor’s degree 0.0 0.0 0.0

 Bachelor’s degree 40.2 37.6 42.8

 Master’s degree, Ph.D. 56.2 60.2 52.3

 Specialist degree 2.1 1.0 3.2

 Other 1.5 1.3 1.7

* Treatment group significantly different from control group at the .05 level.

1 Categories, including no certificate and temporary certificate, combined to protect confidentiality. 

2 A test of the distribution of teachers’ education levels across all five categories yielded a p-value of 0.043.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

64

Table 1.16. Initial characteristics of schools 

Characteristic Total Treatment Control

 School sample size 84 42 42

Student race/ethnicity (percent)

 White (non-Hispanic) 37.3 36.4 38.3

 Black (non-Hispanic) 39.7 40.9 38.5

 Hispanic 18.7 18.0 19.5

 Other 4.2 4.6 3.8

Students eligible for free or reduced-price lunch (percent) 61.3 62.3 60.4

 

Number of students enrolled (mean) 567.1 555.7 578.9

Number of full-time teachers (mean) 38.5 37.3 39.6

Title I status (percent)

 Title I eligible school 74.2 73.2 75.2

 Schoolwide Title I 63.7 61.2 66.2

Lowest grade offered (percent)

 Prekindergarten 58.5 50.4* 66.7

 Kindergarten 41.5 49.6 33.3

Highest grade offered (percent)

 Fifth grade 55.4 58.1 52.8

 Sixth grade 22.0 19.0 25.1

 Eighth grade 22.5 22.9 22.1

Location of school (percent)

 City 56.2 58.6 53.8

 Suburbs 26.7 25.2 28.1

 Rural 17.1 16.2 18.1

Number of years principal has been at this school (mean) 5.5 4.8 6.1

* Treatment group significantly different from control group at the .05 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included

program indicators to account for the sample design and adjusted for clustering at the school level. Data are missing from Love In a 

Big World for student race/ethnicity, percentage of students eligible for free or reduced-price lunch, and number of full-time teachers.

SOURCE: NCES Common Core of Data (2003-04), The Social and Character Development (SACD) Research Program.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

65

The Initial Level of Social and Character Development Activities in the Schools 

During the initial data collection in fall 2004, principals and teachers reported on the SACD activities used in 

the schools and classrooms, the availability of SACD materials and instructional practices used, and the 

professional development teachers received on SACD over the past 12 months. These reports were likely to 

have been influenced by the implementation of six programs (in 36 of the 42 treatment schools) and the 

training of treatment teachers in all schools that preceded the data collection. For this reason, the fall 2004 

data did not provide baseline data on how the use of SACD activities differed between treatment and control 

before the intervention but, instead, provided data on how this use differed early on in the study. 

Table 1.17 shows that most school principals reported activities to promote six social and character 

development goals: violence prevention and peace promotion (93%), social and emotional development 

(95%), character education (96%), tolerance and diversity (86%), risk prevention and health promotion (81%), 

and civic responsibility and community service (90%). In addition, most (91%) also reported activities 

directed toward behavior management. There were no statistically significant differences in the percentages of 

principals who reported, in the treatment group schools and the control group schools, that their school had 

programs or activities to promote each of these goals. 

Most teachers reported the use of these activities in their classroom to promote the six social and character 

development goals and behavior management (ranging from 53% to 87%). From the eight comparisons made 

(including a ―none of the above‖ comparison) less than one significant difference would have been expected. 

Unlike the principals, a significantly larger percentage of treatment teachers than control teachers reported 

using activities to promote three SACD goals: violence prevention and peace promotion activities (70% 

versus 59%), social and emotional development activities (79% versus 68%), and behavior management (90% 

versus 84%). In addition, seven comparisons were made as to whether teachers used the activities for each of 

the six SACD goals and behavior management with all, some, or no students (less than one significant 

difference would be expected by chance). Treatment teachers were statistically significantly more likely to 

report engaging all students, as opposed to some or no students, with activities to promote the same three 

SACD goals (these data are not shown in a table). 

The majority of teachers reported that their schools used schoolwide activities to promote social and 

character development. These activities included morning announcements or videos (reported by 80% of 

teachers), school assemblies (71%), school newspapers or bulletins (67%), special school days (54%), and 

special events (72%). Only 13 percent reported the use of other activities. There were no significant 

differences in the percent of treatment versus control teachers reporting the use of these six activities. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

66

Table 1.17. Principal and teacher initial reports on use of SACD programs or activities 

SACD program or activity Total Treatment Control

 Principal sample size 84 42 42

 Teacher sample size 847 436 411

Principals reporting that school had programs or activities

 to promote the following SACD goals (percent)

 Violence prevention and peace promotion 92.8 92.9 92.7

 Social and emotional development 95.0 95.3 94.7

 Character education 96.4 95.6 97.1

 Tolerance and diversity 85.8 84.7 87.0

 Risk prevention and health promotion 80.8 85.1 76.5

 Civic responsibility and community service 89.7 92.3 87.0

 Behavior management 91.4 93.9 89.0

Teachers reporting on using programs or activities in their 

 class to promote the following SACD goals (percent)

 Violence prevention and peace promotion 64.6 69.8* 59.4

 Social and emotional development 73.2 78.5** 67.9

 Character education 82.5 84.4 80.5

 Tolerance and diversity 63.3 64.4 62.1

 Risk prevention and health promotion 52.9 51.7 54.1

 Civic responsibility and community service 59.2 59.6 58.7

 Behavior management 86.8 89.6* 83.9

 None of the above 2.4 1.6 3.2

Teachers reporting schoolwide use of the following 

 activities to promote SACD (percent)

 Morning announcements or videos 80.2 77.2 83.3

 School assemblies 71.0 69.1 72.9

 School newspapers or bulletins 66.7 62.3 71.1

 Special school days 53.8 53.8 53.9

 Special events 71.5 72.5 70.5

 Other activities 12.8 12.5 13.1

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

67

Teachers reported on their use of a broad range of teaching materials (table 1.18). The materials most used 

with SACD activities were teacher guides (used by 65% of teachers), student materials (used by 50%), and 

children’s literature (used by 50%). Of the seven materials reported on, teachers at treatment schools had 

three reports of use that statistically significantly differed from teachers in control schools (versus less than 

one by chance): teacher guides (72% versus 59%), instructional aids such as games or videos (37% versus 

29%), and other types of materials (11% versus 16%). There were no significant differences in the reported 

use of student materials, children’s literature, giveaway materials, or in not using any of these materials.

Teachers reported using a wide variety of instructional strategies (table 1.18). Treatment teachers reported 

significantly higher use on 5 of the 22 items than control teachers (1 significant difference would have been 

expected by chance). These included role-playing (73% versus 59%), direct instruction of social and character 

development (87% versus 76%), incorporating social and character development into the academic 

curriculum (80% versus 69%), targeted story reading or writing on social and character development themes 

(80% versus 71%), and the average number of strategies used (12.2 versus 11.3).
Chapter 1. The Social and Character Development Multiprogram Evaluation 

68

Table 1.18. Teacher initial reports on use of SACD materials and classroom strategies 

SACD material and strategy Total Treatment Control

 Teacher sample size 847 436 411

Teachers using the following materials in conjunction with 

 social and character development activities (percent)

 Teacher guides (manuals, curricula) 65.3 71.6** 59.1

 Student materials (workbooks, worksheets) 50.1 53.1 47.1

 Instructional aids (games, software, videos) 33.2 37.4* 28.9

 Giveaways (bookmarks, stickers) 47.2 45.3 49.1

 Children’s literature 50.0 53.5 46.6

 Other types of materials 13.2 10.6* 15.8

 Do not use any of the materials listed above 10.1 9.1 11.1

Teachers using any of the strategies listed below to promote

 social and character development in the classroom (percent) 99.8 99.6 100.0

Number of strategies (listed below) used by teachers to promote

 social and character development in the classroom (mean) 11.8 12.2** 11.3

Teachers using each of the following strategies to promote 

 social and character development (percent)

 Role-playing 65.7 72.5** 58.9

 Cooperative learning 96.1 97.0 95.3

 Peer group discussions 87.7 90.3 85.1

 Direct instruction of social and character development 81.6 87.4** 75.9

 Skill training 43.1 46.8 39.5

 Incorporating social and character development into 

 academic curriculum 74.7 80.3** 69.2

 Parent training 7.8 6.8 8.9

 Parent/community involvement in program development 

 or delivery 24.0 26.7 21.4

 Mentoring 36.3 35.8 36.7

 Good behavior notes sent home daily or weekly 75.4 76.7 74.1

 Presenting role models 67.2 68.2 66.2

 Targeted story reading or writing on SACD themes 75.1 79.8* 70.5

 Peer mediation 38.8 40.9 36.7

 Honor roll for positive behavior 51.2 50.4 52.1

 Pledges or recitations on social and character 

 development themes 38.1 38.6 37.6

 Guided visualization 44.9 47.2 42.7

 Student-led/student-assisted instruction 48.5 50.9 46.1

 Journaling 72.0 73.6 70.3

 Time out for negative behavior 84.9 86.5 83.3

 Daily or weekly rewards for positive behavior 92.0 91.6 92.3

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

69

Principals and teachers reported on participation in and amount of SACD training and staff development 

provided over the previous 12 months (table 1.19). This period overlapped with the time that treatment 

schools received training in the SACD programs. Principals reported staff participation rates of 92 percent 

and a mean of 8.3 hours of training, with no significant difference between treatment and control principals. 

On the nine training items asked of teachers, treatment teachers’ reports were statistically significantly higher 

for five of them (versus less than one expected by chance). These included staff participation rates in SACD 

training over the past 12 months (86% for treatment teachers versus 63% for control teachers), number of 

training hours received (9.5 versus 6.4), and participation rates in training in specific areas: social and 

emotional development (39% versus 17%), character education (55% versus 27%), and behavior management 

(42% versus 32%). 

Table 1.19. Principal and teacher initial reports on SACD professional development 

SACD professional development Total Treatment Control

 Principal sample size 84 42 42

 Teacher sample size 847 436 411

Principals reporting that staff participated in social and character

 development training within the past year (percent) 92.0 97.5 86.2

 

Teachers reporting participation in social and character 

 development training within the past 12 months (percent) 74.3 86.1** 62.5

Number of hours of social and character development training 

 principals report were provided to each staff person last year (mean) 8.3 9.4 7.1

Number of hours of social and character development training 

 teachers report receiving during the past 12 months (mean) 7.9 9.5** 6.4

Teachers reporting receiving training in the past 12 months

 in the following areas (percent)

 Violence prevention and peace promotion 22.6 23.8 21.5

 Social and emotional development 27.9 38.5** 17.2

 Character education 40.9 54.8** 26.9

 Tolerance and diversity 18.8 20.8 16.9

 Risk prevention and health promotion 11.3 11.5 11.1

 Civic responsibility and community service 6.6 6.4 6.7

 Behavior management 36.8 41.6* 32.0

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

70

The data on the initial level of SACD activity led to two findings. First, treatment teachers reported greater 

use of and training in SACD activities than control teachers more often than would be expected by chance. 

There are two potential causes for this finding, and the analysis cannot be used to determine whether the 

reason for such a difference was that the two groups did differ on their initial use of SACD activities (i.e., that 

randomization did not create similar treatment and control groups) or that the training of all treatment 

teachers and the implementation of six of the programs before the initial data were collected influenced the 

teacher reports. Because it is likely (though unproven) that the training and implementation affected the 

teacher reports, these data were not considered appropriate for use as a baseline measure of SACD activities 

and training in the treatment schools. 

Second, these data indicate that the control condition for the SACD project was not a ―no treatment‖ control 

but a ―standard practice‖ control. Because the control teachers were not affected by the implementation of 

the SACD programs before data collection, their reports reflect standard practice in the control schools. 

Standard practice at the control schools included reports of 54 percent to 84 percent of teachers using SACD 

activities, 89 percent of teachers reporting the use of specific materials in conjunction with these activities, 

100 percent reporting the use of at least one of the specified instructional strategies, and 63 percent reporting 

participation in SACD training over the past 12 months. 

Year-by-Year Impacts on Use of Social and Character Development 

Activities 

Under the first research questions, the SACD programs were expected to increase the use of SACD activities 

in the treatment schools in comparison to the control schools. Two analyses of this impact were done: one 

based on teacher reports and the other on principal reports. 

Analysis of Teacher Reports 

Every spring, third-, fourth-, and fifth-grade teachers provided information through the Teacher Report on 

Classroom and School about the social and character development activities they used in their classroom. As 

described in the Measures section, 83 variables were derived from individual items on the Teacher Report on 

Classroom and School. These were grouped under six domains and used to determine the difference between 

treatment and control teachers: 

1. The use of SACD activities in their classrooms to promote the six social and character development 

goals (separately and all together) and behavior management, and the use of such activities for at least 

1 hour per week for the same purposes; 

2. The use of SACD activities linked to named SACD programs in their classrooms to promote the six 

social and character development goals (separately and all together), and the use of such activities for 

at least 1 hour per week for the same purposes; 

3. The use of materials and strategies to implement the activities in the classroom; 

4. The use of six schoolwide strategies to promote social and character development; 

5. The use of staff development to support the teachers; and 

6. Staff attitudes toward SACD activities and the use of practices conducive to the development of 

social and character development. 

Teacher Report on Classroom and School consent and completion rates (presented in table 1.9) led to 87 

percent to 90 percent of all teachers having data for the 3 years (varying by 1% to 2% for treatment versus 

control teachers). At the program level, the data available ranged from 83 percent to 100 percent of all 

teachers for the 3 years. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

71

To estimate the impacts of the SACD programs for each of the outcome measures, testing the statistical 

significance of the differences in means between the treatment and the control teachers was used.22 Before 

differences in the means were tested, the data were weighted such that each school received equal weight 

within a program and each program received equal weight in the combined-program estimates. Standard 

errors of the impact estimates accounted both for unequal weighting and for the clustering of teachers within 

schools. 

In addition to estimating the impacts of the SACD programs on the individual outcome measures, the 

impacts on the six domains were also examined. Testing the impact on the domains was done to adjust for 

the multiple comparisons made within each domain in order to address the increased chances of finding a 

spurious outcome when more than one test was done. As a result, this section provides two sets of results: 

(1) the impacts on the individual outcomes unadjusted for multiple comparisons, and (2) the impacts on the 

domains that serve as the multiple comparison adjustment. 23

 

The testing of the significance of the impacts on the domains was based on a set of three heuristics modeled 

on the approach used by the U.S. Department of Education’s What Works Clearinghouse for determining 

whether domain-level effects are statistically significant when there are multiple outcome measures within a 

single domain (What Works Clearinghouse n.d.b). Each domain was checked using the three heuristics, and a 

statistically significant positive impact was found if any of the three were met.24 The three heuristics were as 

follows: 

1. Based on the results from the statistical test of each outcome variable within a domain (unadjusted 

for multiple comparisons), at least half of the impacts were positive and statistically significant and no 

impact was negative and statistically significant. 

2. The omnibus impact for all the outcomes measured together was positive and statistically significant 

on the basis of a multivariate statistical test. 25 

 

22 Three factors contributed to the decision to use differences in means. First, because of random assignment, simple 

treatment-control contrasts provided unbiased estimates of program impacts. Second, only initial values (rather than true 

baseline values) for these outcomes were available to use in a model because training (at all treatment schools) and 

program implementation (at 36 treatment schools) began before data collection. The decision to use initial values in an 

analysis partly depends on whether the initial training and implementation occurring before data collection would be 

expected to have immediate and large impacts on the outcomes (Schochet 2008b). For this analysis, the outcomes are 

based on teacher actions and so would likely be upwardly influenced by the teacher training and short period of teacher 

implementation before pretesting (in contrast to student outcomes, which would be less likely to be so influenced). For 

this reason a model-based analysis using the initial values as covariates was not chosen. Third, preliminary analyses 

indicated no gain in precision from the inclusion of other covariates. 

23 When the SACD evaluation was designed, there was no general agreement in the field about how best to adjust for 

multiple comparisons, and this approach was seen by the SACD Research Consortium as the best method available. For 

a more recent discussion of the issue, see Schochet 2008a. 

24 For ease of discussion, this section describes detecting positive significant impacts. These heuristics were also used to 

detect a significant negative impact on a domain. 

25 A multivariate analysis of variance (MANOVA) procedure (Harris 1975) was used. Suppose there are p domain 

outcomes (the dependent variables), n sample members, and for each dependent variable there is one treatment effect 

parameter and k parameters for other covariates. The MANOVA model can then be expressed as follows: 

Y = Ta + Xb + e

where Y is a nxp matrix of domain outcomes, T is an n x 1 vector of treatment indicators, a is a 1 x p parameter vector of 

treatment effects, X is an n x k matrix of covariates (a limited set of covariates that included indicator variables for the 

programs and the intervention) with associated parameter matrix b, and e is a n x p matrix of error terms that accounts 

for school-level clustering. The omnibus F-test uses the MANOVA model to test the joint null hypothesis of zero 

treatment effects for each domain outcome (that is, H0: a = 0). Wilks’ lambda was used to assess statistical significance.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

72

3. After applying the Benjamini-Hochberg (Benjamini and Hochberg 1995) procedure to each outcome 

under a domain, at least one of the outcomes remained positive and statistically significant and no 

outcome was negative and statistically significant. The Benjamini-Hochberg procedure adjusts 

significance levels downward to account for the multiple testing of impacts. 

A statistically significant impact on a domain was recognized if one or more of the three heuristics was met. 

There could be situations in which one or more of the heuristics indicated a statistically significant impact of 

the intervention on the domain, when the preponderance (or all) of the individual outcomes did not indicate 

statistically significant impacts. For example, this type of situation might have arisen when a test captured a 

latent construct underlying the individual outcomes within the domain; the latent construct might have had 

less statistical noise than did the individual outcomes and thus been more likely to have indicated statistical 

significance. There also could be situations in which one or more of the individual outcomes showed 

statistically significant impacts but none of the heuristics showed a significant impact on the domain. 

Analysis of Principal Interviews 

Every spring principals were individually interviewed about the prevalence and characteristics of the activities 

used to promote social and character development along with the percentage of staff trained in them and 

staff support of their use. As discussed in the Measures section, information from the interviews could be 

used to determine the difference in reports between treatment and control principals regarding (1) the 

percentages of schools using activities to address each of the six SACD goals, (2) the delivery methods for 

these SACD activities, (3) targeting of the activities, (4) teacher professional development, and (5) staff 

support for the promotion of social and character development. 

The interviews had an almost perfect response rate; all principals completed them in Years 1 and 2 and all but 

one control school principal completed them in Year 3. Because of inconsistencies in the quality of 

interviewers’ recording of answers to the open-ended questions during the spring 2005 interviews, the results 

from the spring 2005 open-ended data are not included. 

The interview data on activity level for the six SACD goals were transformed in two ways: (1) an indicator of 

whether the school had any activities that addressed each SACD goal was created, and (2) the number of 

activities in a school that addressed each SACD goal was calculated. Logistic regression was used to 

determine whether there were differences between treatment and control schools in the likelihood of each 

implementation measure that was measured through a dichotomous school-level variable. Ordinary least 

squares regression was used to determine whether there were differences between the treatment and control 

schools in the number of activities to promote SACD goals within schools. The research team-specific 

characteristics were controlled for in the model by including indicator variables for each team. An indicator 

variable of school-level treatment status was included in the model. No other covariates were used in the 

models because of the small sample size. The What Works Clearinghouse heuristics were not used in this 

analysis, and the analysis was not done for each individual SACD program because of the small sample sizes. 

Results From Teacher Reports 

The SACD programs were expected to increase the use of SACD activities in the classroom and school, 

broaden the types of materials and instructional strategies used to implement these activities in the classroom 

and schoolwide, increase the amount of staff development for such activities, and generate positive attitudes 

and schoolwide practices. Eighty-three outcomes were measured each year for a total of 249 comparisons 

between treatment and control teachers tested over the 3 years (with 12 to 13 expected to be found 

statistically significant by chance). The analysis found 127 statistically significant differences in these 249 

comparisons, with all showing greater reported use of SACD activities by treatment teachers. Here, the 

impacts on the outcomes (organized by domain) are described and followed by a discussion of the analysis of 

the impacts on the domains. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

73

Use of Activities and Activities Linked to Named SACD Programs 

Two patterns of findings emerged from a comparison of treatment-control group differences in teacherreported activities to promote social and character development goals in the classroom and the use of 

behavior management. First, treatment teachers were more likely than control teachers to report 

implementing these activities in their classrooms and more likely to report conducting these activities for at 

least 1 hour per week in general and when reporting on activities linked to named SACD programs. Second, 

control group teachers also reported conducting activities to promote social and character development. 

Table 1.20 shows the percentages of teachers who reported activities to promote each of six social and 

character development goals, the percentages who reported activities to promote any one of the six goals, and 

the percentages who reported that they used behavior management techniques. Impact estimates (which 

represent the difference in percentile points between treatment and control teachers who reported positively) 

are presented for each school year. The table is divided into four panels: (1) engagement in any activity, 

(2) engagement in any activity for at least an hour per week on average, (3) engagement in any activity as part 

of a named program, and (4) engagement in any activity as part of a named program for at least an hour a 

week. 

Measuring teacher activity in these diverse ways contributes to an understanding of all activities that a teacher 

considered as promoting one of the six social and character development goals, distinguishes between 

activities that were conducted intensively (defined as at least 1 hour per week)26 from those that might have 

been done only occasionally, and distinguishes between activities that were conducted as part of a named 

program and activities that might have been less structured and had less institutional support. 

The first pattern of a larger percentage of treatment teachers than control teachers reporting engagement in 

SACD activities can be seen throughout table 1.20. In regard to teachers’ reported use of SACD activities 

(panels 1 and 2), 48 comparisons were made over the 3 years, with 3 expected to be significant by chance. A 

significantly greater percentage of treatment teachers reported using SACD activities than control teachers in 

31 of these 48 comparisons. In regard to teachers’ reported use of SACD activities linked to named SACD 

programs (panels 3 and 4), 42 comparisons were made over the 3 years with 2 expected to be significant by 

chance. A significantly greater percentage of treatment teachers reported using the SACD activities in 39 of 

these 42 comparisons. In all 3 years, the percentages of treatment teachers reporting engagement in SACD 

activities were statistically significantly greater than for control teachers in regard to promoting any of the 

SACD goals and promoting four of the six individual SACD goals (violence prevention and peace promotion, 

social and emotional development, character education, and tolerance and diversity). For example, when 

looking at activities to support any SACD goal, the statistically significant difference favoring treatment 

teachers ranged from 5 to 10 percentage points for engagement in any activities (panel 1, Any SACD goal), 7 

to 24 percentage points for engagement in any activity for at least an hour per week (panel 2, Any SACD 

goal), 29 to 34 percentage points for engagement in any activity linked to a named SACD program (panel 3, 

Any SACD goal), and 26 to 34 percentage points for engagement in any activity linked to a named SACD 

program for at least an hour per week (panel 4, Any SACD goal). 

The second pattern of control teachers reporting engagement in SACD activities is most obvious in panel 1 

(Any SACD goal), which shows that 86 percent to 90 percent of control teachers reported engaging in SACD 

activities to promote any SACD goal (and more than 50 percent reported engaging in SACD activities to 

promote each of the six individual SACD goals). More than half of the control teachers reported engaging in 

such activities for at least an hour a week (52% to 76%: panel 2, Any SACD goal) and 36 percent to 43 

percent reported engaging in any activity linked to a named SACD program to promote any SACD goal 

(panel 3, Any SACD goal). 

 

26 Intensity was defined as being for at least 1 hour per week because this was about the amount of time that the SACD 

programs being evaluated considered desirable as part of their models. 
Table 1.20. Impacts on teacher-reported SACD classroom activities 

Panel 1: Engagement in any activities to promote SACD goals

Year 1

(Spring 3rd grade)

Year 2

(Spring 4th grade)

Year 3

(Spring 5th grade)

SACD activity

Treatment Control Impact p-valueTreatment Control Impact p-valueTreatment Control Impact p-value

 Teacher sample size 441 409 425 404 425 396

Violence prevention and peace promotion (percent) 76.7* 64.7 12.0 0.005 76.1* 64.3 11.8 0.002 76.9* 63.0 13.9 0.001

Social and emotional development (percent) 81.3* 64.2 17.1 0.000 84.4* 60.4 24.0 0.000 81.1* 62.5 18.6 0.000

Character education (percent) 92.3* 79.3 13.0 0.000 92.3* 75.7 16.6 0.000 86.8* 72.6 14.2 0.000

Tolerance and diversity (percent) 75.7* 62.1 13.6 0.001 75.7* 59.8 15.9 0.000 73.8* 57.4 16.4 0.000

Risk prevention and health promotion (percent) 60.9 62.2 -1.3 0.795 63.5 61.9 1.6 0.744 63.7 62.3 1.4 0.778

Civic responsibility and community service (percent) 63.4 61.7 1.7 0.687 60.2 58.4 1.8 0.669 60.3 59.1 1.2 0.795

Any SACD goal (percent) 95.3* 88.2 7.1 0.006 95.4* 90.3 5.1 0.007 95.5* 86.0 9.5 0.000

Behavior management (percent) 88.5 85.6 2.9 0.285 92.3* 82.1 10.2 0.000 85.0 82.4 2.6 0.389

Panel 2: Engagement in any activities to promote SACD goals for at least 1 hour per week

Year 1

(Spring 3rd grade)

Year 2

(Spring 4th grade)

Year 3

(Spring 5th grade)

SACD activity

Treatment Control Impact p-valueTreatment Control Impact p-valueTreatment Control Impact p-value

 Teacher sample size 441 409 425 404 425 396

Violence prevention and peace promotion (percent) 39.4* 21.9 17.5 0.000 41.9* 24.9 17.0 0.000 38.0* 24.2 13.8 0.002

Social and emotional development (percent) 46.9* 23.4 23.5 0.000 49.5* 24.7 24.8 0.000 42.6* 25.9 16.7 0.000

Character education (percent) 60.9* 31.3 29.5 0.000 60.4* 30.6 29.8 0.000 50.2* 30.5 19.7 0.000

Tolerance and diversity (percent) 36.7* 21.3 15.4 0.000 36.0* 23.1 12.9 0.001 32.8* 18.4 14.4 0.000

Risk prevention and health promotion (percent) 25.9 20.5 5.4 0.138 28.0 24.1 3.9 0.293 24.5 22.8 1.8 0.657

Civic responsibility and community service (percent) 18.8 14.7 4.0 0.179 19.7 16.1 3.6 0.252 20.6^ 14.6 6.0 0.087

Any SACD goal (percent) 75.2* 51.7 23.5 0.000 83.7* 72.6 11.1 0.003 83.0* 76.0 7.0 0.037

Behavior management (percent) 68.7 66.2 2.5 0.666 75.0* 58.9 16.1 0.001 68.6 64.9 3.7 0.377

See notes at end of table.

74

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.20. Impacts on teacher-reported SACD classroom activities—Continued 

Panel 3: Engagement in activities to promote SACD goals linked to named SACD programs

Year 1

(Spring 3rd grade)

Year 2

(Spring 4th grade)

Year 3

(Spring 5th grade)

SACD activity

Treatment Control Impact p-valueTreatment Control Impact p-valueTreatment Control Impact p-value

 Teacher sample size 441 409 425 404 425 396

Violence prevention and peace promotion (percent) 42.0* 17.0 25.0 0.000 48.1* 17.3 30.8 0.000 48.0* 17.4 30.6 0.000

Social and emotional development (percent) 48.9* 8.3 40.7 0.000 54.8* 11.2 43.6 0.000 48.8* 13.0 35.7 0.000

Character education (percent) 54.4* 15.5 38.8 0.000 58.8* 13.4 45.4 0.000 49.4* 13.3 36.1 0.000

Tolerance and diversity (percent) 34.3* 6.2 28.2 0.000 37.6* 7.2 30.4 0.000 33.4* 5.8 27.7 0.000

Risk prevention and health promotion (percent) 27.7 21.7 6.0 0.151 31.3* 21.7 9.6 0.013 28.5* 20.1 8.4 0.048

Civic responsibility and community service (percent) ‡* ‡ 4.4 0.009 13.3* 4.7 8.6 0.000 13.5* 7.2 6.4 0.021

Any SACD goal (percent) 70.0* 36.3 33.6 0.000 72.4* 42.6 29.8 0.000 68.4* 39.7 28.7 0.000

Panel 4: Engagement in activities to promote SACD goals linked to named SACD programs for at least 1 hour per week

Year 1

(Spring 3rd grade)

Year 2

(Spring 4th grade)

Year 3

(Spring 5th grade)

SACD activity

Treatment Control Impact p-valueTreatment Control Impact p-valueTreatment Control Impact p-value

 Teacher sample size 441 409 425 404 425 396

Violence prevention and peace promotion (percent) 29.4* 8.5 20.9 0.000 31.5* 11.2 20.3 0.000 29.2* 10.1 19.1 0.000

Social and emotional development 34.6* 4.0 30.7 0.000 36.9* 6.3 30.6 0.000 30.0* 6.9 23.1 0.000

Character education (percent) 41.3* 6.3 35.0 0.000 43.3* 6.7 36.5 0.000 33.2* 7.0 26.2 0.000

Tolerance and diversity (percent) 22.8* 1.4 21.3 0.000 24.1* 3.4 20.7 0.000 22.0* 2.7 19.3 0.000

Risk prevention and health promotion (percent) 15.8 11.2 4.6 0.132 18.4 13.9 4.5 0.197 15.8* 9.3 6.5 0.043

Civic responsibility and community service (percent) ‡* ‡ 3.5 0.021 7.6* 1.5 6.1 0.010 8.1* 1.3 6.8 0.024

Any SACD goal (percent) 52.7* 18.6 34.1 0.000 51.8* 23.4 28.5 0.000 46.0* 19.6 26.3 0.000

‡ Reporting standards not met. Value suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

NOTE: Impact is the percentile point difference between treatment and control groups. Weights, which assign equal weight to each school within each of the programs and to each 

program across programs, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

75

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

76

Use of Materials and Strategies 

The SACD programs had a statistically significant positive effect on teacher-reported use of specific materials 

and classroom strategies to promote social and character development (table 1.21). Over the 3 years, 87 

comparisons were made between treatment and control teacher use of materials and strategies, with 4 

expected to be significant by chance. A significantly greater percentage of treatment teachers reported using 

these materials and strategies to support SACD goals in 40 of these comparisons. Treatment teachers were 

significantly more likely than control teachers to report use of four of the five specific materials asked about 

in at least 2 of the 3 years. In all years, treatment teachers were significantly more likely than control teachers 

to report using teacher guides (by 22 to 24 percentage points) and children’s literature (by 9 to 15 percentage 

points). In Years 2 and 3, treatment teachers were significantly more likely to report using student materials 

and instructional aids (by 10 to 12 percentage points). In addition, treatment teachers were significantly less 

likely to report not using any of these materials (by 7 to 10 percentage points) in all years. Control teachers 

reported rates of usage of materials from 36 percent for instructional aids to 60 percent for teacher guides, 

and they reported higher rates for not using any of the materials (13% to 14%). 

Of the 20 teaching strategies asked about, 13 strategies were reported as being used by a statistically 

significant greater percentage of treatment teachers (2 to 22 percentage points more) for at least one of the 

years, and no strategy was reported as being used by a significantly greater percentage of control teachers. 

Treatment teachers reported using, on average, one more strategy than control teachers, and this was a 

significant difference. 
Table 1.21. Impacts on use of SACD classroom materials and teaching strategies 

Year 1

(Spring 3rd grade)

Year 2

(Spring 4th grade)

Year 3

(Spring 5th grade)

SACD material and teaching strategy

Treatment Control Impact p-valueTreatment Control Impact p-valueTreatment Control Impact p-value

 Teacher sample size 441 409 425 404 425 396

Use of SACD materials (percent)

 Teacher guides (manuals, curricula) 82.4* 58.1 24.2 0.000 82.3* 60.4 21.8 0.000 79.9* 58.0 21.9 0.000

 Student materials (workbooks or sheets) 61.3 57.0 4.3 0.289 68.3* 56.1 12.2 0.002 66.2* 55.2 11.0 0.002

 Instructional aids (games, software, videos) 43.9 39.1 4.9 0.294 46.3* 36.7 9.6 0.017 47.6* 35.7 11.9 0.001

 Giveaways (bookmarks, stickers) 52.2 54.2 -1.9 0.688 52.9 47.3 5.6 0.157 49.9 52.6 -2.7 0.485

 Children’s literature 60.7* 46.2 14.6 0.002 60.4* 46.5 13.9 0.002 53.4* 44.6 8.8 0.013

 Other types of materials 8.2 8.5 -0.4 0.854 9.1 12.2 -3.1 0.201 9.0 6.9 2.1 0.360

 Did not use any of these materials 3.8* 14.2 -10.4 0.001 4.3* 13.5 -9.2 0.000 6.3* 13.1 -6.7 0.003

Use of teaching strategies (percent)

 Role-playing 85.5* 65.2 20.3 0.000 86.6* 65.1 21.5 0.000 87.1* 80.0 7.1 0.028

 Cooperative learning 96.3 96.1 0.2 0.883 98.0 96.8 1.3 0.250 99.8* 97.7 2.1 0.022

 Peer group discussions 92.9* 87.7 5.2 0.011 92.5^ 88.2 4.3 0.060 97.5^ 95.1 2.4 0.096

 Direct instruction of SACD 91.8* 78.8 13.0 0.000 93.6* 75.9 17.6 0.000 97.1* 87.9 9.2 0.000

 Skill training 63.0* 43.2 19.8 0.000 62.9* 40.5 22.4 0.000 82.1* 75.5 6.6 0.037

 Incorporating SACD into academic curriculum 86.5* 73.6 12.9 0.000 85.3* 71.3 14.0 0.000 92.9 90.3 2.5 0.225

 Parent training 8.1 8.9 -0.8 0.702 8.6 7.2 1.4 0.531 30.8* 21.5 9.3 0.016

 Parent/community involvement 22.9 27.0 -4.1 0.237 26.5^ 19.8 6.7 0.087 48.3* 39.7 8.6 0.022

 Mentoring 44.8 44.5 0.3 0.909 43.8 43.1 0.7 0.821 68.1* 58.3 9.8 0.016

 Good behavior notes sent home daily or weekly 75.8 72.5 3.3 0.409 78.2 71.9 6.3 0.108 91.7 89.5 2.1 0.270

 Presenting role models 76.4 73.3 3.1 0.335 75.7* 68.4 7.2 0.026 85.3 81.6 3.7 0.175

See notes at end of table.

77

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.21. Impacts on use of SACD classroom materials and teaching strategies—Continued

Year 1

(Spring 3rd grade)

Year 2

(Spring 4th grade)

Year 3

(Spring 5th grade)

SACD material and teaching strategy

Treatment Control Impact p-valueTreatment Control Impact p-valueTreatment Control Impact p-value

Use of teaching strategies—Continued (percent)

 Targeted story reading or writing on social 

 and character development themes 88.2* 75.9 12.3 0.000 88.2* 78.0 10.2 0.000 93.6 90.6 3.0 0.125

 Peer mediation 51.2 45.8 5.4 0.205 54.4 48.5 6.0 0.190 68.8 65.0 3.8 0.337

 Honor roll for positive behavior 55.1 57.9 -2.9 0.569 59.8 57.2 2.6 0.585 73.6 68.3 5.3 0.235

 Pledges or recitations on social and 

 character development themes 41.3 42.8 -1.4 0.802 43.4 44.6 -1.2 0.849 57.3 59.1 -1.8 0.721

 Guided visualization 60.9* 45.7 15.2 0.000 60.5* 47.3 13.2 0.003 70.8* 59.0 11.9 0.001

 Student-led/student-assisted instruction 56.0^ 49.8 6.2 0.094 57.3* 49.0 8.3 0.028 75.9 70.3 5.6 0.113

 Journaling 77.7 72.3 5.5 0.138 76.8 71.4 5.4 0.149 87.1^ 82.1 5.0 0.093

 Time out for negative behavior 86.2 85.3 0.8 0.803 85.5 83.0 2.4 0.349 94.1* 89.0 5.1 0.040

 Daily or weekly rewards for positive 

 behavior 91.0^ 86.8 4.1 0.098 92.1 89.7 2.4 0.307 98.2^ 96.0 2.2 0.092

 Any strategy 100.0 99.7 0.3 † 100.0 100.0 0.0 † 99.8 99.4 0.4 0.429

 Number of strategies (mean) 13.4* 12.3 1.1 0.000 13.6* 12.1 1.5 0.000 15.7* 14.6 1.1 0.000

† Not applicable.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were used in producing the treatment, control, and 

overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

78

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

79

Use of Schoolwide Strategies 

The SACD programs did not have an impact on the reported use of schoolwide strategies for supporting 

social and character development. Teachers were asked to report on the use of six schoolwide strategies to 

promote social and character development. including morning announcements or videos, assemblies, 

newspapers or bulletins, special school days, special events, and other activities. Over the 3 years, 18 

comparisons were made, with 1 expected to be significant by chance. The SACD programs did not have a 

statistically significant effect on the use of any of these schoolwide strategies (these data are not shown in a 

table). 

Participation in Professional Development 

The SACD programs had a statistically significant positive effect on teachers’ reports of participation in 

professional development for activities related to social and character development. Over the 3 years, 27 

comparisons were made, with 1 expected to be significant by chance. A significantly greater percentage of 

treatment teachers reported such professional development in 13 of the comparisons (table 1.22). The 

percentage difference in treatment and control teacher reports on receiving SACD training in the previous 12 

months ranged from 32 percentage points in Year 1 to 9 percentage points in Year 3. Mirroring the decline in 

effect over time, only in Year 1 was there a significant impact on the number of hours of professional 

development received (9 hours versus 4 hours). The significance of the impacts on percentages of teachers 

receiving training to help meet each of the six SACD goals varied by year: impacts were significant for four 

goals in Year 1, one goal in Year 2, and three goals in Year 3, with the differences ranging from 7 to 34 

percentage points. In Year 1, there was a significant difference in reports on behavior management training 

(11 percentage points). 
Table 1.22. Impacts on teacher-reported SACD professional development 

Year 1

(Spring 3rd grade)

Year 2

(Spring 4th grade)

Year 3

(Spring 5th grade)

SACD professional development

Treatment Control Impact p-valueTreatment Control Impact p-valueTreatment Control Impact p-value

 Teacher sample size 441 409 425 404 425 396

SACD training in past 12 months (percent) 85.6* 53.8 31.8 0.000 64.1* 52.4 11.7 0.003 56.6* 47.3 9.3 0.027

 

Hours of SACD training (mean) 9.1* 4.3 4.9 0.000 5.3 4.8 0.6 0.580 4.5 3.7 0.8 0.196

Training by goal (percent)

 Violence prevention and peace promotion 26.9* 17.3 9.6 0.009 18.2 14.3 3.9 0.241 17.8* 10.0 7.8 0.012

 Social and emotional development 36.3* 19.4 17.0 0.000 22.6 18.0 4.6 0.209 18.2* 11.2 7.0 0.044

 Character education 60.9* 26.7 34.2 0.000 36.2* 19.3 16.9 0.000 28.0* 14.8 13.2 0.001

 Tolerance and diversity 21.0* 12.3 8.7 0.017 16.8 19.0 -2.2 0.486 16.8 15.5 1.3 0.741

 Risk prevention and health promotion 13.3 14.3 -1.0 0.738 13.6 14.8 -1.2 0.707 14.4 12.0 2.4 0.404

 Civic responsibility and community service 9.8 7.0 2.7 0.221 7.5 4.8 2.7 0.130 3.9 4.8 -0.9 0.562

 Behavior management 36.4* 25.0 11.4 0.023 26.7 25.6 1.1 0.784 23.5 21.4 2.1 0.613

* Treatment group significantly different from control group at the .05 level. 

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were used in producing the treatment, control, and 

overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

80

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

81

Attitudes and Practices 

The SACD programs had a statistically significant positive effect on teachers’ reports on the use of six 

practices27 considered conducive to the social and character development of students but not on teacher 

attitudes toward social and character development. Twenty-seven comparisons were made over the 3 years, 

with 1 expected to be found significant by chance. A significantly greater percentage of treatment teachers 

reported that these practices were being used in 4 of these comparisons (these data are not shown in a table). 

In all 3 years, a significantly larger percentage of treatment teachers reported students having a voice in school 

governance (9% to 10% versus 4% to 5% of control teachers). In Year 3, treatment teachers reported 

significantly greater use of discipline for promoting development (57% versus 47% of control teachers). 

There were no statistically significant estimated impacts on teachers’ enthusiasm for SACD efforts in their 

schools in any of the years (these data are not shown in a table). For the 3 years, between 67 percent and 71 

percent of treatment and control teachers reported that they were enthusiastic about their schools’ efforts to 

promote social and character development and another 27 percent to 31 percent reported being cooperative 

(but not enthusiastic). Fewer than 2 percent reported an open dislike of the SACD efforts in their schools. 

Domain Results 

As discussed in the Analysis section, a set of heuristics was applied to determine whether the six outcome 

domains were statistically significant after adjustments were made for the multiple tests performed under each 

domain. These heuristics were applied to both the combined-program data and to each individual program’s 

data. Table 1.23 shows the statistically significant impacts on the use of SACD activities by domain for all 

seven programs combined and for each program. In table 1.23, a plus sign stands for a significant positive 

impact on the domain, and numeral superscripts show which heuristics identified the domain as significant. 

For all seven programs combined, a significant positive impact was found for each of the 3 years on the 

reports of use of any SACD activities, the use of SACD activities linked to named SACD programs, the use 

of classroom materials and teaching strategies, and professional development. 

Table 1.23 also shows the impacts on domain by individual program. In all cases, when a program’s impact 

was significant it was also positive. Five of the programs had at least 2 years of significant impacts on 

teachers’ reported use of activities, as did six programs for teachers’ use of activities linked to named SACD 

programs. Two programs had at least 2 years of significant impacts on reported use of classroom materials 

and strategies. No programs showed impacts on reported use of schoolwide strategies or on attitudes and 

practices in at least 2 years. One program showed 2 years of impacts on reported professional development. 

Results From Principal Interviews 

The principal interviews in 2005,28 2006, and 2007 echoed the findings from the teacher survey that there was 

a high usage of SACD activities in both the treatment and control schools (these data are not shown in a 

table). Over the years, 88 percent to 100 percent of treatment principals and 71 percent to 98 percent of 

control principals reported that their schools used SACD activities to promote each of the six SACD goals. 

Classroom teachers were reported as having a major role in delivering SACD activities by 91 percent to 98 

percent of treatment school principals and 64 percent to 90 percent of control school principals. SACD 

activities were being provided universally to all students rather than to targeted populations, as reported by 98 

 

27 These included teacher modeling of positive character and behavior with students, similar modeling with other staff, 

involvement of students in discussions, students having a voice in governance, school encouragement of parent 

involvement, and discipline practices that included promoting development. 

28 Only the results from the closed-ended items are reported for Year 1 because of problems with inter-rater reliability 

regarding the open-ended items. Results from both the closed- and open-ended items are reported for Years 2 and 3. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

82

Table 1.23. Significant impacts on use of SACD activity domains, overall and by program 

SACD activity domain

Program

Any SACD 

activities

Named 

SACD 

program 

activities

Classroom 

materials 

and 

strategies

Schoolwide 

strategies

Professional 

development

Attitudes 

and 

practices

Overall

 Year 1 +

1,2,3 +1,2,3 +2,3 +1,2,3

 Year 2 +

1,2,3 +1,2,3 +1,2,3 +2,3

 Year 3 +

1,2,3 +1,2,3 +1,2,3 +2,3

ABC

 Year 1 +

2

+

1

 Year 2

 Year 3 +

3

CSP

 Year 1 +

2

+

1,3 +3

 Year 2 +

3

 Year 3 +

3

LBW

 Year 1 +

2,3 +3

+

3

+

3

 Year 2 +

3

+

3

+

2

 Year 3

PA

 Year 1 +

2,3 +1,3 +3

+

3

 Year 2 +

1,3

 Year 3 +

3

+

3

PATHS

 Year 1 +

2,3 +1,3 +1,3

 Year 2 +

3

+

1,3

 Year 3 +

1,3

4Rs

 Year 1 +

1,2,3 +1,2,3 +3

+

1,3

 Year 2 +

1,3 +1,3

 Year 3 +

1,3 +1,2,3 +3

+

3

See notes at end of table.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

83

Table 1.23. Significant impacts on use of SACD activity domains, overall and by program—

Continued 

SACD activity domain

Program

Any SACD 

activities

Named 

SACD 

program 

activities

Classroom 

materials 

and 

strategies

Schoolwide 

strategies

Professional 

development

Attitudes 

and 

practices

SS

 Year 1 +

3

+

1,3

+

3

 Year 2 +

1, 3 +1,3

+

3

 Year 3 +

3

+

1,3

+

3

1 Based on univariate statistical tests, at least half of the impacts were positive and statistically significant and no impact was 

negative and statistically significant. 

2 The omnibus impact for all the outcomes measured together was positive and statistically significant on the basis of a multivariate 

statistical test. 

3 At least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after 

applying the Benjamini-Hochberg (1995) procedure to adjust significance levels downward to account for the multiple testing of 

impacts. 

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respecting, and Resolution)

 SS: Second Step

 +: Statistically significant beneficial impact on domain

 Blank cell: Finding of no impact

Significance is based on p ≤ .05. No detrimental impact was found statistically significant at or below the .05 level. Description of 

SACD activity domains and the heuristics used to determine the statistically significant beneficial impact on the domain.

 SACD activities: based on 16 teacher-reported measures on the use of SACD activities in the classroom.

 SACD activities linked to named programs: based on 14 teacher-reported measures on the use of SACD activities associated 

 with a named program in the classroom. 

 Classroom materials and strategies: based on 29 teacher-reported measures, 7 concerning materials used in the classroom 

 and 22 concerning classroom strategies.

 Schoolwide strategies: based on six teacher-reported measures concerning strategies to promote SACD schoolwide.

 Professional development: based on nine teacher-reported measures concerning their participation in SACD-related training

 Attitudes and practices: based on nine teacher-reported measures, three concerning teacher attitudes toward SACD efforts in the

 school and six concerning school practices conducive to the social and character development of students.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

84

percent to 100 percent of treatment school principals and 91 percent to 94 percent of control school 

principals. Unlike the findings from the teacher surveys, the principal surveys did not show a significantly 

greater use of SACD activities among the treatment schools, although this might be due to the inability to 

perform significance tests in seven cases as discussed below. In regard to whether principals of treatment 

schools reported greater use of activities to promote each of the six SACD goals than principals of control 

schools, 18 comparisons were made over the 3 years, with 1 expected to be significant by chance. In two 

cases (both in Year 2), a significantly greater percentage of treatment principals reported the use of SACD 

activities to support SACD goals: (1) tolerance and diversity (98% of treatment school principals versus 83% 

of control school principals, with a p-value of 0.036) and (2) risk prevention and health promotion (88% 

versus 71%, with a p-value of 0.044). These results do not provide evidence in support of the SACD 

programs’ impact on schools’ use of SACD activities because they are small in number and are not replicated 

over multiple years. However, for five of the comparisons significance testing could not be done because 100 

percent of the treatment principals reported that their schools had activities promoting a specific SACD goal. 

In regard to staff development, 27 comparisons were made over the 3 years, with 1 expected to be found 

significant by chance. In two cases (both in Year 2), there was a significant difference favoring the treatment 

schools. First, the mean percentage of teachers per school reported by their principals to have received 

training in social and character development in the past year was significantly higher for treatment schools 

versus control schools (55% versus 31%, with a p-value of 0.003). Second, the percentage of principals 

reporting that only 0 percent to 25 percent of their faculty had received such training treatment was 

significantly smaller among treatment school principals than control school principals (33% versus 67%, with 

a p-value of 0.002). These results do not provide evidence in support of the SACD programs’ impact on 

teacher training because they are small in number and are not replicated over multiple years. However, for 

two of the comparisons significance testing could not be done because 100 percent of the treatment 

principals gave the same report. 

Findings Regarding Use of SACD Activities 

The analysis of use of SACD activities had two main findings: (1) the SACD programs increased the reported 

use of and training in activities to promote students’ social and character development in the classroom 

compared to what reportedly occurred in the control schools, and (2) the reports from the control schools 

identified them as a standard practice control in that their staffs reported the use of and training in similar 

activities as the treatment staff. 

Evidence for the first finding came primarily from the teacher reports. Treatment teachers reported 

significantly greater implementation of activities than control teachers for 127 of the 249 outcomes measured 

and for four of the six SACD activity domains during the 3 years. Evidence for the second finding came from 

both the principal and teacher reports. The principals reported similar levels of SACD activities and teacher 

training among treatment and control schools (though statistical difficulties in testing all the comparisons may 

have contributed to this finding). The treatment teachers’ reports were statistically significantly greater than 

those of the control teachers but the practical differences varied. For example, the significantly greater 

percentage of treatment teachers reporting use of activities to promote any SACD goal was 5 to 10 

percentage points greater than the percentage of control teachers (86% to 90% of whom reported engaging in 

such activities), while the significant difference for the reported use of activities linked to named programs to 

promote any SACD goal ranged between 29 and 34 percentage points (with 36% to 42% of control teachers 

reporting engaging in these activities). 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

85

Year-by-Year Impacts on Students and Perceptions of School Climate 

The second and third of the primary research questions for the SACD evaluation were as follows: 

What is the average effect of the seven universal, school-based, social and character 

development programs on students’ social and emotional competence, behavior, 

academics, and on perceptions of school climate? 

What is the average effect of each specific SACD program on students’ social and 

emotional competence, behavior, academics, and on perceptions of school climate? 

One method to answer these questions was to examine the year-by-year impacts of the SACD programs on 

these outcomes over the 3 years as the students progressed from third through fifth grades. The examination 

of year-by-year impacts entailed three sets of analyses resulting in three sets of impacts. The first set of 

analyses compared the outcomes of treatment and control students from the fall of third grade to the spring 

of third grade. The second set compared the outcomes from the fall of third grade to the spring of fourth 

grade, and the third set compared the fall of third grade to the spring of fifth grade. Within each set of yearby-year analyses, an analysis of all the programs together provided impact results for the set of seven 

coherent, universal, school-based programs, and individual program analysis provided results specific to each 

program. In answering the first question above, the analysis of the combined seven programs was able to 

detect smaller statistically significant impacts because of its larger sample size and associated greater power 

than the analysis of individual SACD programs, which were based on the smaller samples available to answer 

the second question above. 

Analysis 

The random assignment design ensured that the main difference between the treatment and control group 

schools at the point of random assignment was exposure to the mix of classroom-based and other activities 

that comprised the SACD program being tested in each site. Thus, unbiased estimates of the average impacts 

of the SACD programs (relative to the social and character development activities offered in the control 

schools) could be computed as the difference in the average outcomes of students and teachers in the 

treatment and control schools. Because of the repeated cross-sectional design, this approach yielded unbiased 

estimates of the combined effects of the SACD programs on student outcomes and on potential treatmentinduced school entry and exit effects that could have influenced the composition of students and teachers in 

the schools at the follow-up points. 

Statistical precision was a concern for the SACD multiprogram evaluation because of design effects due to 

the clustering of students within schools (students shared the same teachers and school environment) and the 

relatively small numbers of schools available for program-level analyses. Therefore, regression procedures 

were used rather than simple differences-in-means procedures to estimate impacts to improve the statistical 

precision of the estimates and to adjust for differences between the treatment and control groups’ observable 

characteristics due to random selection, study nonconsent, and interview nonresponse.29

Two analyses are discussed in this section. The first is the overall analysis combining results from all 

programs. The overall analysis was done to determine if providing schools with a SACD program (as 

reflected in the average results from seven different programs) improved student outcomes. The seven 

 

29 The use of regression models raised the issue of the distribution of the outcome variables. Calculated p-values rely on 

the assumption of normality, and a violation of this condition may cause biases in p-values, thus leading to invalid 

hypothesis testing. However, if the sample size is sufficiently large, the large sample approximation for the distribution 

of the estimates can be used and the normality assumption can be relaxed as it was for the SACD evaluation. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

86

programs sought to change a common set of student behaviors by providing school-based instruction 

(drawing upon a common set of methods and approaches) through the teacher to all students. The focus of 

the programs differed in terms of specific instructional approaches and student outcomes so they did not 

represent a unified, single approach to improving student social and character development. Therefore the 

combined-program analysis was not an evaluation of a general SACD approach (though the results could 

contribute to such an evaluation) nor does it have any direct application to similar SACD programs not 

included in this study. Second, individual analysis of each program was done to evaluate them separately. 

Besides providing an evaluation of each program, these analyses were used to examine the possibility that 

results for a subset of the programs led to the results from the combined-program analysis. 

The Combined-Program Model and the Program-Level Models 

For the combined-program analysis, pooled impact estimates from all the programs were obtained from each 

year of the study to examine the extent to which, taken together, the seven SACD programs, on average, 

changed student and school outcomes relative to what they would have been otherwise. A hierarchical linear 

model (HLM) was used to estimate regression-adjusted impacts (Bryk and Raudenbush 1992). The basic 

model consisted of two levels that were indexed by students or teachers (i) and schools (s), and where fixed 

site effects were indexed by d: 

0

0 0 1

(1) 1: / :

2: : .

isd sd isd isd

sd sd d sd sd

Level Students Teachers Y X e

Level Schools T Z u

 

    

  

    

In this model, Yisd is an outcome measure (from spring of Year 1, Year 2, or Year 3) for a student or teacher; 

0sd is a school-level random intercept; Xisd are student- or teacher-level initial covariates30; Tsd is a binary 

variable equal to 1 for treatment group schools and 0 for control schools; d are program-specific fixed effects; 

Zsd are school-level initial covariates (and teacher-level initial covariates for student outcomes)31; , 0, 1, and 

 are fixed parameter vectors to be estimated; eisd are assumed to be independent and identically distributed

N(0,2

e) student- or teacher-level random error terms; and usd are independent and identically distributed

N(0,2

u) school-specific error terms that capture the correlation between the outcomes of students (teachers) 

in the same schools and are assumed to be distributed independently of the Level 1 error terms. 

Inserting the Level 2 equation into the Level 1 equation creates the following single level model: 

0 1 (2) [ ]. Y T Z X u e isd sd d sd isd sd isd            

In this formulation, the estimate of the parameter, 1, is the regression-adjusted, combined-program impact 

estimate. The standard error of this estimate accounts for design effects due to the clustering of students in 

schools, as well as precision gains from the inclusion of initial covariates that explain some of the variation in 

 

30 The X vector refers to initial covariates for students when student outcomes are used and to initial covariates for 

teachers when teacher-level outcomes are used. For new entrants, most covariates were considered time-invariate and so 

could be used as initial values when collected later in the study. Imputation was used to estimate the values for timevariate covariates (such as initial scores on outcomes) using information from similar students who were in the sample in 

Year 1. In addition, for students in the sample during Year 1, some of the covariates, such as the student outcome 

scores, could have changed between the start of program implementation and data collection in fall 2004. Sensitivity 

analyses reported in appendix B show that the point estimates of the impacts are similar when the covariates are 

included or excluded in the model. 

31 Conceptually, the Z vector referred to both school-level and teacher-level covariates. However, the final model did not 

use school-level covariates so Z actually includes only teacher-level initial covariates for the analysis of student outcomes. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

87

outcomes between and within schools. T tests were used to gauge the statistical significance of the impact 

estimates. 

This model produces impact estimates that are internally valid but not necessarily externally valid to a broader 

population of sites. This stems from treating site effects as fixed rather than random (between-site variance 

terms are not accounted for in the variance calculations), which was done because sites were selected 

intentionally for the study. In addition, because all classrooms within the study schools were included in the 

evaluation (there was no sampling of classrooms and the model does not include classroom error terms), the 

estimated student-level impact findings generalize to the classrooms in the study schools at the time of the 

evaluation. 

For the evaluation of each program, impact estimates were generated by estimating equation (2) separately for 

each program. Each model included program-specific covariates and random school effects but excluded the 

program fixed effects (d). Sample weights were used in all analyses. F tests were used to test for differences 

in estimated impacts across programs and t tests were used to determine the statistical significance of the 

program-specific impact estimates. 

Covariates 

The X and Z covariates in equation (2) were constructed using the fall 2004 surveys. The covariates were 

selected based on two main criteria: (1) they should adjust for statistically significant treatment and control 

differences at initial data collection; and (2) they should have predictive power across a broad range of 

outcomes. Using stepwise regression procedures, a separate set of covariates were selected for the outcomes 

from each of the four data sources for the combined-program analysis (see appendix B for details). Table 1.24 

lists the covariates used with the outcomes from each report (child, primary caregiver, teacher on student, and 

teacher on classroom and school). These covariates were used for the analysis of the Year 1, Year 2, and Year 

3 data. As a sensitivity check, the same covariate selection process was used with the Year 2 data and the 

model re-estimated using the alternative covariates. The use of the alternative covariates did not lead to 

different patterns of results, although the statistical significance of some of the program-level impact 

estimates changed. For all 3 years, missing covariates were imputed using mean values for nonmissing cases, 

by school, gender, and race/ethnicity. 

For the individual evaluations of each program, covariates were identified using the same procedure as above 

but were allowed to vary between programs. This led to 28 different sets of covariates (seven programs by 

four reports providing outcomes). The covariates used in the models for the specific programs are detailed in 

the program chapters. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

88

Table 1.24. Covariates used with outcomes from each report for combined-program analysis 

Potential covariate

CR 

outcome

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome

 Total number 26 35 28 8

Child-reported 

 Female   

 Hispanic (non-White)   

 Black (non-Hispanic)   

 Other ethnicity   

 Age in years   

 Scales

 Afraid at School

 Altruistic Behavior  

 Empathy 

 Engagement with Learning   

 Negative School Orientation  

 Normative Beliefs About Aggression 

 Sense of School as a Community  

 Problem Behavior 

 Self-Efficacy for Peer Interactions  

 Victimization at School 

Primary caregiver-reported 

 Age in years 

 Completed high school or equivalent   

 Some college   

 Bachelor’s or higher degree   

 Highest level of education in household

 Completed high school or equivalent   

 Some college   

 Bachelor’s or higher degree   

 Mother present in home life  

 Mother and father present   

 Respondent someone other than mother or father   

 Number of people in household   

 Household income: $20,000 to $40,000  

 Household income: $40,000 to $60,000  

 Household income: More than $60,000  

 Income-to-poverty-threshold ratio: Below 135 percent  

 Income-to-poverty-threshold ratio: 135 to 185 percent  

 Full-time employment 

 Part-time employment 

See notes at end of table.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

89

Table 1.24. Covariates used with outcomes from each report for combined-program 

Table 1.24. analysis—Continued

Potential covariate

CR 

outcome

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome

 Parental scales

 APQ—Poor Monitoring and Supervision Subscale

 APQ—Positive Parenting Subscale  

 Child-Centered Social Control

 Confusion, Hubbub, and Order 

 Community Resources

 Community Risk 

 Parent and teacher Involvement

 Child scales

 Altruistic Behavior 

 Positive Social Behavior  

 Problem Behavior  

Teacher-reported 

 Female 

 Hispanic (non-White) 

 Black (non-Hispanic) 

 Other ethnicity 

 Total teaching experience 

 Total experience in current school

 Regular certificate 

 Other certificate 

 Child scales

 Academic Competence and Motivation 

 ADHD-Related Behavior  

 Altruistic Behavior  

 Positive Social Behavior  

 Problem Behavior 

 Parent and Teacher Involvement 

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

 APQ: Alabama Parenting Questionnaire

 : Covariate used

 Blank cell: Covariate not used

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

90

Sample Weights 

Sample weights were used in all analyses, and they were constructed for three main reasons: (1) to give each 

site equal weight in the calculation of combined-program impact estimates, (2) to give each school equal weight 

in each site, and (3) to adjust for missing outcome data due to study nonconsent and survey nonresponse.32

To adjust for missing student data, the assumption was made that students in a specific classroom with spring 

follow-up data were representative of all students in that classroom. Weights were constructed to be inversely 

proportional to the combined consent and response rates within each classroom (see appendix B). These 

weights were constructed separately for original cohort stayers and for new entrants because nonconsent rates 

were higher for the new entrants. For the teacher-level weights, the assumption was made that responding 

teachers were representative of all third-, fourth-, and fifth-grade teachers in that school. The standard errors 

of all estimated impacts were adjusted for design effects due to unequal weighting. 

Statistical Significance and Substantively Important Effects 

Results are provided in effect sizes that were calculated by dividing the estimated impact (the coefficient 

estimated by the regression model) by the standard deviation of the outcome measure for the control group. 

The standard deviation was calculated using the initial data for the weighted control group. The statistical 

significance and substantive importance of each result were identified. A standard two-tailed test was used to 

determine the p-value for the impact coefficient of each outcome measure. Coefficients with p-values of 0.05 

or below were considered statistically significant and identified as such. In addition, coefficients with p-values 

between 0.05 and 0.10 were identified in the tables to identify where there may have been additional 

significant results found if a larger sample had been used. However, these results were not considered in the 

text or compilations of statistically significant results. 

Individual outcomes having non-statistically significant effect sizes of 0.25 or above (and -0.25 or below) were 

identified as having substantively positive (negative) importance, following the practice used by the What 

Works Clearinghouse. Substantive importance may identify impacts of practical importance that might have 

been found statistically significant if the sample size were larger. 

In addition to estimating the impacts of the SACD programs on the individual outcome measures, the 

impacts on the four domains (Social and Emotional Competence, Behavior, Academics, and Perceptions of 

School Climate) were also examined. Testing the impact on the domains was done to adjust for the multiple 

comparisons made within each domain in order to address the increased chances of finding a spurious 

outcome when more than one test was done. As a result, this section reports two sets of results: (1) the 

impacts on the individual outcomes unadjusted for multiple comparisons; and (2) the impacts on the domains 

that serve as the multiple comparison adjustment. 

The testing of the significance of the impacts on the domains was based on a set of four heuristics that were 

modeled on the approach used by the U.S. Department of Education’s What Works Clearinghouse for 

determining whether domain-level effects are statistically significant when there are multiple outcome 

measures within a single domain (What Works Clearinghouse n.d.-b). The first three of these heuristics were 

also applied in the earlier analysis of the SACD programs’ impacts on the use of SACD activities to six 

outcome domains.33 Each domain was checked using the four heuristics and a statistically significant positive 

impact was found if any of the four were met.34 The four heuristics were as follows: 

 

32 This weighting approach produced unbiased estimates of the average treatment effect for the study schools and 

programs that were purposively selected for the evaluation. See appendix B for more information. 

33 The fourth heuristic was appropriate for use with the student-level outcomes but not with the teacher-level use of 

SACD activities outcomes, so it was not applied to the six domains containing the individual outcomes of SACD activity 

use. 

34 For ease of discussion, this section describes detecting positive significant impacts. These heuristics were also used to 

detect any negative impact on a domain. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

91

1. Based on the results from the statistical test of each outcome variable within a domain (unadjusted 

for multiple comparisons), at least half of the impacts were positive and statistically significant and no 

impact was negative and statistically significant. 

2. The omnibus impact for all the outcomes measured together was positive and statistically significant 

on the basis of a multivariate statistical test (see footnote 25). 

3. After applying the Benjamini-Hochberg (Benjamini and Hochberg 1995) procedure to each outcome 

under a domain, at least one of the outcomes remained positive and statistically significant and no 

outcome was negative and statistically significant. The Benjamini-Hochberg procedure adjusts 

significance levels downward to account for the multiple testing of impacts. 

4. The statistical model used to estimate impacts on the individual outcomes was re-estimated using a 

composite of all the outcome variables under a domain. The domain was found significant if the 

impact on the composite was significant. The composite was formed by standardizing each outcome 

variable using its standard deviation, combining the values of the outcome variables, and taking the 

average of the final value. 

Statistical Power 

To assess the statistical power of the combined-program and individual program impact estimates under the 

SACD design, minimum detectable impacts in effect size (MDES) units for each outcome measure were 

calculated. MDES represent the smallest impacts in effect size (standard deviation) units that can be detected 

with a high probability. The MDES were primarily a function of study sample sizes, the degrees of freedom 

available for statistical tests, and design effects due to clustering (Schochet 2005). Clustering effects are 

measured by intraclass correlations (ICCs) that reflect the percentage of the total variance in the outcomes 

that is between clusters (schools). These MDES were calculated at 80 percent power. Thus, it is possible to 

find a statistically significant impact estimate on an outcome when the true impact is smaller than the relevant 

MDES, although the chance that this will occur is less than 80 percent. Similarly, it is possible to find an 

impact estimate that is not statistically significant when the true impact is as large as the MDES, although the 

chance that this will occur is 20 percent or less. 

Table 1.25 displays the MDES using the ICCs from equation (2) that adjusted them using the covariates 

(assuming a two-tailed test and a 5 percent significance level). For the outcomes from the Child Report, the 

MDES fell below 0.1 except for Positive School Orientation in Years 2 and 3, which reached about 0.13. For 

outcomes from the Primary Caregiver Report, MDES fell below 0.05 for all 3 years. MDES for outcomes 

from the Teacher Report on Student ranged from 0.05 to 0.24, while those for the Teacher Report on 

Classroom and School ranged from 0.15 to 0.21. 

The MDES for the individual programs were considerably higher, ranging from 0.1 to 0.9 over the 3 years, 

due to smaller sample sizes. They varied somewhat across sites with no consistent pattern. MDES for the 

specific programs can be found in the program chapters. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

92

Table 1.25. Adjusted minimum detectable effect sizes for combined-program impact evaluation 

Outcome measure–Report Year 1 Year 2 Year 3

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 0.048 0.048 0.035

 Normative Beliefs About Aggression–CR 0.045 0.081 0.076

 Empathy–CR 0.068 0.078 0.063

Behavior Domain

 Altruistic Behavior–CR 0.041 0.057 0.043

 Altruistic Behavior–PCR 0.035 0.044 0.047

 Altruistic Behavior–TRS 0.155 0.238 0.204

 Positive Social Behavior–PCR 0.027 0.042 0.042

 Positive Social Behavior–TRS 0.078 0.115 0.112

 Problem Behavior–CR 0.056 0.086 0.075

 Problem Behavior–PCR 0.047 0.040 0.046

 Problem Behavior–TRS 0.084 0.086 0.101

 ADHD-Related Behavior–TRS 0.071 0.082 0.101

Academics Domain

 Engagement with Learning–CR 0.029 0.036 0.054

 Academic Competence and Motivation–TRS 0.053 0.077 0.097

Perceptions of School Climate Domain

 Positive School Orientation–CR 0.088 0.123 0.128

 Negative School Orientation–CR 0.033 0.070 0.087

 Student Afraid at School–CR 0.048 0.064 0.059

 Victimization at School–CR 0.046 0.063 0.065

 Feelings of Safety–TRCS 0.171 0.179 0.207

 Student Support for Teachers–TRCS 0.151 0.154 0.188

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The minimum detectable effect (MDE) formula used in the calculations is as follows:

 

where sT and sC are the number of treatment and comparison schools; nT and nC are the average number of students per classroom; 

1 is the intraclass correlation (ICC) at the school level; and factor(df) is a constant that depends on the number of degrees of 

freedom (df) available for analysis (and is 2.802 for the pooled analysis). Estimates were adjusted for fixed program effects as well as 

baseline covariates.

SOURCE: The Social and Character Development (SACD) Research Program.

1 1

1 1 1 1 ( ) * ( ) (1 )( )

T C T T C C

MDE factor df

s s s n s n

      
Chapter 1. The Social and Character Development Multiprogram Evaluation 

93

Results 

Table 1.26 provides the estimates of the overall impact on each of the 20 outcome measures over each of the 

3 years.35 For the third-graders in Year 1 (using the spring 2005 data), there was one statistically significant 

impact of the SACD programs, when combined, on the teacher-reported Student Support for Teachers 

measure, which had an effect size of 0.12 standard deviations. Effect sizes ranged from 0.00 to 0.12 (absolute 

value), so there were no substantively important effects. For the fourth-graders in Year 2 (using spring 2006 

data), there was one statistically significant impact, again on the Student Support for Teachers measure, which 

had an effect size of 0.16 standard deviations. The other effect sizes ranged from 0.00 to 0.13 (absolute 

value), and there were no substantively important effects. For the fifth-graders in Year 3 (using spring 2007 

data), there were no statistically significant impacts of the SACD programs on any of the 20 outcome 

measures. Effect sizes ranged from 0.00 to 0.08 (absolute value), and there were no substantively important 

effects. With 2 of 60 coefficients found statistically significant (less than the 3 that would be expected by 

chance), and none found substantively important, the combined-program evaluation provides no support for 

an average SACD program effect on student outcomes. Regarding statistically significant impacts on the 

domains, the fourth heuristic indicated a statistically significant detrimental effect of the SACD programs on 

the domain of Social and Emotional Competence in Years 2 and 3. 

 

35 Sample sizes are not reported in the results tables because they vary by outcome and by year. Table 1.47 provides the 

range of sample sizes for the outcomes within each report by year. 
Table 1.26. Combined-program impacts on outcomes 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment Control Effect

size p-value

Treatment Control Effect

size p-value

Treatment Control Effect

size p-value

Social and Emotional Competence Domain1,2

 Self-Efficacy for Peer Interactions–CR (+) 3.04 3.06 -0.03 0.476 3.18^ 3.22 -0.07 0.096 3.22 3.25 -0.04 0.233

 Normative Beliefs About Aggression–CR (-) 1.29 1.30 -0.01 0.826 1.35 1.36 -0.02 0.667 1.46 1.46 -0.01 0.913

 Empathy–CR (+) 2.32 2.30 0.06 0.197 2.18 2.19 -0.02 0.739 2.08 2.10 -0.05 0.343

Behavior Domain

 Altruistic Behavior–CR (+) 1.24^ 1.29 -0.06 0.094 1.05 1.09 -0.06 0.168 1.04 1.06 -0.04 0.368

 Altruistic Behavior–PCR (+) 2.27^ 2.22 0.07 0.055 2.24 2.24 0.00 0.992 2.25 2.27 -0.02 0.691

 Altruistic Behavior–TRS (+) 1.43 1.39 0.08 0.380 1.36 1.35 0.01 0.935 1.35 1.37 -0.04 0.733

 Positive Social Behavior–PCR (+) 3.02 3.03 -0.01 0.726 3.08 3.05 0.05 0.183 3.10 3.08 0.03 0.436

 Positive Social Behavior–TRS (+) 3.03 3.02 0.02 0.659 3.02 3.04 -0.02 0.747 3.10 3.07 0.03 0.606

 Problem Behavior–CR (-) 0.30 0.30 0.01 0.849 0.34 0.35 0.00 0.940 0.48 0.47 0.03 0.518

 Problem Behavior–PCR (-) 1.56 1.56 0.00 0.914 1.53 1.54 -0.03 0.359 1.53 1.54 -0.04 0.335

 Problem Behavior–TRS (-) 1.46 1.45 0.02 0.620 1.44 1.45 -0.02 0.667 1.45 1.47 -0.04 0.467

 ADHD-Related Behavior–TRS (-) 1.75 1.75 0.00 0.925 1.68 1.70 -0.04 0.474 1.65 1.70 -0.08 0.205

Academics Domain

 Engagement with Learning–CR (+) 3.66 3.68 -0.04 0.193 3.67 3.68 -0.03 0.462 3.59 3.61 -0.04 0.411

 Academic Competence and Motivation–TRS (+) 2.98 3.00 -0.02 0.353 2.96 2.98 -0.02 0.659 2.98 2.98 0.00 0.960

See notes at end of table.

94

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.26. Combined-program impacts on outcomes—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment Control Effect

size p-value

Treatment Control Effect

size p-value

Treatment Control Effect

size p-value

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 2.87 2.85 0.03 0.573 2.66 2.64 0.03 0.717 2.51 2.56 -0.07 0.411

 Negative School Orientation–CR (-) 1.88 1.91 -0.04 0.168 1.96 1.98 -0.03 0.466 2.09 2.10 -0.01 0.857

 Student Afraid at School–CR (-) 2.26 2.31 -0.05 0.165 2.21 2.26 -0.06 0.223 2.20 2.21 -0.01 0.818

 Victimization at School–CR (-) 0.76 0.76 -0.01 0.760 0.66 0.69 -0.04 0.382 0.72 0.73 -0.01 0.913

 Feelings of Safety–TRCS (+) 3.47 3.42 0.06 0.472 3.47 3.36 0.13 0.127 3.31 3.38 -0.08 0.445

 Student Support for Teachers–TRCS (+) 3.50* 3.39 0.12 0.046 3.55* 3.41 0.16 0.036 3.43 3.47 -0.05 0.546

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 Impact on domain found statistically significant and detrimental in Year 2 based on the fourth heuristic, in which the statistical model used to estimate impacts on the individual 

outcomes was re-estimated using a composite of all the outcome variables under a domain. The domain was found significant if the impact on the composite was significant. The 

composite was formed by standardizing each outcome variable using its standard deviation, combining the values of the outcome variables, and taking the average of the final value.

2 Impact on domain found statistically significant and detrimental in Year 3 based on the fourth heuristic.

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models where each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. The effect size 

was calculated by dividing the estimated impact by the standard deviation of the outcome measure for the control group. Significance is based on p ≤ .05. See table 1.5 for information 

about the measures used to create the outcome variables. The number of results found significant was no more than expected by chance.

SOURCE: The Social and Character Development (SACD) Research Program.

95

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

96

The lack of statistically significant beneficial impact estimates at the overall level might have been due to 

beneficial impacts in some programs that were offset by detrimental impacts in others. Estimated impacts 

could have differed across programs if some SACD programs were more effective than others, or if the types 

of children or settings in some programs were more conducive to positive program effects than others. To 

investigate this possibility, differences in estimated impacts across programs were tested, as was the statistical 

significance of program-specific impact estimates. 

Impact estimates, however, did not differ significantly across programs. Using an F test to test for differences 

in estimated impacts across programs led to the finding that none of the differences in pooled impact 

estimates across programs was statistically significant at the 5 percent level in any of the 3 years. This result 

provided evidence that it was not the case that one or two programs drove or masked the estimated impacts 

in the combined data. 

Table 1.27 lists the outcomes on which each individual program had statistically significant impacts in each 

year (a program’s impacts on all the outcomes are given in the program-specific chapters). There were 5 

significant program impacts (3 beneficial and 2 detrimental) in Year 1, 7 (5 beneficial and 2 detrimental) in 

Year 2, and 4 (1 beneficial and 3 detrimental) in Year 3, for a total of 16 (9 beneficial and 7 detrimental) 

program-level impacts. As a comparison, 7 of the 140 comparisons made for each year would be expected to 

be statistically significant by chance, which would produce a total of 21 expected significant impacts. Table 

1.27 also lists the 19 nonsignificant substantive impacts by program; 10 were beneficial and 9 were 

detrimental. 
Table 1.27. Individual program statistically significant impacts and nonsignificant but substantively important impacts 

Statistically significant1 Nonstatistically significant but substantive2

Program

Beneficial impacts Detrimental impacts Beneficial impacts Detrimental impacts

(Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value)

 Total 9 7 10 9

All programs

 Year 1 3 2 2 0

 Year 2 5 2 6 1

 Year 3 1 3 2 8

ABC

 Year 1 Altruistic Behavior

(TRS) (.39) (.026)

 Year 2 Academic Competence Altruistic Behavior Student Support for Teachers (TRCS) 

(CR) (.31) (.011) (CR) (-.20) (0.029) (.27) (.276)

Feelings of Safety

(TRCS) (.75) (.003)

 Year 3 Positive Social Behavior 

(PCR) (.21) (.041)

Feelings of Safety

(TRCS) (.31) (.235)

CSP

 Year 1

 Year 2 Problem Behavior Altruistic Behavior

(PCR) (-.21) (.042) (TRS) (.47) (.132)

Student Afraid at School

(CR) (-.26) (.090)

 Year 3 Altruistic Behavior

(TRS) (-.41) (.132)

Feelings of Safety

(TRCS) (-.36) (.246)

LBW

 Year 1 Altruistic Behavior

(PCR) (.31) (.005)

Student Support for Teachers

(TRCS) (.52) (.022)

 Year 2 Student Support for Teachers Altruistic Behavior

(TRCS) (.28) (.428) (TRS) (-.34) (.270)

See notes at end of table.

97

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.27. Individual program statistically significant impacts and nonsignificant but substantively important impacts—Continued 

Statistically significant1 Nonstatistically significant but substantive2

Program

Beneficial impacts Detrimental impacts Beneficial impacts Detrimental impacts

(Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value)

 Year 3 Engagement with Learning Problem Behavior 

(CR) (-.35) (.030) (CR) (.31) (.223)

Positive School Orientation Student Support for Teachers 

(CR) (-.33) (.047) (TRCS) (-.26) (.543)

Feelings of Safety

(TRCS) (-.70) (.046)

PA

 Year 1 Engagement with Learning Altruistic Behavior

(CR) (-.25) (.017) (TRS) (.27) (.480)

 Year 2 Positive Social Behavior Student Support for Teachers

(PCR) (.24) (.039) (TRCS) (.28) (.113)

Problem Behavior 

(TRS) (-.24) (.048)

 Year 3

PATHS

 Year 1

 Year 2

 Year 3 Altruistic Behavior 

(TRS) (-.31) (.485)

Feelings of Safety 

(TRS) (-.29) (.582)

4Rs

 Year 1 Academic Competence 

(CR) (-.17) (.032)

 Year 2

 Year 3 Feelings of Safety 

(TRS) (-.42) (1.46)

Student Support for Teachers 

(TRCS) (-.35) (.109)

See notes at end of table.

98

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.27. Individual program statistically significant impacts and nonsignificant but substantively important impacts—Continued 

Statistically significant1 Nonstatistically significant but substantive2

Program

Beneficial impacts Detrimental impacts Beneficial impacts Detrimental impacts

(Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value)

SS

 Year 1 Feelings of Safety 

(TRCS) (.37) (.216)

 Year 2 Positive Social Behavior Feelings of Safety 

(PCR) (-.14) (.050) (TRCS) (.39) (.197)

 Year 3 Feelings of Safety 

(TRCS) (.52) (.062)

1 Of the 140 comparisons made for each year, 7 would be expected to be statistically significant at the .05 level by chance (for a total of 21).

2 Defined as impacts that were not statistically significant but were .25 standard deviation units (absolute value) or more in magnitude.

NOTE: Abbreviations are

ABC: Academic and Behavioral Competencies Program

CSP: Competence Support Program

LBW: Love In a Big World

PA: Positive Action

PATHS: Promoting Alternative Thinking Strategies

4Rs: The 4Rs Program (Reading, Writing, Respecting, and Resolution)

SS: Second Step

CR: Child Report

PCR: Primary Caregiver Report

TRS: Teacher Report on Student

TRCS: Teacher Report on Classroom and School

Blank cell: Finding of no impact.

All impact estimates were calculated using regression models in which each school within a program was weighted equally. The standard errors of all estimates account for design 

effects due to unequal weighting and the clustering of students within schools. Significance is based on p ≤ .05. The number of results found significant was no more than expected by 

chance.

SOURCE: The Social and Character Development (SACD) Research Program.

99

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

100

Sensitivity Analyses 

To estimate the multilevel models, it was necessary to make decisions about key model parameter 

specifications and estimation methods. These decisions had the potential to affect the results. A set of 

sensitivity analyses was done to determine if these model assumptions affected the results from the 

combined-program impact analysis. Ten sensitivity tests were done, and these included the following 

variations: 

1. The models were estimated without the initial covariates. 

2. The models were estimated without sample weights and also with weights that were not adjusted for 

nonconsent and nonresponse. 

3. Classroom-level random effects were included in the error structure. 

4. The pairwise matching of schools was accounted for in the error structure. 

5. The pretests were treated as dependent variables rather than as covariates. 

6. The models were estimated using alternative statistical software packages. 

7. Missing outcome measures were imputed using multiple imputation procedures. 

8. Combined-program impact estimates were obtained by averaging the program-level impact estimates. 

9. Restricted sets of covariates were included in the models. 

10. New entrants were excluded from the analysis. 

The number of sensitivity analyses done each year declined as the pattern of impact results from the original 

model proved robust to a variety of model specifications. All 10 sensitivity analyses were done with the Year 

1 data. The first nine sensitivity tests were done with the Year 2 data. The sample of new entrants grew large 

enough in Years 2 and 3 to do separate analyses of new entrants versus stayers (in place of the 10th sensitivity 

test) and these were included in the subgroup analyses discussed in chapter 1. The first, second (except for 

the estimation without weights), third, and fourth sensitivity tests were done using the Year 3 data. 

The results from the sensitivity analyses were similar to those of the original combined-program analysis and 

did not produce additional evidence of impacts on the outcome measures. Of the 10 sensitivity analyses, 5 

found no significant impacts on the 20 outcomes, 4 found one significant impact, and 1 found two significant 

impacts. The sensitivity analyses and their results are discussed in appendix B. 

Year-by-Year Analysis of Impacts on Student Subgroups 

The social and character development evaluation research questions included the expectation that the SACD 

programs, considered together, might be more effective for some groups of children than others. These 

subgroups were based on (1) student gender; (2) student risk factors; (3) student status as a stayer (member of 

the original sample from fall 2004) or a new entrant to the study; and (4) fidelity of implementation in the 

student’s school. Descriptive analyses, discussed earlier, of the initial data confirmed that the characteristics 

used to form the first two subgroups were randomly distributed between the treatment and control groups, 

and found no statistically significant difference in the composition of the treatment and control groups by 

stayers and new entrants. These subgroups were hypothesized to respond differently to the SACD programs 

based on (1) indications in the literature that gender, socioeconomic risk, family risk, and community risk are 

associated with children’s behavioral and academic outcomes; and (2) results from previous evaluations of 

universal school-based SACD interventions, which have found that impacts can differ for children at high 

risk for behavior problems and academic failure, and that impacts can be greater for children who have 

received greater program exposure. 

The identification of gender and of stayer versus new entrant status was straightforward because these were 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

101

clearly distinctive subgroups of boys versus girls and students who were in the sample from the initial data 

collection versus those who joined afterward. Identification of the subgroups defined by initial risk and 

fidelity of implementation required the use of scales. 

To identify students with initial risks, four domains of initial risk were examined: 

1. Socioeconomic risk was measured by the number of the following risk factors the child faced at the 

beginning of the evaluation: (1) the child was in a single-parent household, (2) the child was in a 

low-income household (below 135 percent of the federal poverty level), and (3) the child’s 

primary caregiver was a high school dropout. The risk measure used in the analysis was a 

cumulative score across the three risk variables with the following values: 0 (no risk factors), 1 

(one risk factor), and 2 (two or three risk factors). 

2. Family risk was defined as poor parental supervision and taken from the Alabama: Poor Supervision 

Scale from the Primary Caregiver Report data. 

3. Perceived community risk was defined as perceived neighborhood risk measured through items 

from the Community Risk Scale from the Primary Caregiver Report data. 

4. Child behavior risk was measured by the BASC Conduct Problems Subscale, responses to which 

were collected in both the Teacher Report on Student and the Primary Caregiver Report. 

Thus, there were five risk measures used to examine the four initial risk domains: (1) one measure for 

socioeconomic risk, which consisted of a scale based on three variables; (2) one measure for family risk, 

which was based on one variable; (3) one measure for community risk, which was based on one variable; and 

(4) two measures for child behavior risk. Other variables were considered for each risk measure (e.g., parents’ 

employment status for socioeconomic risk, positive parenting for family risk, and community resources for 

community risk) but, in a regression analysis of the initial data, these variables were not found to predict the 

outcome measures. The five selected risk measures were treated in the analysis as continuous, rather than 

categorical, variables to preserve all risk information, and because there were no well-established rules for 

defining risk cutoffs and groups for which the effects of the SACD evaluation were likely to differ. 

Fidelity of implementation of the SACD programs was measured differently in each of the three years. In 

Year 1, treatment schools were divided into two fidelity groups: (1) high implementers (20 schools) and 

(2) low implementers (22 schools). In Years 2 and 3, schools were again divided into the high (20 schools in 

Year 2, 19 schools in Year 3) and low implementers (22 schools in Year 2, 23 schools in Year 3) but the 

fidelity measure combined the rankings from multiple years into three categories: (1) high implementers for 

all years, (2) low implementers for all years, and (3) mixed implementers (high and low implementers in 

different years). The discussion of the construction of the fidelity of implementation variables noted that the 

fidelity analysis was considered exploratory, both because of how the variable was constructed and because of 

the nonrandom assignment of high and low fidelity. As a result, differences in estimated impacts by fidelity 

group could reflect differences in the level of implementation and/or underlying differences between the 

students, the schools, and the SACD programs. 

Analysis 

The subgroup analyses were conducted at the combined-program level. The 18 student-level outcomes were 

used. The degree of power available for testing the differences between the subgroups was lower than for the 

whole-group analysis and depended on the number and sizes of the subgroups. Therefore the results were 

measured less precisely than for the year-by-year impacts at the combined-program level. Partly as a result of 

this, and because the What Works Clearinghouse heuristics do not specify how to conduct multiple 

comparison tests across subgroups, the heuristics were not applied to examine subgroup impact results by 

domain.36

 

36 The comparison approach used was to assess whether there were differences in impacts across subgroups (e.g., 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

102

Two of the subgroups were discrete binary variables (boy/girl or stayer/new entrant) as was Year 1 fidelity 

(high/low), one subgroup included three discrete binary variables (fidelity for Years 2 and 3), and the initial 

risk variables were continuous (and so were not truly discrete subgroups). For all the subgroup analyses, the 

model used to test the equality of the subgroup impacts can be represented in a unified equation as 

Yisp = γo + γ1Tsp + θp + Xispβ +Gispλ +GispTsp + [usp + eisp] 

where Yisp is an outcome for a student; Tsp is a binary variable equal to 1 for treatment group schools and 0 

for control schools; θp are program-specific fixed effects coefficients to be estimated; Xisp are student- or 

teacher-level baseline covariates that do not include the subgroup of interest; Gisp is the variable to represent 

the subgroup of interest; β, γo, γ1, λ, and  are fixed parameter vectors to be estimated; eisp are assumed to be 

independent and identically distributed N(0,σ2

e) student- or teacher-level random error terms; and usp are 

independent and identically distributed N(0,σ2

u) school-specific error terms that capture the correlation 

between the outcomes of students in the same schools and are assumed to be distributed independently of 

the eisp error terms. When subgroups are discrete, Gisp is a binary variable for the nonexcluded subgroup of 

interest (such as females, when males are the excluded subgroup).37 For the risk variables that were 

continuously defined, Gisp is a continuous variable. 

In this model,  is the coefficient on the interaction term between the subgroup of interest and the treatment 

group indicator variable. The value of  estimates the additional effect on the impact estimate of switching 

subgroups (through a one-unit change in the Gisp variable). A test of the hypothesis that  equals 0 is 

conducted to determine whether the subgroups have statistically significantly different impacts. 

This equation is the same for the subgroup analyses in which there are only two subgroups and for the 

subgroup analyses that use a continuous variable. In the case of three subgroups (such as with the three-tiered 

fidelity categorization), Gisp includes two subgroups (because the third one is excluded), and there are two 

subgroup-by-treatment-status interaction terms. The joint test of whether the coefficients on these terms 

equal 0 is conducted to determine whether the three subgroups have different impacts. For the discrete binary 

variables,  represents the additional effect of changing from boy to girl or from stayer to new entrant. For 

continuous variables,  represents the additional effect of a one-unit change in the Gisp variable. For 

presentation purposes, because one-unit changes in the risk measures might be hard to interpret because the 

risk measures are scales, the point estimates of the impacts are presented at the mean risk level, one standard 

deviation above the mean risk level, and one standard deviation below the mean risk level. 

The subgroup analyses determined if there was a differential impact of the SACD programs on the 

subgroups. In cases where such a difference was found, a secondary analysis was done to obtain more 

information on this difference by testing the statistical significance of the impact estimates for each subgroup 

separately. These supplemental analyses could help identify why a differential impact occurred. For example, 

if a significant differential impact was found that favored girls, the supplemental analysis might determine that 

the difference was due to either a beneficial impact on girls or a detrimental impact on boys.38 Impacts for 

 

impacts for boys versus girls) for each outcome and then to examine impacts on the individual subgroups (e.g., boys). 

The latter were to be examined only when the former were found significant. The alternative approach of applying 

multiple comparisons corrections for all the outcomes under each subgroup was not applied because it would have 

reduced the alpha levels for significance testing to such low values (e.g., from .05 to .005) that there would be little 

statistical power to detect impacts. 

37 For the subgroup analyses, (1) girls were included and boys excluded, (2) new entrants were included and stayers 

excluded, and (3) low fidelity was excluded. 

38 The student gains for some of the outcomes are represented by positive coefficients (an increase in that outcome is 

desired) while for others a gain is represented by a negative coefficient (a decrease is desired). To reduce confusion, 

impacts are described as beneficial or detrimental for a subgroup. A beneficial outcome means that a desired outcome 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

103

discrete (binary or categorical) subgroups were estimated using the equation (2) regression models (described 

in Year-by-Year Impacts on Students and Perceptions of School Climate) separately for each subgroup—for 

example, separately for boys and for girls. For the continuous initial risk variables, equation (2) was estimated 

three times for each risk variable: once for each level of risk reported (mean risk level, one standard deviation 

above the mean risk level, and one standard deviation below the mean risk level). 

It is possible to have a statistically significant difference in estimated impacts between subgroups and not to 

have statistically significant estimated impacts for one or all of the subgroups. In a similar way, there can be 

statistically significant estimated impacts for the subgroups, but no statistically significant difference between 

these impacts. These patterns can occur because the tests of statistical significance are asking slightly different 

questions. For each subgroup-specific regression, the hypothesis that is being tested is whether the impact 

coefficient is statistically significantly different from zero, when the coefficients for the regressor variables are 

fit specifically for that subgroup. For the test of the significance of the difference in the impact estimates, the 

hypothesis is testing whether the difference in the impact estimates for each subgroup is statistically 

significantly different from zero, when the coefficients for the regressor variables are fit for the full sample 

(all subgroups combined). 

Results 

For each subgroup, the results of the primary and supplemental subgroup tests are discussed. The test of the 

significance of the difference in the impact estimates for each subgroup is discussed first. This test directly 

compares the point estimates of the impacts for each level of the subgroup, while taking into account the 

variance around each point estimate. Formally, it is a test of the null hypothesis that the difference in impact 

between subgroup levels equals zero. The test establishes whether differences in impacts between levels of 

the subgrouping variable exist. Second, the results of the t tests from the subgroup-specific regressions are 

discussed. These tests determine the significance of the impact of the intervention on each of the subgroups. 

The findings from the subgroup-specific regressions are discussed only if the test of the significance of the 

difference in the impact estimates results rejects the hypothesis that the difference between the impacts for 

the subgroups is zero (although for completeness, the tables present all the subgroup-specific regression 

results). 

Tables 1.28 through 1.35 present the year-by-year results for the subgroup analyses. In the tables, the p-value 

for the difference in the impact estimates for each subgroup is labeled ―p-Value for Difference‖ for the 

discrete subgroups (gender, stayer/new entrant, and fidelity of implementation) and ―Marginal Effect‖ for the 

initial risk subgroups. The ―Marginal Effect‖ column contains both the p-value and the effect of moving from 

one level of risk up to the next level.39 The other columns give the coefficients converted into effect sizes and 

the p-values from the subgroup-specific regressions, and are labeled by their subgroup (e.g., ―Boys‖ and 

―Girls‖ in table 1.28). 

Gender 

There were 8 cases of significant differences between the impacts of the seven SACD programs on boys 

versus girls (table 1.28, last column under each year). From the 54 comparisons made, 3 significant 

differences would be expected by chance. When examining the results of the t tests from subgroup-specific 

regressions for these 8 cases, 1 case was based on a significant beneficial program impact for boys (an 

increase in Positive Social Behavior in Year 2); 3 were based on significant detrimental program impacts for 

boys (declines in Altruistic Behavior in Years 1 and 2, and Engagement with Learning in Year 2); 1 was based 

 

has increased or an undesired outcome has decreased for that subgroup. A detrimental outcome means that a desired 

outcome has decreased or an undesired outcome has increased for that subgroup. 

39 The marginal effects for the discrete binary subgroups (gender and stayer/new entrant) can be approximated by 

subtracting the effect sizes from the subgroup-specific regressions because the differences between subgroups in 

standard deviation estimates are small. For the fidelity subgroups, this approach can be used to identify pairwise 

differences between the impacts for the low-, mixed-, and high-fidelity subgroups. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

104 

on a significant beneficial impact for girls (a decline in Negative School Orientation in Year 2); and 1 was 

based on a significant detrimental impact for girls (a decline in Academic Competence in Year 1). Of the 

remaining 2, the first showed more beneficial impacts for girls (a gain in Normative Beliefs About Aggression 

in Year 2) while the second showed more beneficial impacts for boys (a decline in ADHD-related behavior in 

Year 3) but neither was linked to a significant program impact on either gender. Because there were few 

significant differences and they did not provide consistent results (e.g., half showed a detrimental program 

impact), these results provide little evidence that the SACD programs, as a group, differentially improved 

outcomes for one gender. 
Table 1.28. Combined-program impacts on child outcomes, by gender 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Boys Girls

p-value 

for 

difference1

Boys Girls

p-value 

for 

difference1

Boys Girls

p-value 

for 

difference1 Scale–Report Effect

size

pvalueEffect

size

pvalueEffect

size

pvalueEffect

size

pvalueEffect

size

pvalueEffect

size

pvalue

Social and Emotional 

 Competence Domain

 Self-Efficacy for Peer 

 Interactions–CR (+) 0.01 0.907 -0.05 0.345 0.558 -0.10^ 0.076 -0.04 0.479 0.281 -0.09^ 0.083 0.01 0.833 0.092^

 Normative Beliefs 

 About Aggression–

 CR (-) -0.01 0.842 -0.01 0.899 0.788 0.04 0.574 -0.10^ 0.095 0.023* -0.04 0.545 0.02 0.768 0.245

 Empathy–CR (+) 0.04 0.500 0.08 0.133 0.705 -0.05 0.401 0.02 0.685 0.165 -0.06 0.248 -0.03 0.661 0.841

Behavior Domain

 Altruistic Behavior–

 CR (+) -0.13* 0.014 0.01 0.921 0.023* -0.13* 0.020 0.00 0.967 0.022* -0.04 0.453 -0.03 0.580 0.784

 Altruistic Behavior–

 PCR (+) 0.07 0.199 0.09^ 0.081 0.754 -0.07 0.173 0.07 0.177 0.063^ -0.10 0.102 0.05 0.343 0.101

 Altruistic Behavior–

 TRS (+) 0.04 0.667 0.09 0.398 0.330 0.01 0.924 0.02 0.918 0.972 -0.01 0.940 -0.03 0.785 0.981

 Positive Social 

 Behavior–PCR (+) 0.00 0.934 -0.01 0.790 0.904 0.10* 0.032 -0.02 0.653 0.033* 0.04 0.512 0.00 0.985 0.826

 Positive Social 

 Behavior–TRS (+) 0.01 0.755 0.01 0.750 0.993 -0.02 0.830 -0.02 0.769 0.978 0.10 0.200 -0.01 0.864 0.058^

 Problem Behavior–

 CR (-) 0.01 0.897 0.00 0.993 0.927 -0.01 0.852 -0.01 0.900 0.810 0.03 0.675 0.02 0.735 0.990

 Problem Behavior–

 PCR (-) 0.02 0.630 -0.01 0.875 0.851 -0.08 0.138 0.03 0.599 0.089^ -0.03 0.697 -0.04 0.493 0.895

 Problem Behavior–

 TRS (-) 0.02 0.715 0.03 0.577 0.865 -0.02 0.720 -0.03 0.601 0.788 -0.08 0.346 -0.03 0.569 0.621

 ADHD-Related 

 Behavior–TRS (-) -0.03 0.518 0.02 0.619 0.254 -0.06 0.346 -0.02 0.745 0.529 -0.15^ 0.064 -0.02 0.736 0.043*

See notes at end of table.

105

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.28. Combined-program impacts on child outcomes, by gender—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Boys Girls

p-value 

for 

difference1

Boys Girls

p-value 

for 

difference1

Boys Girls

p-value 

for 

difference1 Scale–Report Effect

size

pvalueEffect

size

pvalueEffect

size

pvalueEffect

size

pvalueEffect

size

pvalueEffect

size

pvalue

Academics Domain

 Engagement with 

 Learning–CR (+) -0.01 0.872 -0.08 0.133 0.178 -0.10* 0.050 0.05 0.297 0.011* 0.00 0.998 -0.09 0.100 0.317

 Academic 

 Competence and

 Motivation–TRS (+) 0.04 0.331 -0.07* 0.017 0.003* 0.02 0.684 -0.04 0.424 0.242 0.05 0.386 -0.05 0.443 0.093^

Perceptions of School 

 Climate Domain

 Positive School 

 Orientation–CR (+) 0.04 0.529 0.02 0.728 0.593 -0.01 0.872 0.06 0.393 0.081^ -0.09 0.291 -0.04 0.617 0.345

 Negative School 

 Orientation–CR (-) 0.01 0.815 -0.10* 0.040 0.065^ 0.09 0.126 -0.14* 0.011 0.000* -0.01 0.914 0.00 0.943 0.842

 Student Afraid at 

 School–CR (-) -0.02 0.654 -0.08 0.102 0.299 -0.10 0.102 -0.02 0.716 0.214 0.01 0.818 -0.02 0.666 0.445

 Victimization at 

 School–CR (-) -0.02 0.748 -0.01 0.817 0.983 -0.07 0.223 -0.01 0.876 0.262 -0.01 0.850 0.01 0.809 0.683

* Treatment group significantly different from control group at the .05 level.

^ Significantly different from zero at the .10 to > .05 level.

1

From the test of a significant difference between Boys’ effect size and Girls’ effect size.

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models in which each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. The effect size 

was calculated by dividing the estimated impact by the standard deviation of the outcome measure for the control group. 

SOURCE: The Social and Character Development (SACD) Research Program.

106

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

107

Stayer Versus New Entrant 

For Years 2 and 3, no significant differences between the impacts on stayers versus new entrants were found 

(table 1.29). From the 36 comparisons made, 2 outcomes would have been expected to be statistically 

significant by chance. These results provide little evidence that the SACD programs, as a group, differentially 

improved outcomes for stayers or for new entrants. 

The results comparing new entrants and stayers subgroups might have been sensitive to the methods used to 

fill in missing values in the initial data for new entrants. This could be a concern, especially because timevarying characteristics were measured at the start of the evaluation for stayers, but no data were available for 

new entrants. Information collected during the follow-up periods was used to fill in missing values for initial 

data items that were not expected to vary much over time. For characteristics that were likely to vary 

considerably over time, including initial values of the outcome measures, values were imputed based on cell 

means, where cells were defined by students’ schools, gender, and race/ethnicity. To check the sensitivity of 

these results to the imputation of covariates that might vary over time, results were estimated for the stayer 

and new entrant subgroups using only covariates that were unlikely to vary over time. The subgroup results 

were similar (these data are not shown in a table). 
Table 1.29. Combined-program impacts on child outcomes, by stayer versus new entrant 

Year 2 Year 3

(Spring 4th grade) (Spring 5th grade)

Stayers New entrants p-value

for 

difference1

Stayers New entrants p-value

for 

difference1 Scale–Report 

Effect

size p-value

Effect

size p-value

Effect

size p-value

Effect

size p-value

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) -0.06 0.176 -0.14 0.117 0.319 -0.04 0.326 -0.10 0.184 0.723

 Normative Beliefs About Aggression–CR (-) 0.00 0.988 -0.11 0.136 0.140 0.00 0.957 -0.05 0.491 0.542

 Empathy–CR (+) -0.01 0.798 -0.04 0.566 0.785 -0.05 0.353 0.01 0.847 0.794

Behavior Domain

 Altruistic Behavior–CR (+) -0.03 0.516 -0.14^ 0.066 0.132 -0.03 0.496 0.02 0.832 0.966

 Altruistic Behavior–PCR (+) -0.01 0.866 0.02 0.820 0.618 -0.04 0.443 0.01 0.950 0.246

 Altruistic Behavior–TRS (+) 0.02 0.903 -0.01 0.971 0.793 -0.02 0.868 -0.05 0.676 0.420

 Positive Social Behavior–PCR (+) 0.01 0.813 0.13^ 0.093 0.067^ 0.02 0.691 0.02 0.825 0.444

 Positive Social Behavior–TRS (+) -0.02 0.818 -0.06 0.462 0.680 0.05 0.460 -0.03 0.679 0.361

 Problem Behavior–CR (-) -0.01 0.871 0.02 0.827 0.473 0.01 0.852 0.02 0.747 0.380

 Problem Behavior–PCR (-) -0.05 0.244 -0.11 0.160 0.195 -0.05 0.426 -0.04 0.610 0.634

 Problem Behavior–TRS (-) -0.04 0.486 0.04 0.648 0.370 -0.07 0.293 0.02 0.826 0.353

 ADHD-Related Behavior–TRS (-) -0.03 0.590 -0.01 0.880 0.913 -0.09 0.121 0.01 0.873 0.333

Academics Domain

 Engagement with Learning–CR (+) -0.03 0.533 -0.06 0.396 0.773 -0.05 0.331 0.01 0.876 0.443

 Academic Competence and Motivation–TRS (+) -0.07 0.170 -0.05 0.494 0.645 -0.02 0.739 -0.07 0.363 0.275

See notes at end of table.

108

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.29. Combined-program impacts on child outcomes, by stayer versus new entrant—Continued

Year 2 Year 3

(Spring 4th grade) (Spring 5th grade)

Stayers New entrants

p-value for 

difference1

Stayers New entrants

p-value for 

difference1 Scale–Report 

Effect

size p-value

Effect

size p-value

Effect

size p-value

Effect

size p-value

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 0.05 0.561 0.05 0.595 0.225 -0.05 0.526 -0.07 0.430 0.559

 Negative School Orientation–CR (-) -0.05 0.432 0.08 0.280 0.053^ -0.04 0.493 0.05 0.524 0.230

 Student Afraid at School–CR (-) -0.05 0.418 -0.01 0.889 0.676 0.02 0.717 -0.02 0.824 0.329

 Victimization at School–CR (-) -0.05 0.353 -0.04 0.626 0.622 0.01 0.845 0.02 0.830 0.663

^ Significantly different from zero at the .10 to > .05 level.

1

From the test of a significant difference between Stayers’ effect size and New Entrants’ effect size.

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The - signs in parentheses indicate the direction of a beneficial outcome. No findings were found statistically significant at or below the .05 level. All impact estimates were calculated 

using regression models in which each program and school within a program was weighted equally. The standard errors of all estimates account for design effects due to unequal 

weighting and the clustering of students within schools. The effect size was calculated by dividing the estimated impact by the standard deviation of the outcome measure for the 

control group. SOURCE: The Social and Character Development (SACD) Research Program.

109

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

110

Initial Risk 

The impacts of the seven SACD programs, considered together, on initial student risk were examined using 

five risk measures (one measure each for socioeconomic risk, family risk, and perceptions of community risk, 

and two measures for child behavior risk). The significance of the change in the impact estimate over a oneunit change (moving one level between the three risk levels) in the risk measure (termed the marginal effect) was 

tested (see tables 1.30 to 1.34). A total of 270 marginal effects were tested over the 3 years (90 per year), and 

13 to 14 would be expected to be significant by chance. A total of 41 were found significant—13 from Year 

1, 17 from Year 2, and 11 from Year 3. Of these 41 significant findings, 26 showed a more beneficial 

marginal impact for high-risk students and 15 showed a more detrimental impact for high-risk students 

compared to lower risk students. Six of these findings were repeated in at least 2 different years, with 1 of 

these repeated all 3 years (4 showing a more beneficial marginal effect for high-risk students and 2 showing a 

detrimental impact). 

For 24 of the 41 significant marginal effects, there were no significant findings from the subgroup-specific 

regressions. In 12 of the 41 significant marginal effects, the subgroup-specific regressions supported the 

interpretation of the marginal effect (6 results supported a more beneficial impact for high-risk students and 6 

results supported a more detrimental impact for high-risk students). In the remaining 5 significant marginal 

effects, the subgroup-specific regressions supported a different interpretation: (1) a detrimental effect on 

lower risk students rather than a beneficial impact on higher risk students in two cases, and (2) a beneficial 

effect on lower risk students rather than a detrimental impact on higher risk students in three cases. 

The low number, mixed direction (beneficial and detrimental), and lack of repeated findings of the significant 

results provide little evidence of a relationship between risk levels and the impact of the seven SACD 

programs considered together. Discussions of the results for each type of risk follow. 

Socioeconomic Risk 

For initial socioeconomic risk, there were 5 significant marginal effects, all showing detrimental impacts on 

high-risk students. From the 54 comparisons made, 3 significant differences would be expected by chance. 

Table 1.30 presents the impacts of the seven programs together on the outcome measures by level of student 

socioeconomic risk. An example may help in the interpretation of the table. For the outcome Engagement 

with Learning under the Academics domain, in Year 1 the estimated marginal effect of -0.131 indicates that, 

as the risk level increased one unit, the estimated impact of the seven SACD programs together decreased by 

0.131. This estimate of the marginal effect was statistically significantly different from zero, providing 

evidence that the estimated impact of the seven SACD programs, as a group, on this outcome varied across 

socioeconomic risk levels (in this case it was detrimental as Engagement with Learning declined as risk 

increased). The results from the subgroup-specific regressions are provided at three levels of initial risk: 

(1) low, which is one standard deviation below the mean socioeconomic risk level; (2) average, which is the 

mean risk level; and (3) high, which is one standard deviation above the mean risk level. In Year 1, at the low 

level of socioeconomic risk, the estimated impact was 0.041 and nonsignificant. As the level of 

socioeconomic risk increased to the mean level, with all else held equal, the point estimate for the impact 

estimate changed to a nonsignificant –0.049. As the level of socioeconomic risk continued to increase to one 

standard deviation above the mean level, all else held equal, the estimated impact changed to -0.154. The 

point estimate for the impact at the high level of risk was statistically significantly different from zero, 

providing additional evidence that the seven SACD programs, together, had a detrimental impact on 

Engagement with Learning for students at high socioeconomic risk. However, as these results were not 

repeated for Years 2 and 3, no clear pattern of impact on this outcome emerged. 

In addition to a reduction in Engagement with Learning, the high-risk students experienced a detrimental 

program impact on four other outcomes: declines in Altruistic Behavior (Teacher Report in Year 1) and 

Positive School Orientation (Year 1) and gains in Negative School Orientation (Year 1) and Student Afraid at 

School (Year 3). None of the significant marginal effects occurred for more than 1 year. 
Table 1.30. Combined-program impacts on child outcomes, by initial socioeconomic risk 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

(p-value) (p-value) (p-value)

Scale–Report Low1

Average2 High3

Low1

Average2 High3

Low1

Average2 High3

Social and Emotional 

 Competence Domain

 Self-Efficacy for Peer -0.005 -0.027 -0.052 -0.032 -0.075 -0.077^ -0.079 -0.003 -0.065 -0.042 -0.015 0.038

 Interactions–CR (+) (0.922) (0.490) (0.298) (0.433) (0.165) (0.089) (0.161) (0.955) (0.190) (0.284) (0.779) (0.439)

 Normative Beliefs -0.024 0.001 0.026 0.033 -0.002 -0.009 -0.018 -0.012 0.047 0.032 0.015 -0.024

 About Aggression–CR (-) (0.655) (0.982) (0.642) (0.449) (0.976) (0.881) (0.801) (0.810) (0.479) (0.576) (0.822) (0.637)

 Empathy–CR (+) 0.124* 0.077 -0.022 -0.069 -0.009 -0.027 -0.048 -0.031 -0.019 -0.068 -0.126^ -0.081

(0.038) (0.129) (0.713) (0.110) (0.891) (0.632) (0.457) (0.533) (0.777) (0.255) (0.073) (0.122)

Behavior Domain

 Altruistic Behavior– -0.012 -0.054 -0.102* -0.060 -0.081 -0.067 -0.051 0.024 -0.073 -0.031 0.018 0.069

 CR (+) (0.800) (0.169) (0.044) (0.138) (0.139) (0.143) (0.375) (0.628) (0.171) (0.470) (0.744) (0.168)

 Altruistic Behavior– 0.047 0.032 0.014 -0.023 -0.008 0.006 0.022 0.023 0.010 -0.013 -0.040 -0.038

 PCR (+) (0.322) (0.396) (0.781) (0.590) (0.884) (0.900) (0.697) (0.643) (0.882) (0.811) (0.549) (0.495)

 Altruistic Behavior– 0.164 0.085 -0.006 -0.115* 0.035 0.014 -0.012 -0.037 -0.030 -0.052 -0.078 -0.037

 TRS (+) (0.102) (0.365) (0.950) (0.011) (0.827) (0.931) (0.940) (0.492) (0.808) (0.661) (0.526) (0.457)

 Positive Social -0.012 -0.026 -0.041 -0.020 0.025 0.042 0.062 0.029 0.027 0.036 0.045 0.014

 Behavior–PCR (+) (0.751) (0.394) (0.298) (0.555) (0.603) (0.285) (0.207) (0.509) (0.619) (0.429) (0.425) (0.786)

 Positive Social 0.020 -0.002 -0.027 -0.031 0.002 -0.032 -0.072 -0.058 0.058 0.009 -0.050 -0.082^

 Behavior–TRS (+) (0.693) (0.968) (0.594) (0.315) (0.979) (0.670) (0.370) (0.212) (0.459) (0.906) (0.530) (0.089)

 Problem Behavior– -0.029 -0.001 0.031 0.040 0.000 -0.002 -0.006 -0.005 0.028 0.073 0.124 0.073

 CR (-) (0.620) (0.983) (0.597) (0.318) (0.997) (0.971) (0.940) (0.925) (0.708) (0.293) (0.108) (0.151)

 Problem Behavior– -0.015 0.010 0.039 0.037 0.005 -0.013 -0.034 -0.030 -0.070 -0.034 0.009 0.061

 PCR (-) (0.752) (0.801) (0.423) (0.327) (0.910) (0.743) (0.493) (0.509) (0.230) (0.478) (0.877) (0.267)

 Problem Behavior– -0.009 0.052 0.101^ 0.062^ -0.004 -0.006 -0.009 -0.004 -0.043 -0.034 -0.024 0.014

 TRS (-) (0.871) (0.293) (0.066) (0.056) (0.950) (0.914) (0.891) (0.931) (0.572) (0.621) (0.752) (0.768)

 ADHD-Related 0.022 0.020 0.018 -0.003 -0.004 -0.019 -0.038 -0.026 -0.053 -0.067 -0.084 -0.024

 Behavior–TRS (-) (0.645) (0.635) (0.716) (0.923) (0.953) (0.724) (0.546) (0.565) (0.459) (0.297) (0.243) (0.615)

See notes at end of table.

111

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.30. Combined-program impacts on child outcomes, by initial socioeconomic risk—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Effect size at risk level

Marginal 

effect4

(p-value)

Effect size at risk level

Marginal 

effect4

(p-value)

Effect size at risk level

Marginal 

effect4

(p-value)

(p-value) (p-value) (p-value)

Scale–Report Low1

Average2 High3

Low1

Average2 High3

Low1

Average2 High3

Academics Domain

 Engagement with 0.041 -0.049 -0.154** -0.131** -0.037 -0.030 -0.021 0.012 -0.041 -0.046 -0.051 -0.008

 Learning–CR (+) (0.394) (0.202) (0.003) (0.003) (0.421) (0.412) (0.668) (0.796) (0.460) (0.318) (0.377) (0.882)

 Academic Competence and -0.013 -0.002 -0.011 0.017 -0.014 -0.016 -0.018 -0.003 0.029 0.019 0.008 -0.016

 Motivation–TRS (+) (0.688) (0.941) (0.743) (0.532) (0.760) (0.687) (0.703) (0.936) (0.622) (0.713) (0.899) (0.707)

Perceptions of School 

 Climate Domain

 Positive School 0.092 0.011 -0.083 -0.118** 0.058 0.028 -0.008 -0.051 -0.049 -0.090 -0.140 -0.070

 Orientation–CR (+) (0.237) (0.876) (0.286) (0.009) (0.509) (0.732) (0.930) (0.315) (0.609) (0.311) (0.146) (0.178)

 Negative School -0.103^ -0.045 0.023 0.085* -0.019 -0.013 -0.007 0.009 -0.024 0.000 0.029 0.040

 Orientation–CR (-) (0.057) (0.319) (0.672) (0.043) (0.783) (0.827) (0.922) (0.852) (0.745) (0.996) (0.699) (0.427)

 Student Afraid at -0.088 -0.058 -0.024 0.043 -0.043 -0.049 -0.056 -0.010 -0.081 -0.019 0.054 0.103*

 School–CR (-) (0.139) (0.242) (0.688) (0.331) (0.522) (0.410) (0.417) (0.849) (0.224) (0.743) (0.427) (0.047)

 Victimization at -0.027 -0.009 0.012 0.026 -0.029 -0.048 -0.071 -0.033 0.012 0.014 0.017 0.004

 School–CR (-) (0.600) (0.835) (0.814) (0.528) (0.603) (0.303) (0.216) (0.503) (0.838) (0.771) (0.777) (0.938)

* Significantly different from zero at the .05 level.

** Significantly different from zero at the .01 level.

^ Significantly different from zero at the .10 to > .05 level.

1 One standard deviation below the mean risk level.

2 At the mean risk level.

3 One standard deviation above the mean risk level.

4 Change in impact as risk level increases by one unit.

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models in which each program and school within 

a program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. The effect 

size was calculated by dividing the estimated impact by the standard deviation of the outcome measure for the control group. 

SOURCE: The Social and Character Development (SACD) Research Program.

112

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

113

Family Risk 

Over the 3 years, there were 6 statistically significant marginal effects regarding level of family risk (table 

1.31). From the 54 comparisons made, 3 significant differences would be expected by chance. Five of the 

significant results showed beneficial impacts for high-risk students: increases in Empathy (Year 1), 

Engagement with Learning (Year 3), and Academic Competence (Year 3), and declines in Problem Behavior 

(Child Report in Year 3) and Negative School Orientation (Year 3). The results from the subgroup-specific 

regressions provided support for the Empathy finding but offered an alternative explanation for the 

Engagement with Learning finding, as they suggested that the significant marginal effect was due to a 

detrimental effect on low-risk students rather than a beneficial impact on high-risk students. 

The sixth significant marginal effect showed a detrimental impact on high-risk students of an increase in 

Student Afraid at School (Year 2). The subgroup-specific regression results suggested that this significant 

marginal effect was due to a beneficial effect on low-risk students rather than a detrimental impact on highrisk students. 
Table 1.31. Combined-program impacts on child outcomes, by initial family risk 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

(p-value) (p-value) (p-value)

Scale–Report Low1

Average2 High3

Low1

Average2 High3

Low1

Average2 High3

Social and Emotional 

 Competence Domain

 Self-Efficacy for Peer 0.007 -0.031 -0.068 -0.169 -0.063 -0.052 -0.041 0.049 -0.059 -0.020 0.019 0.175

 Interactions–CR (+) (0.890) (0.434) (0.170) (0.215) (0.278) (0.261) (0.470) (0.751) (0.309) (0.641) (0.749) (0.313)

 Normative Beliefs 0.036 -0.008 -0.052 -0.198 0.034 -0.003 -0.040 -0.165 0.072 -0.002 -0.076 -0.333^

 About Aggression–CR (-) (0.532) (0.859) (0.359) (0.178) (0.690) (0.969) (0.635) (0.301) (0.396) (0.974) (0.362) (0.056)

 Empathy–CR (+) 0.000 0.073 0.146* 0.329* -0.056 -0.014 0.028 0.191 -0.084 -0.066 -0.049 0.079

(0.996) (0.171) (0.019) (0.018) (0.400) (0.807) (0.667) (0.213) (0.277) (0.318) (0.523) (0.650)

Behavior Domain

 Altruistic Behavior–CR (+) -0.124* -0.067 -0.010 0.258^ -0.100 -0.039 0.022 0.275^ -0.068 -0.032 0.003 0.160

(0.016) (0.104) (0.847) (0.057) (0.109) (0.446) (0.714) (0.081) (0.299) (0.532) (0.960) (0.367)

 Altruistic Behavior–PCR (+) 0.036 0.033 0.029 -0.017 0.021 0.000 -0.022 -0.097 -0.029 -0.041 -0.053 -0.053

(0.476) (0.416) (0.565) (0.905) (0.720) (0.996) (0.707) (0.532) (0.694) (0.505) (0.469) (0.767)

 Altruistic Behavior–TRS (+) 0.095 0.080 0.064 -0.069 0.016 0.026 0.036 0.045 -0.093 -0.022 0.049 0.321^

(0.353) (0.414) (0.527) (0.609) (0.925) (0.878) (0.836) (0.783) (0.480) (0.863) (0.706) (0.053)

 Positive Social -0.053 -0.025 0.003 0.125 -0.054 -0.005 0.043 0.218^ -0.041 0.010 0.060 0.226

 Behavior–PCR (+) (0.199) (0.439) (0.940) (0.267) (0.279) (0.891) (0.373) (0.094) (0.542) (0.865) (0.364) (0.146)

 Positive Social -0.043 -0.005 0.033 0.171^ -0.064 -0.037 -0.009 0.125 -0.012 0.030 0.072 0.190

 Behavior–TRS (+) (0.407) (0.920) (0.515) (0.077) (0.460) (0.653) (0.917) (0.354) (0.887) (0.712) (0.409) (0.209)

 Problem Behavior–CR (-) 0.051 0.004 -0.060 -0.251^ 0.018 -0.025 -0.069 -0.196 0.141 0.034 -0.073 -0.483**

(0.392) (0.933) (0.314) (0.053) (0.827) (0.737) (0.402) (0.182) (0.106) (0.666) (0.398) (0.004)

 Problem Behavior–PCR (-) -0.007 0.012 0.032 0.089 0.002 0.033 0.065 0.142 0.013 -0.019 -0.052 -0.147

(0.871) (0.740) (0.481) (0.459) (0.974) (0.397) (0.187) (0.299) (0.842) (0.720) (0.426) (0.377)

 Problem Behavior–TRS (-) 0.090 0.053 -0.016 -0.168^ 0.031 -0.013 -0.057 -0.198 0.026 -0.033 -0.093 -0.267^

(0.108) (0.301) (0.780) (0.097) (0.677) (0.854) (0.448) (0.142) (0.770) (0.685) (0.297) (0.082)

 ADHD-Related 0.026 0.016 -0.060 -0.045 -0.007 -0.029 -0.051 -0.100 -0.079 -0.091 -0.103 -0.053

 Behavior–TRS (-) (0.593) (0.714) (0.907) (0.648) (0.919) (0.636) (0.452) (0.460) (0.301) (0.184) (0.177) (0.730)

See notes at end of table.

114

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.31. Combined-program impacts on child outcomes, by initial family risk—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

(p-value) (p-value) (p-value)

Scale–Report Low1

Average2 High3

Low1

Average2 High3

Low1

Average2 High3

Academics Domain

 Engagement with -0.090^ -0.056 -0.022 0.155 -0.081 -0.013 0.054 0.304^ -0.152* -0.067 0.018 0.382*

 Learning–CR (+) (0.086) (0.170) (0.676) (0.294) (0.159) (0.765) (0.338) (0.060) (0.025) (0.206) (0.790) (0.040)

 Academic Competence -0.013 -0.004 0.020 0.075 -0.043 -0.011 0.020 0.140 -0.069 0.003 0.075 0.324*

 and Motivation–TRS (+) (0.694) (0.893) (0.540) (0.386) (0.387) (0.784) (0.686) (0.218) (0.291) (0.953) (0.243) (0.015)

 

Perceptions of School 

 Climate Domain

 Positive School 0.007 0.006 0.006 -0.003 0.002 0.040 0.079 0.173 -0.115 -0.085 -0.054 0.136

 Orientation–CR (+) (0.931) (0.931) (0.941) (0.985) (0.983) (0.647) (0.402) (0.256) (0.245) (0.353) (0.579) (0.414)

 Negative School -0.065 -0.044 -0.023 0.096 0.000 -0.051 -0.101 -0.226 0.052 -0.051 -0.153^ -0.461**

 Orientation–CR (-) (0.272) (0.389) (0.699) (0.478) (0.997) (0.489) (0.211) (0.140) (0.505) (0.461) (0.051) (0.006)

 Student Afraid at -0.096 -0.058 -0.020 0.170 -0.175* -0.084 0.006 0.408* -0.001 0.003 0.007 0.018

 School–CR (-) (0.113) (0.257) (0.735) (0.228) (0.020) (0.192) (0.932) (0.012) (0.989) (0.968) (0.933) (0.918)

 Victimization at -0.055 -0.004 0.048 0.232^ -0.030 -0.042 -0.054 -0.054 0.036 0.008 -0.021 -0.128

 School–CR (-) (0.298) (0.935) (0.361) (0.090) (0.622) (0.396) (0.367) (0.727) (0.604) (0.894) (0.766) (0.451)

* Significantly different from zero at the .05 level.

** Significantly different from zero at the .01 level.

^ Significantly different from zero at the .10 to > .05 level.

1 One standard deviation below the mean risk level.

2 At the mean risk level.

3 One standard deviation above the mean risk level.

4 Change in impact as risk level increases by one unit.

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models in which each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. The effect size 

was calculated by dividing the estimated impact by the standard deviation of the outcome measure for the control group. 

SOURCE: The Social and Character Development (SACD) Research Program.

115

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

116

Perceptions of Community Risk. 

Over the 3 years, there were 10 significant marginal effects regarding level of perception of community risk 

(table 1.32). From the 54 comparisons made, 3 significant differences would be expected by chance. Six of 

the significant results showed beneficial marginal impacts for high-risk students: increases in Positive School 

Orientation (Year 2) and declines in Problem Behavior (Teacher Report in Year 2), ADHD-related behavior 

(Year 2), and Victimization at School (all 3 years). Only 1 of these 6 was supported by the subgroup-specific 

regressions with the finding of a beneficial impact of the decline in Victimization at School (Year 2) for highrisk students. 

The 4 detrimental marginal impacts for high-risk students included declines in Empathy (Year 3) and 

Altruistic Behavior (Primary Caregiver Report in Year 3), and increases in Problem Behavior (Primary 

Caregiver Report in Years 1 and 2). In all four cases, the subgroup-specific regressions supported the finding 

of detrimental impact on all 4 outcomes for high-risk students. 
Table 1.32. Combined-program impacts on child outcomes, by initial perceptions of community risk 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

(p-value) (p-value) (p-value)

Scale–Report Low1

Average2 High3

Low1

Average2 High3

Low1

Average2 High3

Social and Emotional 

 Competence Domain

 Self-Efficacy for Peer -0.025 -0.029 -0.032 -0.005 -0.121^ -0.064 -0.008 0.085 -0.061 -0.024 0.014 0.056

 Interactions–CR (+) (0.642) (0.491) (0.550) (0.923) (0.067) (0.204) (0.905) (0.159) (0.323) (0.596) (0.832) (0.395)

 Normative Beliefs About 0.033 -0.003 -0.040 -0.055 0.005 -0.008 -0.021 0.019 -0.035 -0.012 0.011 0.035

 Aggression–CR (-) (0.582) (0.948) (0.504) (0.328) (0.959) (0.914) (0.807) (0.768) (0.699) (0.873) (0.901) (0.632)

 Empathy–CR (+) 0.087 0.073 0.059 -0.021 0.013 -0.005 -0.022 -0.026 0.080 -0.054 -0.187* -0.200 **

(0.171) (0.153) (0.337) (0.698) (0.860) (0.934) (0.747) (0.671) (0.341) (0.423) (0.023) (0.006)

Behavior Domain

 Altruistic Behavior– -0.035 -0.062 -0.089^ -0.041 -0.060 -0.031 -0.001 0.045 -0.017 -0.023 -0.030 -0.010

 CR (+) (0.523) (0.136) (0.095) (0.422) (0.356) (0.539) (0.990) (0.460) (0.808) (0.647) (0.661) (0.881)

 Altruistic Behavior– 0.030 0.036 0.042 0.008 0.079 0.004 -0.070 -0.112^ 0.129^ -0.024 -0.176* -0.228**

 PCR (+) (0.538) (0.315) (0.386) (0.864) (0.205) (0.927) (0.238) (0.060) (0.082) (0.670) (0.015) (0.001)

 Altruistic Behavior– 0.159 0.096 0.033 -0.094 -0.009 0.027 0.063 0.054 -0.004 -0.013 -0.023 -0.014

 TRS (+) (0.141) (0.333) (0.753) (0.104) (0.961) (0.875) (0.723) (0.451) (0.978) (0.919) (0.871) (0.853)

 Positive Social 0.001 -0.014 -0.030 -0.023 0.044 0.016 -0.012 -0.042 0.088 0.013 -0.063 -0.112^

 Behavior–PCR (+) (0.977) (0.657) (0.476) (0.576) (0.407) (0.693) (0.805) (0.392) (0.220) (0.822) (0.359) (0.076)

 Positive Social 0.002 -0.004 -0.011 -0.010 -0.074 -0.045 -0.015 0.044 0.096 0.038 -0.020 -0.087

 Behavior–TRS (+) (0.966) (0.926) (0.834) (0.800) (0.405) (0.569) (0.859) (0.441) (0.295) (0.631) (0.819) (0.187)

 Problem Behavior– 0.022 -0.008 -0.037 -0.044 -0.022 -0.027 -0.032 -0.008 -0.002 0.032 0.067 0.052

 CR (-) (0.732) (0.880) (0.543) (0.391) (0.798) (0.715) (0.698) (0.900) (0.982) (0.680) (0.457) (0.457)

 Problem Behavior– -0.100^ 0.005 0.109* 0.157*** -0.045 0.028 0.101* 0.109* -0.005 0.000 0.006 0.009

 PCR (-) (0.053) (0.909) (0.028) (0.001) (0.381) (0.446) (0.041) (0.032) (0.940) (0.994) (0.929) (0.896)

 Problem Behavior– 0.044 0.054 0.063 0.014 0.087 -0.001 -0.089 -0.132* -0.087 -0.047 -0.007 0.059

 TRS (-) (0.464) (0.304) (0.278) (0.732) (0.268) (0.994) (0.243) (0.022) (0.370) (0.577) (0.939) (0.374)

 ADHD-Related Behavior– -0.016 0.022 0.028 0.008 0.062 -0.021 -0.104 -0.124* -0.073 -0.102 -0.132 -0.045

 TRS (-) (0.748) (0.605) (0.576) (0.835) (0.389) (0.722) (0.132) (0.029) (0.375) (0.134) (0.100) (0.494)

See notes at end of table.

117

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.32. Combined-program impacts on child outcomes, by initial perceptions of community risk—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

(p-value) (p-value) (p-value)

Scale–Report Low1

Average2 High3

Low1

Average2 High3

Low1

Average2 High3

Academics Domain

 Engagement with -0.016 -0.054 -0.091^ -0.056 -0.020 -0.003 0.014 0.026 -0.052 -0.053 -0.055 -0.002

 Learning–CR (+) (0.766) (0.187) (0.095) (0.302) (0.728) (0.943) (0.804) (0.657) (0.463) (0.313) (0.439) (0.972)

 Academic Competence 0.017 -0.002 -0.014 -0.023 0.036 -0.015 -0.066 -0.077^ 0.058 0.017 -0.024 -0.061

 and Motivation–TRS (+) (0.640) (0.956) (0.695) (0.489) (0.483) (0.719) (0.188) (0.092) (0.396) (0.762) (0.720) (0.275)

Perceptions of School

 Climate Domain

 Positive School 0.011 0.013 0.015 -0.003 -0.086 0.030 0.146 0.174** -0.091 -0.094 -0.098 -0.005

 Orientation–CR (+) (0.895) (0.855) (0.850) (0.957) (0.361) (0.719) (0.111) (0.007) (0.378) (0.293) (0.331) (0.945)

 Negative School -0.047 -0.040 -0.034 0.010 -0.079 -0.055 -0.031 0.036 -0.063 -0.052 -0.041 0.017

 Orientation–CR (-) (0.435) (0.395) (0.561) (0.855) (0.342) (0.432) (0.700) (0.561) (0.435) (0.429) (0.607) (0.805)

 Student Afraid at -0.062 -0.061 -0.060 -0.002 -0.017 -0.078 -0.139^ -0.091 0.003 -0.010 -0.023 -0.020

 School–CR (-) (0.331) (0.233) (0.334) (0.976) (0.836) (0.240) (0.073) (0.157) (0.973) (0.881) (0.777) (0.783)

 Victimization at 0.094^ -0.001 -0.093^ -0.139** 0.068 -0.035 -0.138* -0.154* 0.132^ 0.021 -0.090 -0.166*

 School–CR (-) (0.093) (0.988) (0.091) (0.008) (0.298) (0.485) (0.030) (0.011) (0.066) (0.702) (0.205) (0.016)

* Significantly different from zero at the .05 level.

** Significantly different from zero at the .01 level.

*** Significantly different from zero at the .001 level.

^ Significantly different from zero at the .10 to > .05 level.

1 One standard deviation below the mean risk level.

2 At the mean risk level.

3 One standard deviation above the mean risk level.

4 Change in impact as risk level increases by one unit.

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The - signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models in which each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. The effect size 

was calculated by dividing the estimated impact by the standard deviation of the outcome measure for the control group. 

SOURCE: The Social and Character Development (SACD) Research Program.

118

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

119

Child Behavior Risk (Teacher and Primary Caregiver Reports) 

Over the 3 years, the Teacher Report showed 10 significant marginal effects regarding level of child behavior 

risk (table 1.33). From the 54 comparisons made, 3 significant differences would be expected by chance. 

Seven of the significant results showed beneficial marginal impacts for high-risk students: increases in 

Altruistic Behavior (Primary Caregiver Report in Year 1) and Academic Competence (Years 2 and 3), and 

declines in Normative Beliefs About Aggression (Year 2), Problem Behavior (Child Report and Teacher 

Report in Year 2), and Negative School Orientation (Year 2). Results from the subgroup-specific regressions 

supported the beneficial impacts for higher risk students for Altruistic Behavior and Negative School 

Orientation, and also provided evidence that the marginal effect on Academic Competence (Year 2) was due 

to a detrimental impact on low-risk students. 

The 3 detrimental marginal impacts for high-risk students included declines in Engagement with Learning 

(Year 1) and increases in Problem Behavior (Primary Caregiver Report in Year 3) and Student Afraid at 

School (Year 1). Results from the subgroup-specific regressions supported the interpretation that there were 

detrimental impacts on Engagement with Learning for high-risk students but that the marginal impacts on 

Problem Behavior and Student Afraid at School were due to beneficial impacts on low-risk students. 

When child behavior risk was measured using the Primary Caregiver Report, there were 10 significant 

marginal effects in contrast to the 3 expected by chance from the 54 comparisons (table 1.34). Eight of these 

showed beneficial marginal impacts for high-risk students: increases in Empathy (Years 1 and 2), Altruistic 

Behavior (Teacher Report in Year 2), Positive Social Behavior (Primary Caregiver Report in Year 2), and 

Engagement with Learning (Year 2), and declines in Normative Beliefs About Aggression (Year 2) and 

Negative School Orientation (Years 1 and 2). Results from the subgroup-specific regressions supported the 

interpretation that there were beneficial impacts on Empathy (Year 1) and Negative School Orientation (Year 

2) for high-risk students. The 2 detrimental marginal impacts for high-risk students were found for Problem 

Behavior (Primary Caregiver Report in Years 1 and 3). 
Table 1.33. Combined-program impacts on child outcomes, by initial child behavior risk as reported by teacher 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

(p-value) (p-value) (p-value)

Scale–Report Low1

Average2 High3

Low1

Average2 High3

Low1

Average2 High3

Social and Emotional 

 Competence Domain

 Self-Efficacy for Peer 0.006 -0.037 -0.080 -0.004 -0.040 -0.059 -0.077 -0.002 -0.028 -0.041 -0.053 -0.001

 Interactions–CR (+) (0.904) (0.336) (0.109) (0.169) (0.510) (0.230) (0.228) (0.640) (0.653) (0.397) (0.424) (0.771)

 Normative Beliefs About -0.010 -0.015 -0.019 0.000 0.127 0.007 -0.113 -0.012** 0.014 0.021 0.028 0.001

 Aggression–CR (-) (0.851) (0.742) (0.734) (0.897) (0.134) (0.923) (0.191) (0.005) (0.867) (0.770) (0.741) (0.870)

 Empathy–CR (+) 0.035 0.044 0.052 0.001 -0.093 -0.018 0.057 0.007^ -0.027 -0.061 -0.095 -0.003

(0.575) (0.418) (0.404) (0.786) (0.171) (0.748) (0.416) (0.055) (0.728) (0.351) (0.234) (0.430)

Behavior Domain

 Altruistic Behavior– -0.071 -0.077* -0.083^ -0.001 -0.074 -0.048 -0.022 0.003 -0.036 -0.029 -0.023 0.001

 CR (+) (0.136) (0.036) (0.085) (0.844) (0.227) (0.321) (0.734) (0.510) (0.578) (0.561) (0.740) (0.878)

 Altruistic Behavior– -0.065 0.046 0.156** 0.011** -0.011 -0.025 -0.040 -0.001 0.001 -0.063 -0.128 -0.006

 PCR (+) (0.217) (0.265) (0.003) (0.001) (0.855) (0.565) (0.505) (0.706) (0.987) (0.308) (0.119) (0.187)

 Altruistic Behavior– 0.083 0.119 0.156 0.004 0.013 0.030 0.048 0.002 -0.024 -0.014 -0.004 0.001

 TRS (+) (0.531) (0.353) (0.237) (0.222) (0.941) (0.858) (0.783) (0.673) (0.853) (0.910) (0.975) (0.809)

 Positive Social -0.014 -0.016 -0.017 0.000 -0.028 0.021 0.069 0.005 -0.008 0.021 0.050 0.003

 Behavior–PCR (+) (0.716) (0.600) (0.667) (0.961) (0.588) (0.601) (0.192) (0.148) (0.898) (0.700) (0.477) (0.481)

 Positive Social -0.049 -0.017 0.014 0.003 -0.067 -0.056 -0.045 0.001 0.037 0.038 0.038 0.000

 Behavior–TRS (+) (0.355) (0.717) (0.789) (0.145) (0.433) (0.477) (0.599) (0.745) (0.663) (0.630) (0.663) (0.993)

 Problem Behavior– -0.012 0.036 0.061 0.002 0.075 -0.007 -0.089 -0.008* 0.042 0.023 0.003 -0.002

 CR (-) (0.840) (0.480) (0.312) (0.424) (0.348) (0.922) (0.275) (0.032) (0.621) (0.765) (0.969) (0.630)

 Problem Behavior– -0.019 -0.005 0.009 0.001 0.017 -0.007 -0.031 -0.002 -0.202** -0.051 0.101 0.015***

 PCR (-) (0.698) (0.898) (0.857) (0.619) (0.735) (0.849) (0.549) (0.484) (0.009) (0.423) (0.211) (0.001)

 Problem Behavior– -0.007 0.048 0.088^ 0.004^ 0.064 -0.011 -0.085 -0.007* -0.014 -0.030 -0.046 -0.002

 TRS (-) (0.890) (0.322) (0.099) (0.078) (0.389) (0.872) (0.255) (0.029) (0.873) (0.697) (0.593) (0.670)

 ADHD-Related Behavior– -0.031 0.039 0.048 0.001 0.036 -0.007 -0.051 -0.004 -0.080 -0.071 -0.062 0.001

 TRS (-) (0.512) (0.335) (0.299) (0.689) (0.589) (0.902) (0.456) (0.199) (0.298) (0.296) (0.431) (0.808)

See notes at end of table.

120

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.33. Combined-program impacts on child outcomes, by initial child behavior risk as reported by teacher—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effectd

(p-value)

(p-value) (p-value) (p-value)

Scale–Report Low1

Average2 High3

Low1

Average2 High3

Low1

Average2 High3

Academics Domain

 Engagement with 0.041 -0.051 -0.143** -0.009** -0.069 -0.016 0.038 0.005 0.006 -0.043 -0.092 -0.005

 Learning–CR (+) (0.413) (0.185) (0.006) (0.007) (0.228) (0.711) (0.532) (0.187) (0.925) (0.420) (0.199) (0.272)

 Academic Competence -0.052 -0.022 0.008 0.003 -0.108* -0.032 0.044 0.007** -0.088 -0.006 0.076 0.008*

 and Motivation–TRS (+) (0.123) (0.429) (0.808) (0.111) (0.035) (0.455) (0.391) (0.008) (0.149) (0.909) (0.225) (0.013)

Perceptions of School

 Climate Domain

 Positive School 0.013 -0.012 -0.037 -0.002 0.040 0.043 0.046 0.000 -0.069 -0.065 -0.062 0.000

 Orientation–CR (+) (0.866) (0.867) (0.635) (0.430) (0.668) (0.612) (0.622) (0.933) (0.472) (0.452) (0.525) (0.935)

 Negative School -0.019 -0.012 -0.006 -0.001 0.057 -0.072 -0.202* -0.013** 0.027 -0.048 -0.122 -0.007^

 Orientation–CR (-) (0.749) (0.801) (0.915) (0.843) (0.467) (0.300) (0.013) (0.001) (0.728) (0.478) (0.128) (0.072)

 Student Afraid at -0.140* -0.068 0.004 0.007* -0.041 -0.054 -0.066 -0.001 0.030 0.033 0.035 0.000

 School–CR (-) (0.018) (0.162) (0.942) (0.029) (0.572) (0.384) (0.375) (0.755) (0.705) (0.631) (0.664) (0.949)

 Victimization at -0.008 -0.005 -0.019 -0.001 -0.010 -0.043 -0.076 -0.003 0.045 0.011 -0.023 -0.003

 School–CR (-) (0.873) (0.896) (0.716) (0.670) (0.867) (0.337) (0.208) (0.390) (0.511) (0.841) (0.751) (0.425)

* Significantly different from zero at the .05 level.

** Significantly different from zero at the .01 level.

*** Significantly different from zero at the .001 level.

^ Significantly different from zero at the .10 to > .05 level.

1 One standard deviation below the mean risk level.

2 At the mean risk level.

3 One standard deviation above the mean risk level.

4 Change in impact as risk level increases by one unit.

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The - signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models in which each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. The effect size 

was calculated by dividing the estimated impact by the standard deviation of the outcome measure for the control group. 

SOURCE: The Social and Character Development (SACD) Research Program.

121

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.34. Combined-program impacts on child outcomes, by child behavior risk as reported by primary caregiver 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

(p-value) (p-value) (p-value)

Scale–Report Low1

Average2 High3

Low1

Average2 High3

Low1

Average2 High3

Social and Emotional 

 Competence Domain

 Self-Efficacy for Peer 0.027 -0.031 -0.090^ -0.006^ -0.015 -0.054 -0.093 -0.004 -0.001 -0.022 -0.043 -0.002

 Interactions–CR (+) (0.582) (0.421) (0.072) (0.061) (0.798) (0.247) (0.125) (0.295) (0.986) (0.609) (0.474) (0.599)

 Normative Beliefs 0.020 -0.012 -0.044 -0.003 0.071 -0.010 -0.092 -0.008* -0.017 -0.010 -0.004 0.001

 About Aggression–CR (-) (0.725) (0.802) (0.446) (0.338) (0.403) (0.893) (0.287) (0.036) (0.847) (0.893) (0.965) (0.876)

 Empathy–CR (+) -0.013 0.077 0.166** 0.009** -0.083 -0.007 0.068 0.008* -0.021 -0.063 -0.105 -0.004

(0.838) (0.148) (0.008) (0.005) (0.217) (0.897) (0.318) (0.042) (0.786) (0.352) (0.189) (0.306)

Behavior Domain

 Altruistic Behavior– -0.110* -0.064 -0.017 0.005 -0.090 -0.031 0.029 0.006 -0.074 -0.030 0.014 0.004

 CR (+) (0.029) (0.108) (0.734) (0.128) (0.148) (0.539) (0.651) (0.118) (0.248) (0.547) (0.838) (0.287)

 Altruistic Behavior– 0.081 0.037 -0.007 -0.004 -0.040 0.001 0.042 0.004 -0.030 -0.038 -0.046 -0.001

 PCR (+) (0.199) (0.350) (0.890) (0.178) (0.500) (0.986) (0.485) (0.281) (0.695) (0.540) (0.548) (0.856)

 Altruistic Behavior– 0.077 0.072 0.066 -0.001 -0.049 0.029 0.107 0.008* -0.010 -0.015 -0.019 0.000

 TRS (+) (0.431) (0.439) (0.496) (0.860) (0.780) (0.864) (0.542) (0.050) (0.938) (0.907) (0.885) (0.910)

 Positive Social 0.013 -0.027 -0.066^ -0.004 -0.086^ -0.007 0.072 0.008* -0.013 -0.003 0.008 0.001

 Behavior–PCR (+) (0.745) (0.384) (0.097) (0.127) (0.083) (0.852) (0.147) (0.015) (0.845) (0.961) (0.909) (0.786)

 Positive Social -0.010 -0.009 -0.009 0.000 -0.016 0.038 -0.061 -0.002 0.010 0.033 0.056 0.002

 Behavior–TRS (+) (0.849) (0.835) (0.854) (0.992) (0.854) (0.630) (0.480) (0.483) (0.904) (0.674) (0.522) (0.528)

 Problem Behavior– 0.025 -0.003 -0.031 -0.003 0.014 -0.037 -0.089 -0.005 0.040 0.032 0.024 -0.001

 CR (-) (0.677) (0.953) (0.603) (0.346) (0.865) (0.612) (0.281) (0.143) (0.638) (0.677) (0.781) (0.830)

 Problem Behavior– -0.076 0.003 0.081^ 0.008** -0.021 0.036 0.092^ 0.006 -0.101 -0.017 0.066 0.008*

 PCR (-) (0.109) (0.943) (0.083) (0.006) (0.691) (0.363) (0.080) (0.101) (0.135) (0.749) (0.328) (0.043)

 Problem Behavior– 0.043 0.053 0.062 0.001 -0.010 -0.008 -0.006 0.000 -0.017 -0.044 -0.071 -0.003

 TRS (-) (0.428) (0.288) (0.256) (0.686) (0.894) (0.908) (0.940) (0.949) (0.846) (0.587) (0.427) (0.457)

 ADHD-Related 0.000 0.020 0.041 0.002 -0.002 -0.021 -0.040 -0.002 -0.107 -0.096 -0.085 0.001

 Behavior–TRS (-) (0.996) (0.619) (0.379) (0.357) (0.982) (0.732) (0.563) (0.556) (0.168) (0.163) (0.278) (0.763)

See notes to at end of table.

122

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.34. Combined-program impacts on child outcomes, by child behavior risk as reported by primary caregiver—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

Effect size at risk level Marginal 

effect4

(p-value)

(p-value) (p-value) (p-value)

Scale–Report Low1

Average2 High3

Low1

Average2 High3

Low1

Average2 High3

Academics Domain

 Engagement with -0.074 -0.057 -0.039 0.002 -0.089 -0.004 0.081 0.009* -0.004 -0.057 -0.110 -0.005

 Learning–CR (+) (0.151) (0.149) (0.443) (0.599) (0.127) (0.925) (0.178) (0.028) (0.954) (0.278) (0.109) (0.210)

 Academic Competence -0.001 -0.001 -0.003 0.000 -0.044 -0.013 0.019 0.003 -0.025 0.005 0.034 0.003

 and Motivation–TRS (+) (0.979) (0.970) (0.930) (0.923) (0.365) (0.754) (0.705) (0.247) (0.702) (0.932) (0.600) (0.350)

Perceptions of School

 Climate Domain

 Positive School -0.050 0.008 0.066 0.006^ -0.009 0.053 0.115 0.006^ -0.022 -0.080 -0.138 -0.006

 Orientation–CR (+) (0.534) (0.912) (0.408) (0.067) (0.918) (0.532) (0.215) (0.092) (0.819) (0.367) (0.158) (0.143)

 Negative School 0.049 -0.034 -0.116^ -0.008** 0.055 0.055 -0.178* -0.012** -0.017 -0.054 -0.090 -0.004

 Orientation–CR (-) (0.418) (0.513) (0.055) (0.009) (0.492) (0.492) (0.030) (0.002) (0.829) (0.433) (0.254) (0.341)

 Student Afraid at -0.070 -0.055 -0.041 0.001 -0.073 -0.068 -0.062 0.001 0.068 0.006 -0.056 -0.006

 School–CR (-) (0.247) (0.274) (0.494) (0.653) (0.317) (0.285) (0.406) (0.879) (0.385) (0.932) (0.478) (0.123)

 Victimization at 0.026 -0.005 -0.016 -0.002 -0.032 -0.040 -0.048 -0.001 -0.010 0.010 0.030 0.002

 School–CR (-) (0.621) (0.903) (0.763) (0.508) (0.598) (0.407) (0.432) (0.822) (0.888) (0.865) (0.678) (0.621)

* Significantly different from zero at the .05 level.

** Significantly different from zero at the .01 level.

^ Significantly different from zero at the .10 to > .05 level.

1 One standard deviation below the mean risk level.

2 At the mean risk level.

3 One standard deviation above the mean risk level.

4 Change in impact as risk level increases by one unit.

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models in which each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. The effect size 

was calculated by dividing the estimated impact by the standard deviation of the outcome measure for the control group. 

SOURCE: The Social and Character Development (SACD) Research Program.

123

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

124

Fidelity of Implementation 

There were 5 cases of significant differences between the associations of the SACD programs with the 

outcomes for schools with different levels of fidelity of implementation (table 1.35). From the 54 

comparisons made, 3 significant differences would be expected by chance. When examining the results of the 

t tests from subgroup-specific regressions for these 5 cases, 4 cases were based on significant detrimental 

program associations for low-fidelity schools: declines in Altruistic Behavior (Child Report in Year 1) and 

Positive School Orientation (in Year 3) and increases in Problem Behavior (Child Report in Year 3) and 

Negative School Orientation (in Year 3). One was linked to a beneficial (though not significant) association 

for high-fidelity schools (a decline in Student Afraid at School in Year 3). 

As a sensitivity test for the Year 2 and Year 3 data, the schools were divided into only two categories (high or 

low implementers) and the Year 1-type analysis was done (using two fidelity subgroups based on the most 

recent year of fidelity data; these data are not shown in a table). No significant differences in associations 

were found between high and low implementers in this analysis. Because the few significant findings from the 

first analysis were primarily due to detrimental associations between low fidelity and outcomes (rather than 

beneficial associations between high fidelity and outcomes), and because there were no significant findings in 

the sensitivity analysis, the fidelity analyses provide little support for the hypothesis of a relationship between 

high fidelity and more beneficial outcomes. 
Table 1.35. Combined-program results for child outcomes, by fidelity of implementation 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Effect size

p-value 

for 

difference1

Effect size

p-value 

for 

difference1

Effect size

p-value 

for 

difference1

(p-value) (p-value) (p-value)

Scale–Report Low High Low Mixed High Low Mixed High

Social and Emotional 

 Competence Domain

 Self-Efficacy for Peer 

 Interactions–CR (+)

-0.02 -0.04 0.882 -0.12^ 0.02 -0.13^ 0.730 -0.09 -0.01 -0.07 0.841

(0.758) (0.397) (0.087) (0.827) (0.063) (0.364) (0.909) (0.291)

 Normative Beliefs About 

 Aggression–CR (-)

0.04 -0.06 0.160 0.06 0.04 -0.14 0.068^ 0.17^ -0.04 -0.10 0.069^

(0.438) (0.300) (0.419) (0.584) (0.148) (0.091) (0.640) (0.235)

 Empathy–CR (+) -0.01 0.15* 0.074^ -0.08 -0.03 0.02 0.531 -0.15 -0.08 0.05 0.298

(0.851) (0.024) (0.242) (0.736) (0.797) (0.112) (0.359) (0.449)

Behavior Domain

 Altruistic Behavior–

 CR (+)

-0.13** 0.01 0.033* -0.05 -0.08 -0.08 0.907 -0.05 -0.03 -0.03 0.904

(0.010) (0.810) (0.474) (0.332) (0.336) (0.508) (0.663) (0.589)

 Altruistic Behavior–

 PCR (+)

0.10^ 0.04 0.283 -0.03 0.13 -0.02 0.248 -0.05 0.00 -0.05 0.896

(0.061) (0.407) (0.693) (0.135) (0.660) (0.565) (0.995) (0.482)

 Altruistic Behavior–

 TRS (+)

0.10 0.13 0.888 -0.10 0.07 0.12 0.832 0.25 -0.12 -0.21 0.324

(0.435) (0.322) (0.589) (0.845) (0.648) (0.359) (0.358) (0.366)

 Positive Social 

 Behavior–PCR (+)

0.00 -0.02 0.692 0.05 0.11 -0.01 0.692 0.01 0.08 -0.02 0.719

(0.958) (0.541) (0.499) (0.125) (0.843) (0.910) (0.232) (0.787)

 Positive Social 

 Behavior–TRS (+)

0.05 -0.01 0.463 0.07 -0.22^ 0.01 0.190 0.12 0.02 -0.09 0.444

(0.357) (0.831) (0.439) (0.083) (0.954) (0.183) (0.853) (0.582)

 Problem Behavior–

 CR (-)

0.04 -0.02 0.374 0.04 0.09 -0.09 0.229 0.20** 0.04 -0.10 0.028*

(0.477) (0.748) (0.751) (0.323) (0.287) (0.008) (0.658) (0.236)

 Problem Behavior–

 PCR (-)

-0.03 0.02 0.483 -0.04 -0.12^ -0.04 0.983 0.00 -0.14^ -0.07 0.898

(0.587) (0.759) (0.571) (0.094) (0.425) (0.974) (0.074) (0.275)

 Problem Behavior–

 TRS (-)

0.00 0.02 0.752 -0.05 0.12 -0.08 0.184 -0.03 0.02 -0.14 0.349

(0.962) (0.784) (0.589) (0.173) (0.512) (0.732) (0.805) (0.290)

 ADHD-Related 

 Behavior–TRS (-)

-0.02 -0.01 0.787 -0.05 0.06 -0.07 0.474 -0.09 -0.01 -0.08 0.838

(0.618) (0.874) (0.599) (0.538) (0.446) (0.295) (0.903) (0.226)

See notes at end of table.

125

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.35. Combined-program results for child outcomes, by fidelity of implementation—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Effect size

p-value 

for 

difference1

Effect size

p-value 

for 

difference1

Effect size

p-value 

for 

difference1

(p-value) (p-value) (p-value)

Scale–Report Low High Low Mixed High Low Mixed High

Academics Domain

 Engagement with 

 Learning–CR (+)

-0.02 -0.07 0.442 -0.03 -0.06 -0.05 0.772 -0.18* 0.06 -0.02 0.075^

(0.670) (0.127) (0.640) (0.410) (0.345) (0.050) (0.406) (0.699)

 Academic Competence 

 and Motivation–TRS (+)

0.02 -0.06 0.225 -0.04 0.01 -0.13* 0.559 -0.05 -0.04 -0.09 0.965

(0.630) (0.226) (0.628) (0.926) (0.033) (0.606) (0.643) (0.261)

Perceptions of School

 Climate Domain

 Positive School 

 Orientation–CR (+)

0.02 0.06 0.670 -0.07 0.04 0.09 0.409 -0.32** -0.04 0.18 0.026*

(0.795) (0.488) (0.427) (0.804) (0.478) (0.001) (0.732) (0.242)

 Negative School 

 Orientation–CR (-)

-0.03 -0.06 0.578 0.03 -0.01 -0.06 0.457 0.21* -0.02 -0.19 0.021*

(0.411) (0.231) (0.672) (0.928) (0.485) (0.035) (0.841) (0.102)

 Student Afraid at 

 School–CR (-)

-0.04 -0.05 0.759 -0.02 0.00 -0.09 0.675 0.02 0.11 -0.20^ 0.016*

(0.484) (0.255) (0.780) (0.994) (0.234) (0.823) (0.199) (0.062)

 Victimization at 

 School–CR (-)

-0.02 0.01 0.700 -0.04 -0.12^ 0.00 0.611 0.07 -0.08 0.01 0.349

(0.632) (0.879) (0.658) (0.083) (0.982) (0.557) (0.209) (0.948)

* Significantly different from zero at the .05 level.

** Significantly different from zero at the .01 level.

^ Significantly different from zero at the .10 to > .05 level, two-tailed test.

1 Test of significant difference between Fidelity subgroups’ effect sizes.

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models in which each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. The effect size 

was calculated by dividing the estimated impact by the standard deviation of the outcome measure for the control group. 

SOURCE: The Social and Character Development (SACD) Research Program.

126

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

127

Summary of Results Regarding Year-by-Year Impacts 

The combined-program analysis of the year-by-year impacts did not yield evidence that the seven SACD 

programs improved student outcomes. A small number of findings were statistically significant (but no more 

than expected by chance except for several of the subgroup analyses) or substantively important. These 

results were split into similar numbers of beneficial and detrimental impacts. In the majority of cases, the 

results (both beneficial and detrimental) occurred only in 1 year and were not replicated across the 3 years of 

the study. 

Specifically, 2 of 60 estimated impacts for the seven SACD programs combined for the 3 years were found 

statistically significant (with beneficial impacts) and none were found to be substantively important. The lack 

of findings at the combined-program level was not found to be due to differences among the individual 

programs or such subgroups as males versus females, stayers versus new entrants, and students with different 

levels of initial risk. However, it should be noted that the smaller sample sizes available for these analyses 

reduced the ability to detect smaller statistically significant impacts compared to the analysis of all seven 

programs together. 

For the individual SACD programs, 16 statistically significant impacts were found over the 3 years (9 

beneficial and 7 detrimental) versus 21 that would be expected by chance. None occurred more than once per 

program. In addition, 19 nonsignificant substantively important impacts were found (10 beneficial and 9 

detrimental). One program had a substantively important beneficial impact on an outcome for more than 1 

year. 

The subgroup analyses produced a low number of significant results, mixed impacts (both beneficial and 

detrimental), and a lack of repeated significant results, providing little evidence of a relationship between 

subgroups and the impact of the seven SACD programs considered together. The analysis by gender found 

eight significant differences in impacts but no pattern favoring one gender. No significant results were found 

for the analysis of stayers versus new entrants. For the analysis of five types of different initial risk, 41 

significant marginal effects were found (versus 13 to 14 expected by chance), of which 26 showed more 

beneficial impacts for higher risk students and 15 found more detrimental impacts for higher risk students as 

compared to lower risk students. Six of these findings were repeated in at least 2 different years. 

Impacts on Growth of Student Outcomes 

A growth curve analysis was done with the 3 years of data to complement the three cross-sectional impact 

analyses comparing the outcomes of students in control and treatment schools at the end of their third-grade 

year (spring 2005), fourth-grade year (spring 2006), and fifth-grade year (spring 2007). The growth curve 

analysis examined the average change over time in the child outcomes between fall 2004 and spring 2007. 

Unlike the cross-sectional analysis, this was not a cumulative analysis but an analysis of the differences in 

growth rates between students. Specifically, it examined the estimated impacts on the trajectories of student 

outcomes over time. The growth curve analysis used the same covariates and compared results across the 

same subgroups as the cross-sectional analyses. In comparison to the cross-sectional analysis, the growth 

curve approach used all available data and multiple measures of the outcomes because students at any survey 

point contributed to the impact estimates. 

The growth curve analysis addressed a subset of the research questions regarding the SACD evaluation 

(combined-program and individual program). These included the following: 

1. What were the average impacts of the seven SACD programs together on the average growth over 

time in student outcomes during the study period, from implementation of the programs in fall 2004 

through spring 2007? 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

128

2. What were the impacts on the average growth over time in student outcomes during the study period 

by individual SACD program? 

3. Did the impacts on the average growth over time in student outcomes during the study period differ 

by gender, stayer versus new entrant status, and initial risk factors? 

As in the case of the cross-sectional year-by-year analysis, the combined analysis used for the first question, 

with its larger sample size and associated greater power, would be able to detect smaller statistically significant 

impacts than the analyses for the second and third questions. 

Sample 

The sample of students for the growth curve analysis included all students who were enrolled in one of the 

study schools during the study period, who were eligible to complete a survey at the initial data collection or 

at any of the follow-up survey points, and who had data from one or more data collection points.40 Student 

outcomes were measured at five time points. Initial measures were collected in fall 2004, and follow-up 

measures were collected in spring 2005, fall 2005, spring 2006, and spring 2007. The outcomes included the 

18 child-level outcome measures across the four outcome domains (Social and Emotional Competence, 

Behavior, Academics, and Perceptions of School Climate). Two additional outcomes, Feelings of Safety and 

Student Support for Teacher, were used in the cross-sectional analysis but were not used in the growth curve 

model because they were measured at the teacher level. 

Table 1.36 displays the sample sizes and percentages of sample with a response across the three reports 

(Child, Primary caregiver, and Teacher). The sample sizes and percentages varied across survey instruments 

because of differences in survey consent and completion rates. At each survey period, the teachers were most 

likely to provide data on children (the percentage of the sample universe with responses ranged from 63% to 

66%). The primary caregivers were least likely to provide data on children (the percentage of the sample 

universe with responses ranged from 46% to 58%). The percentage of the sample with responses was similar 

for treatment and control schools in most survey waves. However, it was statistically significantly higher for 

the treatment group for the Child Report and Primary Caregiver Report in fall 2004 and for the Child Report 

in spring 2005. Table 1.36 also shows the percentages of the follow-up sample members who were in the 

initial sample. There was considerable turnover, as evidenced by the decreasing percentage over time of 

students who were in the initial fall 2004 sample. By spring 2007, across the three survey instruments, about 

two-thirds of the sample had been part of the original fall 2004 survey (66% for the Child Report and the 

Primary Caregiver Report, and 68% for the Teacher Report on Student). There were no statistically significant 

differences in the level of turnover by treatment status. 

 

40 Sensitivity analyses discussed later in this section examined the robustness of the findings when using smaller samples 

of students who had more rounds of data. 
Table 1.36. Growth curve analysis sample size, percentage of sample universe, and percentage of initial sample 

Analysis sample size 

(Percentage of sample universe)

Child Report Primary Caregiver Report Teacher Report on Student

Program

Treatment Control TotalTreatment Control TotalTreatment Control Total

Fall 2004 Overall 2,092 1,905 3,997 1,980 1,794 3,774 2,157 1,946 4,103

 (62.1)** (59.5) (60.9) (58.8)** (56.1) (57.5) (64.1) (60.8) (62.5)

Spring 2005

 Overall 2,200 1,967 4,167 1,768 1,613 3,381 2,264 2,029 4,293

 (64.9)*** (61.3) (63.2) (52.2) (50.3) (51.3) (66.8) (63.2) (65.1)

 Percentage of sample in fall 2004 86.4 88.0 87.1 88.0 88.5 88.2 89.7 89.9 89.8

Fall 2005

 Overall 2,145 1,952 4,097 1,761 1,623 3,384 2,189 1,963 4,152

 (65.4) (64.8) (65.1) (53.7) (53.8) (53.8) (66.7) (65.1) (66.0)

 Percentage of sample in fall 2004 74.9 76.3 75.6 74.4 75.9 75.1 77.7 77.7 77.7

Spring 2006

 Overall 2,146 1,926 4,072 1,682 1,517 3,199 2,239 2,014 4,253

 (64.5) (62.4) (63.5) (50.6) (49.1) (49.9) (67.3) (65.2) (66.3)

 Percentage of sample in fall 2004 71.6 73.8 72.6 73.3 74.2 73.7 74.4 75.4 74.9

Spring 2007

 Overall 2,059 1,935 3,994 1,454 1,414 2,875 2,095 1,994 4,089

 (64.9) (62.9) (63.9) (46.1) (46.0) (46.0) (66.0) (64.8) (65.4)

 Percentage of sample in fall 2004 66.0 65.6 65.8 66.6 64.7 65.6 68.7 67.4 68.0

** Treatment group significantly different from control group at the .01 level.

*** Treatment group significantly different from control group at the .001 level.

NOTE: This table is based on categorizing students according to their current treatment status at a point in time. During the course of the study about 2 percent of students changed 

status when they moved from a treatment to a control school (or vice versa). These students were assigned to their baseline treatment status to preserve the random assignment 

design.

SOURCE: The Social and Character Development (SACD) Research Program.

129

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

130

Table 1.37 shows the average scores for each of the 18 outcome measures by treatment status and data 

collection point. All students with a response for an outcome measure are included in the data, regardless of 

when they entered the study schools. Because all students are included, the treatment-control differences 

reflect both behavioral changes within students who have data from more than one survey and compositional 

changes in the children for whom data were collected. These compositional changes were due to the mobility 

of the sample, and table 1.36 shows little evidence of differential mobility by treatment status. Table 1.37 

shows that the outcome measures followed very similar trajectories for the students in the treatment and 

control schools (as described in detail later in this section). This holds true across domains and across the 

three data reports. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

131

Table 1.37. Child-level outcomes, by survey period and treatment group status 

Child-level outcome

Fall 

2004

Spring 

2005

Fall 

2005

Spring 

2006

Spring 

Average outcome of children with data in any wave 2007 1

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–(CR) (+)

 Control group 2.95 3.07 3.16 3.22 3.24

 Treatment group 2.93 3.04 3.14 3.18* 3.22

 Normative Beliefs About Aggression–(CR) (-)

 Control group 1.23 1.30 1.30 1.36 1.45

 Treatment group 1.22 1.29 1.32 1.36 1.46

 Empathy–(CR) (+)

 Control group 2.41 2.30 2.27 2.19 2.10

 Treatment group 2.42 2.32 2.25 2.17 2.08

Behavior Domain

 Altruistic Behavior–(CR) (+)

 Control group 1.45 1.28 1.17 1.09 1.05

 Treatment group 1.45 1.24 1.14 1.05 1.05

 Altruistic Behavior–(PCR) (+)

 Control group 2.29 2.23 2.23 2.23 2.26

 Treatment group 2.28 2.27 2.23 2.25 2.26

 Altruistic Behavior–(TRS) (+)

 Control group 1.36 1.38 1.29 1.34 1.36

 Treatment group 1.40 1.44 1.34* 1.37 1.34

 Positive Social Behavior–(PCR) (+)

 Control group 3.00 3.03 3.07 3.06 3.08

 Treatment group 3.00 3.02 3.06 3.07 3.09

 Positive Social Behavior–(TRS) (+)

 Control group 3.05 3.02 3.01 3.06 3.10

 Treatment group 3.04 3.03 3.02 3.01 3.07

 Problem Behavior–(CR) (-)

 Control group 0.23 0.29 0.28 0.34 0.46

 Treatment group 0.24 0.30 0.31 0.36 0.50

 Problem Behavior–(PCR) (-)

 Control group 1.57 1.57 1.55 1.55 1.55

 Treatment group 1.56 1.55 1.53 1.53 1.53

 Problem Behavior–(TRS) (-)

 Control group 1.35 1.45 1.38 1.44 1.46

 Treatment group 1.35 1.46 1.38 1.45 1.46

 ADHD-Related Behavior–(TRS) (-)

 Control group 1.72 1.74 1.72 1.69 1.68

 Treatment group 1.73 1.75 1.69 1.69 1.67

See notes at end of table.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

132

Table 1.37. Child-level outcomes, by survey period and treatment group status—Continued

Child-level outcome

Fall 

2004

Spring 

2005

Fall

2005

Spring 

2006

Spring 

Average outcome of children with data in any wave 2007 1

Academics Domain

 Engagement with Learning–(CR) (+)

 Control group 3.70 3.68 3.72 3.69 3.61

 Treatment group 3.66 3.66 3.70 3.66 3.58

 Academic Competence and Motivation–(TRS) (+)

 Control group 2.95 3.02 2.93 3.01 3.02

 Treatment group 2.89 2.96 2.89 2.93 2.94

Perceptions of School Climate Domain

 Positive School Orientation–(CR) (+)

 Control group 3.23 2.86 2.81 2.64 2.56

 Treatment group 3.21 2.86 2.82 2.65 2.51

 Negative School Orientation–(CR) (-)

 Control group 1.82 1.90 1.87 1.97 2.09

 Treatment group 1.87 1.89 1.87 1.98 2.10

 Student Afraid at School–(CR) (-)

 Control group 2.38 2.30 2.22 2.25 2.21

 Treatment group 2.43 2.27 2.21 2.22 2.20

 Victimization at School–(CR) (-)

 Control group 0.76 0.77 0.65 0.69 0.73

 Treatment group 0.77 0.75 0.64 0.66 0.73

* Treatment group significantly different from control group at the .05 level.

1 Average outcome is the average unadjusted score on the outcome measure across all students in that wave of data.

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression 

models in which each program and school within a program was weighted equally. The standard errors of all estimates account for 

design effects due to unequal weighting and the clustering of students within schools. The effect size was calculated by dividing the 

estimated impact by the standard deviation of the outcome measure for the control group. This table is based on categorizing 

students according to their current treatment status at a point in time. During the course of the study about 2 percent of students 

changed status when they moved from a treatment to a control school (or vice versa). These students were assigned to their 

baseline treatment status to preserve the random assignment design. The majority of the outcomes (13) were on a scale of 1-4. The 

scales for the others were as follows: Empathy 1-3, Altruistic Behavior (CR) 0-3, Problem Behavior (CR) 0-3, Academic 

Competence and Motivation 1-5, and Victimization at School 0-3.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

133

Analysis 

The Growth Curve Model 

The SACD programs’ impacts over time were estimated using growth curve models by examining treatment 

and control group differences in the trajectories of student outcomes during the follow-up period while 

accounting for clustering at the school level. The growth curve models were estimated using a three-level 

hierarchical linear model, where Level 1 corresponds to time, Level 2 to students, and Level 3 to schools: 

Level 1 Model—Time 

(1) ytis = 0is + 1is(TSItis) + utis

where ytis is the outcome variable of student i in school s at time t, TSItis is the time measured in years 

between implementation of the SACD program and each survey date,41 utis is a mean zero disturbance term, 

0is and 1is are random student-level intercepts and slopes, respectively, and the program-level subscript p is 

omitted for simplicity. The growth curve models estimate the effect of the SACD programs (together and 

separately) on outcomes measured near the start of the implementation, and then at the time of the four 

follow-up data points (spring 2005, fall 2005, spring 2006, and spring 2007). 

Level 2 Model—Students 

(2) 0is = s + Xis + 0is

(3) 1is = s + 1is

where Xis is a vector of student-level characteristics measured initially (the covariates used in the crosssectional analyses), ε0is and ε1is are mean zero disturbance terms, βs and δs are random school-level coefficients, 

and θ is a vector of parameters to be estimated. The age of the child at the initial data collection was included 

as a control in the model. 

Level 3 Model—Schools 

(4) s = 0 + 1Ts + sZ + 0s

(5) s = 0 + 1Ts + 1s

where Ts is an indicator of whether the school is in the treatment group, s are program-specific fixed effects 

(Z indicates each of the seven SACD programs), τ0s and τ1s are mean zero disturbance terms (that are assumed 

to be correlated with each other and with the error term in equations (4) and (5) for the same school, but not 

across schools), and γ0, γ1, λ0, and λ1 are parameters to be estimated. 

When the student- and school-level equations are substituted into equation (5), this results in the following 

unified model: 

(6) ytis = 0 + 1Ts + Xis + sZ + 0(TSItis) + 1Ts(TSItis) + error 

where error = utis + 0is + 0s + 1is(TSItis) + 1s(TSItis). In this formulation, the estimate of the slope, γ1, 

represents the treatment and control group difference in the mean growth of the outcome measure between 

 

41 Time of implementation was set at the program level. These dates were used both in treatment and control schools 

within that program. If the survey collection date for a student was missing in a round of data, it was imputed using the 

average date within the school for that round of data collection. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

134

the first and last measurement points. The estimate of λ1 represents the impact at the start of the evaluation 

and was expected not to differ significantly from zero. A linear specification of the time metric is used 

because the data indicate that the change in the outcomes followed a linear trajectory over time.42

 

The growth curve analysis included the same set of covariates—X in equation (6)—that were used for the 

cross-sectional analysis (table 1.24) to ensure the comparability of the results. The only exception was the 

exclusion of the initial outcome measure because it was used as the outcome measure for the growth curve 

analysis at time 1 (fall 2004). As in the cross-sectional analysis, missing covariates were imputed using mean 

values for nonmissing cases, by school, gender, race, and ethnicity. Similarly, sample weights were used in all 

analyses to (1) give each program equal weight in the calculation of overall impact estimates, and (2) give each 

school equal weight in each program. However, the weights were not adjusted for consent and response 

differences across classrooms or schools because the population of students within the schools changed over 

time as students entered and left the schools. The weights were also constructed to give equal weight to each 

time period. 

Equation (6) was estimated for all programs together and for each program separately. In the program-level 

models, the program-specific fixed effects were removed from the model by dropping Z. The key impact 

parameter in the growth curve model in equation (6) is γ1, which represents the extent to which the growth in 

the outcome during the follow-up period differed between the treatment and control groups. The estimated 

intercept parameter (γ0) is of interest because it signifies a model-generated mean initial value of the outcome, 

and thus provides a base for interpreting the slope parameter. The model generated the following impact 

information for each child outcome: (1) the average intercept value for the treatment and control groups (λ0

+ pλ1 + 

x

θ, where p is the percentage of the sample assigned to the treatment group); (2) the estimated 

growth for the treatment group (γ1 + γ0); (3) the estimated growth for the control group (γ0); and (4) the 

estimated impact of the SACD programs (and each program individually) on growth (γ1).

Growth curve effect sizes were calculated by dividing the estimated impact of the treatment on the outcome 

growth trajectory by the standard deviation of that outcome (which was calculated using the initial data for 

the weighted control group). One year was the unit of time used in the growth curve analysis, so the 

estimated impact of the treatment on the growth trajectory equals the difference between the treatment 

group’s outcome and the control group’s outcome, on average, after one year.43 After dividing by the standard 

deviation of the control group outcome, the effect size measures the number of standard deviations the 

treatment group differs from the control group after one average year of the SACD program. This approach 

was analogous to the effect size calculations for the cross-sectional analysis. The primary difference between 

the effect size for the cross-sectional analysis and the effect size for the growth curve trajectory is the time 

unit used in calculating the numerator. In the cross-sectional analysis, the numerator of the effect size is the 

estimated impact of the treatment on the outcome over the time period between the initial and the follow-up 

survey period. In the growth curve effect size, the numerator is the difference between the treatment and the 

control group after one year of the study, on average. 

The growth curve analysis was extended to examine differences by subgroups. For the subgroup analyses, the 

interaction terms between treatment status, time since implementation, and indicators of membership in 

subgroups were included in equation (6). This produced an estimate of the slope for subgroup members in 

the treatment group, relative to subgroup members in the control group. The same subgroups were used in 

 

42 Three sensitivity tests were done. First the assumption of linearity was tested through the inclusion of a TSI-squared 

term. Second, the potential for learning loss as a result of summer school breaks was modeled by including a time 

indicator that allowed the growth coefficient to decline (or increase) during the summer. Third, child’s age was used in

place of time since implementation to model growth. Child’s age captures both the age of the child at the initial data 

collection and the time since implementation of the program in one variable. 

43 This impact estimate takes into account differences between the initial levels of the outcome for the treatment and 

control groups, differences in their covariates, and the effects of clustering at the school level. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

135

the growth curve analysis as were used in the cross-sectional analyses, and these included (1) student gender; 

(2) new entrants versus members of the original sample (stayers); and (3) levels of initial risk for 

socioeconomic risk, family risk, perceptions of community risk, and child behavior risk. 

Results 

None of the 18 estimated impacts on the trajectories of child outcomes from the seven SACD programs 

when combined were statistically significant (table 1.38). The estimated effect sizes were at or below .07 

(absolute value).44

Estimated impacts could differ across programs if some SACD programs were more effective than others, or 

if the types of children or settings in some programs were more conducive to positive program effects than 

those found in others. Therefore, it is possible that there were no statistically significant impact estimates at 

the overall level because beneficial impacts in some programs were offset by negative or null impacts in 

others. To investigate this possibility, the impacts at the program level were tested. 

Table 1.39 lists the outcomes on which each individual program had statistically significant impacts (a 

program’s impacts on all the outcomes are given in the program-specific chapters). The results indicate that 

the lack of significant impacts in the overall evaluation reflected the lack of significant impacts at the program 

level. Six impacts would be expected to appear significant by chance at the 5 percent significance level, given 

that 126 impact estimates were made. Table 1.39 shows that there were 6 significant outcomes found: 2 with 

beneficial impacts on growth and 4 with detrimental impacts on growth. These outcomes, and the programs 

with which they were associated, follow. CSP showed a significant beneficial impact on growth for 

Victimization at School (Child Report) with an effect size of -.09. PATHS showed a significant beneficial 

impact on growth for Academic Competence (Teacher Report) with an effect size of 0.09. LBW showed a 

significant detrimental impact on growth for Positive School Orientation (Child Report) with an effect size of

-0.13. SS showed three significant detrimental impacts on growth: for Engagement with Learning (Child 

Report) with an effect size of -0.09, Positive Social Behavior (Teacher Report) with an effect size of -0.19, 

and Empathy (Child Report) with an effect size of -0.13. 

 

44 When a squared value of the time since implementation was added to the sensitivity analysis, three of the outcomes 

(child-reported Altruistic Behavior, child-reported Victimization, and child-reported Student Afraid at School), showed a 

significant treatment effect on the growth trajectory. These results indicated that for child-reported Altruistic Behavior, 

the treatment had a short-term detrimental impact that converged with the control group over time. For child-reported 

Victimization at School and Student Afraid at School, the treatment had a short-term beneficial impact that converged 

with the control group over time. Because there were 3 significant results out of 18 and the direction of the effects was 

mixed, the linear specification of the growth curve model was retained. 
Table 1.38. Impacts on growth of child outcomes from combined program analysis 

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2

Treatment 

group

Control 

group

Impact on 

growth3

Effect

size4

Standard 

error of

impact

p-value of 

impact

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 2.95 0.13 0.13 0.00 0.00 0.01 0.942

 Normative Beliefs About Aggression–CR (-) 1.23 0.10 0.07 0.02 0.04 0.01 0.115

 Empathy–CR (+) 2.41 -0.14^ -0.12 -0.02 -0.05 0.01 0.070

Behavior Domain

 Altruistic Behavior–CR (+) 1.41 -0.17 -0.16 -0.01 -0.01 0.02 0.681

 Altruistic Behavior–PCR (+) 2.32 -0.03 -0.03 0.01 0.01 0.01 0.616

 Altruistic Behavior–TRS (+) 1.40 -0.04 0.00 -0.04 -0.07 0.03 0.224

 Positive Social Behavior–PCR (+) 2.99 0.04 0.03 0.01 0.02 0.01 0.217

 Positive Social Behavior–TRS (+) 3.00 0.00 0.01 -0.01 -0.01 0.03 0.671

 Problem Behavior–CR (-) 0.24 0.10 0.08 0.01 0.02 0.01 0.351

 Problem Behavior–PCR (-) 1.58 -0.01 -0.01 0.00 0.00 0.01 0.843

 Problem Behavior–TRS (-) 1.38 0.05 0.05 0.00 0.00 0.01 0.909

 ADHD-Related Behavior–TRS (-) 1.75 -0.02 -0.01 -0.01 -0.01 0.02 0.707

Academics Domain

 Engagement with Learning–CR (+) 3.69 -0.03 -0.03 0.00 -0.01 0.01 0.707

 Academic Competence and Motivation–TRS (+) 2.87 0.02 0.03 -0.01 -0.01 0.02 0.590

See note at end of table.

136

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.38. Impacts on growth of child outcomes from combined-program analysis—Continued

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2

Treatment 

group

Control 

group

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 3.09 -0.24 -0.21 -0.03 -0.03 0.02 0.163

 Negative School Orientation–CR (-) 1.84 0.10 0.09 0.01 0.01 0.02 0.696

 Student Afraid at School–CR (-) 2.38 -0.08 -0.08 0.00 0.00 0.02 0.956

 Victimization at School–CR (-) 0.76 -0.03 -0.02 -0.01 -0.01 0.02 0.368

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 Significantly different from zero at the .05 level.

2 The average score at implementation is calculated across treatment and control groups, using regression models for adjustment on covariates. 

3 Estimated difference between the slope of the treatment and control groups.

4 The slope of the treatment group minus the slope of the control group divided by the standard deviation of the outcome for the multiprogram control group (the standard deviation is 

calculated without accounting for school-level clustering or regression adjustments).

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. No findings were found statistically significant at or below the .05 level. All impact estimates were calculated 

using HLM 6.06. Sample weights were used in all analyses to (1) give each program equal weight within each time period, (2) give each school equal weight in each program (within 

each time period), and (3) give each time period equal weight in the analysis. 

SOURCE: The Social and Character Development (SACD) Research Program.

137

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

138

Table 1.39. Significant impacts from the growth curve analyses of the individual programs 

Program

Significant1beneficial impacts Significant1detrimental impacts

(Report) (Effect size) (p-value) (Report) (Effect size) (p-value)

Total 2 4

ABC

CSP Victimization at School 

(CR) (-.09) (.050)

LBW Positive School Orientation 

(CR) (-.13) (.016)

PA

PATHS Academic Competence

(TRS) (.08) (.048)

4Rs

SS Engagement with Learning

(CR) (-.09) (.021)

Positive Social Behavior 

(TRS) (-.19) (.019)

Empathy 

(CR) (-0.13) (.028)

1

Significantly different from zero at the .05 level.

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

 CR: Child Report

 TRS: Teacher Report on Student

 Blank cell: Finding of no impact

Out of the 126 comparisons made (7 programs times 18 outcomes), 6 would be expected to be statistically significant at the .05 

level by chance. The number of results found significant was no more than expected by chance. All impact estimates were 

calculated using HLM 6.06. Sample weights were used in all analyses to (1) give each program equal weight within each time 

period, (2) give each school equal weight in each program (within each time period), and (3) give each time period equal weight in 

the analysis. 

SOURCE: The Social and Character Development (SACD) Research Program.

The subgroup analyses found little evidence regarding differential impacts of the seven SACD programs, 

combined, on the subgroups’ growth in outcomes. For the gender subgroup analysis, none of the 36 

estimated impacts was statistically significant (table 1.40). For the new entrants analysis, 1 impact (on 

Negative School Orientation) was found to differ significantly and detrimentally for new entrants versus the 

impact on members of the original sample (table 1.41). From the 18 tests done, 1 significant impact would be 

expected by chance for the new entrants analysis. 
Table 1.40. Impacts on growth of child outcomes, by gender, from combined-program analysis 

Boys Girls

Scale–Report 

Impact on 

annual 

growth p-value

Impact on 

annual 

growth p-value

p-value for 

test of impact 

differences

Effect 

size1

Effect

size1

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 0.00 0.01 0.785 0.00 0.00 0.885 0.347

 Normative Beliefs About Aggression–CR (-) 0.01 0.02 0.557 0.02 0.03 0.443 0.630

 Empathy–CR (+) -0.01 -0.02 0.583 -0.02 -0.04 0.282 0.349

Behavior Domain

 Altruistic Behavior–CR (+) 0.01 0.01 0.673 0.00 0.00 0.888 0.153

 Altruistic Behavior–PCR (+) 0.01 0.01 0.612 0.00 0.00 0.864 0.925

 Altruistic Behavior–TRS (+) -0.04 -0.06 0.211 -0.05 -0.10 0.104 0.379

 Positive Social Behavior–PCR (+) 0.01 0.02 0.446 0.00 0.01 0.687 0.933

 Positive Social Behavior–TRS (+) -0.01 -0.01 0.674 0.00 0.00 0.995 0.519

 Problem Behavior–CR (-) 0.02 0.03 0.332 0.00 0.01 0.794 0.851

 Problem Behavior–PCR (-) 0.00 -0.01 0.590 0.00 -0.01 0.498 0.708

 Problem Behavior–TRS (-) -0.01 -0.02 0.685 -0.01 -0.02 0.426 0.932

 ADHD-Related Behavior–TRS (-) -0.03 -0.04 0.219 -0.01 -0.01 0.498 0.293

Academics Domain

 Engagement with Learning–CR (+) 0.01 0.02 0.489 -0.01 -0.01 0.581 0.933

 Academic Competence and Motivation–TRS (+) -0.01 -0.01 0.756 -0.02 -0.02 0.291 0.495

See notes at end of table.

139

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.40. Impacts on growth of child outcomes, by gender, from combined-program analysis—Continued

Boys Girls

Scale–Report 

Impact on 

annual 

growth p-value

Impact on 

annual 

growth p-value

p-value for 

test of impact 

differences

Effect

size1

Effect

size1

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) -0.01 -0.01 0.680 -0.01 -0.01 0.742 0.571

 Negative School Orientation–CR (-) 0.00 0.00 0.970 -0.01 -0.02 0.386 0.408

 Student Afraid at School–CR (-) 0.00 0.00 0.880 -0.04 -0.03 0.149 0.213

 Victimization at School–CR (-) -0.02 -0.02 0.205 0.00 0.00 0.930 0.349

1 The slope of the treatment group minus the slope of the control group divided by the standard deviation of the outcome for the combined-program control group (the standard 

deviation is calculated without accounting for school-level clustering or regression adjustments).

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. No findings were found statistically significant at or below the .05 level. All impact estimates were calculated 

using HLM 6.06. Sample weights were used in all analyses to (1) give each program equal weight within each time period, (2) give each school equal weight in each program (within 

each time period), and (3) give each time period equal weight in the analysis. 

SOURCE: The Social and Character Development (SACD) Research Program.

140

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.41. Impacts on growth of child outcomes for new entrants versus original members, from combined-program analysis 

Original members New entrants

Scale–Report 

Impact on 

annual 

growth

Effect 

size1

Impact on 

annual 

growth

Effect 

size1

p-value for 

test of impact 

p-value p-value differences

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 0.00 0.00 0.947 0.00 0.00 0.918 0.931

 Normative Beliefs About Aggression–CR (-) 0.02 0.05 0.256 0.01 0.02 0.725 0.867

 Empathy–CR (+) -0.02 -0.05 0.107 -0.02 -0.05 0.441 0.475

Behavior Domain

 Altruistic Behavior–CR (+) -0.01 -0.01 0.783 0.01 0.01 0.869 0.251

 Altruistic Behavior–PCR (+) 0.01 0.01 0.693 0.01 0.01 0.742 0.752

 Altruistic Behavior–TRS (+) -0.05 -0.11 0.107 0.02 0.05 0.629 0.764

 Positive Social Behavior–PCR (+) 0.01 0.01 0.442 -0.02 -0.04 0.350 0.563

 Positive Social Behavior–TRS (+) 0.00 0.00 0.881 -0.02 -0.02 0.729 0.254

 Problem Behavior–CR (-) 0.01 0.01 0.732 0.00 0.01 0.922 0.320

 Problem Behavior–PCR (-) 0.00 0.01 0.707 -0.01 -0.02 0.629 0.851

 Problem Behavior–TRS (-) -0.01 -0.01 0.710 -0.01 -0.01 0.838 0.427

 ADHD-Related Behavior–TRS (-) -0.02 -0.02 0.318 0.02 0.02 0.694 0.107

Academics Domain

 Engagement with Learning–CR (+) -0.01 -0.01 0.577 -0.03 -0.04 0.228 0.709

 Academic Competence and Motivation–TRS (+) -0.02 -0.02 0.281 -0.02 -0.02 0.739 0.910

See notes at end of table.

141

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.41. Impacts on growth of child outcomes for new entrants versus original members, from combined-program analysis—

Continued

Original members New entrants

Scale–Report 

Impact on 

annual 

growth

Effect 

size1

Impact on 

annual 

growth

Effect 

size1

p-value for 

test of impact 

p-value p-value differences

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) -0.01 -0.01 0.592 -0.05 -0.06 0.161 0.113

 Negative School Orientation–CR (-) -0.01 -0.01 0.627 0.05 0.06^ 0.067 0.012*

 Student Afraid at School–CR (-) -0.02 -0.02 0.427 -0.05 -0.06 0.304 0.701

 Victimization at School–CR (-) -0.01 -0.01 0.435 0.02 0.02 0.568 0.643

* Significantly different from zero at the .05 level.

^ Significantly different from zero at the .10 to > .05 level.

1 The slope of the treatment group minus the slope of the control group divided by the standard deviation of the outcome for the multisite control group (the standard deviation is 

calculated without accounting for school-level clustering or regression adjustments).

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using HLM 6.06. Sample weights were used in all analyses to (1) give 

each program equal weight within each time period, (2) give each school equal weight in each program (within each time period), and (3) give each time period equal weight in the 

analysis. 

SOURCE: The Social and Character Development (SACD) Research Program.

142

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

143

Few of the growth trajectories differed across levels of initial risk (tables 1.42 through 1.46). An example 

illustrates how to interpret the tables. Table 1.42 shows the impacts on the growth trajectory of the outcome 

scales by the child’s initial level of socioeconomic risk. The first row in the table displays the estimated 

impacts on the growth trajectory of child-reported Self-Efficacy for Peer Interactions at three levels of initial 

risk: (1) at one standard deviation below the mean risk level, (2) at the mean risk level, and (3) at one standard 

deviation above the mean risk level. When a child had a low level of risk (one standard deviation below the 

mean), the difference between the growth trajectories of the treatment group and the control group for SelfEfficacy for Peer Interactions was -0.01. When a child had an average level of risk (at the mean), the 

difference between the treatment group and the control group was -0.01. When a child had a high level of risk 

(one standard deviation above the mean), the difference between the treatment group and the control group 

was 0.00. The interaction term showed that with each one-unit increase in socioeconomic risk, the impact on 

the Self-Efficacy for Peer Interactions trajectory increased by 0.01. In other words, the higher the child’s 

initial level of socioeconomic risk, the more beneficial was the SACD program for the child’s growth in SelfEfficacy for Peer Interactions. However, this difference in the SACD programs’ impact across risk levels was 

not statistically significantly different from zero (interaction term = 0.01; not significant at the .05 level). 

There were 6 outcomes out of 90 estimated impacts for which growth differed significantly across initial risk 

levels (versus 4 to 5 expected by chance). For 4 outcomes, the seven SACD programs combined had more 

beneficial impacts on growth for children with higher initial risk levels. The SACD programs resulted in 

greater growth of parent-reported Altruistic Behavior for children with higher levels of socioeconomic risk 

(interaction term = 0.03 on table 1.42); greater growth on Academic Competence and less growth of childreported Victimization at School among children with higher levels of family risk (interaction term = 0.13 and 

-0.13, respectively, on table 1.43); and less growth of child-reported Victimization at School among children 

with higher levels of perceived community risk (interaction term = -0.06 on table 1.44). 

For two outcomes, the seven SACD programs combined had more detrimental impacts on growth for 

children with higher initial risk levels. The SACD programs resulted in less growth for child-reported 

Empathy among children with greater levels of family risk (interaction term = -0.08 on table 1.43), and 

greater growth on Normative Beliefs About Aggression among children with higher initial levels of teacherreported behavior risks (interaction term = 0.01 on table 1.45). 
Table 1.42. Impacts on growth of child outcomes, by initial socioeconomic risk 

Socioeconomic risk

(Mean = .60; Standard deviation = .71)

Impact on growth at the specified risk level (p-value)

Scale–Report 

One standard 

deviation below mean1 Mean2

One standard 

deviation above mean3

Interaction term4

(p-value)

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) -0.01 -0.01 0.00 0.01

 (0.500) (0.594) (0.500) (0.511)

 Normative Beliefs About Aggression–CR (-) 0.03* 0.03* 0.03^ 0.00

 (0.049) (0.027) (0.079) (0.842)

 Empathy–CR (+) -0.01 -0.02 -0.03^ -0.01

 (0.500) (0.122) (0.059) (0.518)

Behavior Domain

 Altruistic Behavior–CR (+) -0.02 -0.01 0.01 0.02

 (0.500) (0.628) (0.500) (0.341)

 Altruistic Behavior–PCR (+) -0.01 0.01 0.04* 0.03*

 (0.500) (0.344) (0.045) (0.044)

 Altruistic Behavior–TRS (+) -0.03 -0.03 -0.03 0.00

 (0.500) (0.269) (0.214) (0.870)

 Positive Social Behavior–PCR (+) 0.01 0.01 0.02^ 0.01

 (0.500) (0.100) (0.090) (0.314)

 Positive Social Behavior–TRS (+) -0.01 -0.01 -0.01 0.00

 (0.500) (0.599) (0.500) (0.939)

 Problem Behavior–CR (-) 0.01 0.02 0.02 0.01

 (0.500) (0.277) (0.272) (0.677)

 Problem Behavior–PCR (-) -0.01 0.00 0.00 0.01

 (0.261) (0.720) (0.500) (0.324)

 Problem Behavior–TRS (-) 0.00 0.00 0.00 0.00

 (0.500) (0.915) (0.500) (0.879)

 ADHD-Related Behavior–TRS (-) 0.00 -0.01 -0.02 -0.02

 (0.500) (0.716) (0.500) (0.329)

See notes at end of table.

144

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.42. Impacts on growth of child outcomes, by initial socioeconomic risk—Continued

Socioeconomic risk

(Mean = .60; Standard deviation = .71)

Impact on growth at the specified risk level (p-value)

Interaction term4

Scale–Report (p-value)

One standard 

deviation below mean1 Mean2

One standard 

deviation above mean3

Academics Domain

 Engagement with Learning–CR (+) -0.02 -0.01 0.01 0.02

 (0.213) (0.644) (0.500) (0.249)

 Academic Competence and Motivation–TRS (+) -0.01 0.00 0.01 0.01

 (0.500) (0.861) (0.500) (0.559)

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) -0.02 -0.03 -0.04^ -0.01

 (0.500) (0.150) (0.087) (0.518)

 Negative School Orientation–CR (-) 0.01 0.00 -0.01 -0.01

 (0.500) (0.876) (0.500) (0.414)

 Student Afraid at School–CR (-) -0.02 0.00 0.01 0.02

 (0.500) (0.833) (0.500) (0.315)

 Victimization at School–CR (-) -0.01 -0.01 -0.01 0.00

 (0.500) (0.506) (0.500) (0.965)

* Significantly different from zero at the .05 level.

^ Significantly different from zero at the .10 to > .05 level.

1 The difference between the estimates of the treatment and control groups’ rate of growth, for the average student who is one standard deviation below the mean for the risk 

measure.

2

The difference between the estimates of the treatment and control groups’ rate of growth, for the average student who is at the mean for the risk measure.

3 The difference between the estimates of the treatment and control groups’ rate of growth, for the average student who is one standard deviation above the mean for the risk 

measure.

4 Estimated difference between the treatment and control groups in the incremental change in the rate of growth, with each unit increase in the risk measure.

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using HLM 6.06. Sample weights were used in all analyses to (1) give 

each program equal weight within each time period, (2) give each school equal weight in each program (within each time period), and (3) give each time period equal weight in the 

analysis. 

SOURCE: The Social and Character Development (SACD) Research Program.

145

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.43. Impacts on growth of child outcomes, by initial family risk 

Family risk

(Mean = 1.15; Standard deviation = .18)

Impact on growth at the specified risk level (p-value)

Scale–Report

One standard 

deviation below mean1 Mean2

One standard 

deviation above mean3

Interaction term4

(p-value)

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) -0.01 0.00 0.01 0.06

 (0.500) (0.739) (0.500) (0.273)

 Normative Beliefs About Aggression–CR (-) 0.02 0.03* 0.04^ 0.04

 (0.142) (0.047) (0.052) (0.510)

 Empathy–CR (+) 0.00 -0.02 -0.03* -0.08*

 (0.500) (0.115) (0.014) (0.040)

Behavior Domain

 Altruistic Behavior–CR (+) 0.01 -0.01 -0.02 -0.07

 (0.500) (0.715) (0.300) (0.227)

 Altruistic Behavior–PCR (+) 0.00 0.01 0.01 0.02

 (0.500) (0.672) (0.500) (0.799)

 Altruistic Behavior–TRS (+) -0.05^ -0.04 -0.02 0.07

 (0.070) (0.225) (0.500) (0.129)

 Positive Social Behavior–PCR (+) 0.01 0.02^ 0.02* 0.05^

 (0.500) (0.090) (0.018) (0.090)

 Positive Social Behavior–TRS (+) -0.02 -0.01 -0.01 0.04

 (0.500) (0.607) (0.500) (0.443)

 Problem Behavior–CR (-) 0.03^ 0.02 0.01 -0.05

 (0.078) (0.241) (0.500) (0.983)

 Problem Behavior–PCR (-) 0.00 0.00 0.00 0.00

 (0.500) (0.440) (0.500) (0.983)

 Problem Behavior–TRS (-) 0.00 0.00 0.00 0.01

 (0.500) (0.858) (0.500) (0.891 

 ADHD-Related Behavior–TRS (-) 0.00 -0.01 -0.01 -0.02

 (0.500) (0.659) (0.500) (0.685)

See notes at end of table.

146

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.43. Impacts on growth of child outcomes, by initial family risk—Continued

Family risk

(Mean = 1.15; Standard deviation = .18)

Impact on growth at the specified risk level (p-value)

Scale–Report

One standard 

deviation below mean1 Mean2

One standard 

deviation above mean3

Interaction term4

(p-value)

Academics Domain

 Engagement with Learning–CR (+) 0.00 -0.01 -0.01 -0.02

 (0.500) (0.699) (0.500) (0.669)

 Academic Competence and Motivation–TRS (+) -0.03 0.00 0.02 0.13*

 (0.222) (0.844) (0.500) (0.031)

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) -0.01 -0.03 -0.04* -0.09^

 (0.500) (0.170) (0.028) (0.097)

 Negative School Orientation–CR (-) 0.00 0.00 0.00 0.00

 (0.500) (0.898) (0.500) (0.926)

 Student Afraid at School–CR (-) -0.03 -0.01 0.01 0.11

 (0.257) (0.698) (0.500) (0.206)

 Victimization at School–CR (-) 0.01 -0.01 -0.04^ -0.13*

 (0.500) (0.466) (0.059) (0.032)

* Significantly different from zero at the .05 level.

^ Significantly different from zero at the .10 to > .05 level.

1 The difference between the estimates of the treatment and control groups’ rate of growth, for the average student who is one standard deviation below the mean for the risk 

measure.

2 The difference between the estimates of the treatment and control groups’ rate of growth, for the average student who is at the mean for the risk measure.

3 The difference between the estimates of the treatment and control groups’ rate of growth, for the average student who is one standard deviation above the mean for the risk 

measure.

4 Estimated difference between the treatment and control groups in the incremental change in the rate of growth, with each unit increase in the risk measure.

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using HLM 6.06. Sample weights were used in all analyses to (1) give 

each program equal weight within each time period, (2) give each school equal weight in each program (within each time period), and (3) give each time period equal weight in the 

analysis. 

SOURCE: The Social and Character Development (SACD) Research Program.

147

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.44. Impacts on growth of child outcomes, by initial perceptions of community risk 

Community risk

(Mean = 1.52; Standard deviation = .61)

Impact on growth at the specified risk level (p-value)

Scale–Report

One standard 

deviation below mean1 Mean2

One standard 

deviation above mean3

Interaction term4

(p-value)

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) -0.01 0.00 0.01 0.01

 (0.500) (0.829) (0.500) (0.494)

 Normative Beliefs About Aggression–CR (-) 0.04* 0.03* 0.02 -0.02

 (0.024) (0.028) (0.226) (0.384)

 Empathy–CR (+) -0.02 -0.02^ -0.02 0.00

 (0.200) (0.096) (0.145) (0.925)

Behavior Domain

 Altruistic Behavior–CR (+) -0.01 -0.01 0.00 0.01

 (0.500) (0.743) (0.500) (0.669)

 Altruistic Behavior–PCR (+) 0.01 0.01 0.00 -0.01

 (0.500) (0.666) (0.500) (0.665)

 Altruistic Behavior–TRS (+) -0.05 -0.03 -0.02 0.03

 (0.129) (0.258) (0.500) (0.218)

 Positive Social Behavior–PCR (+) 0.02 0.02^ 0.02 0.00

 (0.146) (0.075) (0.208) (0.939)

 Positive Social Behavior–TRS (+) -0.03 -0.01 0.00 0.02

 (0.286) (0.615) (0.500) (0.230)

 Problem Behavior–CR (-) 0.01 0.02 0.02 0.00

 (0.500) (0.255) (0.307) (0.871)

 Problem Behavior–PCR (-) -0.01* 0.00 0.01 0.02^

 (0.035) (0.908) (0.250) (0.051)

 Problem Behavior–TRS (-) 0.01 0.00 -0.01 -0.02

 (0.500) (0.899) (0.500) (0.222)

 ADHD-Related Behavior–TRS (-) 0.02 -0.01 -0.03 -0.04^

 (0.500) (0.728) (0.195) (0.060)

See notes at end of table.

148

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.44. Impacts on growth of child outcomes, by initial perceptions of community risk—Continued

Table 1.44

Community risk

(Mean = 1.52; Standard deviation = .61)

Impact on growth at the specified risk level (p-value)

Scale–Report

One standard 

deviation below mean1 Mean2

One standard 

deviation above mean3

Interaction term4

(p-value)

Academics Domain

 Engagement with Learning–CR (+) -0.01 -0.01 0.00 0.00

 (0.500) (0.661) (0.500) (0.866)

 Academic Competence and Motivation–TRS (+) 0.00 0.00 -0.01 -0.01

 (0.500) (0.968) (0.500) (0.732)

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) -0.06* -0.03^ -0.01 0.04^

 (0.017) (0.095) (0.500) (0.073)

 Negative School Orientation–CR (-) 0.01 0.00 0.00 -0.01

 (0.500) (0.792) (0.500) (0.528)

 Student Afraid at School–CR (-) -0.02 -0.01 0.01 0.02

 (0.500) (0.764) (0.500) (0.510)

 Victimization at School–CR (-) 0.02 -0.01 -0.05* -0.06*

 (0.224) 0.386) (0.035) (0.014)

* Significantly different from zero at the .05 level.

^ Significantly different from zero at the .10 to > .05 level.

1 The difference between the estimates of the treatment and control groups’ rate of growth, for the average student who is one standard deviation below the mean for the risk 

measure.

2 The difference between the estimates of the treatment and control groups’ rate of growth, for the average student who is at the mean for the risk measure.

3 The difference between the estimates of the treatment and control groups’ rate of growth, for the average student who is one standard deviation above the mean for the risk 

measure.

4 Estimated difference between the treatment and control groups in the incremental change in the rate of growth, with each unit increase in the risk measure.

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using HLM 6.06. Sample weights were used in all analyses to (1) give 

each program equal weight within each time period, (2) give each school equal weight in each program (within each time period), and (3) give each time period equal weight in the 

analysis. 

SOURCE: The Social and Character Development (SACD) Research Program.

149

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.45. Impacts on growth of child outcomes, by child behavior risk as reported by teacher 

Teacher-reported child behavior risk

(Mean = 1.79; Standard deviation = 2.04)

Impact on growth at the specified risk level (p-value)

Scale–Report 

One standard 

deviation below mean1 Mean2

One standard 

deviation above mean3

Interaction term4

(p-value)

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 0.00 0.00 -0.01 0.00

 (0.500) (0.699) (0.500) (0.929)

 Normative Beliefs About Aggression–CR (-) 0.01 0.03* 0.05* 0.01*

 (0.500) (0.038) (0.005) (0.047)

 Empathy–CR (+) -0.01 -0.02^ -0.02^ 0.00

 (0.500) (0.098) (0.064) (0.585)

Behavior Domain

 Altruistic Behavior–CR (+) -0.01 -0.01 0.00 0.00

 (0.500) (0.786) (0.500) (0.908)

 Altruistic Behavior–PCR (+) 0.02 0.01 -0.01 -0.01

 (0.188) (0.644) (0.500) (0.228)

 Altruistic Behavior–TRS (+) -0.05^ -0.03 -0.02 0.01

 (0.091) (0.241) (0.500) (0.142)

 Positive Social Behavior–PCR (+) 0.00 0.02^ 0.03* 0.01^

 (0.500) (0.057) (0.017) (0.070)

 Positive Social Behavior–TRS (+) -0.01 -0.02 -0.02 0.00

 (0.500) (0.539) (0.500) (0.755)

 Problem Behavior–CR (-) 0.01 0.02 0.02 0.00

 (0.500) (0.248) (0.237) (0.584)

 Problem Behavior–PCR (-)

-0.01 0.00 0.00 0.00

 (0.307) (0.541) (0.500) (0.431)

 Problem Behavior–TRS (-) 0.01 0.01 0.00 0.00

 (0.500) (0.745) (0.500) (0.563)

 ADHD-Related Behavior–TRS (-) 0.01 -0.01 -0.02 -0.01

 (0.500) (0.760) (0.500) (0.173)

See notes at end of table.

150

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.45. Impacts on growth of child outcomes, by child behavior risk as reported by teacher—Continued

Teacher-reported child behavior risk

(Mean = 1.79; Standard deviation = 2.04)

Impact on growth at the specified risk level (p-value)

Scale–Report 

One standard 

deviation below mean1 Mean2

One standard 

deviation above mean3

Interaction term4

(p-value)

Academics Domain

 Engagement with Learning–CR (+) 0.01 -0.01 -0.03 -0.01

 (0.500) (0.544) (0.159) (0.114)

 Academic Competence and Motivation–TRS (+) -0.03 -0.01 0.02 0.01^

 (0.215) (0.799) (0.500) (0.092)

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) -0.02 -0.03 -0.05* -0.01

(0.500) (0.108) (0.033) (0.257)

 Negative School Orientation–CR (-)

-0.01 0.00 0.02 0.01

 (0.500) (0.762) (0.500) (0.208)

 Student Afraid at School–CR (-) 0.00 -0.01 -0.01 0.00

 (0.500) (0.739) (0.500) (0.896)

 Victimization at School–CR (-)

-0.01 -0.01 -0.01 0.00

 (0.500) (0.518) (0.500) (0.795)

* Significantly different from zero at the .05 level.

^ Significantly different from zero at the .10 to > .05 level.

1 The difference between the estimates of the treatment and control groups’ rate of growth, for the average student who is one standard deviation below the mean for the risk 

measure.

2 The difference between the estimates of the treatment and control groups’ rate of growth, for the average student who is at the mean for the risk measure.

3 The difference between the estimates of the treatment and control groups’ rate of growth, for the average student who is one standard deviation above the mean for the risk 

measure.

4 Estimated difference between the treatment and control groups in the incremental change in the rate of growth, with each unit increase in the risk measure.

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using HLM 6.06. Sample weights were used in all analyses to (1) give 

each program equal weight within each time period, (2) give each school equal weight in each program (within each time period), and (3) give each time period equal weight in the 

analysis. 

SOURCE: The Social and Character Development (SACD) Research Program.

151

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.46. Impacts on growth of child outcomes, by child behavior risk as reported by primary caregiver 

Primary caregiver-reported child behavior risk

(Mean = 3.43; Standard deviation = 2.01)

Impact on growth at the specified risk level (p-value)

Scale–Report 

One standard 

deviation below mean1 Mean2

One standard 

deviation above mean3

Interaction term4

(p-value)

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) -0.01 0.00 0.01 0.00

 (0.500) (0.716) (0.500) (0.929)

 Normative Beliefs About Aggression–CR (-) 0.02 0.03* 0.04* 0.00

 (0.200) (0.036) (0.028) (0.416)

 Empathy–CR (+) -0.01 -0.02 -0.02^ 0.00

 (0.299) (0.114) (0.088) (0.490)

Behavior Domain

 Altruistic Behavior–CR (+) -0.01 0.00 0.00 0.00

 (0.500) (0.803) (0.500) (0.854)

 Altruistic Behavior–PCR (+) 0.02 0.01 0.00 0.00

 (0.500) (0.531) (0.500) (0.472)

 Altruistic Behavior–TRS (+) -0.04 -0.03 -0.03 0.00

 (0.169) (0.246) (0.315) (0.319)

 Positive Social Behavior–PCR (+) 0.01 0.02^ 0.02* 0.00

 (0.500) (0.088) (0.042) (0.178)

 Positive Social Behavior–TRS (+) -0.01 -0.01 -0.02 0.00

 (0.500) (0.577) (0.500) (0.900)

 Problem Behavior–CR (-) 0.01 0.02 0.02 0.00

 (0.500) (0.246) (0.164) (0.429)

 Problem Behavior–PCR (-) -0.01* -0.01 0.00 0.00

 (0.044) (0.419) (0.500) (0.157)

 Problem Behavior–TRS (-) 0.00 0.00 0.00 0.00

 (0.500) (0.886) (0.500) (0.724)

 ADHD-Related Behavior–TRS (-) 0.00 -0.01 -0.02 -0.01

 (0.500) (0.620) (0.500) (0.113)

See notes at end of table.

152

Chapter 1. The Social and Character Development Multiprogram Evaluation
Table 1.46. Impacts on growth of child outcomes, by child behavior risk as reported by primary caregiver—Continued

Primary caregiver-reported child behavior risk

(Mean = 3.43; Standard deviation = 2.01)

Impact on growth at the specified risk level (p-value)

Scale–Report 

One standard 

deviation below mean1 Mean2

One standard 

deviation above mean3

Interaction term4

(p-value)

Academics Domain

 Engagement with Learning–CR (+) -0.01 -0.01 0.00 0.00

 (0.500) (0.646) (0.500) (0.711)

 Academic Competence and Motivation–TRS (+) -0.01 0.00 0.01 0.01

 (0.500) (0.837) (0.500) (0.332)

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) -0.01 -0.03 -0.05* -0.01

 (0.500) (0.126) (0.039) (0.173)

 Negative School Orientation–CR (-) 0.00 0.00 0.00 0.00

 (0.500) (0.902) (0.500) (0.898)

 Student Afraid at School–CR (-) 0.02 -0.01 -0.03 -0.01

 (0.500) (0.791) (0.287) (0.144)

 Victimization at School–CR (-) -0.01 -0.01 -0.02 0.00

 (0.500) (0.469) (0.500) (0.751)

* Significantly different from zero at the .05 level.

^ Significantly different from zero at the .10 to > .05 level.

1

The difference between the estimates of the treatment and control groups’ rate of growth, for the average student who is one standard deviation below the mean for the risk measure.

2 The difference between the estimates of the treatment and control groups’ rate of growth, for the average student who is at the mean for the risk measure.

3 The difference between the estimates of the treatment and control groups’ rate of growth, for the average student who is one standard deviation above the mean for the risk measure.

4 Estimated difference between the treatment and control groups in the incremental change in the rate of growth, with each unit increase in the risk measure.

NOTE: Abbreviations are 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. Sample weights were used in all analyses to (1) give each program equal weight in the calculation of pooled 

impact estimates and (2) give each school equal weight in each program. 

SOURCE: The Social and Character Development (SACD) Research Program.

153

Chapter 1. The Social and Character Development Multiprogram Evaluation
Chapter 1. The Social and Character Development Multiprogram Evaluation 

154

Table 1.47. Sample size ranges for outcome analyses 

Year 1 Year 2 Year 3

Outcome analysis Group Report

Minimum of 

range

Maximum of 

range

Minimum of 

range

Maximum of 

range

Minimum of 

range

Maximum of 

range

Combined program Treatment CR 2,181 2,190 2,143 2,162 2,065 2,081

PCR 1,728 1,760 1,668 1,691 1,452 1,473

TRS 2,205 2,252 2,198 2,247 2,073 2,110

TRCS 436 437 422 423 422 423

Control CR 1,939 1,961 1,869 1,892 1,894 1,903

PCR 1,587 1,611 1,475 1,493 1,363 1,383

TRS 1,950 2,025 1,976 1,989 1,921 1,965

TRCS 408 409 403 404 392 393

Individual programs

 ABC Treatment CR 245 257 234 235 242 245

PCR 194 197 168 171 167 167

TRS 248 250 233 245 261 262

TRCS 54 54 49 49 51 51

Control CR 292 297 308 311 302 307

PCR 217 224 229 229 214 219

TRS 287 305 322 322 322 323

TRCS 66 66 67 67 64 64

 CSP Treatment CR 313 319 312 377 280 288

PCR 239 245 243 246 211 211

TRS 324 325 302 404 265 266

TRCS 50 52 57 91 62 62

Control CR 323 335 306 299 324 329

PCR 279 280 257 209 231 231

TRS 329 344 320 333 324 326

TRCS 50 50 60 82 60 61

 LBW Treatment CR 345 347 338 339 331 332

PCR 298 302 293 296 253 257

TRS 380 382 358 366 355 358

TRCS 79 79 68 69 75 75

Control CR 264 265 238 240 220 221

PCR 229 230 191 191 184 187

TRS 273 284 250 252 240 242

TRCS 62 62 47 47 50 50

 PA Treatment CR 296 298 280 284 258 259

PCR 239 244 212 214 168 170

TRS 275 304 289 302 242 262

TRCS 49 50 43 43 39 39

Control CR 277 280 220 225 241 242

PCR 227 231 181 183 158 158

TRS 280 302 249 251 215 250

TRCS 45 45 39 39 38 38

See notes at end of table.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

155

Table 1.47. Sample size ranges for outcome analyses—Continued

Year 1 Year 2 Year 3

Outcome analysis Group Report

Minimum of 

range

Maximum of 

range

Minimum of 

range

Maximum of 

range

Minimum of 

range

Maximum of 

range

 PATHS Treatment CR 216 219 220 222 225 227

PCR 191 198 181 186 169 174

TRS 232 232 228 229 228 230

TRCS 57 58 55 55 50 50

Control CR 219 220 212 212 202 204

PCR 179 180 166 168 161 164

TRS 224 225 219 219 197 208

TRCS 51 51 57 57 60 60

 4Rs Treatment CR 401 403 372 377 350 355

PCR 265 270 241 246 177 183

TRS 387 403 399 404 360 361

TRCS 87 88 90 91 87 88

Control CR 301 303 292 299 306 309

PCR 221 227 206 209 169 174

TRS 303 306 307 320 321 322

TRCS 85 85 82 82 75 75

 SS Treatment CR 358 361 383 387 375 376

PCR 302 305 330 335 307 311

TRS 357 361 381 383 357 376

TRCS 58 58 59 60 58 58

Control CR 259 261 288 291 293 297

PCR 235 239 245 255 246 250

TRS 254 263 292 293 296 298

TRCS 49 50 51 52 45 45

Subgroups

 Gender Boys CR 1,983 1,998 1,913 1,933 1,899 1,915

PCR 1,594 1,662 1,503 1,526 1,325 1,349

TRS 2,010 2,067 2,027 2,048 1,927 1,965

Girls CR 2,135 2,153 2,099 2,121 2,060 2,069

PCR 1,688 1,716 1,626 1,645 1,473 1,490

TRS 2,108 2,174 2,132 2,174 2,046 2,090

 New entrants Stayers CR † † 3,090 3,124 2,764 2,781

PCR † † 2,444 2,470 1,953 1,976

TRS † † 3,230 3,275 2,809 2,862

New 

 entrants CR † † 921 931 1,196 1,203

PCR † † 699 714 862 879

TRS † † 944 962 1,185 1,217

 Initial socioeconomic risk CR 3,292 3,311 3,649 3,693 3,565 3,584

PCR 2,871 2,916 3,076 3,113 2,737 2,775

TRS 3,312 3,411 3,799 3,843 3,588 3,662

See notes at end of table.

 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

156

Table 1.47. Sample size ranges for outcome analyses—Continued

Year 1 Year 2 Year 3

Outcome analysis Group Report

Minimum of 

range

Maximum of 

range

Minimum of 

range

Maximum of 

range

Minimum of 

range

Maximum of 

range

 Initial family risk CR 3,373 3,392 2,749 2,780 2,459 2,473

PCR 2,893 2,939 2,297 2,322 1,834 1,854

TRS 3,382 3,483 2,873 2,901 2,497 2,547

 Initial community risk CR 3,289 3,309 2,702 2,734 2,413 2,427

PCR 2,839 2,884 2,266 2,290 1,813 1,830

TRS 3,298 3,396 2,820 2,857 2,450 2,499

 Initial child behavior 

 risk (Teacher Report) CR 3,608 3,630 2,936 2,970 2,631 2,647

PCR 2,940 2,988 2,341 2,364 1,870 1,893

TRS 3,567 3,763 3,072 3,111 2,679 2,729

 Initial child behavior 

 risk (Primary 

 Caregiver Report) CR 3,353 3,372 2,732 2,764 2,451 2,464

PCR 2,876 2,923 2,289 2,313 1,828 1,847

TRS 3,365 3,466 2,858 2,886 2,487 2,537

 Fidelity of 

 implementation High CR 2,020 2,038 1,617 1,635 1,224 1,235

PCR 1,675 1,697 1,323 1,340 908 926

TRS 2,018 2,096 1,654 1,679 1,207 1,251

Mixed CR † † 848 851 1,444 1,450

PCR † † 677 684 1,036 1,052

TRS † † 893 904 1,469 1,485

Low CR 2,100 2,113 1,544 1,565 1,289 1,298

PCR 1,640 1,675 1,140 1,158 870 877

TRS 2,137 2,182 1,624 1,652 1,317 1,338

† Not applicable.

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love in a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 1. The Social and Character Development Multiprogram Evaluation 

157

Sensitivity Analyses 

To implement the growth curve analyses, it was necessary to make decisions about key model parameter 

specifications and estimation methods. A set of sensitivity tests using variations in key model assumptions 

examined the original growth curve findings and found them robust. 

Time Metric 

Age of student provides a different time metric by which to measure the growth trajectory. The growth curve 

analysis was redone using age in years at the start of implementation of the program. The 18 impact estimates 

remained insignificant when using age as the time metric. 

Weights 

Weights were used to represent each program equally within each survey period, each school equally within 

each program, and each survey period equally within the analysis. To determine whether the weights affected 

the outcomes, an unweighted analysis was conducted. The 18 impact estimates remained insignificant in the 

unweighted model. 

Fitted Impacts 

―Fitted‖ impact estimates and significance levels were examined at follow-up points corresponding to the 

point-in-time analyses at 6 months, 12 months, 18 months, and 30 months after implementation, which 

correspond roughly to the times of the data collection efforts. Based on the fitted growth curve model, 

statistically significant impact estimates were found for three outcomes (these data are not shown in a table).45

One was in a detrimental direction and two were in a beneficial direction. At 12, 18, and 30 months after 

implementation, the seven SACD programs, together, lowered growth of child-reported Altruistic Behavior 

by .05 (significant at the .01 level), 0.05 (significant at the .01 level), and 0.06 (significant at the 0.05 level), 

respectively. At 12 months, 18 months, and 30 months after implementation, the SACD programs increased 

growth of parent-reported Positive Social Behavior by 0.03, 0.04, and 0.05, respectively (all significant at the 

.05 level). At 12 months, 18 months, and 30 months after implementation, the programs lowered growth of 

child-reported Victimization by 0.04, 0.05, and 0.06, respectively (all significant at the .05 level). 

Fall 2005 Data 

The second follow-up point in fall 2005 occurred shortly after the summer, which could affect the outcome 

measures at fall 2005 through ―learning loss‖ during the summer. The growth curve model was estimated 

using a time indicator that allows the growth coefficient to decline (or increase) following the summer. The 

18 impact estimates remained insignificant after accounting for summer ―learning loss.‖

Missing Outcomes 

An advantage of the growth curve methodology is that it can accommodate all students who have at least one 

round of data. The information that was available for each student was used in estimating the intercept and 

slope, across all students. To assess the robustness of study findings to the assumptions underlying this 

approach, the model was estimated using only those students with (1) all rounds of follow-up data, and 

(2) the last round of follow-up data in spring 2007 and at least one earlier round. The impact estimates 

remained insignificant when using these restricted samples. 

 

45 Point-in-time impacts were calculated by adding (1) the estimated impact of the treatment at initial data collection to 

(2) the estimated average annual impact of the treatment on the growth curve trajectory multiplied by the number of 

years since implementation. For the three significant outcomes, the estimated difference between the treatment and 

control groups at initial data collection, though insignificant, was in the same direction as the estimated slope. Thus, the 

cumulative impact of the estimated difference in the slope between the treatment group and control group, in addition 

to the estimated difference at initial data collection between the treatment group and control group, resulted in 

significant differences at later time points. However, the cumulative impact of the difference in the average annual slope 

between the treatment group and control group (that is, the estimated difference in the change in the outcomes of the 

treatment group and control group) resulted in no significant differences for any of the point-in-time estimates. 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

158

Discussion 

IES and DVP established the SACD Research Program to rigorously evaluate the effects of seven universal, 

schoolwide programs to improve students’ social and character development. The programs evaluated were 

chosen through a peer-review process of submitted applications. A separate experiment using random 

assignment of schools was done for each program. Within each program, participating schools were stratified 

into similar pairs then randomly assigned to receive the treatment of the intervention or continue with 

whatever SACD activities they were using. All students in the elementary grades within the treatment schools 

received the intervention program over the 3 years of the study. For the evaluation, one cohort of students 

was followed from the beginning of third grade through the end of fifth grade. A third party (Mathematica 

Policy Research, Inc.) provided technical assistance in the sampling design, helped design data collection, 

conducted and coordinated the data collection, and designed and conducted the analyses. Twenty outcomes 

were used to measure impacts on students and perceptions of school climate and these were grouped under 

four domains: Social and Emotional Competence, Behavior, Academics, and Perceptions of School Climate. 

The data were analyzed from all seven programs together and by individual program to determine the 

collective and individual program impacts. 

Under such a design, inferences can be made regarding the programs, combined and individually, as the cause 

of any impacts on these domains and the outcomes that define them. Rigorous inferences can be drawn only 

about the specific programs tested in the context (including the schools) in which they were tested. Other 

programs tested in similar or different settings and these programs tested in different settings might have 

yielded different findings. Therefore, inferences cannot be made beyond this set of programs and contexts. 

An analysis of the initial data found few significant differences between the family, teacher, and school 

characteristics of the treatment and control students, a sign that the randomization of the schools had created 

similar treatment and control groups (similar at least on the observed characteristics). Treatment teachers 

reported significantly greater use of and training in SACD activities than control teachers more often than 

would be expected by chance. Because the training of all treatment teachers and the implementation of six of 

the programs occurred before the initial data were collected, it could not be determined whether these 

differences resulted from actual baseline differences between treatment and control schools and teachers (i.e., 

that randomization did not create similar treatment and control groups) or from implementation occurring 

before the initial data collection. The control group for the SACD evaluation was a ―standard practice‖ 

control, not a ―no treatment‖ control, in that control teachers reported the use of SACD activities as well as 

related professional development. 

The SACD programs were found to have a statistically significant impact on the reported use of SACD 

activities in the classroom and school. Treatment teachers reported significantly greater implementation of 

activities than control teachers for 127 of the 249 activity outcomes measured over the 3 years. In addition, in 

each year four of the six SACD activity domains were found to be significantly positively affected by the 

SACD programs. These four domains included the use of any SACD activities, the use of SACD activities 

linked to named programs, the use of classroom materials and instructional strategies when teaching SACD 

activities, and professional development in using SACD activities. The actual differences in reported use of 

SACD activities between the treatment and control groups varied from 5 to 34 percentage points. The results 

provided evidence that implementing the SACD programs increased the reported use of SACD activities in 

the schools. 

To examine the average effects of the seven programs (combined and individually) on student outcomes, two 

sets of analyses were done. The first analyzed the outcomes on a year-by-year basis (from the fall of third 

grade to the spring of third grade, spring of fourth grade, and spring of fifth grade) for the entire sample, the 

individual programs, and a set of subgroups. The second analyzed the impacts on the average annual growth 

in student outcomes for the overall sample, the individual programs, and the subgroups. For both analyses, 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

159

the ability to detect smaller significant effects was greater when examining the programs all together versus 

examining the individual programs or the subgroups because of the differences in sample sizes. 

Neither the year-by-year analysis nor the growth curve analysis found that the seven SACD programs 

improved student outcomes when considered together, individually by program, or for specific subgroups. 

For the combined-program analysis, the year-by-year analysis found fewer significant impacts than expected 

by chance (2 out of 60 estimated impacts) and the growth analysis found no significant impacts. For the 

individual program analysis, the year-by-year analysis found fewer significant impacts than expected by chance 

(16 out of 420 estimated impacts) with 9 having beneficial impacts and 7 having detrimental impacts. The 

growth analysis found the same number of impacts as expected by chance (6 out of 126 estimated impacts) 

with 2 having beneficial impacts and 4 having detrimental impacts. For the subgroup analyses, the year-byyear analysis found more significant impacts than expected by chance for gender (8 out of 54 estimated 

impacts) and initial risk levels (41 out of 270 estimated impacts) but not for stayers versus new entrants. For 

gender, half of the significant impacts showed a beneficial impact of the intervention and half showed a 

detrimental impact. For initial risk level, 26 showed a beneficial impact of the intervention on high-risk 

students and 15 showed a detrimental impact on high-risk students. The growth analysis found fewer 

significant impacts than expected by chance for all the subgroups except those based on initial risk levels, for 

which 6 out of 90 estimated impacts were significant. Four of these 6 significant impacts were beneficial and 

2 were detrimental. In sum, the SACD multiprogram evaluation provides no evidence that the seven 

universal, schoolwide programs improved students’ social and character development. 

Several explanations for this finding can be considered: (1) failure of the conceptualization and design of the 

intervention, (2) weak implementation of the intervention, (3) nonsubstantial differences in the level of 

SACD activities in the treatment and control schools, and (4) methodological limitations of the evaluation. 

Failure of the conceptualization and design of the intervention refers to the possibility that the seven 

programs tested might not have altered students’ social and character development in the expected ways 

because the theories underlying them or the combinations of activities chosen to bring about the desired 

changes in students’ attitudes and behaviors were inadequate for the purpose. For example, one alternative 

view to that adopted in the SACD evaluation is that only a subset of elementary-age children has deficits in 

social behavior and character, and these deficits require a more targeted, more intensive intervention than 

schoolwide programs can provide. Therefore, if a school-based program is to be effective, a combination of 

schoolwide and targeted activities might be required for the intervention to make a significant difference in 

students’ outcomes (e.g., see Conduct Problems Prevention Research Group 1999). These kinds of 

explanations, if confirmed by other studies, would lead to focusing more effort on understanding how social 

and character development occurs among elementary-age children, how this development can be affected, 

and what types of practices in classrooms and schools can be used to bring about desired effects. 

Weak implementation of the intervention refers to the possibility that, although the intervention might have 

been well conceived and well designed, the treatment schools did not implement these practices effectively on 

average. Weakly implemented programs may not have positive impacts on students. The SACD evaluation 

examined seven different SACD programs, each with unique features, and the fidelity rating used measured 

fidelity of implementation relative to the targets established for each specific intervention. In this way, the 

ratings were standardized relative to each site’s program-specific benchmarks and could be compared across 

programs and years. The analysis of the fidelity data found little evidence of a relationship between high 

fidelity and more beneficial outcomes. The number of significant associations found between fidelity and 

beneficial outcomes was higher than expected by chance (5 versus 3 out of 54 estimated impacts) but 4 of the 

5 significant results were due to detrimental associations between low fidelity and outcomes rather than 

beneficial associations between high fidelity and outcomes. The approach used to obtain comparable fidelity 

ratings required two compromises. First, it could not account for differences among the programs’ 

implementation standards—for example, whether programs differed in how difficult they were to implement. 

Second, it provided no information on why implementation was of a certain quality. As a result, the SACD 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

160

evaluation fidelity measure may not provide adequate information regarding the issue of whether low fidelity 

might have been the reason behind the lack of significant findings. The fidelity measures used by each 

research team using team-chosen criteria may provide additional information on how well each program was 

implemented in each treatment school. 

Nonsubstantial differences in the level of SACD activities refers to the possibility that the implementation 

differences between the treatment and control schools were not great enough to generate statistically 

significant differences in student outcomes. Like the treatment schools, the control schools joined the study 

with a willingness to implement a SACD program, showing a willingness to promote social and character 

development. In addition, some of the sites were located in states where legislation required or promoted 

such activities. The control group represented ―standard practice,‖ which included the reported use of SACD 

activities in the classroom. For example, 86 percent to 90 percent of control teachers reported using activities 

to promote any one of the six SACD goals. While a statistically significant larger percentage of teachers in the 

treatment schools reported conducting such activities (95% to 96%) the 5- to 10-percentage-point differences 

may not have been large enough to lead to improved student outcomes. At the same time, the significant 

differences between treatment and control teacher reports were larger for other responses regarding the use 

of SACD activities. For example, the differences between treatment and control teachers regarding the use of 

activities from named programs were 29 to 34 percentage points across the 3 years. These results, plus the 

finding that treatment teachers reported greater use of some instructional materials and methods to promote 

SACD goals, provide evidence that the treatment teachers were making a more intensive effort to promote 

social and character development. 

There are three methodological limitations of the evaluation that may have contributed to the finding of no 

impacts on student outcomes. First, the evaluation relied on self-reported data by teachers and principals 

regarding the use of SACD activities. No observational studies were done to validate these reports. If 

treatment teachers over-reported their use of SACD activities (possibly because they felt an expectation to 

report high use given that a SACD program was being implemented in their school), the impacts of the 

treatment could be misestimated; that is, if there really were no differences in the level of SACD activities 

between the treatment and control groups then a lack of effects might be expected. The percentages of 

treatment teachers (95% to 96%) who reported using any activities to promote one of the six SACD goals 

differed from the percentages who reported using activities from named programs (68% to 72%) for the 

same purpose. This difference does not rule out the possibility of systematic over-reporting; however, it does 

suggest that some teachers were candid in their reporting on their use of the treatment programs. The 

research teams used observations or product measures to check implementation of their specific programs, 

and the results from this work may provide additional evidence about the potential importance of overreporting of implementation by treatment teachers. 

A second methodological limitation was that student-provided data (used for 12 of the 20 outcomes) were 

not available for 36 percent to 39 percent of students, depending on the year, because parents did not provide 

written consent or students did not assent to take part in the study. Primary caregiver data, used for three 

outcomes, were not available for 49 percent to 54 percent of students. It is possible that students included in 

the study differed from those not included due to an absence of data. As the study did not collect descriptive 

data on the nonobserved students, the existence of such differences could not be determined, and how the 

inclusion of these students in the study would have affected the findings is unknown. Given the few 

statistically significant and substantively important impacts found with the existing sample, there would need 

to be a large and consistent impact on the nonobserved students (had they been included) to change the 

findings. For example, because two-thirds of the population who were observed received an average impact 

of zero, the nonobserved one-third would need to have received an average impact of nearly one-third of a 

standard deviation to bring the overall mean to one-tenth of a standard deviation unit. In addition, because 

the subgroup analyses did not find systematically significant impacts, there is no evidence that should these 
Chapter 1. The Social and Character Development Multiprogram Evaluation 

161

missing students come from one of the subgroups (e.g., higher initial risk) their inclusion would change the 

findings. 

A third methodological limitation was the sample size for the individual program evaluations. The combined 

analysis of all seven programs provided a sample size sufficient to detect student-level impacts at MDES 

ranging from 0.03 to 0.23 standard deviations (with more than 75% of them below 0.10 standard deviations). 

The power to detect impacts at the level of the individual programs was more limited (the MDES ranged 

from 0.1 to 0.9 over the 3 years) and individual program-level effects might have been missed. To address this 

limitation at the program level, nonsignificant impacts of at least 0.25 standard deviations were identified as 

substantively important results and only a small number were identified, with an almost equal division into 

beneficial and detrimental effects. 

The seven programs tested in the SACD evaluation were similar in being coherent, universal, school-based 

programs to promote social and character development of third- to fifth-grade students. They were diverse 

both in their specific goals and in their approaches to promoting social and character development for all 

students. In addition, they were evaluated in diverse types of locales in which schools served communities 

with very different ethnic and socioeconomic compositions. On average, the seven programs did not improve 

students’ social and emotional competence, behavior, academic achievement, and student and teacher 

perceptions of school climate. In addition, although the numbers of schools and students in each program 

were not always sufficient to draw firm conclusions at the program level, the patterns of estimated impacts 

for each program were largely similar: students’ outcomes were not affected.
This page is intentionally blank. 
163

Chapter 2. Academic and Behavioral Competencies Program 

University at Buffalo, The State University of New York

(Buffalo Site)

Intervention 

Researchers at the University at Buffalo, The State University of New York (SUNY) (Buffalo site) 

implemented and evaluated the Academic and Behavioral Competencies Program (ABC) (Pelham, Fabiano, and 

Massetti 2005; Smith et al. 2007; Waschbusch, Pelham, and Massetti 2005). ABC was developed at the Center 

for Children and Families at the University at Buffalo, SUNY. This program is designed to develop and 

maintain a positive learning environment that reduces misbehavior and promotes positive skills and 

competencies. It focuses on classroom behavior management and teaching and promoting social skills and 

includes multiple components. Table 2.1 describes ABC’s general characteristics (panel 1), the types of 

instruction and strategies used (panel 2), the professional development provided for those implementing the 

program (panel 3), and the social and character development activities (panel 4) and outcomes (panel 5) 

addressed by the program. 

 ABC components include schoolwide rules, a response/cost system for tracking and responding to 

disruptive student behavior (e.g., behavior color charts), a daily home note or communication tool 

for students who follow the school rules, a weekly reinforcement activity (Fun Friday), and a teacherled social skills program. 

 ABC program staff conduct a program development workshop with a school committee of 

representative teachers, staff, and administrators during the summer prior to initial implementation. 

The workshop is intended to discuss components of the program and to modify the program within 

the designed framework to meet the needs of the school. The committee develops a set of 

schoolwide rules and consequences that form the backbone of the program’s implementation in the 

school. 

 Teachers and school staff are trained in behavior management skills and strategies and on delivery of 

the ABC components and structure. Teacher training covers general classroom behavior 

management skills for reducing disruptive behavior and promoting behavioral competencies and 

strategies specific to ABC. Teachers are taught skills such as building positive relationships with 

students, using labeled praise as social reinforcement, setting appropriate limits, and implementing 

effective discipline strategies. 

 Consistent schoolwide rules are posted and monitored. Students who follow the rules earn daily 

notes, which are used to earn weekly Fun Friday activities and other events and privileges. 

 Behavioral consultants are a standard feature of the ABC program. They spend 20 hours per week at 

the school and provide ongoing observations and consultations with teachers on the use of behavior 

management strategies and ABC components. Consultation with teachers ranges from discussing 

general program implementation to training on how to make program modifications to address the 

needs of children who could benefit from more intensive implementation of the program. 

 Each day for 10 minutes, teachers implement scripted social skills training in the classroom. In these 

lessons, they illustrate the targeted social skill that is intended to enhance problem solving, 

communication, and cooperation, and then encourage practice among students. Teachers have 
Chapter 2. Academic and Behavioral Competencies Program 

164

opportunities to reinforce and promote these social skills throughout the day using ―caught being 

good‖ tickets and prizes.

 The behavioral consultants at the schools also deliver parenting workshops. Seven workshops are 

offered that address general parenting topics such as building positive relationships with children and 

using effective discipline strategies. 

 A targeted recreational/peer skills program is delivered to schools for students who exhibit high rates 

of disruptive behavior in the school and classrooms. The recreational program takes place either 

during the school day or after school, and it focuses on building social competencies, enhancing rulefollowing behavior, and building positive relationships with peers. 

 Individualized programs, typically in the form of modifications of the classwide program (e.g., more 

frequent feedback, changes in goal criteria or rewards) and daily report cards, are developed and 

implemented through consulting with teachers about students who have high rates of disruptive 

behaviors. 
Chapter 2. Academic and Behavioral Competencies Program 

165

Table 2.1. Academic and Behavioral Competencies Program

Panel 1: General characteristics

Target population

 Universal and targeted

Program components

 Peer: In and out of class

 Parent: Daily positive notes and monthly parenting workshops

 Classroom: Teacher training and coaching, behavior management, and lessons

 Schoolwide: Planned events, reward programs, schoolwide rules, and discipline policies

 Community: None or not major focus

 Training: Pretraining and ongoing behavior consultation through school-based consultant

Level of integration

 Add-on curriculum and classroom and schoolwide activities

Flexibility

 Manualized: Manual includes modules for all program components (training, universal, and targeted levels); 

 manual adapted for each school with input from school committees

 Adaptability: Program may be individualized within the framework at the school, classroom, and individual child

 level

See notes at end of table.

Panel 2: Description of instruction and strategies

Classroom

 Lessons

 Who delivers: Teacher

 Activities and tools: Social skill of the day description and classroom or schoolwide role-play

 Content: Social skills, problem solving, classroom management, and peer relationships

 Frequency: Social skills training: daily 5- to 10-minute lessons; reinforcement and application: daily

 throughout the day; recreational peer program: 45 minutes two times per week

 Strategies

 Who delivers: Teacher (universal); consultants and/or counselors (targeted)

 Activities and tools: Schoolwide rules; social reinforcement for positive behavior; classroom management

 procedures; teacher training in classroom management, effective discipline, and building teacher- 

 child relationships; rewards for rule-following and consequences for negative behavior (e.g., time out) 

 Frequency: Daily

Supplement to classroom

 Behavioral consultant services to teachers/classrooms; peer mediation; discipline/office referral policies and 

 procedures; parenting workshops; targeted peer program for disruptive children

Schoolwide activities 

 Schoolwide rules; classroom management procedures; behavior honor roll; schoolwide recognition and 

 incentives; schoolwide discipline policies

See notes at end of table.
Chapter 2. Academic and Behavioral Competencies Program 

166

Table 2.1. Academic and Behavioral Competencies Program––Continued

Panel 3: Professional development

Pre-implementation

 Teachers

 Content: Teacher training in classroom behavioral management concepts and strategies; development of 

 schoolwide rules, discipline policies and procedures, and classroom management policies with school 

 committee 

 Duration: 3 half-days (9 hours total)

 Other

 Content: Staff, same as teacher training

 Duration: 3 half-days (9 hours total)

Ongoing consultation

 Teachers

 Content: Teacher consultation; coaching on use of classroom management; school consultation on policies

 and procedures

 Duration: Monthly (at minimum) or more frequently per teacher or school request

 Other

 Content: Consultation on school discipline policies and procedures, such as lunchroom discipline, bus

 discipline, hallway policies, etc.

 Duration: As needed, with regular follow-up

See notes at end of table.

Panel 4: Activities for SACD goals

Violence prevention and peace promotion  Risk prevention and health promotion

Social and emotional development  Civic responsibility and community service

Character education Behavior management 

Tolerance and diversity

See notes at end of table.

Panel 5: SACD outcomes addressed 

Engagement with Learning Empathy

Academic Competence and Motivation Positive School Orientation

Altruistic Behavior Negative School Orientation

Positive Social Behavior  Student Afraid at School

Problem Behavior  Victimization at School 

Self-Efficacy for Peer Interactions Feelings of Safety 

Normative Beliefs About Aggression Student Support for Teachers 

NOTE: Abbreviations are

 : Activity or outcome addressed

 Blank cell: Activity or outcome not addressed

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 2. Academic and Behavioral Competencies Program 

167

Sample and Random Assignment 

The Buffalo research team recruited a total of 10 public and 2 charter elementary schools. The 12 schools 

were in 3 school districts in upstate New York. The schools were randomly assigned to treatment and control 

conditions prior to the fall 2004 initial data collection.46 A two-step process was used. First, a computergenerated pairwise matching algorithm developed by Mathematica Policy Research, Inc. (MPR) identified the 

best pairwise matches across the 12 schools based on variables identified by the Buffalo research team. The 

variables included (a) school district, (b) percentage of students eligible for free or reduced-price lunch during 

the 2003-04 school year, (c) total number of students enrolled during the 2003-04 school year, (d) percentage 

of minority children enrolled during the 2003-04 school year, and (e) percentage of students achieving 

mastery level or better scores on New York’s fourth-grade English language arts standardized achievement 

test administered during the 2002-03 school year. Second, using a random numbers table, 1 school in each 

matched pair was assigned to either the intervention or the control condition, resulting in 6 schools receiving 

the ABC program and 6 schools acting as control schools. Control schools in this study were a ―business as 

usual‖ comparison, meaning these schools implemented the social and character development activities that 

constituted their standard practice. The level of social and character development practices, including 

activities, materials, teacher strategies, and professional development activities, in both the treatment and 

control schools was documented in order to understand the nature of the comparison being made between 

the intervention schools and the standard practice control schools. Assignment to treatment or control 

condition at the school level limited the risk of contamination between treatment and control classrooms. All 

12 schools remained in the study for the full 3 years. One school converted to a K-8 school but this did not 

affect the students in the study. 

The original student sample (the cohort of students in the third grade in the 12 schools in fall 2004) 

numbered 879 students (380 treatment and 499 control). Table 2.2 documents the change in the sample over 

the three spring follow-up data collection periods. Over time, new entrants to the cohort became a larger 

percentage of the sample, eventually making up 32 percent of the sample by the spring of Year 3. Only in 

Year 1 was there a statistically significant difference between the treatment and control groups in the number 

of new entrants (in this case, fewer new entrants in the treatment group). The percentage of the sample made 

up of the original cohort further declined as students left the schools. By Year 3, approximately 33 percent of 

the original sample had left. There were no significant differences between treatment and control leavers. 

 

46 In Year 2 (fall 2005), the Buffalo research team recruited two more schools, one assigned to treatment and one 

assigned to control, which were followed to the end of the study (spring 2007). A description of this second cohort and 

all relevant findings can be found in appendix A. The data from this second cohort are not included in the analyses and 

results reported in this chapter. 
Table 2.2. Sample––ABC 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Characteristic Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 School sample size 12 6 6 12 6 6 12 6 6 12 6 6

 Student sample size 879 380 499 875 373 502 877 367 510 871 353 518

Stayers † † † 836 363 473 719 308 411 590 245 345

New entrants † † † 39 10 29 158 59 99 281 108 173

 New entrants as a percent of

 spring enrollment † † † 4.5 2.7* 5.8 18.0 16.1 19.4 32.3 30.6 33.4

Total leavers (from original cohort) † † † 43 17 26 160 72 88 289 135 154

 Leavers as a percent of fall 2004 

 enrollment † † † 4.9 4.5 5.2 18.2 18.9 17.6 32.9 35.5 30.9

Number of students per school 

 (mean) 73 63 83 73 62 84 74 62 85 73 59 86

Range of number of students per

 school 48-115 48-84 64-115 47-118 47-83 66-118 41-131 41-80 63-131 45-125 45-77 49-125

† Not applicable.

* Treatment group significantly different from control group at the .05 level. 

SOURCE: The Social and Character Development (SACD) Research Program.

168

Chapter 2. Academic and Behavioral Competencies Program
Chapter 2. Academic and Behavioral Competencies Program 

169

Implementation 

Training 

The intervention teachers received 9 hours of program implementation training prior to the beginning of the 

school year (see panel 3 of table 2.1). If teachers were unable to attend this initial training due to scheduling 

conflicts, they were provided with make-up training by the program staff and behavioral consultants. 

Teachers had access to ongoing program implementation support throughout the school year from 

behavioral consultants. For this study, the behavioral consultants were master’s-level therapists or advanced 

graduate students. On average, behavioral consultants were at the participating schools for 20 hours each 

week, including scheduled activities such as administrative meetings, in-service presentations, and parent 

conferences. Consultation visits typically included classroom observations, an opportunity for teachers to ask 

questions about the curriculum, and implementation feedback from the consultant. In addition to the 

consultation visits, behavioral consultants and ABC staff provided approximately 15 hours per school of 

formal in-service training and group presentations throughout the year.

Data Collection 

MPR’s subcontractor, Decision Information Resources, Inc. (DIR), collected the child, teacher, and school 

data at the Buffalo site. Table 2.3 presents the data collection dates. Data were collected in the fall and spring 

of the first 2 years and the spring of Year 3. The fall 2004 (third-grade) data collection window began on 

October 18, 2004, and ended on November 12, 2004. The average time frame from the beginning of program 

implementation to the beginning of fall data collection was 6 weeks. As a result, initial data collection took 

place after ABC implementation began. Therefore, these data provide a measure of the students, teachers, 

and schools near the beginning of the school year, when ABC had been operating for a relatively short period 

of time. 

The spring data collection window was from April 11, 2005, to May 6, 2005. ABC had been implemented for 

31 weeks at the time of the spring data collection and for 21 weeks from the end of the fall data collection. 

Year 2 followed a similar pattern, with implementation occurring at the start of the school year, fall data 

collection occurring 5 weeks later, and spring data collection occurring 21 weeks after fall data collection (and 

29 weeks after the start of implementation). In spring 2007, data collection occurred 32 weeks after the start 

of implementation. Data collection took from 3 to 5 weeks at each collection point. 
Chapter 2. Academic and Behavioral Competencies Program 

170

Table 2.3. Data collection dates––ABC 

Data collection schedule

Year 1

(Fall 3rd grade)

Year 1

(Spring 3rd grade)

Year 2

(Fall 4th grade)

Year 2

(Spring 4th grade)

Year 3

(Spring 5th grade)

 School sample size 12 12 12 12 12

School year dates 

 First day of school 9/7/04 † 8/15/05 

9/7/052

† 8/15/05

9/6/062

 Start of implementation 9/8/04 † First day † First day

 Last day of school † 6/24/05 † 6/23/06 6/22/07

Data collection 

 Start 10/18/0414/11/05 10/10/0523/27/06 4/18/07

 End 11/12/04 5/6/05 10/27/05 5/1/06 5/8/07

Calendar weeks from 

 program implementation 

 to start of fall 2004 

 data collection 6 † † † †

Calendar weeks from 

 start of school to start of 

 fall 2004 data collection 6 † † 5

2

†

Calendar weeks from 

 end of fall data collection 

 to start spring data 

 collection † 21 † 21 †

Calendar weeks from 

 program implementation 

 to start of spring data 

 collection † 31 † 29 32

† Not applicable.

1 Data collection occurred at two schools on 9/21/04. The calculations that include the start of fall 2004 data collection use the 

10/18/04 date as the start date of the data collection.

2

The two charter schools started earlier than the public schools and had earlier data collection. 

NOTE: The dates provided in the table reflect the more common dates and are used to calculate the week’s data.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 2. Academic and Behavioral Competencies Program 

171

Consent Rates, Completion Rates, and Percentage of Sample With Data 

The actual number of child, primary caregiver, and teacher reports available for analysis was smaller than the 

number in the sample because consent and completion rates were less than 100 percent. Primary caregivers 

had to provide consent before children could complete the Child Report, before their child’s teacher could 

complete the Teacher Report on Student, and before they themselves completed the Primary Caregiver 

Report. Teachers also had to provide consent before completing the Teacher Report on Classroom and 

School. In addition, of those with consent, not all completed their respective reports. Table 2.4 shows the 

consent rates, completion rates, and percentages of sample with data for each of the four reports over the 3 

years. For the Child Report and the two teacher reports, completion rates ranged from 91 percent to 100 

percent. For the Primary Caregiver Report, the completion rates dropped over time from 90 percent to 64 

percent. 

The percentages of the sample with Child Report data ranged from 57 percent to 69 percent over the 3 years, 

with significantly higher percentages of treatment students having data in Years 1 and 3. The percentages of 

students with information from the Teacher Report on Student ranged from 56 percent to 74 percent, with 

significantly higher percentages of treatment students having data in Years 1 and 3. The percentages of 

students with data from the Primary Caregiver Report ranged from 42 percent to 60 percent and declined 

over time. The percentages of teachers with data from the Teacher Report on Classroom and School ranged 

from 78 percent to 92 percent. 
Table 2.4. Consent rates, completion rates, and percentage of sample with data from each report––ABC 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 Student sample size 879 380 499 875 373 502 877 367 510 871 353 518

Child Report (percent)

 Primary caregiver consent rate 63.7 68.7* 59.7 64.0 67.3 61.6 64.7 65.4 64.1 67.7 73.7** 63.7

 Student completion rate 93.2 91.2 95.0 97.5 98.0 97.1 96.3 95.8 96.6 93.6 93.8 93.3

 Students with data159.4 62.7* 56.7 62.5 66.0 60.0 62.3 62.7 62.0 63.4 69.1** 59.5

Primary Caregiver Report (percent)

 Primary caregiver consent rate 63.3 68.2* 59.5 63.9 67.3 61.4 64.7 65.4 64.1 67.7 73.7** 63.7

 Primary caregiver completion rate 89.4 88.0 90.6 75.3 78.1 73.1 70.9 70.0 71.6 65.4 64.2 66.4

 Primary caregivers with data156.6 60.0 53.9 48.1 52.5 44.8 45.8 45.8 45.9 44.3 47.3 42.3

Teacher Report on Student (percent)

 Primary caregiver consent rate 263.7 68.7* 59.7 64.0 67.3 61.6 64.7 65.4 64.1 67.7 73.7** 63.7

 Teacher completion rate 93.9 95.0 93.0 99.3 99.2 99.4 100.0 100.0 100.0 99.2 100.0* 98.5

 Students with data159.7 65.3** 55.5 63.7 66.8 61.4 64.7 65.4 64.1 67.2 73.7* 62.7

Teacher Report on Classroom and School 

 (3rd- to 5th-grade teachers) (percent)

 Teacher consent rate 89.9 96.7* 84.6 92.1 91.8 92.3 91.9 93.2 90.9 85.8 90.0 82.7

 Teacher completion rate 95.2 94.9 95.5 91.4 91.1 91.7 93.6 90.9 95.7 95.0 96.3 94.0

 Teachers with data185.6 91.8 80.8 84.2 83.6 84.6 86.0 84.7 87.0 81.6 86.7 77.8

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

1 Calculated as consent rate x completion rate.

2 The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

172

Chapter 2. Academic and Behavioral Competencies Program
Chapter 2. Academic and Behavioral Competencies Program 

173

Responses from students in the original cohort (stayers) and new entrants in the ABC sample were examined 

to investigate possible differences between the two groups in consent rates, completion rates, and the 

percentages of sample with data that might affect outcome data (table 2.5). There were four statistically 

significant differences between the completion rates of these two groups across the 3 years: in Year 1, stayers 

completed more Child Reports than did new entrants (98% versus 85%); in Years 2 and 3, new entrants 

completed more Primary Caregiver Reports than did stayers (84% versus 68% and 75% versus 62%, 

respectively); and in Year 3, teachers of stayers completed more Teacher Reports on Students than did 

teachers of new entrants (100% versus 97%). 
Table 2.5. Consent rates, completion rates, and percentage of sample with data: Stayers versus new entrants––ABC 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total Stayers

New 

entrants Total Stayers

New

entrants Total Stayers

New 

entrants

 Student sample size 875 836 39 877 719 158 871 590 281

Child Report (percent)

 Primary caregiver consent rate 64.0 63.8 69.2 64.7 65.2 62.0 67.7 68.8 65.5

 Student completion rate 97.5 98.1*** 85.2 96.3 96.4 95.9 93.6 93.3 94.0

 Students with data162.4 62.6 59.0 62.3 62.9 59.5 63.4 64.2 61.6

Primary Caregiver Report (percent)

 Primary caregiver consent rate 63.9 63.8 66.7 64.7 65.2 62.0 67.7 68.8 65.5

 Primary caregiver completion rate 75.3 76.0 61.5 70.9 68.2** 83.7 65.8 61.8** 74.5

 Primary caregivers with data148.1 48.4 41.0 45.8 44.5 51.9 44.5 42.5 48.8

Teacher Report on Student (percent)

 Primary caregiver consent rate264.0 63.8 69.2 64.7 65.2 62.0 67.7 68.8 65.5

 Teacher completion rate 99.3 99.4 96.3 100.0 100.0 100.0 99.2 100.0*** 97.3

 Students with data163.5 63.4 66.7 64.7 65.2 62.0 67.2 68.8 63.7

* Stayers significantly different from new entrants at the .05 level.

** Stayers significantly different from new entrants at the .01 level.

*** Stayers significantly different from new entrants at the .001 level.

1 Calculated as consent rate x completion rate.

2 The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

174

Chapter 2. Academic and Behavioral Competencies Program
Chapter 2. Academic and Behavioral Competencies Program 

175

Fidelity of Implementation

Each year, the six ABC treatment schools were independently rated for quantity and quality of program 

implementation by two raters from the research team. The global measure of fidelity for the multisite study 

was used; inter-rater reliability was measured with Cronbach’s alpha (1.00, 0.98, and 0.99) in all 3 years. The 

ratings were combined into a single consensus rating and used to identify schools with high implementation 

fidelity. In Year 1, four treatment schools were identified as having high fidelity, and in Years 2 and 3, three 

schools were identified as having high fidelity. Cohen’s kappa was used as the measure of agreement when 

identifying schools as high fidelity, and it equaled 1.00 in Years 1 and 2, and 0.67 in Year 3. 

Initial Characteristics 

The initial characteristics of the students, teachers, and schools participating in the evaluation of the ABC 

program were collected from students enrolled in the third grade at the study schools in fall 2004, as well as 

from their primary caregivers and third-grade teachers. In addition, third-, fourth-, and fifth-grade teachers 

and principals in the study schools provided information about activities related to social and character 

development in these schools. Documenting the characteristics of students, teachers, and schools, and their 

status on key outcomes at a point before the intervention had been operating for an extended period, helps to 

determine whether the random assignment of schools to treatment and control status produces groups with 

similar distributions of observed characteristics. As noted in the following discussion, 6 significant differences 

(out of 62 comparisons, with 3 expected to be significant by chance) were found between the treatment and 

control students, teachers, and schools in the level of SACD activity in the classroom and school. 

Characteristics of Children, Their Families, and Communities 

There were no significant differences in the observed student, caregiver, and community characteristics 

between the treatment and control groups (table 2.6). For students, the mean age was 8 years. The sample 

contained almost equal percentages of girls and boys. The sample was ethnically diverse: White non-Hispanic 

students made up 33 percent of the sample, Black non-Hispanic students made up 41 percent, and Hispanic 

students made up 17 percent. 

The sample was also diverse in levels of family income, education levels of primary caregivers of the children 

in the sample, and family situation. Fifty percent of the children lived in a household with an income at 135 

percent of the federal poverty level or lower, which is the income threshold for eligibility for free school 

meals. Thirteen percent of primary caregivers did not complete high school. Half of the children lived with 

both their father and their mother. There were no significant differences between the treatment and control 

groups in these characteristics. 

The mean values of the outcomes for children’s behavior and attitudes as reported by the primary caregivers, 

children, and teachers at initial data collection in fall 2004 are shown in table 2.7. There were no significant 

differences between the treatment and control groups in these scores. 
Chapter 2. Academic and Behavioral Competencies Program 

176 

Table 2.6. Initial characteristics of children, their families, and communities––ABC 

Characteristic Total Treatment Control

 Student sample size 497 228 269

Student demographics

 Gender (percent)

 Male 50.1 46.4 53.9

 Female 49.9 53.6 46.1

 Race/ethnicity (percent)

 White (non-Hispanic) 32.7 26.2 39.2

 Black (non-Hispanic) 41.2 48.6 33.8

 Hispanic 17.3 16.9 17.6

 Other 8.8 8.3 9.4

 Age (in years) (mean) 8.0 8.0 8.0

Primary caregiver and family characteristics

 Primary caregiver’s age (in years) (mean) 35.3 35.9 34.7

 Primary caregiver’s race/ethnicity (percent)

 White (non-Hispanic) 38.3 31.3 45.3

 Black (non-Hispanic) 41.1 46.7 35.6

 Hispanic 14.9 15.4 14.3

 Other 5.7 6.6 4.8

 Primary caregiver’s education (percent)

 Did not complete high school 12.5 11.0 14.0

 Completed high school or equivalent 26.6 24.9 28.2

 Some college 49.4 52.6 46.2

 Bachelor’s or higher degree 11.5 11.4 11.7

 Primary caregiver’s employment (percent)

 Full-time 45.1 43.0 47.1

 Other 54.9 57.0 52.9

 Primary caregiver’s marital status (percent)

 Married 47.3 43.5 51.2

 Other 52.7 56.5 48.8

 Students who live in one household (percent) 93.8 94.0 93.6

 Number of individuals in household (mean) 4.6 4.5 4.7

 Primary caregiver’s relationship to child (percent)

 Mother (stepmother) 86.0 86.7 85.3

 Father (stepfather) 7.7 6.0 9.5

 Other relative/nonrelative 6.3 7.4 5.3

See notes at end of table.
Chapter 2. Academic and Behavioral Competencies Program 

177

Table 2.6. Initial characteristics of children, their families, and communities––ABC––Continued

Characteristic Total Treatment Control

 Student lives with (percent)

 Mother (stepmother) and father (stepfather) 50.4 46.6 54.2

 Mother (stepmother) only; father (stepfather) not present 42.5 44.8 40.3

 Father (stepfather) only; mother (stepmother) not present 1.8 2.1 1.6

 Other relative/nonrelative, parents not present 5.2 6.6 3.9

 Highest education of anyone in household (percent)

 Did not complete high school 8.7 5.2 12.1

 Completed high school or equivalent 23.3 22.5 24.0

 Some college 52.4 56.6 48.2

 Bachelor’s or higher degree 15.6 15.7 15.6

 Total household income (percent)

 Less than $20,000 40.7 39.8 41.6

 $20,000 to $39,999 34.0 35.5 32.4

 $40,000 to $59,999 16.2 17.9 14.5

 $60,000 or more 9.1 6.7 11.5

 Income-to-poverty-threshold ratio—Below 135 percent (percent) 49.7 48.0 51.4

 Income-to-poverty-threshold ratio—135 to 185 percent (percent) 25.2 26.9 23.4

 Income-to-poverty-threshold ratio—Above 185 percent (percent) 25.2 25.1 25.3

 Alabama Parenting Questionnaire––Poor Monitoring and 

 Supervision Subscale (mean) 1.1 1.2 1.1

 Alabama Parenting Questionnaire––Positive Parenting

 Subscale (mean) 3.5 3.5 3.5

 Confusion, Hubbub, and Order Scale (mean) 2.2 2.2 2.2

Community characteristics (mean)

 Community Risks Scale 1.7 1.7 1.7

 Community Resources Scale 2.8 2.8 2.8

 Child-Centered Social Control Scale 3.0 3.0 3.0

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 2. Academic and Behavioral Competencies Program 

178 

Table 2.7. Mean scores and standard deviations for initial outcome measures of sample––ABC 

Total Treatment Control

Outcome measure–Report Range Mean SD Mean SD Mean SD

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 1-4 3.0 0.7 3.0 0.7 3.0 0.7

 Normative Beliefs About Aggression–CR 1-4 1.3 0.5 1.2 0.5 1.3 0.5

 Empathy–CR 1-3 2.3 0.4 2.3 0.4 2.4 0.4

Behavior Domain

 Altruistic Behavior–CR 0-3 1.5 0.9 1.5 0.9 1.5 0.9

 Altruistic Behavior–TRS 1-4 1.3 0.4 1.4 0.4 1.3 0.4

 Altruistic Behavior–PCR 1-4 2.3 0.7 2.3 0.7 2.3 0.7

 Positive Social Behavior–TRS 1-4 2.9 0.7 2.9 0.7 2.9 0.7

 Positive Social Behavior–PCR 1-4 2.9 0.5 2.9 0.5 3.0 0.5

 Problem Behavior–CR 0-3 0.3 0.5 0.3 0.5 0.3 0.5

 Problem Behavior–TRS 1-4 1.4 0.5 1.4 0.4 1.4 0.5

 Problem Behavior–PCR 1-4 1.6 0.3 1.6 0.3 1.6 0.4

 ADHD-Related Behavior–TRS 1-4 1.8 0.6 1.8 0.6 1.8 0.6

Academics Domain

 Academic Competence and Motivation–TRS 1-5 2.8 0.8 2.7 0.9 2.8 0.8

 Engagement with Learning–CR 1-4 3.7 0.6 3.6 0.7 3.7 0.6

Perceptions of School Climate Domain

 Positive School Orientation–CR 1-4 3.2 0.7 3.2 0.7 3.3 0.7

 Negative School Orientation–CR 1-4 1.9 0.7 2.0 0.7 1.9 0.7

 Student Afraid at School–CR 1-4 2.3 1.0 2.4 1.0 2.3 1.0

 Victimization at School–CR 0-3 0.8 0.8 0.7 0.8 0.8 0.8

 Student sample size—PCR 497 228 269

 Student sample size—CR 521 238 283

 Student sample size—TRS 525 248 277

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

 SD: Standard deviation

No statistically significant differences were found between values for treatment and control groups. Weights, which assign equal 

weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests were 

conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at the 

school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 2. Academic and Behavioral Competencies Program 

179

Characteristics of Teachers and Schools 

The third-, fourth-, and fifth-grade teachers at the study schools were predominantly White non-Hispanic 

(80%) and female (90%), had an average of 12.3 years of total teaching experience, and about three-fourths 

(77%) held an advanced or specialist degree (table 2.8). There were no statistically significant differences 

between the treatment and control groups of teachers. 

Data regarding school characteristics were drawn from the Common Core of Data in order to compare 

treatment and control schools. There were no significant differences between the two groups of schools in 

terms of student composition (race/ethnicity and school lunch eligibility), number of students enrolled, 

number of full-time teachers, Title I status, or number of years the principal had been at the school (see table 

2.9). In addition, there were no significant differences between treatment and control schools in terms of 

location (urban, suburban, or rural) or lowest and highest grade offered (these data are not shown in a table). 
Chapter 2. Academic and Behavioral Competencies Program 

180 

Table 2.8. Initial characteristics of teachers in sample––ABC 

Characteristic Total Treatment Control

 Teacher sample size 119 56 63

Gender (percent)

 Male 9.7 9.4 10.1

 Female 90.3 90.6 89.9

Race/ethnicity (percent)

 White (non-Hispanic) 80.1 73.2 86.9

 Other 19.9 26.8 13.1

Number of years teaching experience (mean) 12.3 12.7 11.9

Number of years teaching experience in this school (mean) 4.9 4.5 5.3

Type of teaching certificate (percent)

 Regular state certificate or advanced professional certificate 65.7 63.4 68.0

 Other 34.3 36.6 32.0

Education (percent)

 Bachelor’s degree 23.0 24.0 22.0

 Advanced degree/other 77.0 76.0 78.0

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 2. Academic and Behavioral Competencies Program 

181

Table 2.9. Initial characteristics of schools in sample––ABC

Characteristic Total Treatment Control

 School sample size 12 6 6

Student race/ethnicity (percent)

 White (non-Hispanic) 34.3 29.1 39.6

 Black (non-Hispanic) 48.8 54.9 42.7

 Hispanic 14.8 14.6 15.0

 Other 2.0 1.4 2.7

Students eligible for free or reduced-price lunch (percent) 66.7 68.5 65.3

Number of students enrolled (mean) 679.2 584.3 774.0

Number of full-time teachers (mean) 46.5 41.6 51.5

Title I status (percent)

 Title I eligible school 83.3 83.3 83.3

 Schoolwide Title I 90.0 80.0 100.0

Number of years principal has been at this school (mean) 4.3 4.5 4.2

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: NCES Common Core of Data (2003-04), the Social and Character Development (SACD) Research Program.

In the Teacher Report on Classroom and School, teachers reported on nine dimensions of school 

environment (these data are not shown in a table): feelings of safety, adequacy of resources, student support, 

freedom to teach as desired, affiliation with and ties to colleagues, innovation regarding new approaches to 

teaching, professional interest, participatory decisionmaking, and work pressure. There were no statistically 

significant differences between treatment and control schools in these reports. 

The Level of SACD in the Schools Near the Beginning of the Study 

During the initial data collection period, principals and teachers reported on the SACD activities used in the 

schools and classrooms, the availability of SACD materials, and the professional development provided on 

SACD (see tables 2.10, 2.11, and 2.12, respectively). It is important to note that data collection at the 

beginning of the school year took place 6 weeks after the start of ABC implementation; therefore the SACDrelated activities and practices in the treatment and control schools are a measure of what was happening in 

the study schools near the beginning of the study. The principals at the 12 schools reported on activities to 

promote six social and character development goals (see table 2.10): violence prevention and peace 

promotion (83%), social and emotional development (92%), character education (100%), tolerance and 

diversity (75%), risk prevention and health promotion (83%), and civic responsibility and community service 

(83%). In addition, all 12 principals reported activities directed toward behavior management. There were no 

statistically significant differences between the treatment and control schools in the percentages of principal 

reports, although this may be due to the relatively small principal sample size. The percentages of teachers 

who reported the use of these activities in their classrooms ranged from 63 percent to 93 percent; there were 

no significant differences between treatment and control teachers. With respect to the use of schoolwide 

activities, 59 percent to 81 percent of teachers reported that their schools used such activities, and there were 

no significant differences between treatment and control teachers in reported use. 
Chapter 2. Academic and Behavioral Competencies Program 

182 

Table 2.10. Principal and teacher initial reports on use of SACD programs or activities in 

sample—ABC 

SACD program or activity Total Treatment Control

 Principal sample size 12 6 6

 Teacher sample size 119 56 63

Principals reporting that school had programs or activities 

 to promote the following SACD goals (percent)

 Violence prevention and peace promotion 83.3 66.7 100.0

 Social and emotional development 91.7 83.3 100.0

 Character education 100.0 100.0 100.0

 Tolerance and diversity 75.0 66.7 83.3

 Risk prevention and health promotion 83.3 83.3 83.3

 Civic responsibility and community service 83.3 83.3 83.3

 Behavior management 100.0 100.0 100.0

 None of the above 0.0 0.0 0.0

Teachers reporting on using programs or activities in their

 class to promote the following SACD goals (percent) 

 Violence prevention and peace promotion 74.7 82.1 67.2

 Social and emotional development 81.9 87.1 76.8

 Character education 88.5 93.1 83.8

 Tolerance and diversity 71.6 70.2 73.0

 Risk prevention and health promotion 64.1 66.7 61.6

 Civic responsibility and community service 63.0 71.1 54.9

 Behavior management 93.4 98.6 88.2

 None of the above 0.0 0.0 0.0

Teachers reporting schoolwide use of the following

 activities to promote SACD (percent)

 Morning announcements or videos 80.9 81.1 80.8

 School assemblies 76.2 70.4 82.1

 School newspapers or bulletins 64.4 67.6 61.3

 Special school days 59.2 66.6 51.8

 Special events 78.4 80.3 76.6

 Other activities 17.5 19.4 15.6

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within each of the programs and to each program across programs, were used in producing the 

treatment, control, and overall means. Statistical tests were conducted using regressions that included program indicators to 

account for the sample design and adjusted for clustering at the school level. Sample size may differ for some outcomes due to 

nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 2. Academic and Behavioral Competencies Program 

183

Teachers reported using a broad range of teaching materials (table 2.11). The materials most used in support 

of SACD activities were teacher guides (used by 59% of teachers), giveaways (used by 55%), children’s 

literature (used by 47%), and student materials (used by 46%). There were no significant differences between 

treatment and control teachers in the use of materials. 

Teachers reported using a wide variety of teaching strategies (table 2.11). All teachers in the sample reported 

using any of the 20 strategies, and, on average, they used 12 of the strategies. There were no significant 

differences between treatment and control teachers in the average number of strategies used. In regard to the 

specific strategies, there were 5 significant differences out of 29 comparisons made (with 1 expected to be 

significant by chance). A significantly greater number of treatment teachers reported incorporating SACD 

into the academic curriculum (87% versus 62%), sending home good behavior notes (93% versus 64%), 

presenting role models (76% versus 59%), using an honor roll for positive behavior (76% versus 45%), and 

using time-outs for negative behavior (94% versus 73%). These differences between treatment and control 

teachers at the start of the study may reflect the fact that program implementation at the treatment schools 

began before data collection at the start of the school year. 
Chapter 2. Academic and Behavioral Competencies Program 

184 

Table 2.11. Teacher initial reports on use of SACD materials and classroom strategies in 

sample––ABC

SACD material and classroom strategy Total Treatment Control

 Teacher sample size 119 56 63

Teachers using the following materials in conjunction with

 social and character development activities (percent)

 Teacher guides (manuals, curricula) 59.4 48.1 70.7

 Student materials (workbooks, worksheets) 46.2 46.1 46.2

 Instructional aids (games, software, videos) 26.8 29.3 24.3

 Giveaways (bookmarks, stickers) 55.3 59.5 51.2

 Children’s literature 47.3 52.3 42.3

 Other types of materials 11.4 11.2 11.7

 Do not use any of the materials listed above 13.3 14.3 12.4

Teachers using any of the strategies listed below to promote

 social and character development in the classroom (percent) 100.0 100.0 100.0

Number of strategies (listed below) used by teachers to promote

 social and character development in the classroom (mean) 11.7 12.7 10.8

Teachers using each of the following strategies to promote

 social and character development (percent)

 Role-playing 67.1 63.1 71.1

 Cooperative learning 99.4 100.0 98.7

 Peer group discussions 86.3 91.4 81.1

 Direct instruction of social and character development 82.9 88.8 77.0

 Skill training 47.6 46.2 49.0

 Incorporating social and character development into 

 academic curriculum 74.6 87.2** 62.0

 Parent training 6.0 5.1 6.8

 Parent/community involvement in program development

 or delivery 19.4 19.6 19.2

 Mentoring 30.9 36.1 25.6

 Good behavior notes sent home daily or weekly 78.2 92.6* 63.9

 Presenting role models 67.5 75.6* 59.3

 Targeted story reading or writing on SACD themes 72.6 77.3 67.9

 Peer mediation 36.0 36.9 35.1

 Honor roll for positive behavior 60.0 75.5* 44.6

 Pledges or recitations on social and character 

 development themes 41.1 37.8 44.4

 Guided visualization 45.1 49.9 40.3

 Student-led/student-assisted instruction 42.3 41.4 43.2

 Journaling 70.5 80.3 60.7

 Time out for negative behavior 83.6 94.0* 73.2

 Daily or weekly rewards for positive behavior 93.7 98.1 89.2

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 2. Academic and Behavioral Competencies Program 

185

Principals and teachers reported on participation in and amount of SACD training and staff development 

provided over the previous 12 months (table 2.12). Principals reported higher participation rates than did 

teachers (100% versus 76%), and teachers reported more training hours (11 versus 8). There were no 

significant differences between treatment and control principals and teachers in overall training and hours. 

Out of seven targeted areas for professional development, there was one statistically significant difference in 

the area of behavior management, with more treatment teachers than control teachers reporting attending 

training in this area (71% versus 41%). 

Table 2.12. Principal and teacher initial reports on SACD professional development in sample––

ABC 

SACD professional development Total Treatment Control

 Principal sample size 12 6 6

 Teacher sample size 119 56 63

Principals reporting that staff participated in social and character

 development training within the past year (percent) 100.0 100.0 100.0

Teachers reporting participation in social and character

 development training within the past 12 months (percent) 76.2 81.9 70.5

Number of hours of social and character development training

 principals report were provided to each staff person last year (mean) 7.8 8.0 7.5

Number of hours of social and character development training

 teachers report receiving during the past 12 months (mean) 10.8 11.9 9.6

Teachers reporting receiving training in the past 12 months

 in the following areas (percent)

 Violence prevention and peace promotion 22.5 13.7 31.2

 Social and emotional development 25.2 23.1 27.2

 Character education 37.4 35.5 39.3

 Tolerance and diversity 20.5 17.6 23.5

 Risk prevention and health promotion 6.4 5.8 7.0

 Civic responsibility and community service 3.0 ‡ ‡

 Behavior management 56.0 71.2* 40.7

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.

The data on the initial level of SACD activity illustrate that the control condition was a ―standard practice‖ 

control. Standard practice at the control schools included using SACD activities, materials, and practices, 

along with professional development for staff, at rates and in types and amounts similar to the treatment 

schools. For example, the percentages of teachers who reported using programs or activities to promote 

specific SACD goals ranged from 67 percent to 99 percent in the treatment schools and from 55 percent to 

88 percent in the control schools. However, the 6 significant differences between the treatment and control 

conditions in the use of SACD activities were more than would be expected by chance (3 out of 62 

comparisons), and in all cases they favored the treatment group. Because initial data collection happened after 
Chapter 2. Academic and Behavioral Competencies Program 

186 

program implementation began, these differences may reflect that program implementation and program 

training for staff started before initial data collection. 

Impacts on Use of SACD Activities 

The introduction of the formal ABC program would be expected to increase the use of SACD activities in 

the treatment schools in comparison to the control schools. The analysis of this impact is based on the 

Teacher Report on Classroom and School (TRCS). Every spring, third-, fourth-, and fifth-grade teachers in 

the schools provided information through the TRCS about the social and character development activities 

they used in their classrooms. Specifically, information from the TRCS was used to determine the difference 

between treatment and control teachers in the following areas: 

1. the use of SACD activities in their classrooms overall and by SACD goal; 

2. the use of materials and strategies to implement the SACD activities within classrooms and within 

the entire school; 

3. the use of staff development to support the teachers; and 

4. teacher support for SACD efforts in the school and the use of practices conducive to the social and 

character development of students. 

TRCS consent and completion rates (table 2.4) led to 82 percent to 86 percent of teachers overall having data 

for the 3 years, with greater percentages of treatment teachers providing data in most of the years. To 

estimate intervention impacts for each of the outcomes, the statistical significance of the differences in means 

was assessed. Preliminary analysis using covariates indicated little or no gains in precision. Before testing the 

mean differences, the data were weighted such that each school received equal weight. Standard errors of the 

impact estimates accounted for the clustering of teachers within schools. In addition, a set of heuristics 

(described in chapter 1) was applied to determine whether each outcome domain was statistically significant 

after adjustments were made for the multiple tests conducted. 

Use of Activities 

The percentages of control teachers who reported using any SACD activities in their classrooms ranged from 

85 percent to 97 percent over the 3 years (table 2.13, panel 1). For the six individual SACD goals, the 

percentages varied from 52 percent to 87 percent in Year 1, 54 percent to 71 percent in Year 2, and 42 

percent to 69 percent in Year 3. Control teachers’ use of behavior management activities ranged from 80 

percent to 86 percent over this period. The percentages of control teachers who reported using any SACD 

activities in their classrooms for at least 1 hour per week ranged from 78 percent to 80 percent over the 3 

years (table 2.13, panel 2). For the six individual SACD goals, the range varied from 20 percent to 55 percent 

in Year 1, 14 percent to 38 percent in Year 2, and 14 percent to 35 percent in Year 3. Their use of behavior 

management activities for at least 1 hour per week ranged from 62 percent to 74 percent over this period. 

These findings show that the control schools were using these activities as part of their standard practice 

related to social and character development. 

For teachers’ reported use of any SACD activity (panels 1 and 2), 48 comparisons were made, with 2 expected 

to be significant by chance. The percentages of treatment teachers using any SACD activity were not 

significantly different from the percentages of control teachers in any year (panel 1). A significantly larger 

percentage of treatment teachers reported using civic responsibility activities (impact = 23 percentage points) 

in Year 1. In Year 2, more treatment teachers reported using activities for social and emotional development 

(impact = 24 percentage points), character education (impact = 20 percentage points), tolerance and diversity 

(impact = 24 percentage points), and behavior management (impact = 16 percentage points). There were no 

significant differences between treatment and control teacher reports of using activities for at least 1 hour a 
Chapter 2. Academic and Behavioral Competencies Program 

187

week (panel 2). After the heuristics were applied, the domain for engagement in any SACD activities showed 

the ABC program had statistically significant impacts in Year 1. 

For teachers’ reported use of any named SACD activity (panels 3 and 4), 42 comparisons were made, with 2 

expected to be significant by chance. None of the impact estimates were statistically significant over the 3 

years. The overall impact of the ABC program on the domain for engagement in named SACD activities was 

not significant in any year. 
Table 2.13. Impacts on teacher-reported SACD classroom activities––ABC 

Panel 1: Engagement in any activities to promote SACD goals1

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 63 73 50 67 51 64

Violence prevention and peace promotion

 (percent) 70.3 77.0 -6.8 0.587 74.8 63.3 11.5 0.339 63.5 62.0 1.5 0.882

Social and emotional development

 (percent) 77.7 86.5 -8.8 0.474 85.6* 61.9 23.8 0.044 69.2 55.2 14.0 0.111

Character education (percent) 91.4 83.5 7.9 0.383 90.9* 71.1 19.8 0.044 69.6 69.0 0.6 0.961

Tolerance and diversity (percent) 78.7 71.1 7.7 0.451 77.9* 54.2 23.7 0.014 66.4^ 44.8 21.6 0.072

Risk prevention and health promotion

 (percent) 74.8 80.2 -5.4 0.664 65.5 68.4 -2.9 0.833 57.3 56.6 0.7 0.955

Civic responsibility and community service

 (percent) 74.7* 51.8 22.9 0.038 62.9 53.9 9.1 0.571 65.6^ 41.6 24.0 0.094

Any SACD goal (percent) 94.0 97.1 -3.2 0.454 98.8 92.6 6.2 0.108 93.8 84.6 9.2 0.214

Behavior management (percent) 92.7 79.6 13.1 0.186 96.9* 81.1 15.7 0.022 94.1 85.8 8.3 0.179

See notes at end of table.

188

Chapter 2. Academic and Behavioral Competencies Program
Table 2.13. Impacts on teacher-reported SACD classroom activities––ABC—Continued 

Panel 2: Engagement in any activities to promote SACD goals for at least 1 hour per week

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 63 73 50 67 51 64

Violence prevention and peace promotion

 (percent) 32.9 44.0 -11.2 0.341 41.0 37.2 3.8 0.797 27.5 34.9 -7.4 0.537

Social and emotional development

 (percent) 40.4 54.5 -14.0 0.314 38.2 34.9 3.2 0.827 33.2 29.6 3.5 0.748

Character education (percent) 45.5 50.2 -4.7 0.747 61.2^ 34.1 27.0 0.059 38.5 34.6 4.0 0.723

Tolerance and diversity (percent) 30.8 38.7 -7.9 0.568 23.2 22.2 1.0 0.913 20.1 19.9 0.3 0.968

Risk prevention and health promotion

 (percent) 38.9 46.7 -7.8 0.433 22.9 37.5 -14.6 0.304 17.1 17.2 0.0 0.996

Civic responsibility and community service

 (percent) 25.4 20.1 5.4 0.522 15.6 14.1 1.5 0.886 14.0 14.0 0.0 0.997

Any SACD goal (percent) 74.8 80.1 -5.3 0.560 77.6 78.3 -0.7 0.952 84.0 79.6 4.4 0.567

Behavior management (percent) 78.8 61.5 17.3 0.232 79.9 68.7 11.2 0.454 79.4 73.8 5.6 0.527

See notes at end of table.

189

Chapter 2. Academic and Behavioral Competencies Program
Table 2.13. Impacts on teacher-reported SACD classroom activities––ABC—Continued 

Panel 3: Engagement in activities to promote SACD goals linked to named SACD programs

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 63 73 50 67 51 64

Violence prevention and peace promotion

 (percent) 30.8 28.9 1.8 0.856 26.2 30.2 -4.0 0.670 14.1 24.9 -10.9 0.280

Social and emotional development

 (percent) 28.1 12.5 15.6 0.160 28.4 17.6 10.8 0.264 9.8 16.7 -7.0 0.474

Character education (percent) 33.1 20.6 12.5 0.317 25.9 13.6 12.3 0.219 11.6 13.9 -2.3 0.768

Tolerance and diversity (percent) 20.0 10.1 9.8 0.233 19.4 6.8 12.6 0.125 8.0 7.9 0.2 0.979

Risk prevention and health promotion

 (percent) 21.7 37.2 -15.5 0.156 21.8 37.1 -15.3 0.252 17.1 16.3 0.8 0.941

Civic responsibility and community service

 (percent) ‡ ‡ 1.6 0.717 ‡^ ‡ 9.7 0.059 9.8 8.1 1.7 0.773

Any named activity (percent) 50.8 55.2 -4.4 0.742 42.7 55.1 -12.4 0.244 31.1 46.6 -15.6 0.283

See notes at end of table.

190

Chapter 2. Academic and Behavioral Competencies Program
Table 2.13. Impacts on teacher-reported SACD classroom activities––ABC—Continued

Panel 4: Engagement in activities to promote SACD goals linked to named SACD programs for at least 1 hour per week

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 63 73 50 67 51 64

Violence prevention and peace promotion

 (percent) 25.2 18.4 6.7 0.381 21.8 24.9 -3.2 0.735 8.6 13.4 -4.8 0.537

Social and emotional development

 (percent) 23.6 7.8 15.8 0.141 20.7 11.5 9.2 0.327 6.3 10.0 -3.7 0.553

Character education (percent) 21.9 13.6 8.3 0.420 25.4^ 8.9 16.5 0.073 7.2 9.3 -2.1 0.767

Tolerance and diversity (percent) 8.4 4.3 4.1 0.397 13.7^ 3.9 9.8 0.073 ‡ ‡ -0.8 0.760

Risk prevention and health promotion

 (percent) 13.3^ 29.1 -15.7 0.088 15.8 25.5 -9.7 0.464 5.3 7.3 -2.0 0.706

Civic responsibility and community service

 (percent) 5.3 3.5 1.8 0.690 ‡ ‡ 2.9 0.348 4.1 0.0 4.1 †

Any named activity (percent) 34.4 44.5 -10.1 0.410 32.2 41.0 -8.7 0.498 14.5 20.5 -6.0 0.606

† Not applicable.

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1

In Year 1, the omnibus impact for all the outcomes measured together was positive and statistically significant on the basis of a multivariate statistical test. 

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means. The number of results found 

significant was no more than expected by chance.

SOURCE: The Social and Character Development (SACD) Research Program.

191

Chapter 2. Academic and Behavioral Competencies Program
Chapter 2. Academic and Behavioral Competencies Program 

192

Use of Materials and Strategies 

For use of materials and strategies to support SACD goals, 87 comparisons were made, with 4 expected to be 

significant by chance. A significant impact was found on treatment teachers’ use of giveaways in Year 2 

(impact = 31 percentage points) (table 2.14). Regarding use of instructional techniques, significantly greater 

percentages of treatment teachers reported using journaling in Year 1 (impact = 17 percentage points), 

behavior honor roll in Year 2 (impact = 32 percentage points), and mentoring in Year 3 (impact = 17 

percentage points). No significant impact on the domain of materials and strategies was found in any year. 
Table 2.14. Impacts on use of SACD classroom materials and teaching strategies—ABC 

SACD material and teaching strategy

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 63 73 50 67 51 64

Use of SACD materials (percent)

 Teacher guides (manuals, curricula) 57.9 64.4 -6.5 0.578 57.7 64.7 -7.0 0.528 50.6 47.8 2.8 0.796

 Student materials (workbooks or sheets) 49.6 63.1 -13.5 0.295 57.3 68.0 -10.6 0.261 51.8 51.3 0.5 0.957

 Instructional aids (games, software, videos) 38.3 43.6 -5.3 0.644 37.0 31.2 5.8 0.581 26.9 29.9 -3.0 0.768

 Giveaways (bookmarks, stickers) 58.4 51.9 6.5 0.571 77.3* 46.0 31.2 0.015 61.2^ 43.5 17.7 0.099

 Children’s literature 62.6 47.0 15.6 0.138 58.2 50.4 7.8 0.518 48.5 38.8 9.7 0.420

 Other types of materials 6.5 10.0 -3.5 0.563 10.9 11.4 -0.5 0.937 10.4 11.1 -0.6 0.924

 Did not use any of these materials 7.7 9.3 -1.6 0.818 ‡ ‡ -7.7 0.179 13.5 19.2 -5.7 0.336

Use of teaching strategies (percent)

 Role-playing 77.3 76.3 1.0 0.941 84.8 74.1 10.7 0.316 77.9 71.5 6.4 0.489

 Cooperative learning 97.6 96.6 1.0 0.761 100.0 97.9 2.1 † 100.0 98.9 1.1 †

 Peer group discussions 85.5 89.6 -4.1 0.513 85.0 89.9 -4.9 0.554 94.9 87.6 7.3 0.190

 Direct instruction of SACD 82.1 78.9 3.2 0.755 79.7 67.5 12.2 0.471 90.4 81.7 8.7 0.287

 Skill training 44.9 50.5 -5.6 0.685 47.7 37.4 10.3 0.395 66.3 59.7 6.5 0.437

 Incorporating SACD into academic 

 curriculum 82.5 79.7 2.8 0.776 80.6 71.6 9.1 0.368 82.3 80.6 1.7 0.848

 Parent training 7.1 7.2 -0.1 0.977 17.8 8.3 9.5 0.186 13.0 12.8 0.2 0.984

 Parent and community involvement 20.6 20.7 -0.1 0.988 23.7 10.6 13.1 0.150 39.1 33.9 5.2 0.546

 Mentoring 39.1 41.6 -2.6 0.814 46.5 39.9 6.7 0.518 59.9* 42.7 17.2 0.036

 Good behavior notes sent home daily

 or weekly 96.2^ 80.4 15.9 0.053 91.3^ 72.5 18.8 0.072 94.4 86.4 8.0 0.254

 Presenting role models 74.3 84.5 -10.2 0.344 82.0^ 70.2 11.8 0.066 78.2 78.3 -0.1 0.977

See notes at end of table.

193

Chapter 2. Academic and Behavioral Competencies Program
Table 2.14. Impacts on use of SACD classroom materials and teaching strategies––ABC––Continued

SACD material and teaching strategy

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

Use of teaching strategies (percent)––

 Continued

 Targeted story reading or writing on social

 and character development themes 80.4 76.2 4.3 0.656 79.2 77.2 2.0 0.825 87.7 80.7 7.0 0.278

 Peer mediation 35.7 47.8 -12.1 0.170 52.9^ 34.5 18.4 0.071 62.0 51.7 10.3 0.278

 Honor roll for positive behavior 71.2 57.9 13.3 0.471 76.0* 43.8 32.1 0.037 79.0 65.9 13.1 0.259

 Pledges or recitations on social and

 character development themes 39.4 56.3 -16.9 0.257 34.9 45.2 -10.3 0.542 32.8 61.3 -28.5 0.105

 Guided visualization 49.3 53.6 -4.3 0.740 49.1 41.9 7.1 0.588 51.8 51.1 0.7 0.900

 Student-led/student-assisted instruction 60.2 50.9 9.3 0.218 53.4 47.8 5.6 0.491 65.7 54.2 11.5 0.306

 Journaling 90.4* 73.4 17.0 0.014 83.0 65.6 17.5 0.176 84.6 76.7 7.9 0.290

 Time out for negative behavior 78.4 82.6 -4.3 0.725 78.9 73.1 5.8 0.458 90.6 82.9 7.7 0.408

 Daily or weekly rewards for positive 

 behavior 97.1 93.7 3.3 0.441 95.0 88.4 6.6 0.297 100.0 94.4 5.6 †

 Any strategy 100.0 100.0 † † 100.0 100.0 0.0 † 100.0 100.0 0.0 †

 Number of strategies (mean) 13.0 12.9 0.0 0.980 13.3 11.5 1.8 0.163 14.2 13.4 0.8 0.258

† Not applicable

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means. The number of results found 

significant was no more than expected by chance.

SOURCE: The Social and Character Development (SACD) Research Program.

194

Chapter 2. Academic and Behavioral Competencies Program
Chapter 2. Academic and Behavioral Competencies Program 

195

Regarding the use of schoolwide strategies, 18 comparisons were made between treatment and control 

teacher reports, with 1 expected to be significant by chance. In Year 1, a significantly greater percentage of 

treatment teachers reported that their schools used three of five specific schoolwide strategies (these data are 

not shown in a table): newspapers or bulletins (impact = 29 percentage points), special school days (impact = 

16 percentage points), and special events (impact = 24 percentage points). In Years 2 and 3, no significant 

impacts were found for the individual schoolwide strategies. ABC did have a significant impact on the 

domain of schoolwide strategies in Year 1. 

Participation in Professional Development 

Regarding reported participation in professional development, 27 comparisons were made over 3 years, with 

1 expected to be significant by chance. In all 3 years, the intervention had no statistically significant effects on 

teachers’ participation in professional development for activities related to social and character development 

overall or on training for specific SACD goals (table 2.15). No significant impacts were found on the domain 

in all 3 years. 
Table 2.15. Impacts on teacher-reported SACD professional development––ABC 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD professional development

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 63 73 50 67 51 64

SACD training in past 12 months (percent) 77.3 61.9 15.3 0.293 51.4 46.8 4.6 0.713 40.4 37.9 2.5 0.799

Hours of SACD training (mean) 9.8 8.3 1.4 0.682 7.4 7.2 0.2 0.993 2.5 2.6 -0.2 0.854

Training by goal (percent)

 Violence prevention and peace promotion 13.8 27.7 -13.9 0.154 17.9 14.6 3.2 0.732 13.1 8.4 4.7 0.578

 Social and emotional development 23.6 27.4 -3.8 0.751 14.3 14.6 -0.4 0.966 ‡ ‡ -6.2 0.251

 Character education 33.7 39.3 -5.6 0.691 16.2 19.6 -3.4 0.694 9.1 14.3 -5.2 0.464

 Tolerance and diversity 9.1 19.4 -10.3 0.223 13.3 19.0 -5.6 0.505 ‡ ‡ 1.6 0.749

 Risk prevention and health promotion 6.5 16.6 -10.1 0.110 10.3 5.5 4.8 0.321 13.9 4.3 9.6 0.164

 Civic responsibility and community service 6.5 4.4 2.1 0.679 10.6 6.1 4.6 0.440 ‡ ‡ 0.7 0.794

 Behavior management 53.6 31.6 21.9 0.211 28.6 25.6 3.0 0.829 16.9 20.0 -3.2 0.704

‡ Reporting standards not met. Values suppressed to protect confidentiality.

NOTE: No findings were found statistically significant at or below the .05 level. Weights, which assign equal weight to each school within the program, were used in producing the 

treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

196

Chapter 2. Academic and Behavioral Competencies Program
Chapter 2. Academic and Behavioral Competencies Program 

197

Attitudes and Practices 

Teachers reported on their enthusiasm for SACD efforts in their schools (these data are not shown in a table) 

by indicating enthusiasm, cooperation, or open dislike. They also reported on the SACD practices of teachers 

and staff members in their schools (these data are not shown in a table). These practices included modeling 

positive character and behavior traits with students and fellow teachers, involving students in making 

decisions, giving students a voice in school governance, the school encouraging parent involvement in 

children’s social and character development, and using developmentally appropriate discipline strategies 

rather than punishment for misbehavior. In each year, 9 comparisons were made between treatment and 

control teachers on attitudes and practices (27 total), with 1 expected to be significant by chance. In Year 1 

there was a statistically significant negative impact on enthusiasm (impact = -5 percentage points). There were 

significant positive impacts on encouraging parent involvement in Years 2 and 3 (impact = 20 and 37 

percentage points) and on developmentally appropriate discipline strategies in Year 2 (impact = 26 percentage 

points). ABC had a significant positive effect on the domain in Year 3. 

Year-by-Year Impacts on Students and Perceptions of School Climate 

The primary research question for the ABC evaluation was this: 

What is the average effect of the ABC program on children’s social and emotional competence, 

behavior, academics, and perceptions of school climate? 

The first approach to answering this question was to examine the year-by-year impacts of ABC on these 

student and school climate outcomes over the 3 years as the students progressed from third through fifth 

grades. 

Equation (2) (described in chapter 1) was estimated to provide ABC’s impacts on the 20 outcomes based on 

data from the 12 treatment and control schools. For the ABC evaluation, equation (2) excluded the program 

fixed effects (p) and included program-specific covariates and random school effects covariates. Table 2.16 

lists the covariates used with outcomes from each report in the ABC analysis. 
Chapter 2. Academic and Behavioral Competencies Program 

198

Table 2.16. Covariates used with outcomes from each report for analysis—ABC 

Potential covariate

CR 

outcome

PCR 

outcome

TRS 

outcome

TRCS 

outcome

 Total number 11 13 24 8

Child-reported

 Female   

 Hispanic   

 Black (non-Hispanic)   

 Other ethnicity   

 Age in years   

 Scales

 Afraid at School 

 Altruistic Behavior  

 Empathy

 Engagement with Learning 

 Negative School Orientation 

 Normative Beliefs About Aggression

 Sense of School as a Community 

 Problem Behavior

 Self-Efficacy for Peer Interactions 

 Victimization at School

Primary caregiver-reported 

 Age in years  

 Completed high school or equivalent

 Some college

 Bachelor’s or higher degree

 Highest level of education in household

 Completed high school or equivalent 

 Some college 

 Bachelor’s or higher degree 

 Mother present in home life  

 Mother and father present  

 Respondent someone other than mother or father  

 Number of people in household 

 Household income: $20,000 to $40,000

 Household income: $40,000 to $60,000

 Household income: More than $60,000

 Income-to-poverty-threshold ratio: Below 135 percent

 Income-to-poverty-threshold ratio: 135 to 185 percent

 Full-time employment  

 Part-time employment  

See notes at end of table.
Chapter 2. Academic and Behavioral Competencies Program 

199

Table 2.16. Covariates used with outcomes from each report for analysis––ABC—Continued

Potential covariate

CR 

outcome

PCR 

outcome

TRS 

outcome

TRCS 

outcome

 Parental scales

 APQ-Poor Monitoring and Supervision Subscale   

 APQ-Positive Parenting Subscale

 Child-Centered Social Control

 Confusion, Hubbub, and Order  

 Community Resources

 Community Risk

 Parent and Teacher Involvement

 Child scales

 Altruistic Behavior

 Positive Social Behavior   

 Problem Behavior 

Teacher-reported 

 Female 

 Hispanic 

 Black (non-Hispanic) 

 Other ethnicity 

 Total teaching experience 

 Total experience in current school

 Regular certificate 

 Other certificate 

 Highest degree–bachelor’s 

 Child scales

 Academic Competence and Motivation

 ADHD-Related Behavior

 Altruistic Behavior

 Positive Social Behavior

 Problem Behavior 

 Parent and Teacher Involvement

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

 APQ: Alabama Parenting Questionnaire

 : Covariate used

 Blank cell: Covariate not used

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 2. Academic and Behavioral Competencies Program 

200

To assess the statistical power of the program-level impact estimates, minimum detectable impacts in effect 

size units (MDES) for each outcome measure were calculated for the ABC evaluation (table 2.17). MDES 

represent the smallest impacts in effect size (standard deviation) units that can be detected with a high 

probability (80%). MDES are primarily a function of study sample sizes, the degrees of freedom available for 

statistical tests, and design effects due to clustering (Schochet 2005). For the ABC evaluation, the MDES 

ranged from 0.110 to 0.551 for the child-level outcomes and from 0.396 to 0.629 for the perception of school 

climate outcomes based on the Teacher Report on Classroom and School. In general, the MDES for the 

school climate outcomes were larger than those for the child-level outcomes. 
Chapter 2. Academic and Behavioral Competencies Program 

201

Table 2.17. Adjusted minimum detectable effect sizes for impact evaluation––ABC

Outcome measure–Report Year 1 Year 2 Year 3

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 0.110 0.130 0.173

 Normative Beliefs About Aggression–CR 0.159 0.159 0.315

 Empathy–CR 0.133 0.225 0.262

Behavior Domain

 Altruistic Behavior–CR 0.155 0.110 0.124

 Altruistic Behavior–PCR 0.189 0.170 0.236

 Altruistic Behavior–TRS 0.214 0.551 0.353

 Positive Social Behavior–PCR 0.166 0.148 0.130

 Positive Social Behavior–TRS 0.190 0.423 0.381

 Problem Behavior–CR 0.221 0.184 0.244

 Problem Behavior–PCR 0.231 0.167 0.143

 Problem Behavior–TRS 0.297 0.295 0.304

 ADHD-Related Behavior–TRS 0.297 0.381 0.402

Academics Domain

 Engagement with Learning–CR 0.181 0.110 0.175

 Academic Competence and Motivation–TRS 0.149 0.189 0.320

Perceptions of School Climate Domain

 Positive School Orientation–CR 0.163 0.311 0.320

 Negative School Orientation–CR 0.110 0.189 0.286

 Student Afraid at School–CR 0.242 0.240 0.243

 Victimization at School–CR 0.156 0.110 0.211

 Feelings of Safety–TRCS 0.491 0.396 0.586

 Student Support for Teachers–TRCS 0.474 0.474 0.629

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The minimum detectable effect (MDE) formula used in the calculations is as follows:

where sT and sC are the number of treatment and comparison schools; nT and nC are the average number of students per 

classroom; 1 is the intraclass correlation (ICC) at the school level; and factor(df) is a constant that depends on the number of 

degrees of freedom (df) available for analysis (and is 2.802 for the pooled analysis).

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 2. Academic and Behavioral Competencies Program 

202

Table 2.18 provides the estimates of ABC’s impacts on each of the 20 outcomes over each of the 3 years (60 

comparisons in total, with 3 expected to be statistically significant by chance). Of the 60 results, 5 were 

statistically significant, with 4 having beneficial impacts. ABC had a significant positive impact on thirdgraders’ Altruistic Behavior (Teacher Report on Student, effect size [ES] = 0.39); on fourth-graders’ 

Academic Competence and Motivation (Teacher Report on Student, ES = 0.31) and Feelings of Safety 

(Teacher Report on Classroom and School, ES = 0.75); and on fifth-graders’ Positive Social Behavior 

(Primary Caregiver Report, ES = 0.21). A negative impact was found on fourth-graders’ Altruistic Behavior 

(Child Report, ES = –0.20). ABC had two beneficial impacts that were substantively important but not 

statistically significant: (1) on fourth-graders’ Student Support for Teachers (Teacher Report on Classroom 

and School, ES = 0.27) and (2) on fifth-graders’ Feelings of Safety (Teacher Report on Classroom and 

School, ES = 0.31). Application of the heuristics to adjust for multiple comparisons within each outcome 

domain indicated significant beneficial impacts on the domains of Academics and Perceptions of School 

Climate in Year 2. 
Table 2.18. Impacts on child and school outcomes––ABC 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 3.15 3.11 0.06 0.464 3.31 3.29 0.03 0.770 3.31 3.23 0.11 0.315

 Normative Beliefs About Aggression–CR (-) 1.35 1.30 0.09 0.440 1.39 1.29 0.18 0.141 1.51 1.51 0.00 0.981

 Empathy–CR (+) 2.27 2.26 0.02 0.823 2.08 2.19 -0.21 0.144 2.00 2.01 -0.03 0.859

Behavior Domain

 Altruistic Behavior–CR (+) 1.29 1.35 -0.07 0.449 0.94* 1.09 -0.20 0.029 1.01 0.98 0.04 0.656

 Altruistic Behavior–PCR (+) 2.25 2.15 0.15 0.249 2.14 2.19 -0.06 0.595 2.28 2.32 -0.04 0.767

 Altruistic Behavior–TRS (+) 1.44* 1.29 0.39 0.026 1.24 1.26 -0.03 0.914 1.21 1.22 -0.05 0.789

 Positive Social Behavior–PCR (+) 2.96 2.96 0.00 0.993 3.06^ 2.96 0.16 0.071 3.06* 2.94 0.21 0.041

 Positive Social Behavior–TRS (+) 2.90 2.89 0.02 0.800 3.01 3.02 -0.01 0.962 3.03 3.16 -0.19 0.406

 Problem Behavior–CR (-) 0.36 0.34 0.04 0.713 0.33 0.36 -0.05 0.659 0.51 0.53 -0.03 0.812

 Problem Behavior–PCR (-) 1.60 1.60 0.00 0.976 1.56 1.61 -0.11 0.229 1.58 1.61 -0.08 0.465

 Problem Behavior–TRS (-) 1.53 1.52 0.03 0.850 1.43 1.46 -0.07 0.655 1.54 1.44 0.21 0.262

 ADHD-Related Behavior–TRS (-) 1.84 1.85 -0.01 0.958 1.69 1.77 -0.13 0.525 1.72 1.62 0.17 0.447

Academics Domain1

 Engagement with Learning–CR (+) 3.61 3.61 0.00 0.986 3.68 3.63 0.10 0.281 3.58 3.58 0.00 0.964

 Academic Competence and Motivation–

 TRS (+) 2.81 2.79 0.03 0.719 2.92* 2.65 0.31 0.011 2.76 2.77 -0.02 0.924

See notes at end of table. 

203

Chapter 2. Academic and Behavioral Competencies Program
Table 2.18. Impacts on child and school outcomes––ABC––Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Perceptions of School Climate Domain2

 Positive School Orientation–CR (+) 2.82 2.94 -0.17 0.143 2.67 2.61 0.07 0.668 2.41 2.45 -0.05 0.773

 Negative School Orientation–CR (-) 1.94 1.85 0.16 0.108 1.97 1.96 0.01 0.963 2.16 2.12 0.07 0.680

 Student Afraid at School–CR (-) 2.23 2.28 -0.06 0.707 2.21 2.36 -0.17 0.263 2.19 2.35 -0.19 0.218

 Victimization at School–CR (-) 0.74 0.81 -0.08 0.417 0.58 0.66 -0.11 0.213 0.66 0.82 -0.19 0.151

 Feelings of Safety–TRCS (+) 3.42 3.25 0.21 0.353 3.59* 2.97 0.75 0.003 3.29 3.03 0.31° 0.235

 Student Support for Teachers–TRCS (+) 3.27 3.28 0.00 0.979 3.48 3.26 0.27° 0.276 3.25 3.21 0.05 0.847

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

 Substantive (but nonsignificant at .05 level) effect size of ≥ .25 or ≤ -.25.

1 In Year 2, at least half of the impacts were positive and statistically significant and no impact was negative and statistically significant based on univariate statistical tests. In addition, 

at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) procedure to 

adjust significance levels downward to account for the multiple testing of impacts.

2 In Year 2, at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) 

procedure to adjust significance levels downward to account for the multiple testing of impacts.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All Impact estimates were calculated using regression models, where each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. The effect size 

was calculated by dividing the estimated Impact by the standard deviation of the outcome measure for the control group. See table 1.5 for information about the measures used to 

create the outcome variables.

SOURCE: The Social and Character Development (SACD) Research Program.

204

Chapter 2. Academic and Behavioral Competencies Program
Chapter 2. Academic and Behavioral Competencies Program 

205

Impacts on Child Outcomes Over Time 

ABC impacts on the child outcomes over time were estimated using growth curve models by examining 

treatment and control group differences in the trajectories of student outcomes during the follow-up period 

while accounting for clustering at the school level. The growth curve models are estimated using a three-level 

hierarchical linear model, where Level 1 corresponds to time, Level 2 to students, and Level 3 to schools 

(described in chapter 1). 

Table 2.19 provides the estimates of the ABC impacts on the growth in student outcomes over the 3 years. 

The estimated impacts range in effect size units (absolute value) from 0.01 to 0.12. ABC did not have a 

statistically significant effect on any of the 18 estimated impacts (1 is expected to be significant by chance). 
Table 2.19. Impacts on growth of child outcomes––ABC 

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2Treatment Control

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 3.05 0.13 0.07 0.06 0.07 0.03 0.116

 Normative Beliefs About Aggression–CR (-) 1.28 0.10^ 0.02 0.08 0.12 0.04 0.078

 Empathy–CR (+) 2.35 -0.15^ -0.10 -0.05 -0.11 0.03 0.096

Behavior Domain

 Altruistic Behavior–CR (+) 1.47 -0.20 -0.21 0.01 0.01 0.05 0.908

 Altruistic Behavior–PCR (+) 2.29 -0.04 -0.02 -0.02 -0.02 0.04 0.654

 Altruistic Behavior–TRS (+) 1.33 -0.06 -0.01 -0.05 -0.10 0.05 0.325

 Positive Social Behavior–PCR (+) 2.91 0.05^ 0.00 0.05 0.08 0.03 0.097

 Positive Social Behavior–TRS (+) 2.89 0.05 0.11 -0.07 -0.08 0.06 0.293

 Problem Behavior–CR (-) 0.29 0.08 0.06 0.02 0.04 0.03 0.451

 Problem Behavior–PCR (-) 1.63 -0.01 0.00 -0.01 -0.02 0.02 0.611

 Problem Behavior–TRS (-) 1.45 0.05^ -0.01 0.06 0.11 0.03 0.057

 ADHD-Related Behavior–TRS (-) 1.84 -0.06 -0.05 0.00 -0.01 0.06 0.935

Academics Domain

 Engagement with Learning–CR (+) 3.64 -0.02 -0.01 -0.01 -0.01 0.04 0.835

 Academic Competence and Motivation–TRS (+) 2.70 0.02 0.01 0.01 0.01 0.05 0.858

See notes at end of table.

206

Chapter 2. Academic and Behavioral Competencies Program
Table 2.19. Impacts on growth of child outcomes––ABC––Continued

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2Treatment Control

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 3.12 -0.27 -0.22 -0.05 -0.06 0.05 0.343

 Negative School Orientation–CR (-) 1.85 0.09 0.07 0.02 0.02 0.05 0.691

 Student Afraid at School–CR (-) 2.29 -0.02 0.02 -0.04 -0.04 0.06 0.484

 Victimization at School–CR (-) 0.74 -0.03 0.01 -0.05 -0.05 0.04 0.248

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 Pertains to the estimated slope of the outcome for the treatment or control groups.

2 The average score at implementation is calculated across treatment and control groups, using regression models for adjustment on covariates. 

3 Estimated difference between the slope of the treatment and control groups.

4 Effect size: the slope of the treatment group minus the slope of the control group divided by the standard deviation of the outcome for the program’s control group (the standard 

deviation is calculated without accounting for school-level clustering or regression adjustments).

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

No findings were found statistically significant at or below the .05 level. The +/- signs in parentheses indicate the direction of a beneficial outcome. All Impact estimates were calculated 

using HLM 6.06. Sample weights were used in all analyses to give (1) each school equal weight in each program (within each time period) and (2) each time period equal weight within 

the analysis. See table 1.5 for information about the measures used to create the outcome variables.

SOURCE: The Social and Character Development (SACD) Research Program.

207

Chapter 2. Academic and Behavioral Competencies Program
Chapter 2. Academic and Behavioral Competencies Program 

208

Summary 

As part of the Social and Character Development (SACD) initiative, researchers at the Buffalo site 

implemented and evaluated the ABC program, which consists of multiple components related to classroom 

behavior management and teaching and promoting social skills. Twelve public schools in three school 

districts in upstate New York were recruited by the Buffalo research team and randomly assigned to 

treatment and control conditions to determine the impact of ABC on social and character development 

activities in the schools and on the child outcome domains of Social and Emotional Competence, Behavior, 

Academics, and Perceptions of School Climate. 

Analyses of the initial characteristics of the sample (students, caregivers, communities, teachers, and schools) 

indicated that randomization to treatment and control status produced groups that were similar on the 

observed characteristics near the start of the study (none of the comparisons were statistically significantly 

different). Documentation of the initial level of social and character development activities in the schools 

revealed six differences between the treatment and control teachers and classrooms, more than would be 

expected by chance. 

The data on the initial level of SACD activity led to two findings. First, treatment teachers reported greater 

use of and training in SACD activities than did control teachers, and they did so more often than would be 

expected by chance. There are two potential causes for this finding, and the analysis cannot be used to 

determine whether the reason for such a difference was that the two groups did differ on their initial use of 

SACD activities (i.e., that randomization did not create similar treatment and control groups) or the fact that 

the training of all treatment teachers and the implementation of the program before the initial data were 

collected influenced the teacher reports (program implementation began at the treatment schools on average 

6 weeks before the fall 2004 data collection occurred). Because it is likely (though unproven) that the training 

and implementation affected the teacher reports, these data were not considered appropriate for use as a 

baseline measure of SACD activities and training in the treatment schools. 

Second, these data indicated that the control condition for the SACD project was not a ―no treatment‖ 

control but a ―standard practice‖ control. Because the control teachers were not affected by the 

implementation of the SACD programs before data collection, their reports reflect standard practice in the 

control schools. Standard practice at the control schools included reports of 55 percent to 88 percent of 

teachers using SACD activities, 88 percent of teachers reporting the use of specific materials in conjunction 

with these activities, 100 percent reporting the use of at least one of the specified instructional strategies, and 

71 percent reporting participation in SACD training over the past 12 months. 

Analyses of ABC impacts on the use of SACD activities in the schools revealed few differences between 

treatment and control schools. Five of the 90 program impacts on SACD activities, 4 of the 87 impacts on 

SACD materials and strategies, 3 of the 18 impacts on schoolwide strategies, and 4 of the 27 impacts on 

attitudes and practices (1 negative) were statistically significant. No impacts on professional development 

were found in any of the years. 

Of the 20 child-level outcomes representing the four domains of Social and Emotional Competence, 

Behavior, Academics, and Perceptions of School Climate assessed in each of the 3 years of the study (a total 

of 60 results), 5 were statistically significant. One of these 5 was a detrimental impact and 4 were beneficial 

impacts. There were no effects of the ABC program over time as assessed using growth curve models. 

The SACD evaluation did not find evidence to support the hypothesis that the ABC program had beneficial 

impacts on students’ social and character development. These results could have been due to the inability of 

the program to cause such change, possibly because the theory of action was incomplete or because the 

activities to carry out that theory were not effective. 
Chapter 2. Academic and Behavioral Competencies Program 

209

These results may also be due to the inability of the evaluation to observe such an impact due to the control 

condition, the level of nonparticipation, or the sample size. The control schools continued using their 

standard SACD activities, and these turned out to be high in quantity and broad in scope. New York passed 

the Safe Schools Against Violence in Education Act of 2001 to encourage districts to promote instruction in 

civility, citizenship, and character education, and this may have been a factor in high SACD activity in the 

control schools. The high level of control school activity, combined with relatively few significant impacts by 

the ABC program on the use of SACD activities in the treatment schools, may have caused the resulting 

difference in the amount of SACD activities between the treatment and control schools to be too small to 

cause significant differences in the student outcomes. In addition, about one-third of the students in the 

sample universe did not take part because of nonconsent or noncompletion of the surveys. As a 

determination could not be made as to whether these students significantly differed from those who did take 

part, the evaluation’s results are valid only for the students who took part. If the students not taking part were 

different, and if they would have responded better to the ABC program than to the SACD activities occurring 

in the control schools, then the evaluation could have underestimated the program’s impact. Also, the sample 

size of 12 schools and the resulting higher MDES compared to those for the multiprogram evaluation may 

have reduced the likelihood of detecting statistically significant effects. However, it should be noted that 58 

percent of the MDES for the 60 outcomes used in the year-by-year analysis were below 0.25 (47% were 

below 0.20). In addition, only 2 of the 60 outcomes were found to be substantively important. 
This page is intentionally blank. 
211

Chapter 3. Competence Support Program 

University of North Carolina at Chapel Hill 

(North Carolina Site)

Intervention

Researchers at the University of North Carolina at Chapel Hill (North Carolina site) implemented and 

evaluated the Competence Support Program (CSP). The program, developed at the School of Social Work at the 

University of North Carolina at Chapel Hill, is a combination of three programs: the Making Choices program

(Fraser et al. 2000), the Competence Enhancement Behavior Management program (Farmer et al. 2006), and the Social 

Dynamics Training program (Farmer 2000). It is a universal, classroom-level intervention that functions as an 

add-on curriculum made up of scripted lessons and training for teachers in behavior management and social 

dynamics. Table 3.1 describes CSP’s general characteristics (panel 1), the types of instruction and strategies 

used (panel 2), the professional development provided for those implementing the program (panel 3), and the 

social and character development activities (panel 4) and outcomes (panel 5) addressed by the program. 

 In the third grade, the Making Choices curriculum consists of 28 scripted lessons that focus on social 

problem solving and involve learning about emotions and feelings, identifying social cues, making 

sense of social cues, setting social goals, choosing options, making choices, and acting on choices. 

The third grade is considered a keystone year. In the first and second grades, the Making Choices 

curriculum includes eight scripted lessons per year that prepare children for the third-grade program 

and focus on emotions and simple emotion regulation techniques (e.g., stop and think). In the fourth 

and fifth grades, the program consists of eight scripted lessons per year that focus on the application 

of problem-solving skills to routine peer situations (e.g., social aggression and bullying). Across all 

grades, teachers engage students by using a variety of learning strategies—story reading, discussion, 

role-playing, worksheets, art projects, and games. Lessons are designed to last approximately 45 

minutes. 

 The Competence Enhancement Behavior Management component is an in-service training and direct 

consultation program for teachers and is developed from evidenced-based behavioral and ecological 

best practices for promoting positive classroom behavior. It includes training in establishing 

proactive behavior management aims and goals, establishing productive classroom routines and 

structures, teaching and reinforcing alternative behaviors, building supportive relationships, 

communicating with troubled students, using constructive discipline and natural consequences, and 

preventing and managing behavioral crises. 

 Social Dynamics Training is an in-service teacher training and consultation model that includes training 

teachers to recognize the influence of peer social structures on students’ behavior and to manage 

classroom social dynamics. Teachers are trained to monitor and intervene in processes of social 

aggression, prevent the establishment of social hierarchies that promote jealousy and conflict, 

prevent social roles that promote aggression and violence (e.g., bullies), and monitor and prevent the 

establishment of antisocial and enemy peer groups. 
Chapter 3. Competence Support Program 

212

Table 3.1. Competence Support Program

Panel 1: General characteristics

Target population 

 Universal

Program components

 Peer: In and out of class

 Parent: None

 Classroom: Lessons, behavior management, and social dynamics

 Schoolwide: None or not major focus

 Community: None or not major focus

 Training: Pretraining and ongoing for teachers and counselors

Level of integration

 Add-on curriculum

Flexibility

 Manualized: Scripted lessons for teachers

 Adaptability: Teachers may adapt program to setting

See notes at end of table.

Panel 2: Description of instruction and strategies

Classroom

 Lessons

 Who delivers: Teacher

 Activities and tools: Story reading, discussion, role-playing, worksheets, games, artifacts (e.g., turtle 

 puppet), art projects

 Content: Problem solving, emotion identification and regulation, identifying social cues, goal formation,

 choosing options, behavior regulation

 Frequency: 45 minutes, once per week, 28 lessons

 Strategies

 Who delivers: Teacher

 Activities and tools: Prevention of social hierarchies that promote conflict, social reinforcement for

 positive behavior

 Frequency: Daily

Supplement to classroom

 None

Schoolwide activities 

 None

See notes at end of table.
Chapter 3. Competence Support Program 

213

Table 3.1. Competence Support Program––Continued

Panel 3: Professional development

Pre-implementation

 Teachers

 Content: Training on curriculum implementation, behavior management, and social dynamics

 Duration: 4 hours

 Other

 Content: Principal consultation on development of schoolwide management policy

 Duration: As requested

Ongoing consultation

 Teachers

 Content: Consultation on use of program materials

 Duration: 2 times per month

 Other

 Content: None

 Duration: None

See notes at end of table.

Panel 4: Activities for SACD goals

Violence prevention and peace promotion  Risk prevention and health promotion

Social and emotional development  Civic responsibility and community service

Character education Behavior management 

Tolerance and diversity

See notes at end of table.

Panel 5: SACD outcomes addressed 

Engagement with Learning  Empathy 

Academic Competence and Motivation Positive School Orientation

Altruistic Behavior Negative School Orientation

Positive Social Behavior  Student Afraid at School

Problem Behavior  Victimization at School 

Self-Efficacy for Peer Interactions  Feelings of Safety

Normative Beliefs About Aggression  Student Support for Teachers 

NOTE: Abbreviations are

 : Activity or outcome addressed

 Blank cell: Activity or outcome not addressed

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 3. Competence Support Program 

214

 Teachers receive 4 hours of training on curriculum implementation, behavior management strategies, 

and social dynamics. Principals receive consultation on the development of schoolwide management 

policy, as requested. Teachers receive consultations on the use of program materials twice a month.

 Consultation to school staff is a standard element of CSP. This consultation is provided by school 

personnel, such as guidance counselors, master teachers, school psychologists, or school social 

workers, who are trained to be program specialists. 

Sample and Random Assignment 

The North Carolina research team recruited a total of 10 public elementary schools in 2 school districts in 2 

counties in North Carolina. The 10 schools were randomly assigned to treatment and control conditions prior 

to the fall 2004 data collection.47 A two-step process was used. First, a computer-generated pairwise matching 

algorithm developed by Mathematica Policy Research, Inc. (MPR) was used to identify the best pairwise 

matches across the 10 schools based on variables identified by the North Carolina research team. The 

variables used in the pairwise matching for North Carolina included (a) total number of students; 

(b) percentages of students eligible for free and reduced-price lunch; (c) percentages of students who were 

White/Black/Hispanic/Other; (d) math achievement scores; and (e) reading achievement scores. Then, using 

the flip of a coin, 1 school in each matched pair was assigned to either the intervention or control condition, 

resulting in 5 schools receiving CSP and 5 schools acting as control schools and continuing to implement the 

social and character development activities that constituted their standard practice. Assignment to treatment 

or control condition was at the school level and therefore limited the risk of contamination between 

treatment and control classrooms. 

All 10 schools remained in the study for the full 3 years; however, one K-5 school in the treatment group 

became a K-4 school and continued implementation of the intervention. The cohort of students from that 

school who were in fifth grade during the 2006-07 school year went to an intermediate school that had fifth 

and sixth grades. The intermediate school did not implement the intervention. These fifth-grade students who 

were part of the original cohort were followed to their intermediate school, and they, their primary caregivers, 

and their fifth-grade teachers at the intermediate school provided data that were used in all analyses of the 

impact of CSP on children and on the use of activities related to social and character development in the 

schools. This change in grade configuration, which resulted in the absence of the intervention for that 

subgroup of fifth-graders and their teachers, should be noted when considering the results for the North 

Carolina site. 

The original student sample (the cohort of students in the third grade in the 10 schools in fall 2004) 

numbered 959 students (476 treatment and 483 control). Table 3.2 documents the change in the sample over 

the 3 spring follow-up data collection periods. Over time, new entrants to the cohort became a larger 

percentage of the sample, eventually making up 24 percent of the sample by the spring of Year 3. In Year 2 

there was a statistically significant difference between the treatment and control groups in the number of new 

entrants (in this case, fewer in the control group). The percentage of the sample made up of the original 

cohort further declined as students left the schools. By Year 3, approximately 25 percent of the original 

sample had left. In Years 2 and 3 there was a statistically significant difference between treatment and control 

groups, with a larger percentage of leavers in the treatment group. 

 

47 In Year 2 (fall 2005), the North Carolina research team recruited four more schools from an additional county, two 

assigned to treatment and two assigned to control, which were followed to the end of the study (spring 2007). A 

description of this second cohort and all relevant findings can be found in appendix A. The data from this second 

cohort are not included in the analyses and results reported in this chapter. 
Table 3.2. Sample––CSP 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Characteristic Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 School sample size 10 5 5 10 5 5 10 5 5 1015 5

 Student sample size 959 476 483 975 485 490 969 474 495 947 458 489

Stayers † † † 923 456 467 729 341 388 721 337 384

New entrants † † † 52 29 23 240 133 107 226 121 105

 New entrants as a percent of 

 spring enrollment † † † 5.3 6.0 4.7 24.8 28.1* 21.6 23.9 26.4 21.5

Total leavers (from original cohort) † † † 36 20 16 230 135 95 238 139 99

 Leavers as a percent of fall 2004 

 enrollment † † † 3.8 4.2 3.3 24.0 28.4** 19.7 24.8 29.2** 20.5

Number of students per school

 (mean) 96 95 97 98 97 98 97 95 99 95 92 98

Range of number of students per 

 school 46-148 85-106 46-148 47-155 86-117 47-155 46-144 84-121 46-144 47-139 73-105 47-139

† Not applicable.

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

1 One of the treatment group schools that had been a K-5 school in prior years became a K-4 school and continued implementation of the intervention. The Cohort 1 students, who 

were in fifth grade during the 2006-07 school year, went to an intermediate school that had fifth and sixth grades. The intermediate school did not implement the intervention. These 

fifth-grade students, their primary caregivers, and their teachers were included in the spring 2007 data collection and all analyses based upon spring 2007 data.

SOURCE: The Social and Character Development (SACD) Research Program.

215

Chapter 3. Competence Support Program
Chapter 3. Competence Support Program 

216

Implementation 

Training 

The intervention teachers received 4 hours of program implementation training prior to or within the first 2 

weeks of the beginning of the school year (see panel 3 of table 3.1). Teachers had access to ongoing program 

implementation support throughout the school year from two consultants. One consultant was a program 

specialist who focused on the Making Choices Program. She provided onsite consultation in the form of 

biweekly planning meetings with teachers to go over lesson content and to discuss any implementation issues. 

The second consultant, a program specialist for the Competence Enhancement and Social Dynamics Training

components, conducted biweekly consultation meetings with teacher teams to discuss key concepts in 

program implementation and issues and strategies around specific case examples generated by the teachers. In 

addition, each program specialist met at least once per semester with each principal to report on program 

activities. These two consultants also consulted with principals on the development of schoolwide 

management policy on an as-needed basis. 

Data Collection 

MPR collected the child, teacher, and school data at the North Carolina site. Table 3.3 presents the data 

collection dates. Data were collected in the fall and spring of the first 2 years and the spring of Year 3. The 

fall 2004 data collection began on August 30, 2004, and ended on September 16, 2004. The average time 

frame from the beginning of program implementation to the beginning of fall data collection was 4 weeks. As 

a result, initial data collection took place after CSP implementation began. Therefore, these data provide a 

measure of the students, teachers, and schools near the beginning of the school year, at a time when CSP had 

been operating for a relatively short period of time. The spring data collection window was from April 11, 

2005, to April 29, 2005. CSP had been implemented for 36 weeks at the time of the spring data collection and 

for 30 weeks from the end of the fall data collection. Year 2 followed a similar pattern, with implementation 

occurring at the start of the school year, fall data collection occurring 5 weeks later, and spring data collection 

occurring 27 weeks after fall data collection (and 35 weeks after the start of implementation). In spring 2007, 

data collection occurred 35 weeks after the start of implementation. During the 2005-06 and 2006-07 school 

years, one school operated on a year-round schedule. Data collection took from 3 to 5 weeks at each 

collection point. 
Chapter 3. Competence Support Program 

217

Table 3.3. Data collection dates––CSP 

Data collection schedule

Year 1

(Fall 3rd grade)

Year 1

(Spring 3rd grade)

Year 2

(Fall 4th grade)

Year 2

(Spring 4th grade)

Year 3

(Spring 5th grade)

 School sample size 10 10 10 10 10

School year dates 

 First day of school 7/26/04

8/2/041

† 8/25/052† 8/25/062

 Start of implementation 8/4/04 † First day † First day

 Last day of school † 5/24/05 † 5/23/0626/7/072,3

Data collection 

 Start 8/30/04 4/11/05 9/26/0544/24/06 4/26/07

 End 9/16/04 4/29/05 10/20/05 5/5/06 5/10/07

Calendar weeks from 

 program implementation 

 to start of fall 2004 data 

 collection 4 † † † †

Calendar weeks from 

 start of school to start 

 of fall data collection 5

1

† 5 † †

Calendar weeks from end 

 of fall data collection to 

 start of spring data 

 collection † 30 † 27 †

Calendar weeks from 

 program implementation 

 to start of spring data 

 collection † 36 † 35 352

† Not applicable.

1 Five schools started on 7/26/04 and five started on 8/2/04. The calculation of weeks from the start of school to the start of fall 2004 

data collection uses 7/26/04 as the first day of the school year.

2 During the 2005-06 school year and the 2006-07 school year, one school operated on a year-round schedule.

3 Schools ended between 6/7/07 and 6/8/07, with most ending on 6/7/07.

4 Data collection at one school occurred 3 weeks earlier.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 3. Competence Support Program 

218

Consent Rates, Completion Rates, and Percentage of Sample With Data 

The actual number of student, primary caregiver, and teacher reports available for analysis was smaller than 

the number in the sample because consent and completion rates were less than 100 percent. Primary 

caregivers had to provide consent before children could complete the Child Report, before their child’s 

teacher could complete the Teacher Report on Student, and before they themselves completed the Primary 

Caregiver Report. Teachers also had to provide consent before completing the Teacher Report on Classroom 

and School. Of those with consent, not all completed their respective reports. Table 3.4 shows the consent 

rates, completion rates, and percentages of sample with data for each of the four reports over the 3 years. For 

the Child Report and the Teacher Report on Student, completion rates ranged from 91 percent to 100 

percent. In Year 3, a significantly higher percentage of control than treatment students had consent for all 

three reports on students. For the Primary Caregiver Report, the completion rates dropped over time from 91 

percent to 74 percent. 

The percentages of the sample with Child Report data ranged from 59 percent to 71 percent over the 3 years, 

with a significantly higher percentage of control students having data in Year 3. The percentages of students 

with information from the Teacher Report on Student similarly ranged from 55 percent to 70 percent, with a 

significantly higher percentage of control students having data in Year 3. The percentages of students with 

data from the Primary Caregiver Report ranged from 43 percent to 63 percent and declined over time. In 

Year 1, the percentage of primary caregivers with data was significantly less in the treatment group than in the 

control group (56% versus 63%). The percentages of teachers with data from the Teacher Report on 

Classroom and School ranged from 78 percent to 95 percent. 
Table 3.4. Consent rates, completion rates, and percentage of sample with data from each report––CSP 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 Student sample size 959 476 483 975 485 490 969 474 495 947 4583489

Child Report (percent)

 Primary caregiver consent rate 70.0 68.7 71.2 69.4 68.7 70.2 68.5 67.3 69.7 66.4 60.0*** 72.4

 Student completion rate 92.5 91.4 93.6 97.6 97.9 97.4 97.0 98.4 95.7 98.4 98.5 98.3

 Students with data164.8 62.8 66.6 67.8 67.2 68.4 66.5 66.2 66.7 65.4 59.2*** 71.2

Primary Caregiver Report (percent)

 Primary caregiver consent rate 65.9 63.4 68.3 67.4 66.6 68.2 65.1 63.9 66.3 62.8 57.0*** 68.3

 Primary caregiver completion rate 90.5 88.7 92.1 80.1 76.5* 83.5 80.0 78.5 81.4 74.3 76.2 72.8

 Primary caregivers with data159.6 56.3* 62.9 53.9 50.9 56.9 52.1 50.2 53.9 46.7 43.4 49.7

Teacher Report on Student (percent)

 Primary caregiver consent rate270.0 68.7 71.2 69.4 68.7 70.2 68.5 67.3 69.7 66.4 60.0*** 72.4

 Teacher completion rate 94.9 93.3* 96.5 99.0 98.8 99.1 99.7 100.0 99.4 94.6 91.6*** 96.9

 Students with data166.4 64.1 68.7 68.7 67.8 69.6 68.3 67.3 69.3 62.8 55.0*** 70.1

Teacher Report on Classroom and School 

 (3rd-

 to 5th- grade teachers) (percent)

 Teacher consent rate 100.0 100.0 100.0 100.0 100.0 100.0 99.2 98.5 100.0 100.0 100.0 100.0

 Teacher completion rate 84.1 83.1 85.2 78.5 77.9 79.0 88.6 86.4 90.9 90.6 86.3 95.4

 Teachers with data184.1 83.1 85.2 78.5 77.9 79.0 88.0 85.1 90.9 90.6 86.3 95.4

* Treatment group significantly different from control group at the .05 level.

*** Treatment group significantly different from control group at the .001 level.

1 Calculated as consent rate x completion rate.

2 The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

3 One of the treatment group schools that had been a K-5 school in prior years became a K-4 school and continued implementation of the intervention. The Cohort 1 students, who 

were in fifth grade during the 2006-07 school year, went to an intermediate school that had fifth and sixth grades. The intermediate school did not implement the intervention. These 

fifth-grade students, their primary caregivers, and their teachers were included in the spring 2007 data collection and all analyses based upon spring 2007 data.

SOURCE: The Social and Character Development (SACD) Research Program.

219

Chapter 3. Competence Support Program
Chapter 3. Competence Support Program 

220

Responses from students in the original cohort (stayers) and new entrants in the CSP sample were examined 

to investigate possible differences between the two groups in consent rates, completion rates, and the 

percentages of sample with data that might affect outcome data (table 3.5). In Year 1, stayers completed more 

Child Reports than did new entrants (98% versus 93%), a statistically significant difference. In Years 2 and 3, 

primary caregivers of the stayers had significantly higher consent rates for both child and primary caregiver, 

which led to significantly higher percentages of data on all three reports in these 2 years for the stayers. The 

difference in consent rates for these two groups was significantly different by 14 to 35 percentage points. 
Table 3.5. Consent rates, completion rates, and percentage of sample with data: Stayers versus new entrants––CSP 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total Stayers

New 

entrants Total Stayers

New 

entrants Total Stayers

New 

entrants

 Student sample size 975 923 52 969 729 240 947 721 226

Child Report (percent)

 Primary caregiver consent rate 69.4 69.0 76.9 68.5 77.2*** 42.1 66.4 69.8*** 55.8

 Student completion rate 97.6 98.0* 92.5 97.0 97.0 97.0 98.4 98.4 98.4

 Students with data167.8 67.6 71.2 66.5 74.9*** 40.8 65.4 68.7*** 54.9

Primary Caregiver Report (percent)

 Primary caregiver consent rate 67.4 67.0 75.0 65.1 73.8*** 38.8 62.8 66.2*** 52.2

 Primary caregiver completion rate 80.1 80.1 79.5 80.0 81.2 73.1 75.1 75.1 75.4

 Primary caregivers with data153.9 53.6 59.6 52.1 59.9*** 28.3 47.2 49.7** 39.4

Teacher Report on Student (percent)

 Primary caregiver consent rate269.4 69.0 76.9 68.5 77.2*** 42.1 66.4 69.8*** 55.8

 Teacher completion rate 99.0 98.9 100.0 99.7 99.8 99.0 94.6 94.6 94.4

 Students with data168.7 68.3 76.9 68.3 77.1*** 41.7 62.8 66.0*** 52.7

* Stayers significantly different from new entrants at the .05 level.

** Stayers significantly different from new entrants at the .01 level.

*** Stayers significantly different from new entrants at the .001 level.

1 Calculated as consent rate x completion rate.

2 The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

221

Chapter 3. Competence Support Program
Chapter 3. Competence Support Program 

222

Fidelity of Implementation

Each year, CSP’s five treatment schools were independently rated for quantity and quality of the program’s 

implementation by two raters from the research team. The global measure of fidelity for the multisite study 

was used; inter-rater reliability was measured using Cronbach’s alpha (0.70, 0.80, and 0.99 across the 3 years). 

The ratings were combined into a single consensus rating and used to identify schools with high 

implementation fidelity. In all 3 years, two treatment schools were identified as having high fidelity. Cohen’s 

kappa was used as the measure of agreement when identifying schools as high fidelity, and it equaled 0.62 in 

Year 1, 0.55 in Year 2, and 1.00 in Year 3. 

Initial Characteristics 

This section examines the initial characteristics of the students, teachers, and schools participating in the CSP 

evaluation. These characteristics were collected from students who were enrolled in the third grade at the 

study schools in fall 2004, as well as from their primary caregivers and third-grade teachers. In addition, 

third-, fourth-, and fifth-grade teachers and principals in the study schools provided information about 

activities related to social and character development in these schools. Documenting the characteristics of 

students, teachers, and schools and initial measures of key outcomes at a point before the intervention had 

been operating for an extended period helps to determine whether the random assignment of schools to 

treatment and control status produced treatment and control groups with similar distributions of observed 

characteristics. As noted in the following discussion, 7 significant differences were found in the observed 

characteristics, 6 of which (out of 62 comparisons, with 3 expected to be significant by chance) related to 

differences between the treatment and control groups in the level of SACD activity in the classroom and 

school. 

Characteristics of Children, Their Families, and Communities 

There were no significant differences in the observed student, caregiver, and community characteristics 

between the treatment and control groups (see table 3.6). For students, the mean age was 8.1 years. The 

sample contained roughly equal percentages of girls (53%) and boys (47%). The sample was ethnically 

diverse. White non-Hispanic students made up 47 percent of the sample, Black non-Hispanic students made 

up 35 percent, and Hispanic students made up 10 percent. 

The sample was also diverse in its levels of family income, the education levels of primary caregivers of the 

children in the sample, and family situation. Thirty-three percent of the children lived in a household where 

the income was 135 percent of the federal poverty level or lower, which is the income threshold for eligibility 

for free school meals. Ten percent of the primary caregivers had not completed high school. About twothirds of the children (65%) lived with both their mother and their father. There were no significant 

differences between the treatment and control groups in these characteristics. 

The mean values of the outcomes for children’s behavior and attitudes as reported by the primary caregiver, 

child, and teacher in fall 2004 are shown in table 3.7. There was 1 significant difference (out of 18 

comparisons) in these scores between the treatment and control groups; children in the treatment group 

reported a more negative school orientation than did those in the control group. 
Chapter 3. Competence Support Program 

223

Table 3.6. Initial characteristics of children, their families, and communities––CSP 

Characteristic Total Treatment Control

 Student sample size 572 268 304

Student demographics

 Gender (percent)

 Male 46.9 45.2 48.7

 Female 53.1 54.8 51.3

 Race/ethnicity (percent)

 White (non-Hispanic) 46.6 35.4 57.7

 Black (non-Hispanic) 34.5 43.5 25.4

 Hispanic 9.9 8.7 11.1

 Other 9.1 12.4 5.7

 Age (in years) (mean) 8.1 8.1 8.1

Primary caregiver and family characteristics

 Primary caregiver’s age (in years) (mean) 34.7 34.3 35.1

 Primary caregiver’s race/ethnicity (percent)

 White (non-Hispanic) 51.2 41.0 61.3

 Black (non-Hispanic) 33.4 42.7 24.1

 Hispanic 8.6 6.8 10.4

 Other 6.9 9.5 4.2

 Primary caregiver’s education (percent)

 Did not complete high school 9.5 8.0 11.0

 Completed high school or equivalent 27.7 25.4 30.1

 Some college 47.3 51.0 43.5

 Bachelor’s or higher degree 15.5 15.6 15.4

 Primary caregiver’s employment (percent)

 Full-time 54.1 51.5 56.7

 Other 45.9 48.5 43.3

 Primary caregiver’s marital status (percent)

 Married 64.2 60.7 67.7

 Other 35.8 39.3 32.3

 Students who live in one household (percent) 95.2 95.9 94.6

 Number of individuals in household (mean) 4.5 4.4 4.5

 Primary caregiver’s relationship to child (percent)

 Mother (stepmother) 85.7 84.8 86.5

 Father (stepfather) 8.0 7.8 8.2

 Other relative/nonrelative 6.3 7.4 5.2

See notes at end of table.
Chapter 3. Competence Support Program 

224

Table 3.6. Initial characteristics of children, their families, and communities—CSP—Continued

Table 3.6. 

Characteristic Total Treatment Control

 Student lives with (percent)

 Mother (stepmother) and father (stepfather) 64.6 60.3 68.8

 Mother (stepmother) only; father (stepfather) not present 28.5 32.3 24.8

 Father (stepfather) only; mother (stepmother) not present 1.6 1.4 1.8

 Other relative/nonrelative, parents not present 5.3 6.0 4.6

 Highest education of anyone in household (percent)

 Did not complete high school 6.8 7.3 6.3

 Completed high school or equivalent 21.1 17.5 24.7

 Some college 51.5 54.6 48.4

 Bachelor’s or higher degree 20.7 20.7 20.7

 Total household income (percent)

 Less than $20,000 29.1 30.3 27.9

 $20,000 to $39,999 31.7 35.3 28.2

 $40,000 to $59,999 21.2 18.8 23.6

 $60,000 or more 18.0 15.6 20.3

 Income-to-poverty-threshold ratio—Below 135 percent (percent) 33.1 34.1 32.0

 Income-to-poverty-threshold ratio—135 to 185 percent (percent) 25.1 26.3 24.0

 Income-to-poverty-threshold ratio—Above 185 percent (percent) 41.8 39.6 44.0

 Alabama Parenting Questionnaire—Poor Monitoring and

 Supervision Subscale (mean) 1.2 1.2 1.2

 Alabama Parenting Questionnaire—Positive Parenting

 Subscale (mean) 3.6 3.6 3.6

 Confusion, Hubbub, and Order Scale (mean) 2.2 2.1 2.2

Community characteristics (mean)

 Community Risks Scale 1.2 1.2 1.2

 Community Resources Scale 2.2 2.3 2.2

 Child-Centered Social Control Scale 3.2 3.1 3.2

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 3. Competence Support Program 

225

Table 3.7. Mean scores and standard deviations for initial outcome measures of sample––CSP 

Total Treatment Control

Outcome measure–Report Range Mean SD Mean SD Mean SD

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 1-4 2.9 0.6 2.9 0.6 3.0 0.6

 Normative Beliefs About Aggression–CR 1-4 1.2 0.4 1.2 0.4 1.2 0.4

 Empathy–CR 1-3 2.4 0.4 2.4 0.4 2.4 0.4

Behavior Domain

 Altruistic Behavior–CR 0-3 1.5 0.8 1.5 0.9 1.4 0.8

 Altruistic Behavior–TRS 1-4 1.4 0.5 1.5 0.5 1.3 0.4

 Altruistic Behavior–PCR 1-4 2.3 0.7 2.3 0.7 2.3 0.7

 Positive Social Behavior–TRS 1-4 3.0 0.6 3.0 0.6 3.0 0.6

 Positive Social Behavior–PCR 1-4 3.0 0.5 3.0 0.5 3.0 0.5

 Problem Behavior–CR 0-3 0.2 0.4 0.2 0.4 0.2 0.4

 Problem Behavior–TRS 1-4 1.4 0.4 1.4 0.4 1.3 0.3

 Problem Behavior–PCR 1-4 1.5 0.3 1.5 0.3 1.6 0.3

 ADHD-Related Behavior–TRS 1-4 1.8 0.7 1.8 0.7 1.7 0.6

Academics Domain

 Academic Competence and Motivation–TRS 1-5 3.0 0.8 3.0 0.8 2.9 0.9

 Engagement with Learning–CR 1-4 3.7 0.7 3.6 0.7 3.7 0.6

Perceptions of School Climate Domain

 Positive School Orientation–CR 1-4 3.1 0.7 3.1 0.7 3.1 0.7

 Negative School Orientation–CR 1-4 1.9 0.6 2.0* 0.7 1.8 0.6

 Student Afraid at School–CR 1-4 2.5 0.9 2.5 0.9 2.5 0.9

 Victimization at School–CR 0-3 0.8 0.8 0.9 0.8 0.8 0.8

 Student sample size—PCR 572 268 304

 Student sample size—CR 621 299 322

 Student sample size—TRS 638 306 332

* Treatment group significantly different from control group at the .05 level.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

 SD: Standard deviation

Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall 

means. Statistical tests were conducted using regressions that included program indicators to account for the sample design and 

adjusted for clustering at the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 3. Competence Support Program 

226

Characteristics of Teachers and Schools 

The third-, fourth-, and fifth-grade teachers at the study schools were predominantly White non-Hispanic 

(77%) and female (93%), had an average of 14.8 years of total teaching experience, and about 26 percent held 

an advanced or specialist degree (see table 3.8). There was one statistically significant difference between the 

groups of teachers; control group teachers had been teaching in their current schools for a greater number of 

years on average than had treatment group teachers (7.7 versus 4.9 years). 

Data regarding school characteristics were drawn from the Common Core of Data in order to compare 

treatment and control schools. There were no significant differences between the two groups of schools in 

terms of student composition (race/ethnicity and school lunch eligibility), number of students enrolled, 

number of full-time teachers, Title I status, or number of years the principal had been at the school (table 

3.9). In addition, there were no significant differences between treatment and control schools in terms of 

location (urban, suburban, or rural) or lowest and highest grade offered (these data are not shown in a table). 

Table 3.8. Initial characteristics of teachers in sample––CSP 

Characteristic Total Treatment Control

 Teacher sample size 106 54 52

Gender (percent)

 Male 7.2 7.1 7.4

 Female 92.8 92.9 92.6

Race/ethnicity (percent)

 White (non-Hispanic) 77.1 74.8 79.4

 Other 22.9 25.2 20.6

Number of years teaching experience (mean) 14.8 14.7 14.9

Number of years teaching experience in this school (mean) 6.3 4.9* 7.7

Type of teaching certificate (percent)

 Regular state certificate or advanced professional certificate 95.0 ‡ ‡

 Other 5.0 ‡ ‡

Education (percent)

 Bachelor’s degree 74.3 66.7 81.9

 Advanced degree/other 25.7 33.3 18.1

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and 

overall means. Statistical tests were conducted using regressions that included program indicators to account for the sample design 

and adjusted for clustering at the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 3. Competence Support Program 

227

Table 3.9. Initial characteristics of schools in sample—CSP 

Characteristic Total Treatment Control

 School sample size 10 5 5

Student race/ethnicity (percent)

 White (non-Hispanic) 44.0 34.8 53.3

 Black (non-Hispanic) 41.3 51.0 31.5

 Hispanic 10.3 8.0 12.7

 Other 4.4 6.2 2.5

Students eligible for free or reduced-price lunch (percent) 57.0 63.8 50.2

Number of students enrolled (mean) 626.5 657.8 595.2

Number of full-time teachers (mean) 38.9 40.8 37.0

Title I status (percent)

 Title I eligible school 100.0 100.0 100.0

 Schoolwide Title I 100.0 100.0 100.0

Number of years principal has been at this school (mean) 4.9 2.5 6.8

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: NCES Common Core of Data (2003-04), the Social and Character Development (SACD) Research Program.

In the Teacher Report on Classroom and School, teachers reported on nine dimensions of school 

environment (these data are not shown in a table): feelings of safety, adequacy of resources, student support, 

freedom to teach as desired, affiliation with and ties to colleagues, innovation regarding new approaches to 

teaching, professional interest, participatory decisionmaking, and work pressure. There were no statistically 

significant differences between treatment and control schools in these reports. 

The Level of SACD in the Schools Near the Beginning of the Study 

During the initial data collection, principals and teachers reported on the SACD activities used in the schools 

and classrooms, the availability of SACD materials, and the professional development provided on SACD. It 

is important to note that this data collection at the beginning of the school year took place, on average, 4 

weeks after the start of the CSP implementation; therefore the activities and practices related to SACD in the 

treatment and control schools are a measure of what was happening in the study schools near the beginning 

of the study. Table 3.10 shows that the majority of the school principals reported activities to promote six 

social and character development goals: violence prevention and peace promotion (100%), social and 

emotional development (100%), character education (100%), tolerance and diversity (80%), risk prevention 

and health promotion (80%), and civic responsibility and community service (90%). In addition, all of the 

principals reported activities directed toward behavior management. There were no statistically significant 

differences between the treatment group and the control group in the percentages on principal reports, 

although this may be due to the relatively small principal sample size. Teachers’ reports on the use of these 

activities in their classrooms ranged from 60 percent to 93 percent, with no significant difference between 

treatment and control teachers. With respect to the use of schoolwide activities, 64 percent to 97 percent of 

teachers reported that their schools used such activities, and again there were no significant differences 

between treatment and control teachers in reported use. 
Chapter 3. Competence Support Program 

228

Table 3.10. Principal and teacher initial reports on use of SACD programs or activities in 

sample––CSP 

SACD program or activity Total Treatment Control

 Principal sample size 10 5 5

 Teacher sample size 106 54 52

Principals reporting that school had programs or activities

 to promote the following SACD goals (percent)

 Violence prevention and peace promotion 100.0 100.0 100.0

 Social and emotional development 100.0 100.0 100.0

 Character education 100.0 100.0 100.0

 Tolerance and diversity 80.0 80.0 80.0

 Risk prevention and health promotion 80.0 100.0 60.0

 Civic responsibility and community service 90.0 80.0 100.0

 Behavior management 100.0 100.0 100.0

 None of the above 0.0 0.0 0.0

Teachers reporting on using programs or activities in their 

 class to promote the following SACD goals (percent)

 Violence prevention and peace promotion 60.2 67.9 52.4

 Social and emotional development 76.5 77.4 75.6

 Character education 84.6 88.2 80.9

 Tolerance and diversity 66.4 67.3 65.6

 Risk prevention and health promotion 67.4 60.8 74.1

 Civic responsibility and community service 62.1 62.7 61.6

 Behavior management 93.4 94.1 92.7

 None of the above 0.0 0.0 0.0

Teachers reporting schoolwide use of the following 

 activities to promote SACD (percent)

 Morning announcements or videos 97.2 95.6 98.8

 School assemblies 75.8 74.2 77.3

 School newspapers or bulletins 73.5 67.4 79.6

 Special school days 63.9 57.7 70.0

 Special events 80.5 83.0 78.1

 Other activities 17.2 18.9 15.6

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within each of the programs and to each program across programs, were used in producing the 

treatment, control, and overall means. Statistical tests were conducted using regressions that included program indicators to 

account for the sample design and adjusted for clustering at the school level. Sample size may differ for some outcomes due to 

nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.

Teachers reported using a broad range of teaching materials to support SACD activities (table 3.11), including 

teacher guides (69%), student materials (69%), giveaways (50%), instructional aids (45%), and children’s 

literature (42%). There were statistically significant differences in the use of teacher guides (81% versus 57%) 

and instructional aids (53% versus 38%), with treatment teachers reporting greater use of these two types of 

materials than control teachers. 

Teachers also reported using a wide variety of teaching strategies (table 3.11). All teachers reported using any 

of the 20 strategies asked about, and teachers used an average of 12.3 of the strategies. There were no 

significant differences in the average number of strategies, or in the specific strategies, used by treatment 

versus control teachers. 
Chapter 3. Competence Support Program 

229

Table 3.11. Teacher initial reports on use of SACD materials and classroom strategies in 

sample––CSP 

SACD material and classroom strategy Total Treatment Control

 Teacher sample size 106 54 52

Teachers using the following materials in conjunction with

 social and character development activities (percent)

 Teacher guides (manuals, curricula) 69.0 81.0* 57.0

 Student materials (workbooks, worksheets) 68.8 74.4 63.1

 Instructional aids (games, software, videos) 45.3 53.0* 37.6

 Giveaways (bookmarks, stickers) 50.3 49.9 50.8

 Children’s literature 42.3 35.5 49.0

 Other types of materials 14.6 12.8 16.3

 Do not use any of the materials listed above 8.3 ‡ ‡

Teachers using any of the strategies listed below to promote 

 social and character development in the classroom (percent) 100.0 100.0 100.0

Number of strategies (listed below) used by teachers to promote

 social and character development in the classroom (mean) 12.3 13.1 11.6

Teachers using each of the following strategies to promote

 social and character development (percent)

 Role-playing 62.4 68.1 56.8

 Cooperative learning 92.9 93.0 92.7

 Peer group discussions 86.7 86.8 86.7

 Direct instruction of social and character development 85.7 87.9 83.5

 Skill training 30.2 40.1 20.2

 Incorporating social and character development into 

 academic curriculum 82.2 89.6 74.8

 Parent training 18.7 12.2 25.1

 Parent/community involvement in program development 

 or delivery 30.5 35.7 25.3

 Mentoring 38.7 44.5 32.9

 Good behavior notes sent home daily or weekly 90.3 85.5 95.1

 Presenting role models 71.9 64.2 79.5

 Targeted story reading or writing on SACD themes 75.0 82.5 67.5

 Peer mediation 48.8 56.8 40.8

 Honor roll for positive behavior 57.2 54.0 60.3

 Pledges or recitations on social and character 

 development themes 42.2 53.3 31.1

 Guided visualization 54.4 61.6 47.2

 Student-led/student-assisted instruction 45.0 54.9 35.0

 Journaling 58.9 57.4 60.5

 Time out for negative behavior 94.4 98.3 90.5

 Daily or weekly rewards for positive behavior 96.5 95.6 97.5

‡ Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 3. Competence Support Program 

230

Principals and teachers reported on participation in and amount of SACD training and staff development 

provided over the previous 12 months (table 3.12). Principals reported higher participation rates (100% 

versus 72%) and reported more training hours (8.9 versus 7.2) than did teachers. There was a significant 

difference in the percentages of teachers reporting participation in training, with more treatment teachers 

reporting participation in SACD training than control teachers (93% versus 52%). In addition, a significantly 

greater percentage of treatment teachers than control teachers reported receiving training in three of the 

seven targeted SACD goals: social and emotional development (38% versus 9%), character education (62% 

versus 14%), and behavior management (60% versus 26%). 

Table 3.12. Principal and teacher initial reports on SACD professional development in sample––

CSP 

SACD professional development Total Treatment Control

 Principal sample size 10 5 5

 Teacher sample size 106 54 52

Principals reporting that staff participated in social and character

 development training within the past year (percent) 100.0 100.0 100.0

Teachers reporting participation in social and character development

 training within the past 12 months (percent) 72.2 93.0** 51.5

Number of hours of social and character development training 

 principals report were provided to each staff person last year (mean) 8.9 9.6 8.2

Number of hours of social and character development training 

 teachers report receiving during the past 12 months (mean) 7.2 8.1 6.4

Teachers reporting receiving training in the past 12 months

 in the following areas (percent)

 Violence prevention and peace promotion 7.9 ‡ ‡

 Social and emotional development 23.3 37.8* 8.8

 Character education 37.6 61.6** 13.6

 Tolerance and diversity 20.5 24.0 17.0

 Risk prevention and health promotion 8.6 11.9 5.3

 Civic responsibility and community service 3.0 ‡ ‡

 Behavior management 43.1 60.0* 26.1

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.

The data on the initial levels of SACD activity emphasize that the control condition was a ―standard practice‖ 

control. Standard practice at the control schools included using SACD activities, materials, and practices, 

along with professional development for staff, at rates and in types and amounts similar to the treatment 

schools. For example, the percentages of teachers who reported using programs or activities to promote 

specific SACD goals ranged from 61 percent to 94 percent in the treatment schools and from 52 percent to 

93 percent in the control schools. However, the 6 significant differences between the treatment and control 

conditions in the use of SACD activities was more than expected by chance (3 out of 62 comparisons), and in 
Chapter 3. Competence Support Program 

231

all cases they favored the treatment group. An important point about this difference between treatment and 

control teachers is that this training took place just prior to the start of the school year. Because initial data 

collection happened after program implementation began, these differences may reflect the fact that program 

implementation and program training for staff started before initial data collection. 

Impacts on Use of SACD Activities 

The introduction of the formal CSP program would be expected to increase the use of SACD activities in the 

treatment schools in comparison to the control schools. The analysis of this impact is based on the Teacher 

Report on Classroom and School. Every spring, third-, fourth-, and fifth-grade teachers provided information 

through the Teacher Report on Classroom and School about the social and character development activities 

they used in their classrooms. Specifically, information from the Teacher Report on Classroom and School 

was used to determine the difference between treatment and control teachers in the following areas: 

1. the use of SACD activities in their classrooms overall and by SACD goal; 

2. the use of materials and strategies to implement the SACD activities within classrooms and within 

the entire school; 

3. the use of staff development to support the teachers; and 

4. teacher support for SACD efforts in the school and the use of practices conducive to the social and 

character development of students. 

Teacher Report on Classroom and School consent and completion rates (table 3.4) led to 78 percent to 95 

percent of all teachers having data for the 3 years. To estimate intervention impacts for each of the outcomes, 

testing of the statistical significance of the differences in means was used. Preliminary analysis indicated little 

or no gains in precision from using covariates. Before testing the mean differences, the data were weighted 

such that each school received equal weight. Standard errors of the impact estimates accounted for the 

clustering of teachers within schools. In addition, a set of heuristics (described in chapter 1) was applied to 

determine whether each outcome domain was statistically significant after adjustments were made for the 

multiple tests conducted. 

Use of Activities 

The percentages of control teachers who reported using any SACD activities in their classrooms ranged from 

87 percent to 96 percent over the 3 years (table 3.13, panel 1). For the six individual SACD goals, the ranges 

varied from 67 percent to 84 percent in Year 1, 67 percent to 88 percent in Year 2, and 57 percent to 81 

percent in Year 3. Control teachers’ use of behavior management activities ranged from 85 percent to 91 

percent over this period. The percentages of control teachers who reported using any SACD activities in their 

classrooms for at least 1 hour per week (panel 2) ranged from 63 percent to 82 percent over the 3 years. For 

the six individual SACD goals, the ranges varied from 18 percent to 37 percent in Year 1, 20 percent to 40 

percent in Year 2, and 11 percent to 37 percent in Year 3. Control teachers’ use of behavior management 

activities for 1 hour per week ranged from 75 percent to 79 percent over this period. These findings show 

that the control schools were using these activities as part of their standard practice related to social and 

character development. 

For teachers’ reported use of any SACD activity (panels 1 and 2), 48 comparisons were made, with 2 expected 

to be significant by chance. The percentages of treatment teachers using any SACD activity, or specific SACD 

activities, at any point in the previous year were not significantly different from the percentages of control 

teachers in any year. There was a significant difference between treatment and control teacher reports of 

using behavior management activities for at least 1 hour per week in Year 3, with more control teachers 

reporting these activities (impact = -14 percentage points). After the heuristics were applied, the domain for 

engagement in SACD activities showed CSP had a statistically significant impact in Year 1. 
Chapter 3. Competence Support Program 

232

For teachers’ reported use of any named SACD activity (panels 3 and 4), 42 comparisons were made, with 2 

expected to be significant by chance. Eight of the 14 impact estimates in Year 1, 5 of the 14 in Year 2, and 4 

of the 14 in Year 3 were statistically significant. Impacts on the use of a named activity at any point in the 

previous year (panel 3) were as follows: violence prevention and peace promotion in all 3 years (impact = 28, 

30, and 38 percentage points), social and emotional development in Year 1 (impact = 40 percentage points) 

and Year 2 (impact = 44 percentage points), character education in Year 2 (impact = 37 percentage points) 

and Year 3 (impact = 29 percentage points), tolerance and diversity in Year 1 (impact = 20 percentage points) 

and Year 2 (impact = 33 percentage points), and the use of any named activity in all 3 years (impact = 32, 34, 

and 40 percentage points). In Year 1 there were significant impacts on the use of any named activity for at 

least 1 hour per week (panel 4) for violence prevention and peace promotion (impact = 18 percentage 

points), social and emotional development (impact = 21 percentage points), and character education (impact 

= 23 percentage points). The impact on the use of any named activity for at least 1 hour per week was 

significant in Year 1 (impact = 34 percentage points) and Year 3 (impact = 23 percentage points). The overall 

impact of CSP on the domain for engagement in named SACD activities was significant in all 3 years. 
Table 3.13. Impacts on teacher-reported SACD classroom activities––CSP 

Panel 1: Engagement in any activities to promote SACD goals1

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment3 Control Impact p-value 

 Teacher sample size 65.0 61.0 57.0 60.0 63.0 62.0

Violence prevention and peace promotion

 (percent) 77.4 67.1 10.3 0.287 73.0 72.3 0.8 0.935 72.6 61.8 10.8 0.205

Social and emotional development

 (percent) 88.4 72.5 16.0 0.151 79.0 73.2 5.8 0.358 72.3 65.1 7.2 0.449

Character education (percent) 95.2^ 84.0 11.2 0.097 91.6 87.5 4.1 0.382 83.0 80.6 2.4 0.751

Tolerance and diversity (percent) 84.9 74.7 10.2 0.420 73.5 72.4 1.1 0.795 64.9 60.0 4.9 0.684

Risk prevention and health promotion

 (percent) 76.4 73.8 2.6 0.804 62.0 73.3 -11.2 0.341 66.6 68.8 -2.2 0.844

Civic responsibility and community service

 (percent) 72.4 78.7 -6.3 0.650 65.2 67.1 -1.9 0.819 58.7 56.5 2.2 0.861

Any SACD goal (percent) 96.7 90.8 5.9 0.288 94.3 96.3 -2.0 0.550 94.1 87.3 6.8 0.283

Behavior management (percent) 96.9 90.6 6.3 0.144 89.2 85.2 3.9 0.565 80.3 88.6 -8.3 0.110

See notes at end of table.

233

Chapter 3. Competence Support Program
Table 3.13. Impacts on teacher-reported SACD classroom activities––CSP—Continued 

Panel 2: Engagement in any activities to promote SACD goals for at least 1 hour per week

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment3 Control Impact p-value 

 Teacher sample size 65.0 61.0 57.0 60.0 63.0 62.0

Violence prevention and peace promotion

 (percent) 28.5 29.6 -1.0 0.932 24.9 28.8 -3.9 0.623 26.1 27.4 -1.3 0.867

Social and emotional development

 (percent) 39.2^ 26.7 12.6 0.087 32.5 27.5 4.9 0.654 21.6 24.9 -3.3 0.748

Character education (percent) 55.2^ 36.9 18.2 0.098 48.6 39.6 9.0 0.411 31.3 37.3 -6.0 0.547

Tolerance and diversity (percent) 27.1 18.0 9.0 0.363 24.5 30.9 -6.3 0.496 27.1 11.4 15.7 0.145

Risk prevention and health promotion

 (percent) 35.6 24.7 10.9 0.427 30.1 29.0 1.0 0.904 24.0 29.3 -5.3 0.659

Civic responsibility and community service

 (percent) 27.4 28.3 -1.0 0.933 18.2 20.1 -1.8 0.876 31.8 15.0 16.8 0.133

Any SACD goal (percent) 68.8 62.8 6.0 0.431 76.2 78.1 -1.8 0.824 76.8 81.5 -4.7 0.373

Behavior management (percent) 80.6 78.8 1.8 0.789 70.8 74.6 -3.8 0.720 61.3* 75.3 -14.0 0.046

See notes at end of table.

234

Chapter 3. Competence Support Program
Table 3.13. Impacts on teacher-reported SACD classroom activities––CSP—Continued

Panel 3: Engagement in activities to promote SACD goals linked to named SACD programs2

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment3 Control Impact p-value 

 Teacher sample size 65.0 61.0 57.0 60.0 63.0 62.0

Violence prevention and peace promotion

 (percent) 38.8* 10.5 28.3 0.022 50.4* 20.5 29.9 0.016 51.5* 13.3 38.2 0.003

Social and emotional development

 (percent) ‡* ‡ 39.7 0.003 54.5* 10.3 44.3 0.007 47.0 0.0 47.0 †

Character education (percent) 35.8^ 7.1 28.7 0.063 49.0* 11.7 37.3 0.001 33.7* 4.9 28.8 0.018

Tolerance and diversity (percent) ‡* ‡ 19.6 0.015 ‡* ‡ 32.5 0.014 15.6 0.0 15.6 †

Risk prevention and health promotion

 (percent) 36.8 23.0 13.8 0.381 28.6 23.8 4.8 0.569 26.0 18.9 7.1 0.545

Civic responsibility and community service

 (percent) ‡ ‡ 1.5 † 9.5 8.9 0.6 0.936 6.3 5.6 0.7 0.811

Any SACD goal (percent) 67.8* 35.7 32.1 0.025 74.1* 39.8 34.3 0.005 69.7* 29.9 39.8 0.007

See notes at end of table.

235

Chapter 3. Competence Support Program
Table 3.13. Impacts on teacher-reported SACD classroom activities––CSP—Continued

Panel 4: Engagement in activities to promote SACD goals linked to named SACD programs for at least 1 hour per week

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment3 Control Impact p-value 

 Teacher sample size 65.0 61.0 57.0 60.0 63.0 62.0

Violence prevention and peace promotion

 (percent) ‡* ‡ 17.8 0.009 21.1 14.4 6.7 0.334 19.9 11.8 8.0 0.211

Social and emotional development

 (percent) ‡* ‡ 21.4 0.012 23.9 5.8 18.1 0.145 13.5 0.0 13.5 †

Character education (percent) ‡* ‡ 23.4 0.005 28.0^ 10.9 17.1 0.055 ‡^ ‡ 8.3 0.097

Tolerance and diversity (percent) ‡^ ‡ 8.3 0.057 16.2 0.0 16.2 † 8.4 0.0 8.4 †

Risk prevention and health promotion

 (percent) 18.1 6.4 11.8 0.157 17.0 18.6 -1.6 0.834 13.5 11.8 1.7 0.860

Civic responsibility and community service

 (percent) ‡ ‡ 1.7 † ‡ ‡ -2.9 0.664 0.0 0.0 0.0 †

Any SACD goal (percent) 45.1* 10.8 34.3 0.008 36.8^ 25.5 11.3 0.097 38.7* 16.0 22.7 0.047

† Not applicable.

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 In Year 1, the omnibus impact for all the outcomes measured together was positive and statistically significant on the basis of a multivariate statistical test. 

2 In Year 1, based on univariate statistical tests, at least half of the impacts were positive and statistically significant and no impact was negative and statistically significant. In all 3 

years, at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) 

procedure to adjust significance levels downward to account for the multiple testing of impacts. 

3 One of the treatment group schools that had been a K-5 school in prior years became a K-4 school and continued implementation of the intervention. The Cohort 1 students, who 

were in fifth grade during the 2006-07 school year, went to an intermediate school that had fifth and sixth grades. The intermediate school did not implement the intervention. These 

fifth-grade students, their primary caregivers, and their teachers were included in the spring 2007 data collection and all analyses based upon spring 2007 data.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

236

Chapter 3. Competence Support Program
Chapter 3. Competence Support Program 

237

Use of Materials and Strategies 

For use of materials and strategies to support SACD goals, 87 comparisons were made, with 4 expected to be 

significant by chance. Several significant impacts were found on treatment teachers’ use of materials and 

strategies in all 3 years. In Year 1, more treatment teachers used role-playing (impact = 20 percentage points), 

direct instruction of SACD (impact = 16 percentage points), skill training (impact = 33 percentage points), 

and guided visualization (impact = 36 percentage points) (table 3.14). In Year 2, more treatment teachers 

continued to use direct instruction of SACD (impact = 18 percentage points), and more treatment teachers 

used teacher guides (impact = 19 percentage points) and student materials (impact = 16 percentage points). 

In Year 3, more treatment teachers used teacher guides (impact = 21 percentage points). The average number 

of strategies used was significantly different in Year 1 (by 1.6 activities on average). The impact on the domain 

of materials and strategies was not statistically significant in any of the 3 years. Regarding the use of 

schoolwide strategies, 18 comparisons were made between treatment and control teacher reports, with 1 

expected to be significant by chance. There were no statistically significant differences between treatment and 

control teacher reports on use of schoolwide strategies in any year (these data are not shown in a table). 
Table 3.14. Impacts on use of SACD classroom materials and teaching strategies—CSP 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD material and teaching strategy

Treatment Control Impact p-value Treatment Control Impact p-value Treatment1 Control Impact p-value 

 Teacher sample size 65 61 57 60 63 62

Use of SACD materials (percent)

 Teacher guides (manuals, curricula) 87.1 69.7 17.4 0.109 88.0* 69.4 18.6 0.002 84.1* 62.8 21.3 0.015

 Student materials (workbooks or sheets) 78.4 70.7 7.6 0.405 87.7* 71.8 15.9 0.004 74.6 65.7 9.0 0.221

 Instructional aids (games, software, videos) 49.2 53.9 -4.7 0.690 54.0 46.0 7.9 0.469 60.5^ 40.4 20.1 0.096

 Giveaways (bookmarks, stickers) 48.1 72.5 -24.4 0.136 51.1 52.1 -1.0 0.940 46.4 60.2 -13.9 0.187

 Children’s literature 69.6 48.2 21.4 0.102 68.9 53.8 15.1 0.119 48.7 45.6 3.1 0.637

 Other types of materials 5.3 6.2 -0.9 0.857 ‡ ‡ -5.5 0.296 10.9 11.6 -0.7 0.915

 Did not use any of these materials ‡ ‡ -6.6 0.206 ‡ ‡ -2.5 0.520 ‡ ‡ -8.6 0.111

Use of teaching strategies (percent)

 Role-playing 83.0* 62.6 20.4 0.034 85.7 72.3 13.5 0.111 87.5 79.9 7.6 0.282

 Cooperative learning 91.2^ 98.8 -7.6 0.083 96.0 98.9 -2.9 0.294 100.0 98.7 1.3 †

 Peer group discussions 94.2^ 82.8 11.4 0.064 93.2 88.5 4.7 0.249 98.3 94.1 4.2 0.255

 Direct instruction of SACD 98.3* 82.9 15.5 0.043 97.0* 79.5 17.6 0.013 99.0 95.0 4.1 0.181

 Skill training 69.0* 36.2 32.8 0.023 53.1 40.9 12.1 0.344 80.2 84.3 -4.1 0.533

 Incorporating SACD into academic 

 curriculum 90.6 78.1 12.5 0.248 88.1^ 74.5 13.6 0.082 96.3 93.5 2.8 0.492

 Parent training 10.2 12.8 -2.6 0.700 ‡ ‡ -5.2 0.146 39.6 30.4 9.3 0.367

 Parent/community involvement 25.4 34.5 -9.1 0.397 31.5 23.1 8.4 0.424 51.7 46.9 4.7 0.600

 Mentoring 59.5 48.2 11.3 0.293 53.4 56.7 -3.2 0.863 84.6^ 67.7 16.9 0.080

 Good behavior notes sent home daily

 or weekly 80.3 92.2 -11.9 0.308 86.9 93.0 -6.1 0.128 96.3 97.5 -1.2 0.734

 Presenting role models 73.8 79.1 -5.3 0.596 75.7 67.9 7.8 0.472 85.8 91.0 -5.3 0.394

See notes at end of table.

238

Chapter 3. Competence Support Program
Table 3.14. Impacts on use of SACD classroom materials and teaching strategies––CSP—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD material and teaching strategy

Treatment Control Impact p-value Treatment Control Impact p-value Treatment1 Control Impact p-value 

Use of teaching strategies (percent)—

 Continued

 Targeted story reading or writing on social 

 and character development themes 86.2 75.8 10.4 0.186 95.8 85.4 10.3 0.104 98.0 93.3 4.7 0.278

 Peer mediation 60.3 40.9 19.4 0.288 60.0 59.7 0.3 0.975 72.2 68.0 4.2 0.738

 Honor roll for positive behavior 69.1 70.5 -1.4 0.889 75.8 84.7 -8.9 0.311 81.5 79.8 1.7 0.843

 Pledges or recitations on social and 

 character development themes 51.6 33.7 17.8 0.228 49.6 48.8 0.8 0.962 74.8 65.7 9.1 0.578

 Guided visualization 74.9* 38.5 36.4 0.004 58.5 55.0 3.5 0.673 75.3 59.0 16.2 0.134

 Student-led/student-assisted instruction 67.2 50.9 16.2 0.198 57.8 46.2 11.6 0.245 78.7 75.2 3.5 0.697

 Journaling 66.8 76.0 -9.1 0.469 61.0 62.7 -1.7 0.881 86.2 81.1 5.1 0.484

 Time out for negative behavior 93.7 93.0 0.7 0.905 87.9 88.2 -0.3 0.971 93.2 93.3 -0.1 0.985

 Daily or weekly rewards for positive 

 behavior 94.4 87.9 6.5 0.453 96.4 97.0 -0.6 0.836 100.0 96.3 3.7 †

 Any strategy 100.0 100.0 0.0 † 100.0 100.0 0.0 † 100.0 100.0 0.0 †

 Number of strategies (mean) 14.2* 12.6 1.6 0.029 13.9 13.3 0.6 0.296 16.5 15.5 1.0 0.278

† Not applicable.

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 One of the treatment group schools that had been a K-5 school in prior years became a K-4 school and continued implementation of the intervention. The Cohort 1 students, who 

were in fifth grade during the 2006-07 school year, went to an intermediate school that had fifth and sixth grades. The intermediate school did not implement the intervention. These 

fifth-grade students, their primary caregivers, and their teachers were included in the spring 2007 data collection and all analyses based upon spring 2007 data.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

239
Chapter 3. Competence Support Program 

240

Participation in Professional Development 

Regarding reported participation in professional development, 27 comparisons were made over 3 years, with 

2 expected to be significant by chance. In Year 1, CSP had a statistically significant effect on treatment 

teachers’ participation in professional development, with more treatment teachers reporting SACD training in 

the past 12 months (impact = 39 percentage points) and more hours of training (by 5 hours on average). In 

terms of specific SACD goals, more treatment teachers reported training in character education (impact = 38 

percentage points) and behavior management (impact = 16.3 percentage points). In Year 3, there was a 

significant impact on teachers’ training in violence prevention and peace promotion (impact = 13 percentage 

points). CSP had a significant impact on the domain of professional development in Year 1. 
Table 3.15. Impacts on teacher-reported SACD professional development—CSP 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD professional development1

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 65 61 57 60 63 62

SACD training in past 12 months (percent) 82.0* 43.0 39.0 0.009 54.9 51.1 3.8 0.723 56.1 49.3 6.8 0.524

Hours of SACD training (mean) 7.9* 2.9 5.0 0.006 3.8 6.2 -2.4 0.375 4.9 4.6 0.3 0.899

Training by goal (percent)

 Violence prevention and peace promotion 21.0 8.2 12.9 0.143 8.4 10.8 -2.4 0.753 ‡* ‡ 13.1 0.036

 Social and emotional development 29.4 9.3 20.1 0.118 18.4 18.6 -0.2 0.978 13.3 9.7 3.6 0.561

 Character education 57.6* 19.7 37.9 0.016 32.7 16.2 16.5 0.106 25.8 12.9 13.0 0.122

 Tolerance and diversity 23.5^ 6.9 16.6 0.071 8.1 18.2 -10.2 0.273 21.0 12.9 8.1 0.231

 Risk prevention and health promotion 14.1 8.2 5.9 0.387 8.1 20.1 -12.0 0.171 16.9 11.7 5.3 0.333

 Civic responsibility and community service 15.1 19.7 -4.7 0.601 ‡ ‡ 1.8 0.566 6.4 0.0 6.4 †

 Behavior management 35.6* 19.3 16.3 0.021 27.8 26.2 1.6 0.904 31.3 22.2 9.1 0.351

† Not applicable.

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 In Year 1, at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) 

procedure to adjust significance levels downward to account for the multiple testing of impacts.

2 One of the treatment group schools that had been a K-5 school in prior years became a K-4 school and continued implementation of the intervention. The Cohort 1 students, who 

were in fifth grade during the 2006-07 school year, went to an intermediate school that had fifth and sixth grades. The intermediate school did not implement the intervention. These 

fifth-grade students, their primary caregivers, and their teachers were included in the spring 2007 data collection and all analyses based upon spring 2007 data.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

241

Chapter 3. Competence Support Program
Chapter 3. Competence Support Program 

242

Attitudes and Practices 

Teachers reported on their enthusiasm for SACD efforts in their schools (these data are not shown in a table) 

by indicating enthusiasm, cooperation, or open dislike. They also reported on the SACD practices of teachers 

and staff members in their school (these data are not shown in a table). These practices included modeling 

positive character and behavior traits with students and fellow teachers, involving students in making 

decisions, giving students a voice in school governance, the school encouraging parent involvement in 

children’s social and character development, and using developmentally appropriate discipline strategies 

rather than punishment for misbehavior. Twenty-seven comparisons were made over 3 years, with 1 expected 

to be significant by chance. There were no statistically significant estimated impacts on teachers’ enthusiasm 

for SACD efforts in their schools in any of the years. There were 2 significant differences in the treatment 

and control teacher reports of the overall use of practices conducive to students’ social and character 

development (out of 6 comparisons made in each year, or 18 overall); in Year 1, fewer treatment teachers 

reported modeling positive traits with students (impact = -22 percentage points), and in Year 3 more 

treatment teachers reported using discipline strategies that focused on promoting development rather than 

simply punishing for misbehavior (impact = 25 percentage points). There was no significant impact on the 

domain in any year. 

Year-by-Year Impacts on Students and Perceptions of School Climate 

The primary research question for the CSP evaluation was this: 

What is the average effect of CSP on children’s social and emotional competence, behavior, 

academics, and perceptions of school climate? 

The first approach to answering this question was to examine the year-by-year impacts of CSP on these 

student and school climate outcomes over the 3 years as the students progressed from third through fifth 

grades. 

Equation (2) (described in chapter 1) was estimated to provide CSP impacts on the 20 outcomes using data 

from the 10 treatment and control schools. For the CSP evaluation, equation (2) excluded the program fixed 

effects (p) and included program-specific covariates and random school effects covariates. Table 3.16 lists 

the covariates used with outcomes from each report in the CSP analysis. 
Chapter 3. Competence Support Program 

243

Table 3.16. Covariates used with outcomes from each report for analysis—CSP 

Potential covariate

CR 

outcome

PCR 

outcome

TRS 

outcome

TRCS 

outcome

 Total number 17 24 25 9

Child-reported

 Female   

 Hispanic   

 Black (non-Hispanic)   

 Other ethnicity   

 Age in years   

 Scales

 Afraid at School

 Altruistic Behavior   

 Empathy

 Engagement with Learning  

 Negative School Orientation  

 Normative Beliefs About Aggression 

 Sense of School as a Community 

 Problem Behavior

 Self-Efficacy for Peer Interactions 

 Victimization at School  

Primary caregiver-reported

 Age in years 

 Completed high school or equivalent   

 Some college   

 Bachelor’s or higher degree   

 Highest level of education in household

 Completed high school or equivalent   

 Some college   

 Bachelor’s or higher degree   

 Mother present in home life 

 Mother and father present 

 Respondent someone other than mother or father 

 Number of people in household

 Household income: $20,000 to $40,000  

 Household income: $40,000 to $60,000  

 Household income: More than $60,000  

 Income-to-poverty-threshold ratio: Below 135 percent

 Income-to-poverty-threshold ratio: 135 to 185 percent

 Full-time employment  

 Part-time employment  

See notes at end of table.
Chapter 3. Competence Support Program 

244

Table 3.16. Covariates used with outcomes from each report for analysis—CSP—Continued

Potential covariate

CR 

outcome

PCR 

outcome

TRS

outcome

TRCS 

outcome

 Parental scales

 APQ-Poor Monitoring and Supervision Subscale

 APQ-Positive Parenting Subscale

 Child-Centered Social Control 

 Confusion, Hubbub, and Order 

 Community Resources

 Community Risk

 Parent and Teacher Involvement

 Child scales

 Altruistic Behavior

 Positive Social Behavior 

 Problem Behavior

Teacher-reported 

 Female 

 Hispanic 

 Black (non-Hispanic) 

 Other ethnicity 

 Total teaching experience 

 Total experience in current school 

 Regular certificate 

 Other certificate 

 Highest degree–bachelor’s 

 Child scales

 Academic Competence and Motivation 

 ADHD-Related Behavior 

 Altruistic Behavior 

 Positive Social Behavior

 Problem Behavior 

 Parent and Teacher Involvement

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

 APQ: Alabama Parenting Questionnaire

 : Covariate used

 Blank cell: Covariate not used

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 3. Competence Support Program 

245

To assess statistical power of the program-level impact estimates, minimum detectable impacts in effect size 

units (MDES) for each outcome measure were calculated for the CSP evaluation (table 3.17). MDES 

represent the smallest impacts in effect size (standard deviation) units that can be detected with a high 

probability (80%). MDES are primarily a function of study sample sizes, the degrees of freedom available for 

statistical tests, and design effects due to clustering (Schochet 2005). For the CSP evaluation, the MDES 

ranged from 0.113 to 0.57 for the child-level outcomes based on the Child, Caregiver, and Teacher Report on 

Student and from 0.258 to 0.625 for the school climate outcomes based on the Teacher Report on Classroom 

and School. In general, the MDES for the school climate outcomes were larger than those for the child-level 

outcomes. 
Chapter 3. Competence Support Program 

246

Table 3.17. Adjusted minimum detectable effect sizes for impact evaluation––CSP 

Outcome measure–Report Year 1 Year 2 Year 3

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 0.129 0.184 0.116

 Normative Beliefs About Aggression–CR 0.113 0.154 0.161

 Empathy–CR 0.238 0.114 0.182

Behavior Domain

 Altruistic Behavior–CR 0.146 0.202 0.171

 Altruistic Behavior–PCR 0.216 0.239 0.169

 Altruistic Behavior–TRS 0.185 0.436 0.570

 Positive Social Behavior–PCR 0.172 0.172 0.170

 Positive Social Behavior–TRS 0.310 0.301 0.247

 Problem Behavior–CR 0.182 0.334 0.168

 Problem Behavior–PCR 0.210 0.136 0.226

 Problem Behavior–TRS 0.352 0.304 0.369

 ADHD-Related Behavior–TRS 0.290 0.206 0.250

Academics Domain

 Engagement with Learning–CR 0.117 0.173 0.201

 Academic Competence and Motivation–TRS 0.194 0.142 0.118

Perceptions of School Climate Domain

 Positive School Orientation–CR 0.113 0.310 0.371

 Negative School Orientation–CR 0.162 0.251 0.354

 Student Afraid at School–CR 0.135 0.222 0.215

 Victimization at School–CR 0.210 0.237 0.284

 Feelings of Safety–TRCS 0.541 0.447 0.562

 Student Support for Teachers–TRCS 0.625 0.309 0.258

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The minimum detectable effect (MDE) formula used in the calculations is as follows:

 

 

 

 

 

 

 

 

 

 

 

 

 

 

where sT and sC are the number of treatment and comparison schools; nT and nC are the average number of students per classroom; 

1 is the intraclass correlation (ICC) at the school level; and factor(df) is a constant that depends on the number of degrees of 

freedom (df) available for analysis (and is 2.802 for the pooled analysis).

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 3. Competence Support Program 

247

Table 3.18 provides the estimates of CSP’s impacts on each of the 20 outcomes over each of the 3 years (60 

impacts in total, with 3 expected to be statistically significant by chance). Of the 60 results, 1 was statistically 

significant. A beneficial impact was found on fourth-graders’ Problem Behavior (Primary Caregiver Report, 

effect size [ES] = –0.21). Substantively important but nonsignificant beneficial impacts were found in Year 2 

for Altruistic Behavior (Teacher Report on Student, ES = 0.47) and Student Afraid at School (Child Report, 

ES = -0.26). Substantively important but nonsignificant detrimental impacts were found on fifth-graders’ 

Altruistic Behavior (Teacher Report on Student, ES = -0.41), Positive Social Behavior (Primary Caregiver 

Report, ES = -0.25), and Feelings of Safety (Teacher Report on Classroom and School, ES = -0.36). 

Application of the heuristics to adjust for multiple comparisons within each outcome domain indicated that 

CSP had a significant detrimental impact on the domain of Behavior in Year 1 and a significant detrimental 

impact on the domain of Perceptions of School Climate in Year 3. 
Table 3.18. Impacts on child and school outcomes—CSP 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Treatment1 ControlEffect 

size p-value 

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 3.04 2.97 0.12 0.199 3.15 3.15 0.00 0.978 3.19 3.22 -0.05 0.581

 Normative Beliefs About Aggression–CR (-) 1.30 1.26 0.07 0.401 1.35 1.29 0.12 0.273 1.48 1.52 -0.05 0.624

 Empathy–CR (+) 2.34 2.26 0.17 0.168 2.18 2.23 -0.11 0.206 2.07 2.07 -0.02 0.888

Behavior Domain2

 Altruistic Behavior–CR (+) 1.21 1.30 -0.11 0.221 1.06 1.00 0.08 0.505 0.99 1.03 -0.06 0.606

 Altruistic Behavior–PCR (+) 2.29 2.25 0.04 0.773 2.33 2.31 0.03 0.849 2.25 2.28 -0.04 0.733

 Altruistic Behavior–TRS (+) 1.42 1.41 0.04 0.713 1.39 1.25 0.47° 0.132 1.31 1.55 -0.41° 0.132

 Positive Social Behavior–PCR (+) 3.04 3.01 0.06 0.515 3.11 3.02 0.17 0.125 3.03^ 3.16 -0.25° 0.056

 Positive Social Behavior–TRS (+) 2.95 3.05 -0.16 0.241 3.02 2.93 0.12 0.426 3.09 3.01 0.12 0.386

 Problem Behavior–CR (-) 0.27 0.25 0.04 0.671 0.32 0.26 0.15 0.428 0.52 0.51 0.01 0.929

 Problem Behavior–PCR (-) 1.54 1.57 -0.11 0.330 1.50* 1.56 -0.21 0.042 1.65 1.50 0.20 0.218

 Problem Behavior–TRS (-) 1.42 1.38 0.09 0.566 1.48 1.50 -0.03 0.852 1.48 1.44 0.07 0.712

 ADHD-Related Behavior–TRS (-) 1.73 1.74 -0.03 0.826 1.70 1.74 -0.06 0.585 1.71 1.74 -0.04 0.764

Academics Domain

 Engagement with Learning–CR (+) 3.70 3.68 0.04 0.645 3.71 3.71 -0.01 0.952 3.61 3.58 0.04 0.745

 Academic Competence and Motivation–

 TRS (+) 3.05 2.97 0.08 0.245 2.96 2.93 0.04 0.622 2.99 2.96 0.03 0.680

See notes at end of table.

248

Chapter 3. Competence Support Program
Table 3.18. Impacts on child and school outcomes––CSP—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Treatment1 ControlEffect 

size p-value 

Perceptions of School Climate Domain3

 Positive School Orientation–CR (+) 2.81^ 2.71 0.15 0.081 2.64 2.65 -0.02 0.922 2.37 2.46 -0.13 0.527

 Negative School Orientation–CR (-) 1.90 1.96 -0.11 0.274 1.95 1.98 -0.05 0.699 2.13 2.19 -0.10 0.590

 Student Afraid at School–CR (-) 2.26 2.40 -0.15 0.103 2.17^ 2.38 -0.26° 0.090 2.34 2.38 -0.04 0.728

 Victimization at School–CR (-) 0.84 0.78 0.08 0.529 0.69 0.70 -0.01 0.947 0.74 0.75 -0.02 0.896

 Feelings of Safety–TRCS (+) 3.42 3.44 -0.03 0.927 3.60 3.64 -0.05 0.834 3.35 3.58 -0.36° 0.246

 Student Support for Teachers–TRCS (+) 3.53 3.56 -0.05 0.890 3.55 3.48 0.10 0.551 3.37 3.45 -0.11 0.550

* Treatment group significantly different from control group at the .05 level.

^ Treatment group significantly different from control group at the .10 to > .05 level.

o Substantive (but nonsignificant at .05 level) effect size of ≥ .25 or ≤ -.25.

1 One of the treatment group schools that had been a K-5 school in prior years became a K-4 school and continued implementation of the intervention. The Cohort 1 students, who 

were in fifth grade during the 2006-07 school year, went to an intermediate school that had fifth and sixth grades. The intermediate school did not implement the intervention. These 

fifth-grade students, their primary caregivers, and their teachers were included in the spring 2007 data collection and all analyses based upon spring 2007 data.

2 In Year 1, the statistical model used to estimate impacts on the individual outcomes was re-estimated using a composite of all the outcome variables under a domain. The domain 

was found significant if the impact on the composite was significant. The composite was formed by standardizing each outcome variable using its standard deviation, combining the 

values of the outcome variables, and taking the average of the final value.

3 Impact on domain found statistically significant and negative in Year 3 based on the fourth heuristic in which the statistical model used to estimate impacts on the individual outcomes 

was re-estimated using a composite of all the outcome variables under a domain. The domain was found significant if the impact on the composite was significant. The composite was 

formed by standardizing each outcome variable using its standard deviation, combining the values of the outcome variables, and taking the average of the final value.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All Impact estimates were calculated using regression models where each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. See Table 1.5 for 

information about the measures used to create the outcome variables. The effect size was calculated by dividing the estimated Impact by the standard deviation of the outcome 

measure for the control group. The number of results found significant was no more than expected by chance.

SOURCE: The Social and Character Development (SACD) Research Program.

249

Chapter 3. Competence Support Program
Chapter 3. Competence Support Program 

250

Impacts on Child Outcomes Over Time 

CSP impacts on the child outcomes over time were estimated using growth curve models by examining 

treatment and control group differences in the trajectories of student outcomes during the follow-up period 

while accounting for clustering at the school level. The growth curve models are estimated using a three-level 

hierarchical linear model, where Level 1 corresponds to time, Level 2 to students, and Level 3 to schools 

(described in chapter 1). 

Table 3.19 provides the estimates of CSP impacts on the growth in student outcomes over the 3 years. The 

estimated impacts range in effect size units (absolute value) from 0.00 to 0.30. CSP had a statistically 

significant beneficial effect on 1 of the 18 estimated impacts (1 of 18 is expected to be significant by chance), 

Victimization at School (Child Report), with an effect size of –0.09. There was a substantively important but 

nonsignificant detrimental impact on Altruistic Behavior (Teacher Report on Student). 
Table 3.19. Impacts on growth of child outcomes—CSP 

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2Treatment Control

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 2.94 0.13 0.12 0.00 0.00 0.03 0.974

 Normative Beliefs About Aggression–CR (-) 1.20 0.12 0.06 0.05 0.11 0.04 0.207

 Empathy–CR (+) 2.40 -0.12 -0.12 -0.01 -0.03 0.02 0.560

Behavior Domain

 Altruistic Behavior–CR (+) 1.41 -0.20 -0.16 -0.04 -0.04 0.03 0.208

 Altruistic Behavior–PCR (+) 2.31 -0.01 -0.01 0.01 0.01 0.03 0.872

 Altruistic Behavior–TRS (+) 1.40 -0.08^ 0.07 -0.15 -0.30 0.07 0.075

 Positive Social Behavior–PCR (+) 3.01 0.01 0.04 -0.02 -0.04 0.02 0.247

 Positive Social Behavior–TRS (+) 2.96 0.04 0.01 0.03 0.03 0.05 0.580

 Problem Behavior–CR (-) 0.19 0.12 0.08 0.03 0.07 0.03 0.273

 Problem Behavior–PCR (-) 1.57 0.01 -0.01 0.02 0.05 0.01 0.219

 Problem Behavior–TRS (-) 1.36 0.07 0.05 0.02 0.05 0.03 0.563

 ADHD-Related Behavior–TRS (-) 1.76 -0.02 0.00 -0.01 -0.02 0.03 0.717

Academics Domain

 Engagement with Learning–CR (+) 3.69 -0.02 -0.01 -0.01 -0.01 0.03 0.871

 Academic Competence and Motivation–TRS (+) 2.94 0.01 0.03 -0.02 -0.02 0.03 0.593

See notes at end of table.

251

Chapter 3. Competence Support Program
Table 3.19. Impacts on growth of child outcomes—CSP—Continued

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2Treatment Control

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 3.03 -0.26 -0.19 -0.07 -0.09 0.05 0.213

 Negative School Orientation–CR (-) 1.87 0.08 0.10 -0.01 -0.02 0.04 0.744

 Student Afraid at School–CR (-) 2.41 -0.09 -0.03 -0.06 -0.06 0.03 0.111

 Victimization at School–CR (-) 0.82 -0.09* -0.01 -0.08 -0.09 0.03 0.050

* Treatment group significantly different from control group at the .05 level.

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 Pertains to the estimated slope of the outcome for the treatment or control groups.

2 The average score at implementation is calculated across treatment and control groups, using regression models for adjustment on covariates. 

3 Estimated difference between the slope of the treatment and control groups.

4 Effect size: The slope of the treatment group minus the slope of the control group divided by the standard deviation of the outcome for the program’s control group (the standard 

deviation is calculated without accounting for school-level clustering or regression adjustments).

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using HLM 6.06. Sample weights were used in all analyses to give 

(1) each school equal weight in each program (within each time period) and (2) each time period equal weight within the analysis. See table 1.5 for information about the measures 

used to create the outcome variables.

SOURCE: The Social and Character Development (SACD) Research Program.

252

Chapter 3. Competence Support Program
Chapter 3. Competence Support Program 

253

Summary 

As part of the Social and Character Development (SACD) Research Program, researchers at the North 

Carolina site implemented and evaluated CSP, a program focused on social skills training, social and 

emotional learning, behavior management, and social dynamics management. Ten public schools in two 

school districts in North Carolina were recruited by the research team and randomly assigned to treatment 

and control conditions to determine the impact of CSP on social and character development activities in the 

schools and on the child outcome domains of Social and Emotional Competence, Behavior, Academics, and 

Perceptions of School Climate. 

Analyses of the initial characteristics of the sample (students, caregivers, communities, teachers, and schools) 

indicated that randomization to treatment and control status produced groups that were relatively similar at 

the start of the study (with 2 out of 84 comparisons statistically significantly different, fewer than would be 

expected by chance). The data on the initial level of SACD activity led to two findings. First, treatment 

teachers reported greater use of and training in SACD activities than did control teachers, and they did so 

more often than would be expected by chance (6 out of 62 comparisons, with 3 expected significant by 

chance). There are two potential causes for this finding, and the analysis cannot be used to determine whether 

the reason for such a difference was that the two groups did differ on their initial use of SACD activities (i.e., 

that randomization did not create similar treatment and control groups) or the fact that the training of all 

treatment teachers and CSP implementation began before the initial data were collected (by 4 weeks), which 

may have influenced the teacher reports. Because it is likely (though unproven) that the training and 

implementation affected the teacher reports, these data were not considered appropriate for use as a baseline 

measure of SACD activities and training in the treatment schools. 

Second, these data indicate that the control condition for the SACD project was not a ―no treatment‖ control 

but a ―standard practice‖ control. Because the control teachers were not affected by the implementation of 

the SACD programs before data collection, their reports reflected standard practice in the control schools. 

Standard practice at the control schools included 52 percent to 93 percent of teachers reporting the use of 

any SACD activities, 88 percent of teachers reporting the use of specific materials in conjunction with these 

activities, 100 percent reporting the use of at least one of the specified instructional strategies, and 52 percent 

reporting participation in SACD training over the past 12 months. 

Analyses of CSP impacts on the use of SACD activities in the schools revealed impacts on the use of such 

activities (18 out of 90 comparisons) and related materials and strategies (8 out of 87 comparisons) across the 

3 years, and use of more professional development activities for treatment teachers (5 out of 27 

comparisons). These same measures in the control schools across the 3 years of the study confirm that use of 

these activities in the control schools constituted their standard practice. 

Of the 20 child-level outcomes representing the four domains of Social and Emotional Competence, 

Behavior, Academics, and Perceptions of School Climate assessed in each of the 3 years of the study (a total 

of 60 results), CSP had a statistically significant beneficial impact on Problem Behavior (Primary Caregiver 

Report) in Year 2. A growth curve analysis was used to analyze the change over time in these same outcomes 

between initial data collection and the final outcome data collection at the end of the study. Of the 18 childlevel outcomes assessed, CSP had a significant beneficial impact on the trajectory of 1, Victimization at 

School (Child Report). 

The SACD evaluation did not find evidence to support the hypothesis that CSP had beneficial impacts on 

students’ social and character development. Such results could be caused by the inability of the program to 

cause such change, possibly because the theory of action for the program is incomplete or the activities to 

carry out that theory are not effective. 

However, these results may also be due to the inability of the evaluation to observe such a change due to the 

control condition, differences in initial conditions, the level of nonparticipation, or the sample size. The 
Chapter 3. Competence Support Program 

254

control schools continued using their standard SACD activities, and these turned out to be high in quantity 

and broad in scope. Under the Student Citizen Act of 2001, character education was made a required part of 

the standard curriculum for North Carolina schools starting in the 2002-03 school year, and this may have 

been a factor in high SACD activity in the control schools. While the CSP program had significant positive 

impact on the amount and type of SACD activities, the resulting difference in the amount of SACD activities 

between the treatment and control school may not have been large enough to cause significant differences in 

the student outcomes. 

Second, there were differences between the treatment and control groups that might also be related to 

differences in outcomes. Treatment teachers had less teaching experience in their current schools and 

treatment students reported greater negative school orientation (both statistically significant differences, 

although no more than expected by chance). For Year 3, the treatment students at one school did not receive 

the intervention when they were transferred to a new school for fifth grade when their former school 

dropped fifth grade. 

Third, about one-third of the students in the sample universe did not take part because of nonconsent or 

noncompletion of the surveys. As a determination could not be made as to whether the students not taking 

part differed significantly from those who did take part, the evaluation’s results are valid only for the students 

who took part. If the students not taking part were different, and if they would have responded better to CSP 

than to the SACD activities occurring in the control schools, then the evaluation could have underestimated 

the program’s impact. In addition, significantly lower percentages of treatment teachers and students 

provided data in Year 3, and there were significantly larger percentages of treatment leavers in Years 2 and 3. 

Finally, the sample size of 10 schools and the resulting higher MDES than for the multiprogram evaluation 

may have reduced the likelihood of detecting statistically significant effects. However, it should be noted that 

67 percent of the MDES for the 60 outcomes used in the year-by-year analysis were below 0.25 (42% were 

below 0.20). In addition, 5 of the 60 outcomes were found to be substantively important, but 3 of these had 

detrimental impacts. 
255

Chapter 4. Love In a Big World 

Vanderbilt University

(Tennessee Site)

Intervention 

Researchers at Vanderbilt University (Tennessee site) evaluated the Love In a Big World (LBW) program as 

implemented by program staff. The curriculum is based on research conducted by the Social Development 

Research Group at the University of Washington and on asset development research conducted by the Search 

Institute. This character education program includes classroom lesson plans, staff and principal training, a 

peer recognition program, service projects, assembly programs, motivational morning announcements, and 

newsletters that aim to teach children positive character traits and moral principles and how to apply them in 

their lives. Table 4.1 describes LBW’s general characteristics (panel 1), the types of instruction and strategies 

used (panel 2), the professional development provided for those implementing the program (panel 3), and the 

social and character development activities (panel 4) and outcomes (panel 5) addressed by the program. The 

program is designed to encourage wise decisionmaking, improve peer relationships, and promote dialogue 

about character choices. The program includes the following elements: 

 Adults teach and model character traits such as honesty, kindness, responsibility, self-control, 

moderation, perseverance, respect, fairness, cooperation, and love. 

 Classroom strategies for teaching character traits involve story reading, writing, discussion, song, and 

other classroom activities that occur daily in 10- to 15-minute lessons for 30 weeks. Teachers utilize 

behavior management strategies to reward exhibition of character traits and are encouraged to model 

and reinforce social skills throughout the day. 

 Schoolwide strategies for teaching character traits include weekly announcements, assemblies, service 

projects, parent newsletters, and visual artifacts that illustrate character traits (e.g., posters and 

banners). 

 Principals, teachers, and school staff participate in a workshop on lesson planning and on the 

logistics of program implementation at the start of each school year. In the first year of 

implementation, this workshop is 3 hours long, and in subsequent years it is 1 hour long. 
Chapter 4. Love In a Big World 

256

Table 4.1. Love In a Big World

Panel 1: General characteristics

Target population 

 Universal

Program components

 Peer: In and out of class

 Parent: Contact and involvement

 Classroom: Lessons and behavior management

 Schoolwide: Planned events, modeling, and program artifacts

 Community: None or not major focus

 Training: Pretraining and ongoing

Level of integration

 Add-on curriculum and schoolwide activities

Flexibility

 Manualized: Curriculum guidebook

 Adaptability: Less adaptable

See notes at end of table.

Panel 2: Description of instruction and strategies

Classroom

 Lessons

 Who delivers: Teacher

 Activities and tools: Story reading, writing, interdisciplinary activities, rewards, singing

 Content: Character traits and moral virtues

 Frequency: Daily 10- to 15-minute lessons for 30 weeks

 Strategies

 Who delivers: Teacher

 Activities and tools: Recognition and reinforcement of good behavior; modeling of character traits

 Frequency: Daily

Supplement to classroom

 Parent newsletters

Schoolwide activities 

 Weekly announcements, occasional assemblies, two service projects, program artifacts

See notes at end of table.
Chapter 4. Love In a Big World 

257

Table 4.1. Love In a Big World––Continued

Panel 3: Professional development

Pre-implementation

 Teachers

 Content: Workshop on lesson plans and logistics of program

 Duration: 3 hours

 Other

 Content: Principal and staff training on same material as teachers

 Duration: 3 hours

Ongoing consultation

 Teachers

 Content: Faculty boosters with reminders about program implementation and discussion of challenges

 Duration: 24 weekly sessions; biannual meeting

 Other

 Content: Program coordinator; calls with program developer to improve schoolwide implementation

 Duration: Once every 6 months

See notes at end of table.

Panel 4: Activities for SACD goals

Violence prevention and peace promotion  Risk prevention and health promotion

Social and emotional development  Civic responsibility and community service 

Character education  Behavior management

Tolerance and diversity 

See notes at end of table.

Panel 5: SACD outcomes addressed 

Engagement with Learning Empathy 

Academic Competence and Motivation Positive School Orientation 

Altruistic Behavior  Negative School Orientation 

Positive Social Behavior  Student Afraid at School 

Problem Behavior  Victimization at School

Self-Efficacy for Peer Interactions Feelings of Safety 

Normative Beliefs About Aggression Student Support for Teachers 

NOTE: Abbreviations are

 : Activity or outcome addressed

 Blank cell: Activity or outcome not addressed

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 4. Love In a Big World 

258

Sample and Random Assignment 

The Tennessee research team recruited a total of 12 public elementary schools in 2 school districts in 2 

separate counties in Tennessee. The 12 schools were randomly assigned to treatment and control conditions 

prior to the fall 2004 data collection period.48 A two-step process was used. First, a computer-generated 

pairwise matching algorithm developed by Mathematica Policy Research, Inc. (MPR) identified the best 

pairwise matches across the 12 schools based on (a) school district, (b) percentages of students eligible for 

free or reduced-price lunch, (c) total mobility rate, (d) Tennessee assessment scores, (e) total number of 

students, (f) promotion level, and (g) percentages of students who were members of a minority population. 

Second, using the flip of a coin, 1 school in each matched pair was assigned to either the intervention or 

control condition, resulting in 6 schools receiving LBW and 6 schools acting as control schools that 

continued to implement the social and character development activities that constituted their standard 

practice. Assignment to the treatment or control condition was at the school level and therefore limited the 

risk of contamination between treatment and control classrooms. After the first year, 1 control school 

discontinued involvement because it became a magnet school, resulting in a total of 11 schools in the study, 

with 6 in the treatment condition and 5 in the control condition. 

The original student sample (the cohort of students in the third grade in the 12 schools in fall 2004) 

numbered 986 students (548 treatment and 438 control). Table 4.2 documents the changes in the sample over 

the three spring follow-up data collection points. Over time, new entrants to the cohort became a larger 

percentage of the sample, making up 28 percent of the sample by the spring of Year 3. There were no 

statistically significant differences between the treatment and control groups in the numbers of new entrants. 

The percentage of the sample made up of the original cohort further declined as students left the schools. By 

Year 3, approximately 31 percent of the original sample had left. In Years 1 and 3 there were statistically 

significant differences in ―leavers‖ between treatment and control groups (in both cases, there were fewer in 

the treatment group). 

 

48 In Year 2 (fall 2005), the Tennessee research team recruited two more schools, one assigned to treatment and one 

assigned to control, which were followed to the end of the study (spring 2007). A description of this second cohort and 

all relevant findings can be found in appendix A. The data from this second cohort are not included in the analyses and 

results reported in this chapter. 
Table 4.2. Sample––LBW 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Characteristic Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 School sample size 12 6 6 12 6 6 11 6 5

1

11 6 5

1

 Student sample size 986 548 438 1,007 565 442 959 556 403 944 567 377

Stayers † † † 926 523 403 758 438 320 678 403 275

New entrants † † † 81 42 39 201 118 83 266 164 102

 New entrants as a percent of

 spring enrollment † † † 8.0 7.4 8.8 21.0 21.2 20.6 28.2 28.9 27.1

Total leavers (from original 

 cohort) † † † 60 25 35 228 110 118 308 145 163

 Leavers as a percent of fall 

 2004 enrollment † † † 6.1 4.6* 8.0 23.1 20.1 26.9 31.2 26.5** 37.2

Number of students per school

 (mean) 82 91 73 84 94 74 87 93 76 86 95 75

Range of number of students per

 school 40-138 57-138 40-111 44-141 58-141 44-111 45-151 45-151 45-112 34-164 47-164 34-113

† Not applicable.

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

1 One control group school discontinued involvement because it became a magnet school.

SOURCE: The Social and Character Development (SACD) Research Program.

259

Chapter 4. Love In a Big World
Chapter 4. Love In a Big World 

260

Implementation 

Training 

Teachers, principals, and school staff members in the intervention schools received 3 hours of program 

implementation training in the form of a workshop prior to the beginning of the first school year (table 4.1, 

panel 3). In the second and third years of the study, a 1-hour workshop was provided at the start of the 

school year. Teachers had access to ongoing program implementation support during the school year through 

24 weekly faculty booster sessions that were provided to each school’s program coordinator and could be emailed to the faculty. There were also biannual meetings that included reminders about lesson planning and 

discussions of program implementation challenges. In addition, the school program coordinator received 

monthly (in Year 1) or biannual (in later years, in accordance with school feedback) calls from program staff 

to answer questions and provide assistance or any required materials. 

Data Collection 

MPR collected the multiprogram child, teacher, and school data at the Tennessee site. Table 4.3 shows the 

school year milestones and dates of implementation for the Tennessee site. Data were collected in the fall and 

spring of the first 2 years and the spring of Year 3. The fall 2004 multiprogram data collection began on 

August 23, 2004, and ended on September 9, 2004. Fall data collection occurred before implementation of 

LBW began. The spring data collection window was from May 2, 2005, to May 20, 2005. The program had 

been implemented for 33 weeks at the time of the spring data collection and for 34 weeks from the end of the 

fall data collection. Year 2 followed a similar pattern, with implementation occurring at the start of the school 

year, fall data collection occurring 5 weeks later, and spring data collection occurring 31 weeks after fall data 

collection (and 40 weeks after the start of implementation). In spring 2007, data collection occurred 40 weeks 

after the start of implementation. Data collection took from 3 to 5 weeks at each collection point. 
Chapter 4. Love In a Big World 

261

Table 4.3. Data collection dates––LBW 

Data collection schedule

Year 1

(Fall 3rd grade)

Year 1

(Spring 3rd grade)

Year 2

(Fall 4th grade)

Year 3

(Spring 4th grade)

Year 4

(Spring 4th grade)

 School sample size 12 12 11 10 10

School year dates 

 First day of school 7/14/04;

7/4/041

† 7/26/05; 

8/11/054

† 7/24/06; 

8/10/066

 Start of implementation 9/13/04 † First day † First day

 Last day of school † 5/26/05;

6/2/052

† 5/25/06;

6/1/065

5/25/07;

6/1/077

Data collection 

 Start 8/23/04 5/2/05 8/29/05 5/1/06 5/1/07

 End 9/9/04 5/20/05 9/23/05 5/19/06 5/16/07

Calendar weeks from 

 program implementation 

 to start of fall 2004 data 

 collection †

3

† † † †

Calendar weeks from 

 start of school to start 

 of fall data collection 6

1

† 5

4

† †

Calendar weeks from end 

 of fall data collection to 

 start of spring data 

 collection † 34 † 31 †

Calendar weeks from 

 program implementation 

 to start of spring data 

 collection † 33 † 404406

† Not applicable.

1 Eight schools started on 7/14/04 and four started on 8/4/04. The calculation of weeks from the start of school to the start of fall 

2004 data collection uses 7/14/04 as the first day of the school year.

2 Eight schools ended on 5/26/05 and four ended on 6/2/05. The calculation of weeks from program implementation to the end of the 

school year uses 5/26/05 as the last day of the school year.

3 Program implementation occurred after fall 2004 data collection.

4 Eight schools started on7/25/05 and 7/27/05 and four started on 8/11/05. The calculation of weeks from the start of school to the 

start of data collection uses 7/26/05 as the first day of the school year.

5 Eight schools ended on 5/25/06 and four ended on 6/1/06.

6 Nine schools started between 7/24/06 and 7/26/06 and two started on 8/10/06. The calculation of weeks from the start of school to 

the start of spring 2007 data collection uses 7/26/06 as the first day of the school year.

7 Six schools ended on 5/25/07, three ended on 5/31/07, and two ended on 6/1/07.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 4. Love In a Big World 

262

Consent Rates, Completion Rates, and Percentage of Sample With Data 

The actual number of student, primary caregiver, and teacher reports available for analysis was smaller than 

the number in the sample because consent and completion rates were less than 100 percent. Primary 

caregivers had to provide consent before children could complete the Child Report, before teachers could 

complete the Teacher Report on Student, and before they themselves completed the Primary Caregiver 

Report. Teachers also had to provide consent before completing the Teacher Report on Classroom and 

School. Of those with consent, not all completed their respective reports. Table 4.4 shows the consent rates, 

completion rates, and percentages of sample with data for each of the four reports over the 3 years. For the 

Child Report and two teacher reports, completion rates ranged from 84 percent to 100 percent, so the 

consent rates had the most influence on what percentage of the sample had data. There were no statistically 

significant differences between treatment and control groups in consent rates for any of these reports. For the 

Primary Caregiver Report, the completion rates dropped over time from 95 percent to 74 percent. 

The percentages of the sample with Child Report data ranged from 58 percent to 62 percent over the 3 years. 

The percentages of students with information from the Teacher Report on Student ranged from 62 percent 

to 68 percent. The percentages of students with data from the Primary Caregiver Report ranged from 46 

percent to 65 percent and declined over time. The percentages of teachers with data from the Teacher Report 

on Classroom and School ranged from 84 percent to 97 percent. There were no statistically significant 

differences between treatment and control conditions in percentages of students with data for any of the four 

reports. 
Table 4.4. Consent rates, completion rates, and percentage of sample with data from each report––LBW 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 Student sample size 986 548 438 1,007 565 442 959 556 403 944 567 377

Child Report (percent)

 Primary caregiver consent rate 65.7 67.9 63.0 66.3 68.1 64.0 64.4 65.5 63.0 64.9 66.0 63.4

 Student completion rate 91.5 90.6 92.8 91.6 89.9 94.0 93.7 92.3 95.7 90.2 89.6 91.2

 Students with data160.1 61.5 58.4 60.8 61.4 60.0 60.4 60.4 60.3 58.6 59.1 57.8

Primary Caregiver Report (percent)

 Primary caregiver consent rate 65.7 67.9 63.0 66.2 68.0 64.0 64.2 65.1 63.0 63.9 64.6 62.9

 Primary caregiver completion rate 95.2 95.4 94.9 80.2 79.2 81.6 79.7 81.2 77.6 73.5 70.8 77.6

 Primary caregivers with data162.6 64.8 59.8 53.1 54.0 52.0 51.2 52.9 48.9 46.9 45.7 48.8

Teacher Report on Student (percent)

 Primary caregiver consent rate265.7 67.9 63.0 66.3 68.1 64.0 64.4 65.5 63.0 64.9 66.0 63.4

 Teacher completion rate 97.8 97.6 98.2 99.7 99.5 100.0 100.0 100.0 100.0 98.0 96.8** 100.0

 Students with data164.3 66.3 61.9 66.1 67.8 63.8 64.4 65.5 63.0 63.7 63.8 63.4

Teacher Report on Classroom and School 

 (3rd- to 5th-grade teachers) (percent)

 Teacher consent rate 99.3 100.0 98.5 100.0 100.0 100.0 97.0 97.4 96.3 100.0 100.0 100.0

 Teacher completion rate 89.0 84.0* 95.3 95.9 95.2 96.9 90.6 92.1 88.5 91.9 90.4 94.3

 Teachers with data188.4 84.0 93.8 95.9 95.2 96.9 87.9 89.7 85.2 91.9 90.4 94.3

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

1 Calculated as consent rate x completion rate.

2 The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

263

Chapter 4. Love In a Big World
Chapter 4. Love In a Big World 

264

Responses from students in the original cohort (stayers) and new entrants in the LBW sample were examined 

to investigate possible differences between the two groups in consent rates, completion rates, and the 

percentages of sample with data that might affect outcome data (table 4.5). In all 3 years, stayers had 

significantly higher consent rates (by 18 to 32 percentage points) and higher percentages of sample with data 

(by 8 to 33 percentage points) than did new entrants. 
Table 4.5. Consent rates, completion rates, and percentage of sample with data: Stayers versus new entrants––LBW 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total Stayers

New 

entrants Total Stayers

New 

entrants Total Stayers

New 

entrants

 Student sample size 1,007 926 81 959 758 201 944 678 266

Child Report (percent)

 Primary caregiver consent rate 66.3 68.9*** 37.0 64.4 69.7*** 44.8 64.9 70.1*** 51.9

 Student completion rate 91.6 91.5 93.3 93.7 93.2 96.7 90.2 89.9 91.3

 Students with data160.8 63.1*** 34.6 60.4 64.9*** 43.3 58.6 63.0*** 47.4

Primary Caregiver Report (percent)

 Primary caregiver consent rate 66.2 68.9*** 35.8 64.2 69.4*** 44.8 63.9 68.9*** 51.1

 Primary caregiver completion rate 80.2 80.1 82.8 79.7 80.0 77.8 73.6 71.5* 80.9

 Primary caregivers with data153.1 55.2*** 29.6 51.2 55.5*** 34.8 47.0 49.3* 41.4

Teacher Report on Student (percent)

 Primary caregiver consent rate266.3 68.9*** 37.0 64.4 69.7*** 44.8 64.9 70.1*** 51.9

 Teacher completion rate 99.7 99.8 96.7 100.0 100.0 100.0 98.0 98.3 97.1

 Students with data166.1 68.8*** 35.8 64.4 69.7*** 44.8 63.7 68.9*** 50.4

* Stayers significantly different from new entrants at the .05 level.

*** Stayers significantly different from new entrants at the .001 level.

1 Calculated as consent rate x completion rate.

2 The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

265

Chapter 4. Love In a Big World
Chapter 4. Love In a Big World 

266

Fidelity of Implementation

Each year, LBW’s six treatment group schools were individually rated for quantity and quality of program 

implementation by two raters from the research team. The global measure of fidelity for the multisite study 

was used; inter-rater reliability was measured using Cronbach’s alpha. In Years 2 and 3 these values were 0.83 

and 0.97, respectively; in Year 1, the inter-rater reliability coefficient was -0.08, indicating that there was lack 

of agreement among the two raters, and thus Cronbach’s alpha was not calculated. The ratings were 

combined into a single consensus rating and used to identify schools with high implementation fidelity. In 

Year 1 one treatment school was identified as having high fidelity and in Years 2 and 3 two treatment schools 

were identified as having high fidelity. Cohen’s kappa was used as the measure of agreement when identifying 

schools as high fidelity, and it equaled 1.00 in Years 2 and 3. 

Initial Characteristics 

This section examines the initial characteristics of the students, teachers, and schools participating in the 

LBW evaluation. These characteristics were collected from students who were enrolled in the third grade at 

the study schools in fall 2004, as well as from their primary caregivers and third-grade teachers. In addition, 

third-, fourth-, and fifth-grade teachers and principals in the study schools provided information about 

activities related to social and character development in these schools. Documenting the characteristics of 

students, teachers, and schools and initial measures of key outcomes at a point before the intervention began 

operating helps to determine whether the random assignment of schools to treatment and control status 

produced treatment and control groups with similar distributions of observed characteristics. As noted in the 

following discussion, 4 significant differences between the treatment and control students, teachers, and 

schools were found in the observed characteristics, including the level of SACD activity in the classroom and 

school (3 of the 4 significant differences out of 62 comparisons, with 3 expected to be significant by chance). 

Characteristics of Children, Their Families, and Communities 

There were no significant differences between the treatment and control groups in the observed student, 

caregiver, and community characteristics (table 4.6). For students, the mean age was 8.1 years. The sample 

contained roughly equal percentages of girls (52%) and boys (48%). The sample was ethnically diverse. White 

non-Hispanic students made up 65 percent of the sample, Black non-Hispanic students made up 21 percent, 

and Hispanic students made up 8 percent. 

The sample was also diverse in its levels of family income, education levels of primary caregivers of the 

children in the sample, and family situation. Twenty-eight percent of children lived in a household where the 

income was 135 percent of the federal poverty level or lower, which is the income threshold for eligibility for 

free school meals. Slightly less than 9 percent of primary caregivers had not completed high school. Nearly 

two-thirds of the children (64%) lived with both their mother and their father. There were no significant 

differences between the treatment and control groups in these characteristics. 

The mean values of the outcomes for children’s behavior and attitudes as reported by the primary caregiver, 

child, and teacher at initial data collection in fall 2004 are shown in table 4.7. There was 1 significant 

difference (out of 18 comparisons) between the treatment and control groups: the treatment group had a 

lower Altruistic Behavior score (as reported by caregivers) than did the control group. 
Chapter 4. Love In a Big World 

267

Table 4.6. Initial characteristics of children, their families, and communities—LBW 

Characteristic Total Treatment Control

 Student sample size 617 355 262

Student demographics

 Gender (percent)

 Male 48.0 47.7 48.4

 Female 52.0 52.3 51.6

 Race/ethnicity (percent)

 White (non-Hispanic) 64.9 65.7 64.1

 Black (non-Hispanic) 21.1 22.4 19.8

 Hispanic 7.9 6.1 9.8

 Other 6.1 5.8 6.3

 Age (in years) (mean) 8.1 8.1 8.1

Primary caregiver and family characteristics

 Primary caregiver’s age (in years) (mean) 35.5 35.8 35.2

 Primary caregiver’s race/ethnicity (percent)

 White (non-Hispanic) 70.3 70.0 70.7

 Black (non-Hispanic) 20.6 22.3 19.0

 Hispanic 6.7 6.2 7.2

 Other 2.3 1.5 3.1

 Primary caregiver’s education (percent)

 Did not complete high school 8.8 5.4 12.3

 Completed high school or equivalent 25.8 22.3 29.3

 Some college 40.4 42.2 38.6

 Bachelor’s or higher degree 25.0 30.2 19.8

 Primary caregiver’s employment (percent)

 Full-time 54.5 56.9 52.1

 Other 45.5 43.1 47.9

 Primary caregiver’s marital status (percent)

 Married 63.9 65.9 61.8

 Other 36.1 34.1 38.2

 Students who live in one household (percent) 91.8 91.8 91.8

 Number of individuals in household (mean) 4.3 4.3 4.3

 Primary caregiver’s relationship to child (percent)

 Mother (stepmother) 84.9 86.0 83.8

 Father (stepfather) 10.4 9.6 11.2

 Other relative/nonrelative 4.7 4.4 4.9

See notes at end of table.
Chapter 4. Love In a Big World 

268

Table 4.6. Initial characteristics of children, their families, and communities—

LBW—Continued

Table 4.6. 

Characteristic Total Treatment Control

 Student lives with (percent)

 Mother (stepmother) and father (stepfather) 64.3 66.6 61.9

 Mother (stepmother) only; father (stepfather) not present 28.2 27.3 29.1

 Father (stepfather) only; mother (stepmother) not present 3.8 2.9 4.7

 Other relative/nonrelative, parents not present 3.7 3.1 4.3

 Highest education of anyone in household (percent)

 Did not complete high school 5.6 2.7 8.4

 Completed high school or equivalent 20.5 18.7 22.2

 Some college 40.6 40.8 40.5

 Bachelor’s or higher degree 33.4 37.8 28.9

 Total household income (percent)

 Less than $20,000 24.1 19.5 28.8

 $20,000 to $39,999 22.4 20.2 24.6

 $40,000 to $59,999 19.9 19.8 20.1

 $60,000 or more 33.5 40.5 26.5

 Income-to-poverty-threshold ratio—Below 135 percent (percent) 28.1 24.6 31.6

 Income-to-poverty-threshold ratio—135 to 185 percent (percent) 18.1 14.0 22.1

 Income-to-poverty-threshold ratio—Above 185 percent (percent) 53.8 61.4 46.3

 Alabama Parenting Questionnaire—Poor Monitoring and

 Supervision Subscale (mean) 1.1 1.1 1.2

 Alabama Parenting Questionnaire—Positive Parenting

 Subscale (mean) 3.5 3.6 3.5

 Confusion, Hubbub, and Order Scale (mean) 2.2 2.2 2.2

Community characteristics (mean)

 Community Risks Scale 1.2 1.2 1.2

 Community Resources Scale 2.6 2.6 2.7

 Child-Centered Social Control Scale 3.1 3.2 3.1

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 4. Love In a Big World 

269

Table 4.7. Mean scores and standard deviations for initial outcome measures of sample––LBW 

Total Treatment Control

Outcome measure–Report Range Mean SD Mean SD Mean SD

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 1-4 3.0 0.6 3.0 0.6 3.0 0.7

 Normative Beliefs About Aggression–CR 1-4 1.2 0.4 1.2 0.4 1.2 0.4

 Empathy–CR 1-3 2.5 0.4 2.5 0.4 2.5 0.4

Behavior Domain

 Altruistic Behavior–CR 0-3 1.4 0.9 1.5 0.9 1.4 0.9

 Altruistic Behavior–TRS 1-4 1.3 0.4 1.3 0.4 1.4 0.4

 Altruistic Behavior–PCR 1-4 2.3 0.7 2.2** 0.6 2.4 0.7

 Positive Social Behavior–TRS 1-4 3.2 0.7 3.3 0.7 3.2 0.7

 Positive Social Behavior–PCR 1-4 3.0 0.5 3.1 0.5 3.0 0.5

 Problem Behavior–CR 0-3 0.1 0.3 0.1 0.3 0.2 0.3

 Problem Behavior–TRS 1-4 1.3 0.3 1.3 0.4 1.3 0.3

 Problem Behavior–CR 1-4 1.6 0.3 1.6 0.3 1.6 0.3

 ADHD-Related Behavior–TRS 1-4 1.7 0.6 1.7 0.7 1.6 0.6

Academics Domain

 Academic Competence and Motivation–TRS 1-5 3.1 0.9 3.1 0.9 3.1 0.9

 Engagement with Learning–CR 1-4 3.7 0.5 3.7 0.6 3.8 0.5

Perceptions of School Climate Domain

 Positive School Orientation–CR 1-4 3.3 0.6 3.4 0.6 3.3 0.6

 Negative School Orientation–CR 1-4 1.7 0.6 1.7 0.6 1.6 0.5

 Student Afraid at School–CR 1-4 2.4 0.9 2.4 0.9 2.3 0.9

 Victimization at School–CR 0-3 0.7 0.8 0.7 0.8 0.7 0.8

 Student sample size—PCR 617 355 262

 Student sample size—CR 593 337 256

 Student sample size—TRS 634 363 271

** Treatment group significantly different from control group at the .01 level.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

 SD: Standard deviation

Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall 

means. Statistical tests were conducted using regressions that included program indicators to account for the sample design and 

adjusted for clustering at the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 4. Love In a Big World 

270

Characteristics of Teachers and Schools 

Table 4.8 describes the third-, fourth-, and fifth-grade teachers at the study schools. They were predominantly 

White non-Hispanic (91%) and female (90%) and had an average of 14.9 years of total teaching experience. 

More than half (55%) held an advanced or specialist degree. There were no statistically significant differences 

between the treatment and control groups of teachers. 

Data regarding school characteristics were drawn from the Common Core of Data in order to compare 

treatment and control schools.49 There were no significant differences between the two groups of schools in 

terms of student composition (race/ethnicity and free lunch eligibility), number of students enrolled, number 

of full-time teachers, Title I status, or number of years the principal had been at the school (see table 4.9). In 

addition, there were no significant differences between treatment and control schools in terms of location 

(urban, suburban, or rural) or lowest and highest grade offered (these data are not shown in a table). 

Table 4.8. Initial characteristics of teachers in sample––LBW 

Characteristic Total Treatment Control

 Teacher sample size 129 68 61

Gender (percent)

 Male 10.0 8.6 11.4

 Female 90.0 91.4 88.6

Race/ethnicity (percent)

 White (non-Hispanic) 90.5 92.7 88.4

 Other 9.5 7.3 11.6

Number of years teaching experience (mean) 14.9 14.5 15.2

Number of years teaching experience in this school (mean) 10.0 8.7 11.2

Type of teaching certificate (percent)

 Regular state certificate or advanced professional certificate 95.5 ‡ ‡ 

 Other 4.5 ‡ ‡ 

Education (percent)

 Bachelor’s degree 44.7 40.0 49.4

 Advanced degree/other 55.3 60.0 50.6

‡ Reporting standards not met. Values suppressed to protect confidentiality.

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program

 

49 Common Core of Data information on student race/ethnicity, percentage eligible for free or reduced-price lunch, and 

number of full-time teachers were missing for LBW treatment and control schools. 
Chapter 4. Love In a Big World 

271

Table 4.9. Initial characteristics of schools in sample—LBW 

Characteristic Total Treatment Control

 School sample size 12 6 6

Student race/ethnicity (percent)

 White (non-Hispanic) –– –– ––

 Black (non-Hispanic) –– –– ––

 Hispanic –– –– ––

 Other –– –– ––

Students eligible for free or reduced-price lunch (percent)

Number of students enrolled (mean) 536.2 576.5 495.8

Number of full-time teachers (mean) –– –– ––

Title I status (percent)

 Title I eligible school 66.7 66.7 66.7

 Schoolwide Title I ‡ 0 ‡

Number of years principal has been at this school (mean) 6.7 4.8 8.5

–– Not available.

‡ Reporting standards not met. Values suppressed to protect confidentiality.

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: NCES Common Core of Data (2003-04), the Social and Character Development (SACD) Research Program.

In the Teacher Report on Classroom and School, teachers reported on nine dimensions of school 

environment (these data are not shown in a table): feelings of safety, adequacy of resources, student support, 

freedom to teach as desired, affiliation with and ties to colleagues, innovation regarding new approaches to 

teaching, professional interest, participatory decisionmaking, and work pressure. There were no statistically 

significant differences in these reports between treatment and control schools. 

The Level of SACD in the Schools Near the Beginning of the Study 

During the initial data collection, principals and teachers reported on the SACD activities used in the schools 

and classrooms, the availability of SACD materials, and the professional development provided on SACD. 

Table 4.10 shows that the majority of the school principals reported activities to promote six social and 

character development goals: violence prevention and peace promotion (92%), social and emotional 

development (92%), character education (92%), tolerance and diversity (83%), risk prevention and health 

promotion (82%), and civic responsibility and community service (91%). In addition, all of the principals 

reported activities directed toward behavior management. There were no statistically significant differences 

between treatment and control groups in the percentages, although this may be due to the relatively small 

principal sample size. The percentages of teachers reporting the use of these activities in their classrooms 

ranged from 46 percent to 89 percent, and there were no significant differences between treatment and 

control teacher reports. With respect to the use of schoolwide activities, 42 percent to 83 percent of teachers 

reported that their schools used such activities, and there were no significant differences between treatment 

and control teacher reports. 
Chapter 4. Love In a Big World 

272

Table 4.10. Principal and teacher initial reports on use of SACD programs or activities in 

sample––LBW 

SACD program or activity Total Treatment Control

 Principal sample size 12 6 6

 Teacher sample size 129 68 61

Principals reporting that school had programs or activities

 to promote the following SACD goals (percent)

 Violence prevention and peace promotion 91.7 83.3 100.0

 Social and emotional development 91.7 83.3 100.0

 Character education 91.7 83.3 100.0

 Tolerance and diversity 83.3 66.7 100.0

 Risk prevention and health promotion 81.8 83.3 80.0

 Civic responsibility and community service 90.9 83.3 100.0

 Behavior management 100.0 100.0 100.0

 None of the above 0.0 0.0 0.0

Teachers reporting on using programs or activities in their

 class to promote the following SACD goals (percent)

 Violence prevention and peace promotion 58.2 65.3 51.1

 Social and emotional development 55.2 59.6 50.7

 Character education 68.9 70.3 67.5

 Tolerance and diversity 54.9 58.8 51.0

 Risk prevention and health promotion 46.2 46.9 45.5

 Civic responsibility and community service 55.4 62.2 48.6

 Behavior management 88.8 92.7 84.9

 None of the above 4.1 ‡ ‡

Teachers reporting schoolwide use of the 

 following activities to promote SACD (percent)

 Morning announcements or videos 83.1 80.0 86.1

 School assemblies 61.3 70.5 52.1

 School newspapers or bulletins 62.1 61.1 63.2

 Special school days 42.0 45.2 38.9

 Special events 63.9 71.5 56.3

 Other activities 9.6 10.3 8.8

‡ Reporting standards not met. Values suppressed to protect confidentiality.

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within each of the programs and to each program across programs, were used in producing the 

treatment, control, and overall means. Statistical tests were conducted using regressions that included program indicators to 

account for the sample design and adjusted for clustering at the school level. Sample size may differ for some outcomes due to 

nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 4. Love In a Big World 

273

Teachers reported using a broad range of teaching materials to support SACD activities (table 4.11), including 

teacher guides (46%), student materials (31%), instructional aids (29%), giveaways (32%), and children’s 

literature (41%). There was a statistically significant difference between the groups, with a greater percentage 

of treatment teachers reporting the use of children’s literature (one out of seven materials asked about) than 

control teachers (58% versus 24%). 

Teachers also reported using a wide variety of teaching strategies (table 4.11). Nearly all teachers (99.5%) 

reported using any of the 20 strategies asked about, and teachers used an average of 11.2 strategies. There 

were no significant differences in the average number of strategies, or in the specific strategies, used by 

treatment versus control teachers. 
Chapter 4. Love In a Big World 

274

Table 4.11. Teacher initial reports on use of SACD materials and classroom strategies in 

sample—LBW 

SACD material and classroom /strategy Total Treatment Control

 Teacher sample size 129 68 61

Teachers using the following materials in conjunction with

 social and character development activities (percent)

 Teacher guides (manuals, curricula) 46.4 47.4 45.3

 Student materials (workbooks, worksheets) 30.8 31.9 29.8

 Instructional aids (games, software, videos) 29.0 22.9 35.1

 Giveaways (bookmarks, stickers) 32.0 30.5 33.6

 Children’s literature 41.1 58.1** 24.0

 Other types of materials 11.1 8.1 14.1

 Do not use any of the materials listed above 22.3 22.7 21.9

Teachers using any of the strategies listed below to promote

 social and character development in the classroom (percent) 99.5 99.0 100.0

Number of strategies (listed below) used by teachers to promote

 social and character development in the classroom (mean) 11.2 11.2 11.1

Teachers using each of the following strategies to promote

 social and character development (percent)

 Role-playing 65.6 64.1 67.2

 Cooperative learning 94.7 93.7 95.8

 Peer group discussions 88.0 86.8 89.3

 Direct instruction of social and character development 66.8 70.5 63.0

 Skill training 29.4 32.0 26.7

 Incorporating social and character development into 

 academic curriculum 62.6 66.7 58.6

 Parent training 2.9 ‡ ‡

 Parent/community involvement in program development

 or delivery 14.3 15.9 12.7

 Mentoring 46.8 44.0 49.5

 Good behavior notes sent home daily or weekly 80.2 87.3 73.1

 Presenting role models 60.7 70.2 51.2

 Targeted story reading or writing on SACD themes 63.9 65.8 62.1

 Peer mediation 36.1 35.1 37.1

 Honor roll for positive behavior 43.4 37.1 49.8

 Pledges or recitations on social and character 

 development themes 35.3 39.3 31.3

 Guided visualization 31.1 19.1 43.2

 Student-led/student-assisted instruction 45.8 41.6 50.0

 Journaling 81.5 78.7 84.2

 Time out for negative behavior 95.1 92.1 98.1

 Daily or weekly rewards for positive behavior 89.5 92.0 87.1

‡ Values suppressed to protect confidentiality.

** Treatment group significantly different from control group at the .01 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 4. Love In a Big World 

275

Principals and teachers reported on participation in and amount of SACD training and staff development 

provided over the previous 12 months (table 4.12). Principals reported higher participation rates than did 

teachers (91% versus 73%), although principals and teachers reported virtually the same number of training 

hours (5.8 versus 5.6). There was a significant difference in the percentages of teachers reporting participation 

in SACD training, with more treatment teachers reporting participation than control teachers (84% versus 

62%). In addition, a significantly greater percentage of treatment teachers than control teachers reported 

receiving specific training in one of the seven targeted SACD goals (character education: 48% versus 18%). 

Table 4.12. Principal and teacher initial reports on SACD professional development in sample––

LBW 

SACD professional development Total Treatment Control

 Principal sample size 12 6 6

 Teacher sample size 129 68 61

Principals reporting that staff participated in social and character

 development training within the past year (percent) 90.9 83.3 100.0

Teachers reporting participation in social and character

 development training within the past 12 months (percent) 72.9 83.6* 62.2

Number of hours of social and character development training

 principals report were provided to each staff person last year (mean) 5.8 6.2 5.4

Number of hours of social and character development training

 teachers report receiving during the past 12 months (mean) 5.6 6.4 4.8

Teachers reporting receiving training in the past 12 months

 in the following areas (percent)

 Violence prevention and peace promotion 21.3 26.8 15.8

 Social and emotional development 22.3 21.3 23.3

 Character education 33.1 47.9** 18.3

 Tolerance and diversity 24.9 29.4 20.3

 Risk prevention and health promotion 18.5 20.0 17.0

 Civic responsibility and community service 7.3 5.9 8.6

 Behavior management 28.0 33.9 22.0

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.

The data on the initial level of SACD activity emphasized that the control condition was a ―standard practice‖ 

control. Standard practice at the control schools included using SACD activities, materials, and practices, 

along with professional development for staff, at rates and in types and amounts similar to the treatment 

schools. For example, the percentage of teachers who reported using programs or activities to promote 

specific SACD goals ranged from 47 percent to 93 percent in the treatment schools and from 46 percent to 

85 percent in the control schools. The 3 significant differences between the treatment and control conditions 

in the use of SACD activities equaled the number that would be expected by chance (3 out of 62 

comparisons). 
Chapter 4. Love In a Big World 

276

Impacts on Use of SACD Activities 

The introduction of the formal LBW program would be expected to increase the use of SACD activities in 

the treatment schools in comparison to the control schools. The analysis of this impact is based on the 

Teacher Report on Classroom and School (TRCS). Every spring, third-, fourth-, and fifth-grade teachers 

provided information through the TRCS about the social and character development activities they used in 

their classrooms. Specifically, information from the TRCS was used to determine the difference between 

treatment and control teachers in these areas: 

1. the use of SACD activities in their classrooms overall and by SACD goal; 

2. the use of materials and strategies to implement the SACD activities within classrooms and within 

the entire school; 

3. the use of staff development to support the teachers; and 

4. teacher support for SACD efforts in the school and the use of practices conducive to the social and 

character development of students. 

TRCS consent and completion rates (table 4.4) led to 84 percent to 97 percent of all teachers having data for 

the 3 years, with greater percentages of control teachers providing data in Years 1 and 3. To estimate 

intervention impacts for each of the outcomes, testing of the statistical significance of the differences in 

means was used. Preliminary analysis indicated little or no gains in precision from using covariates. Before the 

mean differences were tested, the data were weighted such that each school received equal weight. Standard 

errors of the impact estimates accounted for the clustering of teachers within schools. In addition, a set of 

heuristics (described in chapter 1) was applied to determine whether each outcome domain was statistically 

significant after adjustments were made for the multiple tests conducted. 

Use of Activities 

The percentages of control teachers who reported using any SACD activities in their classrooms ranged from 

82 percent to 87 percent over the 3 years (table 4.13, panel 1). For the six individual SACD goals, the ranges 

varied from 53 percent to 77 percent in Year 1, 52 percent to 70 percent in Year 2, and 64 percent to 81 

percent in Year 3. Control teachers’ use of behavior management activities ranged from 75 percent to 84 

percent over this period. The percentages of control teachers who reported using any SACD activities in their 

classrooms for at least 1 hour per week (panel 2) ranged from 48 percent to 80 percent over the 3 years. For 

the six individual SACD goals, the percentages ranged from 6 percent to 29 percent in Year 1, 14 percent to 

25 percent in Year 2, and 18 percent to 28 percent in Year 3. Control teachers’ use of behavior management 

activities ranged from 43 percent to 57 percent over this period. These findings show that the control schools 

were using these activities as part of their standard practice related to social and character development. 

For teachers’ reported use of any SACD activity, 48 comparisons were made, with 3 expected to be 

significant by chance (panels 1 and 2). The percentage of treatment teachers using any SACD activity was 

significantly different from control teachers in Year 2 (impact = 17 percentage points). For specific SACD 

activities, there was a significant difference for character education in Year 2 (impact = 24 percentage points) 

and for behavior management activities in Year 3 (impact = 19 percentage points), both favoring treatment 

teachers. Significant differences between treatment and control teachers who reported use of these activities 

for at least 1 hour a week occurred in all years, again favoring treatment teachers. In Year 1, there was an 

impact on social and emotional development (impact = 14 percentage points), character education (impact = 

45 percentage points), tolerance and diversity (impact = 16 percentage points), and any activity (impact = 32 

percentage points). In Year 2, impacts were seen for violence prevention and peace promotion (impact = 18 

percentage points), social and emotional development (impact = 23 percentage points), character education 

(impact = 43 percentage points), risk prevention and health promotion (impact = 16 percentage points), any 

activity (impact = 29 percentage points), and behavior management (impact = 33 percentage points). In Year 
Chapter 4. Love In a Big World 

277

3, there was an impact on behavior management (impact = 26 percentage points). After the heuristics were 

applied to adjust for multiple comparisons, the data showed that LBW had statistically significant impacts in 

Years 1 and 2 on the domain for engagement in SACD activities. 

Regarding the use of named SACD activities, 42 comparisons were made, with 2 expected to be significant by 

chance (panels 3 and 4). Five of the 12 impact estimates in Year 1, 6 of the 12 in Year 2, and 3 of the 12 in 

Year 3 were statistically significant. An impact on character education activities (impact = 44, 59, and 31 

percentage points) and any named activity (impact = 29, 45, and 20 percentage points) were seen in all 3 

years. In Year 2, there was an impact on risk prevention and health promotion (impact = 26 percentage 

points). For named activities used at least 1 hour per week, there were significant impacts on character 

education (impact = 52 percentage points), tolerance and diversity (impact = 21 percentage points), and any 

named activity (impact = 37 percentage points) in Year 1; character education (impact = 43 percentage 

points) and any named activity (impact = 38 percentage points) in Year 2; and character education (impact = 

21 percentage points) in Year 3. LBW had significant impacts on the domain for engagement in named 

SACD activities in Years 1 and 2. 
Table 4.13. Impacts on teacher-reported SACD classroom activities––LBW 

Panel 1: Engagement in any activities to promote SACD goals1

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 85 64 69 47 75 50

Violence prevention and peace promotion

 (percent) 65.6 61.2 4.4 0.698 58.8 51.5 7.3 0.337 72.0 71.9 0.1 0.989

Social and emotional development

 (percent) 66.1 55.5 10.6 0.309 66.6 52.2 14.4 0.102 64.4 74.4 -10.0 0.399

Character education (percent) 92.5 77.0 15.4 0.106 93.8* 69.6 24.3 0.003 84.9 81.4 3.6 0.575

Tolerance and diversity (percent) 66.4 62.7 3.7 0.704 61.6 59.9 1.7 0.862 61.5 70.6 -9.2 0.457

Risk prevention and health promotion

 (percent) 63.3 53.2 10.1 0.119 69.6 52.6 17.0 0.114 69.1 64.2 4.9 0.692

Civic responsibility and community service

 (percent) 55.6 53.1 2.5 0.848 59.3 56.0 3.4 0.791 59.6 67.5 -7.9 0.469

Any SACD goal (percent) 93.5 83.2 10.3 0.190 98.3* 81.6 16.7 0.024 91.4 86.9 4.5 0.457

Behavior management (percent) 85.6 84.0 1.7 0.780 90.9 80.2 10.7 0.211 93.6* 74.7 18.9 0.008

See notes at end of table.

278

Chapter 4. Love In a Big World
Table 4.13. Impacts on teacher-reported SACD classroom activities––LBW—Continued 

Panel 2: Engagement in any activities to promote SACD goals for at least 1 hour per week

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 85 64 69 47 75 50

Violence prevention and peace promotion

 (percent) 26.9 19.9 7.0 0.347 31.9* 14.0 17.8 0.034 25.2 17.8 7.4 0.497

Social and emotional development

 (percent) 32.1* 18.5 13.6 0.016 40.8* 17.5 23.3 0.008 21.3 26.5 -5.2 0.565

Character education (percent) 74.1* 28.7 45.4 0.000 64.2* 21.6 42.5 0.000 37.4^ 21.5 15.8 0.061

Tolerance and diversity (percent) 32.0* 16.1 15.9 0.020 24.9 25.4 -0.5 0.944 16.9 23.2 -6.3 0.512

Risk prevention and health promotion

 (percent) 26.1 16.2 9.9 0.077 29.9* 14.0 15.9 0.046 29.6 28.1 1.5 0.920

Civic responsibility and community service

 (percent) 9.4 5.7 3.7 0.503 25.2 16.8 8.4 0.419 11.4 17.9 -6.6 0.374

Any SACD goal (percent) 79.9* 47.8 32.1 0.001 91.7** 62.8 28.9 0.008 84.0 79.9 4.1 0.609

Behavior management (percent) 58.2 55.4 2.9 0.778 75.5** 43.0 32.5 0.004 82.2* 56.7 25.6 0.025

See notes at end of table.

279

Chapter 4. Love In a Big World
Table 4.13. Impacts on teacher-reported SACD classroom activities––LBW—Continued

Panel 3: Engagement in activities to promote SACD goals linked to named SACD programs2

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 85 64 69 47 75 50

Violence prevention and peace promotion

 (percent) 21.9 22.4 -0.4 0.964 29.1 17.3 11.7 0.281 25.9 13.0 12.9 0.191

Social and emotional development

 (percent) 25.6 10.0 15.6 0.311 31.6 14.4 17.2 0.172 10.4 15.2 -4.8 0.605

Character education (percent) 62.9* 18.5 44.4 0.003 75.4* 16.3 59.2 0.000 39.2* 8.0 31.2 0.047

Tolerance and diversity (percent) 25.4 12.0 13.4 0.241 24.6^ 9.8 14.8 0.060 ‡ ‡ 7.1 0.361

Risk prevention and health promotion

 (percent) 38.5^ 25.8 12.7 0.075 52.3* 25.9 26.4 0.009 37.9 31.2 6.7 0.587

Civic responsibility and community service

 (percent) ‡ ‡ 1.0 † ‡ ‡ 6.4 0.288 5.9 12.3 -6.4 0.401

Any named activity (percent) 69.1* 39.7 29.4 0.002 86.6* 42.1 44.6 0.001 63.0* 43.4 19.6 0.021

See notes at end of table.

280

Chapter 4. Love In a Big World
Table 4.13. Impacts on teacher-reported SACD classroom activities––LBW—Continued

Panel 4: Engagement in activities to promote SACD goals linked to named SACD programs for at least 1 hour per week

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 85 64 69 47 75 50

Violence prevention and peace promotion

 (percent) 17.2^ 8.4 8.8 0.094 ‡* ‡ 18.3 0.016 15.1 5.1 9.9 0.272

Social and emotional development

 (percent) ‡^ ‡ 19.6 0.055 23.9 9.2 14.6 0.107 7.8 5.2 2.6 0.701

Character education (percent) 59.1* 6.9 52.3 0.000 53.0* 8.2 44.8 0.000 ‡* ‡ 21.3 0.014

Tolerance and diversity (percent) 22.6* 1.7 20.9 0.016 14.9 8.8 6.1 0.337 6.9 0.0 6.9 †

Risk prevention and health promotion

 (percent) 23.3^ 15.0 8.3 0.093 22.8 12.6 10.2 0.198 19.6 19.3 0.3 0.975

Civic responsibility and community service

 (percent) ‡ ‡ 1.0 † 4.8 0.0 4.8 † ‡ ‡ -0.2 0.909

Any named activity (percent) 59.7* 23.0 36.7 0.000 60.3* 22.1 38.2 0.000 38.1 20.7 17.4 0.131

† Not applicable.

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

** Treatment group significantly different from control group at the .01 level.

^ Treatment group significantly different than control group at the .10 to > .05 level.

1 In Year 1, the omnibus impact for all the outcomes measured together was positive and statistically significant on the basis of a multivariate statistical test. In Years 1 and 2, at least 

one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) procedure to adjust 

significance levels downward to account for the multiple testing of impacts.

2 In Years 1 and 2, at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg 

(1995) procedure to adjust significance levels downward to account for the multiple testing of impacts.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

281

Chapter 4. Love In a Big World
Chapter 4. Love In a Big World 

282

Use of Materials and Strategies 

For use of materials and strategies to support SACD goals, 87 comparisons were made, with 5 expected to be 

significant by chance. Fifteen significant impacts were found on treatment teachers’ use of materials and 

strategies in all 3 years (table 4.14). In Year 1, more treatment teachers than control teachers used teacher 

guides (impact = 43 percentage points), children’s literature (impact = 38 percentage points), direct 

instruction of SACD (impact = 27 percentage points), skill training (impact = 21 percentage points), good 

behavior notes (impact = 24 percentage points), and targeted story reading and writing on SACD themes 

(impact = 29 percentage points). On average, treatment teachers also used more strategies than control 

teachers (by two strategies). In Year 2, more treatment teachers than control teachers continued to use 

children’s literature (impact = 24 percentage points), direct instruction of SACD (impact = 25 percentage 

points), and skill training (impact = 22 percentage points), and fewer treatment teachers reported not using 

any of the SACD materials (impact = -16 percentage points). In addition, more treatment teachers presented 

role models (impact = 22 percentage points). In Year 3, more treatment teachers used teacher guides (impact 

= 22 percentage points), instructional aids (impact = 17 percentage points), and direct instruction of SACD 

(impact = 17 percentage points). LBW’s impact on the domain of materials and strategies was statistically 

significant in Year 1. 
Table 4.14. Impacts on use of SACD classroom materials and teaching strategies—LBW 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD material and teaching strategy

1

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 85 64 69 47 75 50

Use of SACD materials (percent)

 Teacher guides (manuals, curricula) 87.3* 44.0 43.3 0.001 73.5 59.9 13.6 0.170 67.9* 46.2 21.7 0.028

 Student materials (workbooks or sheets) 45.7 54.8 -9.2 0.337 46.5 38.3 8.2 0.425 46.7 34.7 12.0 0.142

 Instructional aids (games, software, videos) 43.4 40.5 3.0 0.803 40.3 44.1 -3.8 0.791 45.3* 28.5 16.7 0.014

 Giveaways (bookmarks, stickers) 58.2 48.3 9.9 0.402 50.2 35.0 15.2 0.232 39.7 43.2 -3.5 0.750

 Children’s literature 75.0* 36.9 38.2 0.006 60.0* 35.7 24.2 0.012 51.3 51.8 -0.5 0.953

 Other types of materials 10.1 5.0 5.1 0.275 10.2 8.5 1.7 0.726 7.7 6.9 0.8 0.875

 Did not use any of these materials 4.4 20.7 -16.2 0.132 6.9* 22.4 -15.5 0.035 16.8 14.7 2.1 0.776

Use of teaching strategies (percent)

 Role-playing 83.1 72.9 10.2 0.215 70.9 50.2 20.7 0.121 74.6 86.3 -11.6 0.350

 Cooperative learning 94.3 92.7 1.5 0.669 92.8 90.7 2.1 0.680 98.4 95.8 2.7 0.405

 Peer group discussions 95.5 92.7 2.8 0.425 92.1 93.8 -1.8 0.796 98.5 97.1 1.3 0.658

 Direct instruction of SACD 98.1* 71.3 26.8 0.003 94.3* 69.1 25.2 0.008 95.3* 78.7 16.6 0.014

 Skill training 57.8* 36.4 21.4 0.002 57.1* 35.3 21.8 0.047 75.2 61.6 13.6 0.305

 Incorporating SACD into academic 

 curriculum 77.7^ 60.5 17.2 0.085 83.1 71.0 12.0 0.173 85.2 86.8 -1.5 0.816

 Parent training 3.9 7.2 -3.3 0.458 5.1 12.0 -6.9 0.224 22.2 22.3 0.0 1.000

 Parent/community involvement 22.1 25.1 -3.1 0.699 18.8 27.8 -8.9 0.463 37.9 43.5 -5.6 0.638

 Mentoring 65.0 65.6 -0.7 0.956 62.6 61.9 0.6 0.958 72.5 75.1 -2.6 0.834

 Good behavior notes sent home daily

 or weekly 91.0* 67.5 23.5 0.022 84.3 64.5 19.7 0.122 90.7 84.4 6.3 0.136

 Presenting role models 84.9 74.8 10.1 0.115 80.9* 59.4 21.5 0.012 82.3 83.6 -1.3 0.874

See notes at end of table.

283

Chapter 4. Love In a Big World
Table 4.14. Impacts on use of SACD classroom materials and teaching strategies—LBW—Continued

 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD material and teaching strategy

1

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

Use of teaching strategies (percent)––

 Continued

 Targeted story reading or writing on social

 and character development themes 98.0* 69.1 28.9 0.001 86.2 76.6 9.6 0.301 86.6 90.8 -4.2 0.494

 Peer mediation 63.9 47.6 16.2 0.165 51.9 49.5 2.3 0.839 62.8 78.3 -15.6 0.237

 Honor roll for positive behavior 57.8 57.3 0.5 0.955 56.1 59.1 -3.0 0.795 67.2 61.6 5.6 0.529

 Pledges or recitations on social and 

 character development themes 52.7 37.3 15.4 0.241 51.7 43.5 8.1 0.709 49.0 48.0 1.0 0.944

 Guided visualization 62.5 52.2 10.2 0.191 51.5 42.5 9.0 0.477 60.2 57.8 2.4 0.776

 Student-led/student-assisted 

 instruction 44.7 54.6 -9.9 0.269 36.2^ 59.3 -23.1 0.074 71.3 65.8 5.5 0.631

 Journaling 93.0^ 78.8 14.2 0.097 87.4 93.6 -6.2 0.323 90.3 87.5 2.8 0.723

 Time out for negative behavior 95.7^ 86.8 8.9 0.086 92.9 94.3 -1.5 0.759 98.4 89.9 8.5 0.102

 Daily or weekly rewards for positive

 behavior 91.0 78.0 13.0 0.164 96.1^ 83.0 13.2 0.098 95.7 94.4 1.3 0.793

 Any strategy 100.0 100.0 0.0 † 100.0 100.0 0.0 † 98.5 100.0 -1.5 †

 Number of strategies (mean) 14.2* 12.1 2.0 0.002 13.5 12.3 1.2 0.293 15.0 14.7 0.3 0.792

† Not applicable.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 In Year 1, at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) 

procedure to adjust significance levels downward to account for the multiple testing of impacts.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

284

Chapter 4. Love In a Big World
Chapter 4. Love In a Big World 

285

Regarding the use of schoolwide strategies, 18 comparisons were made between treatment and control 

teacher reports, with 1 expected to be significant by chance. There was 1 statistically significant difference 

between treatment and control teachers regarding use of schoolwide strategies in Year 3 (these data are not 

shown in a table), with treatment teachers reporting more use of school assemblies than control teachers 

(impact = 22 percentage points). The overall impact of LBW on the domain for use of schoolwide strategies 

in named SACD activities was significant in Year 2; the omnibus impact for all the outcomes measured 

together was positive and statistically significant on the basis of a multivariate statistical test. 

Participation in Professional Development 

Regarding reported participation in professional development, 27 comparisons were made over 3 years, with 

2 expected to be significant by chance. LBW had a statistically significant effect on treatment teachers’ 

participation in professional development, with more treatment teachers reporting SACD training in the past 

12 months in all years (impact = 29, 31, and 30 percentage points) and more hours of training in Years 1 and 

2 (by 5.4 and 3.3 hours, on average). In terms of specific SACD goals, more treatment teachers reported 

training in character education in all 3 years (impact = 62, 34, and 37 percentage points). LBW had a 

significant impact on the domain of professional development in Year 1. 
Table 4.15. Impacts on teacher-reported SACD professional development––LBW 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD professional development1

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 85 64 69 47 75 50

SACD training in past 12 months (percent) 89.5* 60.8 28.7 0.006 68.9* 37.7 31.2 0.027 73.4* 43.7 30.0 0.014

Hours of SACD training (mean) 8.6* 3.1 5.4 0.000 5.3* 2.1 3.3 0.044 5.6 3.8 1.8 0.292

Training by goal (percent)

 Violence prevention and peace promotion 21.9 17.7 4.2 0.707 17.0 8.1 8.9 0.188 26.2 18.3 7.9 0.498

 Social and emotional development 21.8 16.9 4.8 0.474 16.5 7.8 8.7 0.452 12.7 5.9 6.7 0.339

 Character education 77.5* 15.3 62.2 0.000 45.4* 11.3 34.0 0.037 46.7* 9.8 37.0 0.015

 Tolerance and diversity 31.7 15.0 16.8 0.119 11.3 7.1 4.3 0.566 14.1 17.5 -3.4 0.613

 Risk prevention and health promotion 18.3 12.5 5.8 0.483 22.1 14.8 7.3 0.516 17.5 12.0 5.6 0.339

 Civic responsibility and community service 8.2 5.7 2.5 0.642 ‡ ‡ 2.1 0.678 ‡ ‡ -1.9 0.623

 Behavior management 34.8 22.3 12.5 0.184 26.7 18.0 8.7 0.227 35.1 20.6 14.4 0.344

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

1 In Year 1, at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) 

procedure to adjust significance levels downward to account for the multiple testing of impacts.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means. 

SOURCE: The Social and Character Development (SACD) Research Program.

286

Chapter 4. Love In a Big World
Chapter 4. Love In a Big World 

287

Attitudes and Practices 

Teachers reported on their enthusiasm for SACD efforts in their schools (these data are not shown in a table) 

by indicating enthusiasm, cooperation, or open dislike. They also reported on the SACD practices of teachers 

and staff members in their schools (these data are not shown in a table). The practices included modeling 

positive character and behavior traits with students and fellow teachers, involving students in making 

decisions, giving students a voice in school governance, the school encouraging parent involvement in 

children’s social and character development, and using developmentally appropriate discipline strategies 

rather than punishment for misbehavior. Twenty-seven comparisons were made over 3 years, with 1 expected 

to be significant by chance. There were no statistically significant estimated impacts on teachers’ enthusiasm 

for SACD efforts in their schools in any of the years, nor were there any significant differences in teacher 

reports of the overall use of practices conducive to students’ social and character development. No significant 

impact was found on the domain in any of the 3 years. 

Year-by-Year Impacts on Students and Perceptions of School Climate 

The primary research question for the LBW evaluation was this: 

What is the average effect of LBW on children’s social and emotional competence, behavior, 

academics, and perceptions of school climate? 

The first approach to answering this question was to examine the year-by-year impacts of LBW on these 

student and school climate outcomes over the 3 years as the students progressed from third through fifth 

grades. 

Equation (2) (described in chapter 1) was estimated to provide LBW impacts on the 20 outcomes based on 

data from the 11 schools (6 treatment and 5 control). For the LBW evaluation, equation (2) excluded the 

program fixed effects (p) and included program-specific covariates and random school effects covariates. 

Table 4.16 lists the covariates used, with outcomes from each report in the LBW analysis. 
Chapter 4. Love In a Big World 

288

Table 4.16. Covariates used with outcomes from each report for analysis—LBW 

Potential covariate

CR 

outcome

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome

 Total number 6 24 21 5

Child-reported

 Female   

 Hispanic   

 Black (non-Hispanic)   

 Other ethnicity   

 Age in years   

 Scales

 Afraid at School

 Altruistic Behavior 

 Empathy  

 Engagement with Learning

 Negative School Orientation 

 Normative Beliefs About Aggression

 Sense of School as a Community

 Problem Behavior

 Self-Efficacy for Peer Interactions  

 Victimization at School 

Primary caregiver-reported

 Age in years 

 Completed high school or equivalent

 Some college

 Bachelor’s or higher degree

 Highest level of education in household

 Completed high school or equivalent 

 Some college 

 Bachelor’s or higher degree 

 Mother present in home life

 Mother and father present

 Respondent someone other than mother or father 

 Number of people in household 

 Household income: $20,000 to $40,000  

 Household income: $40,000 to $60,000  

 Household income: More than $60,000  

 Income-to-poverty-threshold ratio: Below 135 percent 

 Income-to-poverty-threshold ratio: 135 to 185 percent  

 Full-time employment  

 Part-time employment  

See notes at end of table.
Chapter 4. Love In a Big World 

289

Table 4.16. Covariates used with outcomes from each report for analysis—LBW—Continued

Potential covariate

CR 

outcome

PCR 

outcome

TRS 

outcome

TRCS 

outcome

 Parental scales

 APQ-Poor Monitoring and Supervision Subscale  

 APQ-Positive Parenting Subscale 

 Child-Centered Social Control

 Confusion, Hubbub, and Order 

 Community Resources 

 Community Risk 

 Parent and Teacher Involvement 

 Child scales

 Altruistic Behavior  

 Positive Social Behavior

 Problem Behavior

Teacher-reported

 Female 

 Hispanic 

 Black (non-Hispanic) 

 Other ethnicity 

 Total teaching experience

 Total experience in current school 

 Regular certificate

 Other certificate

 Highest degree–bachelor’s

 Child scales

 Academic Competence and Motivation

 ADHD-Related Behavior 

 Altruistic Behavior

 Positive Social Behavior 

 Problem Behavior

 Parent and Teacher Involvement

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hypertensive disorder

 APQ: Alabama Parenting Questionnaire 

 : Covariate used

 Blank cell: Covariate not used

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 4. Love In a Big World 

290

To assess the statistical power of the program-level impact estimates, minimum detectable impacts in effect 

size units (MDES) for each outcome measure were calculated for the LBW evaluation (table 4.17). MDES 

represent the smallest impacts in effect size (standard deviation) units that can be detected with a high 

probability (80%). The MDES are primarily a function of study sample sizes, the degrees of freedom available 

for statistical tests, and design effects due to clustering (Schochet 2005). For the LBW evaluation, the MDES 

range from 0.099 to 0.637 for the child-level outcomes based on the Child, Caregiver, and Teacher Report on 

Student and from 0.536 to 0.763 for the school climate outcomes based on the Teacher Report on Classroom 

and School. In general, the MDES for the school climate outcomes were larger than those for the child-level 

outcomes. 
Chapter 4. Love In a Big World 

291

Table 4.17. Adjusted minimum detectable effect sizes for impact evaluation––LBW 

Outcome measure–Report Year 1 Year 2 Year 3

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 0.104 0.107 0.126

 Normative Beliefs About Aggression–CR 0.104 0.178 0.154

 Empathy–CR 0.104 0.154 0.115

Behavior Domain

 Altruistic Behavior–CR 0.153 0.228 0.193

 Altruistic Behavior–PCR 0.116 0.155 0.148

 Altruistic Behavior–TRS 0.345 0.637 0.463

 Positive Social Behavior–PCR 0.184 0.210 0.157

 Positive Social Behavior–TRS 0.099 0.310 0.401

 Problem Behavior–CR 0.104 0.169 0.340

 Problem Behavior–PCR 0.152 0.196 0.135

 Problem Behavior–TRS 0.220 0.104 0.185

 ADHD-Related Behavior–TRS 0.227 0.104 0.295

Academics Domain

 Engagement with Learning–CR 0.104 0.107 0.164

 Academic Competence and Motivation–TRS 0.141 0.270 0.191

Perceptions of School Climate Domain

 Positive School Orientation–CR 0.231 0.202 0.212

 Negative School Orientation–CR 0.237 0.260 0.179

 Student Afraid at School–CR 0.223 0.322 0.283

 Victimization at School–CR 0.297 0.140 0.175

 Feelings of Safety–TRCS 0.763 0.611 0.552

 Student Support for Teachers–TRCS 0.536 0.631 0.726

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The minimum detectable effect (MDE) formula used in the calculations is as follows:

where sT and sC are the number of treatment and comparison schools; nT and nC are the average number of students per 

classroom; ρ1 is the intraclass correlation (ICC) at the school level; and factor(df) is a constant that depends on the number of 

degrees of freedom (df) available for analysis (and is 2.802 for the pooled analysis).

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 4. Love In a Big World 

292

Table 4.18 provides the estimates of LBW’s impacts on each of the 20 outcomes over each of the 3 years (60 

impacts in total, with 3 expected to be statistically significant by chance). Of the 60 results, 5 were statistically 

significant (2 beneficial, 3 detrimental). In Year 1, LBW had beneficial impacts on Altruistic Behavior 

(Primary Caregiver Report, effect size [ES] = 0.31) and Student Support for Teachers (Teacher Report on 

Classroom and School, ES = 0.52). In Year 3, LBW had detrimental impacts on Engagement with Learning 

(Child Report, ES = -0.35), Positive School Orientation (Child Report, ES = -0.33), and Feelings of Safety 

(Teacher Report on Classroom and School, ES = -0.70). There were also four substantively important but 

nonsignificant impacts (one beneficial, three detrimental): In Year 2, there was a detrimental impact on 

Altruistic Behavior (Teacher Report on Student, ES = -0.34) and a beneficial impact on Student Support for 

Teachers (Teacher Report on Classroom and School, ES = 0.28); in Year 3, there were detrimental impacts 

on Problem Behavior (Child Report, ES = 0.31) and Student Support for Teachers (Teacher Report on 

Classroom and School, ES = -0.26). Application of the heuristic to adjust for multiple comparisons within 

each outcome domain indicates that LBW had a significant beneficial impact on the domain of Behavior in 

Year 1 and a significant detrimental impact on the domain of Academics in Year 3. 
Table 4.18. Impacts on child and school outcomes—LBW 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 3.03 3.04 -0.01 0.894 3.19 3.21 -0.04 0.674 3.25 3.24 0.02 0.810

 Normative Beliefs About Aggression–CR (-) 1.24 1.23 0.02 0.809 1.26 1.21 0.12 0.368 1.35 1.28 0.15 0.244

 Empathy–CR (+) 2.37 2.38 -0.02 0.793 2.30 2.30 0.00 0.963 2.21 2.26 -0.13 0.176

Behavior Domain1

 Altruistic Behavior–CR (+) 1.19 1.31 -0.15 0.147 1.02 1.16 -0.19 0.191 0.99 1.13 -0.20 0.121

 Altruistic Behavior–PCR (+) 2.33* 2.13 0.31 0.005 2.22 2.11 0.17 0.152 2.22 2.23 0.00 0.985

 Altruistic Behavior–TRS (+) 1.43 1.41 0.04 0.844 1.29 1.46 -0.34° 0.270 1.33 1.31 0.03 0.914

 Positive Social Behavior–PCR (+) 3.06 3.09 -0.07 0.471 3.12 3.04 0.13 0.250 3.10 3.05 0.09 0.397

 Positive Social Behavior–TRS (+) 3.17 3.21 -0.06 0.343 2.94 3.11 -0.24 0.183 3.27 3.24 0.06 0.786

 Problem Behavior–CR (-) 0.22 0.21 0.05 0.541 0.25 0.23 0.04 0.728 0.41 0.29 0.31° 0.223

 Problem Behavior–PCR (-) 1.58 1.56 0.06 0.484 1.56 1.56 -0.01 0.898 1.52 1.57 -0.13 0.197

 Problem Behavior–TRS (-) 1.43 1.40 0.08 0.488 1.46 1.40 0.14 0.117 1.32 1.38 -0.13 0.263

 ADHD-Related Behavior–TRS (-) 1.76 1.69 0.12 0.263 1.73 1.64 0.13 0.105 1.56 1.65 -0.16 0.344

Academics Domain2

 Engagement with Learning–CR (+) 3.66 3.66 0.01 0.942 3.67 3.69 -0.04 0.652 3.54* 3.68 -0.35 0.030

 Academic Competence and Motivation–

 TRS (+) 3.06 3.09 -0.03 0.666 3.05 2.99 0.06 0.622 3.15 3.17 -0.02 0.871

See notes at end of table.

293

Chapter 4. Love In a Big World
Table 4.18. Impacts on child and school outcomes—LBW—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 2.98 2.94 0.05 0.692 2.77 2.79 -0.03 0.794 2.62* 2.82 -0.33 0.047

 Negative School Orientation–CR (-) 1.76 1.86 -0.17 0.201 1.87 1.89 -0.03 0.846 2.05 1.95 0.18 0.178

 Student Afraid at School–CR (-) 2.28 2.30 -0.02 0.859 2.28 2.22 0.06 0.777 2.26 2.16 0.11 0.508

 Victimization at School–CR (-) 0.73 0.79 -0.07 0.657 0.70 0.71 -0.01 0.929 0.70 0.70 0.00 0.993

 Feelings of Safety–TRCS (+) 3.54 3.59 -0.07 0.793 3.39 3.45 -0.06 0.825 3.32* 3.67 -0.70 0.046

 Student Support for Teachers–TRCS (+) 3.72* 3.37 0.52 0.022 3.57 3.40 0.28° 0.428 3.45 3.58 -0.26° 0.543

* Treatment group significantly different from control group at the .05 level.

o Substantive (but nonsignificant at .05 level) effect size of ≥ .25 or ≤ -.25.

1 In Year 1, at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) 

procedure to adjust significance levels downward to account for the multiple testing of impacts.

2 Based on univariate statistical tests, one of the two unadjusted impacts is statistically significantly negative, indicating a detrimental impact of the intervention on child outcomes in 

this domain in Year 3. 

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models where each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. See table 1.5 for 

information about the measures used to create the outcome variables. The effect size was calculated by dividing the estimated impact by the standard deviation of the outcome 

measure for the control group. The number of results found significant was no more than expected by chance.

SOURCE: The Social and Character Development (SACD) Research Program.

294

Chapter 4. Love In a Big World
Chapter 4. Love In a Big World 

295

Impacts on Child Outcomes Over Time 

LBW’s impacts on the child outcomes over time were estimated using growth curve models by examining 

treatment and control group differences in the trajectories of student outcomes during the follow-up period 

while accounting for clustering at the school level. The growth curve models are estimated using a three-level 

hierarchical linear model, where Level 1 corresponds to time, Level 2 to students, and Level 3 to schools 

(described in chapter 1). 

Table 4.19 provides the estimates of LBW impacts on the growth in student outcomes over the 3 years. The 

estimated impacts range in effect size units (absolute value) from 0.01 to 0.13. One of the 18 estimated LBW 

intervention impacts on the trajectories of child outcomes was statistically significant (no more than expected 

by chance); the program had a detrimental impact on Positive School Orientation (Child Report, ES = -0.13). 
Table 4.19. Impacts on growth of child outcomes—LBW 

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2Treatment Control

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 2.96 0.08 0.12 -0.04 -0.04 0.03 0.298

 Normative Beliefs About Aggression–CR (-) 1.17 0.06 0.04 0.01 0.03 0.03 0.620

 Empathy–CR (+) 2.48 -0.10 -0.10 -0.01 -0.02 0.02 0.693

Behavior Domain

 Altruistic Behavior–CR (+) 1.39 -0.20 -0.16 -0.04 -0.04 0.07 0.591

 Altruistic Behavior–PCR (+) 2.29 -0.08 -0.07 -0.01 -0.01 0.04 0.861

 Altruistic Behavior–TRS (+) 1.40 0.00 -0.01 0.01 0.02 0.07 0.914

 Positive Social Behavior–PCR (+) 3.07 0.04 0.02 0.03 0.04 0.04 0.486

 Positive Social Behavior–TRS (+) 3.12 0.02 -0.01 0.04 0.04 0.09 0.691

 Problem Behavior–CR (-) 0.14 0.06 0.08 -0.02 -0.04 0.04 0.706

 Problem Behavior–PCR (-) 1.59 -0.03 -0.02 -0.01 -0.01 0.02 0.813

 Problem Behavior–TRS (-) 1.36 0.02 0.04 -0.03 -0.06 0.02 0.242

 ADHD-Related Behavior–TRS (-) 1.71 -0.09 -0.01 -0.08 -0.11 0.05 0.144

Academics Domain

 Engagement with Learning–CR (+) 3.75 -0.08 -0.04 -0.04 -0.06 0.03 0.247

 Academic Competence and Motivation–TRS (+) 3.11 0.03 0.08 -0.05 -0.05 0.04 0.263

See notes at end of table.

296

Chapter 4. Love In a Big World
Table 4.19. Impacts on growth of child outcomes—LBW—Continued

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2Treatment Control

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 3.23 -0.31* -0.20 -0.11 -0.13 0.04 0.016

 Negative School Orientation–CR (-) 1.68 0.17^ 0.11 0.06 0.08 0.03 0.090

 Student Afraid at School–CR (-) 2.35 -0.03 -0.07 0.04 0.03 0.03 0.232

 Victimization at School–CR (-) 0.68 0.08^ -0.03 0.11 0.10 0.05 0.061

* Treatment group significantly different from control group at the .05 level.

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 Pertains to the estimated slope of the outcome for the treatment or control groups.

2 The average score at implementation is calculated across treatment and control groups, using regression models for adjustment on covariates. 

3 Estimated difference between the slope of the treatment and control groups.

4 Effect size: The slope of the treatment group minus the slope of the control group divided by the standard deviation of the outcome for the program’s control group (the standard 

deviation is calculated without accounting for school-level clustering or regression adjustments).

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using HLM 6.06. Sample weights were used in all analyses to give (1) 

each school equal weight in each program (within each time period) and (2) each time period equal weight within the analysis. See table 1.5 for information about the measures used 

to create the outcome variables.

SOURCE: The Social and Character Development (SACD) Research Program.

297

Chapter 4. Love In a Big World
Chapter 4. Love In a Big World 

298

Summary 

As part of the Social and Character Development (SACD) Research Program, researchers at the Tennessee 

site implemented and evaluated the LBW program. This program focused on character education by teaching 

children positive character traits and moral virtues and how to apply them in their day-to-day activities. 

Twelve public schools in two school districts in two separate counties in Tennessee were recruited by the 

Tennessee research team and randomly assigned to treatment and control conditions to determine the LBW 

impact on social and character development activities in the schools and on the child outcome domains of 

Social and Emotional Competence, Behavior, Academics, and Perceptions of School Climate. 

Analyses of the initial characteristics of the sample (students, caregivers, communities, teachers, and schools) 

indicated that randomization to treatment and control status produced groups that were relatively similar on 

the observed characteristics at the start of the study (1 statistically significant difference in children’s behavior 

was found out of 83 comparisons, fewer than the 4 that would be expected by chance). Documentation of 

the initial level of social and character development activities in the schools also revealed few differences (3, 

no more than expected by chance out of 62 comparisons) between the treatment and control teachers and 

classrooms. This is important for two reasons: one, it indicates that randomization created comparable groups 

near the start of the study, and two, it shows that treatment and control schools both had high levels of 

SACD activities near the start of the study, indicating that social and character development activities are part 

of the ―standard practice‖ of these schools in Tennessee. Standard practice at all of the schools included 

reports that 46 percent to 89 percent of teachers used SACD activities, 78 percent of teachers used specific 

materials in conjunction with these activities, 96 percent used at least one of the specified instructional 

strategies, and 73 percent participated in SACD training over the past 12 months. 

Analyses of LBW impacts on use of SACD activities in the schools revealed impacts on the use of such 

activities (27 out of 90) and related materials and strategies (14 out of 87) across the 3 years, and use of more 

professional development activities for treatment teachers than control teachers in all years (8 out of 27). 

These same measures in the control schools across the 3 years of the study confirmed that use of these 

activities in the control schools constituted their standard practice. 

Of the 20 child-level outcomes representing the four domains of Social and Emotional Competence, 

Behavior, Academics, and Perceptions of School Climate assessed in each of the 3 years of the study (a total 

of 60 results), 5 were statistically significant (2 beneficial and 3 detrimental). A growth curve analysis was used 

to analyze the change over time in these same outcomes between initial data collection and the final outcome 

data collection at the end of the study. One of the 18 child-level outcomes assessed showed a significant 

detrimental impact of the LBW program. 

The SACD evaluation did not find evidence to support the hypothesis that LBW had beneficial impacts on 

students’ social and character development. Such results could be caused by the inability of the program to 

cause such change, possibly because the theory of action for the program is incomplete or the activities to 

carry out that theory are not effective. 

However, these results may also be due to the inability of the evaluation to observe such a change due to the 

control condition, the level of nonparticipation, or the sample size. The control schools continued using their 

standard SACD activities, and these turned out to be high in quantity and broad in scope. While LBW had a 

significant positive impact on the amounts and types of SACD activities, the resulting difference in the 

amount of SACD activities between the treatment and control schools may not have been large enough to 

cause significant differences in the student outcomes. In addition, about one-third of the students in the 

sample universe did not take part because of nonconsent or noncompletion of the surveys. As a 

determination could not be made as to whether the students not taking part significantly differed from those 

who did take part, the evaluation’s results are valid only for the students who took part. If the students not 

taking part were different, and if they would have responded better to LBW than to the SACD activities 
Chapter 4. Love In a Big World 

299

occurring in the control schools, then the evaluation could have underestimated the program’s impact. Third, 

the sample size of 12 schools and the resulting higher MDES compared to those for the multiprogram 

evaluation may have reduced the likelihood of detecting statistically significant effects. However, it should be 

noted that 70 percent of the MDES for the 60 outcomes used in the year-by-year analysis were below 0.25 

(53% were below 0.20). In addition, none of the 60 outcomes were found to be substantively important. 
This page is intentionally blank. 
301

Chapter 5. Positive Action 

University of Illinois at Chicago/Oregon State University

(Illinois Site)

Intervention 

Researchers at the University of Illinois at Chicago/Oregon State University (Illinois site) evaluated the 

Positive Action (PA) program (Flay, Allred, and Ordway 2001). The PA program consists of a school and home 

curriculum and a school climate program for teaching children about the benefits of physical, intellectual, 

social, and emotional positive actions, and for creating a supportive learning environment at school and at 

home. The curriculum is designed to build children’s self-concept and self-management through identifying 

positive thoughts, feelings, and actions. Table 5.1 describes the PA program’s general characteristics (panel 1), 

the types of instruction and strategies used (panel 2), the professional development provided for those 

implementing the program (panel 3), and the social and character development activities (panel 4) and 

outcomes (panel 5) addressed by the program.

 Teachers conduct 15- to 20-minute lessons, 4 days per week, with students. Lessons focus on 

positive actions for a healthy body and mind, social/emotional actions for managing oneself 

responsibly, ways to get along with others, how to be honest with oneself and others, and setting 

goals for self-improvement. 

 Teachers engage in behavior management practices by recognizing and reinforcing positive behavior 

and by modeling and reinforcing social skills throughout the day. 

 The school climate program is designed to reinforce positive actions throughout the school, using 

activities such as peer tutoring and mentoring, assemblies, service projects, PA days, and visual 

artifacts that illustrate positive actions. 

 The parent and community program includes parent newsletters and parent night, which are 

designed to carry classroom concepts into homes and to help families learn and practice positive 

actions in their communities. 

Teachers and staff receive a 4-hour training on the methods of instruction, role modeling, use of positive 

actions, and use of behavior management strategies at the beginning of the school year and a 2-hour training 

(refresher for teachers who continue in the same schools, new for new teachers) at the beginning of each 

subsequent year. Principals and the PA coordinator for each school receive an additional 2 hours of training 

on the program adoption process in a multischool setting. School counselors receive a kit that includes the 

curriculum and suggestions for activities with children and family groups. Ongoing consultation is provided 

to school staff by the PA research team as needed, as often as once per month. 
Chapter 5. Positive Action 

302

Table 5.1. Positive Action

Panel 1: General characteristics

Target population 

 Universal

Program components

 Peer: In and out of class

 Parent: Training, contact, and involvement

 Classroom: Lessons and behavior management

 Schoolwide: Planned events, includes program coordinator and committee

 Community: Committees and groups

 Training: Pretraining and ongoing

Level of integration

 Add-on curriculum and schoolwide activities

Flexibility

 Manualized: Scripted curriculum and guidebook

 Adaptability: Program staff and principal adapt

See notes at end of table.

Panel 2: Description of instruction and strategies

Classroom

 Lessons

 Who delivers: Teacher

 Activities and tools: Direct instruction, story reading, writing, role-playing, discussion, singing, games,

 worksheets, puppets, plays, poems

 Content: Self-concept, physical health, intellectual growth, emotional and behavioral self-regulation, 

 getting along with others (social skills), self-honesty, self-improvement, goal setting, character traits

 Frequency: 15- to 20-minute lessons, 4 days per week

 Strategies

 Who delivers: Teacher

 Activities and tools: Teaches, practices, recognizes, and models positive actions with curriculum and 

 climate activities and materials

 Frequency: Daily

Supplement to classroom

 Parent newsletters, parent manual with PA activities, letters, strategies, and parent night

Schoolwide activities 

 Occasional assemblies, service projects, PA days and year-end event, principal climate program with 

 reinforcement recognition activities

See notes at end of table.
Chapter 5. Positive Action 

303

Table 5.1. Positive Action––Continued

Panel 3: Professional development

Pre-implementation

 Teachers

 Content: Training on concepts and delivery of curriculum, school climate activities to reinforce positive

 behaviors and parent involvement

 Duration: 1⁄2 day

 Other

 Content: Principal training and appointing committees on coordinating school climate activities with

 family groups

 Duration: 1⁄2 day

Ongoing consultation

 Teachers

 Content: Group session to provide technical assistance and share experiences and challenges; visits

 for technical support

 Duration: Up to 1⁄2 day once per year; up to 1⁄2 day visits to schools monthly

 Other

 Content: Meeting of principals and coordinators to provide technical assistance and share successes

 and challenges

 Duration: 1 session of 3 hours per year

See notes at end of table.

Panel 4: Activities for SACD goals

Violence prevention and peace promotion  Risk prevention and health promotion 

Social and emotional development  Civic responsibility and community service 

Character education  Behavior management 

Tolerance and diversity 

See notes at end of table.

Panel 5: SACD outcomes addressed 

Engagement with Learning  Empathy 

Academic Competence and Motivation  Positive School Orientation 

Altruistic Behavior  Negative School Orientation 

Positive Social Behavior  Student Afraid at School 

Problem Behavior  Victimization at School 

Self-Efficacy for Peer Interactions  Feelings of Safety 

Normative Beliefs About Aggression  Student Support for Teachers 

NOTE: Abbreviations are

 PA: Positive Action

 : Activity or outcome addressed

 Blank cell: Activity or outcome not addressed

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 5. Positive Action 

304

Sample and Random Assignment 

The Illinois research team recruited a total of 14 public elementary schools in a single large, urban school 

district in Illinois. The researchers deliberately selected a sample of high-risk schools from Chicago that had 

low achievement scores and families with low household incomes. The 14 schools were randomly assigned to 

treatment and control conditions prior to the fall 2004 data collection. A two-step process was used. First, a 

computer-generated pairwise matching algorithm developed by Mathematica Policy Research, Inc. (MPR) was 

used to identify the best pairwise matches across the 14 schools based on variables identified by the Illinois 

research team. The variables used in the pairwise matching for Illinois included these: (a) school schedule, 

(b) percentage of White students, (c) percentage of Black (non-Hispanic) students, (d) percentage of Hispanic 

students, (e) percentage of Asian students, (f) achievement scores, (g) percentage of attendance, 

(h) percentage of truancy, (i) percentage of poverty, (j) percentage of mobility, (k) total enrollment, 

(l) percentage of parent participation, (m) quality of teachers, and (n) rate of crime in the community. Second, 

using a random number generator, 1 school in each matched pair was assigned to either the intervention or 

control condition. Seven schools received the PA program and 7 schools acted as control schools and 

continued to implement the social and character development activities that constituted their standard 

practice. Assignment to treatment or control condition was at the school level and therefore limited the risk 

of contamination between treatment and control classrooms. 

The original student sample (the cohort of students in the third grade in the 14 schools in fall 2004) 

numbered 811 students (410 treatment and 401 control). Table 5.2 documents the change in the sample over 

the three spring follow-up data collection periods. Over time, new entrants to the cohort became a larger 

percentage of the sample, eventually making up 39 percent of the sample by the spring of Year 3. There were 

no statistically significant differences between the treatment and control groups in the number of new 

entrants. The percentage of the sample made up of the original cohort further declined as students left the 

schools. By Year 3, approximately 50 percent of the original sample had left. In Year 2 there was a statistically 

significant difference between treatment and control groups in ―leavers,‖ with fewer in the treatment group. 

This difference had disappeared by the end of Year 3. 
Table 5.2. Sample––PA 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Characteristic Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 School sample size 14 7 7 14 7 7 14 7 7 14 7 7

 Student sample size 811 410 401 812 416 396 764 425 339 655 327 328

Stayers † † † 737 377 360 560 302 258 403 201 202

New entrants † † † 75 39 36 204 123 81 252 126 126

 New entrants as a percent of 

 spring enrollment † † † 9.2 9.4 9.1 26.7 28.9 23.9 38.5 38.5 38.4

Total leavers (from original cohort) † † † 74 33 41 251 108 143 408 209 199

 Leavers as a percent of fall 

 2004 enrollment † † † 9.1 8.0 10.2 30.9 26.3** 35.7 50.3 51.0 49.6

Number of students per school

 (mean) 58 59 57 58 59 57 55 61 49 47 47 47

Range of number of students per 

 school 23-101 38-92 23-101 26-103 35-92 26-103 25-83 36-83 25-77 25-74 28-74 25-65

† Not applicable.

** Treatment group significantly different from control group at the .01 level.

SOURCE: The Social and Character Development (SACD) Research Program.

305

Chapter 5. Positive Action
Chapter 5. Positive Action 

306

Implementation 

Training 

Teachers, principals, and school staff members in the intervention schools received a half-day of program 

implementation training prior to the beginning of the first school year (table 5.1, panel 3). The Illinois 

research team provided an additional half-day of training to schools via professional development days in fall 

2004 and in spring 2005, which also allowed continuing teachers and staff time to share stories and concerns 

about program implementation. In addition, the principal and PA coordinator from each school attended an 

annual meeting in the winter of each year to learn more about implementing the schoolwide component of 

the program and to share experiences. 

Data Collection 

MPR’s subcontractor, Decision Information Resources, Inc. (DIR) collected the child, teacher, and school 

data at the Illinois site. Table 5.3 shows the school year milestones and dates of implementation for the 

Illinois site. Data were collected in the fall and spring of the first 2 years and the spring of Year 3. The fall 

2004 data collection began on October 12, 2004, and ended on October 28, 2004. The average time frame 

from the beginning of program implementation to the beginning of fall data collection was 2 weeks. As a 

result, initial data collection took place after implementation of the PA program began. Therefore, these data 

provide a measure of the students, teachers, and schools near the beginning of the school year, at a time when 

the PA program had been operating for a relatively short period of time. The spring data collection window 

was from May 16, 2005, to June 3, 2005. The PA program had been implemented for 33 weeks at the time of 

the spring data collection and for 29 weeks from the end of the fall data collection. Year 2 followed a similar 

pattern, with implementation occurring at the start of the school year, fall data collection occurring 5 weeks 

later, and spring data collection occurring 26 weeks after fall data collection (and 34 weeks after the start of 

implementation). In spring 2007, data collection occurred 35 weeks after the start of implementation. Data 

collection took from 3 to 5 weeks at each collection point. 
Chapter 5. Positive Action 

307

Table 5.3. Data collection dates––PA

Year 1 Year 1 Year 2 Year 2 Year 3

Data collection schedule (Fall 3rd grade) (Spring 3rd grade) (Fall 4th grade) (Spring 4th grade) (Spring 5th grade)

 School sample size 14 14 14 14 14

School year dates 

 First day of school 9/2/04 † 9/6/05 † 9/5/06

 Start of implementation 9/27/04 † First day † First day

 Last day of school † 6/17/05 † 6/16/06 6/15/07

Data collection 

 Start 10/12/04 5/16/05 10/11/05 5/1/06 5/7/07

 End 10/28/04 6/3/05 11/2/05 5/26/06 6/1/07

Calendar weeks from 

 program implementation 

 to start of fall 2004 

 data collection 2 † † † †

Calendar weeks from start 

 of school to start of fall 

 data collection 6 † 5 † †

Calendar weeks from end 

 of fall data collection to 

 start of spring data 

 collection † 29 † 26 †

Calendar weeks from 

 program implementation 

 to start of spring data 

 collection † 33 † 34 35

† Not applicable.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 5. Positive Action 

308

Consent Rates, Completion Rates, and Percentage of Sample With Data 

The actual number of student, primary caregiver, and teacher reports available for analysis was smaller than 

the number in the sample because consent and completion rates were below 100 percent. Primary caregivers 

had to provide consent before children could complete the Child Report, before their child’s teacher could 

complete the Teacher Report on Student, and before they themselves completed the Primary Caregiver 

Report. Teachers also had to provide consent before completing the Teacher Report on Classroom and 

School. Of those with consent, not all completed their respective reports. Table 5.4 shows the consent rates, 

completion rates, and percentages of sample with data for each of the four reports over the 3 years. For the 

Child Report and two teacher reports, completion rates ranged from 78 percent to 100 percent, with no 

statistically significant differences between treatment and control groups in consent rates for any of these 

reports. For the Teacher Report on Student, completion rates ranged from 92 percent to 100 percent, with a 

statistically significant difference between treatment and control groups at two time points: in the fall of 2004 

and the spring of 2005 the treatment group teachers completed fewer reports than the control teachers. For 

the Primary Caregiver Report, the completion rates dropped over time from 92 percent to 64 percent. There 

were no statistically significant differences between treatment and control groups for these reports. 

The percentages of the sample with Child Report data ranged from 66 percent to 78 percent over the 3 years. 

The percentages of students with information from the Teacher Report on Student ranged from 71 percent 

to 80 percent. The percentages of students with data from the Primary Caregiver Report ranged from 49 

percent to 73 percent and declined over time. The percentages of teachers with data from the Teacher Report 

on Classroom and School ranged from 78 percent to 91 percent. There were no statistically significant 

differences between treatment and control conditions in percentages of students with data for any of the four 

reports. 
Table 5.4. Consent rates, completion rates, and percentage of sample with data from each report—PA 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 Student sample size 811 410 401 812 416 396 764 425 339 655 327 328

Child Report (percent)

 Primary caregiver consent rate 78.5 79.0 78.1 76.7 76.4 77.0 72.4 71.1 74.0 79.2 80.1 78.4

 Student completion rate 94.7 93.5 95.8 92.8 93.7 91.8 92.0 94.0 89.6 96.5 97.7 95.3

 Students with data174.4 73.9 74.8 71.2 71.6 70.7 66.6 66.8 66.4 76.5 78.3 74.7

Primary Caregiver Report (percent)

 Primary caregiver consent rate 78.3 79.0 77.6 76.7 76.4 77.0 70.9 69.6 72.6 78.3 79.5 77.1

 Primary caregiver completion rate 92.3 92.0 92.6 76.7 77.0 76.4 73.6 72.6 74.8 64.5 65.0 64.0

 Primary caregivers with data172.3 72.7 71.8 58.9 58.9 58.8 52.2 50.6 54.3 50.5 51.7 49.4

Teacher Report on Student (percent)

 Primary caregiver consent rate278.5 79.0 78.1 76.7 76.4 77.0 72.4 71.1 74.0 79.2 80.1 78.4

 Teacher completion rate 95.0 92.3*** 97.8 97.4 95.9* 99.0 100.0 100.0 100.0 98.8 99.2 98.4

 Students with data174.6 72.9 76.3 74.8 73.3 76.3 72.4 71.1 74.0 78.3 79.5 77.1

Teacher Report on Classroom and School 

 (3rd- to 5th-grade teachers) (percent)

 Teacher consent rate 100.0 100.0 100.0 99.1 98.4 100.0 100.0 100.0 100.0 100.0 100.0 100.0

 Teacher completion rate 89.5 90.0 88.9 83.3 83.3 83.3 86.3 86.0 86.7 83.7 78.0 90.5

 Teachers with data189.5 90.0 88.9 82.6 82.0 83.3 86.3 86.0 86.7 83.7 78.0 90.5

* Treatment group significantly different from control group at the .05 level.

*** Treatment group significantly different from control group at the .001 level.

1 Calculated as consent rate x completion rate.

2 The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

309

Chapter 5. Positive Action
Chapter 5. Positive Action 

310

Responses from students in the original cohort (stayers) and new entrants in the PA sample were examined to 

investigate possible differences between the two groups in consent rates, completion rates, and the 

percentages of the sample with data that might affect outcome data (table 5.5). In Year 2, new entrants had 

significantly higher consent rates and percentages of sample with data than stayers on all three reports (by 12 

to 17 percentage points). In Year 3, new entrants had significantly higher completion rates (by 10 percentage 

points) and percentages of sample with data (by 12 percentage points) than stayers for the Primary Caregiver 

Report. 
Table 5.5. Consent rates, completion rates, and percentage of sample with data: Stayers versus new entrants––PA 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total Stayers Entrants Total Stayers Entrants Total Stayers Entrants

 Student sample size 812 737 75 764 560 204 655 403 252

Child Report (percent)

 Primary caregiver consent rate 76.7 77.3 70.7 72.4 68.9*** 81.9 79.2 77.7 81.7

 Student completion rate 92.8 92.5 96.2 92.0 90.2* 96.4 96.5 96.5 96.6

 Students with data171.2 71.5 68.0 66.6 62.1*** 78.9 76.5 74.9 79.0

Primary Caregiver Report (percent)

 Primary caregiver consent rate 76.7 77.3 70.7 70.9 67.0*** 81.9 78.3 76.2 81.7

 Primary caregiver completion rate 76.7 76.1 83.0 73.6 73.1 74.9 64.9 60.9* 70.9

 Primary caregivers with data158.9 58.9 58.7 52.2 48.9** 61.3 50.8 46.4** 57.9

Teacher Report on Student (percent)

 Primary caregiver consent rate276.7 77.3 70.7 72.4 68.9*** 81.9 79.2 77.7 81.7

 Teacher completion rate 97.4 97.7 94.3 100.0 100.0 100.0 98.8 98.7 99.0

 Students with data174.8 75.6 66.7 72.4 68.9*** 81.9 78.3 76.7 81.0

* Stayers significantly different from new entrants at the .05 level.

** Stayers significantly different from new entrants at the .01 level.

*** Stayers significantly different from new entrants at the .001 level.

1 Calculated as consent rate x completion rate.

2 The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

311

Chapter 5. Positive Action
Chapter 5. Positive Action 

312

Fidelity of Implementation

Each year, PA’s seven treatment schools were independently rated for quantity and quality of program 

implementation by two raters from the research team. The global measure of fidelity for the multisite study 

was used; inter-rater reliability was measured using Cronbach’s alpha (0.85 in Year 1, 0.74 in Year 2, and 0.98 

in Year 3). The ratings were combined into a single consensus rating and used to identify schools with high 

implementation fidelity. In Years 1 and 2, four treatment schools were identified as having high fidelity, and 

in Year 3, three treatment schools were identified as having high fidelity. Cohen’s kappa was used as the 

measure of agreement when identifying schools as high fidelity, and it equaled 0.72 in Year 1, 0.46 in Year 2, 

and 1.00 in Year 3. 

Initial Characteristics 

This section describes the initial characteristics of the students, teachers, and schools participating in the 

evaluation of the PA program. These characteristics were collected from students who were enrolled in the 

third grade at the study schools in fall 2004, as well as from their primary caregivers and third-grade teachers. 

In addition, third-, fourth-, and fifth-grade teachers and principals in the study schools provided information 

about activities related to social and character development in these schools. Documenting the characteristics 

of students, teachers, and schools and initial measures of key outcomes at a point before the interventions 

had been operating for an extended period helps to determine whether the random assignment of schools to 

treatment and control status produced treatment and control groups with similar distributions of observed 

characteristics. As noted in the following discussion, 3 significant differences (out of 62 comparisons, with 3 

expected to be significant by chance) were found between the observed characteristics of the treatment and 

control students, teachers, and schools; all 3 reflected the use of SACD activities in the classroom and school. 

Characteristics of Children, Their Families, and Communities 

There were no significant differences between the treatment and control groups in the observed student, 

caregiver, and community characteristics (table 5.6). For students, the mean age was 8.3 years. The sample 

contained roughly equal percentages of girls (54%) and boys (46%). The sample was ethnically diverse, with 

Black non-Hispanic students making up the majority of the sample (51%). White non-Hispanic students 

made up 6 percent of the sample and Hispanic students made up 37 percent of the sample. 

The sample was diverse in its levels of family income, education levels of primary caregivers of the children in 

the sample, and family situation. For the total sample, 66 percent of the children lived in a household where 

the income was 135 percent of the federal poverty level or lower, which is the income threshold for eligibility 

for free school meals. More than one-quarter (27%) of primary caregivers had not completed high school. 

Fewer than half of the children (47%) lived with both their mother and their father. There were no significant 

differences between the treatment and control groups in these characteristics. 

The mean values of the outcomes for children’s behavior and attitudes as reported by the primary caregiver, 

child, and teacher at initial data collection in fall 2004 are shown in table 5.7. There were no significant 

differences between the treatment and control groups in these scores. 
Chapter 5. Positive Action 

313

Table 5.6. Initial characteristics of children, their families, and communities—PA

Characteristic Total Treatment Control

 Student sample size 586 298 288

Student demographics

 Gender (percent)

 Male 46.3 47.8 44.7

 Female 53.7 52.2 55.3

 Race/ethnicity (percent)

 White (non-Hispanic) 6.0 5.0 7.0

 Black (non-Hispanic) 51.1 51.9 50.2

 Hispanic 37.4 39.2 35.5

 Other 5.5 3.8 7.3

 Age (in years) (mean) 8.3 8.3 8.3

Primary caregiver and family characteristics

 Primary caregiver’s age (in years) (mean) 35.8 35.3 36.3

 Primary caregiver’s race/ethnicity (percent)

 White (non-Hispanic) 8.2 7.3 9.1

 Black (non-Hispanic) 51.2 52.2 50.2

 Hispanic 35.5 36.9 34.0

 Other 5.2 3.6 6.7

 Primary caregiver’s education (percent)

 Did not complete high school 27.4 26.3 28.5

 Completed high school or equivalent 30.9 27.1 34.7

 Some college 35.7 38.5 32.9

 Bachelor’s or higher degree 6.1 8.1 4.0

 Primary caregiver’s employment (percent)

 Full-time 45.4 44.1 46.7

 Other 54.6 55.9 53.3

 Primary caregiver’s marital status (percent)

 Married 41.4 40.6 42.2

 Other 58.6 59.4 57.8

 Students who live in one household (percent) 92.7 93.5 92.0

 Number of individuals in household (mean) 5.1 4.9 5.3

 Primary caregiver’s relationship to child (percent)

 Mother (stepmother) 83.2 84.3 82.1

 Father (stepfather) 10.2 8.7 11.6

 Other relative/nonrelative 6.6 7.0 6.3

See notes at end of table.
Chapter 5. Positive Action 

314

Table 5.6. Initial characteristics of children, their families, and communities––PA––Continued

Table 5.6. 

Characteristic Total Treatment Control

 Student lives with (percent)

 Mother (stepmother) and father (stepfather) 46.6 44.1 49.1

 Mother (stepmother) only; father (stepfather) not present 45.3 47.7 42.8

 Father (stepfather) only; mother (stepmother) not present 3.0 2.7 3.4

 Other relative/nonrelative, parents not present 5.1 5.5 4.8

 Highest education of anyone in household (percent)

 Did not complete high school 18.8 17.1 20.5

 Completed high school or equivalent 31.1 29.7 32.5

 Some college 39.7 42.3 37.1

 Bachelor’s or higher degree 10.4 11.0 9.9

 Total household income (percent)

 Less than $20,000 55.4 53.6 57.2

 $20,000 to $39,999 28.1 29.4 26.7

 $40,000 to $59,999 10.2 10.7 9.8

 $60,000 or more 6.3 6.3 6.4

 Income-to-poverty-threshold ratio—Below 135 percent (percent) 65.7 60.6 70.7

 Income-to-poverty-threshold ratio—135 to 185 percent (percent) 18.5 20.8 16.3

 Income-to-poverty-threshold ratio—Above 185 percent (percent) 15.8 18.6 12.9

 Alabama Parenting Questionnaire—Poor Monitoring and 

 Supervision Subscale (mean) 1.2 1.2 1.2

 Alabama Parenting Questionnaire—Positive Parenting 

 Subscale (mean) 3.5 3.5 3.5

 Confusion, Hubbub, and Order Scale (mean) 2.2 2.1 2.2

Community characteristics (mean)

 Community Risks Scale 1.9 1.8 1.9

 Community Resources Scale 2.7 2.7 2.6

 Child-Centered Social Control Scale 2.8 2.8 2.8

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 5. Positive Action 

315

Table 5.7. Mean scores and standard deviations for initial outcome measures of sample—PA 

Total Treatment Control

Outcome measure–Report Range Mean SD Mean SD Mean SD

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 1-4 2.8 0.6 2.9 0.7 2.8 0.6

 Normative Beliefs About Aggression–CR 1-4 1.3 0.4 1.3 0.5 1.2 0.4

 Empathy–CR 1-3 2.4 0.4 2.4 0.4 2.4 0.4

Behavior Domain

 Altruistic Behavior–CR 0-3 1.6 0.8 1.6 0.8 1.7 0.7

 Altruistic Behavior–TRS 1-4 1.4 0.5 1.5 0.6 1.3 0.4

 Altruistic Behavior–PCR 1-4 2.4 0.7 2.4 0.7 2.5 0.8

 Positive Social Behavior–TRS 1-4 2.9 0.7 2.9 0.7 2.9 0.7

 Positive Social Behavior–PCR 1-4 2.9 0.5 2.9 0.5 2.9 0.5

 Problem Behavior–CR 0-3 0.3 0.5 0.4 0.5 0.3 0.4

 Problem Behavior–TRS 1-4 1.4 0.5 1.4 0.4 1.5 0.5

 Problem Behavior–PCR 1-4 1.6 0.4 1.5 0.3 1.6 0.4

 ADHD-Related Behavior–TRS 1-4 1.7 0.6 1.7 0.6 1.8 0.6

Academics Domain

 Academic Competence and Motivation–TRS 1-5 2.7 0.9 2.7 0.9 2.7 0.9

 Engagement with Learning–CR 1-4 3.5 0.7 3.5 0.7 3.5 0.8

Perceptions of School Climate Domain

 Positive School Orientation–CR 1-4 3.1 0.7 3.1 0.7 3.1 0.7

 Negative School Orientation–CR 1-4 2.1 0.7 2.2 0.6 2.1 0.7

 Student Afraid at School–CR 1-4 2.6 0.9 2.6 0.9 2.6 0.9

 Victimization at School–CR 0-3 0.9 0.8 0.9 0.8 0.9 0.8

 Student sample size—PCR 586 298 288

 Student sample size—CR 603 303 300

 Student sample size—TRS 605 299 306

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

 SD: Standard deviation

No statistically significant differences were found between values for treatment and control groups. Weights, which assign equal 

weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests were 

conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at the 

school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 5. Positive Action 

316

Characteristics of Teachers and Schools 

Table 5.8 describes the third-, fourth-, and fifth-grade teachers at the study schools. Slightly fewer than half 

were White non-Hispanic (49%). Most of the teachers were female (85%) and had an average of 15 years of 

total teaching experience. Slightly fewer than half (47%) held an advanced or specialist degree. There were no 

statistically significant differences between the treatment and control groups of teachers. 

Data regarding school characteristics were drawn from the Common Core of Data in order to compare 

treatment and control schools. There were no significant differences between the two groups of schools in 

terms of student composition (race/ethnicity and school lunch eligibility), number of students enrolled, 

number of full-time teachers, Title 1 status, or number of years the principal had been at the school (table 

5.9). In addition, there were no significant differences between treatment and control schools in terms of 

location (urban, suburban, or rural) or lowest and highest grade offered (these data are not shown in a table). 

Table 5.8. Initial characteristics of teachers in sample—PA

Characteristic Total Treatment Control

 Teacher sample size 102 54 48

Gender (percent)

 Male 14.9 22.2 7.5

 Female 85.1 77.8 92.5

Race/ethnicity (percent)

 White (non-Hispanic) 49.1 40.5 57.7

 Other 50.9 59.5 42.3

Number of years teaching experience (mean) 15.0 15.7 14.3

Number of years teaching experience in this school (mean) 9.0 10.0 8.1

Type of teaching certificate (percent)

 Regular state certificate or advanced professional certificate 90.5 92.9 88.1

 Other 9.5 7.1 11.9

Education (percent)

 Bachelor’s degree 53.1 57.1 49.1

 Advanced degree/other 46.9 42.9 50.9

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 5. Positive Action 

317

Table 5.9. Initial characteristics of schools in sample—PA 

Characteristic Total Treatment Control

 School sample size 14 7 7

Student race/ethnicity (percent)

 White (non-Hispanic) 9.2 9.1 9.4

 Black (non-Hispanic) 55.1 53.6 56.5

 Hispanic 31.9 32.8 31.0

 Other 3.8 4.5 3.1

Students eligible for free or reduced-price lunch (percent) 90.6 92.6 88.7

Number of students enrolled (mean) 532.0 521.1 542.9

Number of full-time teachers (mean) 30.0 28.5 331.5

Title I status (percent)

 Title I eligible school 100.0 100.0 100.0

 Schoolwide Title I 100.0 100.0 100.0

Number of years principal has been at this school (mean) 5.9 5.9 5.9

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: NCES Common Core of Data (2003-04), the Social and Character Development (SACD) Research Program.

In the Teacher Report on Classroom and School, teachers reported on nine dimensions of school 

environment (these data are not shown in a table): feelings of safety, adequacy of resources, student support, 

freedom to teach as desired, affiliation with and ties to colleagues, innovation regarding new approaches to 

teaching, professional interest, participatory decisionmaking, and work pressure. There were no statistically 

significant differences between treatment and control schools in these reports. 

The Level of SACD in the Schools Near the Beginning of the Study 

During the initial data collection, principals and teachers reported on the SACD activities used in the schools 

and classrooms, the availability of SACD materials, and the professional development provided on SACD. 

Table 5.10 shows that the majority of the principals reported activities to promote six social and character 

development goals: violence prevention and peace promotion (100%), social and emotional development 

(93%), character education (93%), tolerance and diversity (93%), risk prevention and health promotion (93%), 

and civic responsibility and community service (100%). In addition, 86 percent of the principals reported the 

use of activities directed toward behavior management. There were no statistically significant differences 

between the treatment group and the control group in the percentages on principal reports, although this may 

be due to the small principal sample size. The percentages of teachers reporting the use of these activities in 

their classrooms ranged from 45 percent to 81 percent, and there were no significant differences between 

treatment and control teachers. With respect to the use of schoolwide activities, 52 percent to 76 percent of 

teachers reported that their schools used such activities. There was a significant difference between treatment 

and control teachers on reports of their use of unspecified schoolwide SACD activities (one out of six 

comparisons), with treatment teachers reporting fewer ―other activities‖ than control teachers (values 

suppressed to protect confidentiality). 
Chapter 5. Positive Action 

318

Table 5.10. Principal and teacher initial reports on use of SACD programs or activities in 

sample—PA 

SACD program or activity Total Treatment Control

 Principal sample size 14 7 7

 Teacher sample size 102 54 48

Principals reporting that school had programs or activities

 to promote the following SACD goals (percent)

 Violence prevention and peace promotion 100.0 100.0 100.0

 Social and emotional development 92.9 100.0 85.7

 Character education 92.9 85.7 100.0

 Tolerance and diversity 92.9 100.0 85.7

 Risk prevention and health promotion 92.9 85.7 100.0

 Civic responsibility and community service 100.0 100.0 100.0

 Behavior management 85.7 100.0 71.4

 None of the above 0.0 0.0 0.0

Teachers reporting on using programs or activities in their

 class to promote the following SACD goals (percent)

 Violence prevention and peace promotion 63.0 60.2 65.8

 Social and emotional development 70.8 70.2 71.3

 Character education 81.0 79.4 82.5

 Tolerance and diversity 57.7 60.7 54.8

 Risk prevention and health promotion 54.6 58.8 50.4

 Civic responsibility and community service 44.6 45.8 43.4

 Behavior management 77.5 76.1 79.0

 None of the above 6.9 6.0 7.7

Teachers reporting schoolwide use of the following

 activities to promote SACD (percent)

 Morning announcements or videos 60.8 62.0 59.6

 School assemblies 75.7 70.4 81.0

 School newspapers or bulletins 67.1 61.8 72.4

 Special school days 52.3 52.9 51.7

 Special events 67.0 65.5 68.4

 Other activities 11.8 ‡* ‡

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 5. Positive Action 

319

Teachers reported using a broad range of teaching materials to support SACD activities (table 5.11), including 

teacher guides (70%), student materials (68%), instructional aids (38%), giveaways (52%), and children’s 

literature (40%). There were no statistically significant differences between treatment and control teachers in 

their use of these materials. 

Teachers also reported using a wide variety of teaching strategies (table 5.11). Nearly all teachers (99%) 

reported using any of the 20 strategies asked about, and teachers used an average of 12.1 of the strategies. 

There were significant differences between treatment and control teacher reports on the use of 2 of the 20 

strategies. One was a significant difference in peer mediation, with treatment teachers using this strategy more 

often than control teachers (54% versus 35%). The other was a significant difference in the use of time out 

for negative behavior, with treatment teachers using this strategy less often than control teachers (72% versus 

87%). 
Chapter 5. Positive Action 

320

Table 5.11. Teacher initial reports on use of SACD materials and classroom strategies in 

sample—PA 

SACD material and classroom strategy Total Treatment Control

 Teacher sample size 102 54 48

Teachers using the following materials in conjunction with

 social and character development activities (percent)

 Teacher guides (manuals, curricula) 69.7 76.2 63.2

 Student materials (workbooks, worksheets) 67.5 76.3 58.6

 Instructional aids (games, software, videos) 37.6 36.8 38.5

 Giveaways (bookmarks, stickers) 52.4 56.3 48.4

 Children’s literature 39.8 36.3 43.2

 Other types of materials 14.1 5.7 22.5

 Do not use any of the materials listed above 8.2 9.6 6.8

Teachers using any of the strategies listed below to promote

 social and character development in the classroom (percent) 99.2 98.4 100.0

Number of strategies (listed below) used by teachers to promote

 social and character development in the classroom (mean) 12.1 12.3 12.0

Teachers using each of the following strategies to promote

 social and character development (percent)

 Role-playing 71.6 71.3 71.9

 Cooperative learning 98.3 98.4 98.2

 Peer group discussions 86.6 89.2 84.0

 Direct instruction of social and character development 83.4 86.3 80.5

 Skill training 41.2 40.0 42.5

 Incorporating social and character development into 

 academic curriculum 78.6 81.4 75.9

 Parent training 3.2 0.0 6.4

 Parent/community involvement in program 

 development or delivery 16.3 17.4 15.2

 Mentoring 36.3 27.5 45.1

 Good behavior notes sent home daily or weekly 66.6 67.0 66.1

 Presenting role models 60.6 53.5 67.7

 Targeted story reading or writing on SACD themes 84.2 85.9 82.6

 Peer mediation 44.7 54.4* 35.1

 Honor roll for positive behavior 64.3 64.5 64.1

 Pledges or recitations on social and character 

 development themes 47.1 50.7 43.6

 Guided visualization 54.8 57.5 52.1

 Student-led/student-assisted instruction 50.8 55.5 46.2

 Journaling 82.4 82.0 82.9

 Time out for negative behavior 79.7 72.1* 87.4

 Daily or weekly rewards for positive behavior 93.8 91.2 96.4

* Treatment group significantly different from control group at the .05 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 5. Positive Action 

321

Principals and teachers reported on participation in and amount of SACD training and staff development 

provided over the previous 12 months (table 5.12). Principals reported higher participation rates (93% versus 

68%) than did teachers, although principals and teachers reported virtually the same number of training hours 

(5.0 versus 5.2). There were no significant differences between treatment and control principals or teachers 

on any of these measures. 

Table 5.12. Principal and teacher initial reports on SACD professional development in sample––

PA

SACD professional development Total Treatment Control

 Principal sample size 14 7 7

 Teacher sample size 102 54 48

Principals reporting that staff participated in social and character

 development training within the past year (percent) 92.9 100.0 85.7

Teachers reporting participation in social and character 

 development training within the past 12 months (percent) 67.5 72.4 62.6

Number of hours of social and character development training 

 principals report were provided to each staff person last year (mean) 5.0 6.2 4.1

Number of hours of social and character development training 

 teachers report receiving during the past 12 months (mean) 5.2 5.6 4.9

Teachers reporting receiving training in the past 12 months

 in the following areas (percent)

 Violence prevention and peace promotion 17.1 16.0 18.2

 Social and emotional development 22.3 28.9 15.7

 Character education 35.6 41.0 30.2

 Tolerance and diversity 15.5 16.0 14.9

 Risk prevention and health promotion 6.7 ‡ ‡

 Civic responsibility and community service 7.3 8.6 6.0

 Behavior management 37.2 33.3 41.2

‡ Reporting standards not met. Values suppressed to protect confidentiality.

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within each of the programs and to each program across programs, were used in producing the 

treatment, control, and overall means. Statistical tests were conducted using regressions that included program indicators to 

account for the sample design and adjusted for clustering at the school level. Sample size may differ for some outcomes due to 

nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.

The data on the initial level of SACD activity emphasize that the control condition was a ―standard practice‖ 

control. Standard practice at the control schools included using SACD activities, materials, and practices, 

along with professional development for staff, at rates and in types and amounts similar to the treatment 

schools. For example, the percentages of teachers who reported using programs or activities to promote 

specific SACD goals ranged from 46 percent to 79 percent in the treatment schools and from 43 percent to 

83 percent in the control schools. The 3 significant differences between the treatment and control conditions 

in the use of SACD activities was the number expected by chance (3 out of 62 comparisons); 2 of these 

differences favored the control group and 1 favored the treatment group. 
Chapter 5. Positive Action 

322

Impacts on Use of SACD Activities 

The introduction of the formal PA program would be expected to increase the use of SACD activities in the 

treatment schools in comparison to the control schools. The analysis of this impact is based on the Teacher 

Report on Classroom and School (TRCS). Every spring, third-, fourth-, and fifth-grade teachers provided 

information through the TRCS about the social and character development activities they used in their 

classrooms. Specifically, information from the TRCS was used to determine difference between treatment and 

control teachers in several areas: 

1. the use of SACD activities in their classrooms overall and by SACD goal; 

2. the use of materials and strategies to implement the SACD activities within classrooms and within 

the entire school; 

3. the use of staff development to support the teachers; and 

4. teacher support for SACD efforts in the school and the use of practices conducive to the social and 

character development of students. 

TRCS consent and completion rates (table 5.4) led to 78 percent to 91 percent of all teachers having data for 

the 3 years. To estimate intervention impacts for each of the outcomes, testing of the statistical significance of 

the differences in means was used. Preliminary analysis indicated little or no gains in precision from using 

covariates. Before testing the mean differences, the data were weighted such that each school received equal 

weight. Standard errors of the impact estimates account for the clustering of teachers within schools. In 

addition, a set of heuristics (described in chapter 1) was applied to determine whether each outcome domain 

was statistically significant after adjustments were made for the multiple tests conducted. 

Use of Activities 

The percentages of control teachers reporting using any SACD activities in their classroom ranged from 83 

percent to 95 percent over the 3 years (table 5.13, panel 1). For the six individual SACD goals, the range 

varied from 38 percent to 71 percent in Year 1, 52 percent to 77 percent in Year 2, and 53 percent to 65 

percent in Year 3. Control teachers’ use of behavior management activities ranged from 76 percent to 82 

percent over this period. The percentages of control teachers who reported using any SACD activities in their 

classrooms for at least 1 hour per week (panel 2) ranged from 36 percent to 77 percent over the 3 years. For 

the six individual SACD goals, the ranges varied from 13 percent to 26 percent in Year 1, 23 percent to 44 

percent in Year 2, and 19 percent to 31 percent in Year 3. Their use of behavior management activities ranged 

from 42 percent to 69 percent over this period. These findings show that the control schools were using these 

activities as part of their standard practice related to social and character development. 

For teachers’ reported use of any SACD activity, 48 comparisons were made, with 2 expected to be 

significant by chance. There were 3 significant impacts in use of specific SACD activities (panels 1 and 2). In 

Years 2 and 3, more treatment teachers than control teachers reported using social and emotional 

development activities (impact = 26 and 30 percentage points), and in Year 3 more treatment teachers 

reported using character education activities (impact = 32 percentage points). More treatment teachers than 

control teachers reported using any activity for at least 1 hour per week in Year 1 (impact = 41 percentage 

points). More treatment teachers also reported using specific activities for at least 1 hour per week: violence 

prevention and peace promotion in Years 1 and 3 (impact = 27 and 26 percentage points), social and 

emotional development in all years (impact = 37, 29, and 44 percentage points), character education in Years 

1 and 3 (impact = 41 and 58 percentage points), and tolerance and diversity in Year 3 (impact = 27 

percentage points). After the heuristics were applied, the domain for engagement in SACD activities showed 

that the PA program had statistically significant impacts in Years 1 and 3. 

For teachers’ reported use of any named SACD activity (panels 3 and 4), 42 comparisons were made, with 2 

expected to be significant by chance. Six of the 12 impact estimates in Year 1, 8 of the 12 in Year 2, and 8 of 
Chapter 5. Positive Action 

323

the 12 in Year 3 were statistically significant. In Year 1, impacts occurred on social and emotional 

development (impact = 39 percentage points, and impact = 35 percentage points for at least 1 hour per 

week), character education (impact = 47 percentage points, and impact = 44 percentage points for at least 1 

hour per week), and any named activity (impact = 47 percentage points, and impact = 47 percentage points 

for at least 1 hour per week). In Year 2, impacts were seen for social and emotional development (impact = 

41 percentage points, and impact = 37 percentage points for at least 1 hour per week), character education 

(impact = 45 percentage points), tolerance and diversity (impact = 32 percentage points), risk prevention and 

health promotion (impact = 35 percentage points, and impact = 30 percentage points for at least 1 hour per 

week), civic responsibility and community service (impact = 19 percentage points), and any named activity for 

at least 1 hour per week (impact = 36 percentage points). In Year 3, there were significant impacts on the use 

of named violence prevention and peace promotion activities (impact = 28 percentage points) for at least 1 

hour per week, social and emotional activities (impact = 37 percentage points, impact = 34 percentage points 

at least 1 hour per week), character education (impact = 49 percentage points, impact = 46 percentage points 

for at least 1 hour per week), tolerance and diversity (impact = 38 percentage points, impact = 31 percentage 

points for at least 1 hour per week), and risk prevention and health promotion (impact = 26 percentage 

points). The overall impact of the PA program on the domain for engagement in named SACD activities was 

significant in all 3 years. 
Table 5.13. Impacts on teacher-reported SACD classroom activities––PA 

Panel 1: Engagement in any activities to promote SACD goals1

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 60 54 43 39 39 38

Violence prevention and peace promotion

 (percent) 66.1 58.5 7.6 0.523 71.6 77.4 -5.8 0.594 68.2 60.7 7.5 0.590

Social and emotion development

 (percent) 67.7 54.7 13.0 0.259 88.7* 62.6 26.1 0.013 83.9* 53.6 30.3 0.019

Character education (percent) 88.9^ 70.9 18.0 0.086 91.1^ 75.2 15.9 0.071 94.4* 62.4 32.0 0.017

Tolerance and diversity (percent) 59.1^ 37.7 21.5 0.086 66.1 64.1 2.0 0.893 77.1 53.1 24.1 0.165

Risk prevention and health promotion

 (percent) 62.0 47.5 14.5 0.128 66.8 70.9 -4.1 0.769 67.1 61.6 5.6 0.748

Civic responsibility and community service

 (percent) 43.7 46.2 -2.4 0.815 55.4 51.7 3.6 0.818 53.5 65.1 -11.6 0.325

Any SACD goal (percent) 91.8 82.6 9.1 0.334 91.1 94.8 -3.6 0.494 96.4 88.8 7.6 0.325

Behavior management (percent) 80.9 81.6 -0.8 0.937 91.7 76.3 15.3 0.139 81.2 81.1 0.1 0.995

See note at end of table.

324

Chapter 5. Positive Action
Table 5.13. Impacts on teacher-reported SACD classroom activities––PA—Continued 

Panel 2: Engagement in any activities to promote SACD goals for at least 1 hour per week

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 60 54 43 39 39 38

Violence prevention and peace promotion

 (percent) 43.6* 17.1 26.5 0.021 46.4 31.4 14.9 0.209 48.1* 22.1 26.0 0.028

Social and emotional development

 (percent) 51.3* 14.7 36.7 0.011 57.2* 28.2 29.0 0.024 64.9* 21.4 43.6 0.007

Character education (percent) 66.9* 26.3 40.6 0.001 64.1 43.5 20.6 0.129 86.4* 28.8 57.6 0.002

Tolerance and diversity (percent) 37.8^ 13.9 23.9 0.081 38.6 28.2 10.4 0.249 46.2* 19.2 27.1 0.006

Risk prevention and health promotion

 (percent) 29.4 13.2 16.2 0.134 43.6^ 25.0 18.6 0.087 37.0 31.0 6.0 0.546

Civic responsibility and community service

 (percent) 13.5 12.5 1.1 0.844 26.9 22.7 4.2 0.687 26.5 27.6 -1.1 0.930

Any SACD goal (percent) 76.3* 35.5 40.7 0.002 84.1 76.6 7.6 0.323 80.3 76.4 3.9 0.724

Behavior management (percent) 59.3 68.5 -9.2 0.525 72.6^ 42.1 30.6 0.066 58.4 65.7 -7.3 0.609

See note at end of table.

325

Chapter 5. Positive Action
Table 5.13. Impacts on teacher-reported SACD classroom activities––PA—Continued 

Panel 3: Engagement in activities to promote SACD goals linked to named SACD programs2

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 60 54 43 39 39 38

Violence prevention and peace promotion

 (percent) 36.9 23.4 13.5 0.152 36.2 22.5 13.7 0.215 47.1^ 16.7 30.5 0.066

Social and emotional development

 (percent) 47.3* 8.3 39.0 0.007 48.4* 7.3 41.1 0.001 49.0* 11.9 37.1 0.019

Character education (percent) 59.2* 11.8 47.4 0.001 ‡* ‡ 45.4 0.001 60.4* 11.6 48.9 0.001

Tolerance and diversity (percent) 31.2 0.0 31.2 † ‡* ‡ 32.2 0.014 45.3* 7.1 38.2 0.002

Risk prevention and health promotion

 (percent) 37.9 19.4 18.4 0.110 48.6* 14.0 34.6 0.002 44.6* 18.4 26.2 0.030

Civic responsibility and community service

 (percent) 5.3 0.0 5.3 † ‡* ‡ 19.1 0.033 23.0 9.5 13.5 0.248

Any named activity (percent) 77.3* 30.7 46.6 0.002 59.1 46.2 12.9 0.260 69.2 42.2 27.0 0.121

See note at end of table.

326

Chapter 5. Positive Action
Table 5.13. Impacts on teacher-reported SACD classroom activities––PA—Continued 

Panel 4: Engagement in activities to promote SACD goals linked to named SACD programs for at least 1 hour per week

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 60 54 43 39 39 38

Violence prevention and peace promotion

 (percent) 31.9^ 14.1 17.8 0.051 26.6 12.5 14.1 0.211 40.4* 12.2 28.2 0.050

Social and emotional development

 (percent) 40.3* 4.8 35.4 0.027 ‡* ‡ 36.9 0.004 43.7* 10.0 33.7 0.014

Character education (percent) 49.2* 5.3 43.8 0.002 43.3 0.0 43.3 † 53.4* 7.5 46.0 0.005

Tolerance and diversity (percent) 28.0 0.0 28.0 † 27.4 0.0 27.4 † 39.0* 7.6 31.4 0.002

Risk prevention and health promotion

 (percent) 26.8^ 10.0 16.8 0.099 37.4* 7.8 29.5 0.031 30.2^ 9.6 20.6 0.060

Civic responsibility and community service

 (percent) 5.3 0.0 5.3 † 18.7 0.0 18.7 † 17.3 7.9 9.4 0.441

Any named activity (percent) 64.1* 17.0 47.1 0.000 52.4* 16.5 35.9 0.018 51.5 30.6 20.8 0.242

† Not applicable.

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 In Year 1, the omnibus impact for all the outcomes measured together was positive and statistically significant on the basis of a multivariate statistical test. In Years 1 and 3, at least 

one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) procedure to adjust 

significance levels downward to account for the multiple testing of impacts.

2 In Years 1 and 2, at least half of the impacts were positive and statistically significant and no impact was negative and statistically significant on the basis of univariate statistical tests. 

In all 3 years, at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg 

(1995) procedure to adjust significance levels downward to account for the multiple testing of impacts.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

327

Chapter 5. Positive Action
Chapter 5. Positive Action 

328

Use of Materials and Strategies 

For use of materials and strategies to support SACD goals, 87 comparisons were made, with 4 expected to be 

significant by chance. Eleven significant impacts were found on treatment teachers’ use of materials and 

strategies in all 3 years. In Year 1, more treatment teachers used teacher guides (impact = 32 percentage 

points), student materials such as workbooks or worksheets (impact = 33 percentage points), giveaways 

(impact = 37 percentage points), and student-led/student-assisted instruction (impact = 19 percentage points) 

(table 5.14). In Year 2, more treatment teachers continued to use teacher guides (impact = 26 percentage 

points), skill training (impact = 28 percentage points), parent/community involvement (impact = 25 

percentage points), targeted story reading or writing on SACD themes (impact = 19 percentage points), and 

time-out for negative behavior (impact = 16 percentage points). On average, treatment teachers also used 

more strategies than control teachers (by 1.5 strategies). In Year 3, a significant impact was seen for treatment 

teachers’ use of giveaways (impact = 23 percentage points). The impact on the domain of materials and 

strategies was statistically significant in Year 1. 
Table 5.14. Impacts on use of SACD classroom materials and teaching strategies—PA 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD material and teaching strategy

1

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 60 54 43 39 39 38

Use of SACD materials (percent)

 Teacher guides (manuals, curricula) 86.7* 54.5 32.1 0.017 86.1* 60.5 25.6 0.012 90.5 66.8 23.7 0.178

 Student materials (workbooks or sheets) 84.8* 51.8 32.9 0.007 72.2 58.3 13.9 0.243 85.1^ 66.8 18.3 0.076

 Instructional aids (games, software, videos) 29.7 34.5 -4.7 0.733 38.4 33.8 4.6 0.630 45.1 55.5 -10.4 0.340

 Giveaways (bookmarks, stickers) 76.7* 39.6 37.1 0.004 68.7 55.6 13.1 0.314 79.0* 56.3 22.8 0.019

 Children’s literature 33.9 36.4 -2.5 0.823 36.0 49.4 -13.5 0.341 34.2 26.4 7.8 0.557

 Other types of materials 6.9 15.2 -8.4 0.230 19.7 16.2 3.5 0.686 ‡ ‡ -5.4 0.265

 Did not use any of these materials ‡^ ‡ -20.5 0.069 ‡ ‡ -5.7 0.299 ‡ ‡ -3.3 0.630

Use of teaching strategies (percent)

 Role-playing 80.1 68.5 11.6 0.191 80.1 72.4 7.7 0.508 83.2 88.9 -5.7 0.542

 Cooperative learning 97.6 96.3 1.3 0.706 100.0 97.1 2.9 † 100.0 100.0 0.0 †

 Peer group discussions 93.1 81.4 11.7 0.113 90.9 88.6 2.3 0.729 97.0 100.0 -3.0 †

 Direct instruction of SACD 86.9 77.6 9.2 0.367 94.4 83.7 10.8 0.073 100.0 92.8 7.2 †

 Skill training 59.7 46.7 13.1 0.187 64.7* 36.7 28.0 0.015 81.1 86.1 -5.0 0.563

 Incorporating SACD into academic 

 curriculum 86.0 75.8 10.2 0.149 80.0 68.7 11.3 0.260 100.0 97.9 2.1 †

 Parent training ‡ ‡ -10.2 0.118 ‡ ‡ 13.4 0.116 51.3^ 27.0 24.3 0.077

 Parent/community involvement 7.8 15.0 -7.2 0.322 32.7* 7.6 25.1 0.011 60.1 39.4 20.7 0.100

 Mentoring 45.8 37.0 8.8 0.445 41.2 41.2 0.0 1.000 76.2 51.4 24.8 0.137

 Good behavior notes sent home daily

 or weekly 60.9 51.8 9.1 0.390 74.9 59.8 15.2 0.157 83.1 87.9 -4.8 0.526

 Presenting role models 68.9 58.8 10.1 0.197 65.4 64.4 1.0 0.925 85.5 74.9 10.6 0.255

See notes at end of table.

329

Chapter 5. Positive Action
Table 5.14. Impacts on use of SACD classroom materials and teaching strategies—PA—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD material and teaching strategy

1

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

Use of teaching strategies (percent)—

 Continued

 Targeted story reading or writing on social

 and character development themes 94.8^ 80.9 13.9 0.083 98.5* 79.6 18.9 0.015 97.6 97.9 -0.2 0.939

 Peer mediation 59.7 56.0 3.8 0.619 61.3 62.1 -0.9 0.917 92.0 80.2 11.9 0.174

 Honor roll for positive behavior 67.3 69.1 -1.7 0.854 58.6 75.4 -16.7 0.108 81.5 89.5 -8.0 0.375

 Pledges or recitations on social and

 character development themes 48.9 54.0 -5.0 0.664 57.2 55.9 1.4 0.934 75.6 72.0 3.7 0.774

 Guided visualization 62.5 53.4 9.0 0.492 69.6 60.7 9.0 0.485 77.1 76.8 0.3 0.977

 Student-led/student-assisted instruction 65.1* 45.9 19.2 0.039 59.6 48.8 10.8 0.265 90.6 79.1 11.5 0.115

 Journaling 81.8 80.3 1.5 0.880 77.6 87.8 -10.2 0.411 95.2 97.6 -2.3 0.631

 Time out for negative behavior 84.3 88.9 -4.6 0.562 93.8* 77.6 16.2 0.031 100.0 97.6 2.4 †

 Daily or weekly rewards for positive

 behavior 91.8 89.9 1.9 0.696 97.6 92.7 4.9 0.344 98.0 97.9 0.1 0.986

 Any strategy 100.0 100.0 0.0 † 100.0 100.0 0.0 † 100.0 100.0 0.0 †

 Number of strategies (mean) 13.3 12.4 1.0 0.202 14.0* 12.5 1.5 0.042 16.9 16.0 0.9 0.170

† Not applicable.

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 In Year 1, at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) 

procedure to adjust significance levels downward to account for the multiple testing of impacts.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

330

Chapter 5. Positive Action
Chapter 5. Positive Action 

331

Regarding the use of schoolwide strategies, 18 comparisons were made between treatment and control 

teacher reports, with 1 expected to be significant by chance. There were 2 statistically significant differences 

between treatment and control teachers’ reported use of schoolwide strategies (these data are not shown in a 

table), with treatment teachers reporting more use of unspecified schoolwide activities than control teachers 

(impact = 13 percentage points) in Year 2, and more use of morning announcements or videos (impact = 36 

percentage points) in Year 3. The overall impact of the PA program on the domain for use of schoolwide 

strategies in named SACD activities was not significant in any year. 

Participation in Professional Development 

Regarding reported participation in professional development, 27 comparisons were made over 3 years, with 

1 expected to be significant by chance. The intervention had a statistically significant effect on treatment 

teachers’ participation in professional development, with more treatment teachers than control teachers 

reporting SACD training in the past 12 months (impact = 35 percentage points) and more hours of training 

(by 4.9 hours on average) in Year 1. In Years 1 and 2, there were significant impacts on training for character 

education (impact = 29 and 28 percentage points). No significant impacts were found on the domain in any 

of the 3 years. 
Table 5.15. Impacts on teacher-reported SACD professional development—PA 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD professional development

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 60 54 43 39 39 38

SACD training in past 12 months (percent) 89.3* 54.1 35.3 0.015 73.0 68.5 4.5 0.623 62.9 58.0 5.0 0.682

Hours of SACD training (mean) 8.0* 3.1 4.9 0.011 4.2 4.3 -0.1 0.944 4.0 4.5 -0.5 0.737

Training by goal (percent)

 Violence prevention and peace promotion 21.2 13.1 8.1 0.447 18.5 24.7 -6.2 0.528 13.0 8.8 4.2 0.543

 Social and emotional development 31.4 21.2 10.2 0.434 24.6 30.4 -5.8 0.654 30.7 13.6 17.1 0.122

 Character education 63.1* 34.2 29.0 0.040 45.8* 18.2 27.6 0.025 35.5^ 13.6 21.9 0.086

 Tolerance and diversity 29.1 10.3 18.8 0.125 16.6 16.4 0.2 0.984 11.8 25.0 -13.2 0.208

 Risk prevention and health promotion 11.2 9.0 2.1 0.778 9.6 15.2 -5.6 0.491 20.2 17.7 2.5 0.822

 Civic responsibility and community service ‡ ‡ 8.2 0.179 7.3 7.5 -0.2 0.974 0.0 6.8 -6.8 †

 Behavior management 42.0 24.8 17.2 0.238 37.0 46.3 -9.3 0.447 38.4 24.8 13.5 0.369

† Not applicable.

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

332

Chapter 5. Positive Action
Chapter 5. Positive Action 

333

Attitudes and Practices 

Teachers reported on their enthusiasm for SACD efforts in their schools (these data are not shown in a table) 

by indicating enthusiasm, cooperation, or open dislike. They also reported on the SACD practices of teachers 

and staff members in their schools (these data are not shown in a table). These practices included modeling 

positive character and behavior traits with students and fellow teachers, involving students in making 

decisions, giving students a voice in school governance, the school encouraging parent involvement in 

children’s social and character development, and using developmentally appropriate discipline strategies 

rather than punishment for misbehavior. Twenty-seven comparisons were made over 3 years, with 1 expected 

to be significant by chance. In Year 1, there was a statistically significant impact on teachers’ enthusiasm for 

SACD efforts in their schools (out of 9 comparisons made), with more treatment teachers reporting 

enthusiasm than control teachers (impact = 4 percentage points). There were no significant differences in 

treatment and control teacher reports of the overall use of practices conducive to students’ social and 

character development for any of the years. There was a significant positive impact on the domain in Year 1.

Year-by-Year Impacts on Students and Perceptions of School Climate 

The primary research question for the PA evaluation was this: 

What is the effect of the PA program on children’s social and emotional competence, 

behavior, academics, and perceptions of school climate? 

The first approach to answering this question was to examine the year-by-year impacts of the PA program on 

these student and school climate outcomes over the 3 years as the students progressed from third through 

fifth grades. 

Equation (2) (described in chapter 1) was estimated to provide the impacts of the PA program on the 20 

outcomes using data from the 14 treatment and control schools. For the PA evaluation, equation (2) excluded 

the program fixed effects (p) and included program-specific covariates and random school effects covariates. 

Table 5.16 lists the covariates used with outcomes from each report in the PA analysis. 
Chapter 5. Positive Action 

334

Table 5.16. Covariates used with outcomes from each report for analysis—PA 

Potential covariate

CR 

outcome

PCR 

outcome

TRS 

outcome

TRCS 

outcome

 Total number 20 26 22 7

Child-reported

 Female   

 Hispanic   

 Black (non-Hispanic)   

 Other ethnicity   

 Age in years   

 Scales

 Afraid at School

 Altruistic Behavior 

 Empathy

 Engagement with Learning 

 Negative School Orientation 

 Normative Beliefs About Aggression   

 Sense of School as a Community

 Problem Behavior   

 Self-Efficacy for Peer Interactions 

 Victimization at School  

Primary caregiver-reported

 Age in years  

 Completed high school or equivalent   

 Some college   

 Bachelor’s or higher degree   

 Highest level of education in household

 Completed high school or equivalent 

 Some college 

 Bachelor’s or higher degree 

 Mother present in home life

 Mother and father present  

 Respondent someone other than mother or father  

 Number of people in household   

 Household income: $20,000 to $40,000

 Household income: $40,000 to $60,000

 Household income: More than $60,000

 Income-to-poverty-threshold ratio: Below 135 percent  

 Income-to-poverty-threshold ratio: 135 to 185 percent  

 Full-time employment  

 Part-time employment  

See notes at end of table.
Chapter 5. Positive Action 

335

Table 5.16. Covariates used with outcomes from each report for analysis––PA––Continued

Potential covariate

CR 

outcome

PCR 

outcome

TRS 

outcome

TRCS 

outcome

 Parental scales

 APQ-Poor Monitoring and Supervision Subscale 

 APQ-Positive Parenting Subscale

 Child-Centered Social Control

 Confusion, Hubbub, and Order   

 Community Resources

 Community Risk 

 Parent and Teacher Involvement

 Child scales

 Altruistic Behavior

 Positive Social Behavior 

 Problem Behavior

Teacher-reported

 Female 

 Hispanic 

 Black (non-Hispanic) 

 Other ethnicity 

 Total teaching experience 

 Total experience in current school

 Regular certificate 

 Other certificate 

 Highest degree–bachelor’s

 Child scales

 Academic Competence and Motivation

 ADHD-Related Behavior  

 Altruistic Behavior  

 Positive Social Behavior 

 Problem Behavior 

 Parent and Teacher Involvement

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

 APQ: Alabama Parenting Questionnaire

 : Covariate used

 Blank cell: Covariate not used

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 5. Positive Action 

336

To assess the statistical power of the program-level impact estimates, minimum detectable impacts in effect 

size units (MDES) for each outcome measure were calculated for the PA evaluation (table 5.17). MDES 

represent the smallest impacts in effect size (standard deviation) units that can be detected with a high 

probability (80%). MDES are primarily a function of study sample sizes, the degrees of freedom available for 

statistical tests, and design effects due to clustering (Schochet 2005). For the PA evaluation, the MDES range 

from 0.096 to 0.953 for the child-level outcomes based on the Child, Caregiver, and Teacher Report on 

Student and from 0.321 to 0.742 for the school climate outcomes based on the Teacher Report on Classroom 

and School. In general, the MDES for the school climate outcomes were larger than those for the child-level 

outcomes. 
Chapter 5. Positive Action 

337

Table 5.17. Adjusted minimum detectable effect sizes for impact evaluation––PA 

Outcome measure–Report Year 1 Year 2 Year 3

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 0.203 0.141 0.168

 Normative Beliefs About Aggression–CR 0.241 0.350 0.203

 Empathy–CR 0.256 0.296 0.178

Behavior Domain

 Altruistic Behavior–CR 0.096 0.278 0.103

 Altruistic Behavior–PCR 0.132 0.162 0.162

 Altruistic Behavior–TRS 0.631 0.953 0.784

 Positive Social Behavior–PCR 0.106 0.165 0.199

 Positive Social Behavior–TRS 0.310 0.392 0.285

 Problem Behavior–CR 0.231 0.291 0.233

 Problem Behavior–PCR 0.159 0.150 0.130

 Problem Behavior–TRS 0.268 0.206 0.244

 ADHD-Related Behavior–TRS 0.204 0.198 0.285

Academics Domain

 Engagement with Learning–CR 0.096 0.154 0.223

 Academic Competence and Motivation–TRS 0.145 0.174 0.263

Perceptions of School Climate Domain

 Positive School Orientation–CR 0.333 0.350 0.373

 Negative School Orientation–CR 0.096 0.259 0.335

 Student Afraid at School–CR 0.117 0.183 0.186

 Victimization at School–CR 0.097 0.393 0.175

 Feelings of Safety–TRCS 0.489 0.384 0.742

 Student Support for Teachers–TRCS 0.468 0.321 0.602

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The minimum detectable effect (MDE) formula used in the calculations is as follows:

where sT and sC are the number of treatment and comparison schools; nT and nC are the average number of students per classroom; 

1 is the intraclass correlation (ICC) at the school level; and factor(df) is a constant that depends on the number of degrees of 

freedom (df) available for analysis (and is 2.802 for the pooled analysis).

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 5. Positive Action 

338

Table 5.18 provides the estimates of the PA program’s impacts on each of the 20 outcomes over each of the 

3 years (60 impacts in total, with 3 expected to be statistically significant by chance). Of the 60 results, 3 were 

statistically significant (2 beneficial and 1 detrimental). In Year 1, the PA program had a detrimental impact 

on Engagement with Learning (Child Report, effect size [ES] = -0.25) and a substantively important but 

nonsignificant beneficial impact on Altruistic Behavior (Teacher Report on Student, ES = 0.27). In Year 2, 

beneficial impacts were found for Positive Social Behavior (Primary Caregiver Report, ES = 0.24) and 

Problem Behavior (Teacher Report on Student, ES = -0.24). Substantively important but nonsignificant 

beneficial impact was found in Years 2 and 3 on Student Support for Teachers (Teacher Report on 

Classroom and School, ES = 0.28 and 0.27). Application of the heuristic to adjust for multiple comparisons 

within each outcome domain indicates that PA had a statistically significant detrimental impact on Academics 

in Year 1. 
Table 5.18. Impacts on child and school outcomes––PA 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 2.99^ 3.13 -0.24 0.090 3.15 3.22 -0.12 0.258 3.20 3.18 0.03 0.823

 Normative Beliefs About Aggression–CR (-) 1.34 1.42 -0.13 0.388 1.48 1.65 -0.21 0.303 1.50 1.60 -0.16 0.264

 Empathy–CR (+) 2.28 2.18 0.21 0.176 2.14 2.07 0.14 0.422 2.05 1.99 0.16 0.255

Behavior Domain

 Altruistic Behavior–CR (+) 1.37 1.45 -0.09 0.289 1.25 1.25 0.00 0.998 1.25 1.16 0.14 0.198

 Altruistic Behavior–PCR (+) 2.42 2.42 0.01 0.927 2.49 2.34 0.19 0.139 2.49 2.38 0.16 0.281

 Altruistic Behavior–TRS (+) 1.59 1.45 0.27o 0.480 1.62 1.50 0.21 0.728 1.40 1.38 0.03 0.958

 Positive Social Behavior–PCR (+) 2.99 2.95 0.07 0.342 3.03* 2.89 0.24 0.039 3.03 2.96 0.14 0.360

 Positive Social Behavior–TRS (+) 2.95 2.92 0.04 0.740 3.00 2.86 0.19 0.365 2.98 2.83 0.20 0.238

 Problem Behavior–CR (-) 0.51 0.47 0.07 0.632 0.59 0.58 0.01 0.948 0.67 0.74 -0.10 0.494

 Problem Behavior–PCR (-) 1.55 1.60 -0.10 0.261 1.57 1.53 0.11 0.320 1.52 1.55 -0.08 0.485

 Problem Behavior–TRS (-) 1.53 1.58 -0.08 0.486 1.48* 1.61 -0.24 0.048 1.51 1.62 -0.16 0.222

 ADHD-Related Behavior–TRS (-) 1.72 1.77 -0.08 0.415 1.69 1.75 -0.11 0.357 1.71 1.78 -0.10 0.533

Academics Domain1

 Engagement with Learning–CR (+) 3.56* 3.69 -0.25 0.017 3.58 3.64 -0.11 0.390 3.56 3.54 0.03 0.862

 Academic Competence and Motivation–

 TRS (+) 2.78 2.80 -0.02 0.783 2.80 2.79 0.01 0.933 2.67 2.70 -0.03 0.818

See notes at end of table.

339

Chapter 5. Positive Action
Table 5.18. Impacts on child and school outcomes—PA—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 2.73 2.65 0.11 0.576 2.44 2.47 -0.04 0.833 2.41 2.32 0.14 0.565

 Negative School Orientation–CR (-) 2.14 2.15 -0.02 0.837 2.17 2.22 -0.08 0.628 2.24 2.30 -0.12 0.558

 Student Afraid at School–CR (-) 2.53 2.55 -0.02 0.788 2.41 2.42 -0.02 0.892 2.35 2.39 -0.05 0.716

 Victimization at School–CR (-) 0.90 0.82 0.10 0.228 0.77 0.67 0.13 0.578 0.95 0.83 0.16 0.244

 Feelings of Safety–TRCS (+) 3.25 3.09 0.15 0.473 3.19 3.31 -0.13 0.494 3.07 3.23 -0.18 0.618

 Student Support for Teachers–TRCS (+) 3.05 2.92 0.13 0.418 3.27 2.99 0.28° 0.113 3.12 2.91 0.27o 0.419

* Treatment group significantly different from control group at the .05 level.

^ Treatment group significantly different from control group at the .10 to > .05 level.

o Substantive (but nonsignificant at .05 level) effect size of ≥ .25 or ≤ -.25.

1 In Year 1, at least half of the impacts were positive and statistically significant and no impact was negative and statistically significant based on univariate statistical tests, and at least 

one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) procedure to adjust 

significance levels downward to account for the multiple testing of impacts. The number of results found significant was no more than expected by chance.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models where each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. See table 1.5 for 

information about the measures used to create the outcome variables. The effect size was calculated by dividing the estimated impact by the standard deviation of the outcome 

measure for the control group. 

SOURCE: The Social and Character Development (SACD) Research Program.

340

Chapter 5. Positive Action
Chapter 5. Positive Action 

341

Impacts on Child Outcomes Over Time 

PA impacts on the child outcomes over time were estimated using growth curve models by examining 

treatment and control group differences in the trajectories of student outcomes during the follow-up period 

while accounting for clustering at the school level. The growth curve models are estimated using a three-level 

hierarchical linear model, where Level 1 corresponds to time, Level 2 to students, and Level 3 to schools 

(described in chapter 1). 

Table 5.19 provides the estimates of the PA program’s impacts on the growth in student outcomes over the 3 

years. The estimated impacts ranged in effect size units (absolute value) from 0.00 to 0.09. None of the 18 

estimated PA intervention impacts on the trajectories of child outcomes was statistically significant (1 in 18 is 

expected to be significant by chance). 
Table 5.19. Impacts on growth of child outcomes—PA

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2Treatment Control

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 2.92 0.14 0.13 0.01 0.02 0.03 0.756

 Normative Beliefs About Aggression–CR (-) 1.28 0.14 0.13 0.01 0.01 0.05 0.911

 Empathy–CR (+) 2.37 -0.15 -0.15 0.00 0.01 0.02 0.925

Behavior Domain

 Altruistic Behavior–CR (+) 1.59 -0.14 -0.19 0.05 0.06 0.03 0.150

 Altruistic Behavior–PCR (+) 2.43 0.01 -0.07 0.08 0.09 0.05 0.100

 Altruistic Behavior–TRS (+) 1.45 0.00 0.00 0.00 -0.01 0.09 0.965

 Positive Social Behavior–PCR (+) 2.92 0.03 -0.01 0.04 0.07 0.03 0.119

 Positive Social Behavior–TRS (+) 2.85 0.02 -0.05 0.07 0.08 0.06 0.310

 Problem Behavior–CR (-) 0.33 0.17 0.13 0.04 0.08 0.05 0.506

 Problem Behavior–PCR (-) 1.59 -0.01 0.01 -0.01 -0.03 0.02 0.392

 Problem Behavior–TRS (-) 1.48 0.05 0.09 -0.04 -0.06 0.03 0.245

 ADHD-Related Behavior–TRS (-) 1.77 0.01 0.03 -0.01 -0.01 0.05 0.796

Academics Domain

 Engagement with Learning–CR (+) 3.60 0.00 0.01 -0.01 -0.01 0.04 0.834

 Academic Competence and Motivation–TRS (+) 2.71 -0.05 -0.02 -0.03 -0.02 0.05 0.609

See notes at end of table.

342

Chapter 5. Positive Action
Table 5.19. Impacts on growth of child outcomes—PA—Continued

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2Treatment Control

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 2.93 -0.24 -0.24 0.00 0.00 0.05 0.977

 Negative School Orientation–CR (-) 2.10 0.05 0.08 -0.03 -0.04 0.03 0.338

 Student Afraid at School–CR (-) 2.59 -0.05 -0.13 0.07 0.09 0.05 0.156

 Victimization at School–CR (-) 0.87 -0.04 -0.02 -0.02 -0.02 0.05 0.639

1 Pertains to the estimated slope of the outcome for the treatment or control groups.

2 The average score at implementation is calculated across treatment and control groups, using regression models for adjustment on covariates. 

3 Estimated difference between the slope of the treatment and control groups.

4 Effect size: the slope of the treatment group minus the slope of the control group divided by the standard deviation of the outcome for the program’s control group (the standard 

deviation is calculated without accounting for school-level clustering or regression adjustments).

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

No findings were found statistically significant at or below the .05 level. The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated 

using HLM 6.06. Sample weights were used in all analyses to give (1) each school equal weight in each program (within each time period) and (2) each time period equal weight within 

the analysis. See table 1.5 for information about the measures used to create the outcome variables.

SOURCE: The Social and Character Development (SACD) Research Program.

343

Chapter 5. Positive Action
Chapter 5. Positive Action 

344

Summary 

As part of the Social and Character Development (SACD) Research Program, researchers at the Illinois site 

implemented and evaluated the PA program. This program focused on teaching children about the benefits 

of physical, intellectual, social, and emotional positive actions with a curriculum designed to enhance 

children’s self-concept and self-regulatory skills through the identification of positive thoughts, feelings, and 

actions. Fourteen public schools in a single large, urban school district in Illinois were recruited by the Illinois 

research team and randomly assigned to treatment and control conditions to determine the impact of the PA 

program on social and character development activities in the schools and on the child outcome domains of 

Social and Emotional Competence, Behavior, Academics, and Perceptions of School Climate. 

Analyses of the initial characteristics of the sample (students, caregivers, communities, teachers, and schools) 

indicated that randomization to treatment and control status produced groups that were similar on the 

observed characteristics at the start of the study. Documentation of the initial level of social and character 

development activities in the schools revealed few differences (3, the number that would be expected by 

chance out of 62 comparisons) between the treatment and control teachers and classrooms. This is important 

for two reasons: one, it indicates that randomization created comparable groups near the start of the study, 

and two, it shows that treatment and control schools both had high levels of SACD activities near the start of 

the study, indicating that social and character development activities are part of the ―standard practice‖ of 

these schools in Illinois. Standard practice at all of the schools included reports of 45 percent to 81 percent of 

teachers using SACD activities, 92 percent of teachers using specific materials in conjunction with these 

activities, 99 percent of teachers using at least one of the specified instructional strategies, and 68 percent of 

teachers participating in SACD training over the past 12 months. 

Analyses of the impacts of the PA program on the use of SACD activities in the schools revealed impacts on 

the use of such activities (33 out of 90) and related materials and strategies (13 out of 87) across the 3 years, 

and use of more professional development activities for treatment teachers in all years (4 out of 27). 

Of the 20 child-level outcomes representing the four domains of Social and Emotional Competence, 

Behavior, Academics, and Perceptions of School Climate assessed in each of the 3 years of the study (a total 

of 60 results), 3 were statistically significant (2 beneficial and 1 detrimental). A growth curve analysis was used 

to analyze the change over time in these same outcomes between initial data collection and the final outcome 

data collection at the end of the study. None of the 18 child-level outcomes assessed showed that PA had a 

significant impact. 

The SACD evaluation did not find evidence to support the hypothesis that PA had beneficial impacts on 

students’ social and character development. Such results could be caused by the inability of the program to 

cause such change, possibly because the theory of action for the program is incomplete or the activities to 

carry out that theory are not effective. 

However, these results may also be due to the inability of the evaluation to observe such a change due to the 

control condition, the level of nonparticipation, or the sample size. The control schools continued using their 

standard SACD activities, and these turned out to be high in quantity and broad in scope. In 2004 Illinois 

passed the Illinois Children’s Mental Health Act (Public Act 93-0495), Section 15(a), which calls on the 

Illinois State Board of Education to ―develop and implement a plan to incorporate social and emotional 

development standards as part of the Illinois Learning Standards.‖ This state-level change may have led 

control schools to implement increasing amounts of SACD activities during the course of this study. While 

the PA program had a significant positive impact on the amount and type of SACD activities, the resulting 

difference between the treatment and control schools in the amount of SACD activities may not have been 

large enough to cause significant differences in the student outcomes. 
Chapter 5. Positive Action 

345

Second, about one-quarter to one-third of the students in the sample universe did not take part (depending 

on year) because of nonconsent or noncompletion of the surveys. As a determination could not be made as 

to whether the students not taking part significantly differed from those who did take part, the evaluation’s 

results are valid only for the students who took part. If the students not taking part were different, and if they 

would have responded better to PA than to the SACD activities occurring in the control schools, then the 

evaluation could have underestimated the program’s impact. Third, the sample size of 14 schools and the 

resulting higher MDES compared to those for the multiprogram evaluation may have reduced the likelihood 

of detecting statistically significant effects. However, it should be noted that 57 percent of the MDES for the 

60 outcomes used in the year-by-year analysis were below 0.25 (42% were below 0.20). 

 
This page is intentionally blank. 
347

Chapter 6. Promoting Alternative Thinking Strategies 

(PATHS) 

The Children’s Institute

(New York/Minnesota site)

Intervention 

Researchers at the Children’s Institute in Rochester (New York/Minnesota site) evaluated the Promoting 

Alternative Thinking Strategies (PATHS) program (Kusche and Greenberg 1994). PATHS is designed to 

enhance social and emotional competence and understanding in children and to develop a caring, prosocial 

context that facilitates educational processes in the classroom. Table 6.1 describes PATHS’ general 

characteristics (panel 1), the types of instruction and strategies used (panel 2), the professional development 

provided for those implementing the program (panel 3), and the social and character development activities 

(panel 4) and outcomes (panel 5) addressed by the program. The program includes the following 

components: 

 The PATHS classroom curriculum is designed to facilitate the development of self-control, 

emotional understanding, positive self-esteem, relationships, and interpersonal problem-solving skills 

in children. For 20 to 30 minutes a day, 3 to 5 days per week, teachers engage students in lessons that 

involve discussion, role-playing, storytelling, and worksheets. Teachers are also encouraged to model 

and reinforce social skills throughout the day. 

 Schoolwide activities include a PATHS party at the end of the year and visual artifacts that illustrate 

aspects of social and emotional competence. 

 Parents are engaged in the program through parent newsletters and homework assignments that are 

to be completed with their children. 

 Teachers, principals, and mental health staff participate in an initial 2-day training that presents the 

concepts of program implementation, lessons on integrating PATHS activities with traditional 

instruction, and methods for applying and transferring social skills to children over the entire school 

day and in the home. Program consultants provide weekly technical assistance on program 

implementation to teachers, a recommended practice in the PATHS program. Consultants hold 

biweekly calls with teachers to discuss implementation and challenges. In addition, a national 

PATHS-certified trainer is available to each intervention school 1 day in the fall and spring. A 1-day 

summer meeting for principals, teachers, and school staff after the first year of implementation 

provides additional technical assistance, addresses feedback on program implementation, and engages 

school staff in planning for the following year. 
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

348

Table 6.1. Promoting Alternative Thinking Strategies

Panel 1: General characteristics

Target population 

 Universal

Program components

 Peer: In class

 Parent: Contact and involvement

 Classroom: Lessons

 Schoolwide: Planned events and program artifacts

 Community: None or not major focus

 Training: Pretraining and ongoing

Level of integration

 Add-on curriculum and schoolwide activities

Flexibility

 Manualized: Scripted curriculum and guidebook

 Adaptability: Less adaptable

See notes at end of table.

 

Panel 2: Description of instruction and strategies

Classroom

 Lessons

 Who delivers: Teacher

 Activities and tools: Direct instruction, storytelling, discussion, role-playing, utilization of artifacts (e.g.,

 posters, 

 turtle puppet), worksheets

 Content: Emotion understanding and control, behavior regulation, problem solving, making friends

 Frequency: 20 to 30 minutes per day, 3 to 5 days per week

 Strategies

 Who delivers: Teacher

 Activities and tools: Modeling of skills

 Frequency: Daily

Supplement to classroom

 Parent newsletter and engagement in homework

Schoolwide activities 

 End-of-year PATHS party and program artifacts

See notes at end of table.
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

349

Table 6.1. Promoting Alternative Thinking Strategies—Continued

Panel 3: Professional development

Pre-implementation

 Teachers

 Content: Training on concepts, curriculum implementation, how to integrate activities with traditional 

 instruction

 Duration: 2 days

 Other

 Content: Principal and school mental health staff, same as teacher training

 Duration: 2 days

Ongoing consultation

 Teachers

 Content: Technical assistance/consultation on implementation; individual meetings with each teacher; 

 attendance at grade-level meetings; planning for following year

 Duration: Weekly consultations; 2-day summer meeting

 Other

 Content: Principal and school staff; model lessons, team teaching, general feedback; planning for 

 following year

 Duration: Biweekly calls; 2-day summer meeting

See notes at end of table.

Panel 4: Activities for SACD goals

Violence prevention and peace promotion  Risk prevention and health promotion 

Social and emotional development  Civic responsibility and community service 

Character education  Behavior management 

Tolerance and diversity 

See notes at end of table.

Panel 5: SACD outcomes addressed 

Engagement with Learning  Empathy 

Academic Competence and Motivation  Positive School Orientation 

Altruistic Behavior  Negative School Orientation 

Positive Social Behavior  Student Afraid at School

Problem Behavior  Victimization at School

Self-Efficacy for Peer Interactions  Feelings of Safety 

Normative Beliefs About Aggression  Student Support for Teachers

NOTE: Abbreviations are

 : Activity or outcome addressed

 Blank cell: Activity or outcome not addressed

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

350 

Sample and Random Assignment 

The Children’s Institute research team recruited a total of 10 public elementary schools, representing 1 school 

district in Minnesota (2 schools) and 2 school districts in New York (8 schools). The 10 schools were 

randomly assigned to treatment and control conditions prior to the fall 2004 data collection.50 A two-step 

process was used, stratified by the three participating school districts. First, a computer-generated pairwise 

matching algorithm developed by Mathematica Policy Research, Inc. (MPR) was used to identify the best 

pairwise matches across the 10 schools based on variables identified by the Children’s Institute research team. 

The variables used in the pairwise matching for the 8 New York schools were as follows: 

(a) enrollment; (b) percentage of White students; (c) percentage of limited-English-proficient students; 

(d) student-teacher ratio; (e) percentage of students eligible for free lunch; (f) percentage of students eligible 

for reduced-price lunch; (g) percentage of students passing English language arts; and (h) percentage of 

students passing math. For the 2 Minnesota schools, the following variables were used: (a) enrollment; 

(b) percentage of White students; (c) number of suspensions per 100 students; (d) percentage of limitedEnglish-proficient students; (e) percentage of students eligible for free lunch; (f) percentage of students 

eligible for reduced-price lunch; (g) stability; (h) language test scores; and (i) math test scores. Second, using 

the flip of a coin, 1 school in each matched pair was assigned to either the intervention or control condition. 

Five schools received the PATHS program and 5 schools acted as control schools and continued to 

implement the social and character development activities that constituted their standard practice. 

Assignment to treatment or control condition was at the school level and therefore limited the risk of 

contamination between treatment and control classrooms. 

The original student sample (the cohort of students in the third grade in the 10 schools in fall 2004) 

numbered 786 students (377 treatment and 409 control). Table 6.2 documents the change in the sample over 

the three spring follow-up data collection periods. Over time, new entrants to the cohort became a larger 

percentage of the sample, eventually making up 30 percent of the sample by the spring of Year 3. There were 

no statistically significant differences between the treatment and control groups in the number of new 

entrants. The percentage of the sample made up of the original cohort further declined as students left the 

schools. By Year 3, approximately 31 percent of the original sample had left. 

 

50 In Year 2 (fall 2005), the Children’s Institute research team recruited four more schools, two assigned to treatment and 

two assigned to control, which were followed to the end of the study (spring 2007). A description of this second cohort 

and all relevant findings can be found in appendix A. The data from this second cohort are not included in the analyses 

and results reported in this chapter. 
Table 6.2. Sample—PATHS 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Characteristic Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 School sample size 10 5 5 10 5 5 10 5 5 10 5 5

 Student sample size 786 377 409 783 374 409 778 373 405 778 378 400

Stayers † † † 747 356 391 636 311 325 543 263 280

New entrants † † † 36 18 18 142 62 80 235 115 120

 New entrants as a percent of 

 spring enrollment † † † 4.6 4.8 4.4 18.3 16.6 19.8 30.2 30.4 30.0

Total leavers (from original cohort) † † † 39 21 18 150 66 84 243 114 129

 Leavers as percent of fall 2004

 enrollment † † † 5.0 5.6 4.4 19.1 17.5 20.5 30.9 30.2 31.5

Number of students per school 

 (mean) 79 75 82 78 75 82 78 75 81 78 76 80

Range of number of students per 

 school 64-108 64-83 65-108 63-105 63-82 64-105 56-102 68-85 56-102 49-97 66-90 49-97

† Not applicable.

NOTE: No statistically significant differences were found between values for treatment and control groups.

SOURCE: The Social and Character Development (SACD) Research Program.

351

Chapter 6. Promoting Alternative Thinking Strategies (PATHS)
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

352

Implementation 

Training 

The intervention teachers received 2 days of program implementation training by PATHS-certified trainers 

prior to the beginning of the school year (table 6.1, panel 3). Teachers had access to ongoing program 

implementation support throughout the school year. Three program consultants went to schools at least once 

a week to visit classrooms, teach lessons, team teach, provide feedback to teachers, answer questions, and 

help with implementation as needed. These consultants were trained by the program developers. Each 

consultant was selected because of prior teaching experience and an ability to form close working 

relationships with teachers, students, and administrators at the schools. One consultant had extensive 

classroom experience, a second had both classroom and administrative experience, and the third had 

classroom experience as well as experience as a volunteer working in an urban community. In addition, a 

program trainer was made available to each intervention school during the initial 2-day training and during the 

1-day summer meeting to answer questions, demonstrate lessons, provide feedback, and help with planning 

for the upcoming year. In addition, the lead national PATHS trainer participated in the biweekly calls 

approximately 50 percent of the time and made occasional visits to the schools during each school year. 

Data Collection 

MPR collected the multiprogram child, teacher, and school data at the Minnesota site, and MPR’s 

subcontractor, Decision Information Resources, Inc. (DIR), collected the multiprogram child, teacher, and 

school data at the New York site. Table 6.3 shows the school year milestones and dates of implementation 

for the New York/Minnesota site. Data were collected in the fall and spring of the first 2 years and the spring 

of Year 3. The fall 2004 multiprogram data collection began on October 18, 2004, and ended on October 28, 

2004. The average time frame from the beginning of program implementation to the beginning of fall data 

collection was 6 weeks. As a result, initial data collection took place after implementation of the PATHS 

program began. Therefore, these data provide a measure of the students, teachers, and schools near the 

beginning of the school year, at a time when the PATHS program had been operating for a relatively short 

period of time. The spring data collection window was from March 28, 2005, to April 28, 2005. PATHS had 

been implemented for 29 weeks at the time of the spring data collection and for 22 weeks from the end of the 

fall data collection. Year 2 followed a similar pattern, with implementation occurring at the start of the school 

year, fall data collection occurring 5 weeks later, and spring data collection occurring 20 weeks after fall data 

collection (and 29 weeks after the start of implementation). In spring 2007, data collection occurred 29 weeks 

after the start of implementation. Data collection took between 3 to 5 weeks at each collection point. 
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

353

Table 6.3. Data collection dates—PATHS 

Data collection schedule

Year 1

(Fall 3rd grade)

Year 1

(Spring 3rd grade)

Year 2 

(Fall 4th grade)

Year 2 

(Spring 4th grade)

Year 3 

(Spring 5th grade)

 School sample size 10 10 10 10 10

School-year dates 

 First day of school 9/7/04 † 9/7/05 † 9/6/061

 Start of implementation 9/8/04 † First day † First day

 Last day of school † 6/23/05 † 6/22/06 6/6/07; 

6/21/072

Data collection 

 Start 10/18/04 3/28/05 10/11/05 3/20/06 3/26/07

 End 10/28/04 4/28/05 10/27/05 4/11/06 4/27/07

Calendar weeks from 

 program implementation 

 to start of fall 2004 

 data collection 6 † † † †

Calendar weeks from 

 start of school to start of 

 fall 2004 data collection 6 † 5 † †

Calendar weeks from 

 end of fall data collection 

 to start of spring data 

 collection † 22 † 20 †

Calendar weeks from 

 program implementation 

 to start of spring data 

 collection † 29 † 29 291

† Not applicable.

1 Schools started between 9/5/06 and 9/7/06. In calculations of time from the start of the school year to the start of spring 2007 data 

collection 9/6/06 was used as the first day of the school year.

2 Two schools ended on 6/6/07 and eight ended on 6/21/07.

SOURCE: The Social and Character Development (SACD) Research Program.

Consent Rates, Completion Rates, and Percentage of Sample With Data 

The actual number of student, primary caregiver, and teacher reports available for analysis was smaller than 

the number in the sample because consent and completion rates were below 100 percent. Primary caregivers 

had to provide consent before children could complete the Child Report, before their child’s teacher could 

complete the Teacher Report on Student, and before they themselves completed the Primary Caregiver 

Report. In the spring of Years 1 and 3, significantly more primary caregivers in the treatment group than in 

the control group provided consent for their children and themselves. Teachers also had to provide consent 

before completing the Teacher Report on Classroom and School. In Year 1, significantly more teachers in the 

treatment group gave consent for both the fall and spring data collection. 

Of those with consent, not all completed their respective reports. Table 6.4 shows the consent rates, 

completion rates, and percentages of sample with data for each of the four reports over the 3 years. For the 
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

354

Child Report and two teacher reports, completion rates ranged from 89 percent to 100 percent, with one 

statistically significant difference in completion rates between treatment and control groups for the Child 

Report in the spring of 2005. In this case, the treatment group had fewer completed reports than the control 

group (94% versus 98%). For the Teacher Report on Student, completion rates ranged from 96 percent to 

100 percent. For the Primary Caregiver Report, the completion rates dropped over time from 95 percent to 

79 percent. There was a statistically significant difference for the Primary Caregiver Report in Year 2, with 

more caregivers in the treatment group completing reports compared to the control group (83% versus 79%). 

The percentages of the sample with Child Report data ranged from 50 percent to 60 percent over the 3 years, 

with a statistically significant difference in Year 3, when more students in the treatment group had data than 

did students in the control group (60% versus 52%). The percentages of students with information from the 

Teacher Report on Student ranged from 53 percent to 62 percent, with statistically significant differences 

between treatment and control conditions in Years 1 and 3, both favoring the treatment group (62% versus 

55% and 60% versus 53%). The percentages of students with data from the Primary Caregiver Report ranged 

from 42 percent to 56 percent. In general, this percentage declined over time, although the highest rates were 

seen in the spring of 2005. In Year 2, the treatment group had significantly more Primary Caregiver Report 

data than did the control group (50% versus 42%). The percentages of teachers with data from the Teacher 

Report on Classroom and School ranged from 70 percent to 92 percent, with a statistically significant 

difference in Year 1, when treatment teachers had more data than control teachers in both the fall (89% 

versus 70%) and spring (92% versus 70%). 
Table 6.4. Consent rates, completion rates, and percentage of sample with data from each report—PATHS 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 Student sample size 786 377 409 783 374 409 778 373 405 778 378 400

Child Report (percent)

 Primary caregiver consent rate 58.4 60.5 56.5 58.5 62.3* 55.0 57.6 60.9 54.6 56.6 60.3* 53.0

 Student completion rate 89.3 89.9 88.7 95.9 94.0* 97.8 96.9 96.9 96.8 98.0 98.7 97.2

 Students with data152.2 54.4 50.1 56.1 58.6 53.8 55.8 59.0 52.8 55.4 59.5* 51.5

Primary Caregiver Report (percent)

 Primary caregiver consent rate 56.6 58.9 54.5 57.3 61.2* 53.8 56.0 59.5 52.8 55.1 58.7* 51.8

 Primary caregiver completion rate 94.6 95.0 94.2 84.2 86.5 81.8 81.2 83.3* 79.0 78.8 77.0 80.7

 Primary caregivers with data153.6 56.0 51.3 48.3 52.9 44.0 45.5 49.6** 41.7 43.4 45.2 41.8

Teacher Report on Student (percent)

 Primary caregiver consent rate258.4 60.5 56.5 58.5 62.3* 55.0 57.6 60.9 54.6 56.6 60.3* 53.0

 Teacher completion rate 96.5 96.9 96.1 99.8 99.6 100.0 100.0 100.0 100.0 99.5 100.0 99.1

 Students with data156.4 58.6 54.3 58.4 62.0* 55.0 57.6 60.9 54.6 56.3 60.3* 52.5

Teacher Report on Classroom and School

 (3rd- to 5th-grade teachers) (percent)

 Teacher consent rate 83.8 93.7** 75.3 83.8 95.2** 74.0 84.3 90.0 79.7 93.2 90.0 95.9

 Teacher completion rate 93.9 94.9 92.7 95.6 96.7 94.4 99.1 100.0 98.3 89.5 90.7 88.6

 Teachers with data178.7 88.9** 69.9 80.1 92.1** 69.9 83.6 90.0 78.4 83.5 81.7 84.9

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

1 Calculated as consent rate x completion rate.

2 The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

355

Chapter 6. Promoting Alternative Thinking Strategies (PATHS)
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

356

Responses from students in the original cohort (stayers) and new entrants in the PATHS sample were 

examined to investigate possible differences between the two groups in consent rates, completion rates, and 

the percentages of the sample with data that might affect outcome data (table 6.5). In Years 1 and 3, consent 

rates for stayers were significantly higher than for new entrants (by 16 to 26 percentage points). These 

differences in consent rates led to similar differences between stayers and new entrants in percentages of the 

sample with data on all three reports (by 14 to 26 percentage points). In Year 2, stayers had a significantly 

higher percentage of Primary Caregiver report data than new entrants (by 10 percentage points). 
Table 6.5. Consent rates, completion rates, and percentage of sample with data: Stayers versus new entrants—PATHS 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total Stayers

New 

entrants Total Stayers

New

entrants Total Stayers

New 

entrants

 Student sample size 783 747 36 778 636 142 778 543 235

Child Report (percent)

 Primary caregiver consent rate 58.5 59.7** 33.3 57.6 59.1 50.7 56.6 61.7*** 44.7

 Student completion rate 95.9 95.7 100.0 96.9 96.3 100.0 98.0 97.9 98.1

 Students with data156.1 57.2** 33.3 55.8 56.9 50.7 55.4 60.4*** 43.8

Primary Caregiver Report (percent)

 Primary caregiver consent rate 57.3 58.5** 33.3 56.0 57.5 49.3 55.1 60.0*** 43.8

 Primary caregiver completion rate 84.2 84.4 75.0 81.2 82.2 75.7 79.3 79.8 77.7

 Primary caregivers with data148.3 49.4** 25.0 45.5 47.3* 37.3 43.7 47.9** 34.0

Teacher Report on Student (percent)

 Primary caregiver consent rate258.5 59.7** 33.3 57.6 59.1 50.7 56.6 61.7*** 44.7

 Teacher completion rate 99.8 99.8 100.0 100.0 100.0 100.0 99.5 99.7 99.0

 Students with data158.4 59.6** 33.3 57.6 59.1 50.7 56.3 61.5*** 44.3

* Stayers significantly different from new entrants at the .05 level.

** Stayers significantly different from new entrants at the .01 level.

*** Stayers significantly different from new entrants at the .001 level.

1 Calculated as consent rate x completion rate.

2 The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

357

Chapter 6. Promoting Alternative Thinking Strategies (PATHS)
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

358

Fidelity of Implementation

Each year, PATHS’ five treatment schools were independently rated for quantity and quality of program 

implementation by two raters from the research team. The global measure of fidelity for the multisite study 

was used; inter-rater reliability was calculated using Cronbach’s alpha (Year 1 = 0.78, Year 2 = 0.93, and Year 

3 = 0.86). The ratings were used to generate a single consensus rating with which schools were identified as 

being high or low in implementation fidelity. In Year 1, three treatment schools were identified as having high 

fidelity, and in Years 2 and 3, two treatment schools were identified as having high fidelity. Cohen’s kappa 

was used as the measure of agreement when identifying schools as high fidelity, and it equaled 0.55 in Year 1, 

0.17 in Year 2, and 0.29 in Year 3. 

Initial Characteristics 

This section examines the initial characteristics of the students, teachers, and schools participating in the 

evaluation of the PATHS program. These characteristics were collected from students who were enrolled in 

the third grade at the study schools in fall 2004, as well as from their primary caregivers and third-grade 

teachers. Third-, fourth-, and fifth-grade teachers and principals in the study schools also provided 

information about activities related to social and character development in these schools. Documenting the 

characteristics of students, teachers, and schools and initial measures of key outcomes at a point before the 

interventions had been operating for an extended period helped to determine whether the random 

assignment of schools to treatment and control status produced treatment and control groups with similar 

distributions of observed characteristics. As noted in the following discussion, there were 10 significant 

differences in the observed characteristics, 8 of which (out of 62 comparisons, with 3 expected to be 

significant by chance) reflected differences between the treatment and control students, teachers, and schools 

in the use of SACD activities in the classroom and school. 

Characteristics of Children, Their Families, and Communities 

There were no significant differences between the treatment and control groups in the observed student, 

caregiver, and community characteristics (table 6.6). For students, the mean age was 8.1 years. The sample 

contained roughly equal percentages of girls (58%) and boys (43%). The sample was ethnically diverse, with 

White non-Hispanic students making up 56 percent of the sample, Black non-Hispanic students making up 

22 percent of the sample, and Hispanic students making up 12 percent of the sample. 

The sample was also diverse in its levels of family income, education levels of primary caregivers of the 

children in the sample, and family situation. For the total sample, 34 percent of children lived in a household 

where the income was 135 percent of the federal poverty level or lower, which is the income threshold for 

eligibility for free school meals. Thirteen percent of primary caregivers had not completed high school. Nearly 

two-thirds of the children (66%) lived with both their mother and their father. There were no significant 

differences between the treatment and control groups in these characteristics. 

The mean values of the outcome measures for children’s behavior and attitudes as reported by the primary 

caregiver, child, and teacher at initial data collection in fall 2004 are shown in table 6.7. There was 1 

significant difference between the treatment and control groups (out of 18 comparisons): children in the 

treatment group reported lower Self-Efficacy for Peer Interaction (2.8 versus 3.0). 
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

359

Table 6.6. Initial characteristics of children, their families, and communities—PATHS 

Characteristic Total Treatment Control

 Student sample size 421 211 210

Student demographics

 Gender (percent)

 Male 42.5 40.5 44.6

 Female 57.5 59.5 55.4

 Race/ethnicity (percent)

 White (non-Hispanic) 56.1 52.7 59.4

 Black (non-Hispanic) 22.3 23.9 20.7

 Hispanic 11.5 12.3 10.7

 Other 10.1 11.0 9.2

 Age (in years) (mean) 8.1 8.1 8.1

Primary caregiver and family characteristics

 Primary caregiver’s age (in years) (mean) 37.1 36.6 37.6

 Primary caregiver’s race/ethnicity (percent)

 White (non-Hispanic) 63.2 60.9 65.5

 Black (non-Hispanic) 20.6 22.9 18.4

 Hispanic 9.9 9.7 10.2

 Other 6.3 6.6 6.0

 Primary caregiver’s education (percent)

 Did not complete high school 12.7 13.0 12.4

 Completed high school or equivalent 18.5 19.0 17.9

 Some college 37.1 39.4 34.8

 Bachelor’s or higher degree 31.7 28.5 34.8

 Primary caregiver’s employment (percent)

 Full-time 51.4 56.7 46.0

 Other 48.6 43.3 54.0

 Primary caregiver’s marital status (percent)

 Married 62.9 63.3 62.5

 Other 37.1 36.7 37.5

 Students who live in one household (percent) 93.7 92.8 94.5

 Number of individuals in household (mean) 4.6 4.6 4.6

 Primary caregiver’s relationship to child (percent)

 Mother (stepmother) 86.1 86.0 86.2

 Father (stepfather) 10.3 10.5 10.1

 Other relative/nonrelative 3.6 3.5 3.7

See notes at end of table.
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

360

Table 6.6. Initial characteristics of children, their families, and communities—PATHS—

Continued

Table 6.6. 

Characteristic Total Treatment Control

 Student lives with (percent)

 Mother (stepmother) and father (stepfather) 66.3 65.7 66.9

 Mother (stepmother) only; father (stepfather) not present 29.2 28.9 29.4

 Father (stepfather) only; mother (stepmother) not present 1.6 ‡ ‡

 Other relative/nonrelative, parents not present 2.9 3.1 2.7

 Highest education of anyone in household (percent)

 Did not complete high school 8.2 8.9 7.4

 Completed high school or equivalent 18.4 17.4 19.4

 Some college 33.8 39.4 28.2

 Bachelor’s or higher degree 39.7 34.3 45.0

 Total household income (percent)

 Less than $20,000 29.1 25.9 32.2

 $20,000 to $39,999 18.3 22.1 14.4

 $40,000 to $59,999 17.2 23.3 11.0

 $60,000 or more 35.5 28.7 42.3

 Income-to-poverty-threshold ratio—Below 135 percent (percent) 33.6 31.7 35.5

 Income-to-poverty-threshold ratio—135 to 185 percent (percent) 14.1 16.6 11.6

 Income-to-poverty-threshold ratio—Above 185 percent (percent) 52.3 51.7 52.8

 Alabama Parenting Questionnaire—Poor Monitoring and 

 Supervision Subscale (mean) 1.1 1.1 1.1

 Alabama Parenting Questionnaire—Positive Parenting

 Subscale (mean) 3.5 3.5 3.5

 Confusion, Hubbub, and Order Scale (mean) 2.2 2.2 2.2

Community characteristics (mean)

 Community Risks Scale 1.4 1.3 1.5

 Community Resources Scale 3.0 2.9 3.0

 Child-Centered Social Control Scale 3.1 3.0 3.2

‡ Reporting standards not met. Values suppressed to protect confidentiality.

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

361

Table 6.7. Mean scores and standard deviations for initial outcome measures of sample—

PATHS 

Total Treatment Control

Outcome measure–Report Range Mean SD Mean SD Mean SD

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 1-4 2.9 0.6 2.8* 0.6 3.0 0.6

 Normative Beliefs About Aggression–CR 1-4 1.2 0.3 1.2 0.4 1.1 0.3

 Empathy–CR 1-4 2.4 0.4 2.4 0.4 2.5 0.4

Behavior Domain

 Altruistic Behavior–CR 0-3 1.3 0.9 1.4 0.9 1.3 0.8

 Altruistic Behavior–TRS 1-4 1.2 0.3 1.2 0.3 1.3 0.3

 Altruistic Behavior–PCR 1-4 2.2 0.7 2.3 0.7 2.2 0.8

 Positive Social Behavior–TRS 1-4 3.0 0.7 3.0 0.7 3.1 0.6

 Positive Social Behavior–PCR 1-4 3.0 0.5 3.0 0.5 3.0 0.5

 Problem Behavior–CR 0-3 0.2 0.3 0.2 0.4 0.2 0.3

 Problem Behavior–TRS 1-4 1.3 0.5 1.4 0.5 1.3 0.4

 Problem Behavior–PCR 1-4 1.6 0.3 1.6 0.3 1.6 0.3

 ADHD-Related Behavior–TRS 1-4 1.8 0.7 1.8 0.7 1.7 0.7

Academics Domain

 Academic Competence and Motivation–TRS 1-5 3.0 0.9 2.9 0.9 3.2 0.9

 Engagement with Learning–CR 1-4 3.8 0.5 3.7 0.5 3.8 0.4

Perceptions of School Climate Domain

 Positive School Orientation–CR 1-4 3.3 0.6 3.3 0.6 3.4 0.5

 Negative School Orientation–CR 1-4 1.7 0.6 1.8 0.6 1.7 0.6

 Student Afraid at School–CR 1-4 2.4 0.9 2.5 0.9 2.3 0.9

 Victimization at School–CR 0-3 0.7 0.8 0.7 0.8 0.7 0.8

 Student sample size—PCR 421 211 210

 Student sample size—CR 410 205 205

 Student sample size—TRS 443 221 222

* Treatment group significantly different from control group at the .05 level.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

 SD: Standard deviation

Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall 

means. Statistical tests were conducted using regressions that included program indicators to account for the sample design and 

adjusted for clustering at the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.

Characteristics of Teachers and Schools 

Table 6.8 describes the third-, fourth-, and fifth-grade teachers at the study schools. The majority were White 

non-Hispanic (88%). Most of the teachers were female (74%), and there was a statistically significant 

difference in the percentages of male teachers, with fewer males in the treatment group than in the control 

group (17% versus 35%). Overall, the teachers had an average of 11.5 years of total teaching experience. The 

majority (77%) held an advanced or specialist degree. 
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

362

Data regarding school characteristics were drawn from the Common Core of Data in order to compare 

treatment and control schools. There were no significant differences between the two groups of schools in 

terms of student composition (race/ethnicity and school lunch eligibility), number of students enrolled, 

number of full-time teachers, Title I status, or number of years the principal had been at the school (see table 

6.9). In addition, there were no significant differences between treatment and control schools in terms of 

location (urban, suburban, or rural) or lowest and highest grade offered (these data are not shown in a table). 

Table 6.8. Initial characteristics of teachers in sample—PATHS 

Characteristic Total Treatment Control

 Teacher sample size 107 56 51

Gender (percent)

 Male 26.1 17.4* 34.8

 Female 73.9 82.6 65.2

Race/ethnicity (percent)

 White (non-Hispanic) 87.7 88.7 86.8

 Other 12.3 11.3 13.2

Number of years teaching experience (mean) 11.5 10.0 13.0

Number of years teaching experience in this school (mean) 7.7 6.5 8.9

Type of teaching certificate (percent)

 Regular state certificate or advanced professional certificate 81.0 82.4 79.6

 Other 19.0 17.6 20.4

Education (percent)

 Bachelor’s degree 23.0 19.9 26.2

 Advanced degree/other 77.0 80.1 73.8

* Treatment group significantly different from control group at the .05 level.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and 

overall means. Statistical tests were conducted using regressions that included program indicators to account for the sample design 

and adjusted for clustering at the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

363

Table 6.9. Initial characteristics of schools in sample—PATHS 

Characteristic Total Treatment Control

 School sample size 10 5 5

Student race/ethnicity (percent)

 White (non-Hispanic) 46.0 48.5 43.6

 Black (non-Hispanic) 35.6 31.1 40.2

 Hispanic 13.1 14.4 11.8

 Other 5.2 5.9 4.4

Students eligible for free or reduced-price lunch (percent) 56.4 53.7 59.0

Number of students enrolled (mean) 538.0 460.6 634.8

Number of full-time teachers (mean) 45.1 42.8 47.4

Title I status (percent)

 Title I eligible school 60.0 60.0 60.0

 Schoolwide Title I 66.7 66.7 66.7

Number of years principal has been at this school (mean) 5.8 4.0 7.6

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: NCES Common Core of Data (2003-04), the Social and Character Development (SACD) Research Program.

In the Teacher Report on Classroom and School, teachers reported on nine dimensions of school 

environment (these data are not shown in a table): feelings of safety, adequacy of resources, student support, 

freedom to teach as desired, affiliation with and ties to colleagues, innovation regarding new approaches to 

teaching, professional interest, participatory decisionmaking, and work pressure. There were no statistically 

significant differences between treatment and control schools in these reports. 

The Level of SACD in the Schools Near the Beginning of the Study 

During the initial data collection period, principals and teachers reported on the SACD activities used in the 

schools and classrooms, the availability of SACD materials, and the professional development provided on 

SACD. Table 6.10 shows that the majority of the school principals reported activities to promote six social 

and character development goals: violence prevention and peace promotion (90%), social and emotional 

development (89%), character education (90%), tolerance and diversity (78%), risk prevention and health 

promotion (60%), and civic responsibility and community service (80%). In addition, 90 percent of the 

principals reported activities directed toward behavior management. There were no statistically significant 

differences between the treatment group and the control group in the percentages reported by principals, 

although this may be due to the small principal sample size. Teachers’ reports of the use of these activities in 

their classrooms ranged from 53 percent to 83 percent, and there were no significant differences between 

treatment and control teachers in their use. With respect to the use of schoolwide activities, 65 percent to 89 

percent of teachers reported that their schools used such activities. There were no significant differences 

between treatment and control teachers in reports of their use of these specific SACD activities. 
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

364

Table 6.10. Principal and teacher initial reports on use of SACD programs or activities in 

sample—PATHS 

SACD program or activity Total Treatment Control

 Principal sample size 10 5 5

 Teacher sample size 107 56 51

Principals reporting that school had programs or activities

 to promote the following SACD goals (percent)

 Violence prevention and peace promotion 90.0 100.0 80.0

 Social and emotional development 88.9 100.0 75.0

 Character education 90.0 100.0 80.0

 Tolerance and diversity 77.8 80.0 75.0

 Risk prevention and health promotion 60.0 60.0 60.0

 Civic responsibility and community service 80.0 100.0 60.0

 Behavior management 90.0 100.0 80.0

 None of the above ‡ ‡ ‡

Teachers reporting on using programs or activities in their

 class to promote the following SACD goals (percent)

 Violence prevention and peace promotion 74.1 78.9 69.2

 Social and emotional development 79.8 83.1 76.5

 Character education 83.1 81.2 85.0

 Tolerance and diversity 62.7 59.8 65.7

 Risk prevention and health promotion 52.9 45.6 60.3

 Civic responsibility and community service 70.1 65.9 74.3

 Behavior management 80.4 85.6 75.2

 None of the above 3.1 ‡ ‡

Teachers reporting schoolwide use of the following 

 activities to promote SACD (percent)

 Morning announcements or videos 83.7 77.3 90.0

 School assemblies 89.4 90.2 88.6

 School newspapers or bulletins 72.1 62.0 82.2

 Special school days 65.2 62.9 67.5

 Special events 81.0 84.5 77.4

 Other activities 16.6 13.9 19.2

‡ Reporting standards not met. Values suppressed to protect confidentiality.

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within each of the programs and to each program across programs, were used in producing the 

treatment, control, and overall means. Statistical tests were conducted using regressions that included program indicators to 

account for the sample design and adjusted for clustering at the school level. Sample size may differ for some outcomes due to 

nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

365

Teachers reported using a broad range of teaching materials to support SACD activities (table 6.11), including 

teacher guides (55%), student materials (51%), instructional aids (28%), giveaways (43%), and children’s 

literature (65%). There were statistically significant differences in the use of two types of SACD materials (out 

of seven comparisons made); more treatment than control teachers used teacher guides (67% versus 43%) 

and fewer treatment than control teachers used giveaways (33% versus 54%). 

Teachers also reported using a wide variety of teaching strategies (table 6.11). All teachers reported using any 

of the 20 strategies asked about, and teachers used an average of 11.7 of the strategies. There was a significant 

difference in the use of role-playing (1 of 20 strategies), with treatment teachers using this strategy more often 

than control teachers (81% versus 44%). 
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

366

Table 6.11. Teacher initial reports on use of SACD materials and classroom strategies in 

sample—PATHS 

SACD material and classroom strategy Total Treatment Control

 Teacher sample size 107 56 51

Teachers using the following materials in conjunction with 

 social and character development activities (percent)

 Teacher guides (manuals, curricula) 55.0 67.3* 42.8

 Student materials (workbooks, worksheets) 51.1 55.6 46.7

 Instructional aids (games, software, videos) 28.0 30.5 25.6

 Giveaways (bookmarks, stickers) 43.4 33.0** 53.9

 Children’s literature 65.3 66.7 63.9

 Other types of materials 15.8 6.5 25.0

 Do not use any of the materials listed above 10.7 9.8 11.7

Teachers using any of the strategies listed below to promote 

 social and character development in the classroom (percent) 100.0 100.0 100.0

Number of strategies (listed below) used by teachers to promote 

 social and character development in the classroom (mean) 11.7 12.1 11.2

Teachers using each of the following strategies to promote 

 social and character development (percent)

 Role-playing 62.5 80.8** 44.2

 Cooperative learning 94.3 95.7 93.0

 Peer group discussions 90.0 93.0 87.0

 Direct instruction of social and character development 82.3 91.5 73.1

 Skill training 49.6 52.6 46.6

 Incorporating social and character development into 

 academic curriculum 67.6 75.2 59.9

 Parent training 5.1 ‡ ‡

 Parent/community involvement in program development or delivery 31.3 33.8 28.8

 Mentoring 25.3 24.2 26.4

 Good behavior notes sent home daily or weekly 78.5 79.7 77.4

 Presenting role models 78.5 76.0 80.9

 Targeted story reading or writing on SACD themes 73.3 78.2 68.4

 Peer mediation 35.7 36.3 35.0

 Honor roll for positive behavior 37.3 39.3 35.4

 Pledges or recitations on social and character development themes 40.7 29.7 51.7

 Guided visualization 44.0 43.8 44.2

 Student-led/student-assisted instruction 51.6 59.4 43.8

 Journaling 68.6 69.0 68.2

 Time out for negative behavior 76.1 81.2 70.9

 Daily or weekly rewards for positive behavior 93.1 92.0 94.1

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

367

Principals and teachers reported on participation in and amount of SACD training and staff development 

provided over the previous 12 months (table 6.12). Principals reported higher participation rates (89% versus 

80%) and more hours of training (10.3 versus 9.5) than did teachers. There were significant differences 

between treatment and control principals on the number of hours of SACD training reported (15 versus 5) 

and between treatment and control teachers on both the percentages reporting participation in SACD 

training (92% versus 68%) and the number of hours of SACD training received (13 versus 6). Treatment 

teachers also reported more training than control teachers on two of the seven specific SACD goals: social 

and emotional development (64.8% versus 15.3%) and character education (53.9% versus 17.2%). 

Table 6.12. Principal and teacher initial reports on use of SACD materials and classroom 

strategies in sample—PATHS 

SACD professional development Total Treatment Control

 Principal sample size 10 5 5

 Teacher sample size 107 56 51

Principals reporting that staff participated in social and character 

 development training within the past year (percent) 88.9 100.0 75.0

Teachers reporting participation in social and character 

 development training within the past 12 months (percent) 79.6 91.7* 67.5

Number of hours of social and character development training 

 principals report were provided to each staff person last year (mean) 10.3 14.8* 4.8

Number of hours of social and character development training 

 teachers report receiving during the past 12 months (mean) 9.5 12.7** 6.2

Teachers reporting receiving training in the past 12 months 

 in the following areas (percent)

 Violence prevention and peace promotion 36.7 30.1 43.4

 Social and emotional development 40.0 64.8** 15.3

 Character education 35.5 53.9* 17.2

 Tolerance and diversity 18.5 22.0 14.9

 Risk prevention and health promotion 11.2 13.6 8.9

 Civic responsibility and community service 8.7 4.8 12.6

 Behavior management 31.4 36.2 26.5

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.

The data on the initial level of SACD activity emphasize that the control condition was a ―standard practice‖ 

control. Standard practice at the control schools included using SACD activities, materials, and practices, 

along with professional development, at rates and in types and amounts similar to the treatment schools. For 

example, the percentages of teachers who reported using programs or activities to promote specific SACD 

goals ranged from 46 percent to 86 percent in the treatment schools and from 60 percent to 85 percent in the 

control schools. However, the 8 significant differences between the treatment and control conditions in the 

use of SACD activities was more than expected by chance (3 out of 62 comparisons), and in 7 of these cases 
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

368

the differences favored the treatment group. This may reflect the fact that program implementation and 

program training for staff started before initial data collection. 

Impacts on Use of SACD Activities 

The introduction of the formal PATHS program would be expected to increase the use of SACD activities in 

the treatment schools in comparison to the control schools. The analysis of this impact is based on the 

Teacher Report on Classroom and School (TRCS). Every spring, third-, fourth-, and fifth-grade teachers 

provided information through the TRCS about the social and character development activities they used in 

their classrooms. Specifically, information from the TRCS was used to determine the difference between 

treatment and control teachers in these areas: 

1. the use of SACD activities in their classrooms overall and by SACD goal; 

2. the use of materials and strategies to implement the SACD activities within classrooms and within 

the entire school; 

3. the use of staff development to support the teachers; and 

4. staff attitudes toward the use of SACD activities. 

TRCS consent and completion rates (table 6.4) led to 70 percent to 92 percent of all teachers having data for 

the 3 years, with a significantly greater percentage of treatment than control teachers providing data in Year 1. 

To estimate intervention impacts for each of the outcome measures, testing of the statistical significance of 

the differences in means was used. Preliminary analysis indicated little or no gains in precision from using 

covariates. Before testing the mean differences, the data were weighted such that each school received equal 

weight. Standard errors of the impact estimates account for the clustering of teachers within schools. In 

addition, to estimate the impact on the individual outcome measures, a set of heuristics was applied to 

determine whether each outcome domain was statistically significant after adjustments were made for the 

multiple tests conducted. 

Use of Activities 

The percentages of control teachers who reported using any SACD activities in their classrooms ranged from 

90 percent to 99 percent over the 3 years (table 6.13, panel 1). For the six individual SACD goals, the ranges 

varied from 71 percent to 86 percent in Year 1, 51 percent to 74 percent in Year 2, and 59 percent to 84 

percent in Year 3. Control teachers’ use of behavior management activities ranged from 89 percent to 93 

percent over this period. The percentages of control teachers who reported using any SACD activities in their 

classroom for at least 1 hour per week (panel 2) ranged from 46 percent to 83 percent over the 3 years. For 

the six individual SACD goals, the ranges varied from 10 percent to 25 percent in Year 1, 11 percent to 40 

percent in Year 2, and 6 percent to 37 percent in Year 3. Control teachers’ use of behavior management 

activities ranged from 71 percent to 73 percent over this period. 

For teachers’ reported use of any SACD activity (panels 1 and 2), 48 comparisons were made, with 2 expected 

to be significant by chance. There were 14 significant impacts on teachers’ reported use of specific SACD 

activities in all 3 years. In Year 1, PATHS had significant impacts on engagement in activities for at least 1 

hour per week: violence prevention and peace promotion (impact = 32 percentage points), social and 

emotional development (impact = 48 percentage points), character education (impact = 28 percentage 

points), and tolerance and diversity (impact = 20 percentage points). In Year 2, there were significant impacts 

both overall and for 1 hour per week on social and emotional development (impact = 38 and 36 percentage 

points), character education (impact = 23 and 38 percentage points), and tolerance and diversity (impact = 33 

and 27 percentage points). There was also a significant impact on violence prevention and peace promotion 

overall (impact = 23 percentage points). In Year 3, PATHS had an impact on social and emotional 

development overall (impact = 24 percentage points) and for at least 1 hour per week (impact = 27 
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

369

percentage points). Civic responsibility and community service activities for at least 1 hour per week were also 

significantly impacted (impact = 18 percentage points). After the heuristics were applied, the domain for 

engagement in SACD activities showed that PATHS had statistically significant impacts in Years 1 and 2. 

For teachers’ reported use of any named SACD activity (panels 3 and 4), 42 comparisons were made, with 2 

expected to be significant by chance. Six of the 12 impact estimates in Year 1, 9 of the 12 in Year 2, and 9 of 

the 12 in Year 3 were statistically significant. In all 3 years, the use of named activities to promote violence 

prevention and peace promotion (impact = 41, 50, and 36 percentage points), social and emotional 

development (impact = 59, 64, and 53 percentage points), character education (impact = 49, 65, and 40 

percentage points), tolerance and diversity (impact = 42, 31, and 34 percentage points), and any named 

activity (impact = 44, 37, and 33 percentage points) were significantly impacted. In Year 2, named activities to 

promote civic responsibility and community service were significantly impacted (impact = 13 percentage 

points). The use of named activities for at least 1 hour per week showed significant impacts on the use of any 

named activity in Year 1 (impact = 42 percentage points). In Years 2 and 3, PATHS had significant impact on 

activities related to social and emotional development (impact = 43 and 44 percentage points), character 

education (impact = 43 and 41 percentage points), and tolerance and diversity (impact = 19 and 24 

percentage points). In Years 1 and 3, there were significant impacts on the use of any named activity for at 

least 1 hour per week (impact = 42 and 40 percentage points). PATHS had a significant impact on the 

domain of engagement in named SACD activities in all 3 years. 
Table 6.13. Impacts on teacher-reported SACD classroom activities—PATHS 

Panel 1: Engagement in any activities to promote SACD goals1

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 60 56 55 57 50 61

Violence prevention and peace promotion

 (percent) 85.0 78.9 6.1 0.554 85.9* 62.4 23.4 0.043 88.2 81.2 7.1 0.446

Social and emotional development 

 (percent) 89.0 71.5 17.4 0.122 93.5* 55.1 38.4 0.006 98.0* 74.5 23.5 0.020

Character education (percent) 87.7 85.8 1.9 0.861 95.5* 72.2 23.3 0.015 95.9^ 83.6 12.3 0.091

Tolerance and diversity (percent) 78.7 70.8 7.9 0.420 88.9* 56.2 32.7 0.025 86.5 72.3 14.2 0.142

Risk prevention and health promotion

 (percent) 68.5 70.5 -2.0 0.916 74.3 73.6 0.8 0.967 80.5 80.4 0.1 0.995

Civic responsibility and community service

 (percent) 65.4 71.7 -6.3 0.566 67.3 51.2 16.1 0.199 66.4 59.4 7.0 0.623

Any SACD goal (percent) 94.3 93.7 0.5 0.922 97.0 90.1 6.8 0.185 100.0 98.5 1.5 †

Behavior management (percent) 88.5 93.1 -4.6 0.472 95.1 89.5 5.6 0.463 79.8 89.4 -9.6 0.193

See notes at end of table.

370

Chapter 6. Promoting Alternative Thinking Strategies (PATHS)
Table 6.13. Impacts on teacher-reported SACD classroom activities—PATHS—Continued 

Panel 2: Engagement in any activities to promote SACD goals for at least 1 hour per week

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 60 56 55 57 50 61

Violence prevention and peace promotion

 (percent) 48.0* 15.9 32.0 0.020 41.6 22.5 19.1 0.121 45.8 33.2 12.6 0.343

Social and emotional development

 (percent) 60.0* 12.5 47.5 0.003 57.1* 21.3 35.8 0.023 59.2* 32.0 27.2 0.012

Character education (percent) 52.6* 25.1 27.5 0.036 63.9* 25.5 38.4 0.001 60.2^ 37.1 23.1 0.078

Tolerance and diversity (percent) 40.9* 21.3 19.6 0.031 43.3* 16.2 27.1 0.043 32.2 25.9 6.3 0.596

Risk prevention and health promotion

 (percent) 32.8 18.5 14.3 0.288 34.5 39.9 -5.4 0.699 30.8 27.2 3.6 0.778

Civic responsibility and community service

 (percent) 19.8 10.0 9.8 0.374 17.1 10.9 6.2 0.239 24.0* 5.6 18.4 0.039

Any SACD goal (percent) 74.7 45.6 29.1 0.053 86.7 82.8 3.9 0.663 82.4 77.0 5.4 0.599

Behavior management (percent) 68.7 72.7 -4.0 0.614 82.2 71.0 11.2 0.442 58.9 70.8 -11.9 0.305

See notes at end of table.

371

Chapter 6. Promoting Alternative Thinking Strategies (PATHS)
Table 6.13. Impacts on teacher-reported SACD classroom activities—PATHS—Continued 

Panel 3: Engagement in activities to promote SACD goals linked to named SACD programs2

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 60 56 55 57 50 61

Violence prevention and peace promotion 

 (percent) 54.4* 13.8 40.6 0.009 60.8* 11.0 49.8 0.001 65.7* 30.2 35.5 0.034

Social and emotional development

 (percent) 65.1* 6.0 59.1 0.003 78.1* 14.2 64.0 0.000 75.9* 22.6 53.3 0.000

Character education (percent) 54.5* 5.3 49.2 0.002 70.2* 5.1 65.2 0.000 63.8* 24.0 39.8 0.033

Tolerance and diversity (percent) ‡* ‡ 42.1 0.006 41.8* 11.0 30.8 0.003 42.5* 8.4 34.1 0.016

Risk prevention and health promotion

 (percent) 25.1 16.2 8.9 0.552 23.3 30.2 -7.0 0.569 25.6 30.4 -4.8 0.698

Civic responsibility and community service

 (percent) 6.6 0.0 6.6 † ‡* ‡ 12.9 0.035 17.8 5.6 12.2 0.229

Any named activity (percent) 66.3* 22.8 43.5 0.014 83.3* 45.9 37.4 0.003 82.3* 49.0 33.3 0.021

See notes at end of table.

 

372

Chapter 6. Promoting Alternative Thinking Strategies (PATHS)
Table 6.13. Impacts on teacher-reported SACD classroom activities—PATHS—Continued 

Panel 4: Engagement in activities to promote SACD goals linked to named SACD programs for at least 1 hour per week

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value

 Teacher sample size 60 56 55 57 50 61

Violence prevention and peace promotion

 (percent) 37.5^ 11.7 25.8 0.100 34.5^ 7.7 26.8 0.062 40.8^ 17.2 23.6 0.094

Social and emotional development 

(percent) 47.3 0.0 47.3 † ‡* ‡ 43.4 0.003 50.6* 6.9 43.7 0.000

Character education (percent) 36.4 0.0 36.4 † ‡* ‡ 43.4 0.004 52.4* 11.7 40.7 0.015

Tolerance and diversity (percent) 29.8 0.0 29.8 † ‡* ‡ 19.2 0.016 ‡* ‡ 24.2 0.015

Risk prevention and health promotion

 (percent) 16.7 7.8 8.9 0.499 10.9 21.7 -10.7 0.388 18.9 7.9 11.0 0.305

Civic responsibility and community service

 (percent) 5.5 0.0 5.5 † 4.6 0.0 4.6 † 12.2 0.0 12.2 †

Any named activity (percent) 52.2* 10.2 42.0 0.040 57.3^ 28.4 28.8 0.060 61.9* 22.2 39.6 0.031

† Not applicable.

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 In Year 1, the omnibus impact for all the outcomes measured together was positive and statistically significant on the basis of a multivariate statistical test. In Years 1 and 2, at least 

one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) procedure to adjust 

significance levels downward to account for the multiple testing of impacts.

2 In all 3 years at least half of the impacts were positive and statistically significant and no impact was negative and statistically significant on the basis of univariate statistical tests, and 

at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) procedure to 

adjust significance levels downward to account for the multiple testing of impacts.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

373

Chapter 6. Promoting Alternative Thinking Strategies (PATHS)
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

374

Use of Materials and Strategies 

For use of materials and strategies to support SACD goals, 87 comparisons were made, with 5 expected to be 

significant by chance. Six significant impacts were found on treatment teachers’ use of materials and 

strategies. In Year 1, more treatment teachers used role-playing (impact = 34 percentage points). In Year 2, 

more treatment teachers used instructional aids (impact = 30 percentage points), children’s literature (impact 

= 30 percentage points), role-playing (impact = 29 percentage points), and direct instruction of SACD 

(impact = 20 percentage points), and these teachers also reported using a greater average number of strategies 

(by 1.6 strategies on average). The PATHS impact on the domain of materials and strategies was not 

statistically significant in any of the 3 years. 
Table 6.14. Impacts on use of SACD classroom materials and teaching strategies—PATHS 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD material and teaching strategy

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 60 56 55 57 50 61

Use of SACD materials (percent)

 Teacher guides (manuals, curricula) 74.8 64.7 10.1 0.409 83.9 56.9 27.0 0.116 89.5 79.0 10.5 0.239

 Student materials (workbooks or sheets) 64.1 66.6 -2.5 0.776 72.0 70.6 1.4 0.891 77.4 72.1 5.3 0.679

 Instructional aids (games, software, videos) 50.0 49.0 1.0 0.952 61.9* 32.4 29.5 0.022 54.2^ 37.2 17.0 0.088

 Giveaways (bookmarks, stickers) 32.6^ 57.8 -25.2 0.064 41.9 37.7 4.2 0.613 35.4 46.8 -11.4 0.344

 Children’s literature 73.3 63.2 10.1 0.432 75.2* 45.5 29.7 0.012 68.1 61.9 6.2 0.471

 Other types of materials 10.4 5.8 4.5 0.468 5.2 7.0 -1.9 0.755 ‡ ‡ 11.6 0.19

 Did not use any of these materials 9.3 7.5 1.8 0.765 ‡ ‡ -9.2 0.125 ‡ ‡ -5.7 0.254

Use of teaching strategies (percent)

 Role-playing 91.6* 58.1 33.5 0.024 90.4* 61.0 29.4 0.002 92.2 84.6 7.6 0.282

 Cooperative learning 98.8 100.0 -1.3 † 100.0 100.0 0.0 † 100.0 100.0 0.0 †

 Peer group discussions 92.1 89.5 2.6 0.590 100.0 91.4 8.6 † 93.5 100.0 -6.5 †

 Direct instruction of SACD 87.9 85.0 2.8 0.757 98.5* 79.0 19.5 0.021 100.0 98.1 1.9 †

 Skill training 54.8 45.3 9.5 0.324 67.9^ 44.1 23.8 0.058 87.8 81.0 6.8 0.360

 Incorporating SACD into academic 

 curriculum 84.5 80.6 3.9 0.572 89.5^ 77.2 12.3 0.072 95.9 92.4 3.4 0.413

 Parent training 5.6 9.6 -4.1 0.388 7.7 8.4 -0.8 0.902 25.7 25.2 0.5 0.963

 Parent/community involvement 20.7 38.4 -17.7 0.212 19.5 37.5 -18.0 0.138 40.3 36.6 3.7 0.725

 Mentoring 29.0 40.4 -11.5 0.289 24.8 24.0 0.8 0.932 54.1 54.4 -0.3 0.961

 Good behavior notes sent home daily 

 or weekly 82.0 73.9 8.1 0.344 81.3 74.5 6.9 0.642 93.1 98.1 -5.1 0.268

 Presenting role models 86.6 81.2 5.4 0.363 75.7 71.6 4.0 0.663 94.1 87.8 6.3 0.184

See notes at end of table.

375

Chapter 6. Promoting Alternative Thinking Strategies (PATHS)
Table 6.14. Impacts on use of SACD classroom materials and teaching strategies—PATHS—Continued

 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD material and teaching strategy

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

Use of teaching strategies (percent)—

 Continued

 Targeted story reading or writing on social

 and character development themes 88.9 84.0 4.9 0.599 90.8 84.6 6.3 0.173 98.2 93.2 5.0 0.212

 Peer mediation 50.4 49.1 1.3 0.935 59.1 46.0 13.1 0.523 67.3 63.7 3.6 0.749

 Honor roll for positive behavior 40.7 40.1 0.5 0.974 45.3 28.3 17.0 0.253 69.3 55.4 13.9 0.341

 Pledges or recitations on social and 

 character development themes 21.8 56.7 -34.9 0.173 47.6 55.7 -8.1 0.722 54.5 75.6 -21.0 0.265

 Guided visualization 58.6 52.4 6.2 0.500 67.3 46.8 20.4 0.116 62.6 56.0 6.6 0.447

 Student-led/student-assisted instruction 54.9 54.4 0.5 0.952 70.3 57.0 13.4 0.185 78.5 70.0 8.5 0.377

 Journaling 75.9 77.9 -1.9 0.853 85.7 79.2 6.5 0.261 84.0 85.5 -1.5 0.778

 Time out for negative behavior 86.9 73.5 13.4 0.239 82.9 75.4 7.6 0.479 89.5 80.2 9.4 0.410

 Daily or weekly rewards for positive 

 behavior 92.2 91.3 0.9 0.889 86.9 95.7 -8.8 0.273 96.6 96.7 -0.1 0.990

 Any strategy 100.0 100.0 0.0 † 100.0 100.0 0.0 † 100.0 100.0 0.0 †

 Number of strategies (mean) 12.9 12.7 0.2 0.715 13.9* 12.3 1.6 0.039 15.6 15.2 0.4 0.477

† Not applicable.

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

376

Chapter 6. Promoting Alternative Thinking Strategies (PATHS)
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

377

Regarding the use of schoolwide strategies, 18 comparisons were made between treatment and control 

teacher reports, with 1 expected to be significant by chance. There was 1 statistically significant difference 

between reports from treatment and control teachers on their use of schoolwide strategies in Year 1 (these 

data are not shown in a table), with fewer treatment teachers reporting use of special events than control 

teachers (impact = -17 percentage points). The overall impact of the PATHS program on the domain for use 

of schoolwide strategies in named SACD activities was not significant in any year. 

Participation in Professional Development 

Regarding reported participation in professional development, 27 comparisons were made over 3 years, with 

2 expected to be significant by chance. In Year 1, the intervention had a statistically significant effect on 

treatment teachers’ participation in professional development, with more treatment teachers than control 

teachers reporting SACD training in the past 12 months (impact = 32 percentage points) and reporting more 

hours of training (by 10.6 hours on average). In terms of specific SACD goals, more treatment teachers 

reported training in violence prevention and peace promotion (impact = 24 percentage points), social and 

emotional development (impact = 47 percentage points), and character education (impact = 46 percentage 

points). In Year 2, PATHS had an impact on social and emotional development training (impact = 17 

percentage points). There was a significant impact on the domain in Year 1. 
Table 6.15. Impacts on teacher-reported SACD professional development—PATHS 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD professional development1

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 60 56 55 57 50 61

SACD training in past 12 months (percent) 91.9* 60.1 31.8 0.020 76.4 66.9 9.5 0.294 63.8 70.0 -6.2 0.461

Hours of SACD training (mean) 14.0* 3.4 10.6 0.000 5.2 4.4 0.7 0.465 7.1 5.5 1.6 0.516

Training by goal (percent)

 Violence prevention and peace promotion 44.8* 20.6 24.2 0.034 18.7 14.6 4.0 0.737 20.3 11.5 8.8 0.432

 Social and emotional development 74.2* 27.5 46.7 0.005 37.6* 20.8 16.9 0.025 29.8 28.2 1.6 0.926

 Character education 67.1* 21.2 45.9 0.002 45.7^ 25.7 20.0 0.056 33.9 16.1 17.8 0.112

 Tolerance and diversity 16.3 17.0 -0.7 0.945 31.4 41.6 -10.2 0.257 41.3 40.1 1.2 0.937

 Risk prevention and health promotion 15.5 23.0 -7.5 0.524 15.5 9.4 6.1 0.536 8.5 18.1 -9.6 0.231

 Civic responsibility and community service 6.5 5.4 1.1 0.829 6.1 7.1 -1.0 0.811 ‡ ‡ -1.6 0.727

 Behavior management 40.5 32.7 7.8 0.697 30.6 27.7 2.9 0.842 22.5 30.7 -8.2 0.554

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 In Year 1, at least half of the impacts were positive and statistically significant and no impact was negative and statistically significant on the basis of univariate statistical tests, and at 

least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) procedure to 

adjust significance levels downward to account for the multiple testing of impacts.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

378

Chapter 6. Promoting Alternative Thinking Strategies (PATHS)
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

379

Attitudes and Practices 

Teachers reported on their enthusiasm for SACD efforts in their schools (these data are not shown in a table) 

by indicating enthusiasm, cooperation, or open dislike. They also reported on the SACD practices of teachers 

and staff members in their schools (these data are not shown in a table). These practices included modeling 

positive character and behavior traits with students and fellow teachers, involving students in making 

decisions, giving students a voice in school governance, the school encouraging parent involvement in 

children’s social and character development, and using developmentally appropriate discipline strategies 

rather than punishment for misbehavior. Twenty-seven comparisons were made over 3 years, with 1 expected 

to be significant by chance. There were no statistically significant impacts on teachers’ enthusiasm for SACD 

efforts in their schools in any of the years, nor were there significant differences in the treatment and control 

teachers’ reports of the overall use of practices conducive to students’ social and character development. 

PATHS had no impact on the domain in any year. 

Year-by-Year Impacts on Students and Perceptions of School Climate 

The primary research question for the PATHS evaluation was this: 

What is the average effect of the PATHS program on children’s social and emotional competence, 

behavior, academic achievement, and perceptions of school climate? 

The first approach to answering this question was to examine the year-by-year impacts of PATHS on these 

student and school climate outcomes over the 3 years as the students progressed from third through fifth 

grades. 

Equation (2) (described in chapter 1) was estimated to provide the impacts of the PATHS program on the 20 

outcome measures using data from the 10 treatment and control schools. For the PATHS evaluation, 

equation (2) excluded the program fixed effects (p) and included program-specific covariates and random 

school effects covariates. Table 6.16 lists the covariates used with outcomes from each report in the PATHS 

analysis. 
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

380

Table 6.16. Covariates used with outcomes from each report for analysis—PATHS 

Potential covariate

CR 

outcome

PCR 

outcome

TRS 

outcome

TRCS 

outcome

 Total number 18 27 27 5

Child-reported

 Female   

 Hispanic   

 Black (non-Hispanic)   

 Other ethnicity   

 Age in years   

 Scales

 Afraid at School 

 Altruistic Behavior

 Empathy

 Engagement with Learning

 Negative School Orientation  

 Normative Beliefs About Aggression

 Sense of School as a Community

 Problem Behavior 

 Self-Efficacy for Peer Interactions   

 Victimization at School 

Primary caregiver-reported

 Age in years 

 Completed high school or equivalent

 Some college

 Bachelor’s or higher degree

 Highest level of education in household

 Completed high school or equivalent   

 Some college   

 Bachelor’s or higher degree   

 Mother present in home life  

 Mother and father present 

 Respondent someone other than mother or father 

 Number of people in household

 Household Income: $20,000 to $40,000   

 Household Income: $40,000 to $60,000   

 Household Income: More than $60,000   

 Income-to-poverty-threshold ratio: Below 135 percent 

 Income-to-poverty-threshold ratio: 135-185 percent 

 Full-time employment   

 Part-time employment   

See notes at end of table.
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

381

Table 6.16. Covariates used with outcomes from each report for analysis—PATHS—Continued

Potential covariate

CR 

outcome

PCR 

outcome

TRS 

outcome

TRCS 

outcome

 Parental scales

 APQ-Poor Monitoring and Supervision Subscale   

 APQ-Positive Parenting Subscale

 Child-Centered Social Control 

 Confusion, Hubbub, and Order

 Community Resources 

 Community Risk  

 Parent and Teacher Involvement  

 Child scales

 Altruistic Behavior

 Positive Social Behavior 

 Problem Behavior 

Teacher-reported 

 Female 

 Hispanic 

 Black (non-Hispanic) 

 Other ethnicity 

 Total teaching experience 

 Total experience in current school

 Regular certificate

 Other certificate

 Highest degree–bachelor’s

 Child scales

 Academic Competence and Motivation

 ADHD-Related Behavior  

 Altruistic Behavior 

 Positive Social Behavior

 Problem Behavior  

 Parent and Teacher Involvement   

NOTES: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

 APQ: Alabama Parenting Questionnaire

 Covariate used

 Blank cell: Covariate not used

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

382

To assess the statistical power of the program-level impact estimates, minimum detectable impacts in effect 

size units (MDES) for each outcome measure were calculated for the PATHS evaluation (table 6.17). MDES 

represent the smallest impacts in effect size (standard deviation) units that can be detected with a high 

probability (80%). MDES are primarily a function of study sample sizes, the degrees of freedom available for 

statistical tests, and design effects due to clustering (Schochet 2005). For the PATHS evaluation, the MDES 

ranged from 0.137 to 0.709 for the child-level outcomes based on the Child, Primary Caregiver, and Teacher 

Report on Student and from 0.652 to 1.040 for the school climate outcomes based on the Teacher Report on 

Classroom and School. In general, the MDES for the school climate outcomes were larger than those for the 

child-level outcomes. 
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

383

Table 6.17. Adjusted minimum detectable effect sizes for impact evaluation—PATHS 

Outcome measure–Report Year 1 Year 2 Year 3

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 0.160 0.210 0.179

 Normative Beliefs About Aggression–CR 0.137 0.201 0.209

 Empathy–CR 0.252 0.171 0.239

Behavior Domain

 Altruistic Behavior–CR 0.230 0.159 0.228

 Altruistic Behavior–PCR 0.239 0.154 0.210

 Altruistic Behavior–TRS 0.423 0.282 0.573

 Positive Social Behavior–PCR 0.147 0.218 0.156

 Positive Social Behavior–TRS 0.188 0.467 0.395

 Problem Behavior–CR 0.137 0.138 0.213

 Problem Behavior–PCR 0.194 0.173 0.156

 Problem Behavior–TRS 0.181 0.419 0.230

 ADHD-Related Behavior–TRS 0.172 0.440 0.230

Academics Domain

 Engagement with Learning–CR 0.140 0.178 0.138

 Academic Competence and Motivation–TRS 0.217 0.169 0.312

Perceptions of School Climate Domain

 Positive School Orientation–CR 0.196 0.611 0.709

 Negative School Orientation–CR 0.256 0.332 0.246

 Student Afraid at School–CR 0.174 0.142 0.249

 Victimization at School–CR 0.164 0.151 0.252

 Feelings of Safety–TRCS 0.652 1.040 1.034

 Student Support for Teachers–TRCS 0.792 0.937 0.913

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The minimum detectable effect (MDE) formula used in the calculations is as follows:

 

 

 

 

 

 

 

 

 

 

 

 

 

 

where sT and sC are the number of treatment and comparison schools; nT and nC are the average number of students per 

classroom; ρ1 is the intraclass correlation (ICC) at the school level; and factor(df) is a constant that depends on the number of 

degrees of freedom (df) available for analysis (and is 2.802 for the pooled analysis).

SOURCE: The Social and Character Development (SACD) Research Program.

 
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

384

Table 6.18 provides the estimates of the PATHS program’s impacts on each of the 20 outcome measures 

over each of the 3 years (60 impacts in total, with 3 expected to be statistically significant by chance). Of the 

60 results, none were statistically significant. In Year 3, substantively important but nonsignificant detrimental 

impacts were found for Altruistic Behavior (Teacher Report on Student, effect size [ES] = -0.31) and Feelings 

of Safety (Teacher Report on Classroom and School, ES = -0.29). Application of the heuristics to adjust for 

multiple comparisons within each outcome domain indicates a statistically significant detrimental effect on 

the Social and Emotional Competence domain in Year 2 and a statistically significant detrimental effect on 

the Academics domain in Year 3. 
Table 6.18. Impacts on child and school outcomes—PATHS 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value

Social and Emotional Competence Domain1

 Self-Efficacy for Peer Interactions–CR (+) 3.06 3.06 0.01 0.953 3.18 3.28 -0.16 0.261 3.21 3.30 -0.16 0.238

 Normative Beliefs About Aggression–CR (-) 1.21 1.23 -0.05 0.606 1.26 1.37 -0.17 0.214 1.41 1.42 -0.01 0.951

 Empathy–CR (+) 2.38 2.34 0.10 0.493 2.21 2.14 0.13 0.245 2.09 2.11 -0.03 0.851

Behavior Domain

 Altruistic Behavior–CR (+) 1.25 1.13 0.15 0.320 1.00 0.96 0.06 0.635 0.98 0.98 0.00 0.991

 Altruistic Behavior–PCR (+) 2.15 2.13 0.03 0.827 2.06 2.19 -0.16 0.140 2.12 2.16 -0.06 0.665

 Altruistic Behavior–TRS (+) 1.26 1.25 0.02 0.932 1.22 1.18 0.17 0.395 1.27 1.37 -0.31° 0.485

 Positive Social Behavior–PCR (+) 2.96 3.03 -0.13 0.159 3.05 3.06 -0.01 0.953 3.04 3.09 -0.08 0.413

 Positive Social Behavior–TRS (+) 3.01 2.98 0.05 0.590 3.14 3.04 0.15 0.503 3.18 3.14 0.06 0.778

 Problem Behavior–CR (-) 0.22 0.27 -0.10 0.220 0.27 0.30 -0.06 0.547 0.43 0.37 0.12 0.385

 Problem Behavior–PCR (-) 1.57 1.54 0.10 0.383 1.53 1.53 0.00 0.987 1.53 1.55 -0.04 0.714

 Problem Behavior–TRS (-) 1.44 1.42 0.03 0.677 1.42 1.42 0.01 0.957 1.36 1.43 -0.14 0.229

 ADHD-Related Behavior–TRS (-) 1.75 1.79 -0.07 0.417 1.62 1.70 -0.13 0.574 1.55 1.69 -0.22 0.126

Academics Domain2

 Engagement with Learning–CR (+) 3.73 3.70 0.05 0.657 3.70 3.69 0.01 0.954 3.61 3.66 -0.11 0.296

 Academic Competence and Motivation–

 TRS (+) 3.13 3.11 0.02 0.831 2.95 3.05 -0.09 0.272 3.11 3.03 0.08 0.621

See notes at end of table.

385

Chapter 6. Promoting Alternative Thinking Strategies (PATHS)
Table 6.18. Impacts on child and school outcomes—PATHS—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 2.96 2.98 -0.03 0.826 2.79 2.67 0.15 0.640 2.66 2.66 0.00 0.997

 Negative School Orientation–CR (-) 1.76 1.81 -0.08 0.587 1.87 1.90 -0.05 0.765 1.93 2.04 -0.19 0.214

 Student Afraid at School–CR (-) 2.17 2.18 -0.01 0.945 2.08 2.09 -0.01 0.909 2.09 2.28 -0.22 0.180

 Victimization at School–CR (-) 0.68 0.65 0.04 0.681 0.65 0.56 0.12 0.306 0.71 0.57 0.20 0.284

 Feelings of Safety–TRCS (+) 3.28 3.30 -0.02 0.932 3.36 3.18 0.20 0.615 3.19 3.40 -0.29° 0.582

 Student Support for Teachers–TRCS (+) 3.54 3.53 0.01 0.980 3.52 3.46 0.07 0.838 3.57 3.70 -0.20 0.631

o Substantive (but nonsignificant at .05 level) effect size of ≥ .25 or ≤ -.25.

1 Impact on domain found statistically significant and detrimental in Year 2 based on the fourth heuristic in which the statistical model used to estimate impacts on the individual 

outcomes was re-estimated using a composite of all the outcome variables under a domain. The domain was found significant if the impact on the composite was significant. The 

composite was formed by standardizing each outcome variable using its standard deviation, combining the values of the outcome variables, and taking the average of the final value.

2 Impact on domain found statistically significant and detrimental in Year 3 based on the fourth heuristic in which the statistical model used to estimate impacts on the individual 

outcomes was re-estimated using a composite of all the outcome variables under a domain. The domain was found significant if the impact on the composite was significant. The 

composite was formed by standardizing each outcome variable using its standard deviation, combining the values of the outcome variables, and taking the average of the final value.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

No findings were found statistically significant at or below the .05 level. The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated 

using regression models where each program and school within a program was weighted equally. The standard errors of all estimates account for design effects due to unequal 

weighting and the clustering of students within schools. See table 1.5 for information about the measured used to create the outcome variables. The effect size was calculated by 

dividing the estimated impact by the standard deviation of the outcome measure for the control group.

SOURCE: The Social and Character Development (SACD) Research Program.

386

Chapter 6. Promoting Alternative Thinking Strategies (PATHS)
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

387

Impacts on Child Outcomes Over Time 

The PATHS program’s impacts on the child outcomes over time were estimated using growth curve models 

by examining treatment and control group differences in the trajectories of student outcomes during the 

follow-up period while accounting for clustering at the school level. The growth curve models are estimated 

using a three-level hierarchical linear model, where Level 1 corresponds to time, Level 2 to students, and 

Level 3 to schools. 

Table 6.19 provides the estimates of the PATHS program’s impacts on the growth in student outcome 

measures over the 3 years. The estimated impacts ranged in effect size units (absolute value) from 0.01 to 

0.12. One of the 18 estimated PATHS intervention impacts on the trajectories of child outcomes was 

statistically significant (no more than expected by chance); this was Academic Competence and Motivation 

(Teacher Report on Student, ES = 0.08). 
Table 6.19. Impacts on growth of child outcomes—PATHS 

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2Treatment Control

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 2.94 0.13 0.14 -0.01 -0.02 0.03 0.693

 Normative Beliefs About Aggression–CR (-) 1.17 0.11 0.08 0.03 0.07 0.04 0.512

 Empathy–CR (+) 2.44 -0.18 -0.13 -0.05 -0.12 0.04 0.179

Behavior Domain

 Altruistic Behavior–CR (+) 1.35 -0.22 -0.13 -0.09 -0.07 0.06 0.187

 Altruistic Behavior–PCR (+) 2.24 -0.05 -0.05 -0.01 -0.01 0.03 0.846

 Altruistic Behavior–TRS (+) 1.19 0.03 0.05 -0.02 -0.05 0.07 0.803

 Positive Social Behavior–PCR (+) 2.96 0.05 0.04 0.01 0.01 0.02 0.698

 Positive Social Behavior–TRS (+) 2.99 0.07 0.04 0.03 0.04 0.07 0.642

 Problem Behavior–CR (-) 0.17 0.09 0.08 0.01 0.02 0.04 0.812

 Problem Behavior–PCR (-) 1.60 -0.03 -0.02 -0.01 -0.01 0.01 0.672

 Problem Behavior–TRS (-) 1.38 0.00 0.01 0.00 -0.01 0.02 0.852

 ADHD-Related Behavior–TRS (-) 1.81 -0.09 -0.06 -0.03 -0.03 0.04 0.535

Academics Domain

 Engagement with Learning–CR (+) 3.76 -0.06 -0.04 -0.02 -0.04 0.03 0.422

 Academic Competence and Motivation–TRS (+) 2.93 0.09* -0.02 0.11 0.08 0.05 0.048

See notes at end of table.

388

Chapter 6. Promoting Alternative Thinking Strategies (PATHS)
Table 6.19. Impacts on growth of child outcomes—PATHS—Continued

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2Treatment Control

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 3.24 -0.25 -0.22 -0.04 -0.05 0.06 0.552

 Negative School Orientation–CR (-) 1.70 0.13 0.08 0.05 0.06 0.05 0.258

 Student Afraid at School–CR (-) 2.31 -0.07 -0.08 0.01 0.01 0.05 0.790

 Victimization at School–CR (-) 0.69 -0.04 -0.02 -0.02 -0.01 0.04 0.719

* Treatment group significantly different from control group at the .05 level.

1 Pertains to the estimated slope of the outcome for the treatment or control groups.

2 The average score at implementation is calculated across treatment and control groups, using regression models for adjustment on covariates. 

3 Estimated difference between the slope of the treatment and control groups.

4 Effect size: the slope of the treatment group minus the slope of the control group divided by the standard deviation of the outcome for the program’s control group (the standard 

deviation is calculated without accounting for school-level clustering or regression adjustments).

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using HLM 6.06. Sample weights were used in all analyses to give (1) 

each school equal weight in each program (within each time period) and (2) each time period equal weight within the analysis. See table 1.5 for information about the measures used 

to create the outcome variables.

SOURCE: The Social and Character Development (SACD) Research Program.

389

Chapter 6. Promoting Alternative Thinking Strategies (PATHS)
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

390

Summary 

As part of the Social and Character Development (SACD) initiative, researchers at the New York/Minnesota 

site implemented and evaluated the PATHS program. This program focused on promoting children’s selfcontrol, emotional understanding, positive self-esteem, relationships, and interpersonal problem-solving skills. 

Ten public schools, eight in New York and two in Minnesota, were recruited by the Children’s Institute 

research team and randomly assigned to treatment and control conditions to determine the impact of the 

PATHS program on social and character development activities in the schools and on the child outcome 

domains of Social and Emotional Competence, Behavior, Academics, and Perceptions of School Climate. 

Analyses of the initial characteristics of the sample (students, caregivers, communities, teachers, and schools) 

indicated that randomization to treatment and control status produced groups that were relatively similar at 

the start of the study (with 2 out of 84 comparisons statistically significantly different, fewer than would be 

expected by chance). The data on the initial level of SACD activity led to two findings. First, treatment 

teachers reported greater use of and training in SACD activities than control teachers, and they did so more 

often than would be expected by chance (8 out of 62 comparisons, with 3 expected significant by chance). 

There are two potential causes for this finding, and the analysis cannot be used to determine whether the 

reason for such a difference was that the two groups did differ on their initial use of SACD activities (i.e., that 

randomization did not create similar treatment and control groups) or the fact that the training of all 

treatment teachers and the implementation of the PATHS program began before the initial data were 

collected (by 6 weeks) influenced the teacher reports. Because it is likely (though unproven) that the training 

and implementation affected the teacher reports, these data were not considered appropriate for use as a 

baseline measure of SACD activities and training in the treatment schools. 

Second, these data indicate that the control condition for the SACD project was not a ―no treatment‖ control 

but a ―standard practice‖ control. Because the control teachers were not affected by the implementation of 

the SACD programs before data collection, their reports reflected standard practice in the control schools. 

Standard practice at the control schools included reports of 60 percent to 85 percent of teachers using any 

SACD activities, 88 percent of teachers using specific materials in conjunction with these activities, 100 

percent of teachers using at least one of the specified instructional strategies, and 68 percent of teachers 

participating in SACD training over the past 12 months. 

Analyses of the impacts of the PATHS program on the level of SACD activities in the schools revealed 

impacts on the use of such activities (39 out of 90) and related materials and strategies (7 out of 87, 1 

negative) across the 3 years, and use of more professional development activities for treatment teachers in the 

first 2 years (6 out of 27). These same measures in the control schools across the 3 years of the study 

confirmed that use of these activities in the control schools constituted their standard practice. 

Of the 20 child-level outcomes representing the four domains of Social and Emotional Competence, 

Behavior, Academics, and Perceptions of School Climate assessed in each of the 3 years of the study (a total 

of 60 results), none were statistically significant. A growth curve analysis was used to analyze the change over 

time in these same outcomes between initial data collection and the final outcome data collection at the end 

of the study. One of the 18 child-level outcomes assessed showed a significant beneficial impact of the 

PATHS program. 

The SACD evaluation did not find evidence to support the hypothesis that PATHS had beneficial impacts on 

students’ social and character development. Such results could be caused by the inability of the program to 

cause such change, possibly because the theory of action for the program is incomplete or the activities to 

carry out that theory are not effective. 

However, these results may also be due to the inability of the evaluation to observe such a change due to the 

control condition, the level of nonparticipation, or the sample size. The control schools continued using their 

standard SACD activities, and these turned out to be high in quantity and broad in scope. While PATHS had 
Chapter 6. Promoting Alternative Thinking Strategies (PATHS) 

391

a significant positive impact on the amount and types of SACD activities, the resulting difference in the 

amount of SACD activities between the treatment and control schools may not have been large enough to 

cause significant differences in the student outcomes. Second, nearly one-half of the students in the sample 

universe did not take part because of nonconsent or noncompletion of the surveys. As a determination could 

not be made as to whether the students not taking part significantly differed from those who did take part, 

the evaluation’s results are valid only for the students who took part. If the students not taking part were 

different, and if they would have responded better to PATHS than to the SACD activities occurring in the 

control schools, then the evaluation could have underestimated the program’s impact. In addition, when 

looking at the percentages of the sample with data across the 3 study years and the four data sources (table 

6.4) there were 6 comparisons out of 16 in which the percentages of data were greater for the treatment 

reports than for the control reports (all statistically significant differences at the .05 level). How these may 

have affected the results is not clear, as the differences between those with data and those without cannot be 

determined. Third, the sample size of 10 schools and the resulting higher MDES compared to those for the 

multiprogram evaluation may have reduced the likelihood of detecting statistically significant effects. 

However, it should be noted that 67 percent of the MDES for the 60 outcomes used in the year-by-year 

analysis were below 0.25 (42% were below 0.20). In addition, only 2 of the 60 outcomes were found to be 

substantively important, and both had detrimental impacts. 
This page is intentionally blank. 
393

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and 

Resolution) 

New York University

(New York City Site)

Intervention 

Researchers at New York University and Fordham University (New York City site) evaluated The 4Rs Program 

(Reading, Writing, Respect, and Resolution) (4Rs) program as implemented by staff developers from the 

Morningside Center for Teaching Social Responsibility, the community-based nonprofit organization that 

developed the program. The 4Rs program integrates the teaching and promotion of social and emotional 

skills and conflict resolution lessons into the language arts curriculum and aims to create a caring classroom 

community. Table 7.1 describes the 4Rs program’s general characteristics (panel 1), the types of instruction 

and strategies used (panel 2), the professional development provided for those implementing the program 

(panel 3), and the social and character development activities (panel 4) and outcomes (panel 5) addressed by 

the program. The program includes two primary components: 

 The 4Rs consists of a comprehensive, seven-unit, 21- to 35-lesson, literacy-based curriculum on 

conflict resolution and social-emotional learning. Lesson units focus on building community, 

understanding and dealing with feelings, respecting others and becoming a better listener, standing 

up for oneself and learning to be assertive, understanding and dealing with conflict, celebrating 

diversity and countering prejudice, and sustaining a caring community and making a difference. Each 

unit is based on a high-quality children’s book, around which discussions, writing, and role-playing 

activities are conducted. Teachers engage students in lessons for 1 hour at least once per week over 

the school year. Teachers are encouraged to model and reinforce social skills throughout the day. 

 Teachers receive 25 hours of intensive introductory professional development designed to 

(a) introduce them to the curricular units and the children’s books, specific lessons, and activities tied 

to each unit; (b) give them an opportunity to practice conflict resolution skills at the adult level 

through role-play and experiential learning; and (c) inspire them to employ the ideas and skills 

embodied in the curriculum in their own lives both professionally and personally. This training is 

followed by ongoing classroom coaching to support teachers in teaching the 4Rs curriculum, with a 

minimum of 12 contacts in each school year. Ongoing coaching encompasses class lesson modeling 

and workshops led by a program staff developer, coplanning and teaching of lessons by the teacher 

and staff developer, and lesson observations and feedback. Staff developers also convene regular 

conferences with teachers in a one-on-one format or with a group of teachers from one grade. 
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

394

Table 7.1. The 4Rs Program (Reading, Writing, Respect, and Resolution)

Panel 1: General characteristics

Target population 

 Universal

Program components

 Peer: In class

 Parent: Contact and involvement

 Classroom: Lessons

 Schoolwide: None or not major focus

 Community: None or not major focus

 Training: Pretraining and ongoing

Level of integration

 Core curriculum

Flexibility

 Manualized: Curriculum guidebook

 Adaptability: Less adaptable

See notes at end of table.

Panel 2: Description of instruction and strategies

Classroom

 Lessons

 Who delivers: Teacher

 Activities and tools: Story reading, role-playing, discussions, writing, reflections, sharing exercises, 

 brainstorming, songs, worksheets

 Content: Literacy focus; building community, feelings, listening, assertiveness, problem solving, diversity, 

 making a difference

 Frequency: 1-hour lessons at least once per week; minimum of 21 lessons

 Strategies

 Who delivers: Teacher

 Activities and tools: Modeling and reinforcement of social skills

 Frequency: Daily

Supplement to classroom

 Family Connections homework and parent workshops

Schoolwide activities 

 None

See notes at end of table.
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

395

Table 7.1. The 4Rs Program (Reading, Writing, Respect, and Resolution)—Continued

Panel 3: Professional development

Pre-implementation

 Teachers

 Content: Course to learn curriculum, improve on own skills, create a vision of community, set consistent 

 Rules, and behavior management

 Duration: 25 hours

 Other

 Content: None

 Duration: None

Ongoing consultation

 Teachers

 Content: Training and support in mediation; negotiation; role-playing; class meetings; behavior 

 management

 Duration: Individualized coaching a minimum of 12 times per year; 3-day summer institute

 Other

 Content: None

 Duration: None

See notes at end of table.

Panel 4: Activities for SACD goals

Violence prevention and peace promotion  Risk prevention and health promotion 

Social and emotional development  Civic responsibility and community service 

Character education  Behavior management 

Tolerance and diversity 

See notes at end of table.

Panel 5: SACD outcomes addressed 

Engagement with Learning  Empathy 

Academic Competence and Motivation  Positive School Orientation

Altruistic Behavior Negative School Orientation

Positive Social Behavior  Student Afraid at School

Problem Behavior  Victimization at School 

Self-Efficacy for Peer Interactions  Feelings of Safety 

Normative Beliefs About Aggression  Student Support for Teachers

NOTE: Abbreviations are

 : Activity or outcome addressed

 Blank cell: Activity or outcome not addressed

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

396

Sample and Random Assignment 

The New York City research team recruited a total of 14 public elementary schools in eight school districts in 

four boroughs in New York City.51 These 14 schools were randomly assigned to treatment and control 

conditions prior to the fall 2004 data collection. A two-step process was used. First, a computer-generated 

pairwise matching algorithm developed by Mathematica Policy Research, Inc. (MPR) was used to identify the 

best pairwise matches across the 14 schools based on variables identified by the New York University 

research team. The variables used in the pairwise matching for the New York City site included the following: 

(a) average spending per student, (b) percentage of part-time special education students, (c) percentage of 

students at level on the English language arts standardized achievement test, (d) attendance, (e) percentage of 

students eligible for free school lunch, (f) percentage of students who were recent immigrants, (g) number of 

students, (h) percentage of Hispanic students, (i) percentage of Asian/Other students, (j) percentage of male 

students, (k) student stability, (l) average days teacher absent, (m) percentage of fully licensed teachers, 

(n) percentage of teachers with more than 2 years of teaching experience, (o) direct total classroom 

instruction expenditure, (p) percentage of foreign-born students, (q) Bloomberg School list, (r) No Child Left 

Behind improvement status school, (s) teacher organizational readiness, and (t) school organizational 

readiness. In early spring 2004, as part of the initial school recruitment process, organizational readiness was 

measured by assessing such important school dimensions as principal leadership style, openness of 

communication, administrative/teacher buy-in, administrative and staff stability, number and degree of other 

programs, demands on teacher time, and amount of professional development. Second, using a computerbased random number generator, 1 school in each matched pair was assigned to either the intervention or 

control condition. Seven schools received the 4Rs program and 7 schools acted as control schools and 

continued to implement the social and character development activities that constituted their standard 

practice. Assignment to treatment or control condition was at the school level and therefore limited the risk 

of contamination between treatment and control classrooms. 

The original student sample (the cohort of students in the third grade in the 14 schools in fall 2004) 

numbered 1,202 students (652 treatment and 550 control). Table 7.2 documents the change in the sample 

over the three spring follow-up data collection periods. Over time, the percentage of new entrants to the 

cohort increased, eventually making up 33 percent of the sample by the spring of Year 3. There were no 

statistically significant differences between the treatment and control groups in the number of new entrants. 

The percentage of the sample made up of the original cohort further declined as students left the schools. By 

Year 3, approximately 41 percent of the original sample had left. 

 

51 The New York University research team recruited 18 schools but only 14 were included in the multiprogram 

evaluation. 
Table 7.2. Sample—4Rs 

Year 1 Year 2 Year 3 Year 4

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Characteristic Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 School sample size 14 7 7 14 7 7 14 7 7 14 7 7

 Student sample size 1,202 652 550 1,194 647 547 1,109 599 510 1,065 556 509

Stayers † † † 1,116 603 513 882 469 413 710 373 337

New entrants † † † 78 44 34 227 130 97 355 183 172

 New entrants as a percent

 of spring enrollment † † † 6.5 6.8 6.2 20.5 21.7 19.0 33.3 32.9 33.8

Total leavers (from original 

 cohort) † † † 86 49 37 320 183 137 492 279 213

 Leavers as percent of fall

 2004 enrollment † † † 7.2 7.5 6.7 26.6 28.1 24.9 40.9 42.8 38.7

Number of students per 

 school (mean) 86 93 79 85 92 78 80 86 73 76 79 73

Range of number of students 

 per school 58-149 58-149 64-97 58-147 58-147 66-94 46-136 52-136 46-86 43-126 49-126 43-101

† Not applicable.

NOTE: No statistically significant differences were found between values for treatment and control groups.

SOURCE: The Social and Character Development (SACD) Research Program.

397

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution)
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

398

Implementation 

Training 

The intervention teachers received 25 hours of program implementation training prior to the beginning of the 

school year (table 7.1, panel 3). On average, teachers received an additional 2.5 days of training during the 

school year. Onsite coaching occurred throughout the year, with the average school receiving about 1 day per 

week of staff development visits. Coaching visits typically included a classroom experience (a demonstration 

lesson by the staff developer, a lesson cofacilitated by the staff developer and the teacher, or a lesson taught 

by the teacher and observed by the staff developer) and a conference with the teacher for debriefing. 

Data Collection 

MPR collected the child, teacher, and school data for the New York City site. Table 7.3 shows the school year 

milestones and dates of implementation for the New York City site. Data were collected in the fall and spring 

of the first 2 years and the spring of Year 3. The fall 2004 data collection began on October 18, 2004, and 

ended on November 19, 2004. The average time frame from the beginning of program implementation to the 

beginning of fall data collection was 4 weeks. As a result, initial data collection took place after 4Rs 

implementation began. Therefore, these data provide a measure of the students, teachers, and schools near 

the beginning of the school year, at a time when the program had been operating for a relatively short period 

of time. The spring data collection window was from March 21, 2005, to April 15, 2005. The 4Rs program 

had been implemented for 26 weeks at the time of the spring data collection and for 17 weeks from the end 

of the fall data collection. Year 2 followed a similar pattern, with implementation occurring at the start of the 

school year, fall data collection occurring 5 weeks later, and spring data collection occurring 20 weeks after 

fall data collection (and 29 weeks after the start of implementation). In spring 2007, data collection occurred 

29 weeks after the start of implementation. Data collection took from 3 to 5 weeks at each collection point. 
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

399

Table 7.3. Data collection dates—4Rs 

Data collection schedule

Year 1

(Fall 3rd grade)

Year 1

(Spring 3rd grade)

Year 2

(Fall 4th grade)

Year 2

(Spring 4th grade)

Year 3

(Spring 5th grade)

 School sample size 14 14 14 14 14

School year dates 

 First day of school 9/7/04 † 9/8/05 † 9/5/06

 Start of implementation 9/20/04 † First day † First day

 Last day of school † 6/28/05 † 6/27/06 6/27/07

Data collection 

 Start 10/18/04 3/21/05 10/11/05 3/27/06 3/26/07

 End 11/19/04 4/15/05 11/16/05 5/8/06 4/26/07

Calendar weeks from 

 program implementation 

 to start of fall 2004 

 data collection 4 † † † †

Calendar weeks from 

 start of school to start of 

 fall 2004 data collection 6 † 5 † †

Calendar weeks from end 

 of fall data collection to 

 start of Spring data 

 collection † 17 † 20 †

Calendar weeks from 

 program implementation 

 to start of spring data 

 collection † 26 † 29 29

† Not applicable.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

400

Consent Rates, Completion Rates, and Percentage of Sample With Data 

The actual number of student, primary caregiver, and teacher reports available for analysis was smaller than 

the number in the sample because consent and completion rates were below 100 percent. Primary caregivers 

had to provide consent before children could complete the Child Report, before their child’s teacher could 

complete the Teacher Report on Student, and before they themselves completed the Primary Caregiver 

Report. Teachers also had to provide consent before completing the Teacher Report on Classroom and 

School. There were no statistically significant differences between treatment and control groups in consent 

rates for any of these reports. 

Of those with consent, not all completed their respective reports. Table 7.4 shows the consent rates, 

completion rates, and percentages of sample with data for each of the four reports over the 3 years. For the 

Child Report and two teacher reports, completion rates ranged from 91 percent to 100 percent, with two 

statistically significant differences between treatment and control groups in completion rates. In Year 3, more 

Child Reports and, in Year 2, more Teacher Reports on Students were completed in the treatment group 

compared to the control group. For the Primary Caregiver Report, the completion rates dropped over time 

from 85 percent to 58 percent. There were no statistically significant differences for the Primary Caregiver 

Report completion rates. 

The percentages of the sample with Child Report data ranged from 54 percent to 64 percent over the 3 years, 

with a statistically significant difference in Year 1, when more students in the treatment group had data 

compared to the control group. The percentages of students with information from the Teacher Report on 

Student ranged from 52 percent to 67 percent, with a statistically significant difference between treatment and 

control conditions in Year 1 favoring the treatment group. The percentages of students with data from the 

Primary Caregiver Report ranged from 32 percent to 42 percent. In general, this percentage declined over 

time, although the highest rates were seen in the spring of 2005. There were no statistically significant 

differences between treatment and control groups for this data. The percentages of teachers with data from 

the Teacher Report on Classroom and School ranged from 89 percent to 98 percent, with no statistically 

significant difference between treatment and control teachers. 
Table 7.4. Consent rates, completion rates, and percentage of sample with data from each report—4Rs 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 Student sample size 1202 652 550 1194 647 547 1109 599 510 1065 556 509

Child Report (percent)

 Primary caregiver consent rate 56.8 58.4 54.9 61.5 64.1 58.3 66.3 67.3 65.1 64.3 64.6 64.0

 Student completion rate 96.6 95.8 97.7 96.2 97.1 95.0 92.4 93.3 91.3 97.1 98.3* 95.7

 Students with data154.9 56.0 53.6 59.1 62.3* 55.4 61.2 62.8 59.4 62.4 63.5 61.3

Primary Caregiver Report (percent)

 Primary caregiver consent rate 47.8 49.4 46.0 52.8 54.9 50.5 58.3 58.8 57.8 57.1 56.3 58.0

 Primary caregiver completion rate 85.7 85.4 86.2 78.9 76.3 82.2 70.3 69.6 71.2 58.1 56.9 59.3

 Primary caregivers with data141.0 42.2 39.6 41.7 41.9 41.5 41.0 40.9 41.2 33.1 32.0 34.4

Teacher Report on Student (percent)

 Primary caregiver consent rate256.8 58.4 54.9 61.5 64.1 58.3 66.3 67.3 65.1 64.3 64.6 64.0

 Teacher completion rate 94.7 95.0 94.4 97.0 97.3 96.6 98.8 100.0*** 97.3 99.7 99.7 99.7

 Students with data153.8 55.5 51.8 59.6 62.4* 56.3 65.5 67.3 63.3 64.1 64.4 63.9

Teacher Report on Classroom and School

 (3rd- to 5th-grade teachers) (percent)

 Teacher consent rate 96.3 98.0 94.5 94.8 95.0 94.6 97.2 96.8 97.6 97.0 98.9 94.9

 Teacher completion rate 96.2 93.9 98.8 95.6 93.7 97.7 100.0 100.0 100.0 99.4 98.9 100.0

 Teachers with data192.7 92.0 93.4 90.6 89.0 92.4 97.2 96.8 97.6 96.4 97.8 94.9

* Treatment group significantly different from control group at the .05 level.

*** Treatment group significantly different from control group at the .001 level.

1 Calculated as consent rate x completion rate.

2 The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

401

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution)
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

402

Responses from students in the original cohort (stayers) and new entrants in the 4Rs sample were examined 

to investigate possible differences between the two groups in consent rates, completion rates, and the 

percentages of sample with data that might affect outcome data (table 7.5). In Years 2 and 3, consent rates for 

the child and the primary caregiver were significantly higher for the stayers than for the new entrants (by 17 

to 23 percentage points), which resulted in significantly larger percentages of data on both the Child Report 

(by 15 to 21 percentage points) and the Teacher Report on Student (by 17 to 23 percentage points). In 

contrast, more primary caregivers in the new entrants group compared to the stayers completed the Primary 

Caregiver report (by 9 to 17 percentage points) in all 3 years. 
Table 7.5. Consent rates, completion rates, and percentage of sample with data: Stayers versus new entrants—4Rs 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total Stayers

New 

entrants Total Stayers

New 

entrants Total Stayers

New 

entrants

 Student sample size 1,194 1,116 78 1,109 882 227 1,065 710 355

Child Report (percent)

 Primary caregiver consent rate 61.5 61.8 56.4 66.3 69.7*** 52.9 64.3 72.1*** 48.7

 Student completion rate 96.2 96.1 97.7 92.4 92.2 93.3 97.1 96.5 98.8

 Students with data159.1 59.4 55.1 61.2 64.3*** 49.3 62.4 69.6*** 48.2

Primary Caregiver Report (percent)

 Primary caregiver consent rate 52.8 53.0 50.0 58.3 60.3** 50.7 57.1 61.8*** 47.6

 Primary caregiver completion rate 78.9 77.9* 94.9 70.3 67.9** 81.7 59.4 56.9* 65.7

 Primary caregivers with data141.7 41.3 47.4 41.0 40.9 41.4 33.9 35.2 31.3

Teacher Report on Student (percent)

 Primary caregiver consent rate261.5 61.8 56.4 66.3 69.7*** 52.9 64.3 72.1*** 48.7

 Teacher completion rate 97.0 97.0 97.7 98.8 98.9 98.3 99.7 99.6 100.0

 Students with data159.6 59.9 55.1 65.5 68.9*** 52.0 64.1 71.8*** 48.7

* Stayers significantly different from new entrants at the .05 level.

** Stayers significantly different from new entrants at the .01 level.

*** Stayers significantly different from new entrants at the .001 level.

1 Calculated as consent rate x completion rate.

2 The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

403

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution)
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

404

Fidelity of Implementation

Each year, the seven 4Rs treatment schools were independently rated for quantity and quality of program 

implementation by two raters from the research team. The global measure of fidelity for the multisite study 

was used; inter-rater reliability was measured using Cronbach’s alpha (0.84 in Year 1, 0.78 in Year 2, and 0.72 

in Year 3). The ratings were combined into a single consensus rating and used to identify schools with high 

implementation fidelity. In Years 1 and 2, two treatment schools and, in Year 3, three treatment schools, were 

identified as having high fidelity. Cohen’s kappa was used as the measure of agreement when identifying 

schools as high fidelity and it equaled 0.70 in Year 1, 0.59 in Year 2, and 0.30 in Year 3. 

Initial Characteristics 

This section examines the initial characteristics of the students, teachers, and schools participating in the 4Rs 

evaluation. These characteristics were collected from students who were enrolled in the third grade at the 

study schools in fall 2004, as well as from their primary caregivers and third-grade teachers. In addition, 

third-, fourth-, and fifth-grade teachers and principals in the study schools provided information about 

activities related to social and character development in these schools. Documenting the characteristics of 

students, teachers, and schools and initial measures of key outcomes at a point before the intervention had 

been operating for an extended period helped to determine whether the random assignment of schools to 

treatment and control status produced treatment and control groups with similar distributions of observed 

characteristics. As noted in the following discussion, there were 8 significant differences in the observed 

characteristics, 7 of which (out of 62 comparisons, with 3 expected to be significant by chance) reflected 

differences between the treatment and control students, teachers, and schools in the school and classroom 

use of SACD activities. 

Characteristics of Children, Their Families, and Communities 

There were no significant differences between the treatment and control groups in the observed student, 

caregiver, and community characteristics (table 7.6). For students, the mean age was 8.1 years. The sample 

contained roughly equal percentages of girls (52%) and boys (48%). The sample was ethnically diverse, with 

White non-Hispanic students making up 5 percent of the sample, Black non-Hispanic students making up 41 

percent of the sample, and Hispanic students making up 46 percent of the sample. 

The sample was also diverse in its levels of family income, education levels of primary caregivers of the 

children in the sample, and family situation. For the total sample, 58 percent of children lived in a household 

where the income was 135 percent of the federal poverty level or lower, which is the income threshold for 

eligibility for free school meals. Twenty-eight percent of primary caregivers had not completed high school. 

Less than half of the children (48%) lived with both their mother and their father. There were no significant 

differences in these characteristics between the treatment and control groups. 

The mean values of the outcomes for children’s behavior and attitudes as reported by the primary caregiver, 

child, and teacher at initial data collection in fall 2004 are shown in table 7.7. There were no significant 

differences between the treatment and control groups. 
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

405

Table 7.6. Initial characteristics of children, their families, and communities—4Rs 

Characteristic Total Treatment Control

 Student sample size 493 275 218

Student demographics

 Gender (percent)

 Male 48.3 49.3 47.3

 Female 51.7 50.7 52.7

 Race/ethnicity (percent)

 White (non-Hispanic) 5.3 7.0 3.6

 Black (non-Hispanic) 40.5 43.4 37.6

 Hispanic 45.9 43.1 48.7

 Other 8.3 6.5 10.1

 Age (in years) (mean) 8.1 8.1 8.2

Primary caregiver and family characteristics

 Primary caregiver’s age (in years) (mean) 35.1 36.0 34.3

 Primary caregiver’s race/ethnicity (percent)

 White (non-Hispanic) 7.1 7.9 6.2

 Black (non-Hispanic) 41.1 44.4 37.8

 Hispanic 43.4 41.0 45.8

 Other 8.4 6.7 10.2

 Primary caregiver’s education (percent)

 Did not complete high school 28.0 27.2 28.8

 Completed high school or equivalent 27.5 25.9 29.1

 Some college 32.2 31.2 33.3

 Bachelor’s or higher degree 12.3 15.8 8.8

 Primary caregiver’s employment (percent)

 Full-time 38.9 37.5 40.3

 Other 61.1 62.5 59.7

 Primary caregiver’s marital status (percent)

 Married 40.3 41.6 39.0

 Other 59.7 58.4 61.0

 Students who live in one household (percent) 91.7 92.1 91.4

 Number of individuals in household (mean) 4.7 4.6 4.9

 Primary caregiver’s relationship to child (percent)

 Mother (stepmother) 86.5 85.9 87.2

 Father (stepfather) 8.6 7.3 9.8

 Other relative/nonrelative 4.9 6.8 3.0

See notes at end of table.
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

406

Table 7.6. Initial characteristics of children, their families, and communities—4Rs—Continued

Characteristic Total Treatment Control

 Student lives with (percent)

 Mother (stepmother) and father (stepfather) 47.9 46.7 49.0

 Mother (stepmother) only; father (stepfather) not present 46.1 47.1 45.0

 Father (stepfather) only; mother (stepmother) not present 2.9 1.9 3.9

 Other relative/nonrelative, parents not present 3.2 4.3 2.0

 Highest education of anyone in household (percent)

 Did not complete high school 21.0 18.4 23.6

 Completed high school or equivalent 28.9 26.9 31.0

 Some college 32.6 32.3 32.9

 Bachelor’s or higher degree 17.4 22.4 12.5

 Total household income (percent)

 Less than $20,000 51.7 51.0 52.5

 $20,000 to $39,999 25.9 24.2 27.7

 $40,000 to $59,999 9.9 8.5 11.3

 $60,000 or more 12.5 16.4 8.6

 Income-to-poverty-threshold ratio—Below 135 percent (percent) 57.8 55.2 60.5

 Income-to-poverty-threshold ratio—135 to 185 percent (percent) 19.3 20.1 18.5

 Income-to-poverty-threshold ratio—Above 185 percent (percent) 22.9 24.8 21.1

 Alabama Parenting Questionnaire–Poor Monitoring and 

 Supervision Subscale (mean) 1.2 1.2 1.2

 Alabama Parenting Questionnaire–Positive Parenting 

 Subscale (mean) 3.5 3.5 3.5

 Confusion, Hubbub, and Order Scale (mean) 2.2 2.1 2.2

Community characteristics (mean)

 Community Risks Scale 2.0 1.9 2.0

 Community Resources Scale 2.8 2.8 2.8

 Child-Centered Social Control Scale 2.7 2.7 2.8

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

407

Table 7.7. Mean scores and standard deviations for initial outcome measures of sample—4Rs 

Total Treatment Control

Outcome measure–Report Range Mean SD Mean SD Mean SD

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 1-4 2.9 0.7 2.9 0.6 2.9 0.7

 Normative Beliefs About Aggression–CR 1-4 1.3 0.5 1.3 0.5 1.3 0.6

 Empathy–CR 1-3 2.4 0.4 2.4 0.4 2.4 0.4

Behavior Domain

 Altruistic Behavior–CR 0-3 1.6 0.8 1.6 0.8 1.6 0.8

 Altruistic Behavior–TRS 1-4 1.5 0.6 1.5 0.6 1.5 0.6

 Altruistic Behavior–PCR 1-4 2.3 0.8 2.4 0.8 2.3 0.8

 Positive Social Behavior–TRS 1-4 2.8 0.7 2.8 0.7 2.8 0.7

 Positive Social Behavior–PCR 1-4 2.9 0.5 3.0 0.5 2.9 0.5

 Problem Behavior–CR 0-3 0.4 0.5 0.3 0.5 0.4 0.6

 Problem Behavior–TRS 1-4 1.5 0.5 1.5 0.6 1.4 0.4

 Problem Behavior–PCR 1-4 1.5 0.3 1.5 0.3 1.5 0.3

 ADHD-Related Behavior–TRS 1-4 1.8 0.7 1.8 0.7 1.8 0.6

Academics Domain

 Academic Competence and Motivation–TRS 1-5 2.6 0.8 2.6 0.9 2.6 0.8

 Engagement with Learning–CR 1-4 3.6 0.7 3.6 0.7 3.7 0.6

Perceptions of School Climate Domain

 Positive School Orientation–CR 1-4 3.0 0.7 3.0 0.8 3.1 0.7

 Negative School Orientation–CR 1-4 2.1 0.6 2.1 0.6 2.1 0.7

 Student Afraid at School–CR 1-4 2.6 0.9 2.6 1.0 2.6 0.9

 Victimization at School–CR 0-3 0.9 0.8 0.8 0.8 0.9 0.8

 Student sample size—PCR 497 228 269

 Student sample size—CR 521 238 283

 Student sample size—TRS 525 248 277

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

 SD: Standard deviation

No statistically significant differences were found between values for treatment and control groups. Weights, which assign equal 

weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests were 

conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at the 

school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

408

Characteristics of Teachers and Schools 

Table 7.8 describes the third-, fourth-, and fifth-grade teachers at the study schools. Half the members of this 

sample were White non-Hispanic (50%), and most were female (86%). They had an average of 8.5 years of 

total teaching experience, and there was a statistically significant difference between the treatment and control 

groups, with control teachers having more teaching experience than treatment teachers (10 years versus 7 

years). The majority of the teachers (78%) held an advanced or specialist degree. 

Data related to school characteristics were drawn from the Common Core of Data in order to compare 

treatment and control schools. There were no significant differences between the two groups of schools in 

terms of student composition (race/ethnicity and school lunch eligibility), number of students enrolled, 

number of full-time teachers, Title I status, or number of years the principal had been at the school (see table 

7.9). In addition, there were no significant differences between treatment and control schools in terms of 

location (urban, suburban, or rural) or lowest and highest grade offered (these data are not shown in a table). 

Table 7.8. Initial characteristics of teachers in sample—4Rs 

Characteristic Total Treatment Control

 Teacher sample size 177 92 85

Gender (percent)

 Male 14.4 14.8 13.9

 Female 85.6 85.2 86.1

Race/ethnicity (percent)

 White (non-Hispanic) 50.0 47.9 52.2

 Other 50.0 52.1 47.8

Number of years teaching experience (mean) 8.5 7.0* 10.0

Number of years teaching experience in this school (mean) 5.6 5.0 6.2

Type of teaching certificate (percent)

 Regular state certificate or advanced professional certificate 68.0 62.4 73.5

 Other 32.0 37.6 26.5

Education (percent)

 Bachelor’s degree 22.0 18.7 25.2

 Advanced degree/other 78.0 81.3 74.8

* Treatment group significantly different from control group at the .05 level.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and 

overall means. Statistical tests were conducted using regressions that included program indicators to account for the sample design 

and adjusted for clustering at the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

409

Table 7.9. Initial characteristics of schools in sample—4Rs 

Characteristic Total Treatment Control

 School sample size 14 7 7

Student race/ethnicity (percent)

 White (non-Hispanic) 6.7 8.8 4.6

 Black (non-Hispanic) 47.1 47.7 46.5

 Hispanic 40.1 37.2 42.9

 Other 6.1 6.3 6.0

Students eligible for free or reduced-price lunch (percent) 85.1 85.9 84.5

Number of students enrolled (mean) 604.2 608.6 599.9

Number of full-time teachers (mean) 44.2 43.4 44.8

Title I status (percent)

 Title I eligible school 92.9 85.7 100.0

 Schoolwide Title I 76.9 83.3 71.4

Number of years principal has been at this school (mean) 6.4 7.4 5.0

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: NCES Common Core of Data (2003-04), the Social and Character Development (SACD) Research Program.

In the Teacher Report on Classroom and School, teachers reported on nine dimensions of school 

environment (these data are not shown in a table): feelings of safety, adequacy of resources, student support, 

freedom to teach as desired, affiliation with and ties to colleagues, innovation regarding new approaches to 

teaching, professional interest, participatory decisionmaking, and work pressure. There were no statistically 

significant differences between treatment and control schools. 

The Level of SACD in the Schools Near the Beginning of the Study 

During the initial data collection, principals and teachers reported on the SACD activities used in the schools 

and classrooms, the availability of SACD materials, and the professional development provided on SACD. 

Table 7.10 shows that the majority of the school principals reported activities to promote six social and 

character development goals: violence prevention and peace promotion (93%), social and emotional 

development (100%), character education (100%), tolerance and diversity (100%), risk prevention and health 

promotion (93%), and civic responsibility and community service (100%). In addition, 64 percent of the 

principals reported activities directed toward behavior management. There were no statistically significant 

differences between the treatment and control schools in the percentages reported by principals, although this 

may be due to the small principal sample size. Teachers reported on use of these activities in their classrooms, 

ranging from 38 percent to 84 percent, and there was one significant difference (out of eight comparisons), 

with more treatment teachers than control teachers reporting the use of Social and Emotional Development 

activities in their classrooms (83% versus 64%). With respect to the use of schoolwide activities, 33 percent to 

61 percent of teachers reported that their schools used such activities. There was one significant difference 

here (out of six comparisons), with more control teachers reporting the use of school newspapers or bulletins 

than treatment teachers (54% versus 33%). 
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

410

Table 7.10. Principal and teacher initial reports on use of SACD programs or activities in 

sample—4Rs 

SACD program or activity Total Treatment Control

 Principal sample size 14 7 7

 Teacher sample size 177 92 85

Principals reporting that school had programs or activities 

 to promote the following SACD goals (percent)

 Violence prevention and peace promotion 92.9 100.0 85.7

 Social and emotional development 100.0 100.0 100.0

 Character education 100.0 100.0 100.0

 Tolerance and diversity 100.0 100.0 100.0

 Risk prevention and health promotion 92.9 100.0 85.7

 Civic responsibility and community service 100.0 100.0 100.0

 Behavior management 64.3 57.1 71.4

 None of the above 0.0 0.0 0.0

Teachers reporting on using programs or activities in their class 

 to promote the following SACD goals (percent)

 Violence prevention and peace promotion 64.4 71.9 56.9

 Social and emotional development 73.4 83.1* 63.8

 Character education 79.3 80.2 78.5

 Tolerance and diversity 70.6 73.9 67.4

 Risk prevention and health promotion 38.0 36.6 39.5

 Civic responsibility and community service 60.2 61.7 58.8

 Behavior management 84.3 88.9 79.8

 None of the above 1.5 ‡ ‡

Teachers reporting schoolwide use of the following 

 activities to promote SACD (percent)

 Morning announcements or videos 59.6 49.3 69.9

 School assemblies 53.7 51.9 55.5

 School newspapers or bulletins 43.5 32.9* 54.1

 Special school days 32.5 29.0 36.0

 Special events 61.4 57.0 65.8

 Other activities 7.0 6.7 7.3

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

411

Teachers reported using a broad range of teaching materials to support SACD activities (table 7.11), including 

teacher guides (73%), student materials (41%), instructional aids (31%), giveaways (43%), and children’s 

literature (77%). There was a statistically significant difference (out of seven comparisons) in the use of 

teacher guides, with more treatment teachers reporting their use than control teachers (88% versus 58%), and 

in the use of unspecified SACD materials, with treatment teachers reporting using other types of materials 

more often than control teachers (18% versus 6%). 

Teachers also reported using a wide variety of teaching strategies (table 7.11). All teachers reported using any 

of the 20 strategies asked about, and teachers used an average of 11.5 of the strategies. There were no 

significant differences between treatment and control teachers in the use of any of these specific strategies. 
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

412

Table 7.11. Teacher initial reports on use of SACD materials and classroom strategies in 

sample—4Rs 

SACD material and classroom strategy Total Treatment Control

 Teacher sample size 177 92 85

Teachers using the following materials in conjunction with

 social and character development activities (percent)

 Teacher guides (manuals, curricula) 72.9 87.5** 58.2

 Student materials (workbooks, worksheets) 41.2 39.6 42.9

 Instructional aids (games, software, videos) 30.6 37.6 23.7

 Giveaways (bookmarks, stickers) 42.9 38.3 47.6

 Children’s literature 77.0 81.7 72.3

 Other types of materials 12.0 18.2* 5.7

 Do not use any of the materials listed above 2.9 ‡ ‡

Teachers using any of the strategies listed below to promote

 social and character development in the classroom (percent) 100.0 100.0 100.0

Number of strategies (listed below) used by teachers to promote

 social and character development in the classroom (mean) 11.5 11.7 11.3

Teachers using each of the following strategies to promote

 social and character development (percent)

 Role-playing 59.7 61.9 57.5

 Cooperative learning 97.8 100.0 95.6

 Peer group discussions 90.6 94.5 86.6

 Direct instruction of social and character development 85.0 89.1 80.8

 Skill training 52.8 48.1 57.5

 Incorporating social and character development into 

 academic curriculum 77.3 81.4 73.1

 Parent training 8.5 8.4 8.6

 Parent/community involvement in program development or delivery 20.5 26.6 14.3

 Mentoring 32.6 37.6 27.6

 Good behavior notes sent home daily or weekly 56.6 49.4 63.8

 Presenting role models 62.2 63.5 60.9

 Targeted story reading or writing on SACD themes 89.7 95.0 84.4

 Peer mediation 40.3 40.9 39.8

 Honor roll for positive behavior 53.6 45.8 61.3

 Pledges or recitations on social and character development themes 25.5 24.2 26.9

 Guided visualization 38.0 33.3 42.8

 Student-led/student-assisted instruction 60.6 64.5 56.6

 Journaling 75.4 79.4 71.4

 Time out for negative behavior 85.5 87.7 83.2

 Daily or weekly rewards for positive behavior 87.7 85.0 90.5

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

413

Principals and teachers reported on participation in and amount of SACD training and staff development 

provided over the previous 12 months (table 7.12). Principals reported higher participation rates (71% versus 

70%) and more hours of training (10.5 versus 9.7) than did teachers. There was a significant difference 

between percentages of treatment and control teachers who reported participating in SACD training, and this 

favored the treatment group (84% versus 56%). The treatment teachers also reported significantly more 

training than did the control teachers on two specific SACD activities (out of seven): Social and Emotional 

Development (51% versus 17%) and Character Education (54% versus 20%). 

Table 7.12. Principal and teacher initial reports on SACD professional development in sample—

4Rs 

SACD professional development Total Treatment Control

 Principal sample size 14 7 7

 Teacher sample size 177 92 85

Principals reporting that staff participated in social and character

 development training within the past year (percent) 71.4 100.0 42.9

Teachers reporting participation in social and character development

 training within the past 12 months (percent) 70.3 84.2* 56.3

Number of hours of social and character development training 

 principals report were provided to each staff person last year (mean) 10.5 12.1 8.9

Number of hours of social and character development training 

 teachers report receiving during the past 12 months (mean) 9.7 12.4 7.1

Teachers reporting receiving training in the past 12 months 

 in the following areas (percent)

 Violence prevention and peace promotion 36.5 43.2 29.8

 Social and emotional development 34.4 51.3** 17.4

 Character education 36.9 54.3** 19.5

 Tolerance and diversity 20.5 28.6 12.4

 Risk prevention and health promotion 12.8 14.0 11.6

 Civic responsibility and community service 13.8 15.1 12.6

 Behavior management 36.6 37.7 35.6

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.

The data on the initial level of SACD activity emphasize that the control condition was a ―standard practice‖ 

control. Standard practice at the control schools included using SACD activities, materials, and practices, 

along with professional development, at rates and in types and amounts similar to the treatment schools. For 

example, the percentages of teachers who reported using programs or activities to promote specific SACD 

goals ranged from 37 percent to 89 percent in the treatment schools and from 40 percent to 80 percent in the 

control schools. However, the 7 significant differences between the treatment and control conditions in the 

use of SACD activities was more than expected by chance (3 out of 62 comparisons), and 6 of these cases 

favored the treatment group. This may reflect the fact that program implementation and program training for 

staff started before initial data collection. 
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

414

Impacts on Use of SACD Activities 

The introduction of the formal 4Rs program would be expected to increase the use of SACD activities in the 

treatment schools in comparison to the control schools. The analysis of this impact was based on the Teacher 

Report on Classroom and School. Every spring, third-, fourth-, and fifth-grade teachers provided information 

through the Teacher Report on Classroom and School about the social and character development activities 

they used in their classroom. Specifically, information from the TRCS was used to determine the difference 

between treatment and control teachers in these areas: 

1. the use of SACD activities in their classrooms overall and by SACD goal; 

2. the use of materials and strategies to implement the SACD activities within classrooms and within 

the entire school; 

3. the use of staff development to support the teachers; and 

4. teacher support for SACD efforts in the school and the use of practices conducive to the social and 

character development of students. 

Teacher Report on Classroom and School consent and completion rates (table 7.4) led to 89 percent to 98 

percent of all teachers having data for the 3 years. To estimate intervention impacts for each of the outcomes, 

testing of the statistical significance of the differences in means was used. Preliminary analysis indicated little 

or no gains in precision from using covariates. Before testing the mean differences, the data were weighted 

such that each school received equal weight. Standard errors of the impact estimates account for the 

clustering of teachers within schools. In addition, a set of heuristics was applied to determine whether each 

outcome domain was statistically significant after adjustments were made for the multiple tests conducted. 

Use of Activities 

The percentages of control teachers who reported using any SACD activities in their classrooms ranged from 

76 percent to 86 percent over the 3 years (table 7.13, panels 1 and 2). For the six individual SACD goals, the 

percentages ranged from 47 percent to 67 percent in Year 1, 44 percent to 66 percent in Year 2, and 43 

percent to 60 percent in Year 3. Control teachers’ use of behavior management activities ranged from 68 

percent to 76 percent over this period. The percentages of control teachers who reported using any SACD 

activities for at least 1 hour per week in their classrooms ranged from 51 percent to 69 percent over the 3 

years. For the six individual SACD goals, the percentages ranged from 15 percent to 36 percent in Year 1, 11 

percent to 28 percent in Year 2, and 11 percent to 31 percent in Year 3. Control teachers’ use of behavior 

management activities ranged from 37 percent to 54 percent over this period. 

For teachers’ reported use of any SACD activity, 48 comparisons were made, with 2 expected to be 

significant by chance. The percentages of treatment teachers using any SACD activity (panel 1) were 

significantly different from the percentages of control teachers in Year 1 (impact = 15 percentage points) and 

Year 3 (impact = 19 percentage points). Significantly larger percentages of treatment teachers reported using 

activities targeting violence prevention and peace promotion (impact = 31, 19, and 31 percentage points), 

social and emotional development (impact = 33, 27, and 24 percentage points), character education (impact = 

26, 18, and 26 percentage points), and tolerance and diversity (impact = 25, 27, and 27 percentage points) in 

all 3 years. In Year 2, there was a significant difference between treatment and control teachers in their use of 

civic responsibility and community service activities, with more control teachers reporting their use (impact = 

-16 percentage points). A similar overall pattern was seen in teachers’ reported use of these activities for at 

least 1 hour per week (panel 2). Treatment teachers’ use of any SACD activity for at least 1 hour per week 

was significant in all 3 years (impact = 32, 17, and 24 percentage points). Violence prevention and peace 

promotion was significantly impacted in Years 2 and 3 (impact = 30 and 34 percentage points). Impacts on 

social and emotional development (impact = 31, 36, and 25 percentage points), character education (impact = 

29, 35, and 29 percentage points), and tolerance and diversity (impact = 30, 34, and 30 percentage points) 
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

415

were found in all 3 years. Civic responsibility and community service was positively significantly impacted in 

Year 2 (impact = 10 percentage points). Behavior management was significantly impacted in Year 3 (impact = 

29 percentage points). After the heuristics were applied, the domain for engagement in SACD activities 

showed that the 4Rs program had statistically significant impacts in all 3 years. 

For teachers’ reported use of any named SACD activity (panels 3 and 4), 42 comparisons were made, with 2 

expected to be significant by chance. Several of the impact estimates were statistically significant over the 3 

years. This pattern holds for the use of named activities at least 1 hour per week. The use of any named 

activity was significantly impacted in all 3 years, both overall and for at least 1 hour per week (impacts ranged 

from 40 to 49 percentage points). Violence prevention and peace promotion was significantly impacted in all 

3 years, both overall and for at least 1 hour per week (impacts ranged from 27 to 44 percentage points), as 

were social and emotional development (impacts ranged from 29 to 43 percentage points), character 

education (impacts ranged from 31 to 45 percentage points), and tolerance and diversity (impacts ranged 

from 21 to 49 percentage points). The use of risk prevention and health promotion activities overall and for 

at least 1 hour per week were significantly impacted in Year 2 (impact = 15 and 11 percentage points). Civic 

responsibility and community service was significantly impacted in all 3 years (impact = 10, 15, and 16 

percentage points). The overall impact of the 4Rs program on the domain for engagement in named SACD 

activities was significant in all 3 years. 
Table 7.13. Impacts on teacher-reported SACD classroom activities—4Rs 

Panel 1: Engagement in any activities to promote SACD goals1

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 94 88 91 82 88 75

Violence prevention and peace promotion 

 (percent) 85.2* 53.9 31.3 0.004 76.5* 57.1 19.4 0.014 84.6* 53.3 31.3 0.000

Social and emotional development

 (percent) 88.2* 55.7 32.5 0.004 80.2* 52.9 27.3 0.010 84.2* 60.1 24.1 0.012

Character education (percent) 92.4* 66.9 25.5 0.007 83.4* 65.6 17.8 0.022 83.2* 57.7 25.6 0.042

Tolerance and diversity (percent) 80.8* 56.0 24.7 0.032 76.1** 49.0 27.1 0.011 83.5* 56.9 26.6 0.001

Risk prevention and health promotion

 (percent) 28.7 46.6 -17.9 0.174 48.2 44.1 4.1 0.550 45.8 42.8 3.0 0.696

Civic responsibility and community service

 (percent) 61.6 56.6 5.0 0.694 44.4* 60.5 -16.1 0.050 47.9 57.4 -9.5 0.355

Any SACD goal (percent) 97.1* 82.1 15.0 0.011 88.7 85.6 3.0 0.575 95.0* 75.6 19.4 0.017

Behavior management (percent) 85.1 73.9 11.2 0.196 89.7^ 75.9 13.8 0.096 80.7^ 67.6 13.1 0.087

See notes at end of table.

416

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution)
Table 7.13. Impacts on teacher-reported SACD classroom activities—4Rs—Continued 

Panel 2: Engagement in any activities to promote SACD goals for at least 1 hour per week

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 94 88 91 82 88 75

Violence prevention and peace promotion 

 (percent) 53.1^ 27.0 26.2 0.053 57.7* 27.9 29.8 0.003 55.4* 21.8 33.7 0.002

Social and emotional development

 (percent) 55.1* 23.7 31.4 0.006 63.8* 27.8 36.0 0.004 54.9* 29.8 25.1 0.028

Character education (percent) 65.9* 36.4 29.4 0.013 61.0* 26.1 34.9 0.000 60.4* 31.0 29.3 0.006

Tolerance and diversity (percent) 51.7* 21.6 30.1 0.008 56.3* 22.7 33.5 0.001 51.4* 21.8 29.6 0.001

Risk prevention and health promotion

 (percent) 5.1 15.1 -10.0 0.124 22.1 14.9 7.2 0.274 17.7 21.0 -3.3 0.723

Civic responsibility and community service

 (percent) 25.2 15.8 9.3 0.228 20.9* 10.7 10.2 0.044 22.1 11.1 11.0 0.147

Any SACD goal (percent) 82.4* 50.8 31.6 0.003 85.7* 69.0 16.7 0.023 88.1* 63.8 24.3 0.017

Behavior management (percent) 70.4^ 53.8 16.6 0.090 65.7 47.9 17.7 0.114 66.3* 37.2 29.1 0.015

See notes at end of table.

417

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution)
Table 7.13. Impacts on teacher-reported SACD classroom activities—4Rs—Continued

Panel 3: Engagement in activities to promote SACD goals linked to named SACD programs2

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 94 88 91 82 88 75

Violence prevention and peace promotion

 (percent) 42.0* 4.8 37.2 0.008 54.0* 12.4 41.6 0.000 52.4* 8.4 44.0 0.000

Social and emotional development 

 (percent) 46.6* 3.9 42.7 0.010 49.8* 9.3 40.5 0.000 52.9* 12.1 40.8 0.015

Character education (percent) 47.7* 3.9 43.8 0.001 53.6* 11.7 41.9 0.002 51.1* 6.6 44.6 0.004

Tolerance and diversity (percent) ‡* ‡ 31.8 0.001 48.4* 6.7 41.7 0.000 ‡* ‡ 49.2 0.000

Risk prevention and health promotion

 (percent) 5.9 8.7 -2.9 0.719 23.0* 8.0 15.0 0.049 16.3 10.3 6.1 0.500

Civic responsibility and community service

 (percent) ‡* ‡ 9.6 0.032 18.2* 3.3 14.9 0.004 ‡* ‡ 15.8 0.040

Any named activity (percent) 60.0* 12.4 47.6 0.006 66.8* 26.7 40.1 0.000 65.3* 21.9 43.4 0.013

See notes at end of table.

418

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution)
Table 7.13. Impacts on teacher-reported SACD classroom activities—4Rs—Continued

Panel 4: Engagement in activities to promote SACD goals linked to named SACD programs for at least 1 hour per week

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 94 88 91 82 88 75

Violence prevention and peace promotion

 (percent) 31.6* 5.1 26.5 0.033 48.8* 9.4 39.4 0.000 ‡* ‡ 40.4 0.005

Social and emotional development

 (percent) 32.8* 4.1 28.7 0.034 45.4* 6.6 38.8 0.003 43.8* 7.3 36.6 0.020

Character education (percent) 35.6* 4.2 31.4 0.006 48.2* 7.8 40.4 0.001 48.3* 3.5 44.8 0.001

Tolerance and diversity (percent) ‡* ‡ 21.3 0.012 40.0* 3.6 36.4 0.002 ‡* ‡ 40.1 0.001

Risk prevention and health promotion

 (percent) ‡ ‡ -4.9 0.264 15.0* 3.7 11.4 0.018 10.1 6.8 3.3 0.660

Civic responsibility and community service

 (percent) ‡^ ‡ 5.9 0.097 13.1 0.0 13.1 † 15.8 0.0 15.8 †

Any named activity (percent) 48.6* 8.7 40.0 0.003 60.1* 16.5 43.6 0.000 59.1* 10.0 49.1 0.005

† Not applicable.

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 In Year 1 at least half of the impacts were positive and statistically significant and no impact was negative and statistically significant based on univariate statistical tests, the omnibus 

impact for all the outcomes measured together was positive and statistically significant on the basis of a multivariate statistical test, and at least one outcome remained positive and 

statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) procedure to adjust significance levels downward to 

account for the multiple testing of impacts. In Years 2 and 3 at least half of the impacts were positive and statistically significant and no impact was negative and statistically significant 

based on univariate tests, and at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the 

Benjamini-Hochberg (1995) procedure to adjust significance levels downward to account for the multiple testing of impacts.

2 In Years 1 and 3 at least half of the impacts were positive and statistically significant and no impact was negative and statistically significant based on univariate statistical tests, the 

omnibus impact for all the outcomes measured together was positive and statistically significant on the basis of a multivariate statistical test, and at least one outcome remained 

positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) procedure to adjust significance levels 

downward to account for the multiple testing of impacts. In Year 2 at least half of the impacts were positive and statistically significant and no impact was negative and statistically 

significant based on univariate tests, and at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the 

Benjamini-Hochberg (1995) procedure to adjust significance levels downward to account for the multiple testing of impacts.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

419

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution)
420

Use of Materials and Strategies 

For use of materials and strategies to support SACD goals, 87 comparisons were made, with 4 expected to be 

significant by chance. Several significant impacts were found on treatment teachers’ use of SACD materials in 

all years (table 7.14). In Year 1, there was an effect on the use of teacher guides (impact = 42 percentage 

points), which was also seen in Year 2 (impact = 37 percentage points) and Year 3 (impact = 27 percentage 

points). In Year 2, there was an impact on the use of student materials (impact = 27 percentage points) and 

another in Year 3 (impact = 15 percentage points). In Years 2 and 3, there was an impact on the use of 

children’s literature (impact = 24 and 27 percentage points). In Year 2, control teachers reported significantly 

greater use of other types of materials (impact = -18 percentage points). Regarding use of instructional 

techniques, there was a significant impact on use of role-playing in all 3 years (impact = 29, 32, and 18 

percentage points). In Year 2, there were significant impacts on the use of direct instruction of SACD (impact 

= 12 percentage points) and on incorporating SACD into the academic curriculum (impact = 19 percentage 

points). Other impacts were seen on targeted story reading or writing (impact = 11 percentage points) and 

student-led/student-assisted instruction (impact = 18 percentage points). In Year 3, there was an impact on 

skill training (impact = 11 percentage points), guided visualization (impact = 21 percentage points), and the 

average number of strategies teachers reported (by 1.3 strategies on average). The 4Rs program had 

significant impacts on the domain of materials and strategies in Years 1 and 3. 
Table 7.14. Impacts on use of SACD classroom materials and teaching strategies—4Rs 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD material and teaching strategy

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 94 88 91 82 88 75

Use of SACD materials (percent)

 Teacher guides (manuals, curricula) 88.9* 46.9 42.0 0.001 87.6* 51.1 36.5 0.000 81.2* 54.3 26.9 0.002

 Student materials (workbooks or sheets) 34.3 39.0 -4.7 0.621 62.5* 35.1 27.3 0.008 59.1* 43.9 15.2 0.029

 Instructional aids (games, software, videos) 31.3 22.3 8.9 0.233 28.0 29.5 -1.6 0.825 40.0^ 25.7 14.3 0.056

 Giveaways (bookmarks, stickers) 39.0 42.7 -3.7 0.530 23.3^ 36.9 -13.6 0.080 42.6 56.2 -13.5 0.170

 Children’s literature 71.4 57.6 13.8 0.117 76.6* 52.5 24.0 0.023 81.1* 54.0 27.2 0.001

 Other types of materials 9.6 7.8 1.7 0.738 3.6* 21.7 -18.1 0.004 ‡ ‡ 3.9 0.236

 Did not use any of these materials ‡* ‡ -16.0 0.009 6.4^ 15.4 -8.9 0.077 4.9 15.8 -11.0 0.100

Use of teaching strategies (percent)

 Role-playing 85.9* 57.3 28.6 0.000 94.4* 62.7 31.6 0.000 94.1* 76.3 17.9 0.020

 Cooperative learning 97.6 95.6 1.9 0.427 99.0 97.6 1.5 0.433 100.0 98.7 1.3 †

 Peer group discussions 94.7 91.7 3.0 0.492 94.4 89.8 4.6 0.324 100.0 96.3 3.7 †

 Direct instruction of SACD 89.2 83.0 6.2 0.441 92.6* 81.0 11.6 0.006 95.3^ 83.1 12.2 0.077

 Skill training 66.2 57.4 8.7 0.282 60.7 52.8 7.9 0.327 90.4* 79.1 11.4 0.032

 Incorporating SACD into academic 

 curriculum 87.1^ 71.2 16.0 0.063 88.6* 69.4 19.1 0.007 96.1 86.1 10.1 0.116

 Parent training 7.6 3.4 4.2 0.278 4.9 7.9 -3.0 0.558 25.1 28.7 -3.6 0.632

 Parent/community involvement 21.2 17.2 4.0 0.551 23.2 13.7 9.4 0.231 47.4 39.8 7.6 0.318

 Mentoring 30.4 28.5 1.9 0.817 35.9 34.0 1.9 0.866 54.4 49.3 5.1 0.591

 Good behavior notes sent home daily 

 or weekly 46.5^ 65.0 -18.6 0.062 55.3 59.4 -4.0 0.650 87.7 80.5 7.2 0.229

 Presenting role models 60.5 65.4 -4.9 0.598 63.5 72.3 -8.8 0.235 78.3 77.5 0.8 0.906

See notes at end of table.

421

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution)
Table 7.14. Impacts on use of SACD classroom materials and teaching strategies—4Rs—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD material and teaching strategy

1

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

Use of teaching strategies (percent)—

 Continued

 Targeted story reading or writing on social

 and character development themes 94.2 85.4 8.8 0.113 98.8* 87.7 11.1 0.034 96.8 93.0 3.7 0.174

 Peer mediation 50.3^ 41.7 8.6 0.085 58.3 46.5 11.8 0.249 73.4 64.1 9.3 0.126

 Honor roll for positive behavior 40.3 56.3 -15.9 0.297 49.0 56.2 -7.1 0.633 68.2 69.2 -1.0 0.924

 Pledges or recitations on social and 

 character development themes 31.9 32.9 -1.0 0.905 27.1 30.7 -3.6 0.693 57.3 50.5 6.8 0.361

 Guided visualization 43.8 38.7 5.1 0.324 54.5 53.2 1.3 0.889 78.2* 57.7 20.5 0.002

 Student-led/student-assisted instruction 53.7 53.8 -0.1 0.993 67.8* 50.2 17.6 0.014 82.4 76.5 5.9 0.455

 Journaling 67.4 69.0 -1.5 0.865 75.6^ 61.5 14.1 0.093 85.1 77.4 7.7 0.275

 Time out for negative behavior 89.5 86.4 3.1 0.501 79.0 82.3 -3.4 0.549 91.0 82.2 8.8 0.181

 Daily or weekly rewards for positive 

 behavior 80.6 86.8 -6.2 0.439 83.5 82.8 0.7 0.901 97.1 95.2 1.8 0.550

 Any strategy 100.0 99.0 1.0 † 100.0 100.0 0.0 † 100.0 98.7 1.3 †

 Number of strategies (mean) 12.2 11.8 0.3 0.612 12.9^ 11.8 1.1 0.073 15.6* 14.3 1.3 0.028

† Not applicable.

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 In Years 1 and 3 and at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the BenjaminiHochberg (1995) procedure to adjust significance levels downward to account for the multiple testing of impacts. 

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

422

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution)
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

423

Regarding the use of schoolwide strategies, 18 comparisons were made between treatment and control 

teacher reports, with 1 expected to be significant by chance. There were no significant impacts on teacher 

reports of schools’ use of specific schoolwide strategies (these data are not shown in a table). 

Participation in Professional Development 

Regarding reported participation in professional development, 27 comparisons were made over 3 years, with 

1 expected to be significant by chance. In Year 1, the 4Rs intervention had a statistically significant impact on 

teachers’ participation in professional development training (impact = 40 percentage points); mean hours of 

SACD training (by 4.8 hours on average); and training in the specific goals of violence prevention and peace 

promotion (impact = 25 percentage points), social and emotional development (impact = 25 percentage 

points), character education (impact = 39 percentage points), and tolerance and diversity (impact = 20 

percentage points) (table 7.15). There was also a significant impact on behavior management training (impact 

= 19 percentage points). The impact on mean hours of SACD training remained significant in Year 3 (by 2.5 

hours on average). The significant impacts on violence prevention and peace promotion training remained in 

Year 2 (impact = 19 percentage points) and Year 3 (impact = 12 percentage points), as did the impact on 

social and emotional development and character education in Year 2 (impact = 16 and 11 percentage points). 

The impact on tolerance and diversity training was seen again in Year 3 (impact = 17 percentage points). 

Significant impact was found on the domain in Years 1 and 3. 
Table 7.15. Impacts on teacher-reported SACD professional development—4Rs 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD professional development1

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 94 88 91 82 88 75

SACD training in past 12 months (percent) 81.6* 41.2 40.4 0.000 63.9^ 48.6 15.3 0.083 54.1^ 30.7 23.4 0.052

Hours of SACD training (mean) 9.3* 4.4 4.8 0.005 7.4 4.8 2.5 0.448 ‡* ‡ 2.5 0.037

Training by goal (percent)

 Violence prevention and peace promotion 44.0* 19.4 24.6 0.003 36.7* 17.4 19.3 0.031 25.0* 12.8 12.3 0.027

 Social and emotional development 40.8* 15.8 25.0 0.003 32.5* 16.1 16.4 0.038 22.6^ 8.3 14.3 0.087

 Character education 50.2* 10.9 39.4 0.003 23.2* 12.3 10.9 0.038 20.1 9.4 10.7 0.133

 Tolerance and diversity 27.8* 8.0 19.8 0.001 23.1 12.1 11.1 0.109 ‡* ‡ 17.3 0.005

 Risk prevention and health promotion 7.1 4.6 2.5 0.468 21.0 18.7 2.3 0.785 12.8 6.1 6.7 0.290

 Civic responsibility and community service 10.2 5.7 4.5 0.358 ‡ ‡ 6.4 0.100 7.3 7.3 0.0 0.992

 Behavior management 35.0* 15.9 19.1 0.042 20.4 18.8 1.6 0.774 14.9 13.4 1.4 0.796

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 In Years 1 and 3 at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg 

(1995) procedure to adjust significance levels downward to account for the multiple testing of impacts. In Year 1 at least half of the impacts were positive and statistically significant 

and no impact was negative and statistically significant based on univariate statistical tests.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

424

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution)
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

425

Attitudes and Practices 

Teachers reported on their enthusiasm for SACD efforts in their schools (these data are not shown in a table) 

by indicating enthusiasm, cooperation, or open dislike. They also reported on the SACD practices of teachers 

and staff members in their schools (these data are not shown in a table). The practices included modeling 

positive character and behavior traits with students and fellow teachers, involving students in making 

decisions, giving students a voice in school governance, school encouragement of parent involvement in 

children’s social and character development, and the use of developmentally appropriate discipline strategies 

rather than punishment for misbehavior. Twenty-seven comparisons were made over 3 years, with 1 expected 

to be significant by chance. There were no statistically significant impacts on teachers’ enthusiasm for SACD 

efforts or reported SACD practices in their schools in any of the years (these data are not shown in a table). 

The 4Rs program had no impact on the domain in any year. 

Year-by-Year Impacts on Students and Perceptions of School Climate 

The primary research question for the 4Rs evaluation was this: 

What is the average effect of the 4Rs program on children’s social and emotional competence, 

behavior, academics, and perceptions of school climate? 

The first approach to answering this question was to examine the year-by-year impacts of the 4Rs program on 

these student and school climate outcomes over the 3 years as the students progressed from third through 

fifth grades. 

Equation (2) (described in chapter 1) was estimated to provide the impacts of the 4Rs program on the 20 

outcomes using data from the 14 treatment and control schools. For the 4Rs evaluation, equation (2) 

excluded the program fixed effects (p) and included program-specific covariates and random school effects 

covariates. Table 7.16 lists the covariates used with outcomes from each report in the 4Rs analysis. 
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

426

Table 7.16. Covariates used with outcomes from each report for analysis—4Rs 

Potential covariate

CR 

outcome

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome

 Total number 10 26 19 6

Child-reported

 Female   

 Hispanic   

 Black (non-Hispanic)   

 Other ethnicity   

 Age in years   

 Scales

 Afraid at School 

 Altruistic Behavior

 Empathy  

 Engagement with Learning 

 Negative School Orientation 

 Normative Beliefs About Aggression  

 Sense of School as a Community

 Problem Behavior 

 Self-Efficacy for Peer Interactions

 Victimization at School 

Primary caregiver-reported

 Age in years   

 Completed high school or equivalent  

 Some college  

 Bachelor’s or higher degree  

 Highest level of education in household

 Completed high school or equivalent

 Some college

 Bachelor’s or higher degree

 Mother present in home life 

 Mother and father present 

 Respondent someone other than mother or father 

 Number of people in household  

 Household income: $20,000 to $40,000 

 Household income: $40,000 to $60,000 

 Household income: More than $60,000 

 Income-to-poverty-threshold ratio: Below 135 percent 

 Income-to-poverty-threshold ratio: 135-185 percent 

 Full-time employment 

 Part-time employment 

See notes at the end of table.
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

427

Table 7.16. Covariates used with outcomes from each report for analysis—4Rs—Continued

Potential covariate

CR 

outcome

PCR 

outcome 

TRS 

outcome 

TRCS 

outcome

 Parental scales

 APQ-Poor Monitoring and Supervision Subscale 

 APQ-Positive Parenting Subscale

 Child-Centered Social Control

 Confusion, Hubbub, and Order  

 Community Resources

 Community Risk 

 Parent and Teacher Involvement

 Child scales

 Altruistic Behavior

 Positive Social Behavior  

 Problem Behavior  

Teacher-reported

 Female 

 Hispanic 

 Black (non-Hispanic) 

 Other ethnicity 

 Total teaching experience 

 Total experience in current school

 Regular certificate

 Other certificate

 Highest degree–bachelor’s 

 Child scales

 Academic Competence and Motivation

 ADHD-Related Behavior 

 Altruistic Behavior

 Positive Social Behavior 

 Problem Behavior

 Parent and Teacher Involvement 

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

 APQ: Alabama Parenting Questionnaire

: Covariate used

 Blank cell: Covariate not used

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

428

To assess the statistical power of the program-level impact estimates, minimum detectable impacts in effect 

size units (MDES) for each outcome measure were calculated for the 4Rs evaluation (table 7.17). MDES 

represent the smallest impacts in effect size (standard deviation) units that can be detected with a high 

probability (80%). MDES are primarily a function of study sample sizes, the degrees of freedom available for 

statistical tests, and design effects due to clustering (Schochet 2005). For the 14 schools included in the 4Rs 

evaluation, the MDES ranged from 0.088 to 0.569 for the child-level outcomes based on the Child, Caregiver, 

and Teacher Report on Student and from 0.180 to 0.581 for the school climate outcomes based on the 

Teacher Report on Classroom and School. In general, the MDES for the school climate outcomes were larger 

than those for the child-level outcomes. 
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

429

Table 7.17. Adjusted minimum detectable effect sizes for impact evaluation—4Rs 

Outcome measure–Report Year 1 Year 2 Year 3

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 0.250 0.090 0.090 

 Normative Beliefs About Aggression–CR 0.148 0.159 0.275 

 Empathy–CR 0.326 0.213 0.180 

Behavior Domain

 Altruistic Behavior–CR 0.118 0.150 0.183 

 Altruistic Behavior–PCR 0.130 0.109 0.164 

 Altruistic Behavior–TRS 0.323 0.540 0.569 

 Positive Social Behavior–PCR 0.104 0.108 0.197 

 Positive Social Behavior–TRS 0.268 0.117 0.354 

 Problem Behavior–CR 0.148 0.253 0.135 

 Problem Behavior–PCR 0.104 0.108 0.161 

 Problem Behavior–TRS 0.213 0.218 0.404 

 ADHD-Related Behavior–TRS 0.160 0.226 0.449 

Academics Domain

 Engagement with Learning–CR 0.202 0.195 0.169 

 Academic Competence and Motivation–TRS 0.159 0.303 0.335 

Perceptions of School Climate Domain

 Positive School Orientation–CR 0.277 0.236 0.294 

 Negative School Orientation–CR 0.088 0.190 0.225 

 Student Afraid at School–CR 0.218 0.235 0.137 

 Victimization at School–CR 0.157 0.116 0.218 

 Feelings of Safety–TRCS 0.501 0.542 0.568 

 Student Support for Teachers–TRCS 0.180 0.581 0.343 

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The minimum detectable effect (MDE) formula used in the calculations is as follows:

 

 

 

 

 

 

 

 

 

 

 

 

 

 

where sT and sC are the number of treatment and comparison schools; nT and nC are the average number of students per 

classroom; ρ1 is the intraclass correlation (ICC) at the school level; and factor(df) is a constant that depends on the number of 

degrees of freedom (df) available for analysis (and is 2.802 for the pooled analysis).

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

430

Table 7.18 provides the estimates of the 4Rs program’s impacts on each of the 20 outcomes over each of the 

3 years (60 impacts in total, with 3 expected to be statistically significant by chance). Of the 60 results, 1 was 

statistically significant. There was a detrimental impact on Academic Competence and Motivation (Teacher 

Report on Student, effect size [ES] = -0.17) in Year 1. In addition, there were substantively important but 

nonsignificant detrimental impacts on Feelings of Safety (Teacher Report on Classroom and School, ES = 

-0.42) and Student Support for Teachers (Teacher Report on Classroom and School, ES = -0.35) in Year 3. 

Application of the heuristics to adjust for multiple comparisons within each outcome domain indicated that 

4Rs had a statistically significant detrimental effect on the Social and Emotional Competence domain in Year 

3 and on the Behavior domain in Year 1. 
Table 7.18. Impacts on child and school outcomes—4Rs 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Social and Emotional Competence Domain1

 Self-Efficacy for Peer Interactions–CR (+) 2.96 3.04 -0.13 0.359 3.10 3.12 -0.04 0.598 3.15 3.21 -0.08 0.338

 Normative Beliefs About Aggression–CR (-) 1.39 1.44 -0.09 0.324 1.47 1.49 -0.03 0.778 1.59 1.65 -0.09 0.545

 Empathy–CR (+) 2.24 2.27 -0.07 0.674 2.08 2.10 -0.05 0.701 1.95 2.00 -0.10 0.382

Behavior Domain2

 Altruistic Behavior–CR (+) 1.30 1.40 -0.13 0.137 1.14 1.22 -0.09 0.365 1.18 1.18 0.00 0.985

 Altruistic Behavior–PCR (+) 2.38 2.38 0.00 0.989 2.36 2.43 -0.11 0.270 2.33 2.43 -0.13 0.375

 Altruistic Behavior–TRS (+) 1.48 1.58 -0.17 0.262 1.48 1.53 -0.10 0.730 1.62 1.55 0.11 0.698

 Positive Social Behavior–PCR (+) 2.97 2.98 -0.03 0.756 3.02 3.04 -0.03 0.757 3.11 3.05 0.13 0.422

 Positive Social Behavior–TRS (+) 2.81 2.78 0.03 0.788 2.84 2.90 -0.08 0.310 2.81 2.81 -0.01 0.969

 Problem Behavior–CR (-) 0.40 0.43 -0.04 0.654 0.48 0.51 -0.05 0.718 0.61 0.59 0.03 0.709

 Problem Behavior–PCR (-) 1.55 1.51 0.11 0.223 1.51 1.53 -0.04 0.720 1.52 1.53 -0.01 0.916

 Problem Behavior–TRS (-) 1.60 1.55 0.08 0.439 1.51 1.55 -0.07 0.516 1.63 1.65 -0.04 0.830

 ADHD-Related Behavior–TRS (-) 1.84 1.83 0.01 0.865 1.69 1.78 -0.15 0.205 1.74 1.87 -0.20 0.357

Academics Domain

 Engagement with Learning–CR (+) 3.63 3.67 -0.08 0.541 3.63 3.64 -0.03 0.815 3.63 3.59 0.07 0.508

 Academic Competence and Motivation–

 TRS (+) 2.69* 2.84 -0.17 0.032 2.80 2.88 -0.10 0.513 2.73 2.89 -0.16 0.309

See notes at end of table.

431

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution)
Table 7.18. Impacts on child and school outcomes—4Rs—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 2.70 2.64 0.08 0.577 2.39 2.38 0.01 0.950 2.31 2.36 -0.08 0.617

 Negative School Orientation–CR (-) 2.04 2.07 -0.06 0.439 2.20 2.19 0.02 0.841 2.26 2.25 0.02 0.862

 Victimization at School–CR (-) 0.83 0.89 -0.08 0.436 0.74^ 0.87 -0.17 0.069 0.75 0.78 -0.04 0.785

 Feelings of Safety–TRCS (+) 3.15 3.22 -0.09 0.606 3.08 3.12 -0.05 0.817 2.93 3.27 -0.42° 0.146

 Student Support for Teachers–TRCS (+) 3.10 3.05 0.06 0.568 3.14 3.12 0.02 0.926 3.08 3.30 -0.35° 0.109

* Treatment group significantly different from control group at the .05 level.

^ Treatment group significantly different from control group at the .10 to > .05 level.

o Substantive (but nonsignificant at .05 level) effect size of ≥ .25 or ≤ -.25.

1 Impact on domain found statistically significant and detrimental in Year 3 based on the fourth heuristic in which the statistical model used to estimate impacts on the individual 

outcomes was re-estimated using a composite of all the outcome variables under a domain. The domain was found significant if the impact on the composite was significant. The 

composite was formed by standardizing each outcome variable using its standard deviation, combining the values of the outcome variables, and taking the average of the final value.

2 Impact on domain found statistically significant and detrimental in Year 1 based on the fourth heuristic in which the statistical model used to estimate impacts on the individual 

outcomes was re-estimated using a composite of all the outcome variables under a domain. The domain was found significant if the impact on the composite was significant. The 

composite was formed by standardizing each outcome variable using its standard deviation, combining the values of the outcome variables, and taking the average of the final value.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The -signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models where each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. See table 1.5 

for information about the measures used to create the outcome variables. The effect size was calculated by dividing the estimated impact by the standard deviation of the outcome 

measure for the control group. The number of results found significant was no more than expected by chance.

SOURCE: The Social and Character Development (SACD) Research Program.

432

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution)
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

433

Impacts on Child Outcomes Over Time 

The 4Rs program’s impacts on the child outcomes over time were estimated using growth curve models by 

examining treatment and control group differences in the trajectories of student outcomes during the followup period while accounting for clustering at the school level. The growth curve models are estimated using a 

three-level hierarchical linear model, where Level 1 corresponds to time, Level 2 to students, and Level 3 to 

schools (described in chapter 1). 

Table 7.19 provides the estimates of the 4Rs program’s impacts on the growth in student outcomes over the 

3 years. The estimated impacts ranged in effect size units (absolute value) from 0.00 to 0.10. None of the 18 

estimated 4Rs intervention impacts on the trajectories of child outcomes was statistically significantly 

different from zero at the 5 percent level (1 in 18 would be expected to be significant by chance).
Table 7.19. Impacts on growth of child outcomes—4Rs 

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2Treatment Control

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 2.89 0.13 0.14 -0.01 -0.01 0.03 0.850

 Normative Beliefs About Aggression–CR (-) 1.31 0.13 0.13 -0.01 -0.01 0.04 0.840

 Empathy–CR (+) 2.37 -0.18 -0.17 -0.01 -0.02 0.03 0.756

Behavior Domain

 Altruistic Behavior–CR (+) 1.51 -0.16 -0.19 0.02 0.03 0.06 0.681

 Altruistic Behavior–PCR (+) 2.39 -0.02 0.01 -0.03 -0.04 0.04 0.405

 Altruistic Behavior–TRS (+) 1.46 0.07 0.00 0.07 0.10 0.10 0.475

 Positive Social Behavior–PCR (+) 2.94 0.02 0.04 -0.02 -0.04 0.02 0.400

 Positive Social Behavior–TRS (+) 2.85 -0.05 0.00 -0.05 -0.06 0.05 0.356

 Problem Behavior–CR (-) 0.36 0.12^ 0.09 0.03 0.04 0.02 0.067

 Problem Behavior–PCR (-) 1.54 0.00 -0.02 0.02 0.04 0.02 0.391

 Problem Behavior–TRS (-) 1.47 0.06 0.11 -0.04 -0.09 0.05 0.361

 ADHD-Related Behavior–TRS (-) 1.77 0.01 0.05 -0.04 -0.05 0.05 0.464

Academics Domain

 Engagement with Learning–CR (+) 3.65 0.01 -0.04 0.05 0.07 0.03 0.136

 Academic Competence and Motivation–TRS (+) 2.61 0.05 0.09 -0.04 -0.04 0.06 0.548

See notes at end of table.

434

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution)
Table 7.19. Impacts on growth of child outcomes—4Rs—Continued

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2Treatment Control

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 2.88 -0.23 -0.25 0.03 0.03 0.05 0.571

 Negative School Orientation–CR (-) 2.05 0.08 0.09 0.00 0.00 0.05 0.947

 Student Afraid at School–CR (-) 2.59 -0.14 -0.15 0.02 0.02 0.05 0.716

 Victimization at School–CR (-) 0.88 -0.06 -0.06 0.00 0.00 0.05 0.930

^ Treatment group significantly different from control group at the .05 level.

1 Pertains to the estimated slope of the outcome for the treatment or control groups.

2 The average score at implementation is calculated across treatment and control groups, using regression models for adjustment on covariates. 

3 Estimated difference between the slope of the treatment and control groups.

4 Effect size: the slope of the treatment group minus the slope of the control group divided by the standard deviation of the outcome for the program’s control group (the standard 

deviation is calculated without accounting for school-level clustering or regression adjustments).

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

No findings were found statistically significant at or below the .05 level. The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were 

calculated using HLM 6.06. Sample weights were used in all analyses to give (1) each school equal weight in each program (within each time period) and (2) each time period equal 

weight within the analysis. See table 1.5 for information about the measures used to create the outcome variables.

SOURCE: The Social and Character Development (SACD) Research Program.

435

Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution)
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

436

Summary 

As part of the Social and Character Development (SACD) initiative, researchers at the New York site 

evaluated the 4Rs program. This program was designed to integrate the teaching and promoting of social and 

emotional skills and conflict resolution into the language arts curriculum and to create a caring classroom 

community. Fourteen public schools in New York City were recruited by the New York research team and 

randomly assigned to treatment and control conditions to determine the impact of 4Rs on social and 

character development activities in the schools and on the child outcome domains of Social and Emotional 

Competence, Behavior, Academics, and Perceptions of School Climate. 

Analyses of the initial characteristics of the sample (students, caregivers, communities, teachers, and schools) 

indicated that randomization to treatment and control status produced groups that were relatively similar at 

the start of the study (with 1 out of 84 comparisons statistically significantly different, fewer than would be 

expected by chance). The data on the initial level of SACD activity led to two findings. First, treatment 

teachers reported greater use of and training in SACD activities than control teachers, and they did so more 

often than would be expected by chance (7 out of 62 comparisons, with 3 expected significant by chance). 

There are two potential causes for this finding, and the analysis cannot be used to determine whether the 

reason for such a difference was that the two groups did differ on their initial use of SACD activities (i.e., that 

randomization did not create similar treatment and control groups) or whether the fact that the training of all 

treatment teachers and the implementation of the 4Rs program began before the initial data were collected 

(by 4 weeks) influenced the teacher reports. Because it is likely (though unproven) that the training and 

implementation affected the teacher reports, these data were not considered appropriate for use as a baseline 

measure of SACD activities and training in the treatment schools. 

Second, these data indicate that the control condition for the SACD project was not a ―no treatment‖ control 

but a ―standard practice‖ control. Because the control teachers were not affected by the implementation of 

the SACD programs before data collection, their reports reflect standard practice in the control schools. 

Standard practice at the control schools included reports of 40 percent to 80 percent of teachers using any 

SACD activities, 96 percent of teachers using specific materials in conjunction with these activities, 100 

percent using at least one of the specified instructional strategies, and 56 percent participating in SACD 

training over the past 12 months. 

Analyses of the impacts of the 4Rs program on the level of SACD activities in the schools revealed impacts 

on the use of such activities (66 out of 90, 1 negative) and related materials and strategies (18 out of 87, 1 

negative) across the 3 years. Analyses also showed more use of professional development activities for 

treatment teachers in all years (13 out of 27). 

Of the 20 child-level outcomes representing the four domains of Social and Emotional Competence, 

Behavior, Academics, and Perceptions of School Climate assessed in each of the 3 years of the study (a total 

of 60 results), 1 showed a statistically significant detrimental impact of the 4Rs intervention. A growth curve 

analysis was used to analyze the change over time in these same outcomes between initial data collection and 

the final outcome data collection at the end of the study. None of the 18 child-level outcomes assessed 

showed a significant impact of the 4Rs program. 

The SACD evaluation did not find evidence to support the hypothesis that 4Rs had beneficial impacts on 

students’ social and character development. Such results could be caused by the inability of the program to 

cause such change, possibly because the theory of action for the program is incomplete or the activities to 

carry out that theory are not effective. 

However, these results may also be due to the inability of the evaluation to observe such a change due to the 

control condition, the level of nonparticipation, or the sample size. The control schools continued using their 

standard SACD activities, and these turned out to be high in quantity and broad in scope. While 4Rs had a 

significant positive impact on the amount and type of SACD activities, the resulting difference in the amount 
Chapter 7. The 4Rs Program (Reading, Writing, Respect, and Resolution) 

437

of SACD activities between the treatment and control schools may not have been large enough to cause 

significant differences in the student outcomes. Second, 38 percent to 45 percent of the students in the 

sample universe did not take part (depending on year) because of nonconsent or noncompletion of the 

surveys. As a determination could not be made as to whether the students not taking part significantly 

differed from those who did take part, the evaluation’s results are valid only for the students who took part. If 

the students not taking part were different, and if they would have responded better to 4Rs than to the SACD 

activities occurring in the control schools, then the evaluation could have underestimated the program’s 

impact. Third, the sample size of 14 schools and the resulting higher MDES compared to those for the 

multiprogram evaluation may have reduced the likelihood of detecting statistically significant effects. 

However, it should be noted that 68 percent of the MDES for the 60 outcomes used in the year-by-year 

analysis were below 0.25 (50% were below 0.20). In addition, only 2 of the 60 outcomes were found to be 

substantively important, and they were detrimental. 

 
This page is intentionally blank. 
439

Chapter 8. Second Step 

University of Maryland

(Maryland Site)

Intervention 

Researchers at the University of Maryland (Maryland site) evaluated the Second Step (SS) program. This

program, one of several disseminated by the Committee for Children, is designed to promote social 

competence and reduce children’s social and emotional problems. Table 8.1 describes the SS program’s 

general characteristics (panel 1), the types of instruction and strategies used (panel 2), the professional 

development provided for those implementing the program (panel 3), and the social and character 

development activities (panel 4) and outcomes (panel 5). SS includes the following components:

 The SS curriculum focuses on developing children’s social and emotional skills in the areas of 

empathy, impulse control and problem solving, and anger management. Through storytelling, 

discussion, and role-playing, teachers deliver 30-minute classroom lessons to students, 1 to 2 days per 

week. 

 Teachers learn to use classroom management practices and to model social-emotional skills to foster 

a positive classroom climate. Teachers give students opportunities to practice social-emotional skills 

in new situations; positively reinforce students’ skill use; and utilize teachable moments to provide 

coaching, constructive feedback, and positive reinforcement to students to support skills used during 

real-life situations. Visual cues are utilized to illustrate aspects of social problem solving. 

 SS extends into the home through newsletters, a family guide and video, and encouragement of 

parent involvement in helping children complete their SS homework. 

 In this evaluation, treatment schools established Character Development Planning Teams (CDPTs), 

made up of counselors, the principal, and other school staff members (e.g., teachers, administrative 

or office personnel) who made decisions about program implementation and other social and 

character development activities. 
Chapter 8. Second Step 

440

Table 8.1. Second Step 

Panel 1: General characteristics

Target population 

 Universal

Program components

 Peer: In class

 Parent: Contact

 Classroom: Lessons and transfer of training

 Schoolwide: Program artifacts and modeling

 Community: None or not major focus

 Training: Pretraining and ongoing

Level of integration

 Add-on curriculum

Flexibility

 Manualized: Curriculum guidebook

 Adaptability: Character development planning teams

See notes at end of table.

Panel 2: Description of instruction and strategies

Classroom

 Lessons

 Who delivers: Teacher

 Activities and tools: Storytelling, discussion, videos, role-playing, anticipation, recall, learning points

 Content: Empathy, impulse control, problem solving, anger management

 Frequency: 30-minute lessons, 1 to 2 days per week

 Strategies

 Who delivers: Teacher

 Activities and tools: Modeling and reinforcement of skills

 Frequency: Weekly lessons, daily generalization

Supplement to classroom

 Take-home letters, homework, family guide, and video

Schoolwide activities 

 Program artifacts, training of school staff

See notes at end of table.
Chapter 8. Second Step 

441

Table 8.1. Second Step—Continued

Panel 3: Professional development

Pre-implementation

 Teachers

 Content: Training on child development, social skills, and curriculum delivery

 Duration: 2 days

 Other

 Content: Character Development Planning Team, school counselor, and other staff; training on child 

 development, social skills, and curriculum content

 Duration: 21⁄2 hours

Ongoing consultation

 Teachers

 Content: Year-end workshop to review student outcome data and plan for following year; ongoing program 

 implementation support by school counselors

 Duration: 1 day

 Other

 Content: School counselor guidance on strategies and challenges; principal and staff review student 

 outcome data and plan for following year

 Duration: 1⁄2-day counselor meetings about monthly; 1-day year-end workshop

See notes at end of table.

Panel 4: Activities for SACD goals

Violence prevention and peace promotion  Risk prevention and health promotion

Social and emotional development  Civic responsibility and community service

Character education  Behavior management

Tolerance and diversity

See notes at end of table.

Panel 5: SACD outcomes addressed 

Engagement with Learning Empathy 

Academic Competence and Motivation Positive School Orientation

Altruistic Behavior  Negative School Orientation

Positive Social Behavior  Student Afraid at School

Problem Behavior  Victimization at School

Self-Efficacy for Peer Interactions  Feelings of Safety

Normative Beliefs About Aggression Student Support for Teachers

NOTE: Abbreviations are

 : Activity or outcome addressed

 Blank cell: Activity or outcome not addressed

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 8. Second Step 

442

Sample and Random Assignment 

The Maryland research team recruited a total of 12 public elementary schools in a single school district in 

Maryland. The 12 schools were randomly assigned to treatment and control conditions prior to the fall 2004 

data collection. A two-step process was used. First, a pairwise matching algorithm developed by Mathematica 

Policy Research, Inc. (MPR) was used to identify the best pairwise matches across the 12 schools based on 

variables identified by the Maryland research team. The variables used in the pairwise matching for the 

Maryland site included the following: (a) school mobility rate, (b) average daily attendance rate, (c) average 

national percentile rank of reading and math scores for fifth-graders, (d) average national percentile rank of 

reading and math scores for third-graders, (e) percentage of students eligible for free and reduced-price lunch, 

(f) school enrollment size, (g) percentage of families in the community with children in female-headed 

households with no husband present, (h) percentage of persons in the community aged 25 years and older 

with bachelor’s degrees, and (i) population density. Second, using a computer-based pseudo random number 

generator, 1 school in each matched pair was assigned to either the intervention or the control condition. Six 

schools received the SS program and 6 schools acted as control schools and continued to implement the 

social and character development activities that constituted their standard practice. Assignment to treatment 

or control condition was at the school level and therefore limited the risk of contamination between 

treatment and control classrooms. 

The original student sample (the cohort of students in the third grade in the 12 schools in fall 2004) 

numbered 944 students (524 treatment and 420 control). Table 8.2 documents the change in the sample over 

the three spring follow-up data collection periods. Over time, new entrants to the cohort became a larger 

percentage of the sample, eventually making up 14 percent of the sample by the spring of Year 3. There were 

no statistically significant differences between the treatment and control groups in the numbers of new 

entrants. The percentage of the sample made up of the original cohort further declined as students left the 

schools. By Year 3, approximately 9 percent of the original sample had left. There was a statistically 

significant difference between treatment and control schools in the percentage of students who had left (i.e., 

―leavers‖) in Year 3; the treatment schools had a larger percentage of leavers than control schools (11% 

versus 8%). 
Table 8.2. Sample—SS 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Characteristic Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 School sample size 12 6 6 12 6 6 12 6 6 12 6 6

 Student sample size 944 524 420 951 528 423 959 533 426 989 533 456

Stayers † † † 918 510 408 826 456 370 855 467 388

New entrants † † † 33 18 15 133 77 56 134 66 68

 New entrants as a percent of 

 spring enrollment † † † 3.5 3.4 3.5 13.9 14.4 13.1 13.5 12.4 14.9

Total leavers (from original cohort) † † † 26 14 12 118 68 50 89 57 32

 Leavers as a percent of fall 2004

 enrollment † † † 2.8 2.7 2.9 12.5 13.0 11.9 9.4 10.9* 7.6

Number of students per school 

 (mean) 79 87 70 79 88 71 80 89 71 82 89 76

Range of number of students per 

 school 46-123 55-123 46-102 47-127 54-127 47-104 39-125 54-25 39-110 41-121 52-121 41-115

† Not applicable.

* Treatment group significantly different from control group at the .05 level.

SOURCE: The Social and Character Development (SACD) Research Program.

Chapter 8. Second Step

443
Chapter 8. Second Step 

444

Implementation 

Training 

The intervention teachers received 2 days of program implementation training prior to the beginning of the 

school year (table 8.1, panel 3). School counselors provided ongoing program implementation support to 

teachers throughout the school year. School counselors received ongoing guidance through half-day monthly 

project meetings from the Maryland research team. In these monthly meetings, the research team provided 

detailed implementation feedback to the school counselors, who then conveyed the information to 

participating teachers on an as-needed basis. 

Teachers participated in a 2-day training that focused on child development, social skills, and program 

implementation. Members of the Character Development Planning Teams (CDPTs) and other school staff 

members (e.g., principal, school counselors) received a condensed, 2.5-hour version of the teacher training. 

During the school year, school counselors participated in monthly half-day meetings with the Maryland 

research team to discuss program implementation. Teachers and CDPT members were invited to participate 

in a 1-day end-of-the-year workshop to review data on student progress and to plan implementation activities 

for the following year. Teams opted to do this after the first year of implementation only. 

Data Collection 

MPR collected the multiprogram child, teacher, and school data for the Maryland site. Table 8.3 shows the 

school year milestones and dates of implementation for the Maryland site. Data were collected in the fall and 

spring of the first 2 years and in the spring of Year 3. The fall 2004 multiprogram data collection began on 

October 11, 2004, and ended on October 29, 2004. The average time frame from the beginning of program 

implementation to the beginning of fall data collection was 5 weeks. As a result, initial data collection took 

place after implementation of the SS program began. Therefore, these data provide a measure of the students, 

teachers, and schools near the beginning of the school year, at a time when the SS program had been 

operating for a relatively short period of time. The spring data collection window was from April 25, 2005, to 

May 13, 2005. The SS program had been implemented for 33 weeks at the time of the spring data collection 

and for 25 weeks from the end of the fall data collection. Year 2 followed a similar pattern, with 

implementation occurring at the start of the school year, fall data collection occurring 8 weeks later, and 

spring data collection occurring 23 weeks after fall data collection (and 34 weeks after the start of 

implementation). In spring 2007, data collection occurred 34 weeks after the start of implementation. Data 

collection took from 3 to 5 weeks at each collection point. 
Chapter 8. Second Step 

445

Table 8.3. Data collection dates—SS

Year 1 Year 1 Year 2 Year 2 Year 3

Data collection schedule (Fall 3rd grade) (Spring 3rd grade) (Fall 4th grade) (Spring 4th grade) (Spring 5th grade)

 School sample size 12 12 12 12 12

School year dates 

 First day of school 8/30/04 † 8/29/05 † 8/28/06

 Start of implementation 9/6/04 † First day † First day

 Last day of school † 6/16/05 † 6/15/06 6/14/07

Data collection 

 Start 10/11/04 4/25/05 10/24/05 4/24/06 4/23/07

 End 10/29/04 5/13/05 11/11/05 5/12/06 5/10/07

Calendar weeks from 

 program implementation 

 to start of fall 2004 

 data collection 5 † † † †

Calendar weeks from 

 start of school to start of 

 fall 2004 data collection 6 † 8 † †

Calendar weeks from 

 end of fall data collection 

 to start of spring data 

 collection † 25 † 23 †

Calendar weeks from 

 program implementation 

 to start of spring data

 collection † 33 † 34 34

† Not applicable.

SOURCE: The Social and Character Development (SACD) Research Program.

Consent Rates, Completion Rates, and Percentage of Sample With Data 

The actual number of student, primary caregiver, and teacher reports available for analysis was smaller than 

the number in the sample because consent and completion rates were less than 100 percent. Primary 

caregivers had to provide consent before children could complete the Child Report, before their child’s 

teacher could complete the Teacher Report on Student, and before they themselves completed the Primary 

Caregiver Report. There were statistically significant differences in consent for child and primary caregiver in 

Year 1; treatment caregivers had higher consent rates than control caregivers. Teachers also had to provide 

consent before completing the Teacher Report on Classroom and School. There were no statistically 

significant differences between treatment and control groups in consent rates for teachers. 

Of those with consent, not all completed their respective reports. Table 8.4 shows the consent rates, 

completion rates, and percentages of sample with data for each of the four reports over the 3 years. For the 

Child Report and two teacher reports, completion rates ranged from 87 percent to 100 percent, with no 

statistically significant difference in completion rates between treatment and control groups. For the Primary 

Caregiver Report, the completion rates dropped over time from 96 percent to 83 percent. There was one 

statistically significant difference for the Primary Caregiver Report completion rate; in the spring of Year 1, 

fewer treatment group caregivers completed reports than did control group caregivers. 
Chapter 8. Second Step 

446

The percentages of the sample with Child Report data ranged from 58 percent to 72 percent over the 3 years, 

with one statistically significant difference in the fall of Year 1. The percentages of students with information 

from the Teacher Report on Student ranged from 60 percent to 72 percent, again with one statistically 

significant difference in the fall of Year 1. The percentages of students with data from the Primary Caregiver 

Report ranged from 55 percent to 66 percent, also with one statistically significant difference in the fall of 

Year 1. The percentages of teachers with data from the Teacher Report on Classroom and School ranged 

from 87 percent to 97 percent, with no statistically significant differences between treatment and control 

teachers. 
Table 8.4. Consent rates, completion rates, and percentage of sample with data from each report—SS 

Year 1 Year 1 Year 2 Year 3

(Fall 3rd grade) (Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 Student sample size 944 524 420 951 528 423 959 533 426 989 533 456

Child Report (percent)

 Primary caregiver consent rate 65.0 68.9** 60.2 65.7 68.6* 62.2 71.1 72.4 69.5 68.4 70.4 66.0

 Student completion rate 95.9 95.6 96.4 99.8 99.7 100.0 99.9 100.0 99.7 99.6 100.0 99.0

 Students with data162.3 65.7 58.0 65.6 68.4 62.2 71.0 72.4 69.2 68.0 70.4 65.4

Primary Caregiver Report (percent)

 Primary caregiver consent rate 65.0 68.9** 60.2 65.7 68.8* 61.9 70.6 71.9 69.0 68.1 70.0 66.0

 Primary caregiver completion rate 95.8 95.6 96.0 87.2 84.3* 91.2 87.6 88.0 87.1 83.2 83.4 83.1

 Primary caregivers with data162.3 65.7 57.8 57.3 58.0 56.5 61.8 63.2 60.1 56.7 58.3 54.8

Teacher Report on Student (percent)

 Primary caregiver consent rate265.0 68.9** 60.2 65.7 68.6* 62.2 71.1 72.4 69.5 68.4 70.4 66.0

 Teacher completion rate 99.7 99.4 100.0 99.8 99.7 100.0 99.6 99.5 99.7 99.7 100.0 99.3

 Students with data164.8 68.5 60.2 65.6 68.4 62.2 70.8 72.0 69.2 68.1 70.4 65.6

Teacher Report on Classroom and School

 (3rd- to 5th-grade teachers) (percent)

 Teacher consent rate 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0

 Teacher completion rate 92.2 90.3 94.4 93.1 93.5 92.6 96.6 96.8 96.2 89.0 90.8 86.8

 Teachers with data192.2 90.3 94.4 93.1 93.5 92.6 96.6 96.8 96.2 89.0 90.8 86.8

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

1 Calculated as consent rate x completion rate.

2 The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

447

Chapter 8. Second Step
Chapter 8. Second Step 

448

Responses from students in the original cohort (stayers) and new entrants in the SS sample were examined to 

investigate possible differences between the two groups in consent rates, completion rates, and the 

percentages of sample with data that might affect outcome data (table 8.5). In Years 1 and 2, the stayers had 

significantly higher consent rates than the new entrants (by 18 to 24 percentage points), and in Year 3, the 

opposite trend was found, with new entrants having significantly higher consent rates than stayers (by 18 

percentage points). These differences in consent rates resulted in a similar pattern of differences for 

percentages of sample with data, with more data for the stayers in Years 1 and 2 on all three reports (by 17 to 

22 percentage points), and more data for the new entrants in Year 3 on all three reports (by 15 to 16 

percentage points). 
Table 8.5. Consent rates, completion rates, and percentage of sample with data: Stayers versus new entrants—SS

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Report Total Stayers

New 

entrants Total Stayers

New 

entrants Total Stayers

New 

entrants

 Student sample size 951 918 33 959 826 133 989 855 134

Child Report (percent)

 Primary caregiver consent rate 65.7 66.4* 45.5 71.7 73.7*** 54.9 68.4 66.0*** 83.6

 Student completion rate 99.8 99.8 100.0 99.9 99.8 100.0 99.6 99.8* 98.2

 Students with data165.6 66.3* 45.5 71.0 73.6*** 54.9 68.0 65.8*** 82.1

Primary Caregiver Report (percent)

 Primary caregiver consent rate 65.7 66.6** 42.4 70.6 73.1*** 54.9 68.1 65.7*** 83.6

 Primary caregiver completion rate 87.2 87.2 85.7 87.6 87.7 86.3 83.4 83.5 83.0

 Primary caregivers with data157.3 58.1* 36.4 61.8 64.2*** 47.4 56.8 54.9** 69.4

Teacher Report on Student (percent)

 Primary caregiver consent rate265.7 66.4* 45.5 71.7 73.7*** 54.9 68.4 66.0*** 83.6

 Teacher completion rate 99.8 99.8 100.0 99.6 99.7 98.6 99.7 100.0** 98.2

 Students with data165.6 66.3* 45.5 70.8 73.5*** 54.1 68.1 66.0*** 82.1

* Stayers significantly different from new entrants at the .05 level.

** Stayers significantly different from new entrants at the .01 level.

*** Stayers significantly different from new entrants at the .001 level.

1 Calculated as consent rate x completion rate.

2 The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

449

Chapter 8. Second Step
Chapter 8. Second Step 

450

Fidelity of Implementation

In Years 2 and 3, the six SS treatment group schools were individually rated for the quantity and quality of the 

program’s implementation by two raters from the research team (in Year 1, only one rater was used). The 

global measure of fidelity for the multisite study was used; inter-rater reliability was measured using 

Cronbach’s alpha (Year 2 = 1.00 and Year 3 = 0.99). The ratings were combined into a single consensus 

rating and used to identify schools with high implementation fidelity. In Year 2, five treatment schools were 

identified as having high fidelity, and in Year 3 four schools were. Cohen’s kappa was used as the measure of 

agreement when identifying schools as high fidelity, and it equaled 1.00 in both years. In Year 1, four 

treatment schools were identified as having high fidelity; however, only one rater was used, and therefore 

inter-rater reliability could not be determined. 

Initial Characteristics 

This section examines the initial characteristics of the students, teachers, and schools participating in the 

evaluation of the SS program. These characteristics were collected from students who were enrolled in the 

third grade at the study schools in fall 2004, as well as from their primary caregivers and third-grade teachers. 

In addition, third-, fourth-, and fifth-grade teachers and principals in the study schools provided information 

about activities related to social and character development in these schools. Documenting the characteristics 

of students, teachers, and schools and initial measures of key outcomes at a point before the interventions 

had been operating for an extended period helped to determine whether the random assignment of schools to 

treatment and control status produced treatment and control groups with similar distributions of observed 

characteristics. As noted in the following discussion, there were 13 significant differences in the observed 

characteristics, 12 of which (out of 62 comparisons, with 3 expected to be significant by chance) were related 

to differences between the treatment and control students, teachers, and schools in the use of SACD 

activities in the classroom and school. 

Characteristics of Children, Their Families, and Communities 

The mean age of students in the overall SS sample was 8.0 years. The sample contained equal percentages of 

girls (50%) and boys (50%). The sample had some ethnic diversity, with White non-Hispanic students making 

up 83 percent of the sample, Black non-Hispanic students making up 7 percent of the sample, and Hispanic 

students making up 5 percent of the sample. 

The sample was also somewhat diverse in its levels of family income, education levels of primary caregivers of 

the children in the sample, and family situation. For the total sample, 5 percent of children lived in a 

household where the income was 135 percent of the federal poverty level or lower, which is the income 

threshold for eligibility for free school meals. About 4 percent of primary caregivers had not completed high 

school. There was a statistically significant difference between the groups, with more control group caregivers 

not having completed high school than treatment group caregivers (5% versus 4%). The majority of the 

children (80%) lived with both their mother and their father. 

The mean values of the outcomes for children’s behavior and attitudes as reported by the primary caregiver, 

child, and teacher at initial data collection in fall 2004 are shown in table 8.7. There were no significant 

differences between the treatment and control groups. 
Chapter 8. Second Step 

451

Table 8.6. Initial characteristics of children, their families, and communities—SS 

Characteristic Total Treatment Control

 Student sample size 588 345 243

Student demographics

 Gender (percent)

 Male 50.4 51.6 49.3

 Female 49.6 48.4 50.7

 Race/ethnicity (percent)

 White (non-Hispanic) 82.9 85.8 80.0

 Black (non-Hispanic) 6.6 5.3 7.8

 Hispanic 4.7 2.9 6.5

 Other 5.8 6.0 5.6

 Age (in years) (mean) 8.0 8.0 8.0

Primary caregiver and family characteristics

 Primary caregiver’s age (in years) (mean) 38.6 38.3 38.9

 Primary caregiver’s race/ethnicity (percent)

 White (non-Hispanic) 84.7 88.4 80.9

 Black (non-Hispanic) 7.0 5.6 8.3

 Hispanic 4.2 2.0 6.4

 Other 4.2 3.9 4.4

 Primary caregiver’s education (percent)

 Did not complete high school 4.1 3.5* 4.6

 Completed high school or equivalent 13.5 18.2 8.8

 Some college 36.1 34.5 37.6

 Bachelor’s or higher degree 46.4 43.8 49.0

 Primary caregiver’s employment (percent)

 Full-time 49.2 50.3 48.0

 Other 50.8 49.7 52.0

 Primary caregiver’s marital status (percent)

 Married 79.3 79.4 79.3

 Other 20.7 20.6 20.7

 Students who live in one household (percent) 96.1 95.0 97.1

 Number of individuals in household (mean) 4.7 4.6 4.8

 Primary caregiver’s relationship to child (percent)

 Mother (stepmother) 90.1 90.1 90.1

 Father (stepfather) 7.6 6.9 8.3

 Other relative/nonrelative 2.3 3.0 1.5

See notes at end of table.
Chapter 8. Second Step 

452

Table 8.6. Initial characteristics of children, their families, and communities—SS—Continued

Table 8.6

Characteristic Total Treatment Control

 Student lives with (percent)

 Mother (stepmother) and father (stepfather) 80.3 79.3 81.2

 Mother (stepmother) only; father (stepfather) not present 15.3 15.9 14.8

 Father (stepfather) only; mother (stepmother) not present 2.8 2.7 3.0

 Other relative/nonrelative, parents not present 1.6 ‡ ‡

 Highest education of anyone in household (percent)

 Did not complete high school 1.4 1.5 1.2

 Completed high school or equivalent 10.0 13.0 7.0

 Some college 31.3 30.5 32.1

 Bachelor’s or higher degree 57.3 54.9 59.7

 Total household income (percent)

 Less than $20,000 2.2 1.7 2.6

 $20,000 to $39,999 10.1 10.3 10.0

 $40,000 to $59,999 11.2 12.3 10.2

 $60,000 or more 76.5 75.7 77.2

 Income-to-poverty-threshold ratio—Below 135 percent (percent) 5.1 4.9 5.3

 Income-to-poverty-threshold ratio—135 to 185 percent (percent) 7.2 7.5 6.8

 Income-to-poverty-threshold ratio—Above 185 percent (percent) 87.7 87.6 87.8

 Alabama Parenting Questionnaire––Poor Monitoring and 

 Supervision Subscale (mean) 1.1 1.1 1.1

 Alabama Parenting Questionnaire––Positive Parenting

 Subscale (mean) 3.5 3.5 3.5

 Confusion, Hubbub, and Order Scale (mean) 2.2 2.3 2.2

Community characteristics (mean)

 Community Risks Scale 1.1 1.1 1.1

 Community Resources Scale 3.1 3.0 3.1

 Child-Centered Social Control Scale 3.5 3.5 3.5

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and 

overall means. Statistical tests were conducted using regressions that included program indicators to account for the sample design 

and adjusted for clustering at the school level.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 8. Second Step 

453

Table 8.7. Mean scores and standard deviations for initial outcome measures of sample—SS

Total Treatment Control

Outcome measure–Report Range Mean SD Mean SD Mean SD

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 1-4 3.0 0.6 3.0 0.6 3.0 0.6

 Normative Beliefs About Aggression–CR 1-4 1.2 0.4 1.2 0.4 1.2 0.4

 Empathy–CR 1-3 2.4 0.4 2.5 0.3 2.4 0.4

Behavior Domain

 Altruistic Behavior–Child 0-3 1.2 0.8 1.2 0.8 1.2 0.8

 Altruistic Behavior–TRS 1-4 1.4 0.5 1.5 0.6 1.4 0.4

 Altruistic Behavior–PCR 1-4 2.1 0.7 2.2 0.7 2.1 0.7

 Positive Social Behavior–TRS 1-4 3.4 0.6 3.4 0.6 3.4 0.6

 Positive Social Behavior–PCR 1-4 3.1 0.5 3.2 0.5 3.1 0.5

 Problem Behavior–CR 0-3 0.1 0.2 0.1 0.2 0.1 0.2

 Problem Behavior–TRS 1-4 1.2 0.3 1.2 0.3 1.2 0.3

 Problem Behavior–PCR 1-4 1.5 0.3 1.5 0.3 1.5 0.3

 ADHD-Related Behavior–TRS 1-4 1.5 0.6 1.5 0.6 1.5 0.6

Academics Domain

 Academic Competence and Motivation–TRS 1-5 3.3 0.8 3.3 0.8 3.4 0.8

 Engagement with Learning–CR 1-4 3.8 0.4 3.8 0.4 3.8 0.4

Perceptions of School Climate Domain

 Positive School Orientation–CR 1-4 3.4 0.5 3.4 0.5 3.3 0.5

 Negative School Orientation–CR 1-4 1.5 0.5 1.5 0.4 1.5 0.5

 Student Afraid at School–CR 1-4 2.1 0.9 2.1 0.9 2.1 0.9

 Victimization at School–CR 0-3 0.6 0.7 0.6 0.7 0.6 0.7

 Student sample size—PCR 588 345 243

 Student sample size—CR 589 345 244

 Student sample size—TRS 612 359 253

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

 SD: Standard deviation

No statistically significant differences were found between values for treatment and control groups. Weights, which assign equal 

weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests were 

conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at the 

school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 8. Second Step 

454

Characteristics of Teachers and Schools 

Table 8.8 describes the third-, fourth-, and fifth-grade teachers at the study schools. The majority were White 

non-Hispanic (92%) and female (96%). They had an average of 12.3 years of total teaching experience, and 

more than half (59%) held an advanced or specialist degree. There were no significant differences between 

the treatment and control groups in these characteristics. 

Data regarding school characteristics were drawn from the Common Core of Data in order to compare 

treatment and control schools. There were no significant differences between the two groups of schools in 

terms of student composition (race/ethnicity and school lunch eligibility), number of students enrolled, 

number of full-time teachers, Title I status, or number of years the principal had been at the school (see table 

8.9). In addition, there were no significant differences between treatment and control schools in terms of 

location (urban, suburban, or rural) or lowest and highest grade offered (these data are not shown in a table). 

Table 8.8. Initial characteristics of teachers in sample—SS

Characteristic Total Treatment Control

 Teacher sample size 107 56 51

Gender (percent)

 Male 4.0 ‡ ‡

 Female 96.0 ‡ ‡

Race/ethnicity (percent)

 White (non-Hispanic) 92.1 93.9 90.3

 Other 7.9 6.1 9.7

Number of years teaching experience (mean) 12.3 12.8 11.8

Number of years teaching experience this school (mean) 7.5 8.2 6.7

 

Type of teaching certificate (percent)

 Regular state certificate or advanced professional certificate 97.5 95.1 100.0

 Other 2.5 4.9 0.0

Education (percent)

 Bachelor’s degree 41.0 36.6 45.4

 Advanced degree/other 59.0 63.4 54.6

‡ Reporting standards not met. Values suppressed to protect confidentiality.

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 8. Second Step 

455

Table 8.9. Initial characteristics of schools in sample—SS

Characteristic Total Treatment Control

 School sample size 12 6 6

Student race/ethnicity (percent)

 White (non-Hispanic) 83.7 88.2 79.2

 Black (non-Hispanic) 10.4 7.1 13.6

 Hispanic 2.2 1.2 3.3

 Other 3.7 3.5 3.8

Students eligible for free or reduced-price lunch (percent) 12.2 11.7 12.6

Number of students enrolled (mean) 453.8 491.8 415.7

Number of full-time teachers (mean) 26.1 27.1 25.1

Title I status (percent)

 Title I eligible school 16.7 16.7 16.7

 Schoolwide Title I 0.0 0.0 0.0

Number of years principal has been at this school (mean) 4.2 4.0 4.4

NOTE: No statistically significant differences were found between values for treatment and control groups. Weights, which assign 

equal weight to each school within the program, were used in producing the treatment, control, and overall means. Statistical tests 

were conducted using regressions that included program indicators to account for the sample design and adjusted for clustering at 

the school level. Sample size may differ for some outcomes due to nonresponse.

SOURCE: NCES Common Core of Data (2003-04), the Social and Character Development (SACD) Research Program.

In the Teacher Report on Classroom and School, teachers reported on nine dimensions of school 

environment (these data are not shown in a table): feelings of safety, adequacy of resources, student support, 

freedom to teach as desired, affiliation with and ties to colleagues, innovation regarding new approaches to 

teaching, professional interest, participatory decisionmaking, and work pressure. There were no statistically 

significant differences between treatment and control schools in these reports. 

The Level of SACD in the Schools Near the Beginning of the Study 

During the initial data collection, principals and teachers reported on the SACD activities used in the schools 

and classrooms, the availability of SACD materials, and the professional development provided on SACD. 

Table 8.10 shows that the majority of the school principals reported activities to promote six social and 

character development goals: violence prevention and peace promotion (92%), social and emotional 

development (100%), character education (100%), tolerance and diversity (92%), risk prevention and health 

promotion (75%), and civic responsibility and community service (83%). In addition, all of the principals 

reported activities directed toward behavior management. There were no statistically significant differences 

between the treatment and the control group schools in the percentages reported by principals, although this 

may be due to the small principal sample size. Teachers reported that use of these activities in their 

classrooms ranged from 47 percent to 92 percent, with three significant differences (out of eight 

comparisons) between treatment and control teachers: larger percentages of treatment teachers than control 

teachers reported the use of social and emotional development (89% versus 61%) and character education 

(99% versus 85%) activities in their classrooms, while more control than treatment teachers reported the use 

of civic responsibility and community service activities (69% versus 48%) in their classrooms. With respect to 

the use of schoolwide activities, 62 percent to 96 percent of teachers reported that their schools used such 

activities. There were no significant differences between treatment and control groups. 
Chapter 8. Second Step 

456

Table 8.10. Principal and teacher initial reports on use of SACD programs or activities in 

sample—SS 

SACD program or activity Total Treatment Control

 Principal sample size 12 6 6

 Teacher sample size 107 56 51

Principals reporting that school had programs or activities 

 to promote the following SACD goals (percent)

 Violence prevention and peace promotion 91.7 100.0 83.3

 Social and emotional development 100.0 100.0 100.0

 Character education 100.0 100.0 100.0

 Tolerance and diversity 91.7 100.0 83.3

 Risk prevention and health promotion 75.0 83.3 66.7

 Civic responsibility and community service 83.3 100.0 66.7

 Behavior management 100.0 100.0 100.0

 None of the above 0.0 0.0 0.0

Teachers reporting on using programs or activities in their class 

 to promote the following SACD goals (percent)

 Violence prevention and peace promotion 57.6 62.2 53.1

 Social and emotional development 74.9 89.0** 60.9

 Character education 91.9 98.6* 85.1

 Tolerance and diversity 58.8 60.5 57.0

 Risk prevention and health promotion 46.9 46.6 47.2

 Civic responsibility and community service 58.7 48.1* 69.3

 Behavior management 89.4 91.5 87.3

 None of the above ‡ 0.0 ‡

Teachers reporting schoolwide use of the following

 activities to promote SACD (percent)

 Morning announcements or videos 96.2 94.8 97.6

 School assemblies 65.1 56.2 73.9

 School newspapers or bulletins 84.1 83.0 85.2

 Special school days 61.8 62.4 61.1

 Special events 68.1 65.4 70.8

 Other activities 10.0 13.7 6.4

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 8. Second Step 

457

Teachers reported using a broad range of teaching materials to support SACD activities (table 8.11), including 

teacher guides (85%), student materials (45%), instructional aids (35%), giveaways (54%), and children’s 

literature (38%). There was one statistically significant difference (out of seven comparisons) in the use of 

instructional aids, with more treatment teachers than control teachers reporting their use (52% versus 18%). 

Teachers also reported using a wide variety of teaching strategies (table 8.11). All teachers reported using any 

of the 20 strategies asked about, and teachers used an average of 12 of the strategies. There were 4 significant 

differences (out of 20 comparisons) between treatment and control teachers in the use of these specific 

strategies. More treatment teachers than control teachers reported using role-playing (98% versus 44%), direct 

instruction of SACD (98% versus 73%), skill training (68% versus 34%), and guided visualization (65% 

versus 29%). 
Chapter 8. Second Step 

458

Table 8.11. Teacher initial reports on use of SACD materials and classroom strategies in the 

sample—SS 

SACD material and classroom strategy Total Treatment Control

 Teacher sample size 107 56 51

Teachers using the following materials in conjunction with 

 social and character development activities (percent)

 Teacher guides (manuals, curricula) 84.9 93.5 76.3

 Student materials (workbooks, worksheets) 45.2 47.8 42.6

 Instructional aids (games, software, videos) 34.9 52.1** 17.8

 Giveaways (bookmarks, stickers) 54.0 49.8 58.3

 Children’s literature 37.5 43.7 31.4

 Other types of materials 13.6 11.6 15.5

 Do not use any of the materials listed above 4.8 ‡ ‡

Teachers using any of the strategies listed below to promote 

 social and character development in the classroom (percent) 100.0 100.0 100.0

Number of strategies (listed below) used by teachers to promote 

 social and character development in the classroom (mean) 12.0 12.6 11.3

Teachers using each of the following strategies to promote 

 social and character development (percent)

 Role-playing 71.0 98.3** 43.6

 Cooperative learning 95.6 98.3 92.9

 Peer group discussions 85.7 90.4 81.0

 Direct instruction of social and character development 85.5 97.9* 73.0

 Skill training 51.1 68.4* 33.8

 Incorporating social and character development into 

 academic curriculum 80.1 80.4 79.8

 Parent training 10.3 11.0 9.7

 Parent/community involvement in program development 

 or delivery 36.0 37.8 34.2

 Mentoring 43.3 37.0 49.7

 Good behavior notes sent home daily or weekly 77.2 75.1 79.3

 Presenting role models 69.3 74.6 64.0

 Targeted story reading or writing on SACD themes 67.1 73.8 60.3

 Peer mediation 30.1 26.2 34.0

 Honor roll for positive behavior 42.7 36.6 48.9

 Pledges or recitations on social and character 

 development themes 34.6 35.0 34.1

 Guided visualization 47.0 65.2* 28.9

 Student-led/student-assisted instruction 43.4 38.6 48.2

 Journaling 66.4 68.7 64.1

 Time out for negative behavior 80.0 80.0 80.1

 Daily or weekly rewards for positive behavior 89.5 87.6 91.5

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 8. Second Step 

459

Principals and teachers reported on participation in and amount of SACD training and staff development 

provided over the previous 12 months (table 8.12). Principals reported higher participation rates (100% 

versus 81%) and more hours of training (9.7 versus 7.6) than did teachers. There was a significant difference 

between treatment and control teachers on the percentages reporting participation in SACD training; this 

favored the treatment group (96% versus 67%). The treatment teachers also reported significantly more 

training than did the control teachers on three out of seven specific SACD areas: violence prevention and 

peace promotion (24% versus 9%), social and emotional development (43% versus 13%), and character 

education (90% versus 50%). 

Table 8.12. Principal and teacher initial reports on SACD professional development in sample—

SS 

SACD professional development Total Treatment Control

 Principal sample size 12 6 6

 Teacher sample size 107 56 51

Principals reporting that staff participated in social and character

 development training within the past year (percent) 100.0 100.0 100.0

Teachers reporting participation in social and character

 development training within the past 12 months (percent) 81.4 95.6** 67.2

Number of hours of social and character development training 

 principals report were provided to each staff person last year (mean) 9.7 8.2 11.6

Number of hours of social and character development training

 teachers report receiving during the past 12 months (mean) 7.6 9.3 5.8

Teachers reporting receiving training in the past 12 months

 in the following areas (percent)

 Violence prevention and peace promotion 16.5 24.2* 8.8

 Social and emotional development 27.7 42.5** 12.9

 Character education 69.9 89.8** 50.0

 Tolerance and diversity 11.6 7.7 15.5

 Risk prevention and health promotion 14.7 13.1 16.4

 Civic responsibility and community service 3.1 ‡ ‡

 Behavior management 25.4 19.0 31.8

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 8. Second Step 

460

The data on the initial level of SACD activity emphasize that the control condition was a ―standard practice‖ 

control. Standard practice at the control schools included using SACD activities, materials, and practices, 

along with professional development, at rates and in types and amounts similar to the treatment schools. For 

example, the percentages of teachers who reported using programs or activities to promote specific SACD 

goals ranged from 47 percent to 99 percent in the treatment schools and from 47 percent to 87 percent in the 

control schools. However, the 12 significant differences between the treatment and control schools in the use 

of SACD activities was more than expected by chance (which would be 3 out of 62 comparisons), and in all 

but 1 case the differences favored the treatment group. This may reflect the fact that program implementation 

and program training for staff started before initial data collection. 

Impacts on Use of SACD Activities 

The introduction of the formal SS program would be expected to increase the use of SACD activities in the 

treatment schools in comparison to the control schools. The analysis of this impact was based on the Teacher 

Report on Classroom and School. Every spring, third-, fourth-, and fifth-grade teachers provided information 

through the Teacher Report on Classroom and School about the social and character development activities 

they used in their classrooms. Specifically, information from the Teacher Report on Classroom and School 

was used to determine the difference between treatment and control teachers in these areas: 

1. the use of SACD activities in their classrooms overall and by SACD goal; 

2. the use of materials and strategies to implement the SACD activities within classrooms and within 

the entire school; 

3. the use of staff development to support the teachers; and 

4. teacher support for SACD efforts in the school and the use of practices conducive to the social and 

character development of students. 

Teacher Report on Classroom and School consent and completion rates (table 8.4) led to 87 percent to 97 

percent of teachers overall having data for the 3 years (with a greater percentage of treatment teachers 

providing data at all but one time point). To estimate intervention impacts for each of the outcomes, testing 

of the statistical significance of the differences in means was used. Preliminary analysis indicated little or no 

gains in precision from using covariates. Before testing the mean differences, the data were weighted such 

that each school received equal weight. Standard errors of the impact estimates account for the clustering of 

teachers within schools. In addition, a set of heuristics was applied to determine whether each outcome 

domain was statistically significant after adjustments were made for the multiple tests conducted. 

Use of Activities 

The percentages of control teachers who reported using any SACD activities in their classrooms ranged from 

81 percent to 92 percent over the 3 years (table 8.13, panel 1). For the six individual SACD goals, the 

percentages ranged from 53 percent to 86 percent in Year 1, 51 percent to 89 percent in Year 2, and 45 

percent to 74 percent in Year 3. Control teachers’ use of behavior management activities ranged from 86 

percent to 96 percent over this period. The percentages of control teachers who reported using any SACD 

activities in their classrooms for at least 1 hour per week (panel 2) ranged from 39 percent to 74 percent over 

the 3 years. For the six individual SACD goals, the percentages ranged from 0 percent to 20 percent in Year 

1, 9 percent to 23 percent in Year 2, and 6 percent to 23 percent in Year 3. Control teachers’ reported use of 

behavior management activities ranged from 67 percent to 77 percent over this period. 

For teachers’ reported use of any SACD activity (panels 1 and 2), 48 comparisons were made, with 2 expected 

to be significant by chance. The percentage of treatment teachers using any SACD activity was not 

significantly different from the percentage of control teachers in any year. There were significant impacts on 

the use of specific activities in all 3 years. In Year 1, there were significant impacts on violence prevention and 
Chapter 8. Second Step 

461

peace promotion (impact = 31 percentage points) and on social and emotional development (impact = 39 

percentage points). The same impacts were seen in Years 2 and 3 (impacts ranging from 25 to 40 percentage 

points). In Year 2, there was a significant impact on tolerance and diversity (impact = 23 percentage points) 

and another in Year 3 (impact = 32 percentage points). In Year 3, there was an impact on character education 

(impact = 22 percentage points). A similar pattern was seen in the reports of engagement in these activities 

for at least 1 hour per week. Significant impacts occurred on violence prevention and peace promotion in 

Year 2 (impact = 37 percentage points), social and emotional development activities in Year 1 (impact = 37 

percentage points) and Year 2 (impact = 40 percentage points), character education in Year 1 (impact = 52 

percentage points) and Year 2 (impact = 37 percentage points), and tolerance and diversity in Year 2 (impact 

= 26 percentage points) and Year 3 (impact = 31 percentage points). The use of any activity for at least 1 

hour per week was significantly impacted in Year 1 (impact = 30 percentage points). After the heuristics were 

applied, the domain for engagement in SACD activities showed that SS had statistically significant impacts for 

all 3 years. 

For teachers’ reported use of any named SACD activity (panels 3 and 4), 42 comparisons were made, with 2 

expected to be significant by chance. Eight of the 14 impact estimates were statistically significant in Year 1, 

with a significant impact on the domain as well. Ten of the 14 tested impacts were significant in Year 2, and 

there was an overall impact on the domain. Seven of the 14 tested impacts were significant in Year 3, again 

with a significant impact on the domain. In all 3 years, impacts were seen on violence prevention (impact = 

53, 69, and 63 percentage points), social and emotional development (impact = 73, 80, and 81 percentage 

points), character education (impact = 45, 53, and 61 percentage points), tolerance and diversity (impact = 49, 

45, and 49 percentage points), and any named activity (impact = 40, 49, and 53 percentage points). For any 

named activity at least 1 hour per week, SS had significant impacts on violence prevention in Year 2 (impact 

= 40 percentage points), social and emotional development in Years 1 and 2 (impact = 46 and 50 percentage 

points), character education in Years 1 and 2 (impact = 46 and 45 percentage points), tolerance and diversity 

in Years 2 and 3 (impact = 27 and 28 percentage points), and any named activity in all 3 years (impact = 49, 

46, and 40 percentage points). After the heuristics were applied, SS had statistically significant impacts on the 

domain for engagement in named SACD activities in all 3 years. 
Table 8.13. Impacts on teacher-reported SACD classroom activities—SS 

Panel 1: Engagement in any activities to promote SACD goals1

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 61 52 60 52 59 46

Violence prevention and peace promotion

 (percent) 87.3* 56.2 31.1 0.017 91.5* 66.2 25.3 0.011 88.5* 50.5 38.0 0.005

Social and emotional development

 (percent) 91.9* 53.0 38.9 0.002 96.7* 64.9 31.8 0.002 94.9* 54.8 40.1 0.004

Character education (percent) 98.1^ 86.4 11.8 0.083 100.0 88.8 11.2 † 96.1* 73.8 22.3 0.028

Tolerance and diversity (percent) 80.9^ 62.0 19.0 0.092 85.1* 62.3 22.7 0.002 76.4* 44.7 31.8 0.022

Risk prevention and health promotion

 (percent) 52.7 63.2 -10.4 0.457 57.9 51.1 6.8 0.502 59.3 62.4 -3.1 0.819

Civic responsibility and community service

 (percent) 70.4 73.7 -3.3 0.751 67.3 68.0 -0.6 0.959 70.5 66.7 3.8 0.732

Any SACD goal (percent) 100.0 87.8 12.2 † 100.0 91.5 8.5 † 97.9^ 80.7 17.2 0.076

Behavior management (percent) 89.4 96.2 -6.8 0.187 92.3 85.7 6.6 0.213 85.4 89.8 -4.4 0.614

See notes at end of table.

462

Chapter 8. Second Step
Table 8.13. Impacts on teacher-reported SACD classroom activities—SS—Continued 

Panel 2: Engagement in any activities to promote SACD goals for at least 1 hour per week

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 61 52 60 52 59 46

Violence prevention and peace promotion

 (percent) 42.7 0.0 42.7 † 50.2* 13.1 37.1 0.026 38.2 12.5 25.7 0.109

Social and emotional development

 (percent) 49.8* 12.4 37.3 0.007 55.9* 16.1 39.8 0.005 48.2^ 16.9 31.3 0.079

Character education (percent) 64.8* 13.2 51.5 0.002 60.2* 23.0 37.1 0.040 44.1^ 23.3 20.9 0.094

Tolerance and diversity (percent) 36.1 19.6 16.5 0.199 41.9* 15.6 26.2 0.042 38.5* 7.4 31.1 0.010

Risk prevention and health promotion

 (percent) 13.9 9.7 4.2 0.527 13.3 8.5 4.8 0.530 16.0 6.4 9.6 0.181

Civic responsibility and community service

 (percent) 11.2 11.8 -0.6 0.934 13.3 17.0 -3.7 0.591 13.9^ 10.6 3.3 0.699

Any SACD goal (percent) 69.2* 39.4 29.7 0.045 83.4^ 61.1 22.2 0.052 85.4 73.9 11.5 0.245

Behavior management (percent) 63.1 76.6 -13.5 0.417 79.3 67.4 11.9 0.331 73.0 77.3 -4.3 0.688

See notes at end of table.

463

Chapter 8. Second Step
Table 8.13. Impacts on teacher-reported SACD classroom activities—SS—Continued 

Panel 3: Engagement in activities to promote SACD goals linked to named SACD programs2

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 61 52 60 52 59 46

Violence prevention and peace promotion

 (percent) 68.0* 15.0 53.0 0.002 76.3* 7.3 69.0 0.001 78.4* 15.6 62.8 0.004

Social and emotional development

 (percent) 86.8* 13.9 72.9 0.001 85.4* 5.8 79.6 0.000 93.4* 12.8 80.6 0.001

Character education (percent) 85.9* 41.4 44.5 0.008 82.0* 29.5 52.6 0.003 85.1* 24.4 60.7 0.003

Tolerance and diversity (percent) 62.4* 13.2 49.2 0.009 55.2* 10.2 45.0 0.000 58.9* 10.1 48.8 0.000

Risk prevention and health promotion

 (percent) 28.0 21.6 6.4 0.520 18.3 12.8 5.5 0.487 30.2 15.6 14.6 0.267

Civic responsibility and community service

 (percent) 5.6 0.0 5.6 † ‡ ‡ -3.8 0.310 13.8 6.8 7.0 0.294

Any named activity (percent) 97.0* 57.3 39.7 0.001 91.0* 42.5 48.5 0.001 97.9* 45.0 53.0 0.004

See notes at end of table.

464

Chapter 8. Second Step
Table 8.13. Impacts on teacher-reported SACD classroom activities—SS—Continued

Panel 4: Engagement in activities to promote SACD goals linked to named SACD programs for at least 1 hour per week

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD activity

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 61 52 60 52 59 46

Violence prevention and peace promotion

 (percent) 42.7 0.0 42.7 † ‡* ‡ 40.0 0.002 37.0 8.7 28.3 0.138

Social and emotional development

 (percent) ‡* ‡ 46.1 0.011 ‡* ‡ 50.4 0.002 48.2^ 9.1 39.1 0.083

Character education (percent) 57.3* 11.1 46.2 0.002 54.4* 9.4 45.0 0.002 42.5^ 12.8 29.7 0.089

Tolerance and diversity (percent) 36.1 0.0 36.1 † ‡* ‡ 26.8 0.005 ‡* ‡ 28.1 0.006

Risk prevention and health promotion

 (percent) ‡ ‡ 7.1 0.187 10.1 7.3 2.9 0.688 ‡ ‡ 9.6 0.119

Civic responsibility and community service 

 (percent) ‡ ‡ 2.7 † ‡ ‡ 0.4 0.904 6.5 0.0 6.5 †

Any named activity (percent) 64.6* 15.7 49.0 0.015 59.5* 13.6 45.9 0.001 57.2* 17.5 39.7 0.039

† Not applicable.

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 In all 3 years at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg 

(1995) procedure to adjust significance levels downward to account for the multiple testing of impacts. In Year 2 at least half of the impacts were positive and statistically significant 

and no impact was negative and statistically significant based on univariate statistical tests.

2

In all 3 years at least half of the impacts were positive and statistically significant and no impact was negative and statistically significant based on univariate statistical tests and at 

least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg (1995) procedure to 

adjust significance levels downward to account for the multiple testing of impacts.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

465

Chapter 8. Second Step
Chapter 8. Second Step 

466

Use of Materials and Strategies 

For use of materials and strategies to support SACD goals, 87 comparisons were made, with 4 expected to be 

significant by chance. In Year 1, significant impacts were found for teachers’ use of teacher guides (impact = 

30 percentage points), instructional aids (impact = 36 percentage points), role-playing (impact = 37 

percentage points), skill training (impact = 59 percentage points), incorporating SACD into the academic 

curriculum (impact = 27 percentage points), presenting role models (impact = 17 percentage points), using 

guided visualization (impact = 43 percentage points), and the average number of strategies used (by 2.7 

strategies on average). In Year 2, SS significantly affected the use of teacher guides (impact = 39 percentage 

points), student materials (impact = 30 percentage points), instructional aids (impact = 24 percentage points), 

direct instruction of SACD (impact = 26 percentage points), skill training (impact = 52 percentage points), 

incorporating SACD into academic curriculum (impact = 21 percentage points), guided visualization (impact 

= 40 percentage points), and average number of strategies used (by 2.7 strategies on average). Significant 

impacts in Year 3 affected use of teacher guides (impact = 45 percentage points), instructional aids (impact = 

28 percentage points), skill training (impact = 17 percentage points), parent training (impact = 36 percentage 

points), parent and community involvement (impact = 25 percentage points), guided visualization (impact = 

38 percentage points), and average number of strategies (by 2.7 strategies on average). The SS impact on the 

domain of materials and strategies was significant in all 3 years. 
Table 8.14. Impacts on use of SACD classroom materials and teaching strategies—SS 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD material and teaching strategy

1

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 61 52 60 52 59 46

Use of SACD materials (percent)

 Teacher guides (manuals, curricula) 93.0* 62.7 30.4 0.000 98.1* 59.1 39.0 0.005 94.7* 50.2 44.5 0.003

 Student materials (workbooks or sheets) 70.0 52.1 17.9 0.126 78.7* 48.6 30.1 0.021 67.1 52.9 14.3 0.192

 Instructional aids (games, software, videos) 64.6* 28.7 35.9 0.005 62.8* 38.6 24.2 0.015 61.5* 33.6 27.9 0.005

 Giveaways (bookmarks, stickers) 52.3 66.6 -14.3 0.343 59.1 66.8 -7.7 0.504 44.1 62.3 -18.2 0.203

 Children’s literature 40.8 35.4 5.4 0.765 46.7 38.7 8.0 0.587 40.6 33.9 6.8 0.381

 Other types of materials 8.5 9.7 -1.1 0.843 10.9 12.2 -1.2 0.875 ‡ ‡ 4.7 0.463

 Did not use any of these materials 0.0 13.2 -13.2 † 0.0 14.7 -14.7 † 0.0 14.1 -14.1 †

Use of teaching strategies (percent)

 Role-playing 97.5* 60.2 37.2 0.008 100.0 62.9 37.1 † 100.0 73.1 27.0 †

 Cooperative learning 97.2 92.9 4.3 0.365 98.3 95.5 2.8 0.404 100.0 91.7 8.3 †

 Peer group discussions 95.4 86.1 9.3 0.153 91.4^ 75.4 16.0 0.064 100.0 90.4 9.6 †

 Direct instruction of SACD 100.0 72.8 27.2 † 97.9* 72.4 25.5 0.015 100.0 86.2 13.8 †

 Skill training 88.5* 29.5 59.1 0.000 89.0* 36.7 52.3 0.000 94.4* 77.8 16.5 0.047

 Incorporating SACD into academic 

 curriculum 96.3* 69.1 27.2 0.003 87.4* 66.9 20.5 0.045 94.0 95.1 -1.1 0.825

 Parent training 20.1 9.7 10.5 0.302 ‡ ‡ 2.4 0.601 ‡* ‡ 35.9 0.002

 Parent/community involvement 41.7 38.3 3.5 0.763 36.1^ 18.5 17.6 0.093 62.2* 37.5 24.6 0.020

 Mentoring 45.7 49.4 -3.7 0.820 42.7 42.5 0.2 0.988 74.7 67.6 7.1 0.569

 Good behavior notes sent home daily 

 or weekly 72.9 76.8 -3.9 0.766 73.6 79.2 -5.6 0.626 96.5 91.5 5.0 0.219

 Presenting role models 85.7* 69.1 16.5 0.009 87.6 73.5 14.1 0.117 92.5 77.9 14.6 0.131

See notes at end of table.

467

Chapter 8. Second Step
Table 8.14. Impacts on use of SACD classroom materials and teaching strategies—SS—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD material and teaching strategy

1

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

Use of teaching strategies (percent)—

 Continued

 Targeted story reading or writing on social 

 and character development themes 75.3 60.1 15.2 0.266 67.6 55.9 11.7 0.303 89.9 85.9 4.0 0.614

 Peer mediation 38.8 37.0 1.9 0.887 38.0 41.4 -3.4 0.822 52.4 49.4 3.0 0.833

 Honor roll for positive behavior 38.6 54.5 -15.8 0.371 58.2 53.0 5.2 0.749 68.1 57.0 11.1 0.554

 Pledges or recitations on social and 

 character development themes 43.2 29.2 14.0 0.451 35.3 32.1 3.2 0.838 56.8 40.9 16.0 0.317

 Guided visualization 73.9* 30.8 43.1 0.000 71.3* 31.5 39.9 0.006 92.0* 54.6 37.5 0.001

 Student-led/student-assisted instruction 46.4 38.3 8.1 0.553 55.1^ 33.3 21.8 0.051 63.4 71.9 -8.5 0.362

 Journaling 68.7 50.0 18.7 0.099 67.4 50.1 17.2 0.106 84.3 68.9 15.4 0.264

 Time out for negative behavior 74.9 86.1 -11.2 0.331 82.9 90.1 -7.2 0.185 96.3 97.1 -0.8 0.857

 Daily or weekly rewards for positive 

 behavior 89.6 80.0 9.5 0.192 89.1 88.6 0.4 0.949 100.0 97.2 2.8 †

 Any strategy 100.0 98.6 1.4 † 100.0 100.0 0.0 † 100.0 97.2 2.8 †

 Number of strategies (mean) 13.8* 11.1 2.7 0.001 13.7* 10.9 2.7 0.002 16.3* 13.6 2.7 0.000

† Not applicable.

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 In all 3 years at least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after applying the Benjamini-Hochberg 

(1995) procedure to adjust significance levels downward to account for the multiple testing of impacts. 

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

468

Chapter 8. Second Step
Chapter 8. Second Step 

469 

Regarding the use of schoolwide strategies, 18 comparisons were made between treatment and control 

teacher reports, with 1 expected to be significant by chance. There was no SS impact on teachers’ reports of 

their schools’ use of schoolwide strategies in any of the years (these data are not shown in a table), nor was 

there a significant impact on the domain in any year. 

Participation in Professional Development 

Regarding reported participation in professional development, 27 comparisons were made over 3 years, with 

1 expected to be significant by chance. In Year 1, SS had an impact on SACD training (impact = 32 

percentage points) and training in the specific goal of character education (impact = 29 percentage points). 

No significant impacts were found on the domain in any of the 3 years. 
Table 8.15. Impacts on teacher-reported SACD professional development—SS 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

SACD professional development

Treatment Control Impact p-value Treatment Control Impact p-value Treatment Control Impact p-value 

 Teacher sample size 61 52 60 52 59 46

SACD training in past 12 months (percent) 87.0* 54.7 32.3 0.009 60.2 48.1 12.1 0.295 44.9 42.8 2.1 0.864

Hours of SACD training (mean) 6.4 4.5 1.9 0.302 4.3 4.8 -0.4 0.859 3.1 2.9 0.1 0.934

Training by goal (percent)

 Violence prevention and peace promotion 21.0 13.5 7.5 0.267 10.6 10.4 0.2 0.973 12.9 9.0 3.9 0.581

 Social and emotional development 32.1 16.8 15.3 0.208 14.4 18.5 -4.1 0.690 ‡ ‡ 10.6 0.140

 Character education 75.5* 46.2 29.3 0.029 43.7 31.5 12.2 0.267 24.4 27.9 -3.4 0.792

 Tolerance and diversity 9.7 9.0 0.7 0.919 13.5 18.4 -4.9 0.647 4.5 8.3 -3.8 0.394

 Risk prevention and health promotion 20.2 25.2 -5.0 0.438 8.6 19.9 -11.3 0.178 11.0 14.6 -3.6 0.614

 Civic responsibility and community service ‡ ‡ 5.0 0.294 ‡ ‡ 5.0 0.358 ‡ ‡ -2.8 0.502

 Behavior management 13.8 27.9 -14.1 0.269 16.2 17.3 -1.1 0.891 4.9^ 18.3 -13.5 0.073

‡ Reporting standards not met. Values suppressed to protect confidentiality.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

NOTE: Weights, which assign equal weight to each school within the program, were used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.

470

Chapter 8. Second Step
Chapter 8. Second Step 

471

Attitudes and Practices 

Teachers reported on their enthusiasm for SACD efforts in their schools (these data are not shown in a table) 

by indicating enthusiasm, cooperation, or open dislike. They also reported on the SACD practices of teachers 

and staff members in their schools (these data are not shown in a table). The practices included modeling 

positive character and behavior traits with students and fellow teachers, involving students in making 

decisions, giving students a voice in school governance, the school encouraging parent involvement in 

children’s social and character development, and using developmentally appropriate discipline strategies 

rather than punishment for misbehavior. Twenty-seven comparisons were made over 3 years, with 1 expected 

to be significant by chance. Out of 9 comparisons made in Year 2, there was 1 significant impact on reported 

enthusiasm (impact = 24 percentage points). There was no impact on the domain in any year. 

Year-by-Year Impacts on Students and Perceptions of School Climate 

The primary research question for the SS evaluation was this: 

What is the average effect of the SS program on children’s social and emotional competence, behavior, 

academics, and perceptions of school climate? 

The first approach to answering this question was to examine the year-by-year impacts of the SS program on 

these student and school climate outcomes over the 3 years as the students progressed from third through 

fifth grades. 

Equation (2) (described in chapter 1) was estimated to provide the SS impacts on the 20 outcomes using data 

from the 12 treatment and control schools. For the SS evaluation, equation (2) excluded the program fixed 

effects (p) and included program-specific covariates and random school effects covariates. Table 8.16 lists 

the covariates used with outcomes from each report in the SS analysis. 
Chapter 8. Second Step 

472

Table 8.16. Covariates used with outcomes from each report for analysis—SS

Potential covariate

CR 

outcome

PCR 

outcome

TRS 

outcome

TRCS 

outcome

 Total number 11 28 21 8

Child-reported

 Female   

 Hispanic   

 Black (non-Hispanic)   

 Other ethnicity   

 Age in years   

 Scales

 Afraid at School  

 Altruistic Behavior

 Empathy  

 Engagement with Learning

 Negative School Orientation  

 Normative Beliefs About Aggression  

 Sense of School as a Community   

 Problem Behavior

 Self-Efficacy for Peer Interactions

 Victimization at School

Primary caregiver-reported

 Age in years 

 Completed high school or equivalent   

 Some College   

 Bachelor’s or higher degree   

 Highest level of education in household 

 Completed high school or equivalent 

 Some college 

 Bachelor’s or higher degree 

 Mother present in home life 

 Mother and father present

 Respondent someone other than mother or father 

 Number of people in household 

 Household income: $20,000 to $40,000 

 Household income: $40,000 to $60,000 

 Household income: More than $60,000 

 Income-to-poverty-threshold ratio: Below 135 percent

 Income-to-poverty-threshold ratio: 135-185 percent

 Full-time employment 

 Part-time employment 

See notes at end of table.
Chapter 8. Second Step 

473

Table 8.16. Covariates used with outcomes from each report for analysis—SS—Continued

Table 8.16.

Potential covariate

CR 

outcome

PCR 

outcome

TRS 

outcome

TRCS 

outcome

 Parental scales

 APQ-Poor Monitoring and Supervision Subscale

 APQ-Positive Parenting Subscale  

 Child-Centered Social Control

 Confusion, Hubbub, and Order

 Community Resources

 Community Risk 

 Parent and Teacher Involvement

 Child scales

 Altruistic Behavior   

 Positive Social Behavior 

 Problem Behavior  

Teacher-reported

 Female 

 Hispanic 

 Black (non-Hispanic) 

 Other ethnicity 

 Total teaching experience 

 Total experience in current school

 Regular certificate 

 Other certificate 

 Highest degree–bachelor’s 

 Child scales

 Academic Competence and Motivation   

 ADHD-Related Behavior 

 Altruistic Behavior  

 Positive Social Behavior

 Problem Behavior

 Parent and Teacher Involvement

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

 APQ: Alabama Parenting Questionnaire

: Covariate used

 Blank cell: Covariate not used

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 8. Second Step 

474

To assess the statistical power of the program-level impact estimates, minimum detectable impacts in effect 

size units (MDES) for each outcome measure were calculated for the SS evaluation (table 8.17). MDES 

represent the smallest impacts in effect size (standard deviation) units that can be detected with a high 

probability (80%). MDES are primarily a function of study sample sizes, the degrees of freedom available for 

statistical tests, and design effects due to clustering (Schochet 2005). For the SS evaluation, the MDES ranged 

from 0.099 to 0.526 for the child-level outcomes based on the Child, Caregiver, and Teacher Report on 

Student and from 0.314 to 0.694 for the school climate outcomes based on the Teacher Report on Classroom 

and School. In general, the MDES for the school climate outcomes were larger than those for the child-level 

outcomes.
Chapter 8. Second Step 

475

Table 8.17. Adjusted minimum detectable effect sizes for impact evaluation—SS 

Outcome measure–Report Year 1 Year 2 Year 3

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 0.156 0.252 0.105

 Normative Beliefs About Aggression–CR 0.103 0.099 0.099

 Empathy–CR 0.156 0.170 0.200

Behavior Domain

 Altruistic Behavior–CR 0.103 0.161 0.133

 Altruistic Behavior–PCR 0.110 0.132 0.139

 Altruistic Behavior–TRS 0.526 0.512 0.481

 Positive Social Behavior–PCR 0.142 0.106 0.108

 Positive Social Behavior–TRS 0.279 0.455 0.356

 Problem Behavior–CR 0.106 0.166 0.099

 Problem Behavior–PCR 0.183 0.142 0.149

 Problem Behavior–TRS 0.275 0.404 0.348

 ADHD-Related Behavior–TRS 0.264 0.296 0.342

Academics Domain

 Engagement with Learning–CR 0.103 0.106 0.099

 Academic Competence and Motivation–TRS 0.103 0.167 0.277

Perceptions of School Climate Domain

 Positive School Orientation–CR 0.192 0.259 0.270

 Negative School Orientation–CR 0.201 0.198 0.123

 Student Afraid at School–CR 0.103 0.197 0.204

 Victimization at School–CR 0.103 0.185 0.209

 Feelings of Safety–TRCS 0.447 0.550 0.481

 Student Support for Teachers–TRCS 0.694 0.314 0.609

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The minimum detectable effect (MDE) formula used in the calculations is as follows:

 

 

 

 

 

 

 

 

 

 

 

 

 

 

where sT and sC are the number of treatment and comparison schools; nT and nC are the average number of students per 

classroom; 1 is the intraclass correlation (ICC) at the school level; and factor(df) is a constant that depends on the number of 

degrees of freedom (df) available for analysis (and is 2.802 for the pooled analysis).

SOURCE: The Social and Character Development (SACD) Research Program.
Chapter 8. Second Step 

476

Table 8.18 provides the estimates of the SS impacts on each of the 20 outcomes over each of the 3 years (60 

impacts in total, with 3 expected to be statistically significant by chance). Of the 60 results, 1 was statistically 

significant: a detrimental impact was found on fourth-graders’ Positive Social Behavior (Primary Caregiver 

Report, effect size [ES] = -0.14). A substantively important but nonsignificant beneficial impact was found in 

all 3 years for Feelings of Safety (Teacher Report on Classroom and School, ES = 0.37, 0.39, and 0.52). 

Application of the heuristics to adjust for multiple comparisons within each outcome domain indicates that 

SS had a statistically significant detrimental impact on the Social and Emotional Competence domain in Year 3. 
Table 8.18. Impacts on child and school outcomes—SS 

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Social and Emotional Competence Domain1

 Self-Efficacy for Peer Interactions–CR (+) 3.09 3.03 0.09 0.337 3.20 3.24 -0.08 0.562 3.24 3.31 -0.12 0.146

 Normative Beliefs About Aggression–CR (-) 1.22 1.20 0.06 0.539 1.23 1.28 -0.10 0.228 1.32 1.26 0.15 0.130

 Empathy–CR (+) 2.38 2.37 0.04 0.649 2.25 2.28 -0.07 0.463 2.20 2.26 -0.14 0.258

Behavior Domain

 Altruistic Behavior–CR (+) 1.03 1.10 -0.09 0.249 0.89^ 1.00 -0.18 0.094 0.88 0.95 -0.12 0.184

 Altruistic Behavior–PCR (+) 2.08 2.10 -0.03 0.671 2.06 2.15 -0.13 0.144 2.02^ 2.13 -0.16 0.098

 Altruistic Behavior–TRS (+) 1.37 1.40 -0.08 0.757 1.30 1.29 0.03 0.919 1.27 1.32 -0.10 0.675

 Positive Social Behavior–PCR (+) 3.18 3.17 0.03 0.739 3.20* 3.28 -0.14 0.050 3.30 3.30 0.01 0.880

 Positive Social Behavior–TRS (+) 3.40 3.32 0.12 0.252 3.25 3.36 -0.17 0.435 3.28 3.31 -0.04 0.800

 Problem Behavior–CR (-) 0.12 0.13 -0.04 0.659 0.20 0.17 0.09 0.406 0.26 0.22 0.12 0.201

 Problem Behavior–PCR (-) 1.53 1.52 0.03 0.727 1.48 1.50 -0.05 0.566 1.47 1.50 -0.10 0.280

 Problem Behavior–TRS (-) 1.26 1.27 -0.02 0.896 1.29 1.26 0.10 0.599 1.33 1.32 0.02 0.905

 ADHD-Related Behavior–TRS (-) 1.59 1.55 0.07 0.517 1.60 1.53 0.13 0.412 1.57 1.57 -0.01 0.969

Academics Domain

 Engagement with Learning–CR (+) 3.73 3.74 -0.03 0.757 3.70 3.74 -0.10 0.210 3.59 3.64 -0.12 0.169

 Academic Competence and Motivation–

 TRS (+) 3.34 3.40 -0.06 0.161 3.34^ 3.49 -0.16 0.055 3.50 3.32 0.19 0.137

See notes at end of table.

477

Chapter 8. Second Step
Table 8.18. Impacts on child and school outcomes—SS—Continued

Year 1 Year 2 Year 3

(Spring 3rd grade) (Spring 4th grade) (Spring 5th grade)

Scale–Report 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Treatment ControlEffect 

size p-value 

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 3.13 3.08 0.09 0.425 2.91 2.89 0.05 0.763 2.83 2.80 0.06 0.720

 Negative School Orientation–CR (-) 1.62 1.65 -0.06 0.571 1.73 1.72 0.03 0.780 1.87 1.83 0.08 0.375

 Student Afraid at School–CR (-) 1.93 2.00 -0.08 0.332 1.87 1.95 -0.10 0.402 1.86 1.82 0.05 0.691

 Victimization at School–CR (-) 0.56 0.62 -0.09 0.283 0.55 0.62 -0.11 0.306 0.62 0.60 0.03 0.833

 Feelings of Safety–TRCS (+) 4.19 4.04 0.37° 0.216 4.10 3.86 0.39° 0.197 4.02^ 3.63 0.52° 0.062

 Student Support for Teachers–TRCS (+) 4.89 4.11 0.13 0.663 4.27 4.22 0.08 0.678 4.17 4.12 0.08 0.768

* Treatment group significantly different from control group at the .05 level.

^ Treatment group significantly different from control group at the .10 to > .05 level.

o Substantive (but nonsignificant at .05 level) effect size of ≥ .25 or ≤ -.25.

1

Impact on domain found statistically significant and detrimental in Year 3 based on the fourth heuristic in which the statistical model used to estimate impacts on the individual 

outcomes was re-estimated using a composite of all the outcome variables under a domain. The domain was found significant if the impact on the composite was significant. The 

composite was formed by standardizing each outcome variable using its standard deviation, combining the values of the outcome variables, and taking the average of the final value.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The - signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models where each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. See table 1.5 for 

information about the measures used to create the outcome variables. The effect size was calculated by dividing the estimated impact by the standard deviation of the outcome measure 

for the control group. The number of results found significant was no more than expected by chance.

SOURCE: The Social and Character Development (SACD) Research Program.

478

Chapter 8. Second Step
Chapter 8. Second Step 

479

Impacts on Child Outcomes Over Time 

SS impacts on the child outcomes over time were estimated using growth curve models by examining 

treatment and control group differences in the trajectories of student outcomes during the follow-up period 

while accounting for clustering at the school level. The growth curve models are estimated using a three-level 

hierarchical linear model, where Level 1 corresponds to time, Level 2 to students, and Level 3 to schools 

(described in chapter 1). 

Table 8.19 provides the estimates of the program’s impacts on the growth in student outcomes over the 3 

years. The estimated impacts ranged in effect size units (absolute value) from 0.00 to 0.19. Three of the 18 

(with 1 expected to be significant by chance) estimated SS impacts on the trajectories of child outcomes were 

statistically significant and showed a detrimental impact of the program on growth: Empathy (Child Report, 

ES = -0.13), Positive Social Behavior (Teacher Report on Student, ES = -0.19), and Engagement with 

Learning (Child Report, ES = -0.09). 
Table 8.19. Impacts on growth of child outcomes—SS 

Average growth in the score per year1

Scale–Report 

Mean score at

implementation2Treatment Control

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 2.95 0.12 0.15 -0.02 -0.03 0.03 0.495

 Normative Beliefs About Aggression–CR (-) 1.20 0.05^ 0.01 0.03 0.07 0.02 0.090

 Empathy–CR (+) 2.43 -0.11* -0.05 -0.06 -0.13 0.02 0.028

Behavior Domain

 Altruistic Behavior–CR (+) 1.21 -0.13 -0.12 -0.01 -0.01 0.04 0.755

 Altruistic Behavior–PCR (+) 2.14 -0.05 -0.02 -0.04 -0.04 0.02 0.159

 Altruistic Behavior–TRS (+) 1.42 -0.04 -0.07 0.03 0.05 0.06 0.666

 Positive Social Behavior–PCR (+) 3.12 0.05 0.08 -0.03 -0.04 0.01 0.101

 Positive Social Behavior–TRS (+) 3.39 -0.10* 0.06 -0.16 -0.19 0.06 0.019

 Problem Behavior–CR (-) 0.09 0.06 0.05 0.00 0.01 0.01 0.711

 Problem Behavior–PCR (-) 1.53 -0.02 -0.02 0.00 0.00 0.01 0.848

 Problem Behavior–TRS (-) 1.20 0.08^ 0.01 0.07 0.17 0.04 0.086

 ADHD-Related Behavior–TRS (-) 1.56 0.02 -0.02 0.05 0.06 0.04 0.285

Academics Domain

 Engagement with Learning–CR (+) 3.80 -0.09* -0.04 -0.05* -0.09 0.02 0.021

 Academic Competence and Motivation–TRS (+) 3.35 0.04 0.03 0.01 0.01 0.04 0.808

See notes at end of table.

480

Chapter 8. Second Step
Table 8.19. Impacts on growth of child outcomes—SS—Continued

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2Treatment Control

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 3.32 -0.22 -0.19 -0.04 -0.05 0.04 0.339

 Negative School Orientation–CR (-) 1.52 0.14 0.12 0.02 0.02 0.02 0.464

 Student Afraid at School–CR (-) 2.08 -0.12 -0.09 -0.03 -0.03 0.04 0.403

 Victimization at School–CR (-) 0.60 0.00 0.00 0.00 0.00 0.03 0.930

* Treatment group significantly different from control group at the .05 level.

^ Treatment group significantly different from control group at the .10 to > .05 level.

1 Pertains to the estimated slope of the outcome for the treatment or control groups.

2 The average score at implementation is calculated across treatment and control groups, using regression models for adjustment on covariates. 

3 Estimated difference between the slope of the treatment and control groups.

4 Effect size: the slope of the treatment group minus the slope of the control group divided by the standard deviation of the outcome for the program’s control group (the standard 

deviation is calculated without accounting for school-level clustering or regression adjustments).

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The - signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using HLM 6.06. Sample weights were used in all analyses to give (1) 

each school equal weight in each program (within each time period) and (2) each time period equal weight within the analysis. See table 1.5 for information about the measures used 

to create the outcome variables. 

SOURCE: The Social and Character Development (SACD) Research Program.

481

Chapter 8. Second Step
Chapter 8. Second Step 

482 

Summary 

As part of the Social and Character Development (SACD) Research Program, researchers at the Maryland site 

evaluated the SS program. This program is designed to promote social competence and reduce children’s 

social and emotional problems. Twelve public schools in a single school district in Maryland were recruited by 

the Maryland research team and randomly assigned to treatment and control conditions to determine the 

impact of SS on social and character development activities in the schools and on the child outcome domains 

of Social and Emotional Competence, Behavior, Academics, and Perceptions of School Climate. 

Analyses of the initial characteristics of the sample (students, caregivers, communities, teachers, and schools) 

indicated that randomization to treatment and control status produced groups that were relatively similar at 

the start of the study (with 1 out of 84 comparisons statistically significantly different, fewer than would be 

expected by chance). The data on the initial level of SACD activity led to two findings. First, treatment 

teachers reported greater use of and training in SACD activities than control teachers, and they did so more 

often than would be expected by chance (11 out of 62 comparisons, with 3 expected to be significant by 

chance). There are two potential causes for this finding, and the analysis cannot be used to determine whether 

the reason for such a difference was that the two groups did differ on their initial use of SACD activities (i.e., 

that randomization did not create similar treatment and control groups) or whether the fact that the training 

of all treatment teachers and the implementation of the SS program began before the initial data were 

collected (by 5 weeks) influenced the teacher reports. Because it is likely (though unproven) that the training 

and implementation affected the teacher reports, these data were not considered appropriate for use as a 

baseline measure of SACD activities and training in the treatment schools. 

Second, these data indicate that the control condition for the SACD project was not a ―no treatment‖ control 

but a ―standard practice‖ control. Because the control teachers were not affected by the implementation of 

the SACD programs before data collection, their reports reflect standard practice in the control schools. 

Standard practice at the control schools included reports of 47 percent to 87 percent of teachers using any 

SACD activities, 92 percent of teachers using specific materials in conjunction with these activities, 100 

percent using at least one of the specified instructional strategies, and 67 percent participating in SACD 

training over the past 12 months. 

Analyses of the SS impacts on level of SACD activities in the schools revealed impacts on the use of SACD 

activities (39 out of 90) and materials and strategies (23 out of 87) across the 3 years. There were impacts on 

two of the professional development activities for treatment teachers in Year 1 (two out of nine). These same 

measures in the control schools across the 3 years of the study confirmed that use of these activities in the 

control schools constituted their standard practice. 

Of the 20 child-level outcomes representing the four domains of Social and Emotional Competence, 

Behavior, Academics, and Perceptions of School Climate assessed in each of the 3 years of the study (a total 

of 60 results), 1 showed a statistically significant detrimental impact of the SS intervention. A growth curve 

analysis was used to analyze the change over time in these same outcomes between initial data collection and 

the final outcome data collection at the end of the study. Three of the 18 child-level outcomes assessed 

showed a significant detrimental impact of the SS program. 

The SACD evaluation did not find evidence to support the hypothesis that SS had beneficial impacts on 

students’ social and character development. Such results could be caused by the inability of the program to 

cause such change, possibly because the theory of action for the program is incomplete or the activities to 

carry out that theory are not effective. 

However, these results may also be due to the inability of the evaluation to observe such a change due to the 

control condition, the level of nonparticipation, or the sample size. The control schools continued using their 

standard SACD activities, and these turned out to be high in quantity and broad in scope. While SS had a 

significant positive impact on the amount and types of SACD activities, the resulting difference in the amount 
Chapter 8. Second Step 

483

of SACD activities between the treatment and control schools may not have been large enough to cause 

significant differences in the student outcomes. Second, 29 percent to 38 percent of the students in the 

sample universe did not take part (depending on year) because of nonconsent or noncompletion of the 

surveys. As a determination could not be made as to whether the students not taking part differed 

significantly from those who did take part, the evaluation’s results are valid only for the students who took 

part. If the students not taking part were different, and if they would have responded better to SS than to the 

SACD activities occurring in the control schools, then the evaluation could have underestimated the 

program’s impact. Third, the sample size of 12 schools and the resulting higher MDES compared to those for 

the multiprogram evaluation may have reduced the likelihood of detecting statistically significant effects. 

However, it should be noted that 63 percent of the MDES for the 60 outcomes used in the year-by-year 

analysis were below 0.25 (58% were below 0.20). In addition, only 3 of the 60 outcomes were found to be 

substantively important; however, they were all beneficial and referred to the same outcome (Feelings of 

Safety). 
This page is intentionally blank.
485

References 

Achenbach, T.M. (1991). Manual for Teacher’s Report Form and 1991 Profile. Burlington: University of Vermont, 

Department of Psychiatry. 

American Psychiatric Association. (2000). Diagnostic and Statistical Manual of Mental Disorders (4th ed., revised 

text). Washington, DC: American Psychiatric Association. 

Battistich, V., Solomon, D., Watson, M., and Schaps, E. (1997). Caring School Communities. Educational 

Psychologist, 32(3): 137-151.

Benjamini, Y., and Hochberg, Y. (1995). Controlling the False Discovery Rate—A New and Powerful 

Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B, 59: 289-300.

Bennett, K.J., Brown, K.S., Boyle, M., Racine, Y., and Offord, D. (2003). Does Low Reading Achievement at 

School Entry Cause Conduct Problems? Social Science and Medicine, 56(11): 2443-2448. 

Brooks-Gunn, J., and Duncan, G.J. (1997). The Effects of Poverty on Children. Children and Poverty, 7(2): 55-

71.

Bryk, A., and Raudenbush, S. (1992). Hierarchical Linear Models for Social and Behavioral Research: Applications and 

Data Analysis Methods. Newbury Park, CA: Sage. 

Carlson, E.A., Sroufe, L.A., Collins, W.A., Jimerson, S., Weinfield, N., Hennighausen, K., Egeland, B., Hyson, 

D.M., Anderson, F., and Meyer, S.E. (1999). Early Environmental Support and Elementary School 

Adjustment as Predictors of School Adjustment in Middle Adolescence. Journal of Adolescent Research, 

14(1): 72-94.

Caspi, A., Taylor, A., Moffitt, T.E., and Plomin, R. (2000). Neighborhood Deprivation Affects Children’s 

Mental Health: Environmental Risks Identified in a Genetic Design. Psychological Science, 11: 338-342.

Catalano, R.F., Berglund, M.L., Ryan, J.A.M., Lonczak, H.S., and Hawkins, J.D. (2004). Positive Youth 

Development in the United States: Research Findings on Evaluations of Positive Youth 

Development Programs. The ANNALS of the American Academy of Political and Social Science, 591(1): 98-

124.

Conduct Problems Prevention Research Group (CPPRG). (1991). Parent & Teacher Involvement Measure-Parent. 

Retrieved from http://www.fasttrackproject.org/techrept/p/ptp. 

Conduct Problems Prevention Research Group (CPPRG). (1999). Initial Impact of the Fast Track Prevention 

Trial for Conduct Problems II: Classroom Effects. Journal of Consulting and Clinical Psychology, 67(5): 

648-657.

Crandall, D., Loucks-Horsley, S., Baucher, J., Schmidt, W., Eisman, J., Cox, P., Miles, M., Huberman, A., 

Taylor, B., Goldberg, J., Shive, G., Thompson, C., and Taylor, J. (1982). People, Policies and Practices: 

Examining the Chain of School Improvement, Vols. 1-10. Andover, MA: Network. 

Crosse, S., Burr, M., Cantor, D., Hagen, C.A., and Hantman, I. (2001). Wide Scope, Questionable Quality: Drug 

and Violence Prevention Efforts in American Schools: Report on the Study on School Violence and Prevention. 

Washington, DC: Planning and Evaluation Service, U.S. Department of Education. 

Dahlberg, L.L. (1998). Youth Violence in the United States: Major Trends, Risk Factors, and Prevention 

Approaches. American Journal of Preventive Medicine, 14(4): 259-272.
References 

486

Dahlberg, L.L., and Simon, T.R. (2006). Predicting and Preventing Youth Violence: Developmental Pathways 

and Risk. In J. Lutzker (Ed.), Preventing Violence: Research and Evidence-Based Intervention Strategies (pp. 97-

124). Washington, DC: American Psychological Association. 

Dahlberg, L.L., Toal, S.B., and Behrens, C.B. (1998). Measuring Violence-Related Attitudes, Beliefs, and Behaviors 

Among Youths: A Compendium of Assessment Tools. Atlanta, GA: Division of Violence Prevention, 

National Center for Injury Prevention, Centers for Disease Control and Prevention. 

Damon, W. (1999). The Moral Development of Children. Scientific American, 281(2): 72-79.

Dempster, A.P., Laird, N.M., and Rubin, D.B. (1977). Maximum Likelihood Estimation From Incomplete 

Data via the EM Algorithm. Journal of the Royal Statistical Society: Series B, 39: 1-38. 

Dinkes, R., Cataldi, E.F., and Lin-Kelly, W. (2007). Indicators of School Crime and Safety: 2007. (NCES 2008021, 

NCJ 219553). U.S. Departments of Education and Justice. Washington, DC: U.S. Government 

Printing Office. 

Dodge, K.A., and Coie, J.D. (1987). Social-Information Processing Factors in Reactive and Proactive 

Aggression in Children’s Peer Groups. Journal of Personality and Social Psychology, 53: 1146-1158.

Dodge, K.A., and Pettit, G.S. (2003). A Biopsychosocial Model of the Development of Chronic Conduct 

Problems in Adolescence. Developmental Psychology, 39: 349-371.

Dunford, F.W., and Elliott, D.S. (1984). Identifying Career Offenders Using Self-Reported Data. Journal of 

Research in Crime and Delinquency, 21: 57-86.

Eisenberg, N. (2000). Emotion, Regulation, and Moral Development. Annual Review of Psychology, 51: 665-697.

Emler, N. (1996). How Can We Decide Whether Moral Education Works? Journal of Moral Education, 25: 117-

126.

Farmer, T.W. (2000). Social Dynamics of Aggressive and Disruptive Behavior in School: Implications for 

Behavior Consultation. Journal of Educational and Psychological Consultation, 11: 299-322. 

Farmer, T.W., Goforth, J.B., Hives, J., Aaron, A., Hunter, F., and Sgmatto, A. (2006). Competence 

Enhancement Behavior Management. Preventing School Failure, 50: 39-44. 

Farrell, A.D., and Camou, S. (2006). School-Based Interventions for Youth Violence Prevention. In R.J. 

Lutzker (Ed.), Preventing Violence: Research and Evidence-Based Intervention Strategies (pp. 125-146). 

Washington, DC: American Psychological Association. 

Farrell, A.D., and Meyer, A.L. (1997). The Effectiveness of a School-Based Curriculum for Reducing 

Violence Among Urban Sixth-Grade Students. American Journal of Public Health, 87: 979-984.

Farrington, D.P. (1989). Early Predictors of Adolescent Aggression and Adult Violence. Violence and Victims, 

4: 79-100. 

Farrington, D.P., and Hawkins, J.D. (1991). Predicting Participation, Early Onset and Later Persistence in 

Officially Recorded Offending. Criminal Behavior and Mental Health, 1: 1-33.

Flannery, D.J., Vazsonyi, A.T., Liau, A.K., Guo, S., Powell, K.E., Atha, H., Vesterdal, W., and Embry, D. 

(2003). Initial Behavior Outcomes for the PeaceBuilders Universal School-Based Violence 

Prevention Program. Developmental Psychology, 39: 292-308.

Flay, B.R. (2002). Positive Youth Development Requires Comprehensive Health Promotion Programs. 

American Journal of Health Behavior, 26(6): 407-424.

Flay, B.R., Allred, C.G., and Ordway, N. (2001). Effects of the Positive Action Program on Achievement and 

Discipline: Two Matched-Control Comparisons. Prevention Science, 2: 71-89.
References 

487

Flay, B.R., Graumlich, S., Segawa, E. Burns, J.L., and Holliday, M.Y. (2004). Effects of 2 Prevention 

Programs on High-Risk Behaviors Among African American Youth: A Randomized Trial. Archives of 

Pediatric & Adolescent Medicine, 158: 377-384.

Forehand, R., Brody, G., Armistead, L., Dorsey, S., Morse, E., Morse, P., and Stock, M. (2000). The Role of 

Community Risks and Resources in the Psychosocial Adjustment of At-Risk Children: An 

Examination Across Two Community Contexts and Two Informants. Behavior Therapy, 31: 395-414.

Fors, S.W., Crepaz, N., and Hayes, D.M. (1999). Key Factors That Protect Against Health Risks in Youth: 

Further Evidence. American Journal of Health Behavior, 23: 368-380.

Fraser, M.W., Nash, J.K., Galinsky, M.J., and Darwin, K.E. (2000). Making Choices: Social Problem-Solving Skills 

for Children. Washington, DC: NASW Press. 

Funk, J., Elliott, R., Bechtoldt, H., Pasold, T., and Tsavoussis, A. (2003). The Attitudes Toward Violence 

Scale. Journal of Interpersonal Violence, 18: 186-196.

Furrer C., and Skinner, E. (2003). Sense of Relatedness as a Factor in Children’s Academic Engagement and 

Performance. Journal of Educational Psychology, 95: 148-162.

Gershoff, E.T., and Aber, J.L. (2006). Neighborhoods and Schools: Contexts and Consequences for the 

Mental Health and Risk Behaviors of Children and Youth. In L. Balta and C. Tamis-LeMonda (Eds.), 

Child Psychology: A Handbook of Issues (2nd ed.) (pp. 611-645). New York: Psychology Press/Taylor & 

Francis. 

Gorman-Smith, D., Tolan, P.H., Henry, D.B., and Florsheim, P. (2000). Patterns of Family Functioning and 

Adolescent Outcomes Among Urban African American and Mexican American Families. Journal of 

Family Psychology, 14: 436-457.

Gorman-Smith, D., Tolan, P.H., Zelli, A., and Huesmann, L.R. (1996). The Relation of Family Functioning to 

Violence Among Inner-City Minority Youths. Journal of Family Psychology, 10: 115-129.

Gottfredson, G.D., and Gottfredson, D.C. (2001). What Schools Do to Prevent Problem Behavior and 

Promote Safe Environments. Journal of Educational and Psychological Consultation, 12: 313-344.

Greenberg, M.T. (2004). Current and Future Challenges in School-Based Prevention: The Researcher 

Perspective. Prevention Science, 5: 5-13.

Greenberg, M.T., Weissberg, R.P., Utne O’Brian, M., Zins, J.E., Fredericks, L., Resnik, H., and Elias, M.J. 

(2003). Enhancing School-Based Prevention and Youth Development Through Coordinated Social, 

Emotional, and Academic Learning. American Psychologist, 58: 466-474.

Gresham, F.M., and Elliott, S.N. (1990). Social Skills Rating System. Circle Pines, MN: American Guidance 

Service.

Griffith, J. (1996). Relation of Parental Involvement, Empowerment, and School Traits to Student Academic 

Performance. Journal of Educational Research, 90(1): 33-41.

Grolnick, W.S., Ryan, R.M., and Deci, E.L. (1991). Inner Resources for Student Motivation: Motivational 

Mediators of Children’s Perceptions of Their Parents. Journal of Educational Psychology, 83: 508-517.

Grossman, D., Neckerman, H.J., Koepsell, T.D., Liu, P., Asher, K., Beland, K., Frey, K., and Rivara, F.P. 

(1997). Effectiveness of a Violence Prevention Curriculum Among Children in Elementary School: A 

Randomized Controlled Trial. Journal of the American Medical Association, 277: 1605-1611.

Guerra, N.G., Huessman, L.R., Tolan, P.H., VanAcker, R. and Eron, L.D. (1995). Stressful Events and 

Individual Beliefs as Correlates of Economic Disadvantage and Aggression Among Urban Children. 

Journal of Consulting and Clinical Psychology, 63: 518-528.

Hahn, R., Fuqua-Whitley, D., Wethington, H., Lowy, J., Crosby, A., Fullilove, M., Johnson, R., Liberman, A., 

Moscicki, E., Price, L., Snyder, S., Tuma, F., Cory, S., Stone, G., Mukhopadhaya, K., Chattopadhyay, 
References 

488

S., and Dahlberg, L. (2007). Effectiveness of Universal School-Based Programs to Prevent Violent 

and Aggressive Behavior: A Systematic Review. American Journal of Preventive Medicine, 33(2S): S114-

S129. 

Harris, R.J. (1975). A Primer of Multivariate Statistics. New York: Academic Press. 

Henry, D.B., Tolan, P.H., and Gorman-Smith, D. (2001). Longitudinal Family and Peer Group Effects on 

Violent and Non-Violent Delinquency. Journal of Child Clinical Psychology,16: 203-220.

Huesmann, L.R., and Guerra, N.G. (1997). Children’s Normative Beliefs About Aggression and Aggressive 

Behavior. Journal of Personality and Social Psychology, 72: 408-419.

Joreskog, K.G., Sorbom, D., Du Toit, S., and Du Toit, M. (2003). LISREL 8: New Statistical Features (3rd 

printing, with revisions). Lincolnwood, IL: Scientific Software International.

Kazdin, A.E. (2001). Behavior Modification in Applied Settings (6th ed.). Belmont, CA: Wadsworth/Thomson 

Learning. 

Kellam, S.G, Ling, X., Merisca, R., Brown, C.H., and Ialong, N. (1998). The Effect of the Level of Aggression 

in the First Grade Classroom on the Course and Malleability of Aggressive Behavior into Middle 

School. Development and Psychopathology, 10: 165-185.

Kusche, C.A., and Greenberg, M.T. (1994). The PATHS Curriculum. Seattle, WA: Developmental Research 

and Programs. 

Larson, R. W. (2000). Toward a Psychology of Positive Youth Development. American Psychologist, 55: 170-

183.

Lewis, L., Parsad, B., Carey, N., Bartfai, N., Farris, E., and Smerdon, B. (1999). Teacher Quality: A Report on the 

Preparation and Qualifications of Public School Teachers (NCES 1999-080). Washington, DC: U.S. 

Department of Education. 

Lickona, T. (1993). The Return of Character Education. Educational Leadership, 51: 6-11.

Limber, S.P., and Small, M.A. (2003). State Laws and Policies to Address Bullying in Schools. School Psychology 

Review, 32: 445-455.

Loeber, R., and Stouthamer-Loeber, M. (1998). Juvenile Aggression at Home and at School. In D.S. Elliott, 

B.A. Hamburg, and K.R. Williams (Eds.), Violence in American Schools (pp. 94-126). Cambridge: 

Cambridge University Press. 

Loney, J., and Milich, R. (1982). Hyperactivity, Inattention, and Aggression in Clinical Practice. In M. 

Wolraich and D. Routh (Eds.), Advances in Developmental and Behavioral Pediatrics (Vol. 2, pp. 113-147). 

Greenwich, CT: JAI Press. 

Malecki, C.K., and Elliot, S.N. (2002). Children’s Social Behaviors as Predictors of Academic Achievement: A 

Longitudinal Analysis. School Psychology Quarterly, 17: 1-23.

Mansfield, W., Alexander, D., and Farris, E. (1991). Fast Response Survey System, Teacher Survey on Safe, Disciplined, 

and Drug-Free Schools (NCES 91-091). National Center for Education Statistics, Office of Educational 

Research and Improvement, U.S. Department of Education. Washington, DC: U.S. Government 

Printing Office. 

Matheny, A.P., Wachs, T.D., Ludwig, J.L., and Phillips, K. (1995). Bringing Order Out of Chaos: 

Psychometric Characteristics of the Confusion, Hubbub, and Order Scale. Journal of Applied 

Developmental Psychology, 16: 429-444.

McCord, J., Widom, C.S., Bamba, M.I., and Crowell, N.A. (2000). Education and Delinquency: Summary of a 

Workshop. Washington, DC: National Academy of Sciences. 
References 

489

McEvoy, A., and Welker, R. (2000). Antisocial Behavior, Academic Failure, and School Climate: A Critical 

Review. Journal of Emotional and Behavioral Disorders, 8: 130-141.

McMahon, R.J., and Frick, P.J. (2005). Evidence-Based Assessment of Conduct Problems in Children and 

Adolescents. Journal of Clinical Child and Adolescent Psychology, 34, 477-505.

Merrell, K., Gueldner, B.A., Ross, S.W., and Isava, D.M. (2008). How Effective Are School Bullying 

Intervention Programs? A Meta-Analysis of Intervention Research. School Psychology Quarterly, 23: 26-

42.

Moffit, T.E. (1993). Adolescence-Limited and Life-Course-Persistent Antisocial Behavior: A Developmental 

Taxonomy. Psychological Review, 100: 674-701.

Murray, D. (1998). Design and Analysis of Group-Randomized Trials. Oxford: Oxford University Press. 

Mytton, J., DiGuiseppi, C., Gough, D., Taylor, R., and Logan, S. (2006). School-Based Secondary Prevention 

Programmes for Preventing Violence. Cochrane Database of Systematic Reviews, Issue 3. doi: 

10.1002/14651858.CD004606.pub2. 

Najaka, S.S., Gottfredson, D.C., and Wilson, D.B. (2001). A Meta-Analytic Inquiry Into the Relationship 

Between Selected Risk Factors and Problem Behavior. Prevention Science, 2: 257-271.

O’Donnell, J., Hawkins, J.D., and Abbott, R.D. (1995). Predicting Serious Delinquency and Substance Use 

Among Aggressive Boys. Journal of Consulting and Clinical Psychology, 63: 529-537.

Orpinas, P., and Frankowski, R. (2001). The Aggression Scale: A Self-Report Measure of Aggressive Behavior 

for Young Adolescents. Journal of Early Adolescence, 21(1): 50-67.

Orpinas, P., and Horne, A.M. (2006). Bullying Prevention: Creating a Positive School Climate and Developing Social 

Competence. Washington, DC: American Psychological Association. 

Orpinas, P., and Kelder, S. (1995). Students for Peace Project: Second Student Evaluation (Unpublished manuscript). 

University of Texas Health Science Center at Houston, School of Public Health, Houston. 

Park-Higgerson, H.-K., Perumean-Chaney, S.E., Bartolucci, A.A., Grimley, D.M., and Singh, K.P. (2008). The 

Evaluation of School-Based Violence Prevention Programs: A Meta-Analysis. Journal of School Health,

78: 465-479.

Pelham, W.E., Fabiano, G.A., and Massetti, G.M. (2005). Evidence-Based Assessment of Attention Deficit 

Hyperactivity Disorder in Children and Adolescents. Journal of Clinical Child and Adolescent Psychology, 

34: 449-476.

Pelham, W.E., Gnagy, E.M., Greenslade, K.E., and Milich, R. (1992). Teacher Ratings of DSM-III-R 

Symptoms for the Disruptive Behavior Disorders. Journal of the American Academy of Child and 

Adolescent Psychiatry, 31: 210-218.

Rentoul, A.J., and Fraser, B.J. (1983). Development of a School-Level Environment Questionnaire. Journal of 

Educational Administration, 21(1): 21–39.

Rey, J.M., Sawyer, M.G., and Prior, M.R. (2005). Similarities and Differences Between Aggressive and 

Delinquent Children and Adolescents in a National Sample. Australian and New Zealand Journal of 

Psychiatry, 39: 366-372. 

Reynolds, C.R., and Kamphaus, R.W. (1998). Behavioral Assessment System for Children. Circle Pines, MN: 

American Guidance Service, Inc. 

Roberts, W., Horn, A., and Battistich, V. (1995). Assessing Students’ and Teachers’ Sense of School as a 

Caring Community. Paper presented at the Annual Meeting of the American Educational Research 

Association, San Francisco, CA. 

Rubin, D. (1987). Multiple Imputation for Nonresponse in Surveys. New York: John Wiley & Sons, Inc. 
References 

490

Samples, F., and Aber, L. (1998). Evaluations of School-Based Violence Prevention Programs. In D.S. Elliott 

and B.A. Hamburg (Eds.), Violence in American Schools: A New Perspective (pp. 217-252). Cambridge: 

Cambridge University Press. 

Sampson, R.J. (2000). A Neighborhood-Level Perspective on Social Change and the Social Control of 

Adolescent Delinquency. In L.J. Crocket and R.K. Silbereisen (Eds.), Negotiating Adolescence in Times of 

Social Change (pp. 178-188). New York: Cambridge University Press. 

Sampson, R.J. Morenoff, J., and Earls, F. (1999). Beyond Social Capital: Spatial Dynamics of Collective 

Efficacy for Children. American Sociological Review, 64: 633-660.

Schochet, P. (2005). Statistical Power for Random Assignment Evaluations of Education Programs. Princeton, NJ: 

Mathematica Policy Research. 

Schochet, P.Z. (2008a). Technical Methods Report: Guidelines for Multiple Testing in Impact Evaluations (NCEE 2008-

4018). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute 

of Education Sciences, U.S. Department of Education. 

Schochet, P.Z. (2008b). The Late Pretest Problem in Randomized Control Trials of Education Interventions (NCEE 

2009-4033). Washington, DC: National Center for Education Evaluation and Regional Assistance, 

Institute of Education Sciences, U.S. Department of Education. 

Schultz, D., Izard, C., and Bear, G. (2004). Children’s Emotion Processing: Relations to Emotionality and 

Aggression. Development and Psychopathology, 16: 371-187.

Schwartz, D., and Proctor, L.J. (2000). Community Violence Exposure and Children’s Social Adjustment in 

the School Peer Group: The Mediating Roles of Emotion Regulation and Social Cognition. Journal of 

Consulting and Clinical Psychology, 68: 670-683.

Seligman, M.E., and Csikszentmihalyi, M. (2000). Positive Psychology: An Introduction. American Psychologist, 

56(1): 5-13.

Shaver, V., and Walls, R.T. (1998). Effect of Title I Parent Involvement on Student Reading and Mathematics 

Achievement. Journal of Research and Development in Education, 31: 90-97.

Shelton, K.K., Frick, P.J., and Wootton, J. (1996). Assessment of Parenting Practices in Families of 

Elementary School-Age Children. Journal of Clinical Child Psychology, 25: 317-329.

Slaby, R.G., and Guerra, N.G. (1988). Cognitive Mediators of Aggression in Adolescent Offenders: 1. 

Assessment. Developmental Psychology, 24: 580-588.

Smith, B.H., Molina, B.S.G., Massetti, G.M., Waschbusch, D.A., and Pelham, W.E. (2007). School-Wide 

Interventions: The Foundation of a Public Health Approach to School-Based Mental Health. In S.W. 

Evans, M. Weist, and Z. Serpell (Eds.), Advances in School-Based Mental Health Interventions: Best Practices 

and Program Models (Vol. 2, pp. 7-2–7-19). Kingston, NJ: Civic Research Institute 

Solomon, D., Battistich, V., Watson, M., Schaps, E., and Lewis, C. (2000). A Six-District Study of Educational 

Change: Direct and Mediated Effects of the Child Development Project. Social Psychology of Education, 

4(1): 3-51. 

Stage, S.A., and Quiroz, D.R. (1997). A Meta-Analysis of Interventions to Decrease Disruptive Classroom 

Behavior in Public Education Settings. School Psychology Review, 26(3): 333-368.

Stringfield, S., Millsap, M., Herman, R., Yoder, N. Brigham, N., Nesselrodt, P., Schaffer, E., Karweit, N. 

Levin, M., and Stevens, R. (1997). Urban and Suburban/Rural Special Strategies for Educating Disadvantaged 

Children: Final Report. Washington, DC: U.S. Department of Education. 
References 

491

Takeuchi, D., Williams, D.R., and Adair, R. (1991). Economic Stress and Children’s Emotional and 

Behavioral Problems. Journal of Marriage and the Family, 53: 1031-1041.

Tolan, P.H., Guerra, N.G., and Kendall, P.C. (1995). A Developmental-Ecological Perspective on Antisocial 

Behavior in Children and Adolescents: Toward a Unified Risk and Intervention Framework. Journal of 

Consulting and Clinical Psychology, 63: 579-584.

Thornberry, T.P., Huizinga, D., and Loeber, R. (2004). The Causes and Correlates Studies: Findings and 

Policy Implications. Juvenile Justice, 9: 3-19.

Trzesniewski, K.H., Moffitt, T.E., Caspi, A., Taylor, A., and Maughan, B. (2006). Revisiting the Association 

Between Reading Achievement and Antisocial Behavior: New Evidence of Environmental 

Explanation From a Twin Study. Child Development, 77: 72-88. 

U.S. Department of Education, National Center for Education Statistics, Common Core of Data. (2003-04). 

Data retrieved from http://nces.ed.gov/ccd. 

Walker, H.M., and Shinn, M.R. (2002). Structuring School-Based Interventions to Achieve Integrated 

Primary, Secondary, and Tertiary Prevention Goals for Safe and Effective Schools (pp. 1-26). In M.R. 

Shin, H.M. Walker, and G. Stoner (Eds.), Interventions for Academic and Behavioral Problems II: Preventive 

and Remedial Approaches. Bethesda, MD: National Association of School Psychologists. 

Waschbusch, D., Pelham, W., and Massetti, G. (2005). The Behavior Education Support and Treatment 

(BEST) School Intervention Program: Pilot Project Data Examining Schoolwide, Targeted-School, 

and Targeted-Home Approaches. Journal of Attention Disorders, 9: 313-322. 

Weissberg, R.P., Kumpfer, K.L., and Seligman, M.E.P. (2003). Special Issue: Prevention That Works for 

Children and Youth: An Introduction. American Psychologist, 58: 425-432.

Wentzel, K.R. (1991). Relations Between Social Competence and Academic Achievement in Early 

Adolescence. Child Development, 62: 1066-1078.

Wentzel, K.R. (1993). Does Being Good Make the Grade? Social Behavior and Academic Competence in 

Middle School. Journal of Educational Psychology, 85: 357-364.

What Works Clearinghouse. (n.d.-a). Character Education. Retrieved August 24, 2007, from 

http://ies.ed.gov/ncee/wwc/reports/Topic.aspx?tid=12. 

What Works Clearinghouse. (n.d.-b). What Works Clearinghouse Intervention Rating Scheme. U.S. Department of 

Education, National Center for Education Evaluation and Regional Assistance. Retrieved May 6, 

2009, from http://ies.ed.gov/ncee/wwc/pdf/rating_scheme.pdf. 

Wheeler, V.A., and Ladd, G.W. (1982). Assessment of Children’s Self-Efficacy for Social Interactions With 

Peers. Developmental Psychology, 18: 795-805.

Wilson, D.B., Gottfredson, D.C., and Najaka, S.S. (2001). School-Based Prevention of Problem Behavior: A 

Meta-Analysis. Journal of Quantitative Criminology, 17: 247-272.

Wilson, S.J., and Lipsey, M.W. (2007). School-Based Interventions for Aggressive and Disruptive Behavior: 

Update of a Meta-Analysis. American Journal of Preventive Medicine, 33(2S): S130-S143. 

Wilson, S.J., Lipsey, M.W., and Derzon, J.H. (2003). The Effects of School-Based Intervention Programs on 

Aggressive Behavior: A Meta-Analysis. Journal of Consulting and Clinical Psychology, 71: 136-149.

Zellman, G.L., and Waterman, J.M. (1998). Understanding the Impact of Parent School Involvement on 

Children’s Educational Outcomes. Journal of Educational Research, 91: 370-380.
A-1 

Appendix A: Analysis of Cohorts 1 and 2 
This page is intentionally blank. 
A-3 

Analysis of Cohorts 1 and 2 

For Years 2 and 3 of the social and character development (SACD) multiprogram evaluation, additional funds 

were provided to enable research teams to add additional schools in order to increase the sample size and the 

corresponding statistical power of the analyses. Four teams added 2 or 4 schools and randomly assigned half 

to the treatment and half to the control groups using the same matching process as with the original schools. 

The additions were as follows: (1) University at Buffalo, The State University of New York (Academic and 

Behavioral Competencies Program [ABC]) added 2 schools, (2) University of North Carolina at Chapel Hill 

(Competence Support Program [CSP]) added 4 schools,1

 (3) Vanderbilt University (Love In a Big World [LBW]) 

added 2 schools, and (4) The Children’s Institute (Promoting Alternative Thinking Strategies [PATHS]) added 4 

schools. As a result, a second set of 12 schools (and a second cohort of students) was added to the SACD 

evaluation for Years 2 and 3, raising the total number of schools involved in the study from 84 to 96 (with the 

second cohort of schools making up 14 percent of the sample).2

 

Data collection at these schools followed the same process and used the same reports as at the first set of 

schools. Initial data collection with third-grade students in the second cohort began in fall 2005 (1 year later 

than the first cohort of third-graders), with the first follow-up at the end of third grade (spring 2006) and the 

second follow-up at the end of fourth grade (spring 2007). Students were followed for 2 years (through third 

and fourth grades) but not through fifth grade as was done with Cohort 1. 

This appendix examines whether an analysis of the larger sample of both cohorts together (all Cohort 1 

students plus all Cohort 2 students) gives similar or different results from those for Cohort 1 alone (presented 

in chapters 1 through 8). The two cohorts were combined into one sample by grouping all third-graders 

together and all fourth-graders together. Two analyses were done. The first compared student outcomes for 

treatment and control students at the end of their fourth-grade year, and the second looked at average yearly 

growth in student outcomes from the beginning of third grade to the end of fourth grade. As Cohort 2 was 

not followed through the fifth grade, both analyses are based on the third- and fourth-grade data only. 

Sample 

There were 7,255 fourth-graders enrolled in one of 95 study schools at the time of data collection in spring of 

the students’ fourth grade (spring 2006 for Cohort 1 and spring 2007 for Cohort 2). These students included 

5,750 original cohort “stayers” who were enrolled in a study school at the time of data collection in fall of 

their third grade and spring of fourth grade, and 1,505 new entrants who enrolled in a study school after the 

data collection in the fall of their third grade but before data collection in the spring of fourth grade. The 

sample universe excluded 1,607 original cohort “leavers” who were enrolled in a study school in fall of third 

grade but not in spring of fourth grade. 

There were 1,064 third-, fourth-, or fifth-grade teachers in spring of fourth grade but 124 either did not give 

consent to participate or did not provide the requested data. As a result, the teacher samples for the spring of 

fourth-grade analysis included 940 third- to fifth-grade teachers (fourth-grade teachers completed the Teacher 

Report on Student and third- to fifth-grade teachers completed the Teacher Report on Classroom and 

School). Teachers who had been in the study in fall of third grade but were excluded from the fourth-grade 

 

1 One of CSP’s Cohort 2 schools stopped implementing the intervention after the first year but remained in the study 

for the second year as a treatment school. 

2

 Of the original 84 schools, one control school dropped out of the study prior to Year 2 when it became a magnet 

school, leaving 95 schools available for the analysis of both cohorts together. 
Appendix A: Analysis of Cohorts 1 and 2 

A-4 

follow-up could either have (1) left the school between fall of third grade and spring of fourth grade or (2) 

remained at the school but discontinued teaching in the third, fourth, or fifth grade. 

The consent and response rates led to the analysis sample sizes and the percentages of the universe of fourthgraders for which there were data (table A.1). The analysis sample for the Child Report was 4,549 (2,395 

treatment group students and 2,154 control group students); the samples for the Primary Caregiver Report 

and Teacher Report on Student were 3,551 (1,856 treatment group students and 1,695 control group 

students) and 4,737 (2,492 treatment group students and 2,245 control group students), respectively. These 

Child Report, Primary Caregiver Report, and Teacher Report on Student samples reflected 63 percent, 49 

percent, and 65 percent of the 7,255 students in the sample universe of students in the fourth grade in study 

schools (in spring 2006 for Cohort 1 and spring 2007 for Cohort 2, including both original cohort stayers and 

new entrants in each cohort). The analysis sample for the Teacher Report on Classroom and School was 940 

teachers (482 treatment group and 458 control group), which was 88 percent of the universe of teachers in 

the study schools. None of the differences between treatment and control groups in the percentage of the 

universe for which data were collected was statistically significant when the data were pooled across 

programs. 
Table A.1. Combined Cohorts 1 and 2: Sample size and percentage of sample universe for fourth-graders, by program and by 

treatment group status 

Analysis sample size (Percent of sample universe)

Child Report Primary Caregiver Report Teacher Report on Student

Teacher Report on 

Classroom and School

Program Treatment Control Total Treatment Control Total Treatment Control Total Treatment Control Total

All programs 2,395 2,154 4,549 1,856 1,695 3,551 2,492 2,245 4,737 482 458 940

(63.9) (61.4) (62.7) (49.5) (48.3) (48.9) (66.5) (64.0) (65.3) (88.6) (88.1) (88.3)

ABC 254 350 604 182 262 444 266 362 628 56 72 128

(59.8)** (63.6) (61.9) (42.8) (47.6) (45.5) (62.6)*** (65.8) (64.4) (77.8) (85.7) (82.1)

CSP 432 429 861 324 348 672 438 442 880 81 82 163

(65.9) (60.2) (62.9) (49.4) (48.8) (49.1) (66.8) (62.0) (64.3) (85.3) (85.4) (85.3)

LBW 357 287 644 314 234 548 385 299 684 77 55 132

(60.9) (62.1) (61.5) (53.6) (50.6) (52.3) (65.7) (64.7) (65.3) (90.6) (87.3) (89.2)

PA 284 225 509 215 184 399 302 251 553 43 39 82

(66.8) (66.4) (66.6) (50.6) (54.3) (52.2) (71.1) (74.0) (72.4) (86.0) (86.7) (86.3)

PATHS 306 265 571 239 201 440 314 273 587 73 77 150

(58.5) (52.2) (55.4) (45.7)** (39.6) (42.7) (60.0) (53.7) (56.9) (85.9) (81.1) (83.3)

4Rs 376 303 679 245 210 455 403 323 726 91 82 173

(62.8) (59.4) (61.2) (40.9) (41.2) (41.0) (67.3) (63.3) (65.5) (96.8) (97.6) (97.2)

SS 386 295 681 337 256 593 384 295 679 61 51 112

(72.4) (69.2) (71.0) (63.2) (60.1) (61.8) (72.0) (69.2) (70.8) (96.8) (96.2) (96.6)

** Treatment group significantly different from control group at the .01 level.

*** Treatment group significantly different from control group at the .001 level.

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies.

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution).

 SS: Second Step

SOURCE: The Social and Character Development (SACD) Research Program.

A-5

Appendix A: Analysis of Cohorts 1 and 2
Appendix A: Analysis of Cohorts 1 and 2 

A-6 

There were several significant differences at the program level. For the ABC program, there was a statistically 

significant treatment-control group difference in the percentages with Child Report data (60% for the 

treatment group and 64% for the control group) and Teacher Report on Student data (63% and 66%, 

respectively). For the PATHS program, there was a statistically significant difference in the percentages with 

Primary Caregiver Report data (46% and 40%, respectively). 

As in the Cohort 1 study, the sample for the evaluation of the two cohorts raised two cautions for the 

interpretation of the results. First, because of non-consent and non-response, the impact estimates might not 

generalize to the full sample universe if the characteristics of students in the study and students not in the 

study differed in ways that correlated with child outcomes. Second, because the study did not follow leavers 

from the original cohort, the intent-to-treat (ITT) impacts could not be estimated using all original cohort 

members. Instead, the evaluation used a repeated cross-sectional analysis approach to estimate impacts; this 

involved including all students in the study schools who had consent and who provided data at each followup point. These unbiased impact estimates for students in the study reflect (1) the extent to which the 

programs improved the outcomes of the average student in the study, and (2) treatment-induced “mobility 

effects” resulting from potential differences in the average outcomes of treatment and control students who 

entered and left the schools after random assignment. 

Initial Data 

The examination of the initial characteristics3 of students, families, teachers, and schools for the combined 

cohorts showed few significant differences between the treatment and control groups at the combinedprogram level and fewer differences among the programs than would be expected by chance. These results 

are similar to those of the Cohort 1-only sample. The treatment and control groups in the combined sample 

were similar along all student and parent/caregiver characteristics except for household size (the treatment 

group students were more likely to come from smaller households: 4.6 people versus 4.7 per household) 

(table A.2). There were no significant differences between treatment and control groups in the mean values of 

the outcomes measuring children’s behavior and attitudes (table A.3). There were no significant differences 

between the treatment and control groups in teachers’ initial characteristics (table A.4), nor were there 

significant differences in the characteristics of treatment group schools and control group schools (though 

sample sizes were small for these comparisons) (table A.5). There were no differences between treatment and 

control groups in teacher-reported measures of the school environment on such dimensions as safety, 

participatory decision making, and work pressure (these data are not shown in a table). 

 

3

 For a list of the characteristics, see Initial Characteristics in chapter 1. 
Appendix A: Analysis of Cohorts 1 and 2 

A-7 

Table A.2. Combined Cohorts 1 and 2: Initial characteristics of children, their families, and 

communities 

Characteristic Total Treatment Control

 Student sample size 4,202 2,202 2,000

Student demographics

 Gender (percent)

 Male 47.4 47.1 47.7

 Female 52.6 52.9 52.3

 Race/ethnicity (percent)

 White (non-Hispanic) 40.7 39.0 42.5

 Black (non-Hispanic) 31.9 33.7 30.2

 Hispanic 19.9 19.8 20.0

 Other 7.4 7.5 7.3

 Age (in years) (mean) 8.1 8.1 8.1

Primary caregiver and family characteristics

 Primary caregiver’s age (in years) (mean) 36.0 35.9 36.1

 Primary caregiver’s race/ethnicity (percent)

 White (non-Hispanic) 44.6 42.8 46.4

 Black (non-Hispanic) 31.9 33.8 30.0

 Hispanic 18.2 18.0 18.3

 Other 5.3 5.4 5.3

 Primary caregiver’s education (percent)

 Did not complete high school 15.5 14.5 16.5

 Completed high school or equivalent 24.7 24.2 25.3

 Some college 39.7 40.7 38.7

 Bachelor’s or higher degree 20.0 20.6 19.5

 Primary caregiver’s employment (percent)

 Full-time 47.5 47.8 47.3

 Part-time 14.6 14.5 14.7

 Student and employed 4.4 4.9 3.9

 Not employed 32.4 31.7 33.0

 Other 1.1 1.1 1.1

 Primary caregiver’s marital status (percent)

 Single 25.0 25.6 24.4

 Married 55.2 54.5 56.0

 Separated or divorced 12.7 12.7 12.7

 Widowed 1.4 1.3 1.5

 Other15.6 5.9 5.4

 Students who live in one household (percent) 93.6 93.5 93.7

 Number of individuals in household (mean) 4.7 4.6* 4.7

See notes at end of table.
Appendix A: Analysis of Cohorts 1 and 2 

A-8 

Table A.2. Combined Cohorts 1 and 2: Initial characteristics of children, their families, and

Table A.2. communities—Continued

Characteristic Total Treatment Control

 Primary caregiver’s relationship to child (percent)

 Mother (stepmother) 85.9 86.0 85.8

 Father (stepfather) 8.9 8.5 9.4

 Grandparent 3.1 3.2 3.0

 Other relative 1.3 1.4 1.3

 Nonrelative 0.8 1.0 0.6

 Student lives with (percent)

 Mother (stepmother) and father (stepfather) 58.4 57.0 59.7

 Mother (stepmother) only; father (stepfather) not present 35.1 36.3 33.9

 Father (stepfather) only; mother (stepmother) not present 2.6 2.4 2.8

 Other relative/nonrelative; parents not present 4.0 4.3 3.6

 Highest education of anyone in household (percent)

 Did not complete high school 10.7 9.6 11.9

 Completed high school or equivalent 22.4 21.7 23.1

 Some college 40.5 42.1 39.0

 Bachelor’s or higher degree 26.3 26.6 26.0

 Total household income (percent)

 Less than $20,000 34.6 33.7 35.4

 $20,000 to $39,999 24.6 25.2 24.1

 $40,000 to $59,999 15.0 15.4 14.5

 $60,000 or more 25.9 25.7 26.0

 Household income-to-poverty ratio (percent)

 Below 135 percent 40.8 39.3 42.3

 135 to 185 percent 18.7 19.3 18.1

 Above 185 percent 40.5 41.4 39.6

 Alabama Parenting Questionnaire—Poor Monitoring 

 and Supervision Subscale (mean) 1.2 1.2 1.2

 Alabama Parenting Questionnaire—Positive Parenting 

 Subscale (mean) 3.5 3.5 3.5

 Confusion, Hubbub, and Order Scale (mean) 2.2 2.2 2.2

Community characteristics (mean)

 Community Risks Scale 1.5 1.5 1.5

 Community Resources Scale 2.7 2.7 2.7

 Child-Centered Social Control Scale 3.1 3.0 3.1

* Treatment group significantly different from control group at the .05 level.

1 Categories combined to protect confidentiality.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included program 

indicators to account for the sample design and adjusted for clustering at the school level.

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix A: Analysis of Cohorts 1 and 2 

A-9 

Table A.3. Combined Cohorts 1 and 2: Mean scores and standard deviations for initial outcome 

measures 

Outcome measure–Report

Total Treatment Control

Range Mean SD Mean SD Mean SD

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 1-4 2.9 0.6 2.9 0.6 2.9 0.6

 Normative Beliefs About Aggression–CR 1-4 1.2 0.5 1.2 0.5 1.2 0.5

 Empathy–CR 1-3 2.4 0.4 2.4 0.4 2.4 0.4

Behavior Domain

 Altruistic Behavior–CR 0-3 1.5 0.8 1.4 0.8 1.5 0.8

 Altruistic Behavior–TRS 1-4 1.4 0.5 1.4 0.5 1.4 0.4

 Altruistic Behavior–PCR 1-4 2.3 0.7 2.3 0.7 2.3 0.8

 Positive Social Behavior–TRS 1-4 3.0 0.7 3.0 0.7 3.0 0.7

 Positive Social Behavior–PCR 1-4 3.0 0.5 3.0 0.5 3.0 0.5

 Problem Behavior–CR 0-3 0.2 0.4 0.3 0.4 0.2 0.4

 Problem Behavior–TRS 1-4 1.4 0.4 1.4 0.4 1.4 0.4

 Problem Behavior–PCR 1-4 1.6 0.3 1.6 0.3 1.6 0.3

 ADHD-Related Behavior–TRS 1-4 1.7 0.7 1.7 0.7 1.7 0.6

Academics Domain

 Academic Competence and Motivation–TRS 1-5 2.9 0.9 2.9 0.9 2.9 0.9

 Engagement with Learning–CR 1-4 3.7 0.6 3.7 0.7 3.7 0.6

Perceptions of School Climate Domain

 Positive School Orientation–CR 1-4 3.2 0.7 3.2 0.7 3.2 0.7

 Negative School Orientation–CR 1-4 1.9 0.7 1.9 0.7 1.9 0.7

 Student Afraid at School–CR 1-4 2.4 0.9 2.4 0.9 2.4 0.9

 Victimization at School–CR 0-3 0.8 0.8 0.8 0.8 0.8 0.8

 Sample size—PCR1 4,202 2,202 2,000

 Sample size—CR

1

 4,486 2,357 2,129

 Sample size—TRS

1

 4,578 2,405 2,173

1 Sample size may differ for some outcomes due to nonresponse.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

 SD: Standard deviation

No findings were found statistically significant at or below the .05 level. Weights, which assign equal weight to each school within 

each of the programs and to each program across programs, were used in producing the treatment, control, overall means, and 

standard deviations. Statistical tests were conducted using regressions that included program indicators to account for the sample 

design and adjusted for clustering at the school level.

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix A: Analysis of Cohorts 1 and 2 

A-10

Table A.4. Combined Cohorts 1 and 2: Initial characteristics of teachers 

Characteristic Total Treatment Control

 Teacher sample size 964 494 470

Gender (percent)

 Male 11.8 12.2 11.4

 Female 88.2 87.8 88.6

Race/ethnicity (percent)

 White (non-Hispanic) 76.0 74.5 77.4

 Black (non-Hispanic) 16.4 17.6 15.1

 Hispanic 4.7 5.7 3.7

 Other 3.0 2.1 3.8

Number of years teaching (mean) 12.7 12.5 12.9

Number of years teaching in this school (mean) 7.3 6.9 7.7

Type of teaching certificate (percent)

 Regular state certificate or advanced professional certificate 84.6 83.7 85.6

 Provisional certificate (for those in alternative certification 

 programs) 5.8 6.7 4.8

 Probationary certificate (for those who have satisfied all 

 requirements except for completing the 

 probationary period) 5.0 5.5 4.5

 Emergency certificate or waiver (for those who must 

 complete a certification program to continue teaching) 0.0 0.0 0.0

 Other14.6 4.0 5.1

Education

 Less than a bachelor’s degree 0.0 0.0 0.0

 Bachelor’s degree 40.3 39.4 41.2

 Master’s degree, Ph.D. 56.1 58.2 54.0

 Specialist degree 2.0 1.0 2.9

 Other 1.7 1.4 1.9

1 Categories, including ―no certificate‖ and ―temporary certificate,‖ combined to protect confidentiality. 

NOTE: No findings were found statistically significant at or below the .05 level. Weights, which assign equal weight to each school 

within each of the programs and to each program across programs, were used in producing the treatment, control, and overall 

means. Statistical tests were conducted using regressions that included program indicators to account for the sample design and 

adjusted for clustering at the school level.

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix A: Analysis of Cohorts 1 and 2 

A-11

Table A.5. Combined Cohorts 1 and 2: Initial characteristics of schools 

Characteristic Total Treatment Control

 School sample size 96 48 48

Student race/ethnicity (percent)

 White (non-Hispanic) 36.4 36.6 36.1

 Black (non-Hispanic) 40.9 40.4 41.5

 Hispanic 18.8 18.7 18.9

 Other 3.9 4.3 3.5

Students eligible for free or reduced-price lunch (percent) 63.2 64.1 62.4

Number of students enrolled (mean) 552.1 541.3 563.3

Number of full-time teachers (mean) 35.7 34.4 36.9

Title I status (percent)

 Title I eligible school 76.9 76.1 77.6

 Schoolwide Title I 68.4 66.8 69.9

Lowest grade offered (percent)

 Prekindergarten 59.1 51.4 67.0

 Kindergarten 40.9 48.6 33.0

Highest grade offered (percent)

 5th grade 54.0 57.5 50.5

 6h grade 23.5 20.2 26.8

 8th grade 21.5 20.3 22.7

Location of school (percent)

 City 57.2 58.7 55.7

 Suburbs 23.7 22.4 25.1

 Rural 19.1 18.9 19.2

Number of years principal has been at this school (mean) 5.5 5.0 5.9

NOTE: No findings were found statistically significant at or below the .05 level. Weights, which assign equal weight to each school 

within each of the programs and to each program across programs, were used in producing the treatment, control, and overall 

means. Statistical tests were conducted using regressions that included program indicators to account for the sample design and 

adjusted for clustering at the school level. Data are missing from Love In a Big World for student race/ethnicity, percentage of 

students eligible for free or reduced-price lunch, and number of full-time teachers.

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix A: Analysis of Cohorts 1 and 2 

A-12

Large percentages of principals and teachers (treatment and control) reported using social and character 

development activities and receiving professional development in their use. The percentages reported were 

very similar to those reported for Cohort 1, so only a selection of indicators are shown in table A.6. Where 

there were differences, they favored the treatment teachers, and all the differences that appeared here also 

appeared in the Cohort 1-only data. These differences may have reflected the fact that the programs had been 

implemented in most of the study schools before the teachers were surveyed about their use of activities. 
Appendix A: Analysis of Cohorts 1 and 2 

A-13

Table A.6. Combined Cohorts 1 and 2: Teacher initial reports on use of SACD activities and 

training in SACD activities 

Teacher report Total Treatment Control

Teachers reporting on using programs or activities in their 

 class to promote the following SACD goals (percent)

 Violence prevention and peace promotion 64.4 68.2* 60.7

 Social and emotional development 72.2 76.3* 68.0

 Character education 82.8 84.2 81.4

 Tolerance and diversity 61.9 62.4 61.4

 Risk prevention and health promotion 52.0 50.8 53.1

 Civic responsibility and community service 59.1 58.2 60.0

 Behavior management 87.3 89.1 85.4

 None of the above 2.4 1.9 2.9

Teachers using the following materials in conjunction with social 

 and character development activities (percent)

 Teacher guides (manuals, curricula) 66.0 72.0** 60.1

 Student materials (workbooks, worksheets) 50.1 51.4 48.9

 Instructional aids (games, software, videos) 32.6 36.0* 29.1

 Giveaways (bookmarks, stickers) 48.0 45.5 50.4

 Children’s literature 49.7 52.5 46.9

 Other types of materials 12.9 10.9 14.9

 Do not use any of the materials listed above 10.1 9.8 10.4

Use of instructional strategies to promote SACD (percent)

 Teachers using any of 20 instructional strategies to promote 

 social and character development in the classroom (percent) 99.8 99.6 100.0

 Number of the 20 strategies used by teachers to promote 

 social and character development in the classroom (mean) 11.7 12.1* 11.4

Professional development on promoting SACD (percent)

 Teachers reporting participation in social and character 

 development training within the past 12 month (percent) 73.7 84.7*** 62.7

 Number of hours of social and character development 

 training teachers report receiving during the past 12 

 months (mean) 7.6 8.9** 6.3

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

*** Treatment group significantly different from control group at the .001 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were

used in producing the treatment, control, and overall means. Statistical tests were conducted using regressions that included 

program indicators to account for the sample design and adjusted for clustering at the school level. Sample size may differ for some 

outcomes due to nonresponse.

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix A: Analysis of Cohorts 1 and 2 

A-14

Cohort 2 significantly differed in some ways from Cohort 1 when comparing students, teachers, and schools 

(these data are not shown in a table), although some of these comparisons were based on small samples sizes 

for Cohort 2. At the combined-program level, there were no statistically significant differences between 

Cohort 1 and Cohort 2 students in terms of gender, race/ethnicity, or age. However, students from Cohort 1 

were reported to come from less disadvantaged families than students in Cohort 2. Cohort 1 parents or 

primary caregivers were statistically significantly more likely to have a bachelor’s degree or higher, be 

employed, be married, live in a household that included both a mother (or stepmother) and a father (or 

stepfather), have a higher income-to-poverty threshold ratio, and have a lower average community risks scale 

score (as reported by primary caregivers). With respect to the outcome measures, Cohort 1 students had 

significantly higher Academic Competence and Motivation, higher Engagement with Learning, fewer 

Normative Beliefs About Aggression, and a more Positive Orientation Toward School, although they had 

lower Altruistic Behavior scores (Primary Caregiver Report). The characteristics of teachers were similar in 

Cohort 1 and Cohort 2, and there were no statistically significant differences between the cohorts in teachers’ 

gender, race/ethnicity, years of experience, or education. The Cohort 1 schools had a statistically significantly 

greater average number of enrolled students and a greater average number of full-time teachers but there 

were no differences in the characteristics of the students at the schools, such as the distributions of students 

across racial/ethnic groups or percentages of students eligible for free or reduced-price lunches. Teacherreported measures of the school environment on such dimensions as safety, participatory decision making, 

and work pressure showed no statistically significant differences between Cohort 1 and Cohort 2. There were 

few significant differences in the use of SACD activities reported by the teachers in both cohorts. More 

Cohort 1 teachers reported using activities for tolerance and diversity (a difference of 10 percentile points). 

More Cohort 2 teachers reported using character education (with a difference of less than 2 percentile points) 

and behavior management activities (a difference of 7 percentile points). Teachers reported no significant 

differences in the use of materials and strategies for SACD activities and their participation in professional 

development; however, there was a difference between the cohorts in teacher reports on modeling positive 

social and character traits with students; this favored Cohort 1 (87 % versus 81 %). 

The addition of the Cohort 2 schools did not have a large impact on the percentages of data available for the 

analysis. Table A.7 compares the consent rates, completion rates, and percentages of sample with data for 

each report for Cohort 1 only versus Cohorts 1 and 2 combined. The rates for Cohorts 1 and 2 combined are 

very similar to those of Cohort 1 only. Significant differences between treatment and control groups occur, in 

most cases, similarly in both. In two cases, the combined cohort data show significant differences between 

the treatment and control groups (Child Report students with data and students with Teacher Report on 

Student data, both in spring fourth grade) that were not found for Cohort 1 only. However, the actual 

magnitudes of the differences are very similar for both. 
Table A.7. Cohort 1 versus combined Cohorts 1 and 2: Comparison of consent rates, completion rates, and percentage of sample 

with data 

Cohort 1 Cohorts 1 and 2 Cohort 1 Cohorts 1 and 2

(Fall 3rd grade) (Fall 3rd grade) (Spring 4th grade) (Spring 4th grade)

Report Total

Treatment Control TotalTreatment Control TotalTreatment Control TotalTreatment Control

 Student sample size 6,567 3,367 3,200 7,357 3,764 3,593 6,415 3,327 3,088 7,255 3,747 3,508

Child Report (percent)

 Primary caregiver consent rate 65 67** 63 65 67*** 63 67 67 66 66 67 64

 Student completion rate 94 93* 94 93 95 94 95 96 95 96 96 95

 Students with data161 62* 60 61 63** 59 63 65 62 63 64* 61

Primary Caregiver Report (percent)

 Primary caregiver consent rate 63 64** 61 62 64*** 60 64 65 63 63 64 62

 Primary caregiver completion rate 92 92 92 92 91 92 78 77 78 77 77 77

 Primary caregivers with data157 59** 56 57 59* 56 50 51 49 49 50 48

Teacher Report on Student (percent)

 Primary caregiver consent rate265 67*** 63 65 67*** 63 67 67 66 66 67 64

 Teacher completion rate 96 96 96 96 95 97 100 100 99 100 100 99

 Students with data162 64** 61 62 64** 60 66 67 65 65 67* 64

Teacher Report on Classroom and School

 (3rd- to 5th-grade teachers) (percent)

 Teacher consent rate 96 98*** 92 95 98*** 93 95 97 94 95 95 94

 Teacher completion rate 91 90 93 91 90 93 94 94 94 93 93 93

 Teachers with data187 88 86 87 88 86 90 90 89 88 89 88

* Treatment group significantly different from control group at the .05 level.

** Treatment group significantly different from control group at the .01 level.

*** Treatment group significantly different from control group at the .001 level.

1 Calculated as consent rate x completion rate.

2

The primary caregiver consent rates for the Child Report and the Teacher Report on Student are identical, as the primary caregiver gave consent to both together.

SOURCE: The Social and Character Development (SACD) Research Program.

A-15

Appendix A: Analysis of Cohorts 1 and 2
Appendix A: Analysis of Cohorts 1 and 2 

A-16

The opportunity for the teams to add the Cohort 2 schools to the SACD multiprogram evaluation was 

planned as a means to increase the statistical power of the analyses by increasing the sample size. In practice, 

Cohort 2 added 12 schools to the sample of 84 schools. The addition of new schools also opened the 

possibility for a change in the intraclass correlations (the ICC, a measure of the percentage of the total 

variance in the outcomes that is between schools). An increase in the sample size would reduce the minimum 

detectable impacts in effect size units (MDES) for each outcome measure; however, changes in the ICCs 

could offset those reductions. In practice, the MDES for the outcome measures both increased and 

decreased, with 17 outcomes having changes in the thousandths of a point and 3 having changes in the 

hundredths (the largest being a decline of .027). Table A.8 compares the MDES for Cohort 1 versus the 

combined Cohorts 1 and 2. 
Appendix A: Analysis of Cohorts 1 and 2 

A-17

Table A.8. Comparison of Cohort 1 and Cohorts 1 and 2: Adjusted minimum detectable effect 

sizes for fourth-grade outcomes for combined-program evaluation 

Outcome measure–Report Cohort 1 Cohorts 1 and 2

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interaction–CR 0.048 0.049

 Normative Beliefs About Aggression–CR 0.081 0.076

 Empathy–CR 0.078 0.071

Behavior Domain

 Altruistic Behavior–CR 0.057 0.056

 Altruistic Behavior–PCR 0.044 0.039

 Altruistic Behavior–TRS 0.238 0.211

 Positive Social Behavior–PCR 0.042 0.048

 Positive Social Behavior–TRS 0.115 0.113

 Problem Behavior–CR 0.086 0.090

 Problem Behavior–PCR 0.040 0.045

 Problem Behavior–TRS 0.086 0.082

 ADHD-Related Behavior–TRS 0.082 0.083

Academics Domain

 Engagement with Learning–CR 0.036 0.042

 Academic Competence and Motivation–TRS 0.077 0.084

Perceptions of School Climate Domain

 Positive School Orientation–CR 0.123 0.113

 Negative School Orientation–CR 0.070 0.067

 Student Afraid at School–CR 0.064 0.063

 Victimization at School-CR 0.063 0.071

 Feelings of Safety–TRCS 0.179 0.157

 Student Support for Teachers–TRCS 0.154 0.152

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The minimum detectable effect (MDE) formula used in the calculations is as follows:

 

 

 

 

 

 

 

 

 

 

 

 

 

 

where sT and sC are the number of treatment and comparison schools; nT and nC are the average number of students per 

classroom; ρ1 is the intraclass correlation (ICC) at the school level; and factor(df) is a constant that depends on the number of 

degrees of freedom (df) available for analysis (and is 2.802 for the pooled analysis). Estimates were adjusted for fixed program 

effects as well as baseline covariates.

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix A: Analysis of Cohorts 1 and 2 

A-18

Analysis 

The analyses used for the combined cohort data were similar to those used with the Cohort 1 data and 

described in chapter 1. Adjustments were made to account for fewer data collection time points. For example, 

the analyses of the impacts on the use of SACD activities in the classroom and the school and of the impact 

on student outcomes used the same analytical methods as did the year-by-year analysis of Cohort 1 fourthgrade student outcomes. 

In addition, a similar growth curve analysis was done with the combined cohort data, using student results 

from three time points (fall third grade, spring third grade, and spring fourth grade) rather than the five time 

points used with the Cohort 1 growth analysis. For this reason, the combined cohort growth analysis is not 

directly comparable to the Cohort 1 growth analysis. 

Impacts on Use of SACD Activities 

The analysis was based on the Teacher Report on Classroom and School data collected when the students 

were in the fourth grade (school year 2005-06 for Cohort 1 and school year 2006-07 for Cohort 2). In 

addition to information about the teachers’ characteristics and backgrounds, this questionnaire asked third-, 

fourth-, and fifth-grade teachers in treatment and control group schools to provide information about the 

social and character development activities they used in their classrooms and about the climate in their 

schools. 

The impacts of the SACD programs on teachers’ reported use of SACD activities for the combined cohort 

analysis were consistent with results from the analyses of Cohort 1 data when students were in the fourth 

grade. Compared to control group teachers, treatment group teachers were more likely to report engaging in 

activities to promote social and character development on 12 out of 16 outcomes and to report engaging in 

similar activities linked to named programs on 14 out of 14 outcomes (table A.9). They also reported using 

more materials and strategies to promote social and character development for 13 out of 29 outcomes (table 

A.10) and more participation in SACD-related professional development for 2 out of 9 outcomes (table 

A.11). For engaging in any SACD activities, the difference between treatment and control group reports was 

small but significant (95% versus 91%), and for engaging in any SACD activities linked to named programs, 

the difference was larger (72% versus 43%). For professional development, 66 percent of treatment teachers 

reported receiving SACD training over the past 12 months versus 51 percent of control teachers. Treatment 

teachers were not more likely to report teacher and staff attitudes that might be conducive to promoting 

social and character development (these data are not shown in a table). 
Appendix A: Analysis of Cohorts 1 and 2 

A-19

Table A.9. Combined Cohorts 1 and 2: Impacts on classroom activities 

Classroom activity Treatment Control Impact p-value

 Teacher sample size 481 459

Engagement in any activities to promote SACD goals1 (percent)

 Activities

 Violence prevention and peace promotion 76.4* 66.1 10.3 0.002

 Social and emotional development 84.5* 61.2 23.4 0.000

 Character education 91.9* 77.6 14.3 0.000

 Tolerance and diversity 74.9* 61.9 13.0 0.000

 Risk prevention and health promotion 63.6 60.1 3.6 0.389

 Civic responsibility and community service 60.7 60.0 0.7 0.872

 Any activity 95.0* 90.8 4.2 0.025

 Behavior management 92.1* 83.3 8.8 0.000

 At least 1 hour per week of activities

 Violence prevention and peace promotion 41.9* 25.3 16.7 0.000

 Social and emotional development 49.2* 25.7 23.5 0.000

 Character education 59.4* 31.7 27.7* 0.000

 Tolerance and diversity 35.9* 22.9 13.0 0.000

 Risk prevention and health promotion 26.9 22.0 4.9 0.174

 Civic responsibility and community service 20.8 16.1 4.7 0.111

 Any activity 83.7* 73.2 10.5 0.002

 Behavior management 74.9* 61.5 13.3 0.002

Engagement in named activities to promote SACD goals1 (percent)

 Named activities

 Violence prevention and peace promotion 46.7* 18.1 28.6 0.000

 Social and emotional development 53.2* 11.6 41.6 0.000

 Character education 56.4* 14.9 41.5 0.000

 Tolerance and diversity 36.4* 8.2 28.2 0.000

 Risk prevention and health promotion 30.5* 18.9 11.5 0.001

 Civic responsibility and community service 14.4* 5.9 8.5 0.000

 Any named activity 71.5* 42.5 29.0 0.000

 At least 1 hour per week of named activities

 Violence prevention and peace promotion 31.5* 11.7 19.8 0.000

 Social and emotional development 36.1* 7.0 29.1 0.000

 Character education 41.8* 8.1 33.6 0.000

 Tolerance and diversity 24.2* 4.2 20.0 0.000

 Risk prevention and health promotion 17.4* 10.8 6.6 0.035

 Civic responsibility and community service 8.1* 2.2 6.0 0.003

 Any named activity 50.7* 22.6 28.1 0.000

* Treatment group significantly different from control group at the .05 level. 

1

Impact on domain found statistically significant after adjustments made for multiple comparisons within domain.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were 

used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix A: Analysis of Cohorts 1 and 2 

A-20

Table A.10. Combined Cohorts 1 and 2: Impacts on use of SACD classroom materials and 

teaching strategies 

SACD material and teaching strategy Treatment Control Impact p-value

 Teacher sample size 481 459

Use of SACD materials (percent)

 Teacher guides (manuals, curricula) 80.4* 59.5 20.9 0.000

 Student materials (workbooks or sheets) 66.5* 54.3 12.3 0.001

 Instructional aids (games, software, videos) 46.2* 38.4 7.8 0.029

 Giveaways (bookmarks, stickers) 53.1 49.2 3.9 0.250

 Children’s literature 57.8 48.2 9.6 0.027

 Other types of materials 9.1 12.3 -3.3 0.147

 Did not use any of these materials 4.6* 13.1 -8.5 0.000

Use of teaching strategies (percent)

 Role-playing 88.0* 67.4 20.5 0.000

 Cooperative learning 98.2^ 96.3 2.0 0.075

 Peer group discussions 92.4 89.1 3.3 0.111

 Direct instruction of SACD 93.4* 78.0 15.4 0.000

 Skill training 65.3* 44.2 21.2 0.000

 Incorporating SACD into academic curriculum 86.3* 73.3 13.0 0.000

 Parent training 11.7 9.7 2.0 0.466

 Parent and community involvement 30.7* 22.0 8.7 0.023

 Mentoring 46.0 44.8 1.1 0.741

 Good behavior notes sent home daily or weekly 79.5 74.3 5.2 0.143

 Presenting role models 75.2^ 69.6 5.6 0.080

 Targeted story reading or writing on social 

 and character development themes 88.5* 78.9 9.6 0.000

 Peer mediation 54.4 52.4 2.0 0.635

 Honor roll for positive behavior 62.0 59.9 2.1 0.639

 Pledges or recitations on SACD themes 48.0 44.9 3.1 0.598

 Guided visualization 60.9* 48.2 12.6 0.002

 Student-led/student-assisted instruction 59.6* 48.5 11.1 0.002

 Journaling 77.9 72.3 5.6 0.117

 Time out for negative behavior 86.1 85.3 0.8 0.720

 Daily or weekly rewards for positive behavior 93.0 90.2 2.8 0.190

 Any strategy (percent) 100.0 100.0 0.0 †

 Number of strategies (mean) 13.8* 12.4 1.5 0.000

† Not applicable.

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were

used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix A: Analysis of Cohorts 1 and 2 

A-21

Table A.11. Combined Cohorts 1 and 2: Impacts on teacher-reported SACD professional 

development 

SACD professional development1Treatment Control Impact p-value

 Teacher sample size 481 459

SACD training in past 12 months (percent) 65.9* 51.3 14.6 0.000

Hours of SACD training (mean) 5.8 4.9 0.9 0.404

Training by goal (percent)

 Violence prevention and peace promotion 19.1 14.8 4.3 0.180

 Social and emotional development 23.0^ 16.6 6.4 0.066

 Character education 35.1* 18.4 16.8 0.000

 Tolerance and diversity 17.5 18.9 -1.4 0.650

 Risk prevention and health promotion 14.2 14.4 -0.2 0.933

 Civic responsibility and community service 7.4 5.3 2.0 0.232

 Behavior management 29.6 25.0 4.6 0.210

* Treatment group significantly different from control group at the .05 level. 

^ Treatment group significantly different from control group at the .10 to > .05 level.

1

Impact on domain found statistically significant after adjustments made for multiple comparisons within domain for all years.

NOTE: Weights, which assign equal weight to each school within each of the programs and to each program across programs, were

used in producing the treatment, control, and overall means.

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix A: Analysis of Cohorts 1 and 2 

A-22

The impacts of the programs on the SACD activity domains were consistent with results from the analyses of 

Cohort 1 data when students were in the fourth grade. Table A.12 shows that statistically significant larger 

percentages of treatment group teachers than control group teachers reported engaging in activities to 

promote social and character development, using more materials and strategies to promote social and 

character development, and participating in related professional development. Treatment group teachers were 

not more likely to report changes in the school environment or in teacher and staff attitudes that might be 

conducive to promoting social and character development. 

Table A.12. Combined Cohorts 1 and 2: Significant impacts on domains of use of SACD activity 

overall and by program 

SACD activity domain

Program

SACD 

activities

SACD activities 

linked to named 

programs

Classroom

materials

and strategies

Schoolwide

strategies

Professional 

development

Attitudes

and 

practices

Overall +

1,2,3 +1,2,3 +1,2,3 +2,3

ABC +

3

CSP +

3

+

2

LBW +

3

+

3

+

2

+

3

PA +

1,3

PATHS +

1,3 +1,3

4Rs +

1,3 +1,3

SS +

1,3 +1,3 +3

 

1

Based on univariate statistical tests, at least half of the impacts were positive and statistically significant and no impact was 

negative and statistically significant. 

2 The omnibus impact for all the outcomes measured together was positive and statistically significant on the basis of a multivariate 

statistical test. 

3

At least one outcome remained positive and statistically significant and no outcome was negative and statistically significant after 

applying the Benjamini-Hochberg (1995) procedure to adjust significance levels downward to account for the multiple testing of 

impacts.

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

 + Statistically significant beneficial impact on domain

 Blank cell: Finding of no impact

No detrimental impact was found statistically significant at or below the .05 level. For a description of the heuristics used to 

determine the statistically significant beneficial impact on the domain, see chapter 1.

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix A: Analysis of Cohorts 1 and 2 

A-23

Impacts on Student Outcomes: Year-by-Year and Growth Analyses Results 

Year-by-Year Analysis 

None of the 20 estimated combined-program impacts on fourth-grade child and school outcomes, based on 

the combined samples of Cohorts 1 and 2, were statistically significantly different from zero at the 5 percent 

level (table A.134). This varies from the finding of one statistically significant impact (for Student Support for 

Teachers) in the Cohort 1 analysis. The combined-sample analysis agreed with the Cohort 1 analysis that none 

of the impacts was substantively important. The estimated impacts in effect size units ranged from a 

detrimental effect of -0.06 (for child-reported Altruistic Behavior) to a beneficial effect of 0.12 standard 

deviations (for Student Support for Teachers). When the What Works Clearinghouse heuristics5

 were applied 

to the domains to adjust for multiple comparisons within each outcome domain, the fourth heuristic 

identified the significance of a detrimental impact on children’s Social and Emotional Competence (similar to 

the finding in the Cohort 1 analysis). 

 

4

 Sample sizes are not reported in the Results tables because they vary by outcome. Table A.20 provides the range of 

sample sizes for the outcomes within each report. 

5

 See the chapter 1 section headed Year-by-Year Impacts on Students and Perceptions of School Climate, subsection 

Statistical Significance and Substantively Important Effects, for a discussion of these heuristics. 
Appendix A: Analysis of Cohorts 1 and 2 

A-24

Table A.13. Combined Cohorts 1 and 2: Combined-program impacts on outcomes for fourthgraders 

Scale–Report Treatment Control 

Effect 

size1p-value2

Social and Emotional Competence Domain3

 Self-Efficacy for Peer Interactions–CR (+) 3.17 3.20 -0.05 0.194

 Normative Beliefs About Aggression–CR (-) 1.36 1.37 -0.01 0.849

 Empathy–CR (+) 2.18 2.19 -0.02 0.699

Behavior Domain

 Altruistic Behavior–CR (+) 1.06 1.10 -0.06 0.154

 Altruistic Behavior–PCR (+) 2.27 2.25 0.03 0.364

 Altruistic Behavior–TRS (+) 1.38 1.36 0.05 0.686

 Positive Social Behavior–PCR (+) 3.06 3.04 0.05 0.177

 Positive Social Behavior–TRS (+) 3.02 3.05 -0.04 0.533

 Problem Behavior–CR (-) 0.37 0.36 0.01 0.901

 Problem Behavior–PCR (-) 1.54 1.55 -0.04 0.248

 Problem Behavior–TRS (-) 1.45 1.46 -0.01 0.800

 ADHD-Related Behavior–TRS (-) 1.69 1.70 -0.02 0.624

Academics Domain

 Engagement with Learning–CR (+) 3.66 3.67 -0.02 0.680

 Academic Competence and Motivation–TRS (+) 2.97 2.99 -0.02 0.651

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 2.65 2.62 0.04 0.508

 Negative School Orientation–CR (-) 1.99 2.00 -0.02 0.653

 Student Afraid at School–CR (-) 2.24 2.28 -0.04 0.402

 Victimization at School–CR (-) 0.68 0.71 -0.03 0.533

 Feelings of Safety–TRCS (+) 3.42 3.36 0.08 0.325

 Student Support for Teachers–TRCS (+) 3.51 3.41 0.12 0.099

1 The effect size was calculated by dividing the estimated impact by the standard deviation of the outcome measure for the control 

group. 

2 The p-value is from a two-tailed t test to gauge the statistical significance of the impact estimate. 

3 Impact on domain found statistically significant and detrimental based on the fourth heuristic, in which the statistical model used to 

estimate impacts on the individual outcomes was re-estimated using a composite of all the outcome variables under a domain. The 

domain was found significant if the impact on the composite was significant. The composite was formed by standardizing each 

outcome variable using its standard deviation, combining the values of the outcome variables, and taking the average of the final 

value.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. No findings were found statistically significant at or below 

the .05 level. All impact estimates were calculated using regression models in which each program and school within a program was 

weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of 

students within schools. 

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix A: Analysis of Cohorts 1 and 2 

A-25

The lack of statistically significant beneficial impact estimates at the combined-program level might have been 

due to beneficial impacts in some programs that were offset by negative impacts in others. Differences in 

estimated impacts across programs were tested for as were the statistical significance of program-specific 

impact estimates. 

With two exceptions, the 20 impact estimates did not significantly differ by program. For 18 of the 20 

outcomes, the differences in pooled impact estimates across programs were not statistically significant at the 5 

percent level. For these outcomes, one or two programs were not driving or masking the estimated impacts in 

the combined program data. For the other two outcomes (Academic Competence and Motivation, and 

Feelings of Safety) the ABC program showed a beneficial, statistically significant result so the hypothesis that 

the estimated impacts were equal across programs was rejected. 

From the analyses of the individual SACD programs, 6 of 140 program impact estimates were statistically 

significant (versus 7 that would be expected by chance and the 7 that were found in the fourth-grade analysis 

of Cohort 1). Four of these were beneficial and 2 were detrimental (versus the 5 beneficial and 2 detrimental 

found in the analysis of Cohort 1). For the three programs that did not add a second cohort (Positive Action 

[PA], the 4Rs Program [4Rs], and Second Step [SS]), results remained the same as in the Cohort 1 analysis (2 

significant beneficial impacts, 1 significant detrimental impact, and 2 non-significant but substantively 

important beneficial impacts: see table 1.27 in chapter 1). For the four programs that included a second 

cohort, table A.14 identifies the statistically significant and the non-significant substantively important results 

from the combined cohort analysis and compares them with the similar results found in the Cohort 1 only 

analysis (table A.15 compares all the results). 

With respect to the ABC program, statistically significant beneficial impacts were found on Academic 

Competence and Motivation (effect size of 0.35) and on Feelings of Safety (effect size of 0.67), and these 

align with findings from the Cohort 1 analysis. However, the combined cohort analysis did not find a 

significant detrimental impact on Altruistic Behavior (Child Report), which did appear in the Cohort 1 

analysis. With respect to CSP, the combined cohort findings showed a statistically significant detrimental 

impact on Empathy (effect size of -0.18), which did not occur in the Cohort 1 analysis. CSP’s Cohort 1 

beneficial impact on Problem Behavior disappeared in the combined cohort analysis. LBW and PATHS had 

no significant impacts in either the Cohort 1 or the combined cohort analyses. 

With respect to non-significant but substantively important results, when the combined cohort analyses were 

compared to the Cohort 1 analyses, ABC maintained its beneficial impact for Student Support for Teachers 

(effect size of .40). CSP lost its beneficial effect for Altruistic Behavior and Student Afraid at School. LBW 

lost its beneficial impact for Student Support for Teachers and its detrimental effect on Altruistic Behavior 

(Teacher Report on Student), and it gained a detrimental effect on Positive Social Behavior (Teacher Report 

on Student) (effect size of -.25). PATHS gained a beneficial impact on Altruistic Behavior (Teacher Report on 

Student). 
Table A.14. Comparison of individual programs’ statistically significant impacts and nonsignificant substantive impacts between 

Cohort 1 and combined Cohort 1 and Cohort 2 fourth-graders 

Statistically significant1 Nonstatistically significant but substantive2

Program3

Beneficial impacts Detrimental impacts Beneficial impacts Detrimental impacts

(Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value)

Total

 Cohort 1 3 1 4 1

 Cohorts 

 1 and 2

2 1 2 1

ABC

 Cohort 1 Academic Competence Altruistic Behavior Student Support for Teachers 

(TRCS) (.27) (.276) (CR) (.31) (.011) (CR) (-.20) (.029)

Feelings of Safety

(TRCS) (.75) (.003)

 Cohorts Academic Competence Student Support for Teachers 

(TRCS) (.40) (.105)

 1 and 2 (CR) (.35) (.002)

Feelings of Safety

(TRCS) (.67) (.001)

CSP

 Cohort 1 Problem Behavior Altruistic Behavior

(PCR) (-.21) (.042) (TRS) (.47) (.132)

Student Afraid at School

(CR) (-.26) (.090)

 Cohorts Empathy

 1 and 2 (CR) (-.18) (.041)

LBW

 Cohort 1 Student Support for Teachers Altruistic Behavior

(TRCS) (.28) (.428) (TRS) (-.34) (.270)

 Cohorts Positive Social Behavior

 1 and 2 (TRS) (-.25) (.205)

See notes at end of table.

A-26

Appendix A: Analysis of Cohorts 1 and 2
Table A.14. Comparison of individual programs’ statistically significant impacts and nonsignificant substantive impacts between 

Cohort 1 and combined Cohort 1 and Cohort 2 fourth-graders—Continued

Statistically significant1 Nonstatistically significant but substantive2

Beneficial impacts Detrimental impacts Beneficial impacts Detrimental impacts

Program3(Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value) (Report) (Effect size) (p-value)

PATHS

 Cohort 1

 Cohorts Altruistic Behavior

 1 and 2 (TRS) (.34) (.250)

1 Of the 80 comparisons (4 programs with 20 outcomes apiece) made, 4 would be expected to be statistically significant at the .05 level by chance. 

2 Defined as impacts that were not statistically significant but were .25 standard deviation units (absolute value) or more in magnitude.

3 Only the four programs with a Cohort 2 are shown. The results for the other programs remained the same as those discussed in chapter 1.

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World 

 PATHS: Promoting Alternative Thinking Strategies

 CR: Child Report

 TRS: Teacher Report on Student

 PCR: Primary Caregiver Report

 TRCS: Teacher Report on Classroom and School

 Blank cell: Finding of no impact

All impact estimates were calculated using regression models in which each school within a program was weighted equally. The standard errors of all estimates account for design 

effects due to unequal weighting and the clustering of students within schools. 

SOURCE: The Social and Character Development (SACD) Research Program.

A-27

Appendix A: Analysis of Cohorts 1 and 2
A-28

Appendix A: Analysis of Cohorts 1 and 2

Table A.15. Cohort 1 and combined Cohorts 1 and 2: Comparison of program effects 

Panel 1: ABC Program 

Cohort 1 Combined Cohorts 1 and 2

(Spring 4th grade) (Spring 4th grade)

Scale–Report

Treatment ControlEffect 

size p-value

Treatment ControlEffect 

size p-value

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 3.31 3.29 0.03 0.770 3.27 3.24 0.04 0.697

 Normative Beliefs About Aggression–CR (-) 1.39 1.29 0.18 0.141 1.38^ 1.28 0.18 0.082

 Empathy–CR (+) 2.08 2.19 -0.21 0.144 2.09 2.17 -0.17 0.156

Behavior Domain

 Altruistic Behavior–CR (+) 0.94* 1.09 -0.20 0.029 1.00 1.13 -0.16 0.121

 Altruistic Behavior–PCR (+) 2.14 2.19 -0.06 0.595 2.19 2.23 -0.04 0.710

 Altruistic Behavior–TRS (+) 1.24 1.26 -0.03 0.914 1.24 1.25 -0.03 0.920

 Positive Social Behavior–PCR (+) 3.06^ 2.96 0.16 0.071 3.01 2.90 0.18 0.187

 Positive Social Behavior–TRS (+) 3.01 3.02 -0.01 0.961 3.05 2.95 0.13 0.504

 Problem Behavior–CR (-) 0.33 0.36 -0.05 0.659 0.36 0.42 -0.10 0.545

 Problem Behavior–PCR (-) 1.56 1.61 -0.11 0.229 1.58 1.65 -0.16 0.198

 Problem Behavior–TRS (-) 1.43 1.46 -0.07 0.655 1.44 1.49 -0.11 0.403

 ADHD-Related Behavior–TRS (-) 1.69 1.77 -0.13 0.525 1.68 1.77 -0.16 0.341

Academics Domain

 Engagement with Learning–CR (+) 3.68 3.63 0.10 0.281 3.69^ 3.60 0.14 0.093

 Academic Competence and Motivation–TRS (+) 2.92* 2.65 0.31 0.011 2.93* 2.63 0.35 0.002

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 2.67 2.61 0.07 0.668 2.62 2.56 0.08 0.639

 Negative School Orientation–CR (-) 1.97 1.96 0.01 0.963 2.01 2.01 -0.01 0.960

 Student Afraid at School–CR (-) 2.21 2.36 -0.17 0.263 2.23 2.39 -0.17 0.211

 Victimization at School–CR (-) 0.58 0.66 -0.11 0.213 0.71 0.70 0.02 0.923

 Feelings of Safety–TRCS (+) 3.59* 2.97 0.75 0.003 3.51* 2.94 0.67 0.001

 Student Support for Teachers–TRCS (+) 3.48 3.26 0.27° 0.276 3.49 3.15 0.40° 0.105

See notes at end of table.
A-29

Appendix A: Analysis of Cohorts 1 and 2 

Table A.15. Cohort 1 and combined Cohorts 1 and 2: Comparison of program effects—Continued

Panel 2: CSP

Cohort 1 Combined Cohorts 1 and 2

(Spring 4th grade) (Spring 4th grade)

Scale–Report 

Treatment ControlEffect 

size p-value

Treatment ControlEffect 

size p-value

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 3.15 3.15 0.00 0.978 3.16 3.13 0.05 0.475

 Normative Beliefs About Aggression–CR (-) 1.35 1.29 0.12 0.273 1.42 1.29 0.24 0.133

 Empathy–CR (+) 2.18 2.23 -0.11 0.206 2.15* 2.23 -0.18 0.041

Behavior Domain

 Altruistic Behavior–CR (+) 1.06 1.00 0.08 0.505 1.01 1.00 0.01 0.889

 Altruistic Behavior–PCR (+) 2.33 2.31 0.03 0.849 2.34 2.28 0.08 0.452

 Altruistic Behavior–TRS (+) 1.39 1.25 0.47° 0.132 1.40 1.30 0.24 0.263

 Positive Social Behavior–PCR (+) 3.11 3.02 0.17 0.125 3.11 3.02 0.15 0.153

 Positive Social Behavior–TRS (+) 3.02 2.93 0.12 0.426 3.02 3.05 -0.05 0.702

 Problem Behavior–CR (-) 0.32 0.26 0.15 0.428 0.36 0.27 0.21 0.254

 Problem Behavior–PCR (-) 1.50* 1.56 -0.21 0.042 1.50^ 1.56 -0.18 0.069

 Problem Behavior–TRS (-) 1.48 1.50 -0.03 0.852 1.46 1.44 0.04 0.784

 ADHD-Related Behavior–TRS (-) 1.70 1.74 -0.06 0.585 1.70 1.68 0.04 0.733

Academics Domain

 Engagement with Learning–CR (+) 3.71 3.71 -0.01 0.952 3.67 3.68 -0.02 0.889

 Academic Competence and Motivation–TRS (+) 2.96 2.93 0.04 0.622 3.04 3.09 -0.05 0.484

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 2.64 2.65 -0.02 0.922 2.56 2.58 -0.02 0.898

 Negative School Orientation–CR (-) 1.95 1.98 -0.05 0.699 2.02 2.00 0.04 0.794

 Student Afraid at School–CR (-) 2.17^ 2.38 -0.26° 0.090 2.26 2.31 -0.06 0.697

 Victimization at School–CR (-) 0.69 0.70 -0.01 0.947 0.69 0.71 -0.03 0.756

 Feelings of Safety–TRCS (+) 3.60 3.64 -0.05 0.834 3.63 3.65 -0.02 0.919

 Student Support for Teachers–TRCS (+) 3.55 3.48 0.10 0.551 3.59 3.55 0.06 0.634

See notes at end of table.
Table A.15. Cohort 1 and combined Cohorts 1 and 2: Comparison of program effects—Continued

Panel 3: LBW

Cohort 1 Combined Cohorts 1 and 2

(Spring 4th grade) (Spring 4th grade)

Scale–Report 

Treatment ControlEffect 

size p-value

Treatment ControlEffect

size p-value

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 3.19 3.21 -0.04 0.674 3.20 3.17 0.05 0.602

 Normative Beliefs About Aggression–CR (-) 1.26 1.21 0.12 0.368 1.28 1.21 0.19 0.124

 Empathy–CR (+) 2.30 2.30 -0.01 0.963 2.30 2.31 -0.03 0.752

Behavior Domain

 Altruistic Behavior–CR (+) 1.02 1.16 -0.20 0.191 1.00 1.13 -0.18 0.192

 Altruistic Behavior–PCR (+) 2.22 2.11 0.17 0.152 2.26^ 2.11 0.23 0.092

 Altruistic Behavior–TRS (+) 1.29 1.46 -0.34° 0.270 1.34 1.41 -0.16 0.607

 Positive Social Behavior–PCR (+) 3.12 3.04 0.13 0.250 3.10 3.03 0.12 0.262

 Positive Social Behavior–TRS (+) 2.94 3.11 -0.23 0.183 2.95 3.13 -0.25° 0.205

 Problem Behavior–CR (-) 0.25 0.23 0.04 0.728 0.26 0.25 0.02 0.854

 Problem Behavior–PCR (-) 1.56 1.56 -0.01 0.898 1.56 1.57 -0.03 0.796

 Problem Behavior–TRS (-) 1.46 1.40 0.13 0.117 1.49^ 1.39 0.22 0.063

 ADHD-Related Behavior–TRS (-) 1.73 1.64 0.13 0.105 1.78^ 1.68 0.16 0.077

Academics Domain

 Engagement with Learning–CR (+) 3.67 3.69 -0.04 0.652 3.64 3.68 -0.08 0.370

 Academic Competence and Motivation–TRS (+) 3.05 2.99 0.07 0.622 3.09 2.94 0.15 0.378

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 2.77 2.79 -0.03 0.794 2.77 2.82 -0.07 0.551

 Negative School Orientation–CR (-) 1.87 1.89 -0.03 0.846 1.90 1.90 -0.01 0.960

 Student Afraid at School–CR (-) 2.27 2.23 0.06 0.777 2.27 2.22 0.06 0.727

 Victimization at School–CR (-) 0.70 0.71 -0.01 0.929 0.69 0.70 -0.02 0.828

 Feelings of Safety–TRCS (+) 3.39 3.45 -0.06 0.825 3.35 3.53 -0.24 0.349

 Student Support for Teachers–TRCS (+) 3.57 3.40 0.28° 0.428 3.53 3.41 0.20 0.494

See notes at end of table.

A-30

Appendix A: Analysis of Cohorts 1 and 2
Table A.15. Cohort 1 and combined Cohorts 1 and 2: Comparison of program effects—Continued

Panel 4: PATHS

Cohort 1 Combined Cohorts 1 and 2

(Spring 4th grade) (Spring 4th grade)

Scale–Report

Treatment ControlEffect 

size p-value

Treatment ControlEffect 

size p-value

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 3.18 3.28 -0.16 0.261 3.12^ 3.26 -0.21 0.084

 Normative Beliefs About Aggression–CR (-) 1.26 1.37 -0.17 0.214 1.28 1.37 -0.14 0.251

 Empathy–CR (+) 2.21 2.14 0.13 0.245 2.23 2.16 0.13 0.317

Behavior Domain

 Altruistic Behavior–CR (+) 1.00 0.96 0.06 0.635 1.10 1.03 0.08 0.360

 Altruistic Behavior–PCR (+) 2.06 2.19 -0.16 0.140 2.18 2.21 -0.04 0.756

 Altruistic Behavior–TRS (+) 1.22 1.18 0.17 0.395 1.35 1.24 0.34° 0.250

 Positive Social Behavior–PCR (+) 3.05 3.06 -0.01 0.953 3.03 3.03 -0.01 0.878

 Positive Social Behavior–TRS (+) 3.14 3.04 0.15 0.503 3.06 3.06 0.00 0.997

 Problem Behavior–CR (-) 0.27 0.30 -0.06 0.547 0.34 0.31 0.06 0.656

 Problem Behavior–PCR (-) 1.53 1.53 0.00 0.987 1.54 1.54 0.02 0.869

 Problem Behavior–TRS (-) 1.42 1.42 0.01 0.957 1.46 1.47 -0.01 0.935

 ADHD-Related Behavior–TRS (-) 1.62 1.70 -0.13 0.574 1.65 1.73 -0.14 0.571

Academics Domain

 Engagement with Learning–CR (+) 3.70 3.69 0.01 0.954 3.68 3.69 -0.03 0.758

 Academic Competence and Motivation–TRS (+) 2.95 3.05 -0.09 0.272 2.85^ 3.03 -0.21 0.093

See notes at end of table.

A-31

Appendix A: Analysis of Cohorts 1 and 2
A-32

Appendix A: Analysis of Cohorts 1 and 2

Table A.15. Cohort 1 and combined Cohorts 1 and 2: Comparison of program effects—Continued

Panel 4: PATHS

Cohort 1 Combined Cohorts 1 and 2

(Spring 4th grade) (Spring 4th grade)

Scale–Report

Treatment ControlEffect 

size p-value

Treatment ControlEffect 

size p-value

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 2.79 2.67 0.15 0.640 2.81 2.67 0.18 0.461

 Negative School Orientation–CR (-) 1.87 1.90 -0.05 0.765 1.95 1.93 0.02 0.825

 Student Afraid at School–CR (-) 2.08 2.09 -0.01 0.909 2.22 2.20 0.02 0.844

 Victimization at School–CR (-) 0.65 0.56 0.12 0.306 0.69 0.63 0.08 0.382

 Feelings of Safety–TRCS (+) 3.36 3.18 0.20 0.615 3.09 3.11 -0.01 0.962

 Student Support for Teachers–TRCS (+) 3.52 3.46 0.07 0.838 3.28 3.37 -0.11 0.719

* Treatment group significantly different from control group at the .05 level.

^ Treatment group significantly different from control group at the .10 to > .05 level.

o Substantive (but nonsignificant at .05 level) effect size of > .25 or < -.25.

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program 

 CSP: Competence Support Program 

 LBW: Love In a Big World 

 PATHS: Promoting Alternative Thinking Strategies 

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

 ADHD: Attention deficit hyperactivity disorder

- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models in which each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. The effect size 

was calculated by dividing the estimated impact by the standard deviation of the outcome measure for the control group. 

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix A: Analysis of Cohorts 1 and 2 

A-33

Eight subgroup analyses were done (as discussed in chapter 1) to determine whether the SACD programs 

might be more effective for some groups of children than for others. These subgroups were defined by 

gender, stayers versus new entrants, five types of risk faced by the children, and fidelity of implementation. 

The combined cohort subgroup analyses found a total of 20 significant impacts compared to the 22 found in 

the Cohort 1 subgroup analyses. The combined cohort analysis found 14 of the same significant impacts as 

the Cohort 1 analysis, failed to find 8 of the Cohort 1 impacts, and found 6 different significant impacts that 

did not appear in the Cohort 1 analyses. Of the 8 results that were significant in the Cohort 1 analyses but not 

in the combined cohort analyses, 7 were related to risk subgroups (5 beneficial impacts for high-risk students 

and 2 detrimental impacts for high-risk students), and 1 impact was beneficial for girls. Of the 6 new findings 

in the combined cohort analyses, 4 occurred in the stayer versus new entrant subgroup (2 beneficial impacts 

and 2 detrimental impacts for new entrants), 1 was a detrimental impact for high-risk students, and 1 was a 

beneficial impact for high-fidelity schools. 

Overall, the differences between the subgroup findings from the combined cohort analyses versus the Cohort 

1 analysis are primarily less evidence that impacts are more beneficial for higher risk students and mixed 

evidence on the impacts for stayers versus new entrants. Table A.16 compares the significant subgroup 

findings from the combined cohort analysis to those of the Cohort 1 analysis, showing each subgroup on a 

separate panel (exception: no panel displays the socioeconomic risk subgroup analysis as no significant results 

were found in either the Cohort 1 or the combined cohort analyses). 
Table A.16. Cohort 1 versus Cohorts 1 and 2: Comparison of significant impacts by subgroup 

Panel 1: Gender

Cohort 1 Cohorts 1 and 2

Boys Girls p-value for 

test of 

differences

Boys Girls p-value for 

test of 

Scale–Report differences 1

Effect 

size p-value

Effect 

size p-value

Effect 

size p-value

Effect 

size p-value

Normative Beliefs About Aggression–CR (-) 0.04 0.574 -0.10^ 0.095 0.023* 0.05 0.436 -0.08 0.165 0.022*

Altruistic Behavior–CR (+) -0.13* 0.020 -0.00 0.967 0.022* -0.10^ 0.080 -0.04 0.491 0.293

Positive Social Behavior–PCR (+) 0.10* 0.032 -0.02 0.653 0.033* 0.10* 0.036 -0.02 0.658 0.021*

Engagement with Learning–CR (+) -0.10* 0.050 0.05 0.297 0.011* -0.07 0.183 0.05 0.306 0.035*

Negative School Orientation–CR (-) 0.09 0.126 -0.14* 0.011 0.000* 0.06 0.303 -0.10^ 0.083 0.001**

See notes at end of table.

Panel 2: Stayers and new entrants

Cohort 1 Cohorts 1 and 2

Scale–Report1

Stayers Entrants p-value for 

test of 

differences

Stayers Entrants p-value for 

test of 

differences

Effect 

size p-value

Effect 

size p-value

Effect 

size p-value

Effect 

size p-value

Altruistic Behavior–CR (+) -0.03 0.516 -0.14^ 0.066 0.132 -0.03 0.565 -0.14^ 0.055 0.041*

Positive Social Behavior–PCR (+) 0.01 0.813 0.13^ 0.093 0.067^ 0.00 0.970 0.13* 0.050 0.009**

Problem Behavior–PCR (-) -0.05 0.244 -0.11 0.160 0.195 -0.00 0.993 -0.14^ 0.074 0.005**

Negative School Orientation–CR (-) -0.05 0.432 0.08 0.280 0.053^ -0.06 0.297 0.06 0.467 0.047*

See notes at end of table.

A-34

Appendix A: Analysis of Cohorts 1 and 2
A-35

Appendix A: Analysis of Cohorts 1 and 2

Table A.16. Cohort 1 versus Cohorts 1 and 2: Comparison of significant impacts by subgroup—Continued 

Panel 3: Family risk

Cohort 1 Cohorts 1 and 2

Scale–Report 

Low risk1

Average 

risk2 High risk3

Marginal 

effect4Low Risk1

Average 

risk2 High risk3

Marginal 

effect4

(p-value) (p-value) (p-value) (p-value) (p-value) (p-value) (p-value) (p-value)

Student Afraid at School–CR (-) -0.175* -0.084 0.006 0.408* -0.157* -0.063 0.031 0.414*

 (0.020) (0.192) (0.932) (0.012) (0.029) (0.311) (0.660) (0.007)

See notes at end of table.

Panel 4: Community risk

Cohort 1 Cohorts 1 and 2

Scale–Report 

Low risk1

Average 

risk2 High risk3

Marginal 

effect4Low risk1

Average 

risk2 High risk3

Marginal 

effect4

(p-value) (p-value) (p-value) (p-value) (p-value) (p-value) (p-value) (p-value)

Problem Behavior–PCR (-) -0.045 0.028 0.101* 0.109* -0.062 0.028 0.118* 0.136**

 (0.381) (0.446) (0.041) (0.032) (0.208) (0.442) (0.013) (0.005)

Problem Behavior–TRS (-) 0.087 -0.001 -0.089 -0.132* 0.087 0.020 -0.048 -0.101^

 (0.268) (0.994) (0.243) (0.022) (0.254) (0.764) (0.511) (0.060)

ADHD-Related Behavior–TRS (-) 0.062 -0.021 -0.104 -0.124* 0.048 -0.004 -0.057 -0.079

 (0.389) (0.722) (0.132) (0.029) (0.481) (0.938) (0.380) (0.130)

Positive School Orientation–CR (+) -0.086 0.030 0.146 0.174** -0.051 0.039 0.129 0.135*

 (0.361) (0.719) (0.111) (0.007) (0.560) (0.607) (0.124) (0.023)

Victimization at School–CR (-) 0.068 -0.035 -0.138* -0.154* 0.054 -0.026 -0.106 -0.120*

 (0.298) (0.485) (0.030) (0.011) (0.432) (0.639) (0.106) (0.037)

See notes at end of table.
A-36

Appendix A: Analysis of Cohorts 1 and 2

Table A.16. Cohort 1 versus Cohorts 1 and 2: Comparison of significant impacts by subgroup—Continued 

Panel 5: Initial child behavior risk as reported by teacher 

Cohort 1 Cohorts 1 and 2

Scale–Report 

Low risk1

Average 

risk2 High risk3

Marginal 

effect4Low risk1

Average 

risk2 High risk3

Marginal 

effect4

(p-value) (p-value) (p-value) (p-value) (p-value) (p-value) (p-value) (p-value)

Normative Beliefs About Aggression–

 CR (-)

0.127 0.007 -0.113 -0.012** 0.146^ 0.030 -0.086 -0.011**

 (0.134) (0.923) (0.191) (0.005) (0.068) (0.671) (0.285) (0.003)

Problem Behavior–CR (-) 0.075 -0.007 -0.089 -0.008* 0.077 0.013 -0.052 -0.006^

 (0.348) (0.922) (0.275) (0.032) (0.320) (0.856) (0.507) (0.068)

Problem Behavior–TRS (-) 0.064 -0.011 -0.085 -0.007* 0.075 0.022 -0.030 -0.005^

 (0.389) (0.872) (0.255) (0.029) (0.323) (0.746) (0.690) (0.099)

Academic Competence and Motivation–

 TRS (+)

-0.108* -0.032 0.044 0.007** -0.119* -0.046 0.027 0.007**

 (0.035) (0.455) (0.391) (0.008) (0.017) (0.277) (0.587) (0.006)

Negative School Orientation–CR (-) 0.057 -0.072 -0.202* -0.013** 0.066 -0.047 -0.160* -0.011**

 (0.467) (0.300) (0.013) (0.001) (0.364) (0.457) (0.031) (0.002)

See notes at end of table.
A-37

Appendix A: Analysis of Cohorts 1 and 2

Table A.16. Cohort 1 versus Cohorts 1 and 2: Comparison of significant impacts by subgroup—Continued 

Panel 6: Initial child behavior risk as reported by primary caregiver 

Cohort 1 Cohorts 1 and 2

Scale–Report 

Low risk1

Average 

risk2 High risk3

Marginal 

effect4Low risk1

Average 

risk2 High risk3

Marginal 

effect4

(p-value) (p-value) (p-value) (p-value) (p-value) (p-value) (p-value) (p-value)

Normative Beliefs About Aggression–

 CR (-)

0.071 -0.010 -0.092 -0.008* 0.080 0.010 -0.061 -0.007^

 (0.403) (0.893) (0.287) (0.036) (0.325) (0.989) (0.460) (0.062)

Empathy–CR (+) -0.083 -0.007 0.068 0.008* -0.056 -0.003 0.049 0.005

 (0.217) (0.897) (0.318) (0.042) (0.375) (0.948) (0.445) (0.138)

Altruistic Behavior–TRS (+) -0.049 0.029 0.107 0.008* -0.004 0.060 0.123 0.006^

 (0.780) (0.864) (0.542) (0.050) (0.979) (0.693) (0.429) (0.095)

Positive Social Behavior–PCR (+) -0.086^ -0.007 0.072 0.008* -0.080 -0.015 0.051 0.006*

 (0.083) (0.852) (0.147) (0.015) (0.106) (0.701) (0.310) (0.037)

Problem Behavior–PCR (-)

-0.021 0.036 0.092^ 0.006 -0.028 0.037 0.103* 0.007*

 (0.691) (0.363) (0.080) (0.101) (0.571) (0.323) (0.043) (0.048)

Engagement with Learning–CR (+) -0.089 -0.004 0.081 0.009* -0.071 0.016 0.102^ 0.009*

 (0.127) (0.925) (0.178) (0.028) (0.196) (0.707) (0.070) (0.018)

Negative School Orientation–CR (-) 0.055 0.055 -0.178* -0.012** 0.056 -0.044 -0.143^ -0.010**

 (0.492) (0.492) (0.030) (0.002) (0.447) (0.501) (0.056) (0.006)

See notes at end of table.
A-38

Appendix A: Analysis of Cohorts 1 and 2

Table A.16. Cohort 1 versus Cohorts 1 and 2: Comparison of significant impacts by subgroup—Continued 

Panel 7: Fidelity of implementation

Cohort 1 Cohorts 1 and 2

Scale–Report 

Low effect 

size

Mixed effect 

size

High effect 

size p-value for 

test of 

differences

Low effect 

size

Mixed effect 

size

High effect 

size p-value for 

test of 

differences

Effect 

size p-value

Effect 

size p-value

Effect 

size p-value

Effect 

size p-value

Effect 

size p-value

Effect 

size p-value

Normative Beliefs About 

 Aggression–CR (-) 0.06 0.419 0.04 0.584 -0.14 0.148 0.068^ 0.11 0.174 0.05 0.547 -0.14 0.129 0.042*

* Significantly different from zero at the .05 level.

** Significantly different from zero at the .01 level.

^ Significantly different from zero at the .10 to > .05 level.

1 One standard deviation below the mean risk level.

2 At the mean risk level.

3 One standard deviation above the mean risk level.

4 Change in impact as risk level increases by one unit.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models in which each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. P-values for the 

effect sizes are shown within the parentheses below the effect size.

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix A: Analysis of Cohorts 1 and 2 

A-39

Growth Analysis 

Regarding the growth analysis, none of the 18 estimated combined program impacts on growth in fourthgrade child and school outcomes was statistically significantly different from zero at the 5 percent level (table 

A.17). The lack of statistically significant beneficial impact estimates at the combined-program level did not 

appear to be due to beneficial impacts in some programs or subgroups that were offset by negative impacts in 

others. 
Table A.17. Combined Cohorts 1 and 2: Impacts on growth of child outcomes from combined-program analysis 

Average growth in the score per year1

Scale–Report

Mean score at 

implementation2

Treatment 

group

Control 

group

Impact on 

growth3

Effect 

size4

Standard 

error of

impact

p-value of 

impact

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 2.92 0.18 0.18 0.00 0.00 0.02 0.854

 Normative Beliefs About Aggression–CR (-) 1.24 0.09 0.08 0.00 0.01 0.03 0.885

 Empathy–CR (+) 2.41 -0.16 -0.15 -0.01 -0.02 0.02 0.705

Behavior Domain

 Altruistic Behavior–CR (+) 1.45 -0.25 -0.24 -0.02 -0.02 0.03 0.549

 Altruistic Behavior–PCR (+) 2.31 -0.06 -0.05 -0.01 -0.02 0.02 0.559

 Altruistic Behavior–TRS (+) 1.40 -0.02 0.01 -0.04 -0.08 0.05 0.498

 Positive Social Behavior–PCR (+) 2.98 0.05 0.04 0.01 0.03 0.01 0.371

 Positive Social Behavior–TRS (+) 3.03 -0.04 0.01 -0.05 -0.07 0.04 0.205

 Problem Behavior–CR (-) 0.25 0.08 0.08 0.00 0.01 0.02 0.827

 Problem Behavior–PCR (-) 1.57 -0.02 -0.01 -0.01 -0.02 0.01 0.486

 Problem Behavior–TRS (-) 1.38 0.06 0.07 -0.01 -0.02 0.02 0.654

 ADHD-Related Behavior–TRS (-) 1.74 -0.01 0.00 -0.01 -0.02 0.03 0.624

Academics Domain

 Engagement with Learning–CR (+) 3.67 -0.01 -0.02 0.01 0.02 0.02 0.591

 Academic Competence–TRS (+) 2.90 0.04 0.06 -0.02 -0.02 0.03 0.460

See notes at end of table.

A-40

Appendix A: Analysis of Cohorts 1 and 2
Table A.17. Combined Cohorts 1 and 2: Impacts on growth of child outcomes from combined-program analysis—Continued 

Average growth in the score per year1

Scale–Report 

Mean score at 

implementation2

Treatment

group

Control 

group

Impact on 

growth3

Effect 

size4

Standard 

error of 

impact

p-value of 

impact

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 3.13 -0.31 -0.33 0.02 0.03 0.03 0.365

 Negative School Orientation–CR (-) 1.87 0.08 0.09 -0.01 -0.02 0.02 0.528

 Student Afraid at School–CR (-) 2.42 -0.13 -0.08 -0.05 -0.05 0.03 0.102

 Victimization at School–CR (-) 0.79 -0.08 -0.03 -0.04 -0.05 0.03 0.134

1 Pertains to the estimated slope of the outcome for the treatment or control groups.

2 The average score at implementation was calculated across treatment and control groups, using regression models for adjustment on covariates. 

3 Estimated difference between the slope of the treatment and control groups.

4 The slope of the treatment group minus the slope of the control group divided by the standard deviation of the outcome for the multisite control group (the standard deviation was 

calculated without accounting for school-level clustering or regression adjustments).

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. No findings were found statistically significant at or below the .05 level. All impact estimates were calculated 

using HLM 6.06. Sample weights were used in all analyses to (1) give each program equal weight within each time period, (2) give each school equal weight in each program (within 

each time period), and (3) give each time period equal weight in the analysis. 

SOURCE: The Social and Character Development (SACD) Research Program.

A-41

Appendix A: Analysis of Cohorts 1 and 2
Appendix A: Analysis of Cohorts 1 and 2 

A-42

From the program-level analysis, 12 impact estimates were found to be statistically significant out of 126 

results (more than the 6 that would be expected by chance). These are listed in table A.18, which shows that 4 

were beneficial impacts on growth in student outcomes, and 8 were detrimental impacts. 

Table A.18. Combined Cohorts 1 and 2: Statistically significant impacts from the growth curve 

analyses of the individual programs

Program

Significant beneficial impacts

(Report) (Effect size) (p-value)

Significant detrimental impacts

(Report) (Effect size) (p-value)

Total 4 8

ABC Positive Social Behavior

(PCR) (0.20) (0.046)

CSP Victimization at School

(CR) (-0.18) (0.038)

Academic Competence

(TRS) (-0.15) (0.025)

Normative Beliefs About Aggression

(CR) (0.33) (0.031)

LBW Problem Behavior

(CR) (0.14) (0.031)

Normative Beliefs About Aggression

(CR) (0.13) (0.030)

PA Altruistic Behavior

(PCR) (0.19) (0.021)

PATHS

4Rs Problem Behavior

(TRS) (-0.21) (0.048)

SS Academic Competence

(TRS) (-0.15) (0.013)

Positive Social Behavior

(TRS) (-0.35) (0.008)

Problem Behavior

(TRS) (0.40) (0.010)

ADHD-Related Behavior

(TRS) (0.28) (0.000)

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

 Blank cell: Finding of no impact

All impact estimates were calculated using HLM 6.06. Sample weights were used in all analyses to (1) give each program equal 

weight within each time period, (2) give each school equal weight in each program (within each time period), and (3) give each time 

period equal weight in the analysis. 

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix A: Analysis of Cohorts 1 and 2 

A-43

The subgroup analyses found 12 impact estimates that significantly differed by subgroup out of the 126 

results (more than the 6 that would be expected by chance). Table A.19 shows the statistically significant 

impacts for the subgroups. The gender analysis found 1 significant beneficial impact for girls: a decline in 

Negative School Orientation. New entrants had detrimental impacts for growth in both Positive School 

Orientation and Negative School Orientation. For the initial risk analyses, 9 impacts were found to be 

significant, with 8 showing beneficial impacts on growth for high-risk students and 1 showing detrimental 

impacts on high-risk students. Subgroup-specific regression results showed that 2 of the beneficial impacts 

were actually due to detrimental growth effects for low-risk students (Academic Competence in regards to 

child behavior risk as reported by both the teacher and the primary caregiver). In addition, they showed that 

the one finding of a detrimental impact on high-risk students was due to a beneficial impact on low-risk 

students (Problem Behavior reported in the Primary Caregiver Report in regards to community risk). 
Appendix A: Analysis of Cohorts 1 and 2 

A-44

Table A.19. Combined Cohorts 1 and 2: Statistically significant impacts from the growth curve 

analyses of the subgroups 

Subgroup Outcome Impact on subgroup

Gender Negative School Orientation

 (p-value of impact = 0.033)

Beneficial to girls

New entrants/stayers Positive School Orientation

 (p-value of impact = 0.049)

Detrimental to new entrants

Negative School Orientation

 (p-value of impact = 0.034)

Socioeconomic risk Academic Competence

 (Interaction term = 0.05)

 (p-value of impact = 0.032)

Beneficial to high risk

Family risk Victimization at School

 (Interaction term = -0.20)

 (p-value of impact = 0.026)

Beneficial to high risk

Community risk Problem Behavior (PCR)

 (Interaction term = 0.04)

 (p-value of impact = 0.013)

Detrimental to high risk due to 

 beneficial for low risk

Problem Behavior (TRS)

 (Interaction term = -0.05)

 (p-value of impact = 0.025)

Beneficial to high risk

Positive School Orientation

 (Interaction term = 0.11)

 (p-value of impact = 0.001)

Beneficial to high risk

Child behavior risk (TRS) Academic Competence

 (Interaction term = 0.03)

 (p-value of impact = 0.002)

Beneficial to high risk due to 

 detrimental for low risk

Positive Social Behavior (PCR)

 (Interaction term = 0.02)

 (p-value of impact = 0.016)

Beneficial to high risk

Child behavior risk (PCR) Academic Competence

 (Interaction term = 0.02)

 (p-value of impact = 0.010)

Beneficial to high risk due to

 detrimental for low risk

Positive Social Behavior (PCR)

 (Interaction term = 0.02)

 (p-value of impact = 0.002)

Beneficial to high risk

NOTE: Abbreviations are

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

All impact estimates were calculated using HLM 6.06. Sample weights were used in all analyses to (1) give each program equal 

weight within each time period, (2) give each school equal weight in each program (within each time period), and (3) give each time 

period equal weight in the analysis. 

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix A: Analysis of Cohorts 1 and 2 

A-45

Summary 

As a result of an effort to increase the sample size and corresponding statistical power of the SACD 

evaluation, a second cohort of 12 schools was added in Year 2. Six of the schools implemented one of four 

SACD programs while the other six acted as their control schools. Overall, the analyses of the combined 

cohort sample produced results similar to those of the analyses of the Cohort 1 data. 

Responses from students, primary caregivers, and teachers from the original 83 and the 12 new schools 

during the year students were in the fourth grade were combined. The combined treatment and control 

groups remained similar, as they were in the Cohort 1 study, although the Cohort 2 students were significantly 

different from the Cohort 1 students in that they came from more disadvantaged households and 

communities. The analyses done with the fourth-grade Cohort 1 students were replicated with the students of 

Cohorts 1 and 2 combined. 

The results of the analysis of teachers’ activities and professional development, pooled across programs, were 

similar to the results for the Cohort 1 data. These results showed that statistically significant larger 

percentages of treatment group teachers than control group teachers reported engaging in activities to 

promote social and character development, using materials and strategies to promote social and character 

development, and participating in related professional development. Treatment group teachers were no more 

likely to report changes in the school environment or in teacher and staff’s attitudes that might be conducive 

to promoting social and character development. 

The findings from the combined cohort study were similar to those from the Cohort 1 study. In the year-byyear analysis, the seven SACD combined programs had no significant impact on the 20 outcomes as 

compared to the impact on one outcome (Student Support for Teachers) found in the analysis of Cohort 1. 

Regarding the domains, the analysis found a significant detrimental effect on the domain of Social and 

Emotional Competence. The lack of impacts at the combined-program level was not due to beneficial 

impacts of some programs being offset by detrimental impacts of other programs. For 18 of the 20 

outcomes, there were no statistically significant differences in impacts across the programs (the two 

exceptions were due to significant positive impacts by the ABC Program). 

The individual program impacts for the four programs that had Cohort 2 students showed little or no 

improvement over those of the Cohort 1 analysis. One program lost a significant beneficial impact and gained 

a significant detrimental impact (CSP). One program lost a substantively important beneficial impact and a 

substantively important detrimental impact and gained a substantively important detrimental impact (LBW). 

One program gained a substantively important beneficial impact (PATHS), and one program lost a significant 

detrimental effect (ABC). 

The findings from subgroup analyses of the combined cohorts were the same for 14 out of 22 impacts that 

were found significant in the Cohort 1 subgroup analysis. The differences between the findings were primarily 

that the analyses of the combined cohorts provided less evidence of more beneficial impacts for higher risk 

students and mixed evidence regarding impacts for stayers versus new entrants. 

For the growth curve analysis, the results, though not directly comparable to the Cohort 1 analysis, were 

similar. The seven SACD programs, combined, had no significant impact on the 18 outcomes. The individual 

program analysis found a small number of significant impacts on student growth in outcomes but two-thirds 

of these were detrimental impacts. One difference was that the initial risk analyses had slightly more findings 

than would be expected by chance, and the majority of these showed more beneficial impacts for higher risk 

students. 
Appendix A: Analysis of Cohorts 1 and 2 

A-46

Table A.20. Sample size ranges for outcome analyses 

Outcome analysis Group Report

Minimum 

of range

Maximum 

of range

Combined programs Treatment CR 2,389 2,409

PCR 1,833 1,859

TRS 2,450 2,498

TRCS 478 479

Control CR 2,098 2,121

PCR 1,651 1,671

TRS 2,208 2,221

TRCS 458 459

Individual programs

 ABC Treatment CR 258 259

PCR 182 185

TRS 259 271

TRCS 55 55

Control CR 342 345

PCR 257 257

TRS 357 357

TRCS 72 72

 CSP Treatment CR 428 436

PCR 323 326

TRS 420 444

TRCS 81 81

Control CR 406 416

PCR 339 341

TRS 433 433

TRCS 82 82

 LBW Treatment CR 359 360

PCR 312 316

TRS 379 387

TRCS 75 76

Control CR 282 284

PCR 225 227

TRS 295 297

TRCS 56 56

 PA Treatment CR 280 284

PCR 212 214

TRS 289 302

TRCS 43 43

Control CR 220 225

PCR 181 183

TRS 249 251

TRCS 39 39

See notes at end of table.
Appendix A: Analysis of Cohorts 1 and 2 

A-47

Table A.20. Sample size ranges for outcome analyses—Continued

Outcome analysis Group Report

Minimum 

of range

Maximum 

of range

 PATHS Treatment CR 305 308

PCR 233 239

TRS 315 315

TRCS 74 74

Control CR 262 263

PCR 198 199

TRS 271 271

TRCS 76 76

 4Rs Treatment CR 372 377

PCR 241 246

TRS 399 404

TRCS 90 91

Control CR 292 299

PCR 206 209

TRS 307 320

TRCS 82 82

 SS Treatment CR 383 387

PCR 330 335

TRS 381 383

TRCS 59 60

Control CR 288 291

PCR 245 255

TRS 292 293

TRCS 51 52

Subgroups

 Gender Boys CR 2,124 2,144

PCR 1,643 1,669

TRS 2,240 2,261

Girls CR 2,363 2,386

PCR 1,825 1,845

TRS 2,400 2,441

 New entrants Stayers CR 3,461 3,495

PCR 2,699 2,731

TRS 3,607 3,651

New entrants CR 1,026 1,036

PCR 785 799

TRS 1,051 1,069

 Initial socioeconomic risk CR 4,062 4,106

PCR 3,407 3,448

TRS 4,218 4,261

See notes at end of table.
Appendix A: Analysis of Cohorts 1 and 2 

A-48

Table A.20. Sample size ranges for outcome analyses—Continued

Outcome analysis Group Report

Minimum 

of range

Maximum 

of range

 Initial family risk CR 3,068 3,099

PCR 2,531 2,561

TRS 3,195 3,222

 Initial community risk CR 3,019 3,051

PCR 2,501 2,529

TRS 3,140 3,166

 Initial child behavior risk 

 (Teacher Report) CR 3,288 3,322

PCR 2,582 2,610

TRS 3,430 3,468

 Initial child behavior risk 

 (Primary Caregiver 

 Report) CR 3,050 3,082

PCR 2,522 2,551

TRS 3,179 3,206

 Fidelity of implementation High CR 1,749 1,767

PCR 1,423 1,442

TRS 1,788 1,813

Mixed CR 848 851

PCR 677 684

TRS 893 904

Low CR 1,887 1,909

PCR 1,381 1,402

TRS 1,974 2,001

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 TRCS: Teacher Report on Classroom and School

SOURCE: The Social and Character Development (SACD) Research Program.
B-1 

Appendix B: Technical Notes on the Development of 

Outcome Measures, Selection of Covariates, Construction of 

Sample Weights, and Sensitivity Analyses 
This page is intentionally blank. 
B-3 

Technical Notes on the Development of Outcome Measures, 

Selection of Covariates, Construction of Sample Weights, 

and Sensitivity Analyses 

This technical appendix supplements the Study Design and Methodology section of chapter 1 with additional 

details on the methods used for the SACD impact analysis. It provides information about the development of 

the outcome measures, the selection of covariates that were included in the regression models, the sample 

weights, and the sensitivity analysis used to examine the robustness of the impact estimates to alternative 

parameter assumptions and specifications. 

Development of the Outcome Measures 

This section describes the procedures used to create and validate the survey-based child and school outcome 

measures. The majority of the discussion concerns the 18 student-level outcome measures, with the 2 schoollevel outcome measures discussed at the end of the section. The development of the outcome measures was 

directed by the Social and Character Development (SACD) Statistical Workgroup, which was made up of 

representatives from the National Center for Education Research, Institute of Education Sciences (IES), U.S. 

Department of Education; the Division of Violence Prevention in the National Center for Injury Prevention 

and Control, Centers for Disease Control and Prevention (CDC); SACD grantees; Mathematica Policy 

Research, Inc. (MPR); and the University of Missouri-St. Louis. Each outcome measure is a scale composite 

of multiple Likert-type items, derived from initial (fall 2004) survey responses to the Child Report, Primary 

Caregiver Report, and the Teacher Report on Student.1

 For example, the five student-level measures created 

from the Teacher Report on Student are Positive Social Behavior, Problem Behavior, Altruistic Behavior, 

ADHD-Related Behavior, and Academic Competence and Motivation. A list of all scales for each respondent 

group is provided in table B.1. 

The surveys were either self-administered or, for some primary caregivers, administered as a computerassisted telephone interview. The items included in the initial versions of the teacher-, primary caregiver-, and 

child-report surveys were largely taken from instruments used in previous research. Four measures (child 

responsibility, child and teacher perceptions of school safety, and primary caregiver perceptions of 

community resources) were generated by SACD Consortium members due to a lack of adequate existing 

measures of those constructs. Preliminary analyses of the measures administered in the initial wave of data 

collection revealed that, although most of the scales performed adequately and as expected, some predefined 

scales had poor internal consistency (Cronbach’s alphas  0.60 for primary caregiver report on the Behavior 

Assessment System for Children [BASC] Conduct Problems Subscale and child report on the Nonconflict 

Subscale items of the Children’s Self-Efficacy for Peer Interaction Scale). Furthermore, a small number of the 

scale or subscale scores were highly intercorrelated (for example, r  0.80 for the correlation between teacher 

report on the Responsibility Scale and teacher report on the Social Competence Scale). Consequently, the 

SACD Statistical Workgroup conducted more rigorous analyses of the items to determine whether a set of 

outcome measures with better psychometric characteristics could be derived. 

 

1

 In fall 2004, data collection began at an original cohort of schools and focused on students who were beginning third 

grade. Students at these schools were followed through spring of fifth grade. A second cohort of schools was 

subsequently included in the study; data collection at these schools focused on students who started third grade in fall 

2005; these students were followed through the spring of fourth grade. The analyses discussed here pertain to the first 

cohort of schools and students. 
Appendix B: Technical Notes 

B-4 

Table B.1. Scales and internal consistency for child outcomes, by report type 

Scale (Cronbach’s α)

Teacher Report on Student Child Report

 Positive Social Behavior (0.97) Positive School Orientation (0.86)

 Problem Behavior (0.95) Engagement with Learning (0.84)

 Altruistic Behavior (0.89) Student Afraid at School (0.79)

 ADHD-Related Behavior (0.91) Negative School Orientation (0.78)

 Academic Competence and Motivation (0.95) Self-Efficacy for Peer Interaction (0.83)

 Normative Beliefs About Aggression (0.83)

Primary Caregiver Report Empathy (0.78)

 Positive Social Behavior (0.93) Altruistic Behavior (0.88)

 Problem Behavior (0.86) Problem Behavior (0.86)

 Altruistic Behavior (0.88) Victimization at School (0.86)

NOTE: Internal consistency is Cronbach’s α (alpha), measured using a random sample of one-half of the respondents to the fall 

2004 survey administration. Consistency remained similarly high at all waves. Listwise deletion of cases with missing data was used 

in all analyses. ADHD is the abbreviation for attention deficit hyperactivity disorder.

SOURCE: The Social and Character Development (SACD) Research Program.

The approach the SACD Statistical Workgroup used was, first, to conduct exploratory factor analyses. Using 

half of the sample, selected randomly, researchers looked at the individual items of the fall 2004 survey 

separately for each group of respondents (teachers, primary caregivers, and children). These analyses were 

conducted separately for each group of respondents because the survey instruments for teachers, primary 

caregivers, and children contained different sets of items. Although a small number of measures were 

administered to more than one respondent group (see below), most of the measures were administered to 

only a single group of respondents. Potential outcome scales identified in these exploratory analyses were 

then subjected to confirmatory factor analyses with the remaining half of the sample (hereafter referred to as 

the “validation sample”) using structural equation modeling procedures.

For each of the three survey respondent groups, a set of exploratory factor analyses was undertaken using the 

principal axis method. Based on both conceptual and empirical (such as eigenvalues) criteria, varying numbers 

of factors were extracted and rotated in these analyses, and then compared with respect to the conceptual 

clarity of the factors, the nature and extent of cross-loading of items, the extent to which factors were defined 

by a very small number of items (or single items), and parsimony (the absence of multiple factors appearing 

to assess the same basic construct). On the basis of these analyses, it was decided that the item responses of 

teachers, primary caregivers, and children were optimally represented by 5, 3, and 10 underlying factors, 

respectively. The names and internal consistencies of the scales are listed in table B.1.2

 

Confirmatory Factor Analyses of Student-Level Outcome Measures 

The factors identified in the exploratory analyses were subjected to empirical confirmation using the 

validation samples. Analyses of model fit were conducted using LISREL software for the analysis of 

covariance structures. The focus was on two measures of model fit: the comparative fit index (CFI) and the 

root mean square error of approximation (RMSEA). The CFI runs from 0 to 1 and gauges the percentage 

improvement in reproducing the input covariance matrix obtained by using the proposed measurement 

model as compared to a null model that assumes no shared variance (and thus no latent factors). Higher 

 

2

 A small number of items were dropped from further analyses because they did not load on any of the identified factors 

(loadings were not ≥ 0.30 on any factor). Specifically, 8 of 91 items in the child survey (9%), 3 of 74 items in the teacher 

survey (4%), and 6 of 59 items in the primary caregiver survey (10%) were dropped from analysis. Items that crossloaded on other factors (an item was considered to cross-load on two or more factors when factor loadings were greater 

than 0.30) were included on the factor for which the loading was stronger. 
Appendix B: Technical Notes 

B-5 

values represent improved fit and, while all fit standards are somewhat arbitrary, researchers have variously 

suggested either .90 or .95 as cutoffs for good fit. The RMSEA is an index of lack of fit (based on the χ2) per 

degree of freedom. The RMSEA also runs from 0 to 1, but higher values reflect poorer fit. Suggested cutoffs 

for acceptable fit vary from .08 to .10. 

For the teacher, the primary caregiver, and the child respondents, the hypothesized factor representations 

provided a good fit to the data, as estimated using LISREL structural equations modeling software. As shown 

in table B.2, for the 71 items in the teacher survey, the confirmatory factor analysis of five latent factors 

yielded a CFI value of 0.98 and an RMSEA of 0.090 (90% confidence interval [CI] = 0.089, 0.091). For the 53 

primary caregiver items, the confirmatory factor analysis of three latent factors yielded a CFI of 0.94 and an 

RMSEA of 0.087 (90% CI = 0.086, 0.088); for the 83 child items, the confirmatory factor analysis of 10 latent 

factors yielded a CFI of 0.91 and an RMSEA of 0.060 (90% CI = 0.059, 0.061). 
Appendix B: Technical Notes 

B-6 

Table B.2. Fit indexes from confirmatory factor analyses, five-factor measurement model: First 

four waves of data collection, by respondent type 

Respondent type/Time of data collection Sample size RMSEA CFI

Teachers (5 factors)

 Initial 2,040 0.090 0.980

 Follow-up 1 3,538 0.114 0.967

 Follow-up 2 3,410 0.102 0.969

 Follow-up 3 3,580 0.107 0.970

 Follow-up 4 3,178 0.110 0.969

Primary Caregivers (3 factors)

 Initial 1,389 0.087 0.940

 Follow-up 1 2,643 0.079 0.956

 Follow-up 2 2,745 0.081 0.957

 Follow-up 3 2,639 0.082 0.954

 Follow-up 4 2,355 0.084 0.957

Children (10 factors)

 Initial 1,314 0.060 0.910

 Follow-up 1 3,074 0.064 0.943

 Follow-up 2 3,247 0.067 0.947

 Follow-up 3 3,307 0.066 0.952

 Follow-up 4 3,412 0.066 0.954

NOTE: Abbreviations are

 CFI: Comparative fit index

 RMSEA: Root mean square error of approximation

Listwise deletion of cases with missing data was used in all analyses. Confirmatory factor models were estimated using the LISREL 

structural equation modeling package, with latent factor variances fixed to 1.0 to scale the factors. Initial data were collected in fall 

2004. Follow-up 1 was in spring 2005, Follow-up 2 was in fall 2005, Follow-up 3 was in spring 2006, and Follow-up 4 was in spring 

2007. The fall 2004 results are from analyses of random samples of one-half of the fall 2004 respondents of each respective 

respondent type. Results for the follow-ups are derived from analyses of full samples (after listwise deletion).

SOURCE: The Social and Character Development (SACD) Research Program.

Robustness Checks 

The analyses of the fall 2004 data indicated that, for the sample as a whole, the child outcome variables could 

be adequately represented by a relatively small number of scales. Further analyses were performed to test the 

robustness of the model (1) for major demographic subgroups, (2) across samples at different study sites, (3) 

to treatment of missing data, (4) to different estimation approaches, and (5) across different survey waves. 

Demographic and Site Checks 

A series of multigroup confirmatory analyses were conducted to investigate whether the identified 

measurement models held across major child demographic groups (gender and race/ethnicity) and across the 

seven individual SACD programs. In each case, a measurement model in which all factor loadings and factor 

covariances were constrained to be equal across groups (but error variances were allowed to vary) provided a 
Appendix B: Technical Notes 

B-7 

good fit to the multigroup data.3 The basic measurement model thus proved to be valid across child gender, 

across child race/ethnicity, and for each of the seven SACD programs. 

Missing Items 

A second set of confirmatory analyses were conducted to examine whether the measurement models were 

robust to differences in the treatment of missing item-response data. Primary analyses were restricted to 

observations with no missing items. Although preliminary analyses indicated that missing responses to 

individual survey items were infrequent among each of the respondent groups (no more than 5% of the 

responses were missing for any survey item), listwise deletion had a cumulative effect of excluding one-third 

or more of the respondents in each group. Analyses of missing data showed that respondents with valid 

scores on the outcome measures were similar to those with missing data, suggesting that data were missing at 

random. Nonetheless, to investigate potential biases due to exclusion of cases with missing responses to 

survey items, the confirmatory factor analyses were re-run with missing item responses imputed using the 

Expectation-Maximization (EM) algorithm.4

 For each respondent group, the use of EM imputation resulted 

in a fit as good as or better than that of models on samples that used listwise deletion. 

Estimation Procedures 

Approaches to evaluating model fit vary across software packages. LISREL uses a “weighted least squares” 

chi-square independence model. By contrast, another structural equation modeling package, EQS, uses a 

“minimum fit function” chi-square independence model. For equivalent models, LISREL and EQS should 

produce the same normal theory chi-square and RMSEA values but would produce different measures of 

relative fit, such as CFI, due to the use of different chi-square independence models. Analyses performed 

using EQS showed that, as expected, LISREL yielded a higher CFI for each model than EQS.5 However, the 

other fit indexes, which are not based on choice of chi-square independence model, were actually better as 

estimated by EQS than LISREL, and both software packages indicated that the measurement models 

provided a good fit to the data. 

Successive Data Waves 

Finally, it is possible that the SACD programs might influence not only the mean scores on the measured 

outcomes but also the correlations among the measured outcomes, and thus have an impact on the quality of 

the measurement model. Analyses were performed to test the continuing fit of the model with each new wave 

of data. In addition to presenting measures of model fit from the initial survey wave, table B.2 also presents 

corresponding measures for each of the succeeding follow-up waves for the original cohort. Those results 

confirm that the model has held up well over time. There was minimal variation in either of the fit statistics 

across survey waves. 

 

3

 All multigroup confirmatory factor analyses were conducted using the random half-samples of respondents used in the 

initial confirmatory analyses. 

4

 The EM algorithm (see Dempster, Laird, and Rubin 1977) is an iterative, maximum-likelihood procedure that uses 

information from observed data to successively estimate, evaluate, and re-estimate values for missing data until it 

converges on a unique maximum-likelihood estimate. 

5

 The independence model specifies that the covariance matrix of the observed variables is a diagonal matrix with all 

observed variables uncorrelated. Although the computational formulas are complicated (see Joreskog et al. 2003), the 

weighted least squares independence model chi-square (LISREL default) yields larger values than the minimum fit 

function independence model chi-square (EQS default). As a result, incremental fit indexes such as the CFI will be larger 

in LISREL than in EQS, because all such indexes are computed as some function of the chi square (1  [model chisquare/independence chi-square]). As should be the case, when the same chi-square independence model is used to 

evaluate model fit in LISREL and EQS, the two software packages produce virtually identical estimates of CFI and all 

other model fit indexes that are evaluated against the independence model. 
Appendix B: Technical Notes 

B-8 

Construct and Reporter-Based Variance 

The measurement models described above all derived measures separately for each respondent group 

(children, primary caregivers, and teachers). However, comparable measures were found among the 

respondent groups—such as Altruistic Behavior, Positive Social Behavior, and Problem Behavior. A series of 

exploratory “multitrait, multirespondent” (MTMR) analyses were undertaken to investigate the extent of 

convergence across respondents in these commonly measured outcomes—whether responses predominantly 

reflected common perceptions of child behaviors across respondents or whether they were largely 

idiosyncratic to the respondent. That is, these efforts represented an attempt to distinguish construct variance 

from systematic variance due to the respondent (as well as random measurement error). 

Systematic differences between respondents may represent reporter bias or may reflect differences associated 

with the different types of information available to each respondent when making judgments. For example, 

primary caregivers and teachers differ in their overall familiarity with a child and in the nature of the 

situations in which they are able to observe the child’s behavior. These differences in available information 

are quite likely to influence their judgments about the child’s characteristics. Thus it is important to 

acknowledge that systematic variance between respondents might not necessarily indicate error in 

observation, recollection, or reporting, but rather they could reflect real differences in child behavior in 

different environments and contexts. 

The SACD Statistical Workgroup conducted multiple analyses with “construct” latent variables (e.g., Problem 

Behavior, Altruistic Behavior) and “reporter” latent variables (e.g., primary caregiver). The central finding of 

these MTMR analyses was that for the set of outcomes measured, there was substantial variance attributable 

to both the respondent and the constructs being measured. Thus, for example, the correlations between a 

single respondent’s assessments across constructs—such as a teacher’s perceptions of a child’s Problem 

Behavior and Altruistic Behavior—were similar to the correlations between respondents for the same 

construct—such as the teacher’s assessment of that child’s Problem Behavior and the corresponding 

assessment by the primary caregiver. In addition, the path coefficients for the construct and reporter latent 

variables were of generally similar magnitude to one another. The data used in these analyses do not permit 

determination of whether the latent reporter effects reflected bias on the part of respondents or differences in 

the information available to different respondents. 

Psychometric Properties of School-Level Outcome Measures 

In addition to the individual, student-level outcome measures described above, two school-level measures of 

possible SACD program outcomes were included in the impact analyses. These measures were included in 

separate teacher surveys (Teacher Report on Classroom and School) administered in fall 2004 and each 

follow-up period, and the measures focused on assessments of the school environment. One of the measures, 

Student Support for Teachers, assessed teacher perceptions of student positive and negative behavior and 

teacher-student relations; the other, Feelings of Safety, assessed teacher perceptions of the extent to which 

students were not fearful of being psychologically or physically harmed at school. Principal components 

analyses revealed that both measures were unidimensional, with all items having large positive loadings (≥ 

.69) on the first unrotated component and the component accounting for 62 percent to 77 percent of the 

variance in item responses. Similarly, internal consistency analyses revealed that both measures were highly 

reliable, with alpha coefficients between .89 and .91 at each survey wave for the Student Support for Teachers 

index and between .88 and .90 at each wave for the Feelings of Safety index. 

Selection of Covariates 

This section provides details on the selection of covariates for estimating regression-adjusted intervention 

impacts on the key child and school outcomes. The approach described here significantly benefited from 

discussions with the SACD Statistical Workgroup, composed of principal investigators across the seven 

SACD study sites and representatives of IES and CDC, in addition to staff at MPR and MPR’s subcontractor, 

University of Missouri-St. Louis. This approach was used to choose the covariates for the Year 1 analysis and 
Appendix B: Technical Notes 

B-9 

made use of the Year 1 outcome data. The covariates that were selected using the Year 1 data were also used 

in the Year 2 and Year 3 analyses to ensure comparability of results.6

Objectives in Selecting Covariates 

Because the primary goal of the multiprogram study was to estimate program impacts, criteria were adopted 

for selecting covariates to improve the precision and accuracy of the impact estimates. The goal was not to 

build structural models explaining how the SACD interventions work or to identify mediating variables. 

Therefore, there was no focus on the signs or interpretations of the estimated coefficients on the covariates. 

The random assignment design ensured that the main difference between treatment and control group 

schools at baseline was the opportunity to receive SACD program services. Therefore, simple differences in 

mean outcomes between the treatment and control groups provided unbiased impact estimates of the offer of 

SACD program services on key outcomes. However, regression models were used to estimate the impacts for 

the following two reasons: 

To Increase the Precision of Impact Estimates 

Statistical power was a particularly important issue for the SACD evaluation because of clustering effects due 

to the correlation of student outcomes within schools. This was particularly true at the site level due to the 

small number of schools. The inclusion of baseline covariates in the regression models increased the precision 

of the impact estimates (that is, reduced intraclass correlations) by explaining some of the variation in mean 

outcomes across schools. The inclusion of school-level covariates (which varied only between schools), 

however, also resulted in a loss in degrees of freedom available for statistical tests, which reduced statistical 

power. This precision loss was important at the site level, where there were relatively small numbers of 

schools. 

To Adjust for Baseline Differences Between the Treatment and Control Group Schools 

Although random assignment produced statistically equivalent treatment and control groups, there were some 

residual differences in the average characteristics of the two research groups due to random sampling. In the 

benchmark models, MPR adjusted for these residual differences using regression models. The inclusion of 

these baseline variables, however, affected the impact estimates only if they were also correlated with the 

outcome variables. Thus, as discussed in the following section, a covariate with a treatment-control group 

difference at baseline was selected only if it also had some predictive power in the regression models. 

Tailoring Covariates to Specific Outcomes and Sites 

The options for selecting covariates lie between picking a set that is the same across all outcomes and sites 

and picking a set that is tailored specifically for each outcome within each site. While the former is 

advantageous because of its parsimony, there is the risk of losing precision by including covariates that might 

not be correlated with the particular outcome (due to reductions in degrees of freedom) or by excluding 

covariates with predictive power. Completely tailoring sets for each outcome and site maximizes precision but 

is computationally intensive. Therefore, recognizing that outcomes obtained from a specific reporter tend to 

be somewhat correlated, and balancing the trade-offs between being too parsimonious and too unwieldy, 

covariate sets were chosen that differed by site but were the same within reporter in each site and for the pooled 

analysis. Thus, the same set of covariates was used to estimate impacts on child-reported outcomes in a 

particular site (or for the pooled analysis), but the set was allowed to vary across sites (and similarly for 

outcomes from the Primary Caregiver Report, Teacher Report on Student, and Teacher Report on Classroom 

and School). 

 

6

 As a sensitivity analysis for the use of the covariates selected with the Year 1 outcome data in the analyses of all 3 years 

of data, the covariates were reselected with the same approach but using the Year 2 outcome data. For the combinedprogram analysis, the results of the Year 1 and Year 2 analyses were not sensitive to which set of covariates were 

included. There were several differences in the individual program analyses. 
Appendix B: Technical Notes 

B-10

The one exception to the rule of uniform sets within a reporter was that the baseline measure of the outcome 

(pretest) was always included as a covariate in the regression models. As noted in the following discussion, 

stepwise regressions found that the pretests explained a relatively large proportion of the variance in the 

outcomes. 

Implementation of the Covariate Selection Procedure 

Table B.3 lists the Year 1 (spring 2005) outcome variables for the analysis. The outcomes include 10 measures 

from the Child Report, three measures from the Primary Caregiver Report, five measures from the Teacher 

Report on Student, and two measures from the Teacher Report on Classroom and School. These measures 

were constructed from original scale items using factor analytic methods. 
Appendix B: Technical Notes 

B-11

Table B.3. Child and school outcomes used for covariate selection and the percentage missing 

at spring 2005 (Year 1) analysis 

Outcome

Percent 

missing 

in combinedprogram

sample

Average 

percent 

missing 

across 

programs

Minimum 

missing 

across 

programs

Maximum 

missing 

across

programs

Child-reported

 Positive School Orientation 4.97 4.92 0.64 8.40

 Engagement with Learning 4.57 4.53 0.64 8.25

 Student Afraid at School 5.08 5.04 0.80 8.70

 Negative School Orientation 5.01 4.97 0.80 8.40

 Self-Efficacy for Peer Interactions 5.06 5.03 0.96 8.55

 Normative Beliefs About Aggression 5.22 5.17 0.80 8.55

 Empathy 5.10 5.06 0.80 8.40

 Altruistic Behavior 5.08 5.04 0.80 8.70

 Problem Behavior 5.20 5.14 1.28 8.55

 Victimization at School 5.06 5.03 1.12 8.55

Primary caregiver-reported

 Positive Social Behavior 22.53 22.14 12.96 33.02

 Problem Behavior 22.51 22.12 12.96 32.75

 Altruistic Behavior 23.79 23.43 14.08 34.24

Teacher-reported

 Positive Social Behavior 1.86 1.77 0.15 4.06

 Problem Behavior 1.70 1.61 0.15 4.06

 ADHD-Related Behavior 1.75 1.66 0.15 4.06

 Altruistic Behavior 4.48 4.33 0.87 10.77

 Academic Competence and Motivation 1.68 1.57 0.16 4.60

 Student Support for Teachers 0.59 0.65 0.00 2.91

 Feelings of Safety 0.59 0.57 0.00 1.15

NOTE: ADHD is the abbreviation for attention deficit hyperactivity disorder.

SOURCE: The Social and Character Development (SACD) Research Program.

Table B.4 lists the full set of candidate covariates constructed from the baseline (fall 2004) data at the time of 

the Year 1 (spring 2005) analysis. (Note that because the first set of data was collected after program 

implementation began at many sites, these data are referred to as “initial data” rather than “baseline data” in 

the main body of this report.) The set of potential covariates included three types of baseline variables.7

 First, 

the covariates included child demographic measures and, from the Child Report, pretest scores. Second, the 

covariates included primary caregiver characteristics and child pretest scores from the Primary Caregiver 

Report. Finally, the covariates included child and primary caregiver involvement pretest scores from the 

Teacher Report on Student. 

 

7

 Including covariates to measure the characteristics of teachers was considered but this approach was rejected for the 

analysis of child outcomes because teachers were likely to change over the 3-year follow-up period, and because the 

SACD interventions might affect the types of teachers who entered and exited the study schools. Teacher demographic 

characteristics were included as covariates for the analysis of the two teacher-reported outcomes from the TRCS. 
Appendix B: Technical Notes 

B-12

Table B.4. Potential covariates and the percentage missing at spring 2005 (Year 1) analysis 

Potential covariate

Percent 

missing 

in combinedprogram

sample

Average 

percent 

missing

across 

programs 

Minimum 

missing 

across 

programs

Maximum 

missing 

across 

programs

Child-reported 

 Female 1.08 1.06 0.00 2.25

 White (non-Hispanic) 9.03 8.64 2.56 21.79

 Black (non-Hispanic) 9.03 8.64 2.56 21.79

 Hispanic 9.03 8.64 2.56 21.79

 Other ethnicity 9.03 8.64 2.56 21.79

 Age (in years) 1.08 1.06 0.00 2.25

 Scales

 Student Afraid at School 13.61 13.47 8.64 17.19

 Altruistic Behavior 13.86 13.73 8.64 17.05

 Empathy 13.93 13.80 8.80 17.05

 Engagement with Learning 13.59 13.44 8.32 17.19

 Negative School Orientation 13.70 13.56 8.64 17.19

 Normative Beliefs About Aggression 13.75 13.61 8.64 17.05

 Positive School Orientation 13.49 13.35 8.32 17.19

 Problem Behavior 14.00 13.86 8.80 17.05

 Self-Efficacy for Peer Interactions 13.82 13.69 8.80 17.05

 Victimization at School 14.05 13.90 8.80 17.05

Primary caregiver-reported 

 Age (in years) 10.57 10.04 3.04 27.47

 Completed high school education or equivalent 9.01 8.62 2.56 22.19

 Some college 9.01 8.62 2.56 22.19

 Bachelor’s or higher degree 9.01 8.62 2.56 22.19

 Highest level of education in household—

 Completed high school or equivalent 9.08 8.67 2.56 22.46

 Highest level of education in household—

 Some college 9.08 8.67 2.56 22.46

 Highest level of education in household—

 Bachelor’s or higher degree 9.08 8.67 2.56 22.46

 Mother present in home life 8.80 8.41 2.56 21.92

 Mother and father present 8.83 8.43 2.56 22.06

 Respondent someone other than mother 

 or father 8.92 8.53 2.56 22.19

 Number of people in household 9.01 8.62 2.56 22.06

 Household income: $20,000 to $40,000 11.63 11.14 3.52 27.20

 Household income: $40,000 to $60,000 11.63 11.14 3.52 27.20

 Household income: More than $60,000 11.63 11.14 3.52 27.20

 Income-to-poverty-threshold ratio—

 Below 135 percent 11.61 11.12 3.52 27.33

 Income-to-poverty-threshold ratio—

 Between 135 and 185 percent 11.61 11.12 3.52 27.33

 Income-to-poverty-threshold ratio—

 Above 185 percent 11.61 11.12 3.52 27.33

See notes at end of table.
Appendix B: Technical Notes 

B-13

Table B.4. Potential covariates and the percentage missing at spring 2005 (Year 1) analysis—

Continued

Potential covariate

Percent 

missing 

in combinedprogram

sample

Average 

percent 

missing 

across 

programs 

Minimum 

missing 

across 

programs

Maximum 

missing 

across 

programs

 Full-time employment 9.13 8.73 2.72 22.60

 Part-time employment 9.13 8.73 2.72 22.60

 Parental scales

 APQ–Poor Monitoring and Supervision Subscale 18.94 18.15 8.32 39.78

 APQ–Positive Parenting Subscale 18.97 18.18 8.48 39.51

 Child-Centered Social Control 18.87 18.08 8.32 39.51

 Confusion, Hubbub, and Order 19.24 18.45 8.80 39.78

 Community Resources 19.52 18.70 9.44 40.60

 Community Risk 20.99 20.19 9.60 43.30

 Parent and Teacher Involvement 21.10 20.41 11.36 42.49

 Child scales

 Altruistic Behavior 18.90 18.08 8.48 39.51

 Positive Social Behavior 18.69 17.88 8.32 39.11

 Problem Behavior 19.29 18.52 8.80 39.24

Teacher-reported 

 Child scales 

 Academic Competence and Motivation 11.33 10.95 5.28 19.76

 ADHD-Related Behavior 11.33 10.96 5.12 19.22

 Altruistic Behavior 12.55 12.13 6.24 21.52

 Positive Social Behavior 11.49 11.12 5.60 19.49

 Problem Behavior 11.33 10.96 5.12 19.22

 Parent involvement

 Parent-teacher involvement 20.74 20.24 10.65 30.31

 Teacher characteristics (for school outcomes only) 

 Teacher—Female 0.00 0.00 0.00 0.00

 Teacher—Black (non-Hispanic) 0.24 0.27 0.00 0.97

 Teacher—Hispanic 0.24 0.27 0.00 0.97

 Teacher—Other race 0.24 0.27 0.00 0.97

 Total experience 0.00 0.00 0.00 0.00

 Experience at current school 0.12 0.15 0.00 1.05

 Regular certification 0.00 0.00 0.00 0.00

 Other certification 0.00 0.00 0.00 0.00

 Teacher’s highest degree—Bachelor’s degree 0.00 0.00 0.00 0.00

NOTE: Abbreviations are

 ADHD: Attention deficit hyperactivity disorder

 APQ: Alabama Parenting Questionnaire

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-14

Handling of Missing Values 

Before selecting covariates for Year 1 (spring 2005) impact analyses, the missing values of potential covariates 

and outcomes were considered. Tables B.3 and B.4 list the percentages of observations with missing values 

for each spring 2005 outcome measure and each potential covariate. The analysis to select covariates used a 

base for the child-related outcomes of all children with either a Child Report, a Primary Caregiver Report, or 

a Teacher Report on Student at the first follow-up (and excluded study non-consenters). This universe of 

children who had data on at least one of the three child-level reports (the Child Report, the Primary Caregiver 

Report, or the Teacher Report on Student) included 4,351 observations and varied from 460 to 739 

observations across programs. The base for the school outcomes included 850 third-, fourth-, and fifth-grade 

teachers with Teacher Report on Classroom and School data and varied from 95 to 174 teachers across 

programs. Outcome measures were constructed if about 80 percent or more of pertinent scale items were 

completed.8

 Thus, missing outcomes pertain to two types of cases in the sample universe: (1) survey 

respondents who completed fewer than 80 percent of scale items, and (2) survey non-respondents. 

For the analysis, missing values for covariates were imputed using mean cell imputation procedures, but 

missing values were not imputed for outcome measures. The following steps were used for the covariate 

imputations in the child-level data: 

 For covariates unlikely to vary over time, such as demographic characteristics, data that became 

available during follow-up data collection were used before conducting the cell mean imputations. 

Except for baseline values of the scales, these measures included all child and primary caregiver 

characteristics, such as the race/ethnicity of a child, the educational attainment of the primary 

caregiver, and whether the household income was within a certain range of income. Thus, if fall 2004 

data on a covariate that was unlikely to vary over time were unavailable, spring 2005 data were used 

when available. 

 Imputation cells were defined on the basis of school, gender, and race/ethnicity. Sample members 

were allocated to cells defined by school, gender, and, race/ethnicity (White non-Hispanic, Black 

non-Hispanic, Hispanic, and other). Excluded from the imputation analysis were cases with missing 

values for the variables used to define cells for the cell mean imputations, such as cases for which a 

child’s gender or race/ethnicity was unknown. 

 Imputations for cases with missing covariates in a particular cell were obtained using mean values for 

cases with nonmissing covariate values in the same cell. Where cases with missing covariates also had 

missing values for the data that defined the cells (such as school, race/ethnicity, or sex), the 

imputations were performed within broader cells defined by the nonmissing characteristics. In 

addition, if a cell size was below five, the definition of the cell was sequentially broadened until a cell 

size of at least five was obtained. 

 Race/ethnicity was imputed using school and gender. After imputations on other covariates were 

performed, cases with missing values for race/ethnicity were imputed using mean cell imputations, 

where cells were defined by school and gender. 

For teacher covariates used in estimating impacts on Teacher Report on Classroom and School school-level 

outcomes, a similar procedure was followed:

 For all covariates, the fall 2004 data were used if available. 

 If fall 2004 data were missing, the spring 2005 data were used. 

 

8

 The specific cutoff that was used for the construction of outcome measures depended on the number of scale items. 

For example, a cutoff of 75 percent (3 out of 4) was used when the outcome was based on four scale items. 
Appendix B: Technical Notes 

B-15

 If a value for the covariate was still missing, imputation was based on the mean value for all teachers 

in the school. 

Filling in missing values of baseline data had two closely related benefits. First, the selection of covariates for 

the Year 1 impact analyses could be based on as many records as possible. After missing baseline covariate 

data were filled in through the use of follow-up data or cell mean imputation, the covariates used in the 

benchmark models were selected through a stepwise procedure that used Year 1 outcomes as dependent 

variables (see Stepwise Regression Procedures later in this appendix). Second, and similarly, after covariates 

were selected, the estimation procedure used to generate impact results could be based on as many records as 

possible. In this way, estimating the results was not hampered by missing initial data. 

Filling in missing values for baseline data for the Year 2 and Year 3 impact analyses. After the benchmark model 

covariates were selected and the Year 1 analyses were conducted, additional follow-up data became available 

for the 2005-06 and 2006-07 school years. Data from these follow-up periods were used to fill in missing 

baseline covariate data for new students who entered the study schools after the spring 2005 data collection 

and for students who had been in the study schools from the start of the study (thus avoiding the need to use 

cell mean imputation). Tables B.5 and B.6 show the percentages of observations with missing values for each 

spring 2006 (Year 2) outcome measure and each potential covariate, and tables B.7 and B.8 show the 

percentages of observations with missing values for each spring 2007 (Year 3) outcome measure and each 

potential covariate.

The procedure to fill in missing data for the covariates used in the Year 2 (spring 2006) and the Year 3 (spring 

2007) regression analyses was similar to that used for the Year 1 (spring 2005) analyses. If fall 2004 initial data 

were available, then those data were used. If not, then spring 2005 data on covariates, such as the 

race/ethnicity of the child and the educational attainment of the primary caregiver, were used if possible (that 

is, if the spring 2005 data item was available and if the data item was unlikely to vary over time). If fall 2004 

data and spring 2005 data were unavailable, then fall 2005 data were used to fill in the baseline data. Next, 

spring 2006 data were used. Finally, spring 2007 data were used. If the information was unavailable from all 

these data collection efforts, or if the data item was likely to vary over time (as is the case with pretest data), 

the cell mean imputation process was used to fill in the missing data. 
Appendix B: Technical Notes 

B-16

Table B.5. Child and school outcomes used for covariate selection and the percentage missing 

at spring 2006 (Year 2) analysis 

Outcome

Percent

missing 

in combinedprogram 

sample

Average 

percent

missing 

across 

programs

Minimum 

missing 

across 

programs

Maximum 

missing 

across 

programs

Child-reported

 Positive School Orientation 5.48 5.43 1.17 9.22

 Engagement with Learning 5.10 5.06 0.73 8.32

 Student Afraid at School 5.50 5.43 1.03 8.15

 Negative School Orientation 5.50 5.44 0.73 9.04

 Self-Efficacy for Peer Interactions 5.64 5.59 1.17 9.04

 Normative Beliefs About Aggression 5.85 5.77 1.32 9.38

 Empathy 5.74 5.66 0.88 9.65

 Altruistic Behavior 6.04 5.95 1.32 9.65

 Problem Behavior 6.02 5.93 1.32 9.51

 Victimization at School 6.09 6.00 1.47 9.65

Primary caregiver-reported

 Positive Social Behavior 25.47 25.24 13.49 38.18

 Problem Behavior 25.47 25.24 13.49 38.18

 Altruistic Behavior 26.43 26.20 15.69 39.27

Teacher-reported

 Positive Social Behavior 0.87 0.82 0.00 1.63

 Problem Behavior 0.89 0.85 0.00 1.63

 ADHD-Related Behavior 0.84 0.80 0.00 1.63

 Altruistic Behavior 2.29 2.18 0.16 4.51

 Academic Competence and Motivation 1.03 0.95 0.00 2.17

 Student Support for Teachers 0.36 0.33 0.00 0.89

 Feelings of Safety 0.36 0.37 0.00 0.89

NOTE: ADHD is the abbreviation for attention deficit hyperactivity disorder.

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-17

Table B.6. Potential covariates and the percentage missing at spring 2006 (Year 2) analysis 

Potential covariate

Percent 

missing 

in combinedprogram 

sample

Average 

percent 

missing 

across 

programs 

Minimum 

missing 

across 

programs

Maximum 

missing 

across 

programs

Child-reported 

 Female 0.47 0.48 0.00 0.90

 White (non-Hispanic) 6.53 6.26 2.20 15.35

 Black (non-Hispanic) 6.53 6.26 2.20 15.35

 Hispanic 6.53 6.26 2.20 15.35

 Other ethnicity 6.53 6.26 2.20 15.35

 Age (in years) 0.23 0.24 0.00 0.54

 Scales

 Student Afraid at School 27.57 27.54 23.61 34.36

 Altruistic Behavior 27.76 27.74 23.90 34.72

 Empathy 27.79 27.76 24.05 34.72

 Engagement with Learning 27.55 27.53 23.31 34.54

 Negative School Orientation 27.62 27.59 23.76 34.36

 Normative Beliefs About Aggression 27.65 27.62 23.90 34.36

 Positive School Orientation 27.48 27.46 23.46 34.36

 Problem Behavior 27.81 27.78 24.05 34.72

 Self-Efficacy for Peer Interactions 27.69 27.66 24.05 34.36

 Victimization at School 27.88 27.85 24.05 34.90

Primary caregiver-reported 

 Age (in years) 6.84 6.53 2.64 16.30

 Completed high school education or equivalent 6.46 6.18 2.05 15.49

 Some college 6.46 6.18 2.05 15.49

 Bachelor’s or higher degree 6.46 6.18 2.05 15.49

 Highest level of education in household—

 Completed high school or equivalent 6.44 6.17 2.05 15.35

 Highest level of education in household—

 Some college 6.44 6.17 2.05 15.35

 Highest level of education in household—

 Bachelor’s or higher degree 6.44 6.17 2.05 15.35

 Mother present in home life 6.44 6.16 2.05 15.49

 Mother and father present 6.44 6.16 2.05 15.49

 Respondent someone other than mother 

or father 6.48 6.20 2.05 15.76

 Number of people in household 6.55 6.29 2.35 15.49

 Household income: $20,000 to $40,000 8.12 7.77 2.79 19.43

 Household income: $40,000 to $60,000 8.12 7.77 2.79 19.43

 Household income: More than $60,000 8.12 7.77 2.79 19.43

 Income-to-poverty-threshold ratio—

 Below 135 percent 8.22 7.86 3.08 19.57

 Income-to-poverty-threshold ratio—

 Between 135 and 185 percent 8.22 7.86 3.08 19.57

See notes at end of table.
Appendix B: Technical Notes 

B-18

Table B.6. Potential covariates and the percentage missing at spring 2006 (Year 2) analysis— 

Continued

Potential covariate

Percent 

missing 

in combinedprogram

sample

Average 

percent 

missing 

across 

programs 

Minimum 

missing 

across 

programs

Maximum 

missing 

across 

programs

 Income-to-poverty-threshold ratio—

 Above 185 percent 8.22 7.86 3.08 19.57

 Full-time employment 6.55 6.27 2.05 15.76

 Part-time employment 6.55 6.27 2.05 15.76

 Parental scales

 APQ—Poor Monitoring and Supervision 

 Subscale 31.60 30.99 23.02 50.68

 APQ—Positive Parenting Subscale 31.65 31.04 23.17 50.41

 Child-Centered Social Control 31.44 30.85 22.73 50.14

 Confusion, Hubbub, and Order 31.84 31.24 23.46 50.27

 Community Resources 31.93 31.31 24.00 50.68

 Community Risk 32.89 32.27 23.90 52.58

 Parent and teacher Involvement 33.47 32.92 25.37 52.85

 Child scales

 Altruistic Behavior 31.53 30.91 23.17 50.54

 Positive Social Behavior 31.32 30.70 23.02 50.00

 Problem Behavior 31.91 31.30 23.61 50.41

Teacher-reported 

 Child scales 

 Academic Competence and Motivation 25.63 25.39 19.33 34.54

 ADHD-Related Behavior 25.61 25.38 19.11 34.72

 Altruistic Behavior 26.66 26.40 19.11 37.43

 Positive Social Behavior 25.77 25.53 19.11 34.72

 Problem Behavior 25.61 25.38 19.11 34.72

 Parent involvement 

 Parent and teacher involvement 33.47 33.06 22.44 43.34

 Teacher characteristics (for school 

 outcomes only) 

 Teacher—Female 0.00 0.00 0.00 0.00

 Teacher—Black (non-Hispanic) 0.48 0.41 0.00 1.16

 Teacher—Hispanic 0.48 0.41 0.00 1.16

 Teacher—Other race 0.48 0.41 0.00 1.16

 Total experience 0.24 0.30 0.00 1.22

 Experience at current school 0.24 0.30 0.00 1.22

 Regular certification 0.00 0.00 0.00 0.00

 Other certification 0.00 0.00 0.00 0.00

 Teacher’s highest degree—

 Bachelor’s degree 0.00 0.00 0.00 0.00

NOTE: Abbreviations are

 ADHD: Attention deficit hyperactivity disorder

 APQ: Alabama Parenting Questionnaire

SOURCE: The Social and Character Development (SACD) Research Program. 
Appendix B: Technical Notes 

B-19

Table B.7. Child and school outcomes used for covariate selection and the percentage 

missing at spring 2007 (Year 3) analysis 

Outcome

Percent 

missing 

in combinedprogram 

sample

Average 

percent 

missing 

across 

programs

Minimum 

missing 

across 

programs

Maximum 

missing 

across 

programs

Child-reported

 Positive School Orientation 4.33 4.34 0.44 9.79

 Engagement with Learning 4.07 4.06 0.74 9.79

 Student Afraid at School 4.43 4.44 0.30 9.95

 Negative School Orientation 4.33 4.35 0.74 9.95

 Self-Efficacy for Peer Interactions 4.41 4.42 0.59 9.79

 Normative Beliefs About Aggression 4.36 4.35 0.74 9.95

 Empathy 4.41 4.41 0.44 9.79

 Altruistic Behavior 4.53 4.53 0.74 9.79

 Problem Behavior 4.62 4.63 0.44 9.79

 Victimization at School 4.65 4.64 0.89 9.95

Primary caregiver-reported

 Positive Social Behavior 31.28 31.04 16.89 48.10

 Problem Behavior 31.25 31.02 16.89 47.96

 Altruistic Behavior 32.22 31.98 18.07 49.56

Teacher-reported

 Positive Social Behavior 2.00 1.99 0.58 6.20

 Problem Behavior 1.90 1.89 0.44 6.04

 ADHD-Related Behavior 1.97 1.96 0.58 6.36

 Altruistic Behavior 3.83 4.04 0.73 11.95

 Academic Competence and Motivation 1.88 1.84 0.15 5.88

 Student Support for Teachers 0.61 0.63 0.00 1.90

 Feelings of Safety 0.85 0.83 0.00 2.40

NOTE: ADHD is the abbreviation for attention deficit hyperactivity disorder.

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-20

Table B.8. Potential covariates and the percentage missing at spring 2007 (Year 3) analysis 

Potential covariate

Percent 

missing 

in combinedprogram 

sample

Average 

percent 

missing 

across 

programs 

Minimum 

missing 

across 

programs

Maximum 

missing 

across 

programs

Child-reported 

 Female 0.60 0.61 0.15 1.47

 White (non-Hispanic) 7.44 7.32 1.93 17.49

 Black (non-Hispanic) 7.44 7.32 1.93 17.49

 Hispanic 7.44 7.32 1.93 17.49

 Other ethnicity 7.44 7.32 1.93 17.49

 Age (in years) 12.91 13.14 6.81 19.85

 Scales

 Student Afraid at School 34.48 34.68 27.41 43.55

 Altruistic Behavior 34.67 34.88 27.26 43.74

 Empathy 34.70 34.90 27.41 43.74

 Engagement with Learning 34.43 34.64 27.11 43.74

 Negative School Orientation 34.53 34.72 27.41 43.55

 Normative Beliefs About Aggression 34.58 34.78 27.26 43.55

 Positive School Orientation 34.41 34.61 27.11 43.55

 Problem Behavior 34.70 34.90 27.41 43.74

 Self-Efficacy for Peer Interactions 34.63 34.83 27.41 43.55

 Victimization at School 34.79 34.99 27.41 43.93

Primary caregiver-reported 

 Age (in years) 7.95 7.82 1.93 18.51

 Completed high school education or equivalent 7.39 7.27 1.93 17.64

 Some college 7.39 7.27 1.93 17.64

 Bachelor’s or higher degree 7.39 7.27 1.93 17.64

 Highest level of education in household—

 Completed high school or equivalent 7.44 7.31 2.07 17.49

 Highest level of education in household—

 Some college 7.44 7.31 2.07 17.49

 Highest level of education in household—

 Bachelor’s or higher degree 7.44 7.31 2.07 17.49

 Mother present in home life 7.37 7.24 1.93 17.64

 Mother and father present 7.37 7.24 1.93 17.64

 Respondent someone other than mother

 or father 7.44 7.31 1.93 17.64

 Number of people in household 7.56 7.45 2.07 17.49

 Household income: $20,000 to $40,000 9.03 8.87 2.67 21.28

 Household income: $40,000 to $60,000 9.03 8.87 2.67 21.28

 Household income: More than $60,000 9.03 8.87 2.67 21.28

 Income-to-poverty-threshold ratio—

 Below 135 percent 9.15 9.00 2.67 21.28

 Income-to-poverty-threshold ratio—

 Between 135 and 185 percent 9.15 9.00 2.67 21.28

 Income-to-poverty-threshold ratio—

 Above 185 percent 9.15 9.00 2.67 21.28

 Full-time employment 7.49 7.36 1.93 17.78

 Part-time employment 7.49 7.36 1.93 17.78

See notes at end of table.
Appendix B: Technical Notes 

B-21

Table B.8. Potential covariates and the percentage missing at spring 2007 (Year 3) analysis—

Continued

Potential covariate

Percent 

missing 

in combinedprogram 

sample

Average 

percent 

missing 

across 

programs 

Minimum 

missing 

across 

programs

Maximum 

missing across 

programs

 Parental scales

 APQ—Poor Monitoring and Supervision 

 Subscale 38.02 37.73 26.67 55.69

 APQ—Positive Parenting Subscale 38.07 37.79 26.81 55.54

 Child-Centered Social Control 37.92 37.65 26.37 55.25

 Confusion, Hubbub, and Order 38.29 38.01 27.11 55.25

 Community Resources 38.43 38.13 27.56 55.83

 Community Risk 39.25 38.94 27.41 57.73

 Parent and Teacher Involvement 39.66 39.45 28.59 57.43

 Child scales

 Altruistic Behavior 37.92 37.62 26.81 55.69

 Positive Social Behavior 37.76 37.46 26.67 55.10

 Problem Behavior 38.31 38.03 27.26 55.25

Teacher-reported 

 Child scales 

 Academic Competence and Motivation 32.65 32.67 24.89 42.77

 ADHD-Related Behavior 32.65 32.69 24.89 43.16

 Altruistic Behavior 33.57 33.58 25.93 45.47

 Positive Social Behavior 32.80 32.82 25.33 43.16

 Problem Behavior 32.65 32.69 24.89 43.16

 Parent involvement 

 Parent and teacher involvement 39.73 39.51 29.25 49.52

 Teacher characteristics (for school 

 outcomes only) 

 Teacher—Female 0.24 0.23 0.00 1.60

 Teacher—Black (non-Hispanic) 0.49 0.45 0.00 1.60

 Teacher—Hispanic 0.49 0.45 0.00 1.60

 Teacher—Other race 0.49 0.45 0.00 1.60

 Total experience 0.37 0.32 0.00 1.60

 Experience at current school 0.49 0.45 0.00 1.60

 Regular certification 0.00 0.00 0.00 0.00

 Other certification 0.00 0.00 0.00 0.00

 Teacher’s highest degree—

 Bachelor’s degree 0.24 0.23 0.00 1.60

NOTE: Abbreviations are

 ADHD: Attention deficit hyperactivity disorder

 APQ: Alabama Parenting Questionnaire

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-22

Stepwise Regression Procedures

For the selection of covariates to be used in the benchmark impact models, stepwise regression methods were 

used to identify potential covariates with significant explanatory power.9

 The treatment binary variable 

(measuring estimated impacts); program-specific binary variables (for the pooled models); and the age, 

gender, and race/ethnicity of the child were included (these covariates were separately tested as to whether 

they would be selected under the implementation rules). At each step of the stepwise procedure, the variable 

with the smallest p-value below a preset threshold level was included in the model while variables already 

selected were evaluated to see if any could be removed; the variable with a p-value greater than the critical pvalue of 0.32 and whose removal would least lower the adjusted R2

 value was removed. The critical p-value 

was set at 0.32 to correspond to a t-statistic of 1, which is the smallest value of the t-statistic at which the 

addition of a variable in a model increases the adjusted R2 value.10

 

Rules for Selecting Covariates Using Year 1 Data 

The rules for selecting covariates had two key features: (1) they identified covariates that had a treatmentcontrol group difference at baseline and that showed some explanatory power in the regression models, and 

(2) they identified covariates that had no treatment-control group difference at baseline and that had 

substantial explanatory power in the regression models for a majority of outcomes. Specifically, based on 

regressions using spring 2005 (Year 1) outcomes, a unique set of covariates by reporter and program were 

selected using sequential rules. First, a variable was included that had a statistically significant treatmentcontrol group baseline difference in the variable, and that was selected in the final stepwise model for at least 

one of the outcomes for a given reporter. Second, a variable with no treatment-control group difference at 

baseline was included if it was selected by the stepwise regressions for about 60 percent or more of the 

outcomes. Third, covariates were included to increase face validity. That is, variables that measured age, 

gender, race, and ethnicity of the child (or of the teacher for Teacher Report on Classroom and School 

outcomes) were included if they had not already been selected. Finally, for completeness, covariates were 

selected if they completed a set of categorical variables if one of the categories had been previously selected. 

Tables B.9 through B.12 list the sets of covariates selected as a result of these rules. The covariates are listed 

separately for the combined-program analysis and the program-specific analyses. Table B.9 provides details 

for all child-reported outcomes, table B.10 provides details for all primary caregiver-reported outcomes, table 

B.11 does the same for all teacher-reported child outcomes, and table B.12 does so for all teacher-reported 

school outcomes. 

Key features of the selected covariates are as follows: 

 While baseline values of the scale under consideration were always selected as covariates, many scales 

were also selected across outcomes and reporters. For example, 7 of the 10 child-reported scales and 

2 of the 6 teacher-reported scales were selected as covariates for child-reported outcomes in the 

combined-program sample (table B.9). Similarly, for primary caregiver-reported outcomes, 5 of the 

10 child-reported scales, 2 of the 3 primary caregiver-reported scales, and 3 of the 6 teacher-reported 

scales were selected as covariates in the combined-program sample (table B.10). Finally, at the 

combined-program level, 3 of the 10 child-reported scales, all 3 primary caregiver-reported scales, 

and 4 of the 6 teacher-reported scales were selected as covariates for the teacher-reported child-level 

outcomes (table B.11). 

 

9

 The PROC REG procedure in SAS was used with the SELECTION option set to STEPWISE. 

10 The stepwise procedure in SAS did not adjust for clustering effects. Thus, after selecting covariates from the stepwise 

procedure, these covariates were used to re-estimate the models in SAS PROC MIXED (which accounts for school-level 

clustering effects) and the final covariate sets were refined accordingly. Because of small clustering effects, however, 

these refinements were minor. 
Appendix B: Technical Notes 

B-23

 Most demographic and socioeconomic characteristics of the primary caregiver were selected as 

covariates for primary caregiver-reported outcomes. However, these measures were selected less 

consistently for the child- and teacher-reported outcomes. 

 Considerable variation existed in the selected covariates across programs. This result likely reflected 

differences in study populations across programs. 

 The baseline value of the outcome under consideration typically explained the highest proportion of 

variance in the outcome among all covariates. Using spring 2005 data, tables B.13 and B.14 compare 

the adjusted R2 values of the models that included only the corresponding baseline pretest value with 

models that included the final selected set of covariates. Results are reported for models run at the 

child and school levels. The tables indicate that R2 values did not increase substantially when 

covariates other than the pretests were included in the models. For Year 2 (spring 2006 data), tables 

B.15 and B.16 show similar information and results, and tables B.17 and B.18 show this information 

for Year 3 (spring 2007 data). 
Appendix B: Technical Notes 

B-24

Table B.9. Covariates selected for child-reported outcomes, for the combined-program sample 

and program-specific samples 

Potential covariate

Combinedprogram

sample ABC PA SS 4Rs CSP PATHS LBW

 Total number of covariates 26 11 20 11 10 17 18 6

Child-reported 

 Female        

 Black (non-Hispanic)        

 Hispanic        

 Other ethnicity        

 Age (in years)        

 Scales

 Student Afraid at School

 Altruistic Behavior    

 Empathy 

 Engagement with Learning  

 Negative School Orientation   

 Normative Beliefs About Aggression  

 Positive School Orientation  

 Problem Behavior 

 Self-Efficacy for Peer Interactions  

 Victimization at School   

Primary caregiver-reported 

 Age (in years) 

 Completed high school 

 education or equivalent    

 Some college    

 Bachelor’s or higher degree    

 Highest level of education in household—

 Completed high school or equivalent    

 Highest level of education in household—

 Some college    

 Highest level of education in household—

 Bachelor’s or higher degree    v

 Mother present in home life

 Mother and father present 

 Respondent someone other than mother

 or father 

 Number of people in household   

 Household income: $20,000 to $40,000  

 Household income: $40,000 to $60,000  

 Household income: More than $60,000  

See notes at end of table.
Appendix B: Technical Notes 

B-25

Table B.9. Covariates selected for child-reported outcomes, for the combined-program sample 

 and program-specific samples—Continued

Potential covariate

Combinedprogram

sample ABC PA SS 4Rs CSP PATHS LBW

 Income-to-poverty-threshold ratio—

 Below 135 percent 

 Income-to-poverty-threshold ratio—

 Between 135 and 185 percent 

 Full-time employment   

 Part-time employment   

 Parental scales

 APQ—Poor Monitoring and Supervision 

 Subscale  

 APQ—Positive Parenting Subscale

 Child-Centered Social Control

 Confusion, Hubbub, and Order 

 Community Resources

 Community Risk 

 Parent and Teacher Involvement

 Child scales

 Altruistic Behavior  

 Positive Social Behavior  

 Problem Behavior 

Teacher-reported 

 Child scales

 Academic Competence and Motivation 

 ADHD-Related Behavior   

 Altruistic Behavior

 Positive Social Behavior 

 Problem Behavior

 Parent and Teacher Involvement 

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

 ADHD: Attention deficit hyperactivity disorder

 APQ: Alabama Parenting Questionnaire

 : Covariate used

 Blank cell: Covariate not used

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-26

Table B.10. Covariates selected for primary caregiver-reported child outcomes, for the 

combined-program and program-specific samples 

Potential covariate

Combinedprogram

sample ABC PA SS 4Rs CSP PATHS LBW

 Total number 35 13 26 28 26 24 27 25

Child-reported

 Female        

 Black (non-Hispanic)        

 Hispanic        

 Other ethnicity        

 Age (in years)        

 Scales

 Student Afraid at School  

 Altruistic Behavior 

 Empathy   

 Engagement with Learning  

 Negative School Orientation      

 Normative Beliefs About Aggression    

 Positive School Orientation  

 Problem Behavior   

 Self-Efficacy for Peer Interactions     

 Victimization at School  

Primary caregiver-reported 

 Age (in years)    

 Completed high school education or equivalent     

 Some college     

 Bachelor’s or higher degree     

 Highest level of education in household—

 Completed high school or equivalent     

 Highest level of education in household—

 Some college     

 Highest level of education in household—

 Bachelor’s or higher degree     

 Mother present in home life    

 Mother and father present     

 Respondent someone other than mother

 or father     

 Number of people in household    

 Household income: $20,000 to $40,000      

 Household income: $40,000 to $60,000      

 Household income: More than $60,000      

 Income-to-poverty-threshold ratio—

 Below 135 percent     

 Income-to-poverty-threshold ratio—

 Between 135 and 185 percent     

 Full-time employment     

 Part-time employment     

See notes at end of table.
Appendix B: Technical Notes 

B-27

Table B.10. Covariates selected for primary caregiver-reported child outcomes, for the 

 combined-program and program-specific samples—Continued

Potential covariate

Combinedprogram

sample ABC PA SS 4Rs CSP PATHS LBW

 Parental scales

 APQ—Poor Monitoring and Supervision 

 Subscale     

 APQ—Positive Parenting Subscale   

 Child-Centered Social Control 

 Confusion, Hubbub, and Order      

 Community Resources 

 Community Risk    

 Parent and Teacher Involvement 

 Child scales 

 Altruistic Behavior 

 Positive Social Behavior   

 Problem Behavior    

Teacher-reported 

 Child scales 

 Academic Competence and Motivation  

 ADHD-Related Behavior   

 Altruistic Behavior     

 Positive Social Behavior 

 Problem Behavior   

 Parent and Teacher Involvement 

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

 ADHD: Attention deficit hyperactivity disorder

 APQ: Alabama Parenting Questionnaire

 : Covariate used

 Blank cell: Covariate not used

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-28

Table B.11. Covariates selected for teacher-reported child outcomes, for the combined-program 

and program-specific samples 

Potential covariate

Combinedprogram

sample ABC PA SS 4Rs CSP PATHS LBW

 Total number of covariates 28 24 22 21 20 25 27 21

Child-reported 

 Female        

 Black (non-Hispanic)        

 Hispanic        

 Other ethnicity        

 Age (in years)        

 Scales 

 Student Afraid at School   

 Altruistic Behavior    

 Empathy   

 Engagement with Learning    

 Negative School Orientation   

 Normative Beliefs About Aggression   

 Positive School Orientation   

 Problem Behavior  

 Self-Efficacy for Peer Interactions   

 Victimization at School   

Primary caregiver-reported 

 Age (in years)    

 Completed high school education or equivalent     

 Some college     

 Bachelor’s or higher degree     

 Highest level of education in household—

 Completed high school or equivalent    

 Highest level of education in household—

 Some college    

 Highest level of education in household—

 Bachelor’s or higher degree    

 Mother present in home life     

 Mother and father present    

 Respondent someone other than mother 

 or father

   

 Number of people in household    

 Household income: $20,000 to $40,000   

 Household income: $40,000 to $60,000   

 Household income: More than $60,000   

 Income-to-poverty-threshold ratio—

 Below 135 percent  

 Income-to-poverty-threshold ratio—

 Between 135 and 185 percent  

 Full-time employment      

 Part-time employment      

See notes at end of table.
Appendix B: Technical Notes 

B-29

Table B.11. Covariates selected for teacher-reported child outcomes, for the combined- 

 program and program-specific samples—Continued

Potential covariate

Combinedprogram

sample ABC PA SS 4Rs CSP PATHS LBW

 Parental scales

 APQ—Poor Monitoring and Supervision 

 Subscale   

 APQ—Positive Parenting Subscale  

 Child-Centered Social Control 

 Confusion, Hubbub, and Order   

 Community Resources 

 Community Risk  

 Parent and Teacher Involvement  

 Child scales 

 Altruistic Behavior   

 Positive Social Behavior      

 Problem Behavior  

Teacher-reported 

 Child scales

 Academic Competence and Motivation  

 ADHD-Related Behavior     

 Altruistic Behavior   

 Positive Social Behavior   

 Problem Behavior   

 Parent and Teacher Involvement   

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

 ADHD: Attention deficit hyperactivity disorder

 APQ: Alabama Parenting Questionnaire

 : Covariate used

 Blank cell: Covariate not used

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-30

Table B.12. Covariates selected for teacher-reported school outcomes, for the combinedprogram and program-specific samples 

Potential covariate

Combinedprogram

sample ABC PA SS 4Rs CSP PATHS LBW

 Total number 8 8 7 8 6 9 5 5

Teacher-reported 

 Female        

 Black (non-Hispanic)        

 Hispanic        

 Other ethnicity        

 Total teaching experience       

 Total experience in current school  

 Regular certificate     

 Other certificate     

 Highest degree—Bachelor’s     

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program

 CSP: Competence Support Program

 LBW: Love In a Big World

 PA: Positive Action

 PATHS: Promoting Alternative Thinking Strategies

 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step

 : Covariate used

 Blank cell: Covariate not used

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-31

Table B.13. Adjusted R2 values from models using spring 2005 (Year 1) outcomes, with pretest 

only and with the full covariate set, for the combined-program sample 

Child level Collapsed to school level

Outcome Pretest only

Full 

covariate 

set Pretest only

Full 

covariate 

set

Child-reported 

 Empathy 0.19 0.24 0.34 0.56

 Student Afraid at School 0.13 0.18 0.64 0.76

 Altruistic Behavior 0.20 0.23 0.60 0.67

 Engagement with Learning 0.09 0.13 0.33 0.49

 Negative School Orientation 0.20 0.26 0.83 0.88

 Problem Behavior 0.26 0.31 0.71 0.82

 Self-Efficacy for Peer Interactions 0.20 0.21 0.22 0.54

 Victimization at School 0.20 0.21 0.47 0.60

 Normative Beliefs About Aggression 0.11 0.15 0.39 0.67

 Positive School Orientation 0.18 0.24 0.71 0.83

Primary caregiver-reported 

 Altruistic Behavior 0.22 0.23 0.53 0.65

 Positive Social Behavior 0.43 0.46 0.74 0.83

 Problem Behavior 0.37 0.40 0.51 0.49

Teacher-reported (child)

 ADHD-Related Behavior 0.52 0.54 0.64 0.71

 Academic Competence and Motivation 0.60 0.62 0.85 0.88

 Altruistic Behavior 0.27 0.28 0.32 0.28

 Positive Social Behavior 0.55 0.58 0.81 0.87

 Problem Behavior 0.51 0.54 0.78 0.84

Teacher-reported (school)

 Student Support for Teachers 0.62 0.63 0.91 0.91

 Feelings of Safety 0.47 0.48 0.78 0.80

NOTE: ADHD is the abbreviation for attention deficit hyperactivity disorder. 

SOURCE: The Social and Character Development (SACD) Research Program.
Table B.14. Adjusted R2

 values from models using spring 2005 (Year 1) outcomes with pretest only and with the full covariate set, by 

site 

LBW CSP SS ABC PATHS PA 4Rs

Outcome

Pretest

only

Full 

set

Pretest

only

Full 

set

Pretest

only

Full 

set

Pretest

only

Full 

set

Pretest

only

Full 

set

Pretest

only

Full 

set

Pretest

only

Full 

set

Child-reported 

 Empathy 0.21 0.24 0.19 0.27 0.25 0.26 0.22 0.24 0.19 0.26 0.11 0.18 0.13 0.17

 Student Afraid at School 0.05 0.08 0.10 0.15 0.20 0.21 0.09 0.13 0.15 0.20 0.07 0.11 0.09 0.13

 Altruistic Behavior 0.17 0.18 0.20 0.22 0.24 0.28 0.30 0.32 0.22 0.25 0.10 0.13 0.13 0.14

 Engagement with Learning 0.13 0.16 0.03 0.04 0.11 0.15 0.09 0.15 0.11 0.14 0.06 0.10 0.12 0.14

 Negative School Orientation 0.19 0.20 0.14 0.16 0.18 0.21 0.15 0.17 0.20 0.32 0.08 0.15 0.14 0.20

 Problem Behavior 0.23 0.26 0.18 0.26 0.28 0.30 0.29 0.33 0.28 0.35 0.10 0.17 0.27 0.32

 Self-Efficacy for Peer Interactions 0.19 0.20 0.14 0.14 0.25 0.27 0.23 0.23 0.25 0.26 0.16 0.21 0.17 0.17

 Victimization at School 0.18 0.20 0.17 0.18 0.21 0.22 0.19 0.21 0.27 0.27 0.16 0.18 0.13 0.13

 Normative Beliefs About Aggression 0.06 0.06 0.07 0.15 0.02 0.03 0.14 0.19 0.12 0.17 0.05 0.11 0.19 0.20

 Positive School Orientation 0.13 0.12 0.11 0.17 0.20 0.20 0.23 0.27 0.13 0.26 0.07 0.16 0.20 0.27

Primary caregiver-reported 

 Altruistic Behavior 0.14 0.20 0.18 0.22 0.23 0.34 0.19 0.24 0.26 0.32 0.19 0.22 0.20 0.25

 Positive Social Behavior 0.35 0.39 0.42 0.46 0.47 0.50 0.54 0.56 0.50 0.54 0.40 0.46 0.32 0.36

 Problem Behavior 0.33 0.40 0.38 0.43 0.50 0.53 0.37 0.40 0.43 0.51 0.40 0.44 0.22 0.26

Teacher-reported (child)

 ADHD-Related Behavior 0.48 0.54 0.52 0.53 0.55 0.57 0.51 0.54 0.57 0.59 0.46 0.54 0.50 0.54

 Academic Competence and 

 Motivation 0.53 0.56 0.61 0.63 0.70 0.72 0.51 0.55 0.69 0.72 0.51 0.56 0.54 0.57

 Altruistic Behavior 0.17 0.19 0.34 0.36 0.28 0.34 0.28 0.33 0.36 0.38 0.15 0.20 0.31 0.35

 Positive Social Behavior 0.51 0.56 0.42 0.47 0.55 0.59 0.51 0.56 0.58 0.64 0.54 0.60 0.52 0.54

 Problem Behavior 0.38 0.42 0.47 0.50 0.47 0.51 0.45 0.52 0.67 0.70 0.50 0.56 0.50 0.52

Teacher-reported (school)

 Student Support for Teachers 0.42 0.43 0.43 0.44 0.31 0.36 0.64 0.63 0.70 0.70 0.61 0.62 0.50 0.52

 Feelings of Safety 0.26 0.29 0.22 0.30 0.32 0.36 0.52 0.52 0.48 0.49 0.38 0.36 0.53 0.53

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program PA: Positive Action

 CSP: Competence Support Program PATHS: Promoting Alternative Thinking Strategies

 LBW: Love In a Big World 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step ADHD: Attention deficit hyperactivity disorder

SOURCE: The Social and Character Development (SACD) Research Program.

B-32

Appendix B: Technical Notes
Appendix B: Technical Notes 

B-33

Table B.15. Adjusted R2 values from models using spring 2006 (Year 2) outcomes, with pretest 

only and with the full covariate set, for the combined-program sample 

Child level Collapsed to school level

Outcome Pretest only

Full 

covariate 

set Pretest only

Full 

covariate 

set

Child-reported 

 Empathy 0.11 0.16 0.32 0.48

 Student Afraid at School 0.07 0.12 0.51 0.65

 Altruistic Behavior 0.09 0.11 0.34 0.42

 Engagement with Learning 0.03 0.07 0.25 0.30

 Negative School Orientation 0.13 0.19 0.71 0.72

 Problem Behavior 0.16 0.24 0.62 0.77

 Self-Efficacy for Peer Interactions 0.10 0.13 0.36 0.58

 Victimization at School 0.08 0.10 0.16 0.18

 Normative Beliefs About Aggression 0.08 0.14 0.30 0.57

 Positive School Orientation 0.13 0.18 0.63 0.69

Primary caregiver-reported 

 Altruistic Behavior 0.19 0.21 0.43 0.67

 Positive Social Behavior 0.32 0.37 0.42 0.71

 Problem Behavior 0.27 0.30 0.37 0.52

Teacher-reported (child)

 ADHD-Related Behavior 0.23 0.29 0.31 0.46

 Academic Competence and Motivation 0.37 0.42 0.62 0.78

 Altruistic Behavior 0.06 0.09 0.09 0.01

 Positive Social Behavior 0.22 0.32 0.35 0.64

 Problem Behavior 0.24 0.31 0.62 0.76

Teacher-reported (school)

 Student Support for Teachers 0.49 0.49 0.85 0.85

 Feelings of Safety 0.37 0.38 0.68 0.69

NOTE: ADHD is the abbreviation for attention deficit hyperactivity disorder. 

SOURCE: The Social and Character Development (SACD) Research Program.
Table B.16. Adjusted R2

 values from models using spring 2006 (Year 2) outcomes with pretest only and with the full covariate set 

selected for each program, by program 

ABC PA SS 4Rs CSP PATHS LBW

Outcome

Pretest

only

Full 

set

Pretest

only

Full 

set

Pretest

only

Full 

set

Pretest

only

Full 

set

Pretest

only

Full 

set

Pretest

only

Full 

set

Pretest

only

Full 

set

Child-reported 

 Empathy 0.12 0.17 0.04 0.12 0.09 0.17 0.07 0.12 0.06 0.12 0.12 0.22 0.08 0.10

 Student Afraid at School 0.07 0.12 0.01 0.04 0.03 0.08 0.03 0.04 0.02 0.06 0.08 0.18 0.01 0.04

 Altruistic Behavior 0.10 0.13 0.06 0.08 0.06 0.11 0.05 0.06 0.09 0.14 0.11 0.12 0.05 0.07

 Engagement with Learning 0.03 0.04 0.02 0.04 0.04 0.05 0.03 0.09 0.01 0.05 0.05 0.10 0.03 0.05

 Negative School Orientation 0.04 0.07 0.04 0.10 0.08 0.12 0.03 0.09 0.03 0.06 0.20 0.29 0.08 0.09

 Problem Behavior 0.13 0.20 0.04 0.16 0.10 0.16 0.15 0.23 0.06 0.15 0.15 0.28 0.06 0.11

 Self-Efficacy for Peer Interactions 0.10 0.15 0.05 0.17 0.08 0.10 0.09 0.09 0.09 0.10 0.17 0.21 0.10 0.11

 Victimization at School 0.10 0.11 0.04 0.07 0.12 0.11 0.07 0.07 0.06 0.11 0.11 0.11 0.04 0.06

 Normative Beliefs About Aggression 0.07 0.12 0.03 0.09 0.01 0.06 0.08 0.10 0.02 0.09 0.04 0.16 0.14 0.15

 Positive School Orientation 0.12 0.17 0.06 0.24 0.03 0.03 0.07 0.12 0.03 0.10 0.17 0.33 0.06 0.06

Primary caregiver-reported 

 Altruistic Behavior 0.15 0.16 0.18 0.19 0.22 0.26 0.17 0.17 0.13 0.12 0.24 0.32 0.10 0.15

 Positive Social Behavior 0.36 0.41 0.35 0.39 0.33 0.42 0.16 0.19 0.26 0.33 0.36 0.39 0.33 0.42

 Problem Behavior 0.34 0.36 0.31 0.32 0.27 0.33 0.09 0.12 0.23 0.30 0.41 0.45 0.26 0.32

Teacher-reported (child)

 ADHD-Related Behavior 0.22 0.25 0.14 0.24 0.27 0.33 0.20 0.26 0.25 0.29 0.33 0.36 0.20 0.28

 Academic Competence and Motivation 0.35 0.42 0.27 0.34 0.39 0.47 0.27 0.31 0.30 0.42 0.54 0.57 0.30 0.35

 Altruistic Behavior 0.02 0.05 0.00 0.08 0.00 0.04 0.00 0.03 0.02 0.05 0.01 0.00 0.02 0.10

 Positive Social Behavior 0.17 0.27 0.19 0.30 0.16 0.29 0.26 0.32 0.14 0.26 0.33 0.46 0.12 0.21

 Problem Behavior 0.20 0.28 0.20 0.30 0.18 0.32 0.20 0.25 0.25 0.29 0.36 0.46 0.17 0.21

Teacher-reported (school)

 Student Support for Teachers 0.28 0.26 0.53 0.54 0.30 0.28 0.31 0.29 0.29 0.31 0.56 0.56 0.18 0.23

 Feelings of Safety 0.42 0.42 0.48 0.50 0.12 0.11 0.36 0.35 0.12 0.18 0.41 0.41 0.16 0.21

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program PA: Positive Action

 CSP: Competence Support Program PATHS: Promoting Alternative Thinking Strategies

 LBW: Love In a Big World 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step ADHD: Attention deficit hyperactivity disorder

SOURCE: The Social and Character Development (SACD) Research Program.

B-34

Appendix B: Technical Notes
Appendix B: Technical Notes 

B-35

Table B.17. Adjusted R2 values from models using spring 2007 (Year 3) outcomes, with pretest 

only and with the full covariate set, for the combined-program sample 

Child level Collapsed to school level

Outcome Pretest only

Full 

covariate 

set Pretest only

Full 

covariate 

set

Child-reported 

 Empathy 0.09 0.14 0.40 0.67

 Student Afraid at School 0.06 0.12 0.51 0.70

 Altruistic Behavior 0.06 0.08 0.41 0.56

 Engagement with Learning 0.02 0.07 0.01 0.29

 Negative School Orientation 0.11 0.17 0.53 0.64

 Problem Behavior 0.14 0.23 0.55 0.74

 Self-Efficacy for Peer Interactions 0.06 0.08 0.15 0.30

 Victimization at School 0.06 0.07 0.25 0.28

 Normative Beliefs About Aggression 0.06 0.11 0.35 0.46

 Positive School Orientation 0.11 0.16 0.47 0.67

Primary caregiver-reported 

 Altruistic Behavior 0.16 0.20 0.40 0.74

 Positive Social Behavior 0.26 0.32 0.54 0.69

 Problem Behavior 0.19 0.23 0.31 0.64

Teacher-reported (child)

 ADHD-Related Behavior 0.20 0.26 0.05 0.69

 Academic Competence and Motivation 0.32 0.37 0.49 0.70

 Altruistic Behavior 0.06 0.07 0.10 0.20

 Positive Social Behavior 0.22 0.30 0.30 0.60

 Problem Behavior 0.22 0.28 0.42 0.61

Teacher-reported (school)

 Student Support for Teachers 0.47 0.47 0.78 0.77

 Feelings of Safety 0.30 0.30 0.50 0.48

NOTE: ADHD is the abbreviation for attention deficit hyperactivity disorder.

SOURCE: The Social and Character Development (SACD) Research Program.
Table B.18. Adjusted R2

 values from models using spring 2007 (Year 3) outcomes, with pretest only and with the full covariate set 

selected for each program, by program 

LBW CSP SS ABC PATHS PA 4Rs

Outcome

Pretest

only

Full 

set

Pretest

only

Full 

set

Pretest

only

Full 

set

Pretest

only

Full 

set

Pretest

only

Full 

set

Pretest

only

Full 

set

Pretest

only

Full 

set

Child-reported 

 Empathy 0.08 0.13 0.07 0.14 0.06 0.15 0.05 0.06 0.03 0.11 0.04 0.11 0.03 0.09

 Student Afraid at School 0.03 0.06 0.01 0.05 0.03 0.12 0.04 0.10 0.06 0.17 0.01 0.07 0.02 0.05

 Altruistic Behavior 0.04 0.08 0.08 0.08 0.05 0.09 0.05 0.08 0.03 0.09 0.04 0.05 0.03 0.04

 Engagement with Learning 0.06 0.07 0.00 0.04 0.09 0.13 0.02 0.04 0.05 0.12 0.02 0.07 0.01 0.07

 Negative School Orientation 0.05 0.08 0.04 0.10 0.07 0.14 0.07 0.08 0.19 0.28 0.04 0.12 0.01 0.05

 Problem Behavior 0.12 0.17 0.05 0.16 0.07 0.15 0.11 0.13 0.22 0.33 0.02 0.18 0.07 0.17

 Self-Efficacy for Peer Interactions 0.04 0.05 0.08 0.11 0.05 0.09 0.07 0.11 0.09 0.12 0.04 0.09 0.05 0.06

 Victimization at School 0.02 0.02 0.05 0.07 0.07 0.09 0.04 0.07 0.13 0.17 0.02 0.07 0.02 0.03

 Normative Beliefs About Aggression 0.07 0.12 0.01 0.14 0.01 0.06 0.01 0.05 0.03 0.13 0.03 0.08 0.07 0.12

 Positive School Orientation 0.07 0.07 0.02 0.08 0.02 0.07 0.09 0.12 0.09 0.28 0.04 0.18 0.02 0.07

Primary caregiver-reported 

 Altruistic Behavior 0.08 0.21 0.15 0.16 0.13 0.24 0.13 0.19 0.20 0.28 0.09 0.16 0.17 0.17

 Positive Social Behavior 0.22 0.29 0.29 0.33 0.31 0.35 0.23 0.28 0.35 0.40 0.22 0.28 0.12 0.19

 Problem Behavior 0.23 0.26 0.19 0.24 0.21 0.28 0.17 0.23 0.25 0.28 0.18 0.20 0.09 0.17

Teacher-reported (child)

 ADHD-Related Behavior 0.19 0.25 0.25 0.32 0.26 0.28 0.14 0.20 0.24 0.28 0.13 0.25 0.15 0.26

 Academic Competence and Motivation 0.28 0.33 0.25 0.36 0.35 0.41 0.27 0.32 0.34 0.40 0.20 0.27 0.27 0.33

 Altruistic Behavior 0.00 0.02 0.06 0.11 0.01 0.05 0.00 0.01 0.01 0.05 0.03 0.07 0.00 0.06

 Positive Social Behavior 0.14 0.22 0.21 0.32 0.15 0.27 0.11 0.15 0.27 0.41 0.17 0.29 0.19 0.24

 Problem Behavior 0.19 0.24 0.19 0.26 0.09 0.21 0.15 0.19 0.36 0.48 0.24 0.31 0.13 0.18

Teacher-reported (school)

 Student Support for Teachers 0.11 0.14 0.19 0.22 0.27 0.27 0.37 0.43 0.55 0.54 0.58 0.59 0.23 0.22

 Feelings of Safety 0.21 0.20 0.17 0.27 0.08 0.11 0.31 0.30 0.32 0.29 0.39 0.41 0.19 0.20

NOTE: Abbreviations are

 ABC: Academic and Behavioral Competencies Program PA: Positive Action

 CSP: Competence Support Program PATHS: Promoting Alternative Thinking Strategies

 LBW: Love In a Big World 4Rs: The 4Rs Program (Reading, Writing, Respect, and Resolution)

 SS: Second Step ADHD: Attention deficit hyperactivity disorder

SOURCE: The Social and Character Development (SACD) Research Program.

B-36

Appendix B: Technical Notes
Appendix B: Technical Notes 

B-37

Construction of Sample Weights 

Two sets of weights were constructed for the year-by-year SACD analysis: (1) base weights assigned equal 

weight to each program and to each school within a program, and (2) adjusted weights modified the base 

weights for study nonconsent, interview nonresponse, and students who entered the study schools after fall 

2004 (new entrants). The weighting approach means that the combined-program impacts were the simple 

averages of the program impacts (based on the same covariates within the regression), the program impacts 

were the simple averages of the school impacts, and the school impacts were weighted averages of the 

impacts in each classroom, with weights proportional to classroom size. The weighting approach was chosen 

to generate the average impact across schools (and programs) in the sample. This approach was grounded in 

the study design, in which random assignment was conducted at the school level within programs. This 

weighting approach produced unbiased estimates of the average treatment effect for the study schools and 

programs that were purposively selected for the evaluation. This approach was consistent with an analysis in 

which the 84 school means were used in the regression models (that is, a between-school analysis), and the 

model included treatment-by-program interactions that were weighted equally to estimate the overall crosssite impact. 

Base Weights 

For each spring (2005-07) follow-up instrument, base weights were constructed with the following properties: 

 Within each school, each survey respondent had the same weight. 

 Within each program, the weights in each school summed to the same value. 

 The weights in each program summed to the same value (equal to the average program sample size). 

 The sum of all the weights was equal to twice the original sample size (because, through random 

assignment, the control and treatment groups each represented the sample). 

To satisfy these objectives, the base weights for the student-level outcome measures were calculated using the 

following formula: 

 

 

 

 

 

where baseweightsp is the weight for a child respondent in school s and program p,

nsp

 is the average number 

of child respondents per study school, nsp is the number of respondents in school s and program p,

schp

is 

the average number of schools per program, and schp is the number of schools in program p. 

Equation (1) was also used to construct base weights for the school-level outcome measures from the 

Teacher Report on Classroom and School data. However in this case, the first term on the right-hand side of 

equation (1) refers to the number of third-, fourth-, and fifth-grade teacher respondents in the school instead 

of the number of child respondents. That is, the numerator of the first term is the average number of teacher 

respondents per study school, and the denominator is the number of teacher respondents in school s and 

program p. 

Base weights were constructed using the total number of respondents who completed each instrument, 

regardless of the incidence of missing data for specific variables in the instruments. 

Adjusted Weights 

To account for study nonconsent and interview nonresponse, adjusted weights were constructed for the 

student-level and the school-level outcome measures so that the impact estimates could be generalized to the 
Appendix B: Technical Notes 

B-38

full sample universe, rather than to those with follow-up data only. For the student-level outcome measures, 

the original cohort children (stayers) in a classroom who responded to an instrument were assumed to be 

representative of all original cohort children (stayers) in that classroom. This approach adjusted for different 

survey response rates across classrooms and used the following equation: 

 

 

 

 

 

 

Thus, adj_wt_childcsp is the adjusted weight based on enrollment for an original cohort child (stayer) in school s

and program p, 

ncsp

 refers to the average number of fifth-grade stayers per classroom c in school s at 

program p, 

ncsp

 refers to the number of stayer respondents in a specific classroom c in school s at program p, 

csp

 refers to the average number of classrooms in schools across programs, and 

sp

c

 refers to the number of 

classrooms in school s at program p. 

Because new entrants were less likely to get parental consent than the original cohort of children, the adjusted 

weights for the child assessments were constructed separately for the two groups of children. There were very 

few new entrants within some classrooms, so the nonresponse and nonconsent adjustments were conducted 

at the school level for this group. The equation for the adjusted weights for new entrants, which does not 

include a classroom-level component, is as follows: 

 

 

 

 

 

where adj_wt_new_entrantsp is the adjusted weight based on enrollment for a new entrant child in school s and 

program p,

nsp

 is the average population of new entrants per study school, and nsp is the number of new 

entrant respondents in school s and program p. 

For the school-level outcome measures, the responding teachers were assumed to be representative of third-, 

fourth-, and fifth-grade teachers in that school, which means that there is no classroom-level component in 

the construction of the Teacher Report on Classroom and School adjusted weights. The equation for these 

weights is as follows: 

 

 

 

 

 

where 

nsp

 is the average population of teachers per study school and nsp is the number of teacher 

respondents in school s and program p.

Weights for the Growth Curve Analysis

The growth curve analysis included data from every survey wave. Weights were constructed specifically for 

the growth curve analysis in order to maintain the equal representation of programs in the analysis. The 

growth curve weights were constructed only at the child level because the growth curve analysis was only 

performed for child-level outcomes. Growth curve weights were constructed with the following properties: 

 Within each school and survey period, each student had the same weight. 

 Within each program and survey period, the weights in each school summed to the same value. 

 Within each survey period, the weights in each program summed to the same value. 

 The weights within each survey period summed to the same value for all survey periods. 
Appendix B: Technical Notes 

B-39

The formula for the growth weights is as follows: 

 

 

 

 

 

 

 

where GC

weightt i p , ,

is the weight for students at survey period t, in school i in program p; 

nt kids ,

 represents 

the average number of students in a school at survey period t; nt,i,s represents the number of students in 

school i in program s at survey period t;

scht

 represents the average number of schools in a program at 

survey period t; and scht,s represents the number of schools in program s at survey period t. The average 

number of students in a survey period is represented by 

r

, while rt represents the number of students within 

survey period t. 

Sensitivity Analysis 

The hierarchical linear modeling (HLM) framework for estimating the impacts of the SACD programs on key 

child and school outcomes estimated impacts using regression models, where the standard errors of the 

impact estimates accounted for design effects due to school-level clustering and precision gains due to the 

inclusion of baseline covariates in the models (Bryk and Raudenbush 1992). 

To implement the HLM framework, it was necessary to make decisions about key model parameter 

specifications and estimation methods. A set of sensitivity analyses was done to determine if these model 

assumptions affected the results from the combined-program impact analysis for the 18 student-level 

outcome measures. Ten sensitivity tests were done and these included the following variations: 

 No initial covariates were included in the regression models. 

 The sample weights were not adjusted for study nonconsent and interview nonresponse and when 

models were estimated without weights. 

 The error structure included classroom-level random effects. 

 The error structure accounted for the pairwise matching of schools. 

 The pretests were treated as dependent variables rather than as covariates. 

 The impacts were estimated using alternative estimation routines (statistical software packages). 

 Missing outcome measures were imputed using multiple imputation procedures. 

 Combined-program impact estimates were obtained by averaging the program-level impact estimates. 

 Various sets of covariates were included in the regression models. 

 New entrants were excluded from the analysis. 

The number of sensitivity analyses done declined each year as the pattern of impact results from the original 

model proved robust to a variety of model specifications. All 10 sensitivity analyses were done with the Year 

1 data. The first 9 sensitivity tests were done with the Year 2 data. The sample of new entrants grew large 

enough in Years 2 and 3 to do separate analyses of new entrants versus stayers (in place of the 10th sensitivity 

test) and these were included in the subgroup analyses discussed in chapter 1. The 1st, 2nd (except for the 

estimation without weights), 3rd, and 4th sensitivity tests were done using the Year 3 data. The sensitivity 

analyses and their results are discussed below. 
Appendix B: Technical Notes 

B-40

Excluding Covariates 

Under the experimental design, the inclusion of baseline covariates was not required to obtain unbiased 

impact estimates. However, covariates were used because they increased the precision of the impact estimates 

by explaining some of the variations in the outcome measures between schools and across students within 

schools. In addition, covariates adjusted for residual differences between the baseline characteristics of 

treatment and control group members due to random sampling, study nonconsent, and interview 

nonresponse. 

As a specification test, the combined-program models were estimated with no covariates except for site 

binary variables (table B.19). As with the original impact estimates, no significant impacts were found from 

these simple differences-in-means estimates. 
Appendix B: Technical Notes 

B-41

Table B.19. Combined-program impacts in effect size units when baseline covariates are 

excluded from the models 

Year 1 Year 2 Year 3

(Spring 2005) (Spring 2006) (Spring 2007)

Scale–Report Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) -0.05 0.281 -0.07 0.164 -0.04 0.295

 Normative Beliefs About Aggression–CR (-) -0.01 0.766 -0.01 0.912 0.03 0.673

 Empathy–CR (+) 0.06 0.298 -0.04 0.524 -0.07 0.254

Behavior Domain

 Altruistic Behavior–CR (+) -0.06 0.295 -0.06 0.208 -0.02 0.626

 Altruistic Behavior–PCR (+) 0.05 0.274 0.01 0.836 -0.01 0.861

 Altruistic Behavior–TRS (+) 0.12 0.253 0.02 0.910 -0.05 0.695

 Positive Social Behavior–PCR (+) -0.01 0.898 0.04 0.532 0.01 0.907

 Positive Social Behavior–TRS (+) 0.02 0.830 -0.05 0.597 -0.01 0.917

 Problem Behavior–CR (-) 0.02 0.741 0.03 0.738 0.08 0.341

 Problem Behavior–PCR (+) -0.04 0.483 -0.07 0.142 -0.06 0.283

 Problem Behavior–TRS (+) 0.03 0.762 0.00 0.989 0.00 0.994

 ADHD-Related Behavior–TRS (-) 0.01 0.871 -0.01 0.867 -0.03 0.610

 

Academics Domain

 Engagement with Learning–CR (+) -0.06 0.183 -0.05 0.217 -0.06 0.237

 Academic Competence and 

 Motivation–TRS (+) -0.06 0.333 -0.07 0.298 -0.07 0.290

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 0.01 0.933 0.00 0.993 -0.09 0.363

 Negative School Orientation–CR (-) -0.02 0.735 0.01 0.942 0.02 0.753

 Student Afraid at School–CR (-) -0.04 0.480 -0.05 0.488 0.00 0.981

 Victimization at School–CR (-) -0.02 0.637 -0.04 0.458 0.00 0.978

1 Impacts are in effect size (standard deviation) units and were calculated by dividing the estimated impact by the standard deviation 

of the outcome measure for the control group. 

2 The p-value is from a two-tailed t test to gauge the statistical significance of the impact estimate.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. No findings were found statistically significant at or below 

the .05 level. All impact estimates were calculated using regression models, where each program and school within a program was 

weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of 

students within schools. 

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-42

Employing Alternative Weighting Schemes 

As discussed earlier in this appendix, the impact models were estimated using sample weights. The weights 

were constructed so that each site and each school within a site were given equal weight in the analysis, and 

survey respondents were assumed to represent not only themselves but also study nonconsenters and survey 

nonrespondents in the same classroom (by new entrant/original cohort status). 

To examine the sensitivity of results to different weighting schemes, models were estimated using weights not 

adjusted for study nonconsent and survey nonresponse for each of the 3 years (table B.20). In addition, 

models were estimated without weights for Years 1 and 2 (table B.21). The alternative weighting schemes did 

not change the statistical significance of the estimated impacts with one exception: a detrimental impact was 

found for Altruistic Behavior (CR) in Year 1 when weights were not adjusted for study nonconsent and 

survey nonresponse. 
Appendix B: Technical Notes 

B-43

Table B.20. Combined-program impacts in effect size units when weights are not adjusted for 

study nonconsent or survey nonresponse 

Year 1 Year 2 Year 3

(Spring 2005) (Spring 2006) (Spring 2007)

Scale–Report Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) -0.05 0.236 -0.07^ 0.073 -0.04 0.298

 Normative Beliefs About Aggression–CR (-) 0.00 0.950 -0.02 0.773 0.02 0.788

 Empathy–CR (+) 0.05 0.272 0.00 0.962 -0.03 0.529

Behavior Domain

 Altruistic Behavior–CR (+) -0.07* 0.041 -0.04 0.384 -0.03 0.452

 Altruistic Behavior–PCR (+) 0.06^ 0.078 -0.02 0.530 0.04 0.445

 Altruistic Behavior–TRS (+) 0.06 0.483 -0.01 0.959 -0.07 0.548

 Positive Social Behavior–PCR (+) 0.00 0.870 0.05 0.134 0.04 0.321

 Positive Social Behavior–TRS (+) 0.01 0.767 -0.02 0.748 0.04 0.522

 Problem Behavior–CR (-) 0.01 0.819 0.00 0.942 0.04 0.396

 Problem Behavior–PCR (+) 0.00 0.985 -0.06 0.119 -0.03 0.504

 Problem Behavior–TRS (+) 0.03 0.503 -0.04 0.402 -0.06 0.279

 ADHD-Related Behavior–TRS (-) 0.01 0.866 -0.05 0.347 -0.13 0.122

 

Academics Domain

 Engagement with Learning–CR (+) -0.05 0.154 -0.01 0.721 -0.04 0.308

Academic Competence and 

 Motivation–TRS (+) -0.02 0.365 0.00 0.929 0.01 0.924

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 0.03 0.547 0.04 0.634 -0.07 0.387

 Negative School Orientation–CR (-) -0.03 0.320 -0.04 0.365 -0.02 0.726

 Student Afraid at School–CR (-) -0.05 0.211 -0.07 0.145 0.00 0.970

 Victimization at School–CR (-) 0.00 0.940 -0.04 0.391 0.00 0.959

* Significantly different from zero at the .05 level.

^ Significantly different from zero at the .10 to > .05 level.

1 Impacts are in effect size (standard deviation) units and were calculated by dividing the estimated impact by the standard deviation 

of the outcome measure for the control group. 

2 The p-value is from a two-tailed t test to gauge the statistical significance of the impact estimate.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression 

models, where each program and school within a program was weighted equally. The standard errors of all estimates account for

design effects due to unequal weighting and the clustering of students within schools. 

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-44

Table B.21. Combined-program impacts in effect size units when weights are not used 

Year 1 Year 2

(Spring 2005) (Spring 2006)

Scale–Report Impact1

p-value of 

impact2Impact1

p-value of 

impact2

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) -0.05 0.211 -0.07 0.124

 Normative Beliefs About Aggression–CR (-) -0.01 0.825 0.00 0.973

 Empathy–CR (+) 0.04 0.474 -0.04 0.551

Behavior Domain

 Altruistic Behavior–CR (+) -0.07 0.190 -0.07 0.181

 Altruistic Behavior–PCR (+) 0.04 0.402 -0.01 0.858

 Altruistic Behavior–TRS (+) 0.11 0.298 0.01 0.925

 Positive Social Behavior–PCR (+) 0.00 0.974 0.01 0.851

 Positive Social Behavior–TRS (+) 0.01 0.934 -0.05 0.571

 Problem Behavior–CR (-) 0.02 0.740 0.02 0.827

 Problem Behavior–PCR (+) -0.02 0.672 -0.07 0.112

 Problem Behavior–TRS (+) 0.04 0.644 -0.01 0.946

 ADHD-Related Behavior–TRS (-) 0.03 0.636 -0.02 0.765

 

Academics Domain

 Engagement with Learning–CR (+) -0.07 0.120 -0.04 0.301

 Academic Competence and Motivation–TRS (+) -0.05 0.410 -0.06 0.314

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 0.01 0.932 0.01 0.881

 Negative School Orientation–CR (-) -0.01 0.889 -0.01 0.870

 Student Afraid at School–CR (-) -0.04 0.483 -0.06 0.336

 Victimization at School–CR (-) -0.02 0.725 -0.05 0.246

1 Impacts are in effect size (standard deviation) units and were calculated by dividing the estimated impact by the standard deviation 

of the outcome measure for the control group. 

2 The p-value is from a two-tailed t test to gauge the statistical significance of the impact estimate.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. No findings were found statistically significant at or below 

the .05 level. All impact estimates were calculated using regression models, where each program and school within a program was 

weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of 

students within schools. 

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-45

Including Classroom-Level Random Effects in the Error Structure 

In the original models, adjustments were made for the clustering of students within schools but not for the 

clustering of students within classrooms. This is because classrooms were not sampled for the study; instead, 

all classrooms within the study schools were included in the evaluation. Thus, the impact estimates 

generalized to the set of classrooms and teachers that were in the study schools at the time of the evaluation. 

An alternative view would be that the estimated impact findings generalized to a broader, but unknown, 

population of classrooms within the study schools. To accommodate this view, classroom-level random 

effects were included in the models using a three-level HLM model that was indexed by students (i), 

classrooms (c), schools (s), and fixed site effects (d):

 

 

 

In this model, Yicsd is an outcome measure for a student; 0csd is a classroom-level random intercept; 00sd is 

school-level random intercept; Xicsd are student-level baseline covariates; Wcsd are teacher-level baseline 

covariates; Tsd is a binary variable equal to 1 for treatment group schools and 0 for control group schools; Gd

are binary variables for each grantee; Zsd are school-level baseline covariates11; , , 0, 1, d, and  are 

parameter vectors to be estimated; eicsd are assumed to be iid N(0,2

e) student- or teacher-level random error 

terms; csd are iid N(0,2

) classroom-specific error terms that are assumed to be distributed independently of 

the Level 1 error terms; and usd are iid (0,2

u) school-specific error terms that are assumed to be distributed 

independently of the Level 1 and 2 error terms. 

Inserting the Level 2 and 3 equations into the Level 1 equation yields the following unified model: 

 

In this formulation, the estimate of the parameter, 1, is the regression-adjusted, multisite impact estimate. 

This equation is similar to the two-level benchmark model in equation (1) in the main report except that it 

includes the random classroom effect csd in the error term. As with the original impact estimates, no 

significant impacts were found when addressing the clustering of students within classroom (table B.22). 

 

11 In practice, school-level covariates were not used in estimation. 
Appendix B: Technical Notes 

B-46

Table B.22. Combined-program impacts in effect size units when accounting for classroom-level 

clustering 

Year 1 Year 2 Year 3

(Spring 2005) (Spring 2006) (Spring 2007)

Scale–Report Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) -0.02 0.548 -0.07^ 0.055 -0.06 0.114

 Normative Beliefs About Aggression–CR (-) -0.01 0.796 -0.01 0.776 0.00 0.970

 Empathy–CR (+) 0.06 0.186 -0.02 0.746 -0.05 0.254

Behavior Domain

 Altruistic Behavior–CR (+) -0.06 0.115 -0.06 0.162 -0.03 0.408

 Altruistic Behavior–PCR (+) 0.07^ 0.066 -0.01 0.871 -0.02 0.609

 Altruistic Behavior–TRS (+) 0.15^ 0.077 0.05 0.545 -0.03 0.671

 Positive Social Behavior–PCR (+) -0.01 0.632 0.04 0.264 0.02 0.526

 Positive Social Behavior–TRS (+) 0.01 0.703 0.01 0.849 0.05 0.400

 Problem Behavior–CR (-) 0.00 0.993 0.00 0.940 0.05 0.299

 Problem Behavior–PCR (+) 0.01 0.876 -0.03 0.336 -0.03 0.435

 Problem Behavior–TRS (+) 0.03 0.506 -0.03 0.598 -0.02 0.678

 ADHD-Related Behavior–TRS (-) 0.00 0.935 -0.05 0.344 -0.03 0.492

Academics Domain

 Engagement with Learning–CR (+) -0.04 0.240 -0.02 0.643 -0.03 0.509

 Academic Competence and 

 Motivation–TRS (+) -0.01 0.614 -0.01 0.772 0.00 0.928

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 0.03 0.554 0.04 0.611 -0.08 0.310

 Negative School Orientation–CR (-) -0.05 0.180 -0.04 0.451 -0.01 0.885

 Student Afraid at School–CR (-) -0.06 0.144 -0.08 0.101 -0.01 0.867

 Victimization at School–CR (-) -0.02 0.661 -0.05 0.321 0.00 0.958

^ Significantly different from zero at the .10 to > .05 level.

1 Impacts are in effect size (standard deviation) units and were calculated by dividing the estimated impact by the standard deviation 

of the outcome measure for the control group. 

2 The p-value is from a two-tailed t test to gauge the statistical significance of the impact estimate.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. No findings were found statistically significant below the 

.05 level. All impact estimates were calculated using regression models, where each program and school within a program was 

weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of 

students within schools. 

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-47

Accounting for the Pairwise Matching of Schools 

Under an experimental design where the random assignment of units is conducted within strata, it is 

customary to account for the stratified design in the analysis by including binary variables of the strata as 

covariates in the regression models. Under a clustered design, the inclusion of these variables will reduce 

design effects to the extent that they are correlated with the outcome measures. Stated differently, under 

stratified designs, intraclass correlations (ICCs) pertain to design effects within strata. Thus, stratified sampling 

will reduce ICCs if mean school outcomes tend to be more similar within strata than across strata. 

This procedure, however, cannot be used for a design where random assignment is conducted for units 

within matched pairs. This is because when pair binary variables and pair-by-treatment interaction terms are 

included as model covariates, the school-level variance term represents the extent to which mean school 

outcomes vary within pairs and within a research group (Murray 1998). Thus, because there is only one 

treatment and one control group unit per pair, there are not enough degrees of freedom to estimate the 

within-pair variance terms. 

To account for this problem, the pairs were treated as another HLM level. Specifically, the following threelevel HLM model was estimated, where pairs were indexed by p:

 

 

 

In this model, pd and pd are independent and identically distributed N(0,

2

) and independent and identically 

distributed N(0,

2

) pair-level random error terms, respectively. Recursively inserting the Level 2 and 3 

equations into the Level 1 equation yields the following unified model: 

 

Under this model, the key estimatable component of the variance expression for the impact estimate is 

(

2

/P), where P is the total number of pairs. This variance component signifies the extent to which impacts

vary across pairs. In contrast, the leading term in the variance expression for the benchmark approach is 

(2

2

u/P), which signifies the extent to which mean outcomes vary across schools. 

It is unclear a priori whether the pairwise matching model will produce more precise impact estimates than 

the original approach. On the one hand, precision gains could occur if 

2

 < 2

2

u which will likely be the case 

as long as there is some correlation between mean outcomes for the treatment and control group schools 

within the same pairs (that is, if the matching was somewhat “successful”). On the other hand, precision 

losses will occur due to a reduction in the number of degrees of freedom available for analysis, because there 

are only half as many pairs as there are schools. Thus, it is an empirical issue as to which approach will yield 

more precise impact estimates. 

Accounting for pairwise matching in the analysis did not change the statistical significance of the estimated 

impacts with one exception: a detrimental impact was found for Altruistic Behavior (CR) in Year 1 (table 

B.23). 
Appendix B: Technical Notes 

B-48

Table B.23. Combined-program impacts in effect size units when accounting for pairwise 

matching 

Year 1 Year 2 Year 3

(Spring 2005) (Spring 2006) (Spring 2007)

Scale–Report Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) -0.03 0.386 -0.07 0.135 -0.05 0.217

 Normative Beliefs About Aggression–CR (-) 0.00 0.951 0.00 0.957 0.01 0.876

 Empathy–CR (+) 0.05 0.302 -0.02 0.742 -0.05 0.343

Behavior Domain

 Altruistic Behavior–CR (+) -0.07* 0.036 -0.03 0.444 -0.03 0.477

 Altruistic Behavior–PCR (+) 0.05 0.118 -0.03 0.420 -0.06 0.157

 Altruistic Behavior–TRS (+) 0.08 0.374 0.02 0.910 -0.02 0.868

 Positive Social Behavior–PCR (+) -0.02 0.528 0.01 0.772 0.02 0.606

 Positive Social Behavior–TRS (+) 0.01 0.709 -0.03 0.618 0.05 0.470

 Problem Behavior–CR (-) 0.01 0.797 -0.01 0.906 0.02 0.759

 Problem Behavior–PCR (+) -0.01 0.814 -0.02 0.602 -0.05 0.371

 Problem Behavior–TRS (+) 0.03 0.444 -0.02 0.720 -0.05 0.405

 ADHD-Related Behavior–TRS (-) 0.02 0.648 -0.02 0.712 -0.09 0.105

 

Academics Domain

 Engagement with Learning–CR (+) -0.05^ 0.083 -0.03 0.508 -0.06 0.201

 Academic Competence and 

 Motivation–TRS (+) -0.03 0.345 -0.04 0.338 -0.02 0.718

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 0.01 0.852 0.04 0.395 -0.05 0.424

 Negative School Orientation–CR (-) -0.03 0.303 -0.07 0.192 -0.03 0.543

 Student Afraid at School–CR (-) -0.06^ 0.089 -0.05 0.328 0.04 0.440

 Victimization at School–CR (-) 0.00 0.999 -0.03 0.370 0.01 0.791

* Significantly different from zero at the .05 level.

^ Significantly different from zero at the .10 to > .05 level.

1 Impacts are in effect size (standard deviation) units and were calculated by dividing the estimated impact by the standard deviation 

of the outcome measure for the control group. 

2 The p-value is from a two-tailed t test to gauge the statistical significance of the impact estimate.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression 

models, where each program and school within a program was weighted equally. The standard errors of all estimates account for

design effects due to unequal weighting and the clustering of students within schools. 

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-49

Treating Pretest Measures as Dependent Variables 

In the original model, initial measures of the outcomes were included as covariates in the regression models. 

An alternative approach was to treat these pretests as dependent variables. This approach was implemented in 

two ways using Year 1 and Year 2 data. First, models were estimated where the dependent variable was 

defined as the difference between posttest and pretest scores (that is, gain scores), which yields a difference-indifference estimator.12 Second, the pretest and posttest scores were stacked as separate observations and the 

model included time-by-treatment interaction terms as covariates as shown here: 

 

where 

 Yisd is an outcome measure for student (or teacher) i; 

 Xisd are student-level (or teacher-level) baseline covariates; 

 Tsd is a binary variable equal to 1 for treatment group schools and 0 for control group schools; 

 d are site-specific fixed effects; 

 Zsd are school-level baseline covariates;13

 Post is a binary variable that equals 1 for the posttest observations and 0 for the pretest observations; 

 , 0, 1, 2, 3, and  are fixed parameter vectors to be estimated; and 

 eisd are independent and identically distributed N(0,2

e) student-level (or teacher-level) random error 

terms. 

Furthermore, usd and ηsd are independent and identically distributed N(0,2

u) and N(0,2η) school-level random 

error terms, respectively, that capture the correlations between the outcomes of students in the same schools 

and time periods and that are distributed independently of the student-level (or teacher-level) error terms. In 

this formulation, the estimate of γ3 is the difference-in-difference impact estimate, and the leading term in the 

variance expression for the impact estimate is a function of 2

η, which signifies the extent to which gain 

scores vary across schools within programs. 

For each estimation approach, the new entrants were excluded from the analysis sample because of missing 

pretest scores.14 Thus, the models were estimated using only original cohort members. 

In the original model, none of the 36 estimated impacts was found to be significant. The treatment of the 

pretests as dependent variables rather than as covariates gave similar results with two exceptions. When using 

the gain score as the dependent variable, a statistically significant beneficial impact was found for Negative 

School Orientation (in Year 2 with an effect size of -0.15) and Student Afraid at School (in Year 1 with an 

effect size of 0.09) (table B.24). When using pretests and posttests as dependent variables, a beneficial impact 

on Negative School Orientation was found in Year 2 (effect size of -0.16) and on Student Afraid at School in 

Year 1 (effect size of 0.10) (table B.25). 

 

12 In these models, the pretests were excluded from the model covariates, although other covariates remained the same 

as in the benchmark models. 

13 For teacher-level outcomes, the Z variables included teacher-level baseline covariates. Although the initial model 

specified the use of school-level covariates, none were used in the empirical estimation. 

14 Using the mean-imputed pretests for the new entrants would yield standard errors that are biased downward. 
Appendix B: Technical Notes 

B-50

Table B.24. Combined-program impacts in effect size units when dependent variable is the 

pretest-posttest difference (gain score) 

Year 1 Year 2

(Spring 2005) (Spring 2006)

Scale–Report Impact1

p-value of 

impact2Impact1

p-value of 

impact2

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 0.01 0.883 -0.04 0.342

 Normative Beliefs About Aggression–CR (-) 0.02 0.628 0.01 0.864

 Empathy–CR (+) 0.03 0.663 -0.01 0.805

Behavior Domain

 Altruistic Behavior–CR (+) -0.09^ 0.057 -0.04 0.482

 Altruistic Behavior–PCR (+) 0.04 0.356 -0.04 0.430

 Altruistic Behavior–TRS (+) 0.08 0.585 -0.08 0.603

 Positive Social Behavior–PCR (+) -0.02 0.569 0.03 0.546

 Positive Social Behavior–TRS (+) -0.03 0.599 -0.08 0.360

 Problem Behavior–CR (-) 0.03 0.605 -0.03 0.587

 Problem Behavior–PCR (+) 0.01 0.882 0.01 0.872

 Problem Behavior–TRS (+) 0.08 0.249 0.00 0.998

 ADHD-Related Behavior–TRS (-) 0.06 0.320 0.00 0.989

Academics Domain

 Engagement with Learning–CR (+) -0.03 0.444 0.03 0.539

 Academic Competence and Motivation–TRS (+) -0.04 0.470 -0.04 0.540

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 0.00 0.989 0.07 0.239

 Negative School Orientation–CR (-) -0.06 0.126 -0.15* 0.007

 Student Afraid at School–CR (-) -0.09* 0.028 -0.04 0.433

 Victimization at School–CR (-) 0.00 0.973 -0.02 0.639

* Significantly different from zero at the .05 level.

^ Significantly different from zero at the .10 to > .05 level.

1 Impacts are in effect size (standard deviation) units and were calculated by dividing the estimated impact by the standard deviation 

of the outcome measure for the control group. 

2 The p-value is from a two-tailed t test to gauge the statistical significance of the impact estimate.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression 

models, where each program and school within a program was weighted equally. The standard errors of all estimates account for

design effects due to unequal weighting and the clustering of students within schools. 

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-51

Table B.25. Combined-program impacts in effect size units when posttest and pretest are treated 

as separate dependent variables 

Year 1 Year 2

(Spring 2005) (Spring 2006)

Scale–Report Impact1

p-value of 

impact2Impact1

p-value of 

impact2

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) 0.01 0.818 -0.04 0.417

 Normative Beliefs About Aggression–CR (-) 0.03 0.486 0.02 0.799

 Empathy–CR (+) 0.02 0.708 -0.03 0.654

Behavior Domain

 Altruistic Behavior–CR (+) -0.09^ 0.062 -0.07 0.341

 Altruistic Behavior–PCR (+) 0.03 0.567 -0.04 0.461

 Altruistic Behavior–TRS (+) 0.07 0.607 -0.09 0.627

 Positive Social Behavior–PCR (+) -0.03 0.544 0.02 0.598

 Positive Social Behavior–TRS (+) -0.02 0.645 -0.08 0.369

 Problem Behavior–CR (-) 0.03 0.521 -0.02 0.786

 Problem Behavior–PCR (+) 0.01 0.785 0.02 0.715

 Problem Behavior–TRS (+) 0.04 0.345 0.01 0.912

 ADHD-Related Behavior–TRS (-) 0.04 0.390 0.00 0.988

Academics Domain

 Engagement with Learning–CR (+) -0.05 0.419 0.01 0.843

 Academic Competence and Motivation–TRS (+) -0.01 0.683 -0.03 0.535

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) -0.01 0.827 0.07 0.311

 Negative School Orientation–CR (-) -0.06 0.238 -0.16* 0.024

 Student Afraid at School–CR (-) -0.10* 0.043 -0.05 0.394

 Victimization at School–CR (-) 0.00 0.944 -0.02 0.692

* Significantly different from zero at the .05 level.

^ Significantly different from zero at the .10 to > .05 level.

1 Impacts are in effect size (standard deviation) units and were calculated by dividing the estimated impact by the standard deviation 

of the outcome measure for the control group. 

2 The p-value is from a two-tailed t test to gauge the statistical significance of the impact estimate.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression 

models, where each program and school within a program was weighted equally. The standard errors of all estimates account for

design effects due to unequal weighting and the clustering of students within schools. 

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-52

Using Alternative Software Packages 

The original models were estimated using the PROC MIXED procedure in SAS. To check the sensitivity of 

the results, the impacts were estimated using alternative statistical packages—SUDAAN PROC REGRESS 

and HLM6—that account for clustering effects in slightly different ways than SAS PROC MIXED. SAS 

PROC MIXED uses a maximum-likelihood approach to general linear mixed models, whereas the SUDAAN 

procedure is based on the Taylor-series linearization method, combined with variance estimation formulas 

specific to the sample design. Similar to SAS PROC MIXED, HLM6 uses a restricted maximum likelihood 

approach to estimate the parameters, but HLM6 uses the Expectation-Maximization algorithm to obtain the 

estimates, whereas SAS PROC MIXED uses a ridge-stabilized Newton-Raphson algorithm. 

The three statistical packages produced similar impact and standard error estimates (table B.26) with one 

exception. When SUDAAN was used, the estimate for Altruistic Behavior (Primary Caregiver Report) 

showed a beneficial statistically significant effect in Year 1 (with an effect size of 0.07). 
Appendix B: Technical Notes 

Table B.26. Combined-program impacts in effect size units when using different statistical software packages 

Year 1 Year 2

(Spring 2005) (Spring 2006)

SUDAAN HLM6 SUDAAN HLM6

Scale–Report Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) -0.02 0.518 -0.04 0.291 -0.07^ 0.084 -0.06 0.209

 Normative Beliefs About Aggression–CR (-) -0.01 0.875 0.01 0.896 -0.03 0.603 0.03 0.624

 Empathy–CR (+) 0.06 0.148 0.05 0.223 -0.02 0.729 -0.01 0.779

Behavior Domain

 Altruistic Behavior–CR (+) -0.06^ 0.092 -0.06 0.105 -0.06 0.144 -0.03 0.430

 Altruistic Behavior–PCR (+) 0.07* 0.037 0.05 0.213 0.00 0.963 -0.02 0.605

 Altruistic Behavior–TRS (+) 0.08 0.288 0.08 0.480 0.06 0.629 -0.09 0.617

 Positive Social Behavior–PCR (+) -0.01 0.707 0.00 0.901 0.04 0.178 0.01 0.787

 Positive Social Behavior–TRS (+) 0.02 0.625 0.01 0.812 -0.02 0.728 -0.03 0.691

 Problem Behavior–CR (-) 0.01 0.872 0.04 0.436 -0.01 0.893 0.02 0.756

 Problem Behavior–PCR (+) 0.00 0.940 -0.04 0.308 -0.03 0.300 -0.02 0.633

 Problem Behavior–TRS (+) 0.02 0.572 0.02 0.662 -0.01 0.800 -0.03 0.543

 ADHD-Related Behavior–TRS (-) 0.00 0.942 0.02 0.696 -0.02 0.497 -0.02 0.768

Academics Domain

 Engagement with Learning–CR (+) -0.04 0.182 -0.07^ 0.053 -0.03 0.400 -0.03 0.521

 Academic Competence and Motivation–TRS (+) -0.02 0.365 -0.08^ 0.053 -0.02 0.498 -0.05 0.282

See notes at end of table.

B-53

Appendix B: Technical Notes
Table B.26. Combined-program impacts in effect size units when using different statistical software packages—Continued

Year 1 Year 2

(Spring 2005) (Spring 2006)

SUDAAN HLM6 SUDAAN HLM6

Scale–Report Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 0.03 0.467 0.00 0.973 0.05 0.432 0.03 0.703

 Negative School Orientation–CR (-) -0.05 0.127 -0.01 0.869 -0.04 0.344 -0.06 0.364

 Student Afraid at School–CR (-) -0.06 0.127 -0.04 0.391 -0.06 0.166 -0.04 0.500

 Victimization at School–CR (-) -0.01 0.765 0.02 0.648 -0.04 0.370 -0.03 0.506

* Significantly different from zero at the .05 level.

^ Significantly different from zero at the .10 to > .05 level.

1 Impacts are in effect size (standard deviation) units and were calculated by dividing the estimated impact by the standard deviation of the outcome measure for the control group. 

2 The p-value is from a two-tailed t test to gauge the statistical significance of the impact estimate.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models, where each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. 

SOURCE: The Social and Character Development (SACD) Research Program. 

B-54

Appendix B: Technical Notes
Appendix B: Technical Notes 

B-55

Imputing Missing Outcomes 

In the original models, missing outcome data were not imputed for (1) study nonconsenters, (2) survey 

nonrespondents, and (3) survey respondents who did not provide enough information to construct the 

outcomes. Instead, adjustments for missing outcome data were made using sample weights so that the impact 

estimates could be generalized to the full sample universe (and not just to survey respondents). 

As a sensitivity test, models were estimated with imputed outcome data. Nearly all students with missing data 

were study nonconsenters for whom baseline data were not available. Thus, imputation methods that used 

covariates as predictor variables could not be used. Instead, missing outcome data were imputed using a 

hotdeck imputation procedure (Rubin 1987), where a missing value for a child was replaced by a nonmissing 

value for a randomly selected child in the same classroom. Five imputed data sets were generated and 

estimated impacts (Ii) and variances (Vi) were made for each one. The overall impact estimates were 

calculated as the average of the five impact estimates. Variances of the estimated impacts were obtained using 

the following formula: 

 

 

 

 

where 

_

V

 is the average of the Vis across the five data sets, m = 5 is the number of imputed data sets, and B is 

the following between-imputation variance estimate: 

 

 

 

 

 

Because child-level covariates were not available for study nonconsenters, the regression models were 

estimated without covariates, except for program-level binary variables. 

As with the original impact estimates, no significant impacts were found when using the multiple imputation 

approach (table B.27). 
Appendix B: Technical Notes 

B-56

Table B.27. Combined-program impacts in effect size units when using imputation procedures 

for missing outcome data 

Year 1 Year 2

(Spring 2005) (Spring 2006)

Scale–Report Impact1

p-value of 

impact2Impact1

p-value of 

impact2

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) -0.05 0.293 -0.06 0.296

 Normative Beliefs About Aggression–CR (-) -0.01 0.883 -0.02 0.733

 Empathy–CR (+) 0.06 0.375 -0.04 0.543

Behavior Domain

 Altruistic Behavior–CR (+) -0.06 0.307 -0.07 0.274

 Altruistic Behavior–PCR (+) 0.06 0.305 -0.01 0.831

 Altruistic Behavior–TRS (+) 0.12 0.278 0.05 0.722

 Positive Social Behavior–PCR (+) -0.01 0.891 0.01 0.859

 Positive Social Behavior–TRS (+) 0.01 0.868 -0.06 0.530

 Problem Behavior–CR (-) 0.03 0.697 0.02 0.803

 Problem Behavior–PCR (+) -0.05 0.428 -0.06 0.248

 Problem Behavior–TRS (+) 0.03 0.724 0.01 0.913

 ADHD-Related Behavior–TRS (-) 0.01 0.831 -0.02 0.743

Academics Domain

 Engagement with Learning–CR (+) -0.06 0.236 -0.05 0.265

 Academic Competence and Motivation–TRS (+) -0.05 0.440 -0.08 0.211

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 0.00 0.981 0.02 0.821

 Negative School Orientation–CR (-) -0.02 0.788 0.00 0.954

 Student Afraid at School–CR (-) -0.04 0.549 -0.05 0.511

 Victimization at School–CR (-) -0.02 0.661 -0.06 0.279

1 Impacts are in effect size (standard deviation) units and were calculated by dividing the estimated impact by the standard deviation 

of the outcome measure for the control group. 

2 The p-value is from a two-tailed t test to gauge the statistical significance of the impact estimate.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. No findings were found statistically significant at or below 

the .05 level. All impact estimates were calculated using regression models, where each program and school within a program was 

weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of 

students within schools. 

SOURCE: The Social and Character Development (SACD) Research Program.
Appendix B: Technical Notes 

B-57

Averaging Program-Level Impact Estimates 

An alternative approach for estimating combined-program impacts was to calculate the simple average of the 

seven program-specific impact estimates. This procedure was done in two ways. First, the seven program-bytreatment interaction terms (that is, p*Tsp terms) were added as covariates to the original combined-program 

model, and the treatment status binary variable was excluded from the model. In this formulation, the 

parameter estimate on a specific program-by-treatment interaction term was the impact estimate for that 

program. Using this approach, the combined-program impact estimate was calculated as the simple average of 

the parameter estimates on the seven interaction terms, and the associated standard error (SE) was calculated 

using the following formula: 

 

 

 

 

 

 

 

 

 

 

where Vp is the estimated variance of the impact in program p and Cjk is the estimated covariance between the 

impacts in programs j and k. 

In the second approach, combined-program impact estimates were calculated by averaging the impact 

estimates from the HLM models that were estimated separately by program. Equation (13) was used to 

calculate the standard errors with Cjk set to 0 (because observations were independent across programs). This 

approach differed slightly from the first approach for several reasons. First, a different set of baseline 

covariates was used for each program-specific model, whereas a common covariate set was used in the 

pooled model. Second, in the pooled model, a single (average) clustering effect was applied to all programs, 

whereas, in the program-specific models, the clustering effects were allowed to vary across programs. 

As table B.28 shows, the combined-program impact estimates from these two approaches in which programspecific impact estimates were calculated directly and then averaged were very similar to the original results 

that relied on weights to give each program equal weight in the analysis. The only difference was that a 

beneficial significant impact (effect size of 0.07) was found for Positive Social Behavior (PCR) in Year 2 when 

using the average of the estimates from the seven program-level models. 
Table B.28. Combined-program impacts in effect size units using averages of program-specific impacts 

Year 1 Year 2

(Spring 2005) (Spring 2006)

Program by 

treatment

Seven separate 

estimates

Program by 

treatment

Seven separate

estimates

Scale–Report Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) -0.02 0.543 -0.01 0.717 -0.07 0.107 -0.06 0.145

 Normative Beliefs About Aggression–CR (-) -0.01 0.871 -0.01 0.754 -0.02 0.748 -0.01 0.535

 Empathy–CR (+) 0.06 0.176 0.07 0.150 -0.02 0.721 -0.02 0.596

Behavior Domain

 Altruistic Behavior–CR (+) -0.06 0.107 -0.07^ 0.057 -0.06 0.193 -0.08 0.100

 Altruistic Behavior–PCR (+) 0.07^ 0.058 0.07^ 0.095 0.00 0.965 -0.01 0.810

 Altruistic Behavior–TRS (+) 0.08 0.378 0.07 0.460 0.02 0.902 0.06 0.868

 Positive Social Behavior–PCR (+) -0.01 0.725 -0.01 0.774 0.05 0.177 0.07* 0.037

 Positive Social Behavior–TRS (+) 0.01 0.709 0.01 0.860 -0.02 0.768 0.00 0.957

 Problem Behavior–CR (-) 0.01 0.892 0.01 0.886 0.00 0.998 0.02 0.879

 Problem Behavior–PCR (+) 0.00 0.930 0.01 0.811 -0.03 0.341 -0.04 0.271

 Problem Behavior–TRS (+) 0.02 0.607 0.03 0.545 -0.01 0.782 -0.02 0.513

 ADHD-Related Behavior–TRS (-) 0.00 0.958 0.00 0.970 -0.03 0.508 -0.05 0.416

Academics Domain

 Engagement with Learning–CR (+) -0.04 0.217 -0.04 0.345 -0.03 0.490 -0.03 0.555

 Academic Competence and Motivation–TRS (+) -0.02 0.433 -0.02 0.448 -0.02 0.653 0.01 0.835

See notes at end of table.

B-58

Appendix B: Technical Notes
Table B.28. Combined-program impacts in effect size units using averages of program-specific impacts—Continued

Year 1 Year 2

(Spring 2005) (Spring 2006)

Program by 

treatment

Seven separate 

estimates

Program by 

treatment

Seven separate 

estimates

Scale–Report Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 0.03 0.586 0.04 0.438 0.03 0.735 0.03 0.695

 Negative School Orientation–CR (-) -0.05 0.167 -0.05 0.209 -0.03 0.503 -0.02 0.653

 Student Afraid at School–CR (-) -0.06 0.171 -0.06 0.140 -0.06 0.218 -0.09 0.101

 Victimization at School–CR (-) -0.01 0.793 -0.01 0.763 -0.04 0.431 -0.02 0.639

* Significantly different from zero at the .05 level.

^ Significantly different from zero at the .10 to > .05 level.

1 Impacts are in effect size (standard deviation) units and were calculated by dividing the estimated impact by the standard deviation of the outcome measure for the control group. 

2 The p-value is from a two-tailed t test to gauge the statistical significance of the impact estimate.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. All impact estimates were calculated using regression models, where each program and school within a 

program was weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of students within schools. 

SOURCE: The Social and Character Development (SACD) Research Program.

B-59

Appendix B: Technical Notes
Appendix B: Technical Notes 

B-60

Excluding New Entrants 

About 7.5 percent of children in the spring 2005 follow-up sample were new entrants who enrolled in the 

study schools after fall 2004. For several reasons, the inclusion of these children in the analysis sample could 

have affected the impact estimates. First, new entrants were exposed to the SACD interventions for a shorter 

period than were original cohort children, which could have depressed the impact estimates. Second, new 

entrants may have had different characteristics than did original cohort members, which could have led to 

different program effects across the two groups. For example, the description of consent rates in chapter 1 

showed that new entrants had lower consent rates (and percentages of the sample with data) than did original 

cohort members, potentially reflecting such differences. Third, the new entrants in the treatment schools 

could have differed from those in the control schools if the SACD interventions had an effect on school 

entry.15 Finally, the imputation of baseline pretest scores for the new entrants (based on mean pretest scores 

of original cohort members in the same class) could have influenced the impact estimates. 

As a sensitivity test, models were estimated that excluded new entrants from the analysis sample using Year 1 

and Year 2 data. As with the original impact estimates, no significant impacts were found when new entrants 

were excluded (table B.29). 

For Years 2 and 3, the numbers of new entrants grew large enough to do separate analyses of new entrants 

and original members of the sample. These analyses are discussed in chapter 1 as part of the subgroup 

analyses. They include the finding of no significant impacts on the 18 outcomes for either group and the 

finding of no significant differences between the impacts on the new entrants versus the original cohort 

members. 

 

15 Similarly, the original cohort stayers in the treatment and control group schools could differ if the interventions had an 

effect on school exit rates. 
B-61

Table B.29. Combined-program impacts in effect size units when excluding new entrants 

Year 1 Year 2

(Spring 2005) (Spring 2006)

Scale–Report Impact1

p-value of 

impact2Impact1

p-value of 

impact2

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) -0.06 0.139 -0.06 0.176

 Normative Beliefs About Aggression–CR (-) 0.00 0.908 0.00 0.996

 Empathy–CR (+) 0.06 0.260 -0.01 0.798

Behavior Domain

 Altruistic Behavior–CR (+) -0.07^ 0.080 -0.03 0.516

 Altruistic Behavior–PCR (+) 0.05 0.184 -0.01 0.807

 Altruistic Behavior–TRS (+) 0.14 0.204 0.02 0.903

 Positive Social Behavior–PCR (+) -0.02 0.620 0.01 0.813

 Positive Social Behavior–TRS (+) 0.01 0.822 -0.03 0.651

 Problem Behavior–CR (-) 0.01 0.887 -0.02 0.759

 Problem Behavior–PCR (+) -0.04 0.411 -0.05 0.244

 Problem Behavior–TRS (+) 0.03 0.438 -0.03 0.534

 ADHD-Related Behavior–TRS (-) 0.02 0.721 -0.03 0.590

Academics Domain

 Engagement with Learning–CR (+) -0.08^ 0.064 -0.03 0.533

 Academic Competence and Motivation–TRS (+) -0.07 0.108 -0.07 0.170

 

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 0.01 0.927 0.05 0.561

 Negative School Orientation–CR (-) -0.01 0.805 -0.05 0.432

 Student Afraid at School–CR (-) -0.06 0.167 -0.05 0.359

 Victimization at School–CR (-) -0.01 0.879 -0.05 0.353

^ Significantly different from zero at the .10 to > .05 level.

1 Impacts are in effect size (standard deviation) units and were calculated by dividing the estimated impact by the standard deviation of 

the outcome measure for the control group. 

2 The p-value is from a two-tailed t test to gauge the statistical significance of the impact estimate.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The + signs in parentheses indicate the direction of a beneficial outcome. No findings were found statistically significant at or below the 

.05 level. All impact estimates were calculated using regression models, where each program and school within a program was 

weighted equally. The standard errors of all estimates account for design effects due to unequal weighting and the clustering of 

students within schools. 

SOURCE: The Social and Character Development (SACD) Research Program.

Including Restricted Sets of Covariates in the Regression Models 

Two sets of regression models were estimated using restricted sets of covariates. In the first set, the covariates 

included only the pretest of the outcome measure. In the second set, the covariates included only the child 

and primary caregiver demographic measures (listed in the first section of this appendix). As with the original 

impact estimates, no significant impacts were found when using the restricted sets of covariates (table B.30).
Table B.30. Combined-program impacts in effect size units using restricted sets of covariates (pretest of outcomes only and child and 

primary caregiver demographic measures only) 

Year 1 Year 2

(Spring 2005) (Spring 2006)

Covariates for 

pretest of 

outcomes only

Covariates for 

child and 

primary caregiver

Covariates for 

pretest of 

outcomes only

Covariates for 

child and 

primary caregiver

Scale–Report Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2

Social and Emotional Competence Domain

 Self-Efficacy for Peer Interactions–CR (+) -0.02 0.520 -0.05 0.240 -0.06 0.155 -0.08^ 0.082

 Normative Beliefs About Aggression–CR (-) 0.00 0.911 -0.02 0.549 -0.01 0.919 -0.02 0.656

 Empathy–CR (+) 0.06 0.271 0.07 0.144 -0.03 0.589 -0.03 0.625

Behavior Domain

 Altruistic Behavior–CR (+) -0.07 0.103 -0.07 0.125 -0.07 0.125 -0.07 0.122

 Altruistic Behavior–PCR (+) 0.06 0.148 0.04 0.307 0.01 0.866 0.00 0.919

 Altruistic Behavior–TRS (+) 0.07 0.424 0.13 0.236 0.01 0.964 0.02 0.914

 Positive Social Behavior–PCR (+) -0.01 0.745 0.01 0.893 0.05 0.230 0.06 0.219

 Positive Social Behavior–TRS (+) 0.01 0.864 0.04 0.568 -0.05 0.517 -0.03 0.719

 Problem Behavior–CR (-) 0.01 0.788 0.00 0.938 0.01 0.848 0.00 0.966

 Problem Behavior–PCR (+) 0.01 0.825 -0.04 0.431 -0.03 0.470 -0.08^ 0.062

 Problem Behavior–TRS (+) 0.03 0.593 0.00 0.948 0.00 0.937 -0.02 0.765

 ADHD-Related Behavior–TRS (-) 0.00 0.917 0.00 0.933 -0.02 0.664 -0.04 0.661

 

Academics Domain

 Engagement with Learning–CR (+) -0.04 0.267 -0.05 0.194 -0.03 0.363 -0.04 0.246

 Academic Competence and Motivation–TRS (+) -0.01 0.597 -0.04 0.440 -0.02 0.701 -0.05 0.340

See notes at end of table.

B-62

Appendix B: Technical Notes
Table B.30. Combined-program impacts in effect size units using restricted sets of covariates (pretest of outcomes only and child 

Table B.30. and primary caregiver demographic measures only)—Continued

Year 1 Year 2

(Spring 2005) (Spring 2006)

Covariates for 

pretest of 

outcomes only

Covariates for 

child and 

primary caregiver

Covariates for 

pretest of 

outcomes only

Covariates for 

child and 

primary caregiver

Scale–Report Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2Impact1

p-value of 

impact2

Perceptions of School Climate Domain

 Positive School Orientation–CR (+) 0.02 0.747 0.02 0.772 0.02 0.824 0.01 0.887

 Negative School Orientation–CR (-) -0.05 0.354 -0.04 0.496 -0.02 0.729 -0.01 0.868

 Student Afraid at School–CR (-) -0.05 0.258 -0.04 0.397 -0.05 0.424 -0.05 0.394

 Victimization at School–CR (-) -0.02 0.609 -0.03 0.593 -0.04 0.420 -0.05 0.389

^ Significantly different from zero at the .10 to > .05 level.

1 Impacts are in effect size (standard deviation) units and were calculated by dividing the estimated impact by the standard deviation of the outcome measure for the control group. 

2 The p-value is from a two-tailed t test to gauge the statistical significance of the impact estimate.

NOTE: Abbreviations are

 CR: Child Report

 PCR: Primary Caregiver Report

 TRS: Teacher Report on Student

 ADHD: Attention deficit hyperactivity disorder

The +/- signs in parentheses indicate the direction of a beneficial outcome. No findings were found statistically significant below the .05 level. All impact estimates were calculated 

using regression models, where each program and school within a program was weighted equally. The standard errors of all estimates account for design effects due to unequal 

weighting and the clustering of students within schools. 

SOURCE: The Social and Character Development (SACD) Research Program.

B-63

Appendix B: Technical Notes