WWC Review of the Report &quot;Assessing the Effectiveness of First Step to Success: Are Short-Term Results the First Step to Long-Term Behavioral Improvements.&quot; What Works Clearinghouse Single Study Review
Author: What Works Clearinghouse (ED)
Date: 2013-09-00
Type: Reports - Evaluative
Subjects: Program Effectiveness,Program Evaluation,Behavior Modification,Student Behavior,Behavior Problems,At Risk Students,Elementary School Students,Comparative Analysis,Parent Participation,Parent Education,Learner Engagement,Academic Achievement,Screening Tests,Coaching (Performance),Grade 1,Grade 2,Grade 3,Effect Size,Intervention,Family School Relationship

The study reported in this paper examined the effects of &quot;First Step to Success (First Step),&quot; a school- and home-based program intended to improve outcomes for students with moderate to severe behavior problems who may be at risk for academic failure. The study took place in six locations across five states: San Jose, California; Tampa, Florida; Cook County, Illinois; Eugene and Springfield, Oregon; and Huntington, West Virginia. Researchers randomly assigned 48 elementary schools either to receive the &quot;First Step&quot; program or to continue implementing regular instruction. Within each of the 48 schools, researchers used teacher-administered behavioral assessments to identify students who were eligible for the study. The three students with the highest scores on a systematic screening procedure used to measure externalizing behavior were identified for inclusion in the study, and additional high-ranked students were approached if any of the top three students did not provide consent. The final analysis sample contained between 117 and 134 students in the intervention condition and between 125 and 139 students in the comparison condition, depending on the outcome. Students in the intervention condition received the &quot;First Step&quot; intervention, a 3-month program that consists of a universal screening, classroom-based behavior coaching, and an at-home parent education program. Students in the comparison group received business-as-usual services. Study authors measured the effects of &quot;First Step&quot; by comparing parent, teacher, and researcher assessments of student behavior for students in the intervention and comparison groups. Results for three measures are presented in this WWC report: (a) academic engaged time, defined as the proportion of time a student is academically involved, (b) problem behavior, and (c) academic competence. The study authors reported that &quot;First Step&quot; increased student academic engaged time, increased teacher assessment of academic competence, and had no impact on parental assessment of problem behavior. Using unimputed data provided by the authors in response to a query, the WWC determined that none of the effects from analyses that met standards were statistically significant. However, the effect size for academic engaged time was determined to be substantively important (greater than 0.25 standard deviations). The research described in this report meets WWC evidence standards with reservation. Its strength is that it is a randomized controlled study. While schools were randomized to the intervention and comparison conditions, the students who were selected to participate in the study may have differed systematically across the intervention and comparison schools. Teachers' selection of students for the study and parents' consent for the study both occurred after randomization and, therefore, teachers' selection and parental consent could have been affected by knowledge of the school's research condition. Because of these selection and consent issues, the study was reviewed as a quasi-experimental design by the WWC. The study demonstrated baseline equivalence of the analysis samples for the three outcomes presented in this WWC report and, therefore, this evidence meets WWC standards with reservations. There were seven additional outcomes that did not meet WWC standards. [The following study is the focus of this &quot;Single Study Review:&quot; Sumi, W. C., Woodbridge, M. W., Javitz, H. S., Thornton, S. P., Wagner, M., Rouspil, K., Yu, J. W., Seeley, J. R., Walker, H. M., Golly, A., Small, J. W., Feil, H. G., &amp; Severson, H. H. (2013). &quot;Assessing the effectiveness of First Step to Success: Are short-term results the first step to long-term behavioral improvements? &quot;Journal of Emotional and Behavioral Disorders,&quot; 21 (1), 66-78. Retrieved from http://ebx.sagepub.com/ content/early/2012/01/18/1063426611429571.] The following are appended: (1) Study details; (2) Outcome measures for each domain; and (3) Study findings for each domain. A glossary of terms is included. (Contains 4 endnotes.)