1



# S UPPORTING	S TA

TEMENT	B: REQUEST	FOR	CLEARANCE	OF	INFORMA TION COLLECTION	FORMS	FOR “Understanding	the	Impact	of	Providing Information	to	Parents	about	the	Role	of	Algebra

II:	An	Opportunistic	Study” January 2014

**Submitted to: Submitted by**  : U.S. Department of Education SEDL Institute of Education Sciences 4700 Mueller Blvd. 555 New Jersey

Ave. NW, Rm. 308 Austin, TX 78723 Washington, DC 20208 Phone: (800) 476 - 6861

4700 Mueller Blvd. Austin, TX 78723 800 -476 - 6861 www.relsouthwest.org This publication was prepared for the Institute of Education Sciences (IES) under contract ED -IES - 12 - C - 00012 by Regional Educational Laboratory Southwest, administered by SEDL. The content of the publication does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government. The publication is in the public domain. Authorization to reproduce in whole or in part for educational purposes is granted.2



## Contents 1. Respondent Universe and Sampling Methods ................................ ................................ .......................... 6 2. Description of Procedures for the Collection of Information ................................ ................................ ... 8 Data Sources ................................ ................................ ................................ ................................ ............. 8 3. Description of Procedures for Maximizing Response Rates ................................ ................................ ..... 9 4. Description of Tests, Procedures, and Methods ................................ ................................ ..................... 10 5. Individuals Con sulted on Statistical Aspects of the Design and Key Staff ................................ .............. 11 Attachment B - 1. School Agreement to Participate Form ................................ ................................ ........... 1334

The U.S. Department of Education (ED) requests clearance for data collection under the Office of Management and Budget (OMB) clearance agreement (OMB number [IES to complete]) for activities related to the Regional Educational Laboratory (REL) Program. ED, in consultation with SEDL, intends to conduct a timely, policy relevant opportunistic experiment to determine if directly providing parents/guardians, prior to students’ selection of their courses, with information on the importance of completing algebra II for college access and success has an impact on the percentage of students who enroll in and complete algebra II by the end of their junior year. In 2013, the governor of Texas signed House Bill (HB) 5 — the Foundation High School Program — into law . HB changed the high school graduation requirements and reduced the number of required state secondary school exams for public high school students in Texas . Prior to enactment of HB 5, high school students were required take four courses each in English, math ematics (including algebra II) , science, and social studies — earning the credits they needed to be admitted to most state universities and colleges. Now, only one of the three new graduation plans 1, created for the Foundation High School Program requires co mpletion of algebra II in order to graduate. Despite the change in high school graduation requirements, t he public universities in Texas will still require students to have completed algebra I, geometry, and algebra II in order to gain admission Thus, a relevant question is whether providing parents/guardians with information about the importance of completing algebra II will help prevent students who aspire to postsecondary degrees from opting to skip algebra II in high school. This study builds on rese arch being conducted in a companion study that investigates the trends in the percentage of students in a school who complete algebra II by the end of their junior year for the graduating classes of 2009 - 10 through 2016 - 17. The results of this study will provide Texas E ducation Agency (TEA) and the Texas Higher Education Coordinating Board (THECB) with information about the role that information dissemination can play in influencing students’ course selection and completion. Information from this study wi ll provide TEA with valuable information that will inform their attempts to stem the number of students who wish to go on to postsecondary education, but may not be aware of the importance completing algebra II has toward this goal. The intervention is rel atively inexpensive, so if it is successful, it would demonstrate the benefit of low - cost interventions for influencing student course selection. TEA will be able to use this information to answer questions about changes in course taking patterns and cours e failure rates that may

1 The STEM endorsement included in the Foundation plus Endorsement Plan also requires completion of algebra II.5

arise from parents, education practitioners and administrators, policymakers, and researchers. It will also allow them to respond to inquiries regarding any variation in how districts responded to the HB 5 graduation requirements. Additionally, if the results of the study show that schools in which students’ parents were provided with information about the role of algebra II in college access and success have higher algebra II completion rates, TEA could recommend that districts co nduct an information campaign similar to this one. The Texas Association for School Administrators has already expressed an interest in providing the materials to its members — all school principals in Texas. Similarly, if this study shows positive results, TEA could produce materials for districts to disseminate to parents when statewide policy changes occur. This would allow for a clear and consistent message that is currently lacking, as schools and districts must create their own materials based on their own interpretations of the policy. Finally, results of our study will provide evidence to other states about the impact of providing parents/guardians with information about course completion. Due to the size and demographic makeup, many states look to Texas for guidance when making decisions about implementing statewide policies and programs. The findings of this study should be of particular interest to those states. OMB approval is being requested to recruit public high schools in Texas to participat e in this study, as well as to collect extant student data records from TEA . This study will address the following research questions: **Confirmatory Research Questions** 1. Does providing information about the role of algebra II in college access and success to the parents/guardians of students entering grade 9 after the enactment of House Bill 5 have an impact on: a. The percentage of students in a school who enroll in algebra II by the end of their junior year? b. The percentage of students in a school who complete algebra II by the end of their junior year? **Exploratory Research Questions** 2. Are there differential impacts of providing information about the role of algebra II in college access and success on students’ algebra II enrollment for:6 a. Schools with a high percentage of minority students? b. Schools with a high percentage of low- income students? c. Schools with a high percentage of low- achieving students? 3. Are there differential impacts of providing information about the role of algebra II in college access and success on students’ algebra II completion for : a. Schools with a high percentage of minority students? b. Schools with a high percentage of low- income students? c. Schools with a high percentage of low- achieving students? 4. To what extent are sc hools implementing the intervention with fidelity? D ata collection for this project consists of recruitment activities (e.g., telephone calls, email messages) and collection of extant data files from TEA . Specifically, in this OMB clearance package, ED is requesting clearance for the foll owing data collection approach :  Recruitment activities, including telephone calls and email messages, targeted at public school districts and public high schools in Texas
 Extant data collection consisting of student records data to be obtained from the Texas Education Agency ED believes that the data collections for which clearance is being requested represent the bare minimum necessary to determine if providing parents/guardians, prior to students’ selection of their courses, with information on the importance of completing algebra II for college access and success has an impact on the percentage of students who enroll in and complete algebra II by the end of their junior year.



### 1. Respondent Universe and Sampling Meth **ods Respondent Universe**  In this study, ED’s contractor, REL Southwest will investigate differences in algebra II enrollment and completion by the end of the junior year between students in study schools who receive information about the role of algebra II for college access (treatment group) and students in study schools who do not (control group) . To do so, ED’s contractor will follow the grade 9 students who begin high school in study schools during the 2014– 15 school year — the first cohort of students re quired to graduate under the state’s new diploma plans — and assess7

differences in algebra II enrollment and completion by the end of junior year between students in treatment schools and students in control schools. All grade 9 students, including special education and English learners who enroll in either algebra I or ge ometry will be included in the study. All public high schools in the state of Texas that are not automatically placing their incoming grade 9 students into the Distinguished graduation plan 2 will be recruited to participate in the study. **Power** For this study, ED’s contractor wish es to be able to detect at least a 5 percentage point difference in algebra II completion between our treatment and control groups — it is not believe d that a difference smaller than this would be meaningful. We used Optimal Design version 3.01 (Spybrook, Bloom, Congdon, Hill, Liu, Martinez & Raudenbush, 2013) t o determine the number of schools needed for our study . Within the software package, ED’s contractor estimated a two - level cluster RCT design with binary outcomes 3, with the following assumptions: Probability of success in treatment group: 0.70 Probability of success in control group: 0.65 Number of students in each school: 170 (median number of students in grade 9 in Texas public high schools) Plausible value range: 0.60 — 0.90 Power: 0.80 Thus, for this study, ED’s contractor determined that it will need to include a total of 144 schools in the RCT, with 50 percent of the schools allocated to the treatment and control groups, to be able to detect a 5 percentage point difference between treatment and control schools 4

. **Random Assignment** Once schools have been consented into the study, ED’s contractor will use cluster analysis to assign schools to strata for randomization. Cluster analysis allows for creation clusters of schools that are similar across multiple variables — rather than just one or two — which should increase precision. Conducting random assignment within strata reduces the likelihood of obtaining treatment and control groups that differ significantly on cov ariates of interest. For this study, we will use k- means clustering, which creates k strata and then assigns each unit to

2 The Distinguished graduation plan requires students to complete algebra II in order to earn a Distinguished Level of Achievement. 3 The Optimal Design software does not allow for inclusion of estimates of the amount of variation taken into account by eit her level - 1 or level - 2 variables. As such, there is no ICC estimate, and the power for this study is likely an underestimate. 4 A difference of less than 5 percentage points between treatment and control groups is not meaningful.8

one stratum, so that a measure of similarity is maximized. Since the covariates are both categorical and continuous, how close the uni ts are to one another will be determined using the general similarity measure (distance) outlined in Glower (1971). To determine the optimal number of strata, ED’s contractor will generate and evaluate results for up to 20 strata and then plot the between - cluster variability in an elbow plot, which is similar to a scree plot in principal components analysis. ED’s contractor will use this plot to determine where the rate of change in the proportion of variance accounted for by each additional strata slows. T here are 6 school level covariates that are likely to predict whether or not students will complete algebra II . These covariates, shown in Table 1 , will be used in the cluster analysis to define strata. **Table 1**  **. Covariates used for cluster analysis** Percent Black, White, Hispanic, Other School size Percent special education Average incoming test scores in math and reading ( that is, grade 8 ) School region ( such as suburban, rural, urban) Percent English language learners After the schools have been place d into strata, one - half of the high schools selected from each strata will be randomly assigned to serve in the treatment group, while the other half of the schools within each strata will serve as the control group for the impact study. 

### 2. Description of **Procedures for the Collection of Information** ED’s contractor will recruit schools, manage data collection and ensure quality and timeliness. ED’s contractor proposes to collect secondary data only. This study will utilize extant student level dat a collect ed and archived by TEA. OMB clearance is being sought for extant data collection, as well as study recruitment activities. **Data Sources** To answer research questions 1 through 3 , ED’s contractor will utilize longitudinal student - and school - level data sets available from TEA . Specifically, t he study will use data from TEA’s Public Education Information Management System (PEIMS), statewide assessment files, and Texas Academic Performance Report (TAPR) files. PEIMS contains student - level data on student enrollment and demographic characteristics, special program participation, and course completion data. TAPR contains organizational data for schools and districts. To answer research question 4, ED’s contractor will use the e nvelopes mailed to the contractor by participating high schools.9



### 3. Description of Procedures for Maximizing Response Rates ED is committed to obtaining complete data for this evaluation. T his impact study relies on administrative data. ED’s contractor an ticipates a 95 - percent response rate from TEA on student variables in the administrative data. That is, ED’s contractor assumes 5 percent or less of the data will be missing on key variables. ED’s contractor has a strong working relationship with represent atives from TEA . All administrative data files will be reviewed for consistency and completeness. The response rate from schools regarding the returned envelopes will be, by definition, 100 percent. Based on its prior experience with recruiting schools t o participate in RCTs, ED’s contractor expects to have minimal difficulty recruiting high schools to participate in the study . ED’s contractor will utilize a two - part process to make schools aware of the project, invite them to participate, and answer questions that schools might have about the study. Recruitment will be conducted via email and telephone . Around the third week of October, ED’s contractor will se nd a n email message to all high school principals in the state introducing the study and inviting them to participate. The emails will outline the goals of the study, the content of the informational materials, and the school’s role in information dissemin ation if randomized into the treatment condition . Principals will be made aware that this study is a randomized controlled trial (RCT) and that only half of participating schools will receive the informational materials and be involved in information disse mination initially; however, they will also be informed that their schools will receive the materials, if desired, after the conclusion of the study — the 2017 - 18 school year. Interested principals will be directed to contact REL Southwest for additional information. A follow- up email will be sent to all non responding principals one week after the initial email , with a second follow- up e - mail sent to all non - responding principals one week after the first follow- up . One week after distribution of the second follow- up email, if enough schools have not agree to participate in the study, REL Southwest will begin conduct ing telephone recruiting calls. REL Southwest anticipate s that it will be able to recruit abo ut 80 high schools via email. REL Southwest also anticipate s needing to contact approximately 150 high school principals via telephone in order to recruit an additional 64 high schools. To re cruit additional high schools, they will begin by contacting other high schools within the same districts as those that have already agreed to participate. If this does not result in a sufficient number of high schools, random numbers will be assigned to each of the high schools in Texas that have not responded to the recruitment emails, and the schools will be sorted in chronological order by these random numbers. Starting with the smallest number, high schools will be contacted via telephone and asked to participate in the study .10

During the recruiting process, REL Southwest researchers will ensure that princi pals understand the nature of the study and the responsibilities of participating schools. Once principals have agreed to participate in the study, they will be asked to sign a memorandum of understanding/consent form (Attachment B - 1 ) indicating that their school agrees to participate in the study. While ED’s contractor does not intend to provide principals or schools with incentives to participate in the study, the cost of materials and material dissemination will be reimburse d . As part of the recruitin g process principals w ill be informed that they will be provide d with all of the materials necessary to distribute the informational brochure s . In January, p rincipals will be asked to provide ED’s contractor with the number of current students who entered grade 9 during the 2014– 15 school year . Principals will be provide d with enough study materials for that number of students, plus a few extra. That is, ED’s contractor will send each participating school a box of postage - paid envelopes containing the infor mational brochures, and we will ask principals to have their administrative staff create mailing labels, affix the labels to the envelopes, and have the envelopes picked up during the regular mail service. Schools will not need to supply any resources othe r than staff time. Along with these materials, schools will be provided with one postage - paid envelope addressed to ED’s contractor. The envelope will contain the name of the school from which it was sent, and schools will be asked to mail this envelope at the same time as the informational materials. ED’s contractor will use these envelopes to determine which high schools have mailed the informational materials and when they were sent. If schools have not mailed the informational materials within one wee k of the mailing deadline, ED’s contractor will conduct reminder telephone calls to high school principals. Schools will be reimbursed up to $150.



### 4. Description of Tests, Procedures, and Methods Data analyses for this study will consist of a confirmatory analysis examining the impact of providing parents/guardians with information about the role of algebra II in college access and success on algebra II course enrollment and completion by the end of junior year and a set of exploratory analyses investigating differences in the effect of providing information about the role of algebra II for college access and success on students’ mathematics course completion for various subgroups. To answer our c onfirmatory research question , we will utilize multilevel models to compare algebra II enrollment and completion rates by the end of junior year for students in schools that received information about the role of algebra II in college access and success and students in schools that did not receive this information. To answer our exploratory research questions, we11

will include interaction terms into the model us ed to analyze our confirmatory research question in order to look for differential impacts of providing information about the role of algebra II for college access and success for schools with high minority populations and high low- income populations. We w ill also assess whether there are differential impacts in schools with a higher percentage of low- achieving schools . Although the specific email messages and telephone scripts that will be used in this study have not been pilot tested, they are modeled em ail messages and telephone scripts that ED’s contractor has used for recruitment in other similar randomized controlled trials. Only minor changes have been made to reflect the nuances of this study. The time burden estimate used here is based on experienc es from these similar studies. The informational materials to be used in the study were pilot tested with parents of students who were enrolled in grade 9 during the 2013 - 14 school year — these parents and students cannot be included in this study. Several different versions of the brochure were pilot testedwith nine parents. These parents were asked to review the brochure. Then, they were asked to identify which brochure was the easiest to read, easiest to understand, most eye catching, and most informative. We used feedback from these parents to create the final version of the brochure. 

### 5. Individuals Consulted on Statistical Aspects of the Design and Key Staff The following individuals were consulted on the statistical, data collection, and analytic a spects of this study through REL Southwest’s Technical Working Group (TWG): **Dan Goldhaber, Ph.D.** Director, CALDER (National Center for Analysis of Longitudinal Data in Education Research) Vice President, American Institutes for Research (AIR) Director, C enter for Education Data & Research (CEDR), University of Washington Bothell Co - Editor, Education Finance and Policy 3876 Bridge Way N,Suite 201 Seattle, WA 98103 Ph: 206- 547 - 1562 Fax: 206- 547 - 1641 E - mail: dgoldhab@uw.edu **Geoffrey Borman, Ph.D.** Professor of Education, University of Wisconsin — Madison12

Deputy Director of the University of Wisconsin's Predoctoral Interdisciplinary Research Training Program Senior Researcher, Consortium for Policy Research in Education. 348 Education Building 1000 Bascom Mall Madison, WI 53706 - 1326 Ph: 608- 263 - 3688 Fax: 608 - 265 - 3135 E - mail: gborman@education.wisc.edu **Johannes M. (Hans) Bos, Ph.D.** Vice President and Program Director, International Development, Evaluation, and Research (IDER) Program American Institutes for Research 2800 Campus Drive, Suite 200 San Mateo, CA 94403 Ph: 650- 843 - 8100 Fax: 650- 843 - 8200 E - mail: jbos@air.org **W. Steven Barnett, Ph.D.** Board of Governors Professor and Director of the National Institute for Early Education Research Rutgers University 73 Easton Avenue New Brunswick, NJ 08901 Ph: 848- 932 -4350 x23132 Fax: 732 - 932 -4360 E - mail: sbarnett@nieer.org13 

### Attachment B**- 1. School Agreement to Participate Form**