1



# S UPPORTING	S TA

TEMENT	B: REQUEST	FOR	CLEARANCE	OF	INFORMA TION COLLECTION	FORMS	FOR “Understanding	the	Impact	of	Providing Information	to	Parents	about	the	Role	of	Algebra

II:	An	Opportunistic	Study” January 2014

**Submitted to: Submitted by**  : U.S. Department of Education SEDL Institute of Education Sciences 4700 Mueller Blvd. 555 New Jersey

Ave. NW, Rm. 308 Austin, TX 78723 Washington, DC 20208 Phone: (800) 476 - 6861

4700 Mueller Blvd. Austin, TX 78723 800 -476 - 6861 www.relsouthwest.org This publication was prepared for the Institute of Education Sciences (IES) under contract ED -IES - 12 -C -00012 by Regional Educational Laboratory Southwest, administered by SEDL. The content of the publication does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government. The publication is in the public domain. Authorization to reproduce in whole or in part for educational purposes is granted.2



## Contents 1. Respondent Universe and Sampling Methods ................................ ................................ ........... 6 2. Description of Procedures for the Collection of Information ................................ ..................... 8 Data Sources ................................ ................................ ................................ ................................ 8 3. Description of Procedures for Maximizing Response Rates ................................ ....................... 9 4. Description of Tests, Procedures, and Methods ................................ ................................ ....... 10 5. Individuals Consulted on Statistical Aspects of the Design and Key Staff ................................ 1134

The U.S. Department of Education (ED) requests clearance for data collection under the Office of Management and Budget (OMB) clearance agreement (OMB number [IES to complete]) for activities related to the Regional Educational Laboratory (REL) Program. ED, in consultation with SEDL, intends to conduct a timely, policy relevant opportunistic experiment to determine if directly providing parents/guardians, prior to students’ selection of their courses, with information on the importance of completing algebra II for college access and success has an impact on the percentage of students who enroll in and complete algebra II by the end of their junior year. In 2013, the governor of Texas signed House Bill (HB) 5 — the Foundation High School Program — into law . HB changed the high school graduation requirements and reduced the number of required state secondary school exams for public high school students in Texas . Prior to enactment of HB 5, high school students were required take four courses each in English, math ematics (including algebra II) , science, and social studies — earning the credits they needed to be admitted to most state universities and colleges. Now, only one of the three new graduation plans 1, created for the Foundation High School Program requires co mpletion of algebra II in order to graduate. Despite the change in high school graduation requirements, t he public universities in Texas will still require students to have completed algebra I, geometry, and algebra II in order to gain admission Thus, a relevant question is whether providing parents/guardians with information about the importance of completing algebra II will help prevent students who aspire to postsecondary degrees from opting to skip algebra II in high school. This study builds on resear ch being conducted in a companion study that investigates the trends in the percentage of students in a school who complete algebra II by the end of their junior year for the graduating classes of 2009 - 10 through 2016 - 17. The results of this study will provide Texas E ducation Agency (TEA) and the Texas Higher Education Coordinating Board (THECB) with information about the role that information dissemination can play in influencing students’ course selection and completion. Information from this study wil l provide TEA with valuable information that will inform their attempts to stem the number of students who wish to go on to postsecondary education, but may not be aware of the importance completing algebra II has toward this goal. The intervention is rela tively inexpensive, so if it is successful, it would demonstrate the benefit of low- cost interventions for influencing student course selection. TEA will be able to use this information to answer questions about changes in course taking patterns and course failure rates that may

1 The STEM endorsement included in the Foundation plus Endorsement Plan also requires completion of algebra II.5

arise from parents, education practitioners and administrators, policymakers, and researchers. It will also allow them to respond to inquiries regarding any variation in how districts responded to the HB 5 graduation requirements. Additionally, if the results of the study show that schools in which students’ parents were provided with information about the role of algebra II in college access and success have higher algebra II completion rates, TEA could recommend that districts con duct an information campaign similar to this one. The Texas Association for School Administrators has already expressed an interest in providing the materials to its members — all school principals in Texas. Similarly, if this study shows positive results, T EA could produce materials for districts to disseminate to parents when statewide policy changes occur. This would allow for a clear and consistent message that is currently lacking, as schools and districts must create their own materials based on their o wn interpretations of the policy. Finally, results of our study will provide evidence to other states about the impact of providing parents/guardians with information about course completion. Due to the size and demographic makeup, many states look to Te xas for guidance when making decisions about implementing statewide policies and programs. The findings of this study should be of particular interest to those states. OMB approval is being requested to recruit public high schools in Texas to participate in this study, as well as to collect extant student data records from TEA . This study will address the following research questions: **Confirmatory Research Questions** 1. Does providing information about the role of algebra II in college access and success t o the parents/guardians of students entering grade 9 after the enactment of House Bill 5 have an impact on: a. The percentage of students in a school who enroll in algebra II by the end of their junior year? b. The percentage of students in a school who complete algebra II by the end of their junior year? **Exploratory Research Questions** 2. Are there differential impacts of providing information about the role of algebra II in college access and success on students’ algebra II enrollment for:6 a. Schools with a high percentage of minority students? b. Schools with a high percentage of low- income students? c. Schools with a high percentage of low- achieving students? 3. Are there differential impacts of providing information about the role of algebra II in college access and success on students’ algebra II completion for : a. Schools with a high percentage of minority students? b. Schools with a high percentage of low- income students? c. Schools with a high percentage of low- achieving students? 4. To what extent are sc hools implementing the intervention with fidelity? D ata collection for this project consists of recruitment activities (e.g., telephone calls, email messages) and collection of extant data files from TEA . Specifically, in this OMB clearance package, ED is requesting clearance for the foll owing data collection approach :  Recruitment activities, including telephone calls and email messages, targeted at public school districts and public high schools in Texas
 Extant data collection consisting of student record s data to be obtained from the Texas Education Agency ED believes that the data collections for which clearance is being requested represent the bare minimum necessary to determine if providing parents/guardians, prior to students’ selection of their courses, with information on the importance of completing algebra II for college access and success has an impact on the percentage of students who enroll in and complete algebra II by the end of their junior year.



### 1. Respondent Universe and Sampling Methods **Respondent Universe** In this study, ED’s contractor, REL Southwest will investigate differences in algebra II enrollment and completion by the end of the junior year between students in study schools who receive information about the role of algebra II for college access (treatment group) and students in study schools who do not (control group) . To do so, ED’s contractor will follow the grade 9 students who begin high school in study schools during the 2014– 15 school year — the first cohort of students required to graduate under the state’s new diploma plans — and assess7

differences in algebra II enrollment and completion by the end of junior year between students in treatment sch ools and students in control schools. All grade 9 students, including special education and English learners who enroll in either algebra I or geometry will be included in the study. All public high schools in the state of Texas that are not automatically placing their incoming grade 9 students into the Distinguished graduation plan 2 will be recruited to participate in the study. **Power** For this study, ED’s contractor wish es to be able to detect at least a 5 percentage point difference in algebra II completion between our treatment and control groups — it is not believe d that a difference smaller than this would be meaningful. We used Optimal Design version 3.01 (Spybrook, Bloom, Congdon, Hill, Liu, Martinez & Raudenbush, 2013) t o determine the number of scho ols needed for our study . Within the software package, ED’s contractor estimated a two - level cluster RCT design with binary outcomes 3, with the following assumptions: Probability of success in treatment group: 0.70 Probability of success in control group: 0.65 Number of students in each school: 170 (median number of students in grade 9 in Texas public high schools) Plausible value range: 0.60 — 0.90 Power: 0.80 Thus, for this study, ED’s contractor determined that it will need to include a total of 144 schoo ls in the RCT, with 50 percent of the schools allocated to the treatment and control groups, to be able to detect a 5 percentage point difference between treatment and control schools 4

. **Random Assignment** Once schools have been consented into the study, ED’s contractor will use cluster analysis to assign schools to strata for randomization. Cluster analysis allows for creation clusters of schools that are similar across multiple variables — rather than just o ne or two — which should increase precision. Conducting random assignment within strata reduces the likelihood of obtaining treatment and control groups that differ significantly on covariates of interest. For this study, we will use k- means clustering, whic h creates k strata and then assigns each unit to

2 The Distinguished graduation plan requires students to complete algebra II in order to earn a Distinguished Level of Achievement. 3 The Optimal Design software does not allow for inclusion of estimates of the amount of variation taken into account by either level - 1 or level - 2 variables. As such, there is no ICC estimate, and the power for this study is likely an underestimate. 4 A difference of less than 5 percentage points between treatment and control groups is not meaningful.8

one stratum, so that a measure of similarity is maximized. Since the covariates are both categorical and continuous, how close the units are to one another will be determined using the general similarity mea sure (distance) outlined in Glower (1971). To determine the optimal number of strata, ED’s contractor will generate and evaluate results for up to 20 strata and then plot the between - cluster variability in an elbow plot, which is similar to a scree plot in principal components analysis. ED’s contractor will use this plot to determine where the rate of change in the proportion of variance accounted for by each additional strata slows. There are 6 school level covariates that are likely to predict whether or not students will complete algebra II . These covariates, shown in Table 1 , will be used in the cluster analysis to define strata. **Table 1**  **. Covariates used for cluster analysis** Percent Black, White, Hispanic, Other School size Percent special education Average incoming test scores in math and reading ( that is, grade 8 ) School region ( such as suburban, rural, urban) Percent English language learners After the schools have been placed into strata, one - half of the high schools selected from each strata w ill be randomly assigned to serve in the treatment group, while the other half of the schools within each strata will serve as the control group for the impact study. 

### 2. Description of Procedures for the Collection of Information ED’s contractor will recr uit schools, manage data collection and ensure quality and timeliness. ED’s contractor proposes to collect secondary data only. This study will utilize extant student level dat a collected and archived by TEA. OMB clearance is being sought for extant data c ollection, as well as study recruitment activities. **Data Sources** To answer research questions 1 through 3 , ED’s contractor will utilize longitudinal student - and school - level data sets available from TEA . Specifically, t he study will use data from TEA’s Public Education Information Management System (PEIMS), statewide assessment files, and Texas Academic Performance Report (TAPR) files. PEIMS contains student - level data on student enrollment and demographic characteristic s, special program participation, and course completion data. TAPR contains organizational data for schools and districts. To answer research question 4, ED’s contractor will use the envelopes mailed to the contractor by participating high schools.9



### 3. Des **cription of Procedures for Maximizing Response Rates** ED is committed to obtaining complete data for this evaluation. T his impact study relies on administrative data. ED’s contractor anticipates a 95 - percent response rate from TEA on student variables in th e administrative data. That is, ED’s contractor assumes 5 percent or less of the data will be missing on key variables. ED’s contractor has a strong working relationship with representatives from TEA . All administrative data files will be reviewed for cons istency and completeness. The response rate from schools regarding the returned envelopes will be, by definition, 100 percent. Based on its prior experience with recruiting schools to participate in RCTs, ED’s contractor expects to have minimal difficulty recruiting high schools to participate in the study . ED’s contractor will utilize a two - part process to make schools aware of the project, invite them to participate, and answer questions that schools might have about the study. Recruitment wi ll be conducted via email and telephone . Around the third week of October, ED’s contractor will send a n email message to all high school principals in the state introducing the study and inviting them to participate. The emails will outline the goals of th e study, the content of the informational materials, and the school’s role in information dissemination if randomized into the treatment condition . Principals will be made aware that this study is a randomized controlled trial (RCT) and that only half of p articipating schools will receive the informational materials and be involved in information dissemination initially; however, they will also be informed that their schools will receive the materials, if desired, after the conclusion of the study — the 2017 - 18 school year. Interested principals will be directed to contact REL Southwest for additional information. A follow- up email will be sent to all non responding principals one week after the initial email , with a second follow- up e - mail sent to all non - res ponding principals one week after the first follow- up . One week after distribution of the second follow- up email, if enough schools have not agree to participate in the study, REL Southwest will begin conduct ing telephone recruiting calls. REL Southwest an ticipate s that it will be able to recruit abo ut 80 high schools via email. REL Southwest also anticipate s needing to contact approximately 150 high school principals via telephone in order to recruit an additional 64 high schools. To re cruit additional high schools, they will begin by contacting other high schools within the same districts as those that have already agreed to participate. If this does not result in a sufficient number of high schools, random numbers will be assigned to each of the high schools in Texas that have not responded to the recruitment emails, and the schools will be sorted in chronological order by these random numbers. Starting with the smallest number, high schools will be contacted via telephone and asked to participate in t he study .10

During the recruiting process, REL Southwest researchers will ensure that principals understand the nature of the study and the responsibilities of participating schools. Once principals have agreed to participate in the study, they will be aske d to sign a memorandum of understanding/consent form indicating that their school agrees to participate in the study. While ED’s contractor does not intend to provide principals or schools with incentives to participate in the study, the cost of materia ls and material dissemination will be reimburse d . As part of the recruiting process principals w ill be informed that they will be provide d with all of the materials necessary to distribute the informational brochure s . In January, p rincipals will be asked t o provide ED’s contractor with the number of current students who entered grade 9 during the 2014– 15 school year . Principals will be provide d with enough study materials for that number of students, plus a few extra. That is, ED’s contractor will send each participating school a box of postage - paid envelopes containing the informational brochures, and we will ask principals to have their administrative staff create mailing labels, affix the labels to the envelopes, and have the envelopes picked up during th e regular mail service. Schools will not need to supply any resources other than staff time. Along with these materials, schools will be provided with one postage - paid envelope addressed to ED’s contractor. The envelope will contain the name of the school from which it was sent, and schools will be asked to mail this envelope at the same time as t he informational materials. ED’s contractor will use these envelopes to determine which high schools have mailed the informational materials and when they were sent. If schools have not mailed the informational materials within one week of the mailing dead line, ED’s contractor will conduct reminder telephone calls to high school principals.



### 4. Description of Tests, Procedures, and Methods Data analyses for this study will consist of a confirmatory analysis examining the impact of providing parents/guardians with information about the role of algebra II in college access and success on algebra II course enrollment and completion by the end of junior year and a set of exploratory analyses investigating differences in the effect of providing information about the role of algebra II for college access and success on students’ mathematics course completion for various subgroups. To answer our c onfirmatory research question , we will utilize multilevel models to compare algebra II enrollment and completion rates by the end of junior year for students in schools that received information about the role of algebra II in college access and success an d students in schools that did not receive this information. To answer our exploratory research questions, we will include interaction terms into the model used to analyze our confirmatory research11

question in order to look for differential impacts of prov iding information about the role of algebra II for college access and success for schools with high minority populations and high low- income populations. We will also assess whether there are differential impacts in schools with a higher percentage of low- achieving schools . Although the specific email messages and telephone scripts that will be used in this study have not been pilot tested, they are modeled email messages and telephone scripts that ED’s contractor has used for recruitment in other similar randomized controlled trials. Only minor changes have been made to reflect the nuances of this study. The time burden estimate used here is based on experiences from these similar studies. The informational materials to be used in the study were pilot te sted with parents of students who were enrolled in grade 9 during the 2013 - 14 school year — these parents and students cannot be included in this study. Several different versions of the brochure were pilot tested were pilot tested with nine parents. These p arents were asked to review the brochure. Then, they were asked to identify which brochure was the easiest to read, easiest to understand, most eye catching, and most informative. We used feedback from these parents to create the final version of the broch ure. 

### 5. Individuals Consulted on Statistical Aspects of the Design and Key Staff The following individuals were consulted on the statistical, data collection, and analytic aspects of this study through REL Southwest’s Technical Working Group (TWG): **Dan Goldhaber, Ph.D.** Director, CALDER (National Center for Analysis of Longitudinal Data in Education Research) Vice President, American Institutes for Research (AIR) Director, Center for Education Data & Research (CEDR), University of Washington Bothell Co - Editor, Education Finance and Policy 3876 Bridge Way N,Suite 201 Seattle, WA 98103 Ph: 206- 547 - 1562 Fax: 206- 547 - 1641 E - mail: dgoldhab@uw.edu **Geoffrey Borman, Ph.D.** Professor of Education, University of Wisconsin — Madison Deputy Director of the University of Wisconsin's Predoctoral Interdisciplinary Research Training Program12

Senior Researcher, Consortium for Policy Research in Education. 348 Ed ucation Building 1000 Bascom Mall Madison, WI 53706 - 1326 Ph: 608- 263 - 3688 Fax: 608 - 265 - 3135 E - mail: gborman@education.wisc.edu **Johannes M. (Hans) Bos, Ph.D.** Vice President and Program Director, International Development, Evaluation, and Research (IDER) Program American Institutes for Research 2800 Campus Drive, Suite 200 San Mateo, CA 94403 Ph: 650- 843 - 8100 Fax: 650- 843 - 8200 E - mail: jbos@air.org **W. Steven Barnett, Ph.D.** Board of Governors Professor and Director of the National Institute for Early Education Research Rutgers University 73 Easton Avenue New Brunswick, NJ 08901 Ph: 848- 932 -4350 x23132 Fax: 732 - 932 -4360 E - mail: sbarnett@nieer.o rg