MONTH YEAR 

# OMB Clearance Request

Deliverable 8.2 Promoting Student Success in Algebra I PREPARED BY: American Institutes for Research ® 1000 Thomas Jefferson Street, NW, Suite 200 Washington, DC 20007
- 3835 Windwalker Corporation 1945 Old Gallows Road, Suite 600 Tysons Corner, VA 22182 

### PREPARED FOR: U.S. Department of Education Office of Elementary and Secondary Education


### February 26 , 2013 Education, Human Development, and the Workforce1



# OMB Clearance Request **Parts A and B Deliverable 8.2** 

**February 2013**

1000 Thomas Jefferson Street NW Washington, DC 20007
- 3835 202.403.5000 | TTY 877.334.3499 www.air.org Copyright © 201 3 American Institutes for Research. All rights reserved.1 

# Contents **Introduction ................................**  **................................ ................................**  **.......................... 2**  **Promoting Student Success in Algebra I ................................**  **................................ ..................**  **3** Study Overview ................................ ................................ ................................ ........................... 3 Conceptual Framework ................................ ................................ ................................ ............... 5 Case Study Research Questions ................................ ................................ ................................ .. 8 Case Study Sampling Design ................................ ................................ ................................ ..... 10 Case Study Data Collection Procedures ................................ ................................ .................... 10 Case Study Analytic Approach ................................ ................................ ................................ .. 11 **References ................................**  **................................ ................................**  **........................... 16** 2 

### Introduction The Office of Elementary and Secondary Education (OESE) of the U.S. Department of Education (ED) requests OMB clearance for the case study component of the Promoting Student Success in Algebra I (PSSA) Study. The purpose of this study is to provide actionable information and greater insight into the knowledge and resources required to enable student success in mathematics betwe en grades 6 through 9, with an emphasis on helping all students complete Algebra I by the end of the ninth grade. To this end, PSSA will conduct literature reviews as well as focus group and case study data collections to generate a series of user
-friendl y technical assistance briefs that showcase promising practices in mathematics professional development, instructional coaching, instructional practices, curricular alignment, and expanded learning opportunities/double
- dose algebra. Clearance is requested for the case study component of PSSA, including its study design, sampling strategy, data collection, and analytic approach. This submission also includes the clearance request for the case study data collection instruments. This package contains three maj or sections: (1) Promoting Student Success in Algebra I study

 Study Overview  Conceptual Framework  Case Study Research Questions  Case Study Sampling Design  Case Study Data Collection Methods  Case Study Analytic Approach (2) Supporting Statement for Paperwork Reduction Act Submission

 Justification (Part A)  Description of Statistical Methods (Part B) (3) A ppendix A – Instructional Practices Protocols Appendix B – Curricular Alignment Protocols Appendix C – Professional Development Protocols Appendix D – Instructiona l Coaching Protocols Appendix E – Expanded Learning/Double
- Does Algebra Protocols3 

### Promoting Student Success in A**lgebra I Study Overview**  The Promoting Student Success in Algebra I (PSSA) study 1 aims to provide policy
- makers and practitioners with a deeper understanding of how instructional practices, professional development, instructional coaching, curriculum alignment, and expanded learning/double
- dose algebra for struggling students can serve as possible avenues for improving student succes s in mathematics and particularly Algebra I, a critical gateway course for which student success is a strong predictor of high
- school graduation. Recent emphases on the rigor and coherence of mathematics standards for preK
- 12 students, brought about by th e Common Core State Standards in Mathematics (CCSSM) (National Governors Association Center for Best Practices [NGACBP] & Council of Chief State School Officers [CCSSO], 2010) and other mathematics initiatives (e.g., National Council for Teachers of Mathem atics, 2000), has placed increased demands on teachers for providing instruction that supports a broad definition of student proficiency in mathematics. In addition to teaching the procedures of mathematics, which is how proficiency in mathematics is often narrowly defined, teachers are now being asked to ensure students: 1) understand the concepts that underlie those procedures; and 2) demonstrate practices that are associated with deep, active learning of mathematics. This focus on concepts and practices represents a significant shift for student expectations in mathematics, especially in Algebra I courses, which have traditionally focused on algebraic manipulations. This shift — coupled with the challenges that schools and districts face in ensuring that a ll students successfully complete Algebra I — underscores the need for high
- quality information to guide educators and policymakers in making careful and thoughtful decisions about how to structure instruction and supports to promote student success in algeb ra. To this end, the PSSA study will incorporate research findings, school
- based perspectives from education practitioners, and case studies of district and school sites that are implementing exemplary initiatives in five topical areas that represent commo n leverage points for addressing student needs in mathematics. This work will make an important contribution by producing actionable information for educators and policymakers about how to promote success in Algebra I for all students while simultaneously increasing the demands on teacher effectiveness and student performance in preparation for the Common Core State Standards for Mathematics (CCSSM). The study centers around three major research questions, which are presented in Exhibit 1.

1 The contractors for this study are the America n Institutes for Research (AIR) and Windwalker Corporation.4

Exhibit 1. Prom oting Student Success in Algebra I Study Research Questions R Q1: What is the evidence to support practices or strategies in the areas of instructional practices, professional development, instructional coaching, curricular alignment, and expanded learning/ double-dose algebra for promoting student success in Algebra I?

 What features of each practice and its implementation hold promise for promoting student success in Algebra I?

 What contextual factors support or undermine the success of each practice or stra tegy? R Q2: What do district- and school - based representatives think about evidence for practices or strategies in these areas?

 Does the evidence align with practitioners’ experience with each practice or strategy?

 To what extent do school and district perspectives differ across urban and rural educational contexts? RQ3: What does exemplary implementation of each practice or strategy look like in districts or schools with demonstrated improvement in student outcomes?

 What are the “active ingredients” critical to success of the practice or strategy?

 To what extent does implementation differ across urban and rural educational contexts? To investigate these research questions, the Promoting Student Success in Algebra I (PSSA) study features the following three components: (1) **Literature Reviews:** To address RQ1, the study team is conducting comprehensive literature reviews and writing corresponding briefs on the five topical areas of interest in this study: instructional practices, professional develo pment for mathematics teachers, instructional coaching, curriculum alignment, and expanded learning/double
- dose algebra for struggling students. The objective of each literature review is to examine the potential utility of each topical area for promoting student success in Algebra I and to produce clear and concise summaries of what is known about these areas from existing literature. (2) **School -**  **Based Perspective Briefs:** To gather school
- based perspectives on the research findings identified in the literature reviews (RQ2), the study team will assemble a Technical Working Group (TWG) consisting of seven district
- level mathematics coordinators and seven Grade 6
- 9 mathematics teachers and convene these individuals for a two
- day TWG meeting. The meeting will serve as a vehicle for obtaining these policymakers’ and practitioners’ insights on how research findings identified through the literature reviews are or can be used to inform school
level policies and classroom
- level instructional practices that promote stude nt success in Algebra I. During the TWG meeting, the study team will conduct semi structured discussions with TWG members in order to provide opportunities for these individuals to interact with research findings related to the study’s five topical areas a nd make connections to practice, thereby increasing the utility and relevance of the literature review findings. Because each of the semi - structured discussions will focus on one of the five different topical areas, fewer than 10 individuals will participa te in each session and topic , thus, OMB clearance will not be requested for this component of the data collection. Following the TWG meeting, the study team will analyze and present themes from these discussions in five school
- based perspectives briefs (on e for each topical area), which will provide actionable information to educators and policymakers on using the five topical areas as avenues for enhancing student learning in mathematics.5

(3) **Case Studies:** Building on the information collected from the literat ure reviews and TWG meeting, the study team will identify and conduct in
- depth case studies of 10 sites that are implementing exemplary programs or initiatives in one of the study’s five topical areas of interest (two sites for each topical area) (RQ3). The goals of the case studies are to (1) document how successful Algebra I programs are developed; (2) detail how such programs are staffed and managed; (3) describe contextual factors important to the program’s development, implementation, and sustainability; and (4) develop user
-fri endly, technical assistance products for policy
- makers and practitioners seeking to promote student success in mathematics in grades six through nine and ensuring that students complete Algebra I by the end of ninth grade. 

### Conceptual Framework The PSSA stu dy is guided by a conceptual framework (represented visually in Exhibit 2 below), that situates the five topical areas of interest in this study — instructional practices, professional development, instructional coaching, curricular alignment, and expanded learning/double
- dose algebra — as direct and indirect pathways to student success in Algebra I. **Instructional Practices** . Among the more direct pathways to promoting student learning in algebra are the instructional strategies that teachers use to engage stu dents with algebraic content. Although there are vigorous debates in mathematics education about the merits of particular instructional approaches — e.g., inquiry- based teaching vs. direct instruction — there is widespread agreement that mathematically profici ent students possess both procedural fluency and conceptual understanding. Recent research and recommendations by national panels have highlighted the importance of learning mathematical skills and concepts simultaneously, rather than in isolation (Kilpat rick et al., 2001; NMAP, 2008; NGACBP & CCSSO, 2010). Accordingly, the PSSA study will examine instructional practices that mutually promote procedural fluency and conceptual understanding in Algebra I specifically, and in mathematics more generally. That is, it will focus on the practices that tie procedural fluency to conceptual understanding rather than practices that focus separately on procedural fluency or conceptual understanding. **Professional Development** . Instructional practices can be improved to b etter target student learning needs through the use of instructional supports such as professional development and/or instructional coaching. Broadly, the term “professional development” can be used to describe any opportunity in which a teacher can furthe r develop his or her knowledge, skills, dispositions, and/or teaching practice. Professional development can vary considerably in form (e.g., single
- day workshops, summer institutes, professional learning communities) and focus (e.g., development of conte nt knowledge, skill in implementing new instructional strategies, sharing of activities). This study will focus on professional development that supports teachers in the design and implementation of instruction that promotes student success in Algebra I. **Instructional Coaching** . Research makes clear that effective professional development must directly relate to work in the classroom and include a mechanism by which teachers receive feedback (e.g., Jerald, 2012). For this reason, many schools and district s have begun to identify ways for teachers to work with coaches. Mathematics coaching is increasingly common, particularly in elementary schools (e.g., Association of Mathematics Teacher Educators, 2010) and has been shown to have an impact on mathematics student achievement in those grades (e.g., Campbell & Malkus, 2012). Within elementary and middle schools, mathematics coaches may be responsible for a number of activities including planning for and providing professional development interventions; supp orting teacher collaboration for grade
- level planning; observing and co
-teaching with teachers; teaching model lessons for other teachers to observe; analyzing6

standardized assessment data; and teaching only mathematics within an elementary school. For th is project, instructional coaching will focus on the range of activities in which a coach, specialist, mentor, and/or teacher leaders works with other teachers to support the instructional activities (e.g., planning, implementing, and evaluating the impact of mathematics instruction) within a school or schools. **Curricular Alignment** . As school districts search for ways to raise student success rates in Algebra I, they may look to students’ preparation to enroll in Algebra I. Ideally, students would enter Al gebra I having already attained the skills and understandings needed to be successful in the course. Thus, ensuring preparation for Algebra I has implications for the mathematics curriculum from Kindergarten through Grade 8. For this study, curriculum al ignment will refer to the scope and sequence of standards for student learning that support preparation for Algebra I. Work around the curriculum alignment topical area will identify skills and understandings that support a strong preparation for Algebra I and provide recommendations for ensuring learning standards addressing those skills and understandings are sequenced in a coherent manner throughout Grades K through 8. **Expanded Learning/Double -**  **Dose Algebra** . Because Algebra I is increasingly required for graduation, districts and schools now offer a range of opportunities to better prepare (e.g., summer bridge programs that improve students’ general mathematics and pre
- algebraic knowledge) and support (e.g., double
- dose algebra, afterschool enrichment pro grams) student success in the course. Expanded learning opportunities (ELOs) can encompass a wide range of enrichment activities which, according to Redd and colleagues, “provide academic and other learning supports to young people and their families durin g or outside of schools or outside of regular operating school
- day hours (often to supplement in
- school learning)” (2012, p. 10). This definition of expanded learning options does not directly include “double
dose algebra” programs, which are typically offered during the school day by replacing an elective course with a supplemental algebra course, and are not accelerated in that they provide more time for exposure to the curricular content presented in regular algebra courses. However, due to the prevale nce of this practice as a means of supporting students who are struggling in Algebra I, the PSSA study’s expanded learning topical area will also include a focus on double
- dose algebra so that the study can shed light on research findings as well as distri ct and school experiences in leveraging this practice to improve student learning. For the case studies that are the subject of this clearance request, the PSSA study team will focus exclusively on double
- dose algebra in its investigation of the expanded l earning topical area.7 Exhibit 2. Conceptual Framework

As represented in Exhibit 2 above, these five pathways to bolstering student success in Algebra I do not operate in a vacuum but rather take place in specific school
-, district
- and state
- level contexts that influence how the pathways are developed, operationalized, and sustained over time. At the school
- level, the capacity to develop and implement strategies to improve student learning assumes many forms, from strong school leadership to teacher knowl edge and skills to structures that support collaboration. In addition, the degree of coherence across the school’s multiple strategies, their divergence from past practices, and the level of buy
- in among program stakeholders are also qualities likely to influence the depth of implementation and eventual effectiveness of new programs to support student learning in mathematics. Moreover, contextual factors such as changes in the school’s student population, turnover among staff, and tight fiscal constraints can also shape how initiatives are implemented. Beyond the school, systemic influences often also mediate the interpretation, selection, and implementation of new initiatives. At the district
- level, capacities such as administrators’ expertise in mathemati cs instruction, data systems quality, and central office staffing
- levels may shape districts’ engagement with schools and strategies around supporting student success in algebra. Moreover, histories of reform churn or unstable district leadership may contribute to a challenging environment in which to foster change. Factors related to the district’s locale in a rural or urban environment— for example, schools’ proximity to the district office and to each other, the pool of available teachers, and the district’s access to external resources and supports — can present further considerations for the types of initiatives that are adopted and how they unfold. Lastly, at the state
- level, such factors as state policies related to standards and assessments (e.g., the s tate’s involvement in the Common Core State Standards); mandates related to curriculum, instruction,8

and mathematics course
-taking requirements; and provision of professional learning supports can hold important implications for the design and operation of new mathematics supports. Given the significance of both the features of new mathematics initiatives and the contexts in which they occur, the study team’s approach to the case studies will address questions related to the structure, activities, and procedures associated with practices in the five topical areas of interest as well as questions related to factors that facilitate or inhibit the use or effectiveness of those practices. 

### Case Study Research Questions As noted above, the subject of this OMB clearance request is PSSA’s series of five topical area case studies. The case studies are designed to address five focused sets of case study research questions that are grounded in the study’s overall conceptual framework. These research questions explo re factors associated with the successful implementation of programs or initiatives in each of the five topical areas, including (a) actions taken to implement the program/initiative; (b) processes used to develop and select the program/initiative; (c) con textual factors that enable and constrain successful implementation; and (d) indicators that are used to describe the effectiveness of the program/initiative. Exhibit 3 presents the research questions that will guide the data collection and analysis for th e case studies in each of the five topical areas.9 Exhibit 3. Promoting Success in Algebra I Case Study Research Questions
 **Instructional Practices** o How do teachers prepare for and deliver instruction that promotes both conceptual understanding and procedur al fluency? o Why was this instructional approach selected and how was it developed? o What contextual factors enable and constrain the successful implementation of this type of instruction? o How to teachers adjust instruction for students who do not have the prerequisite skills for Algebra I? o What measures are used to describe the effectiveness of this type of instruction?  **Professional Development** o What types of professional development activities promote teacher knowledge, teacher practice and student success in Algebra I? o Why was this professional development program selected and how was it developed? o What factors enable and constrain the implementation of these types of professional development activities? o How does this professional develop support s tudents with limited preparation for Algebra I o What measures are used to describe the effectiveness of these types of professional development activities?  **Instructional Coaching** o How are instructional coaching programs that promote student success developed? o How are instructional coaching programs that promote student success in Algebra I implemented and supported? o How do instructional coaching programs i mpact instruction? o How does instructional coaching meet the needs of students who have limited p reparation for Algebra I? o How is the impact of instructional coaching evaluated and how does this evaluation inform program modifications ? o How are challenges associated with implementing an instructional coaching program to support student success in Algeb ra I overcome?  **Curricular Alignment** o How are curricular frameworks that are vertically aligned to support student preparation for Algebra I developed? o How are curricular frameworks that are vertically aligned to support student preparation for Algebra I implemented? o How is the impact of implementation of curricular frameworks that are vertically aligned to support student preparation in Algebra I evaluated and how does this evaluation inform change/modification? o How are challenges associated with developi ng and implementing a curricular framework that is vertically aligned to support student preparation for Algebra I overcome?  **Double -**  **Dose Algebra** o What are the components of a successful double
- dose algebra program? o What is the process by which students are required or encouraged to enroll in the double
- dose algebra program? o How do the instructional activities in double
- dose algebra enrich learning for students struggling in Algebra I? o What curricular resources and professional development do districts and schools provide for double
- dose algebra teachers? o How do districts and schools support the success of the double
- dose algebra program? o How do programs individualize their instruction and focus to students’ specific needs and challenges? o What components of these programs do students see as critical contributions to their academic success? o What challenges do schools or districts face in implementing high
- quality double
- dose algebra courses?10



### Case Study Sampling Design The proposed samples for the case stu dies are presented in Exhibit 4. A detailed discussion of the sampling design is provided in the Supporting Statement for Paperwork Reduction Act Submission, Part B of this package. Briefly, each topical area case study will feature a purposive sample of two “sites” that are identified as implementing an exemplary program or initiative in that topical area. Because the topical areas have varying implications for actors at different levels of the school system, the units of analysis — and therefore the type of “site” targeted — differs across the five areas. Case studies for the instructional practice topical area will focus on two school sites, with individual teachers as embedded units of analysis. Other topical areas will feature two district sites, given the anticipated prominence of a district
- level role in developing and implementing programs in those areas. The instructional coaching and double
- dose algebra case studies will feature two district sites as well as two schools nested within each of those distr icts to all ow the study team to explore school
- level factors in greater depth.

Exhibit 4. Proposed Samples for Case Studies in the Five Topical Areas Topical Area Sample Instructional Practices Two schools Professional Development Two districts (or schools if professional development decisions are made at the school level) Instructional Coaching Two districts and two schools nested in those districts Curricular Alignment Two districts Double
- Dose Algebra Two districts and two schools nested in those districts 

### Case Study Data Collection Procedures The data collection activities for the case studies will vary by topical area but generally include site visit interviews and focus groups, telephone interviews, observations of classrooms and teacher professional learning activities, and document data collection. Exhibit 5 below presents a summary of the data collection activities for the case studies in each of the five topical areas. A more detailed discussion of these procedures is provided in the S upporting Statement for Paperwork Reduction Act Submission, Part B section of this package.11 Exhibit 5 . Case Study Data Collection Activities Topical Area Case Study Data Collection Activities Instructional Practices District Interviews Principal Interview Math Department Chair/Teacher Leader Interview Math Teacher Interviews/Focus Groups Student Focus Group Classroom Observations Document Collection Professional Development District Interviews External Provider Interviews (If Applicable) Math Coach Interviews (If Applicable) Principal Focus Groups Math Department Chair/Teacher Leader Focus Groups Math Teacher Focus Groups Observation of Professional Development Session Document Collection Instructional Coaching District Official Interviews Mat h Coach Interviews Principal Interview Math Department Chair/Teacher Leader Interview Math Teacher Interviews Observations of Coaching Sessions Document Collection Curricular Alignment State Official Interview District Official Interviews Math Coach Interview/focus group (if applicable) Math Department Chair/Teacher Leader Focus Groups Math Teacher Focus Groups Document Collection Double Dose Algebra District Interviews Principal Interview Math Department Chair/Teacher Leader Interview Double
- Dose Al gebra Teacher Interviews/Focus Groups Algebra Teacher Interviews/Focus Groups (if applicable) Student Focus Group Classroom Observations Document Collection 

### Case Study Analytic Approach For these case studies to be of value, they must be grounded in rigorous methods of qualitative research. Experts in qualitative research methods make clear that well
- planned, systematic, and transparent qualitative data collection and analysis techniques yield reliable, transferable findings (Anfara, Brown & Mangione, 2002; Creswell, 1998). Qualitative site visit data will be analyzed through a carefully
- structured five
- step analytic process guided by the study’s research questions and conceptual framework (See Exhibit 6). The process is designed to build in reliabili ty and validity into the case study process, by both building a chain of evidence and using triangulation to identify themes (Yin, 2003). Steps one through four of the analytic process will be conducted for each case study site visit, and step five will r equire an analysis across the sites visited for each topical area. In step one, the preliminary data12

capture will occur immediately following each site visit and will require researchers to enter information into a data capture template. During step two, site visit interview and focus group data will be coded by researchers using a code book and a qualitative analysis software program such as Atlas.Ti, NVivo, or HyperResearch. In step three, within
- site data will be analyzed across interviews, focus group s, and observations (where applicable) within a single site to triangulate data across respondents and data sources. In step four, a site narrative guided by a common outline template will combine analysis from steps 1 - 3 and will also include context
- spec ific observations that clarify critical school, district and state (where applicable) dynamics. Finally, in step five, site narratives will be analyzed collectively using rubrics to determine cross
- cutting themes.

Exhibit 6: Overview of Five-Step Qualitat ive Analysis Process

**Reliability and Validity Strategies** . Research suggest s there are several strategies that, if embedded into qualitative data collection and analysi s processes , can improve the reliability and validity of data analysis (e.g., Yin, 1992, 2003). Reliability relates to the replicability of analyses. One of the standards by which the rigor of a quantitative analysis is measured is that of replicability: that is, with access to the d ata, another researcher could replicate analyses and come to the same conclusion. There is a corresponding concept for qualitative research — that of an “audit trail.” An audit trail is documentation that creates a chain of evidence that may be inspected b y other researchers who seek to reproduce the findings (see Lincoln & Guba, 1985). In the analytic steps for these case studies , the research team will record decision rules, provide supporting evidence for all steps of the analytic process, and document a ll aspects of analysis such that an external researcher could verify the conclusions, if he or she were to have access to the data. Internal validity concerns conducting the study in a way such that inferences from the data are accurate. W hen writing the c ase study reports, the primary intent is to be descriptive, limiting inferences. For example, when documenting implementation, the study team will frame findings in such a way that makes it clear that they are synthesizing stakeholder reports. Even when reporting on an a ssociation between variables, the researchers will seek to ground such findings in stakeholders reports, for example, “Administrators and teachers reported that implementation was more challenging in classes with a high proportion of Englis h learners.” Consistency of reports across multiple stakeholders and through different modes of data collection (often referred to as “triangulation” of data) is another strategy for bolstering the internal validity of a study. The following is a more in
- depth discussion of each of the five steps of the analytic process, including measures to promote validity and reliability. 

### 1. **Preliminary Data Capture** In order to capture initial impressions about case study districts and schools, which can sometimes be lost in the time that elapses between site visits and recording field notes, the study team has designed a data capture field workbook that will be aligned with the key dimensions and qualities identified in the study’s conceptual framework. This workbook wil l allow field researchers to catalog responses to interviews and Preliminary data capture Interview and focus

group coding Within -site analysis Site narrative Cross -site analysis13

focus groups immediately following each site visit. First impressions regarding case study sites’ development and implementation of programs or strategies in the topical area of interest will be recorded consistently across sites. This will ensure that all field teams are collecting data consistently and that no data sources or areas of interest are overlooked. As a step that bridges data collection and data analysis, the data capture workbook will also allow the research team to identify emerging themes and to focus on these as the study progresses. Use of this capture template will be on
-going, and more detailed analysis of this data during step three through five will produce quantified ana lysis on key indicators both within and across sites. Researchers will be trained in the use of the data capture field workbook and will participate in regular meetings to discuss site
-visits and issues related to the use of the data work book. Additiona lly, two lead researchers will review the field workbooks to identify possible gaps in data collection and to inform additional guidance provided during these meetings. This review will ensure consistency in reporting and analysis across sites. 

### 2. **Coding of Site Visit Interview**  **and Focus Group**  **Data** One important way in which qualitative researchers seek to establish the reliability of analyses is through a rigorous coding process. Site visit interview s and focus group s will be audio
- recorded, transcribed, re viewed for accuracy, and then coded using a qualitative analysis software package such as NVivo . Qualitative analysis software programs facilitate the analysis of qualitative data by enabling researchers to systematically review data using codes that are assigned to specific portions of transcripts and field notes. It also allows the research team to organize and categorize data within
- site or across sites. To guide the coding process, a coding book organized by the constructs and domains identified in t he study’s conceptual framework and research questions will be developed. The coding book will detail how each code should be applied and include examples from the data that illustrate each code. Grounding the codes in definitions that are supported by t he research and anchored by behavioral descriptions will also support the contstruct validity of the coding process. Researchers will be trained to use the coding book and qualitative analysis software. To improve the reliability of coding guidance, the p rocess will be piloted, allowing the research team to gain consensus about the meaning of specific codes and thereby improve the reliability and consistency of the coding process. Each transcript will be coded by two analysts, and the inter
- rater reliability will be measured for all coding tasks. In addition, r egular meetings of the researchers analyzing the data will help improve consistency across coding, and spot
- checks of coding conducted by a lead researcher will improve inter
rater reliability. 

### 3. **Within -** **Site Analysis** After all within
- site data have been cataloged and coded using the data capture workbook and qualitative analysis software, the research team will use data displays to organize the data for each site by theme and quantify evidence across respondents. By cataloging data across respondents and data sources for each site, the study team will be able to look for themes and substantiate patterns that emerge within a particular respondent group (e.g., district administrators, principals, teachers) as well as across respondent groups from a particular site. The study team will also be able to determine the prevalence of particular themes or ideas and identify areas where respondent reports diverge. This process will strengthen the reliabil ity of the analysis by ensuring that within
- site findings are triangulated and persistent across sources. Although this study is not seeking to identify causal relationships, validity is nonetheless and important consideration. Towards this end, the study team will only report findings in a14

way that clearly grounds findings in reports that are consistent across respondents. The analytic process and sources of data collection allow for triangulation of data, which will allow researchers to verify the observ ations in the data (Stake, 2000). Last, in each step of the analysis process, procedures for ensuring inter- rater reliability measures will be implemented, and any identified discrepancies will be resolved and discussed at the team meetings (Armstrong, Gos ling, Weinman, & Marteau, 1997). 

### 4. **Site Narrative** The primary purpose of the site narrative is to develop a cohesive, comprehensive summary for each site that integrates the data from steps 1
- 3, but also includes important contextual data that would be diffi cult to measure using the above analysis tools. These narratives will be 5
- 10 page summaries of each site that convey how programs or initiatives in the topical area of interest were developed, implemented, evaluated, and sustained. Additionally, site narr atives will identify school
-, district
-, and state
- level contextual factors that are sometimes difficult to quantify using rubrics. Each site narrative will be reviewed by a lead researcher to ensure there is consistency in reporting across sites. 

### 5. **Cross -** **S ite Analysis**  The study team will conduct cross
- site analyses to identify emergent themes, associations, and processes across the two sites for each topical area. The study team will develop structured data displays (e.g., tables, case
- ordered matrices) t hat will facilitate the systematic identification of cross
- site trends and themes and ultimately highlight promising practices. Cross
- site analyses will include explorations into the implications of state
-, district
-, and school
- level contextual features — s uch as urbanicity, size, policy environments — and the reported implementation and success of the sites’ efforts to support students in learning Algebra I. Moreover, by comparing the experiences of sites within different contexts, the study team will be able to explore conditions that facilitated or constrained the sites’ efforts.1516 

### References Alliance for Excellent Education. (2007). The high cost of high school dropouts: What the nation pays for inadequate high schools . Washington, DC: Author. Retrieved from http://www.all4ed.org/files/archive/publications/HighCost.pdf Armstrong, D., Gosling., A., Weinman, J., and Marteau, T. (1997). The place of inter
- rater reliability in qualitative research: An empirical study. Sociology , 31(1) 597
- 606. Anfara , V. A., Brown, K. M., & Mangione, T. L. (2002). Qualitative analysis on stage: Making the research process more public. Educational Researcher, 31(7), 28
- 38. Association of Mathematics Teacher Educators (2010). Standards for elementary specialists: A reference for teacher credentialing and degree programs . San Diego, CA: Author. Bridgeland, J. M., DiIulio, J. J., & Morison, K. B. (2006). The silent epidemic: Perspectives of high school dropouts . Washington, DC: Civic Enterprises & Peter D. Hart Research Associates. Campbell, P. F., & Malkus, N. N. (2011). The impact of elementary mathematics coaches on student achievement. Elementary School Journal, 111(3), 430–454. Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five desig ns . Thousand Oaks, CA: Sage . Greene, J. P., & Winters, M. A. (2005). The effect of residential school choice on public high school graduation rates (Education Working Paper 9). New York: Manhattan Institute for Policy Research. Ham, S., & Walter, E. (1999) . Getting to the right algebra: The Equity 2000 Initiative in Milwaukee Public Schools . New York: MDRC. Helfand, D. (2006, January 30). A formula for failure in L.A. schools. Los Angeles Times, p. 6. Retrieved from http://www.latimes.com/news/local/la
- me - d ropout30jan30,0,1678653.story Higgins, L. (2008, May 27). Algebra I stumping high school freshmen: Class of 2011 confronts tougher state requirements. Detroit Free Press . Retrieved from http://mymassp.com/files/Algebra%20I%20stumping%20high%20school%20fres hmen.pdf Jerald, C. D. (2012). Movin’ it and improvin’ it! Using both education strategies to increase teaching effectiveness. Journal for Research in Mathematics Education , 37(1), 33
- 64. Kilpatrick, J., Swafford, J., & Findell, B. (eds.) (2001). Adding it up: Helping children learn mathematics . Washington, D.C.: National Academy Press. Levin, H., Belfield, C., Muennig, P., & Rouse, C. (2007). The public returns to public educational investments in African American males. Economics of Education Review, 26(6), 699–708. National Governors Association Center for Best Practices & Council of Chief State School Officers (2010). Common core state standards mathematics. Washington, DC.17

National Mathematics Advisory Panel (2008). Foundations for success: The fina l report of the national mathematics advisory panel . Washington, DC: U.S. Department of Education. National Council of Teachers of Mathematics (2000). Principles and standards for school mathematics . Alexandria, VA: NCTM. Oriheula, Y. R. (2006). Algebra I and other predictors of high school dropout (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses. (Accession Order No. AAI3249717). Redd, Z., Boccanfuso, C., Walker, K., Princiotta, D., Knewstub, D., & Moore, K. (2012). Expanding time for learning both inside and outside of the classroom: A review of the evidence base . Washington, DC: Child Trends. Rouse, C. E. (2005, October). Labor market consequences of an inadequate education . Paper presented at the symposium on the Social Costs of Inadequate Education, Teachers College, Columbia University, New York, NY. Silver, D., Saunders, M., & Zarate, E. (2008). What factors predict high school graduation in the Los Angeles Unified School District (California Dropout Research Project Report 14 ). Santa Barbara: University of California. Retrieved from http://www.hewlett.org/uploads/files/WhatFactorsPredict.pdf Stak e, R. (2000). Chapter 16: Case studies. In Handbook of Qualitative Research , 2nd Edition, edited by Denzin, N. and Lincoln, Y. Thousa nd Oaks, CA: Sage Publishing. Stillwell, R. (2010). Public school graduates and dropouts from the Common Core of Data: School year 2007 –08 (NCES 2010
- 341). Washington, DC: National Center for Education Statistics, Institute of Education Sciences, U.S. Depa rtment of Education. U.S. Department of Education. (1997). Mathematics equals opportunity (White paper prepared for Education Secretary Richard W. Riley). Washington, DC: Author. Retrieved from http://www.ed.gov/pubs/math/index.html Yin, R. K. (2003). Case study research, design and methods, 3rd ed. Newbury Park: Sage Publications. Yin, R.K. (1992). Evaluation: A singular craft. Paper presented at the annual meeting of the American Evaluation Association, Seattle, WA.