MONTH YEAR

OMB Clearance Request

Deliverable 8.2

Promoting Student Success in Algebra I

PREPARED BY:

American Institutes for Research®

1000 Thomas Jefferson Street, NW, Suite 200

Washington, DC 20007-3835

Windwalker Corporation

1945 Old Gallows Road, Suite 600

Tysons Corner, VA 22182

PREPARED FOR:

U.S. Department of Education

Office of Elementary and Secondary Education

February 26, 2013

Education, Human Development, 

and the Workforce
1

OMB Clearance Request

Parts A and B

Deliverable 8.2

February 2013

1000 Thomas Jefferson Street NW

Washington, DC 20007-3835

202.403.5000 | TTY 877.334.3499

www.air.org

Copyright © 2013 American Institutes for Research. All rights reserved.
1

Contents

Introduction ..........................................................................................................................2

Promoting Student Success in Algebra I..................................................................................3

Study Overview ........................................................................................................................... 3

Conceptual Framework............................................................................................................... 5

Case Study Research Questions.................................................................................................. 8

Case Study Sampling Design ..................................................................................................... 10

Case Study Data Collection Procedures.................................................................................... 10

Case Study Analytic Approach .................................................................................................. 11

References...........................................................................................................................16
2

Introduction

The Office of Elementary and Secondary Education (OESE) of the U.S. Department of Education (ED) 

requests OMB clearance for the case study component of the Promoting Student Success in Algebra I 

(PSSA) Study. The purpose of this study is to provide actionable information and greater insight into the 

knowledge and resources required to enable student success in mathematics between grades 6 through 

9, with an emphasis on helping all students complete Algebra I by the end of the ninth grade. To this end, 

PSSA will conduct literature reviews as well as focus group and case study data collections to generate a 

series of user-friendly technical assistance briefs that showcase promising practices in mathematics 

professional development, instructional coaching, instructional practices, curricular alignment, and 

expanded learning opportunities/double-dose algebra.

Clearance is requested for the case study component of PSSA, including its study design, sampling 

strategy, data collection, and analytic approach. This submission also includes the clearance request for 

the case study data collection instruments.

This package contains three major sections:

(1) Promoting Student Success in Algebra I study

 Study Overview

 Conceptual Framework

 Case Study Research Questions

 Case Study Sampling Design

 Case Study Data Collection Methods

 Case Study Analytic Approach

(2) Supporting Statement for Paperwork Reduction Act Submission

 Justification (Part A)

 Description of Statistical Methods (Part B)

(3) Appendix A – Instructional Practices Protocols

Appendix B – Curricular Alignment Protocols

Appendix C – Professional Development Protocols

Appendix D – Instructional Coaching Protocols

Appendix E – Expanded Learning/Double-Does Algebra Protocols
3

Promoting Student Success in Algebra I

Study Overview

The Promoting Student Success in Algebra I (PSSA) study1aims to provide policy-makers and practitioners 

with a deeper understanding of how instructional practices, professional development, instructional 

coaching, curriculum alignment, and expanded learning/double-dose algebra for struggling students can 

serve as possible avenues for improving student success in mathematics and particularly Algebra I, a 

critical gateway course for which student success is a strong predictor of high-school graduation. 

Recent emphases on the rigor and coherence of mathematics standards for preK-12 students, brought 

about by the Common Core State Standards in Mathematics (CCSSM) (National Governors Association 

Center for Best Practices [NGACBP] & Council of Chief State School Officers [CCSSO], 2010) and other 

mathematics initiatives (e.g., National Council for Teachers of Mathematics, 2000), has placed increased 

demands on teachers for providing instruction that supports a broad definition of student proficiency in 

mathematics. In addition to teaching the procedures of mathematics, which is how proficiency in 

mathematics is often narrowly defined, teachers are now being asked to ensure students: 1) understand 

the concepts that underlie those procedures; and 2) demonstrate practices that are associated with deep, 

active learning of mathematics. This focus on concepts and practices represents a significant shift for 

student expectations in mathematics, especially in Algebra I courses, which have traditionally focused on 

algebraic manipulations. This shift—coupled with the challenges that schools and districts face in ensuring 

that all students successfully complete Algebra I—underscores the need for high-quality information to 

guide educators and policymakers in making careful and thoughtful decisions about how to structure 

instruction and supports to promote student success in algebra.

To this end, the PSSA study will incorporate research findings, school-based perspectives from education 

practitioners, and case studies of district and school sites that are implementing exemplary initiatives in 

five topical areas that represent common leverage points for addressing student needs in mathematics. 

This work will make an important contribution by producing actionable information for educators and 

policymakers about how to promote success in Algebra I for all students while simultaneously increasing 

the demands on teacher effectiveness and student performance in preparation for the Common Core 

State Standards for Mathematics (CCSSM). The study centers around three major research questions, 

which are presented in Exhibit 1.

 

1

The contractors for this study are the American Institutes for Research (AIR) and Windwalker Corporation.
4

Exhibit 1. Promoting Student Success in Algebra I Study Research Questions

RQ1: What is the evidence to support practices or strategies in the areas of instructional practices, professional 

development, instructional coaching, curricular alignment, and expanded learning/double-dose algebra for promoting 

student success in Algebra I?

 What features of each practice and its implementation hold promise for promoting student success in Algebra 

I?

 What contextual factors support or undermine the success of each practice or strategy?

RQ2: What do district- and school-based representatives think about evidence for practices or strategies in these 

areas?

 Does the evidence align with practitioners’ experience with each practice or strategy?

 To what extent do school and district perspectives differ across urban and rural educational contexts?

RQ3: What does exemplary implementation of each practice or strategy look like in districts or schools with 

demonstrated improvement in student outcomes?

 What are the “active ingredients” critical to success of the practice or strategy?

 To what extent does implementation differ across urban and rural educational contexts?

To investigate these research questions, the Promoting Student Success in Algebra I (PSSA) study features 

the following three components:

(1) Literature Reviews: To address RQ1, the study team is conducting comprehensive literature 

reviews and writing corresponding briefs on the five topical areas of interest in this study: 

instructional practices, professional development for mathematics teachers, instructional 

coaching, curriculum alignment, and expanded learning/double-dose algebra for struggling 

students. The objective of each literature review is to examine the potential utility of each topical 

area for promoting student success in Algebra I and to produce clear and concise summaries of 

what is known about these areas from existing literature.

(2) School-Based Perspective Briefs: To gather school-based perspectives on the research findings 

identified in the literature reviews (RQ2), the study team will assemble a Technical Working Group 

(TWG) consisting of seven district-level mathematics coordinators and seven Grade 6-9 

mathematics teachers and convene these individuals for a two-day TWG meeting. The meeting 

will serve as a vehicle for obtaining these policymakers’ and practitioners’ insights on how 

research findings identified through the literature reviews are or can be used to inform schoollevel policies and classroom-level instructional practices that promote student success in Algebra 

I. During the TWG meeting, the study team will conduct semi structured discussions with TWG 

members in order to provide opportunities for these individuals to interact with research findings 

related to the study’s five topical areas and make connections to practice, thereby increasing the 

utility and relevance of the literature review findings. Because each of the semi-structured 

discussions will focus on one of the five different topical areas, fewer than 10 individuals will 

participate in each session and topic, thus, OMB clearance will not be requested for this 

component of the data collection. Following the TWG meeting, the study team will analyze and 

present themes from these discussions in five school-based perspectives briefs (one for each 

topical area), which will provide actionable information to educators and policymakers on using 

the five topical areas as avenues for enhancing student learning in mathematics.
5

(3) Case Studies: Building on the information collected from the literature reviews and TWG meeting, 

the study team will identify and conduct in-depth case studies of 10 sites that are implementing 

exemplary programs or initiatives in one of the study’s five topical areas of interest (two sites for 

each topical area) (RQ3). The goals of the case studies are to (1) document how successful Algebra 

I programs are developed; (2) detail how such programs are staffed and managed; (3) describe 

contextual factors important to the program’s development, implementation, and sustainability; 

and (4) develop user-friendly, technical assistance products for policy-makers and practitioners 

seeking to promote student success in mathematics in grades six through nine and ensuring that 

students complete Algebra I by the end of ninth grade.

Conceptual Framework

The PSSA study is guided by a conceptual framework (represented visually in Exhibit 2 below), that 

situates the five topical areas of interest in this study— instructional practices, professional development, 

instructional coaching, curricular alignment, and expanded learning/double-dose algebra—as direct and 

indirect pathways to student success in Algebra I. 

Instructional Practices. Among the more direct pathways to promoting student learning in algebra are the 

instructional strategies that teachers use to engage students with algebraic content. Although there are 

vigorous debates in mathematics education about the merits of particular instructional approaches—e.g., 

inquiry-based teaching vs. direct instruction—there is widespread agreement that mathematically 

proficient students possess both procedural fluency and conceptual understanding. Recent research and 

recommendations by national panels have highlighted the importance of learning mathematical skills and 

concepts simultaneously, rather than in isolation (Kilpatrick et al., 2001; NMAP, 2008; NGACBP & CCSSO, 

2010). Accordingly, the PSSA study will examine instructional practices that mutually promote procedural 

fluency and conceptual understanding in Algebra I specifically, and in mathematics more generally. That 

is, it will focus on the practices that tie procedural fluency to conceptual understanding rather than 

practices that focus separately on procedural fluency or conceptual understanding.

Professional Development. Instructional practices can be improved to better target student learning 

needs through the use of instructional supports such as professional development and/or instructional 

coaching. Broadly, the term “professional development” can be used to describe any opportunity in which 

a teacher can further develop his or her knowledge, skills, dispositions, and/or teaching practice. 

Professional development can vary considerably in form (e.g., single-day workshops, summer institutes, 

professional learning communities) and focus (e.g., development of content knowledge, skill in 

implementing new instructional strategies, sharing of activities). This study will focus on professional 

development that supports teachers in the design and implementation of instruction that promotes 

student success in Algebra I.

Instructional Coaching. Research makes clear that effective professional development must directly relate 

to work in the classroom and include a mechanism by which teachers receive feedback (e.g., Jerald, 2012). 

For this reason, many schools and districts have begun to identify ways for teachers to work with coaches. 

Mathematics coaching is increasingly common, particularly in elementary schools (e.g., Association of 

Mathematics Teacher Educators, 2010) and has been shown to have an impact on mathematics student 

achievement in those grades (e.g., Campbell & Malkus, 2012). Within elementary and middle schools, 

mathematics coaches may be responsible for a number of activities including planning for and providing 

professional development interventions; supporting teacher collaboration for grade-level planning; 

observing and co-teaching with teachers; teaching model lessons for other teachers to observe; analyzing 
6

standardized assessment data; and teaching only mathematics within an elementary school. For this 

project, instructional coaching will focus on the range of activities in which a coach, specialist, mentor, 

and/or teacher leaders works with other teachers to support the instructional activities (e.g., planning, 

implementing, and evaluating the impact of mathematics instruction) within a school or schools.

Curricular Alignment. As school districts search for ways to raise student success rates in Algebra I, they 

may look to students’ preparation to enroll in Algebra I. Ideally, students would enter Algebra I having 

already attained the skills and understandings needed to be successful in the course. Thus, ensuring 

preparation for Algebra I has implications for the mathematics curriculum from Kindergarten through 

Grade 8. For this study, curriculum alignment will refer to the scope and sequence of standards for 

student learning that support preparation for Algebra I. Work around the curriculum alignment topical 

area will identify skills and understandings that support a strong preparation for Algebra I and provide 

recommendations for ensuring learning standards addressing those skills and understandings are 

sequenced in a coherent manner throughout Grades K through 8.

Expanded Learning/Double-Dose Algebra. Because Algebra I is increasingly required for graduation, 

districts and schools now offer a range of opportunities to better prepare (e.g., summer bridge programs 

that improve students’ general mathematics and pre-algebraic knowledge) and support (e.g., double-dose 

algebra, afterschool enrichment programs) student success in the course. Expanded learning 

opportunities (ELOs) can encompass a wide range of enrichment activities which, according to Redd and 

colleagues, “provide academic and other learning supports to young people and their families during or 

outside of schools or outside of regular operating school-day hours (often to supplement in-school 

learning)” (2012, p. 10). This definition of expanded learning options does not directly include “doubledose algebra” programs, which are typically offered during the school day by replacing an elective course 

with a supplemental algebra course, and are not accelerated in that they provide more time for exposure 

to the curricular content presented in regular algebra courses. However, due to the prevalence of this 

practice as a means of supporting students who are struggling in Algebra I, the PSSA study’s expanded 

learning topical area will also include a focus on double-dose algebra so that the study can shed light on 

research findings as well as district and school experiences in leveraging this practice to improve student 

learning. For the case studies that are the subject of this clearance request, the PSSA study team will focus 

exclusively on double-dose algebra in its investigation of the expanded learning topical area.
7

Exhibit 2. Conceptual Framework

As represented in Exhibit 2 above, these five pathways to bolstering student success in Algebra I do not 

operate in a vacuum but rather take place in specific school-, district- and state-level contexts that 

influence how the pathways are developed, operationalized, and sustained over time. At the school-level, 

the capacity to develop and implement strategies to improve student learning assumes many forms, from 

strong school leadership to teacher knowledge and skills to structures that support collaboration. In 

addition, the degree of coherence across the school’s multiple strategies, their divergence from past 

practices, and the level of buy-in among program stakeholders are also qualities likely to influence the 

depth of implementation and eventual effectiveness of new programs to support student learning in 

mathematics. Moreover, contextual factors such as changes in the school’s student population, turnover 

among staff, and tight fiscal constraints can also shape how initiatives are implemented. 

Beyond the school, systemic influences often also mediate the interpretation, selection, and 

implementation of new initiatives. At the district-level, capacities such as administrators’ expertise in 

mathematics instruction, data systems quality, and central office staffing-levels may shape districts’ 

engagement with schools and strategies around supporting student success in algebra. Moreover, 

histories of reform churn or unstable district leadership may contribute to a challenging environment in 

which to foster change. Factors related to the district’s locale in a rural or urban environment—for 

example, schools’ proximity to the district office and to each other, the pool of available teachers, and the 

district’s access to external resources and supports—can present further considerations for the types of 

initiatives that are adopted and how they unfold.

Lastly, at the state-level, such factors as state policies related to standards and assessments (e.g., the 

state’s involvement in the Common Core State Standards); mandates related to curriculum, instruction, 
8

and mathematics course-taking requirements; and provision of professional learning supports can hold 

important implications for the design and operation of new mathematics supports.

Given the significance of both the features of new mathematics initiatives and the contexts in which they 

occur, the study team’s approach to the case studies will address questions related to the structure, 

activities, and procedures associated with practices in the five topical areas of interest as well as 

questions related to factors that facilitate or inhibit the use or effectiveness of those practices.

Case Study Research Questions

As noted above, the subject of this OMB clearance request is PSSA’s series of five topical area case 

studies. The case studies are designed to address five focused sets of case study research questions that 

are grounded in the study’s overall conceptual framework. These research questions explore factors 

associated with the successful implementation of programs or initiatives in each of the five topical areas, 

including (a) actions taken to implement the program/initiative; (b) processes used to develop and select 

the program/initiative; (c) contextual factors that enable and constrain successful implementation; and 

(d) indicators that are used to describe the effectiveness of the program/initiative.

Exhibit 3 presents the research questions that will guide the data collection and analysis for the case 

studies in each of the five topical areas.
9

Exhibit 3. Promoting Success in Algebra I Case Study Research Questions

 Instructional Practices

o How do teachers prepare for and deliver instruction that promotes both conceptual understanding and 

procedural fluency?

o Why was this instructional approach selected and how was it developed? 

o What contextual factors enable and constrain the successful implementation of this type of instruction?

o How to teachers adjust instruction for students who do not have the prerequisite skills for Algebra I?

o What measures are used to describe the effectiveness of this type of instruction?

 Professional Development 

o What types of professional development activities promote teacher knowledge, teacher practice and student 

success in Algebra I?

o Why was this professional development program selected and how was it developed? 

o What factors enable and constrain the implementation of these types of professional development activities?

o How does this professional develop support students with limited preparation for Algebra I

o What measures are used to describe the effectiveness of these types of professional development activities?

 Instructional Coaching 

o How are instructional coaching programs that promote student success developed?

o How are instructional coaching programs that promote student success in Algebra I implemented and 

supported?

o How do instructional coaching programs impact instruction?

o How does instructional coaching meet the needs of students who have limited preparation for Algebra I?

o How is the impact of instructional coaching evaluated and how does this evaluation inform program 

modifications?

o How are challenges associated with implementing an instructional coaching program to support student 

success in Algebra I overcome?

 Curricular Alignment 

o How are curricular frameworks that are vertically aligned to support student preparation for Algebra I 

developed?

o How are curricular frameworks that are vertically aligned to support student preparation for Algebra I 

implemented?

o How is the impact of implementation of curricular frameworks that are vertically aligned to support student 

preparation in Algebra I evaluated and how does this evaluation inform change/modification?

o How are challenges associated with developing and implementing a curricular framework that is vertically 

aligned to support student preparation for Algebra I overcome?

 Double-Dose Algebra

o What are the components of a successful double-dose algebra program?

o What is the process by which students are required or encouraged to enroll in the double-dose algebra 

program?

o How do the instructional activities in double-dose algebra enrich learning for students struggling in Algebra I?

o What curricular resources and professional development do districts and schools provide for double-dose 

algebra teachers?

o How do districts and schools support the success of the double-dose algebra program?

o How do programs individualize their instruction and focus to students’ specific needs and challenges?

o What components of these programs do students see as critical contributions to their academic success?

o What challenges do schools or districts face in implementing high-quality double-dose algebra courses?
10

Case Study Sampling Design

The proposed samples for the case studies are presented in Exhibit 4. A detailed discussion of the 

sampling design is provided in the Supporting Statement for Paperwork Reduction Act Submission, Part B 

of this package.

Briefly, each topical area case study will feature a purposive sample of two “sites” that are identified as 

implementing an exemplary program or initiative in that topical area. Because the topical areas have 

varying implications for actors at different levels of the school system, the units of analysis—and 

therefore the type of “site” targeted—differs across the five areas. Case studies for the instructional 

practice topical area will focus on two school sites, with individual teachers as embedded units of analysis. 

Other topical areas will feature two district sites, given the anticipated prominence of a district-level role 

in developing and implementing programs in those areas. The instructional coaching and double-dose 

algebra case studies will feature two district sites as well as two schools nested within each of those 

districts to allow the study team to explore school-level factors in greater depth. 

Exhibit 4. Proposed Samples for Case Studies in the Five Topical Areas

Topical Area Sample

Instructional Practices Two schools

Professional Development Two districts (or schools if professional development decisions are made at 

the school level)

Instructional Coaching Two districts and two schools nested in those districts

Curricular Alignment Two districts

Double-Dose Algebra Two districts and two schools nested in those districts

Case Study Data Collection Procedures

The data collection activities for the case studies will vary by topical area but generally include site visit 

interviews and focus groups, telephone interviews, observations of classrooms and teacher professional 

learning activities, and document data collection. Exhibit 5 below presents a summary of the data 

collection activities for the case studies in each of the five topical areas. A more detailed discussion of 

these procedures is provided in the Supporting Statement for Paperwork Reduction Act Submission, Part 

B section of this package. 
11

Exhibit 5. Case Study Data Collection Activities

Topical Area Case Study Data Collection Activities

Instructional Practices District Interviews

Principal Interview

Math Department Chair/Teacher Leader Interview

Math Teacher Interviews/Focus Groups

Student Focus Group

Classroom Observations

Document Collection

Professional Development District Interviews

External Provider Interviews (If Applicable)

Math Coach Interviews (If Applicable)

Principal Focus Groups

Math Department Chair/Teacher Leader Focus Groups

Math Teacher Focus Groups

Observation of Professional Development Session

Document Collection

Instructional Coaching District Official Interviews

Math Coach Interviews

Principal Interview

Math Department Chair/Teacher Leader Interview

Math Teacher Interviews

Observations of Coaching Sessions

Document Collection

Curricular Alignment State Official Interview

District Official Interviews

Math Coach Interview/focus group (if applicable)

Math Department Chair/Teacher Leader Focus Groups

Math Teacher Focus Groups

Document Collection

Double Dose Algebra District Interviews

Principal Interview

Math Department Chair/Teacher Leader Interview

Double-Dose Algebra Teacher Interviews/Focus Groups

Algebra Teacher Interviews/Focus Groups (if applicable)

Student Focus Group

Classroom Observations

Document Collection

Case Study Analytic Approach

For these case studies to be of value, they must be grounded in rigorous methods of qualitative research. 

Experts in qualitative research methods make clear that well-planned, systematic, and transparent 

qualitative data collection and analysis techniques yield reliable, transferable findings (Anfara, Brown & 

Mangione, 2002; Creswell, 1998). Qualitative site visit data will be analyzed through a carefully-structured 

five-step analytic process guided by the study’s research questions and conceptual framework (See Exhibit 

6). The process is designed to build in reliability and validity into the case study process, by both building 

a chain of evidence and using triangulation to identify themes (Yin, 2003). 

Steps one through four of the analytic process will be conducted for each case study site visit, and step 

five will require an analysis across the sites visited for each topical area. In step one, the preliminary data 
12

capture will occur immediately following each site visit and will require researchers to enter information 

into a data capture template. During step two, site visit interview and focus group data will be coded by 

researchers using a code book and a qualitative analysis software program such as Atlas.Ti, NVivo, or 

HyperResearch. In step three, within-site data will be analyzed across interviews, focus groups, and 

observations (where applicable) within a single site to triangulate data across respondents and data 

sources. In step four, a site narrative guided by a common outline template will combine analysis from 

steps 1-3 and will also include context-specific observations that clarify critical school, district and state 

(where applicable) dynamics. Finally, in step five, site narratives will be analyzed collectively using rubrics 

to determine cross-cutting themes. 

Exhibit 6: Overview of Five-Step Qualitative Analysis Process

Reliability and Validity Strategies. Research suggests there are several strategies that, if embedded into 

qualitative data collection and analysis processes, can improve the reliability and validity of data analysis 

(e.g., Yin, 1992, 2003). Reliability relates to the replicability of analyses. One of the standards by which 

the rigor of a quantitative analysis is measured is that of replicability: that is, with access to the data, 

another researcher could replicate analyses and come to the same conclusion. There is a corresponding 

concept for qualitative research—that of an “audit trail.” An audit trail is documentation that creates a 

chain of evidence that may be inspected by other researchers who seek to reproduce the findings (see 

Lincoln & Guba, 1985). In the analytic steps for these case studies, the research team will record decision 

rules, provide supporting evidence for all steps of the analytic process, and document all aspects of 

analysis such that an external researcher could verify the conclusions, if he or she were to have access to 

the data.

Internal validity concerns conducting the study in a way such that inferences from the data are accurate. 

When writing the case study reports, the primary intent is to be descriptive, limiting inferences. For 

example, when documenting implementation, the study team will frame findings in such a way that 

makes it clear that they are synthesizing stakeholder reports. Even when reporting on an association 

between variables, the researchers will seek to ground such findings in stakeholders reports, for example, 

“Administrators and teachers reported that implementation was more challenging in classes with a high 

proportion of English learners.” Consistency of reports across multiple stakeholders and through different 

modes of data collection (often referred to as “triangulation” of data) is another strategy for bolstering 

the internal validity of a study.

The following is a more in-depth discussion of each of the five steps of the analytic process, including 

measures to promote validity and reliability.

1. Preliminary Data Capture

In order to capture initial impressions about case study districts and schools, which can sometimes be lost

in the time that elapses between site visits and recording field notes, the study team has designed a data 

capture field workbook that will be aligned with the key dimensions and qualities identified in the study’s 

conceptual framework. This workbook will allow field researchers to catalog responses to interviews and 

Preliminary data 

capture

Interview and focus 

group coding

Within-site 

analysis Site narrative Cross-site analysis
13

focus groups immediately following each site visit. First impressions regarding case study sites’ 

development and implementation of programs or strategies in the topical area of interest will be 

recorded consistently across sites. This will ensure that all field teams are collecting data consistently and 

that no data sources or areas of interest are overlooked. As a step that bridges data collection and data 

analysis, the data capture workbook will also allow the research team to identify emerging themes and to 

focus on these as the study progresses. Use of this capture template will be on-going, and more detailed 

analysis of this data during step three through five will produce quantified analysis on key indicators both 

within and across sites. 

Researchers will be trained in the use of the data capture field workbook and will participate in regular 

meetings to discuss site-visits and issues related to the use of the data workbook. Additionally, two lead 

researchers will review the field workbooks to identify possible gaps in data collection and to inform 

additional guidance provided during these meetings. This review will ensure consistency in reporting and 

analysis across sites.

2. Coding of Site Visit Interview and Focus Group Data

One important way in which qualitative researchers seek to establish the reliability of analyses is through 

a rigorous coding process. Site visit interviews and focus groups will be audio-recorded, transcribed, 

reviewed for accuracy, and then coded using a qualitative analysis software package such as NVivo. 

Qualitative analysis software programs facilitate the analysis of qualitative data by enabling researchers to 

systematically review data using codes that are assigned to specific portions of transcripts and field notes. 

It also allows the research team to organize and categorize data within-site or across sites. To guide the 

coding process, a coding book organized by the constructs and domains identified in the study’s 

conceptual framework and research questions will be developed. The coding book will detail how each 

code should be applied and include examples from the data that illustrate each code. Grounding the 

codes in definitions that are supported by the research and anchored by behavioral descriptions will also 

support the contstruct validity of the coding process.

Researchers will be trained to use the coding book and qualitative analysis software. To improve the 

reliability of coding guidance, the process will be piloted, allowing the research team to gain consensus 

about the meaning of specific codes and thereby improve the reliability and consistency of the coding 

process. Each transcript will be coded by two analysts, and the inter-rater reliability will be measured for 

all coding tasks. In addition, regular meetings of the researchers analyzing the data will help improve

consistency across coding, and spot-checks of coding conducted by a lead researcher will improve interrater reliability. 

3. Within-Site Analysis

After all within-site data have been cataloged and coded using the data capture workbook and qualitative 

analysis software, the research team will use data displays to organize the data for each site by theme and 

quantify evidence across respondents. By cataloging data across respondents and data sources for each 

site, the study team will be able to look for themes and substantiate patterns that emerge within a 

particular respondent group (e.g., district administrators, principals, teachers) as well as across 

respondent groups from a particular site. The study team will also be able to determine the prevalence of 

particular themes or ideas and identify areas where respondent reports diverge. This process will 

strengthen the reliability of the analysis by ensuring that within-site findings are triangulated and 

persistent across sources. Although this study is not seeking to identify causal relationships, validity is 

nonetheless and important consideration. Towards this end, the study team will only report findings in a 
14

way that clearly grounds findings in reports that are consistent across respondents. The analytic process 

and sources of data collection allow for triangulation of data, which will allow researchers to verify the 

observations in the data (Stake, 2000). Last, in each step of the analysis process, procedures for ensuring 

inter-rater reliability measures will be implemented, and any identified discrepancies will be resolved and 

discussed at the team meetings (Armstrong, Gosling, Weinman, & Marteau, 1997).

4. Site Narrative

The primary purpose of the site narrative is to develop a cohesive, comprehensive summary for each site 

that integrates the data from steps 1-3, but also includes important contextual data that would be difficult 

to measure using the above analysis tools. These narratives will be 5-10 page summaries of each site that 

convey how programs or initiatives in the topical area of interest were developed, implemented, 

evaluated, and sustained. Additionally, site narratives will identify school-, district-, and state-level 

contextual factors that are sometimes difficult to quantify using rubrics. Each site narrative will be 

reviewed by a lead researcher to ensure there is consistency in reporting across sites. 

5. Cross-Site Analysis 

The study team will conduct cross-site analyses to identify emergent themes, associations, and processes 

across the two sites for each topical area. The study team will develop structured data displays (e.g., 

tables, case-ordered matrices) that will facilitate the systematic identification of cross-site trends and 

themes and ultimately highlight promising practices. Cross-site analyses will include explorations into the 

implications of state-, district-, and school-level contextual features—such as urbanicity, size, policy 

environments—and the reported implementation and success of the sites’ efforts to support students in 

learning Algebra I. Moreover, by comparing the experiences of sites within different contexts, the study 

team will be able to explore conditions that facilitated or constrained the sites’ efforts. 
15
16

References

Alliance for Excellent Education. (2007). The high cost of high school dropouts: What the nation pays for 

inadequate high schools. Washington, DC: Author. Retrieved from 

http://www.all4ed.org/files/archive/publications/HighCost.pdf

Armstrong, D., Gosling., A., Weinman, J., and Marteau, T. (1997). The place of inter-rater reliability in 

qualitative research: An empirical study. Sociology, 31(1) 597-606.

Anfara, V. A., Brown, K. M., & Mangione, T. L. (2002). Qualitative analysis on stage: Making the research 

process more public. Educational Researcher, 31(7), 28-38.

Association of Mathematics Teacher Educators (2010). Standards for elementary specialists: A reference for 

teacher credentialing and degree programs. San Diego, CA: Author.

Bridgeland, J. M., DiIulio, J. J., & Morison, K. B. (2006). The silent epidemic: Perspectives of high school 

dropouts. Washington, DC: Civic Enterprises & Peter D. Hart Research Associates.

Campbell, P. F., & Malkus, N. N. (2011). The impact of elementary mathematics coaches on student 

achievement. Elementary School Journal, 111(3), 430–454.

Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five designs. Thousand Oaks, 

CA: Sage.

Greene, J. P., & Winters, M. A. (2005). The effect of residential school choice on public high school graduation 

rates (Education Working Paper 9). New York: Manhattan Institute for Policy Research.

Ham, S., & Walter, E. (1999). Getting to the right algebra: The Equity 2000 Initiative in Milwaukee Public 

Schools. New York: MDRC.

Helfand, D. (2006, January 30). A formula for failure in L.A. schools. Los Angeles Times, p. 6. Retrieved from 

http://www.latimes.com/news/local/la-me-dropout30jan30,0,1678653.story

Higgins, L. (2008, May 27). Algebra I stumping high school freshmen: Class of 2011 confronts tougher state 

requirements. Detroit Free Press. Retrieved from 

http://mymassp.com/files/Algebra%20I%20stumping%20high%20school%20freshmen.pdf

Jerald, C. D. (2012). Movin’ it and improvin’ it! Using both education strategies to increase teaching 

effectiveness. Journal for Research in Mathematics Education, 37(1), 33-64.

Kilpatrick, J., Swafford, J., & Findell, B. (eds.) (2001). Adding it up: Helping children learn mathematics. 

Washington, D.C.: National Academy Press.

Levin, H., Belfield, C., Muennig, P., & Rouse, C. (2007). The public returns to public educational investments in 

African American males. Economics of Education Review, 26(6), 699–708.

National Governors Association Center for Best Practices & Council of Chief State School Officers (2010). 

Common core state standards mathematics. Washington, DC.
17

National Mathematics Advisory Panel (2008). Foundations for success: The final report of the national 

mathematics advisory panel. Washington, DC: U.S. Department of Education.

National Council of Teachers of Mathematics (2000). Principles and standards for school mathematics. 

Alexandria, VA: NCTM.

Oriheula, Y. R. (2006). Algebra I and other predictors of high school dropout (Doctoral dissertation). Retrieved 

from ProQuest Dissertations and Theses. (Accession Order No. AAI3249717).

Redd, Z., Boccanfuso, C., Walker, K., Princiotta, D., Knewstub, D., & Moore, K. (2012). Expanding time for 

learning both inside and outside of the classroom: A review of the evidence base. Washington, DC: 

Child Trends.

Rouse, C. E. (2005, October). Labor market consequences of an inadequate education. Paper presented at the 

symposium on the Social Costs of Inadequate Education, Teachers College, Columbia University, New 

York, NY.

Silver, D., Saunders, M., & Zarate, E. (2008). What factors predict high school graduation in the Los Angeles 

Unified School District (California Dropout Research Project Report 14). Santa Barbara: University of 

California. Retrieved from http://www.hewlett.org/uploads/files/WhatFactorsPredict.pdf

Stake, R. (2000). Chapter 16: Case studies. In Handbook of Qualitative Research, 2nd Edition, edited by Denzin, 

N. and Lincoln, Y. Thousand Oaks, CA: Sage Publishing.

Stillwell, R. (2010). Public school graduates and dropouts from the Common Core of Data: School year 2007–08

(NCES 2010-341). Washington, DC: National Center for Education Statistics, Institute of Education 

Sciences, U.S. Department of Education.

U.S. Department of Education. (1997). Mathematics equals opportunity (White paper prepared for Education 

Secretary Richard W. Riley). Washington, DC: Author. Retrieved from 

http://www.ed.gov/pubs/math/index.html

Yin, R. K. (2003). Case study research, design and methods, 3rd ed. Newbury Park: Sage Publications.

Yin, R.K. (1992). Evaluation: A singular craft. Paper presented at the annual meeting of the American 

Evaluation Association, Seattle, WA.