Supporting Statement for Paperwork Reduction Act Submission

Description of Statistical Methods (Part B)

1. Sampling Design

Fundamental to internal validity is ruling out extraneous factors that could compromise the 

interpretation of data. Researchers do so by seeking to isolate the factors under investigation from 

any confounding variables. From this perspective, internal validity is a concern to the sampling 

design of the proposed case studies. In selecting sites for the case study component, the study team 

will avoid sites in which the innovative Algebra I program is implemented in conjunction with 

numerous other programs. Although unlikely, tor the purposes of maintaining internal validity, the 

ideal site would be implementing a program in isolation. With this ideal in mind, the study team 

will attempt to rule out as many confounding variables as possible when identifying promising 

programs. In addition to doing so for each site, the study team will identify cross-site variables to 

avoid selecting schools or sites that are unique in some way (e.g., all magnet schools).

To generate a sample that will yield informative and varied data, the study team proposes to 

generate a purposive sample of 10 sites with appropriate variation. As described earlier, each of the 

five topics that comprise the case study component of this study has a different unit of analysis, and 

hence, a different “site” that will be the focus of data collection (see Exhibit B.1). For example, the 

primary unit of analysis for the topic of curricular alignment is the district; hence, the study will 

identify two districts to which the study team will conduct site visits.

Exhibit B.1: Unit(s) of Analysis, by Topic

Topic Primary Unit of Analysis Embedded Unit of Analysis

Professional Development District* School, and teachers within schools

Instructional Practices School Teacher

Instructional Coaching District School, and teachers within schools

Curricular Alignment District School, and teachers within schools

Double-Dose Algebra District School, and teachers within schools

*In all cases in which the district is identified as the unit of analysis, the study team may opt to focus on an 

individual school, or policy decisions related to professional development, coaching, curriculum, or double-dose 

Algebra were made at the school level.

With regard to the site selection criteria, there are two levels that will guide the identification of 

cases: (1) cross-case criteria, designed to achieve appropriate variation within the set of cases as a 

whole, and (2) topic-specific criteria, which reflect the promising and innovative practices identified 

through the literature reviews. Indeed, the development of these criteria has been informed by the 

study team’s work on the literature reviews, in which analysts have detailed the distinguishing 

features of each topic.

Cross-site criteria:

The study team will seek as much variation as possible among observable state, district, and school 

characteristics which may be associated with implementation patterns and success in Algebra I. To 
the extent possible, given limitations of a relatively small sample, the full set of 10 sites will reflect 

the following criteria:

 Criterion 1: Sites will represent exemplars of the particular topic for which they are 

identified (see specific quality criteria listed below). That is, their practices in each topic 

area will reflect those that are consistent with the literature review for each topic. 

Moreover, each site should provide some evidence of improved outcomes. While the study 

team cannot set a specific achievement threshold or percentage point gain (in none of the 

topics is the literature consistent enough to warrant this), each nominated site should 

provide documentation that shows improvement, whether in student performance on 

standardized tests, student pass rates in Algebra I, student grades in Algebra I, student 

engagement, or other measures as appropriate for each topic.

 Criterion 2: Sites will reflect geographic diversity, both with regard to regional location 

within the United States as well as urbanicity. For each topical area, the study team will 

identify two sites representative of the primary unit of analysis (districts or schools) that 

are in different regions of the United States. For each topic, one site will be in an urban 

location and the other in a rural setting.

 Criterion 3: Sites will reflect diversity of enrolled students. The study team will structure 

the case studies so that they reflect students of diverse ethnic, racial and socioeconomic 

backgrounds—recognizing that students who struggle in Algebra I are disproportionately

students of color or of low socio-economic status. Thus, for each site selected at least half of 

the students in the identified sites will be ethnically or racially diverse or at least 40 percent 

of its student population participating in the National School Lunch Program.

Topic-specific criteria:

Each of the topics has been the focus of a careful literature review by members of the study team. 

As a part of this process, analysts have identified practices associated with each that exemplify 

promising, innovative, or research-supported best practices. When identifying sites for the case 

studies, study team members will screen nominated sites to ensure they will provide evidence of 

these practices and programmatic structural features. Details associated with each are listed 

below.

 Instructional Practices: Selected sites will be schools with an established track record of

teaching algebra for conceptual understanding and procedural fluency. More specifically, 

the study will include schools that systematically promote this type of teaching for all 

algebra teachers in the building and have some evidence of improved student outcomes 

(e.g., algebra end-of-course exam scores have gone up since the school implemented this 

type of instruction). Schools may attribute success in delivering this type of instruction to 

the types of instructional materials they use (e.g., high-leverage tasks and/or curricula that 

they have purchased or developed on their own), the types of professional development 

activities they offer or have received (e.g., common lesson planning, peer observation, 

professional provided by the district or outside consultants, etc.) or other factors.
 Professional Development: Selected sites will be districts with professional development 

programs targeted at improving the teaching and learning of Algebra I or courses that 

promote algebraic thinking (e.g., programs that develop algebraic thinking in grades 6-8). 

The professional development activities may include workshops or seminars that are 

developed and delivered by district staff and/or professional learning communities that 

form within or across schools in the district. The focus of the professional development 

activities may include specific content (e.g., understanding the big ideas of algebra, linear 

relationships, etc.), pedagogical techniques (e.g., questioning strategies, use of multiple 

representations of algebraic concepts, etc.), instructional materials (e.g., how to implement 

adopted materials or other lessons and tasks focused on algebraic concepts) and 

assessments (e.g., how to implement formative assessment lessons). The district should 

have a systematic approach to implementing and supporting these types of professional 

development activities.

 Instructional Coaching: Selected sites will be districts that are implementing a mathematics 

coaching program focused on the improvement of teachers’ mathematics

instruction. Although mathematics coaches within these districts may be responsible for a 

number of activities, the majority of their time should be spent working directly with 

teachers to plan, deliver, and/or evaluate mathematics instruction. Ideally, the district 

would have established a policy for hiring/training mathematics coaches to ensure 

individuals in these roles are leaders with knowledge of mathematics content, mathematics 

teaching and learning, and teacher leadership. In addition, the district should have a 

support structure in place to provide coaches with the time and resources needed to work 

with teachers in this capacity.

 Curricular Alignment: Selected sites will be districts with an established curricular 

framework that is vertically aligned to prepare students for Algebra I. More specifically, this 

curricular framework should specify a series of standards that are vertically aligned to 

support students in the development of skills and understandings needed for Algebra 

I. These include, but are not limited to, an understanding of whole number and fraction 

operations as well as the ability to reason algebraically. The standards within the curricular 

framework should be clearly stated and organized to support a connected progression of 

skills and understandings that will ultimately prepare students for Algebra I.

 Double-Dose Algebra: Selected sites will be districts that have (1) potentially promising 

outcomes for students, in the short- and or longer-term, and (2) provide professional 

development and curricular frameworks for double-dose algebra teachers. Specifically, 

promising outcomes could be demonstrated by improvements in Algebra I course failure 

rates or grades, standardized test scores in math and reading (in any grade in high school), 

on-time graduation and/or college matriculation rates. Recent evidence demonstrates that 

double-dose algebra may have delayed effects on standardized test scores and that 

improvements may extend beyond math achievement to reading if double-dose algebra 

courses emphasize written and oral communication of mathematic concepts. In addition, 

high-quality double-dose algebra problems must go beyond extending students’ learning 

time for algebra and provide specific guidance and curricular materials for enriching and 

supporting learning for struggling students in double-dose algebra courses. These materials 
should be accompanied with appropriate professional development opportunities for 

teachers.

Site Identification Procedures:

To identify potential sites, AIR staff will contact other individuals or organizations that work directly 

with schools, districts, and states in the implementation of instructional strategies. To identify 

prospective sites, AIR will first pinpoint programs or initiatives related to each topic area. 

Specifically, the study team will draw on six sources to identify programs, including (1) the 

literature reviews associated with this project, (2) information obtained from focus group 

interviews, (3) a national call to Algebra I programs and listservs, (4) an Internet search for 

programs using key terms associated with each topic, (5) AIR’s professional contacts, and (6) 

referrals from ED. For example, the study team will reach out to organizations and researchers who 

have conducted studies related to math education. The contractors’ colleagues in the Regional 

Comprehensive Centers, Content Centers, the Regional Education Laboratories, or the California 

Collaborative will be important sources for suggesting potential sites. In addition, AIR staff have 

professional contacts in many school districts (including Boston, Chicago, San Francisco, and DC) 

and we could solicit nominations from district administrators (for school-level practices). 

AIR will enter all information about each site into a Microsoft Access database that will track each 

program/site, its characteristics, and its progression through the selection process. Upon identifying 

5–10 programs per topic area through the channels highlighted above, AIR will review any extant 

data or marketing materials available to determine the extent to which each program has 

undergone internal or external evaluation and corresponding evidence for effectiveness, and 

include this information in the database. 

The study team will first filter each nominated site against the cross-site and site-specific criteria 

described above. After identifying the set of schools that pass first screening process, the study 

team will collect additional data, primarily through extant sources, to determine if the site is likely 

to yield rich data. The identification of the final sample will be an iterative process, through which 

the team will identify potential sites, consider the features of the districts and states in which they 

are nested, and re-evaluate the candidate districts and schools. The final set of sites will be 

reviewed by ED.

As part of another aspect of this study, the contractors have identified a set of exemplar districts 

and schools that will provide feedback on the literature review documents through informal focus 

groups. These sites, which have already been screened by the study team, are likely candidates for 

at least some of the case study sites (see Exhibit B.2).

Exhibit B.2. Potential Sites for Case Studies

State District or School

Illinois Chicago Public Schools

California Elk Grove Public Schools

Maine RUS-19 School District

Maryland Howard County Schools

North Carolina Charlotte-Mecklenburg
The study team also will identify a set of 10 replacement sites in the event that some prove to be

inappropriate for inclusion in the case studies (e.g., substantial turnover in student population) or 

decline to participate.

2. Procedures for Data Collection

The data collection procedures of each of the main components of the study (site visits, principal 

interviews, teacher surveys, and state interviews) are discussed in detail below.

Data Collection Activities by Topic

Because each of the topics for which the contractors are to conduct case studies reflects a different 

purpose, practices, and implementation, the data collection activities will vary for each. The 

common elements are that each will involve interviews or focus groups at the district and school 

level, and each will involve some form of document collection. All focus groups and interviews are 

expected to require one hour, while classroom observations could range from 60 to 90 minutes, 

depending on the organization of the school day. The specific numbers and roles of interviewees 

and focus group participants are detailed below.

Instructional Practices. The purpose of the case study of instructional practices is to understand 

strategies, processes, and policies in schools that have made an effort to blend procedural and 

conceptual foci in their Algebra I instruction. Hence, the school is the unit of analysis for this topic, 

and the study team will visit two such schools. However, the contractors intend to collect data 

from the district level, to better understand that context in which each school is situated. Because 

the contractors intend to conduct classroom observations, each site visit for this topic will last two 

or three days, with two site teams, each including one site visit lead and one site visit associate.

Exhibit B.3: Data Collection Activities for Instructional Practices

Instructional Strategies. Unit of Analysis: School

Level Respondent Total n of 

respondents

District (n=2) Interview with district official with primary responsibility for mathematics 2

School (n=2) Observations of up to 5 teachers, using the Mathematical Quality of Instruction 

observation protocol

10

Interviews with each teacher who is observed 10

Focus group with remaining Algebra I teachers in the school (if there are more than 5) Variable

Principal interview 2

Interview with math department chair or Algebra team lead 2

Interview with instructional coach, if applicable 2

Student focus groups (approximately 6 students) 12

Documents Teacher assignments 10

Lesson plans 12

Schoolwide planning documents with regard to the instruction of Algebra I Variable
Curricular Alignment. The purpose of the curricular alignment case study is to describe districts that 

have developed and implemented a curricular framework that promotes the skills and 

understanding needed to be successful in Algebra I prior to enrollment in an Algebra I course. The 

unit of analysis for this topic is the school district because this is the level at which the curricular 

framework and pacing guides are established, as well as the vision with regard to Algebra I 

instruction. The study team will visit two districts in two different states. Each site visit for this topic 

will last two days, with two site visitors: one site visit lead, and one site visit associate.

Exhibit B.4: Data Collection Activities for Curricular Alignment

Curricular Alignment. Unit of Analysis: 

District

Level Respondent Total n of 

respondents

State (n=2) Interview with the state official with responsibility for mathematics curricular 

standards

2

District (n=2) Interview with the associate superintendent for curriculum and instruction; or 

superintendent in small districts

2

Interview with math leaders at the district level 2-4

Interview with the district administrator with responsibility for professional 

development for math teachers

2

School Focus group of teachers who teach classes prior to Algebra I (elementary, across 

schools)

5-6

Focus group of teachers who teach classes prior to Algebra I (middle, across 

schools)

6-8

Focus group of teachers of Algebra I (cross-schools) 6-8

Focus group of math coordinators (or department chairs, depending on school 

organizational structure) from different schools

4-5

Documents Curricular framework

variable

Pacing guides

Additional instructional resources that support these ideas and concepts in 

the early grades

Professional Development. The purpose of the professional development case study is to explore 

innovative practices in terms of professional learning opportunities designed to deepen teachers’ 

content knowledge, conceptual understanding, and instructional capacity with regard to Algebra I. 

Because professional development policies are often mapped out at the district level, the school 

district is the unit of analysis for this topic. However, it is critical that the study team collect data 

from the school and teacher level as well, to explore within-school participation in professional 

development. The study team will visit two districts in two states. Each site visit for this topic will 

last two days, with two site visitors: one site visit lead, and one site visit associate.

Exhibit B.5: Data Collection Activities for Professional Development

Professional Development. Unit of Analysis: 

District
Level Respondent Total n of 

respondents

State (n=2) Official with responsibility for mathematics professional development policy 2

District (n=2) District Math Coordinator (elementary, secondary) 4

District PD Coordinator (elementary, secondary) 4

District official with responsibility for curriculum and instruction 2

External PD provider such as university faculty, regional or county office person (if 

applicable)

2

School Focus group of teachers from secondary schools (across up to 5 schools in the 

district)

5

Focus group of principals from secondary schools(across up to 5 schools in the 

district)

5

Focus group of school mathematics coordinators (elementary) 5

Focus group of high school mathematics department chairs 5

Documents

Professional development documents (handouts, outline/agenda, PowerPoint 

presentations, district professional development plan, template for 

teacher/school professional development plans) variable

Observations Observe a professional development session (if possible)

Instructional Coaching. The purpose of this case study of instructional coaching is to understand 

how districts and schools use mathematics coaches to support teachers as they continue to 

improve their mathematics teaching. In addition to learning more about the role of a mathematics 

coach as they interact with teachers, the study seeks to learn more about the policies and 

structures that support the training, development, and activities of those coaches. The study team 

is interested in how the coaching program was developed, the supports that contributed to its 

success, and the challenges that were overcome. Hence, the district is the unit of analysis for this 

topic. The study team will conduct site visits to two districts, and will include data collection 

activities in two schools in each district. The site visits will last three days and will be conducted by 

one site visit lead and one site visit associate.

Exhibit B.6: Data Collection Activities for Instructional Coaching

Instructional Coaching. Unit of Analysis: 

District

Level Respondent Total n of 

respondents

District (n=2) Interview with District Mathematics Coordinator 2

Interview with district official responsible for math coaching 2

School (n=4) Interviews with math coaches (in each school) 4

Interviews with principals (in each school) 4

Interviews with math department chair or teacher leader (in each school) 4

Focus groups with math teachers in each school (up to 5 teachers) 20

Documents Coach training manuals

variable Coach job description/guidelines
Coaching materials (e.g., rubrics, forms used to provide feedback, forms to 

monitor teacher progress, schedules)

Double-Dose Algebra. The purpose of this case study is to explore the structures and 

implementation processes associated with instructional programs in which students receive 

additional Algebra I instruction. For example, students may participate in a traditional Algebra I 

class, but then attend a second Algebra I class that reinforces what they have learned. The district is 

the unit of analysis for this topic. The study team will conduct site visits to two districts, and will 

include data collection activities in two schools in each district. Each site visit for this topic will last 

two days, with two site visitors: one site visit lead, and one site visit associate.

Exhibit B.7: Data Collection Activities for Double-Dose Algebra

Double-Dose Algebra. Unit of Analysis: 

District

Level Respondent Total n of 

respondents

District (n=2) District Math Coordinator 2

District official who oversees curriculum and instruction 2

School (n=4) Principal interview (high school) 4

Math Department Chair/Teacher Leader interview 4

Double Dose Algebra Teacher Focus Groups (up to 5 teachers in each school) 20

Student Focus Groups (up to 8 students in each school – students selected by 

teachers as having benefited from double-dose)

32

Observations of double-dose algebra classrooms and their associated standard 

Algebra I classroom/session (1 observation of double-dose algebra and associated 

algebra class session for each of four schools)

4

Documents Example lesson plans from each double-dose algebra teacher in each school (2 

lesson plans and a “syllabus” for up to 5 teachers in each school)

variable District and school guidance on course placement policies for double-dose algebra

District and school curricular guides for double-dose algebra class sessions

Site Visits

Protocol Development

Interview and Focus Group Protocols. Two contractors, AIR and Windwalker, have developed 

protocols to guide the case study interviews and focus groups. These protocols include both 

open-ended questions and more closed-ended questions. The open-ended questions will encourage 

in-depth responses, and incorporate flexibility so that the study team can follow threads of 

conversation to their logical conclusion, while the closed-ended questions will ensure that similar 

types of data are collected across the case study schools. The study team recognizes the utility of 

including broad questions to initiate discussion, followed by focused probes to ascertain insights in 

important areas. In developing protocol questions, the study team has sought to avoid language 

that may be loaded, leading, or likely to yield socially desirable responses.
Interview and focus group protocols have been designed to fully explore the evaluation questions 

without placing undue burden on the respondents. The contractors began the protocol 

development process by creating a theory of action for each topic, which reflects the key constructs 

and contextual variables on which the study will need to collect data. Developing a theory of action 

for each topic – grounded in the relevant literature for each – helped ensure that the protocols, as a 

set, cover the necessary topics. As a result, the full set of interview protocols, focus group 

protocols, and extant data sources will generate sufficient information to address all evaluation 

questions. 

The contractors anticipate that site visitors will adapt the protocols for the context and particular 

intervention models being implemented in the site visited schools. To facilitate this process, the 

protocols include suggested probes, topics that that the interview should “listen for,” and notes to 

indicate where the interviewer should adapt the protocol to the particular respondent’s 

circumstances. Although each question contains many possible probes, the interviewer is not 

required (or expected) to ask each of these. Rather, the probes are intended to ensure the 

interviewer is attuned to the full range of possible topics and variables to be addresses in the 

responses, and – in rare cases – to prompt a less communicative interviewee.

[Note to reviewer: this paragraph reflects our intent for piloting activities, although we have not 

piloted instruments yet, we are retaining this text as a placeholder.] Following publication of this 

OMB submission for public comment, the study team conducted piloting interviews with fewer than 

10 district officials, principals, and teachers in three locations. Study team members asked these 

educators and administrators to react to (1) the overall organization, flow, and length of the 

interview, (2) the clarity of the interview wording and language, (3) specific questions that were 

unclear or difficult to answer, and (4) any recommended changes. The study team then used the 

responses from these pilots to modify the protocols to make them as suitable as possible for the 

actual site visits. 

Classroom Observation Protocol. As part of the case study of instructional practices, the study team 

will conduct a small set of classroom observations using instruments that AIR has developed and 

successfully implemented in other projects focused on math teaching and learning. For example, 

AIR has a low-inference instrument that captures the frequency with which teachers focus on 

concepts, procedures or both. We will also have one math content expert on each visit that will 

record running notes in each of the classrooms observed. These notes will be used to further 

illustrate behaviors captured in the observation protocol. 

Request for Documents. The plans for each case study topic include some collection of documents, 

as relevant for the specific topic. Because each topic requires a different set of documents, the 

study team has prepared a separate request for documents for each of the five topics. In general 

terms there are two sorts of requests: 

(1) A general request, in which the study team requests a straightforward set of documents 

that are generally readily accessible, such as a district’s curriculum framework, pacing 

guides, or a training manual.
(2) A request with detailed guidelines. For example, the request for teacher assignments 

includes specific guidelines on how many assignments are to be submitted by each teacher, 

the focus of the assignment, and the intended level of cognitive demand for the student.

Prior to conducting site visits, AIR will ensure that all data collection procedures (including informed 

consent forms, in Appendices A - E of this document) are approved by its Institutional Review Board 

(IRB).

Training and Preparation

Training for Site Visitors. The study team will train the site visit team so that each member brings to 

his or her on-site work a consistent understanding of the relevant policies, the study, and the data 

collection needs. Prior to the first wave of data collection, all site visit staff from AIR and 

Windwalker will convene in Washington DC for a one-day training session. The site visit team leads 

from AIR and Windwalker will jointly develop and conduct the training so that consistent messages 

and training content are delivered across organizations. The one-day training will cover many 

topics, including the major concepts and big ideas of algebra, evaluation questions, site visit

approach and activities, data collection on site (including brief review of interview protocols), and 

coordinating site visit work.

For the training, the site visit task leaders will develop a site visit checklist that will outline all tasks 

the site visitors need to perform before, during, and after each visit. All site visit team members will 

adhere to this checklist to ensure that visits are conducted efficiently, professionally, and 

consistently. There will be regular meetings of the site visit task leaders from AIR and Windwalker

as well as weekly email updates and periodic larger meetings of the site visit staff. 

Preparation for Site Visits. Prior to each site visit, the study team will compile and review all extant 

background information on the school. Although some document data must be collected on site, 

Web-based resources (primarily state, district and school Web sites) include important contextual 

information on the history, priorities, performance, community support, and staffing of each 

school. Staff will review these closely before each visit and will enter preliminary information in a 

data capture template.

Administration of the Site Visits

Prior to the site visit, researchers will conduct the interview with the state department of education 

official for the topics that include interviews at this level. While one researcher will ask questions of 

the respondent, the second researcher will take notes.

Two-person site visit teams will be instructed to be flexible in determining how to organize each 

school visit, being mindful of district and school context and the need to collect data that answer 

specific evaluation questions. However, the site visits will have a standard set of features, including 

interviews and focus groups with state, district, and school-level respondents; all topics will include 

the collection of document data as well.

Three of the topics include cross-school focus groups which will include five or six respondents 

across schools of a given level (elementary or secondary). To convene these focus groups, AIR and 

Windwalker will work with a district administrator to facilitate communications with school-level 
respondents, and to identify a central location with convenient access for required respondents. In 

addition, the contractors will be as flexible as possible with regard to the timing of interviews and 

focus groups, so as not to disrupt school schedules.

For four of the case study topics, the contractors have determined that two-day site visits will allow 

adequate time for interviews and focus groups with district officials and school administrators, 

teachers, coaches, and students. For one topic, the site visit activities may require a three-day site 

visit, with four staff, largely due to the time required for the classroom observations. If the 

instructional periods are one hour (as in Exhibit B.8, below) a two-day visit should be possible, but if 

class periods are 90 minutes (as is the case in high schools with block scheduling) then a three-day 

site visit may be necessary.

Exhibit B.8. Sample Site Visit Schedule: Instructional Strategies

Site Visit to Sample District: Two Teams of Two Researchers on Site

Site Visit Team #1 Site Visit Team #2

Day 1 Day 1

Travel in morning, at district central office in afternoon Travel in morning, at school in afternoon

1:00 – 2:00 Interview with District official responsible 

for math instruction

1:00 – 2:00 Observation of Teacher 1

2:00 -3:00 Interview with District official responsible for 

math coaches, if applicable

2:00 – 3:00 Interview with Teacher 1

3:00 – 4:00 Meet with district administrative staff to 

collect document data, as necessary

3:00 –4:00 Confer with administrative staff to collect 

document data

Day 2 Day 2

8:00 – 9:00 Observation of Teacher 2 8:00 – 9:00 Observation of Teacher 4

9:00 – 10:00 Interview with Teacher 2 9:00 – 10:00 Interview with Teacher 4

10:00 – 11:00 Observation of Teacher 3 10:00 -11:00 Observation of Teacher 5

11:00 – 11:30 Break 11:00 – 12:00 Interview with Teacher 5

11:30 – 12:30 Math coach interview 12:30 – 1:00 Lunch

12:30 – 1:00 Lunch 1:00 – 2:00 Teacher focus group

1:00 – 2:00 Principal interview 2:00 – 2:30 Break

2:00 – 3:00 Interview with Teacher 3 2:30 – 3:30 Student focus group

3:00 – 4:00 Math department chair interview 3:30 – 4:00 Confirm collection of teacher assignments

Each site visit team will consist of two researchers (for professional development, instructional 

coaching, curricular alignment and double-dose Algebra) or four researchers (for instructional 

strategies), from either AIR or Windwalker. Having the researchers work as a team during the site 

visits will enhance the consistency and reliability of data gathered. The same pair of researchers will 

be responsible for scheduling and conducting the visits, and both site visitors will attend all 

interviews if possible. However, site visitors may conduct interviews separately if this is necessary 

to collect information from all respondents. Because of this, each interview and focus group will be 

audio recorded. In preparation for each visit, site visitors will review notes from interviews with 

state officials, review extant data on each site, consider the reasons for which each jurisdiction was 

selected for inclusion in the sample, review other relevant documentation, and annotate each 

section of the individual interview protocols accordingly. Further, these notes will be used to guide 

the wording of each question. The use of experienced interviewers, coupled with careful 

preparation, will ensure that interviews are not “canned” or overly formal. The lead site visitor for 
each case will have ultimate responsibility for completing and submitting the specified set of 

standard deliverables on the checklist for each site visit or follow-up.

Throughout the process of data collection and reporting, the contractors will make all efforts to 

protect the privacy of respondents participating in the site visits. The study team will not identify by 

name any of the interviewees; nor will the study team attribute quotes. Although the study team

will identify the names of states in the final reporting of case studies, districts and schools will be 

identified by pseudonyms. 

Collection of Documents and Files

All of the case study topics include the collection of some document data, including schoolwide 

planning documents, professional development training guides, curricular framework, and pacing 

guides, among others. Two documents in particular – lesson plans and teacher assignments – will 

require more specific guidelines for the data collection team.

First, the study team plans to collect two sets of lesson plans from each teacher whose class is 

observed: (1) plans for the classes in which teachers are introducing a new topic, and (2) plans for 

the days during which students are preparing for a test. In addition to the plans themselves, the 

study team will collect associated slide presentations, handouts, and other supporting materials. 

Second, the study team will collect two sets of teacher assignments from each teacher whose class 

is observed. One assignment should be what the teacher considers to be a “typical” assignment, 

while the second should be a more challenging assignment. The teacher should also submit the 

answer key and details regarding how they would grade the assignments – that is, the number of 

points possible for each part as well as the way in which points would be allocated or reduced 

based on student work.

3. Methods to Maximize Response Rate

Data collection is a complicated process that requires careful planning. The research team has

developed interview protocols, focus group protocols and questionnaires that are appropriately 

tailored to the respondent group and are designed to place as little burden on respondents as 

possible. The team has also piloted core data collection instruments to ensure they are user-friendly 

and easily understandable, all of which increases participants’ willingness to participate in the data 

collection activities and thus increases response rates.

In addition to careful instrument design, a high response rate among the case study sites may be 

assured through careful recruitment materials. These recruitment materials will emphasize the 

social incentive to respondents by stressing the importance of the data collections as part of a study 

of innovative practices that will provide much-needed information to districts and schools. AIR’s 

experience in past evaluations has demonstrated the importance and value of building a consensus 

of support with participating districts. This leads to districts and schools that have the capacity, 

willingness, and commitment to cooperate fully with the research and data collection 

responsibilities. Investing in site development at the front end reduces problems in the back end, 

helping to ensure smooth implementation of the evaluation.
4. Expert Review and Piloting Procedures

[Note to reviewer: this paragraph reflects our intent for piloting activities, although we have not 

piloted instruments yet, we are retaining this text as a placeholder.] Following publication of this 

OMB submission for public comment, the study team conducted piloting interviews with district 

officials, principals, and teachers in three states. Study team members asked these educators and 

administrators to react to (1) the overall organization, flow, and length of the interview, (2) the 

clarity of the interview wording and language, (3) specific questions that were unclear or difficult to 

answer, and (4) any recommended changes. The study team then used the responses from these 

pilots to modify the protocols to make them as suitable as possible for the actual site visits.

5. Individuals and Organizations Involved in Project

AIR is the contractor for the Study of Promoting Student Success in Algebra I. The project director is 

Dr. Kirk Walters, who is supported by an experienced team of educators leading the major tasks of 

the study (see Exhibit B.9 for a list of key staff and responsibilities). 

During data collection and particularly during the initial phase of analysis, the contractors will draw 

on the cross-staffing of some key members of the study, including the project director, deputy 

project director, and team leaders, to combine findings across these data sources to create the 

synergies that are at the core of the case study design. Contact information for the individuals and 

organizations involved in the study is presented in Exhibit B.9.

Exhibit B.9. Organizations, Individuals Involved in Project

Responsibility Organization Contact Name Telephone 

Number

Project Director AIR Dr. Kirk Walters (202) 403-5838

Deputy Director AIR Dr. Nicholas Sorensen (312) 283-2318

Task Lead: TWG Meeting and 

Senior Math Consultant

AIR Dr. Toni Smith (202) 403-5926

Senior Math Content Advisor AIR Dr. Steven Leinwand (202) 403-6926

Task Lead: Case Studies AIR Dr. Kerstin Carlson Le Floch (202) 403-5649

Task Lead: Qualitative 

Analysis

AIR Andrea Boyle (650) 843-8294

Windwalker Jennifer Grossman (703) 970-3518

Windwalker Meschelle Linjean (703) 970-3554

Windwalker Lauren Malone (703) 970-3542

Windwalker Alison McNeil (703) 970-3528

Windwalker Samantha Schwartz (703) 970-3519