### Supporting Statement for Paperw**ork Reduction A ct Submission**  **D escription of Statistical Methods (Part B)**  **1. Sampling Design**  Fundamental to internal validity is ruling out extraneous factors that could compromise the interpretation of data. Researchers do so by seeking to isolate the factors under investigation from any confounding variables. From this perspective, internal vali dity is a concern to the sampling design of the proposed case studies. In selecting sites for the case study component, the study team will avoid sites in which the innovative Algebra I program is implemented in conju n ction with numerous other programs. Al though unlikely, tor the purposes of maintaining internal validity, the ideal site would be implementing a program in isolation. With this ideal in mind, t he study team will attempt to rule out as many confounding variables as possible when identifying promising programs. In additi on to doing so for each site, the study team will identify cross

-site variables to avoid select ing schools or sites that are unique in some way (e.g., all magnet schools). To generate a sample that will yield informative and vari ed data, the study team propose s to generate a purposive sample of 10 sites with appropriate variation. As described earlier, each of the five topics that comprise the case study component of this study has a different unit of analysis, and hence, a different “site” that will be the focus of data collection (see Exhibit B.1). For example, the primary unit of analysis for the topic of curricular alignment is the district; hence, the study will identify two districts to which the study team will conduct site visits. Exhibit B.1: Unit(s) of Analysis, by Topic Topic Primary Unit of Analysis Embedded Unit of Analysis Professional Development District* School, and teachers within schools Instructional Practices School Teacher Instructional Coaching District School, and teachers within schools Curricular Alignment District School, and teachers within schools Double

- Dose Algebra District School, and teachers within schools *In all cases in which the district is identified as the unit of analysis, the study t eam may opt to focus on an individual school, or policy decisions related to professional development, coaching, curriculum, or double

- dose Algebra were made at the school level. With regard to the site selection criteria, there are two levels that will gu ide the identification of cases: (1) cross

- case criteria, designed to achieve appropriate variation within the set of cases as a whole, and (2) topic

-specific criteria, which reflect the promising and innovative practices identified through the literature reviews. Indeed, the development of these criteria has been informed by the study team ’ s work on the literature reviews, in which analysts have detailed the distinguishing features of each topic
. Cross
-site criteria: The study team will seek as much vari ation as possible among observable state, district, and school characteristics which may be associated with implementation patterns and success in Algebra I. Tothe extent possible, given limitations of a relatively small sample, t he full set of 10 sites w ill reflect the following criteria :

 **Criterion 1: Sites will represent exemplars** of the particular topic for which they are identified (see specific quality criteria listed below). That is, their practices in each topic area will reflect those that are consistent with the literature review for each topic. Moreover, each site should provide some evidence of improved outcomes. While the study team cannot set a specific achievement threshold or percentage point gain (in none of the topics is the literature consistent enough to warrant this), each nominated site should provide documenta tion that shows improvement, whether in student performance on standardized tests, student pass rates in Algebra I, student grades in Algebra I, student engagement, or other measures as appropriate for each topic.

 **Criterion 2: Sites will reflect geographic diversity** , both with regard to regional location within the United States as well as urbanicity. For each topical area, the study team will identify two sites representative of the primary unit of analysis (districts or schools) that are in different re gions of the United States. For each topic, one site will be in an urban location and the other in a rural setting.

 **Criterion 3: Sites will reflect diversity of enrolled students**
. The study team will structure the case studies so that they reflect students of diverse ethnic, racial and socioeconomic backgrounds
— recognizing that students who struggle in Algebra I are disproportionate ly students of color or of low socio

- economic status . Thus, for each site selected at least half of the students in the identified sites will be ethnically or racially diverse or at least 40 percent of its student population participating in the National School Lunch Program. Topic
-specific criteria : Each of the topics has been the focus of a careful literature review by m embers of the study team. As a part of this process, analysts have identified practices associated with each that exemplify promising, innovative, or research

-supported best practices. When identifying sites for the case studies, study team members will screen nominated sites to ensure they will provide evidence of these practices and programmatic structural features. Details associated with each are listed below.

 **Instructional Practices:** Selected sites will be scho ols with an established track record of teaching algebra for conceptual understanding and procedural fluency. More specifically, the study will include schools that systematically promote this type of teaching for all algebra teachers in the building and have some evidenc e of improved student o utcomes (e.g., algebra end

- of

- course exam scores have gone up since the school implemented this type of instruction). Schools may attribute success in delivering this type of instruction to the types of instructional materials they use (e.g., high

- leverage tasks and/or curricula that they have purchased or developed on their own), the types of professional development activities they offer or have received (e.g., common lesso n planning, peer observation, professional provided by the district or outside cons ultants, etc.) or other factors. **Professional Development:** Selected sites will be districts with professional development programs targeted at improving the teaching and learning of Algebra I or courses that promote algebraic thinking (e.g., programs that develop algebraic thinking in grades 6

-8). The professional development activities may include workshops or seminars that are developed and delivered by district staff and/or professional learning communities that form within or across schools in the dist rict. The focus of the professional development activities may include specific content (e.g., understanding the big ideas of algebra, linear relationships, etc.), pedagogical techniques (e.g., questioning strategies, use of multiple representations of al gebraic concepts, etc.), instructional materials (e.g., how to implement adopted materials or other lessons and tasks focused on algebraic concepts) and assessments (e.g., how to implement formative assessment lessons). The district should have a systemat ic approach to implementing and supporting these types of professional development activities.

 **Instructional Coaching** : Selected sites will be districts that are implementing a mathematics coaching program focused on the improvement of teachers’ mathematics instruction. Although mathematics coaches within these districts may be responsible for a number of activities, the majority of their time should be spent working directly with teachers to plan, deliver, and/or evaluate mathematics instruction. Ideally, the district would have established a policy for hiring/training mathematics coaches to ensure individuals in these roles are leaders with knowledge of mathematics content, mathematics teaching and learning, and teacher leadership. In addition, the distr ict should have a support structure in place to provide coaches with the time and resources needed to work with teachers in this capacity.

 **Curricular Alignment:** Selected sites will be districts with an established curricular framework that is vertically al igned to prepare students for Algebra I. More specifically, this curricular framework should specify a series of standards that are vertically aligned to support students in the development of skills and understandings needed for Algebra I. These include , but are not limited to, an understanding of whole number and fraction operations as well as the ability to reason algebraically. The standards within the curricular framework should be clearly stated and organized to support a connected progression of s kills and understandings that will ultimately prepare students for Algebra I.

 **Double -**  **Dose Algebra** : Selected sites will be districts that have (1) potentially promising outcomes for students, in the sh ort- and or longer

-term , and (2) provide professional development and curricular frameworks for double

- dose algebra teachers. Specifically, promising outcomes could be demonstrated by improvements in Algebra I course failure rates or grades, standardized test scores in math and reading (in any grade in high s chool), on -time graduation and/or college matriculation rates. Recent evidence demonstrates that double

- dose algebra may have delayed effects on standardized test scores and that improvements may extend beyond math achievement to reading if double

- dose a lgebra courses emphasize written and oral communication of mathematic concepts. In addition, high

- quality double

- dose algebra problems must go beyond extending students’ learning time for algebra and provide specific guidance and curricular materials for e nriching and supporting learning for struggling students in double

- dose algebra courses. These materialsshould be accompanied with appropriate professional development opportunities for teachers. Site Identification Procedures : To identify potential site s, AIR staff will contact other individuals or organizations that work directly with schools, districts, and states in the implementation of instructional strategies. To identify prospective sites, AIR will first pinpoint programs or initiatives related to each topic area. Specifically, the study team will draw on six sources to identify programs, including (1) the literature reviews associated with this project, (2) information obtained from focus group interviews, (3) a national call to Algebra I progra ms and listservs, (4) an Internet search for programs using key terms associated with each topic, (5) AIR’s professional contacts, and (6) referrals from ED. For example, the study team will reach out to organizations and researchers who have conducted stu dies related to math education. The c o ntractors’ co lleagues in the Regional Comprehensive Centers, Content Centers, the Regional Education Laboratories, or the California Collaborative will be important sources for suggesting potential sites. In addition, AIR staff have professional contacts in many school districts (including Boston, Chicago, San Francisco, and DC) and we could solicit nominations from district administrators (for school

- level practices). AIR will enter all information about each site in to a Microsoft Access database that will track each program/site, its characteristics, and its progression through the selection process. Upon identifying 5– 10 programs per topic area through the channels highlighted above, AIR will review any extant data or marketing materials available to determine the extent to which each program has undergone internal or external evaluation and corresponding evidence for effectiveness, and include this information in the database. The study team will first filter each nominated site against the cross -site and site

-specific criteria described above. After identifying the set of schools that pass first screening process, the study team will collect additional data, primarily through extant sources, to determine if the site is likely to yield rich data. The identification of the final sample will be an ite rative process, through which the team will identify potential sites, consider the features of the districts and states in which they are nested, a nd re

- evaluate the candidate districts and schools. The final set of sites will be reviewed by ED. As part of another aspect of this study, the contractors have identified a set of exemplar districts and schools that will provide feedback on the literature review documents through informal focus groups. These sites, which have already been screened by the study team, are likely candidates for at least some of the case study sites (see Exhibit B.2). Exhibit B.2 . Potential Sites for Case Studies State Distri ct or School Illinois Chicago Public Schools California Elk Grove Public Schools Maine RUS
- 19 School District Maryland Howard County Schools North Carolina Charlotte

- MecklenburgThe study team also will identify a set of 10 replacement sites in the event that some prove to be inappropriate for inclusion in the case studies (e.g., substantial turnover in studen t population ) or decline to participate. 

### 2. **Procedures for Data Collection** The data collection procedures of each of the main components of the study ( site visits, principal interviews, teacher surveys, and state interviews) are discussed in detail below. 

### Data Collection Activities by Topic Because each of the topics for which the contractors are to conduct case studies reflects a different purpose, practices, and implementation, the data collection activities will vary for each. The common elements are that each will involve interviews or focus groups at the district and school level, and each will involve some form of document collection. All focus groups and interviews are expected to require one hour, while classroom observations could range from 60 to 90 minutes, depending on the organization of the school day. The specific numbers and roles of interviewees and focus group participants are detailed below. **Instructional Practices**  . The purpose of the case study of instructional practices is to understand strategies, processes, and policies in schools that have made an effort to blend procedural and conceptual foci in their Algebra I inst ruction . Hence, the school is the unit of analysis for this topic, and the study team will visit two such schools. However, the contractors intend to collect data from the district level, to better understand that context in which each school is situated . Because the contractors intend to conduct classroom observations, each site visit for this topic will last two or three days, with two site teams, each including one site visit lead and one site visit associate.

Exhibit B.3: Data Collection Activities for Instructional Practices Instructional Strategies. Unit of Analysis: School Level Respondent Total n of respondents District (n=2) Interview with district official with primary responsibility for math ematics 2 School (n=2) Observations of up to 5 teachers, using the Mathematical Quality of Instruction

observation protocol 10 Interviews with each teacher who is observed 10 Focus group with remaining Algebra I teachers in the school (if there are more than 5) Variable Principal interview 2 Interview with math department chair or Algebra team lead 2 Interview with instructional coach, if applicable 2 Student focus groups (approximately 6 students) 12 Documents Teacher assignments 10

Lesson plans 12 Schoolwide planning documents with regard to the instruction of Algebra I Variable**Curricular Alignment**
. The purpose of the curricular alignment case study is to describe districts that have developed and implemented a curricular framework that promotes the skills and understanding needed to be successful in Algebra I prior to enrollment in an Algebra I course. The unit of analysis for this topic is the school district because this is the level at which the curricular framework and pacing guides are established, as well as the vision with regard to Algebra I instruction. The study team will visit two districts in two different states. Each site visit for this topic will last two days, with two site visitors: one site visit lead, and one site visit associate.

Exhibit B.4: Data Collection Activities for Curricular Alignment Curricular Alignment. Unit of Analysis: District Level Respondent Total n of respondents State (n=2) Interview with the state official with responsibility for mathematics curricular

standards 2 District (n=2) Interview with the associate superintendent for curriculum and instruction; or

superintendent in small districts 2 Interview with math leaders at the district level 2
-4 Interview with the district administrator with responsibility for professional development for math teachers 2
School Focus group of teachers who teach classes prior to Algebra I (elementary, across

schools) 5
- 6 Focus group of teachers who teach classes prior to Algebra I (middle, across schools) 6
-8 Focus group of teachers of Algebra I (cross

- schools) 6
-8 Focus group of math coordinators (or department chairs, depending on school organizational structure) from different schools 4
- 5 Documents Curricular framework variable

Pacing guides Additional instructional resources that support these ideas and concepts in the early grades **Professional Development**
. The purpose of the professional development case study is to explore innovative practices in terms of professional learning opportunities designed to deepen teachers’ content knowledge, conceptual understanding, and instructional capacity with regard to A lgebra I. Because professional development policies are often mapped out at the district level, the school district is the unit of analysis for this topic. However, it is critical that the study team collect data from the school and teacher level as well , to explore within

-school participation in professional development. The study team will visit two districts in two states. Each site visit for this topic will last two days, with two site visitors: one site visit lead, and one site visit associate.

Exhi bit B.5: Data Collection Activities for Professional Development Professional Development. Unit of Analysis: DistrictLevel Respondent Total n of respondents State (n=2) Official with responsibility for mathematics professional development policy 2 District (n=2) District Math Coordinator (elementary, secondary) 4

District PD Coordinator (elementary, secondary) 4 District official with responsibility for curriculum and instruction 2 External PD provider such as university faculty, regional or county office person (if applicable) 2
School Focus group of teachers from secondary schools (across up to 5 schools in the

district) 5 Focus group of principals from secondary schools(across up to 5 schools in the district) 5 Focus group of school mathematics coordinators (elementary) 5 Focus group of high school mathematics department chairs 5 Documents Professional development documents (handouts, outline/agenda, PowerPoint

presentations, district professional development plan, template for teacher/school professional development plans) variable Observations Observe a professional development session (if possible)
**Instructional Coaching** 
. The purpose of this case study of instructional coaching is to understand how districts and schools use mathematics coaches to support teachers as they continue to improve their mathematics teaching. In addition to learning more about the role of a mathematics coach as they interact with teachers, the study seeks to learn more about the policies and structures that support the training, development, and activities of those coaches. The study team is interested in how the coaching program was developed, the supports that contributed to its success, and the challenges that were overcome. Hence, the district is the unit of analysis for this topic. The study team will conduct site visits to two districts, and will include data collection activitie s in two schools in each district. The site visits will last three days and will be conducted by one site visit lead and one site visit associate.

Exhibit B.6: Data Collection Activities for Instructional Coaching Instructional Coaching. Unit of Analysis: District Level Respondent Total n of respondents District (n=2) Interview with District Mathematics Coordinator 2

Interview with district official responsible for math coaching 2 School (n=4) Interviews with math coaches (in each school) 4

Interviews with principals (in each school) 4 Interviews with math department chair or teacher leader (in each school) 4 Focus groups with math teachers in each school (up to 5 teachers) 20 Documents Coach training manuals variable

Coach job description/guidelinesCoaching materials (e.g., rubrics, forms used to provide feedback, forms to monitor teacher progress, schedules)
**Double -**  **Dose Algebra**
. The purpose of this case study is to explore the structures and implementation processes associated with instructional programs in which students receive additional Algebra I instruction. For example, students may participate in a traditional Algebra I class, but then attend a second Algebra I class that reinforces what they have learned. T he district is the unit of analysis for this topic. The study team will conduct site visits to two districts, and will include data collection activities in two schoo ls in each district. Each site visit for this topic will last two days, with two site visitors: one site visit lead, and one site visit associate.

Exhibit B.7: Data Collection Activities for Double-Dose Algebra Double
- Dose Algebra. Unit of Analysis: District Level Respondent Total n of respondents District (n=2) District Math Coordinator 2

District official who oversees curriculum and instruction 2 School (n=4) Principal interview (high school) 4

Math Department Chair/Teacher Leader interview 4 Double Dose Algebra Teacher Focus Groups (up to 5 teachers in each school) 20 Student Focus Groups (up to 8 students in each school
– students selected by teachers as having benefited from double

- dose) 32 Observations of double

- dose algebra classrooms and their associated standard Algebra I classroom/session (1 observation of double

- dose algebra and associated algebra class session for each of four schools) 4

Documents Example lesson plans from each double

- dose algebra teacher in each school (2

lesson plans and a “syllabus” for up to 5 teachers in each school) variable District and school guidance on course placement policies for double

- dose algebra District and school curricular guides for double

- dose algebra class sessions


### Site Visits **Protocol Development Interview and Focus Group Protocols.**  Two contractors, AIR and Windwalker, have developed protocols to guide the case study interviews and focus groups. These protocols include both open - ended questions and more closed

- ended questions . The open

- ended questions will encou rage in
- depth responses, and incorporate flexibility so that the study team can follow threads of conversation to their logical conclusion, while the closed

- ended questions will ensure that similar types of data are collected across the case study schools. The study team recognizes the utility of including broad questions to init iate discussion, followed by focused probes to ascertain insights in important areas. In developing protocol questions, the study team has sought to avoid language that may be loaded, leading, or likely to yield socially desirable responses.Interview and focus group protocols have been designed to fully explore the evaluation questions without placing undue burden on the respondents. The contractors began the protocol development process by creating a theory of action for each topic, which reflects the key constructs and contextual variables on which the study will need to collect data
. Developing a theory of action for each topic
– grounded in the relevant literature for each
– helped ensure that the protocols, as a set, cover the necessary topics. As a result, the full set of interview protocols, focus group protocols, and extant data sources will generate sufficient information to address all evaluation questions. Th e contractors anticipate that site visitors will adapt the protocols for the context and particular intervention models being implemented in the site visited schools. To facilitate this process, the protocols include suggested probes, topics that that the interview should “listen for,” and notes to indicate where the interviewer should adapt the protocol to the particular respondent’s circumstances. Although each question contains many possible probes, the interviewer is not required (or expected) to ask each of these. Rather, the probes are intended to ensure the interviewer is attuned to th e full range of possible topics and variables to be addresses in the responses, and
– in rare cases
– to prompt a less communicative interviewee. [Note to reviewer: this paragraph reflects our intent for piloting activities, although we have not piloted instruments yet, we are retaining this text as a placeholder.] Following publication of this OMB submission for public comment, the study team conducted piloting interviews with fewer than 10 district officials, principals, and teachers in three locations. Study team members asked these educators and administrators to react to (1) the overall organization, flow, and length of the interview, (2) the clarity of the interview wording and language, (3) specific questions that were unclear or difficult to answer, and (4) any recommended changes. The study team then used the responses from these pilots to modify the protocols to make them as suitable as possible for the actual site visits. **Classroom Observation**  **Protocol .**  As part of the case study of instructional practices, t he study team will conduct a small set of classroom observations using instruments that AIR has developed and successfully implemented in other projects focused on math teaching and learning. For exam ple, AIR has a low

- inference instrument that captures the frequency with which teachers focus on concepts, procedures or both. We will also have one math content expert on each visit that will record running notes in each of the classrooms observed. These notes will be used to further illustrate behaviors captured in the observation protocol. **Request for Documents .**  The plans for each case study topic include some collection of documents, as relevant for the specific topic. Because each topic requires a di fferent set of documents, the study team has prepared a separate request for documents for each of the five topics. In general terms there are two sorts of requests: (1) **A general request**, in which the study team requests a straightforward set of documents t hat are generally readily accessible, such as a district’s curriculum framework, pacing guides, or a training manual.(2) **A request with detailed guidelines** . For example, the request for teacher assignments includes specific guidelines on how many assignments are to be submitted by each teacher, the focus of the assignment, and the intended level of cognitive demand for the student. Prior to conducting site visits, AIR will ensure that all data collection procedures (including informed consent forms, in Appendi ces A
- E of this document) are approved by its Institutional Review Board (IRB). **Training and Preparation Training for Site Visitors.**  The study team will train the site visit team so that each member brings to his or her on

-site work a consistent understa nding of the relevant policies, the study, and the data collection needs. Prior to the first wave of data collection, all site visit staff from AIR and Windwalker will convene in Washington DC for a one

- day training session. The site visit team leads from AIR and Windwalker will jointly develop and conduct the training so that consistent messages and training content are delivered across organizations. The one - day training will cover many topics, including the major concepts and big ideas of algebra, evalu ation questions, site visit approach and activities, data collection on site (including brief review of interview protocols), and coordinating site visit work. For the training, the site visit task leaders will develop a site visit checklist that will outl ine all tasks the site visitors need to perform before, during, and after each visit. All site visit team members will adhere to this checklist to ensure that visits are conducted efficiently, professionally, and consistently. There will be regular meetings of the site visit task leaders from AIR and Windwalker as well as weekly email updates and periodic larger meetings of the site visit staff. **Preparation for Site Visits.** Prior to each site visit, the study team will compile and review all extant background information on the school. Although some document data must be collected on site, Web

- based resources (primarily state, district and school Web sites) include important contextual information on the history, priorities, performance, community su pport, and staffing of each school. Staff will review these closely before each visit and will enter preliminary information in a data capture template. **Administration of the Site Visits** Prior to the site visit, researchers will conduct the interview with the state department of education official for the topics that include interviews at this level. While one researcher will ask questions of the respondent, the second researcher will take notes. Two
- person s ite visit teams will be instructed to be flexibl e in determining how to organize each school v isit, being mindful of district and school context and the need to collect data that answer specific evaluation questions. However, the site visits will have a standard set of features, including interviews and focus groups with state, district, and school

- level respondents; all topics will include the collection of document data as well. Three of the topics include cross

-school focus groups which will include five or six respondents across schools of a given l evel (elementary or secondary). To convene these focus groups, AIR and Windwalker will work with a district administrator to facilitate communications with school

- levelrespondents, and to identify a central location with convenient access for required re spondents. In addition, the contractors will be as flexible as possible with regard to the timing of interviews and focus groups, so as not to disrupt school schedules. For four of the case study topics, the contractors have determined that two

- day site v isits will allow adequate time for interviews and focus groups with district officials and school administrators, teachers, coaches, and students. For one topic, the site visit activities may require a three

- day site visit, with four staff, largely due to the time required for the classroom observations
. If the instructional periods are one hour (as in Exhibit B.8, below) a two

- day visit should be possible, but if class periods are 90 minutes (as is the case in high schools with block scheduling) then a three - day site visit may be necessary.

Exhibit B.8. Sample Site Visit Schedule : Instructional Strategies Site Visit to Sample District: Two Teams of Two Researchers on Site Site Visit Team #1 Site Visit Team #2 Day 1 Day 1 Travel in morning

, at district central office in afternoon Travel in morning , at school in afternoon 1:00
– 2:00 Interview with District official responsible for math instruction 1:00
– 2:00 Observation of Teacher 1 2:00
- 3:00 Interview with District official responsible for math coaches, if applicable 2:00
– 3:00 Interview with Teacher 1 3:00
– 4:00 Meet with district administrative staff to collect document data, as necessary 3:00
–4:00 Confer with administrative staff to collect document data Day 2 Day 2 8:00
– 9:00 Observation of Teacher 2 8:00
– 9:00 Observation of Teacher 4 9:00
– 10:00 Interview with Teacher 2 9:00
– 10:00 Interview with Teacher 4 10:00
– 11:00 Observation of Teacher 3 10:00
- 11:00 Observation of Teacher 5 11:00
– 11:30 Break 11:00
– 12:00 Interview with Teacher 5 11:30
– 12:30 Math coach interview 12:30
– 1:00 Lunch 12:30
– 1:00 Lunch 1:00
– 2:00 Teacher focus group 1:00
– 2:00 Principal interview 2:00
– 2:30 Break 2:00
– 3:00 Interview with Teacher 3 2:30
– 3:30 Student focus group 3: 00
– 4:00 Math department chair interview 3:30
– 4:00 Confirm collection of teacher assignments Each si te visit team will consist of two researchers (for professional development, instructional coaching, curricular alignment and double

- dose Algebra) or four researchers (for instructional strategies), from either AIR or Windwalker. Having the researchers work as a team during the site visits will enhance the consistency and reliability of data gathered. The same pair of researchers will be responsible for scheduling and conducting the visits , and both site visitors will attend all interviews if possible. However, site visitors may conduct interviews separately if this is necessary to collect information from all respondents. Because of this, e ach interview and focus group will be audio recorded. In preparation for each visit, site visitors will review notes from interviews with state officials, review extant data on each site, consider the reasons for which each jurisdiction was selected for inclusion in the sample, review other relevant documentation, and annotate each section of the individual interview protocols accordingly. Further, these notes will be used to guide the wording of each question. The use of experienced interviewers, coupled with careful preparation, will ensure that interviews are not “canned ” or overly formal. The lead site visitor foreach case will have ultimate responsibility for completing and submitting the specified set of standard deliverables on the checklist for each site visit or follow

- up. Throughout the process of data collection and reporting, the contractors will make all efforts to protect the privacy of respondents participating in the site visits. Th e study team will not identify by name any of the interviewees; nor will t h e study team attribute quotes. Although the study team will identify the names of states in the final reporting of case studies, districts and schools will be identified by pseudonyms. **Collection of Documents and Files** All of the case study topics include the collection of some document data, including schoolwide planning documents, professional development training guides, curricular framework, and pacing guides, among others. Two documents in particular
– lesson plans and teacher assignments
– will require more specific guidelines for the data collection team. First, the study team plans to collect two sets of lesson plans from each teacher whose class is observed: (1) plans for the classes in which teachers are introd ucing a new topic, and (2) plans for the days during which students are preparing for a test. In addition to the plans themselves, the study team will collect associated slide presentations, handouts, and other supporting materials. Second, the study te am will collect two sets of teacher assignments from each teacher whose class is observed. One assignment should be what the teacher considers to be a “typical” assignment, while the second should be a more challenging assignment. The teacher should also submit the answer key and details regarding how they would grade the assignments
– that is, the number of points possible for each part as well as the way in which points would be allocated or reduced based on student work. 

### 3. **Methods to Maximize Response Ra te**  Data collection is a complicated process that requires careful planning. The research team has developed interview protocols, focus group protocols and questionnaires that are appropriately tailored to the respondent group and are designed to place as l ittle burden on respondents as possible. The team has also pilot ed core data collection instruments to ensure they are user

-friendly and easily understandable, all of which increases participants’ willingness to participate in the data collection activitie s and thus increases response rates. In addition to careful instrument design, a high response rate among the case study sites may be assured through careful recruitment materials. These recruitment materials will emphasize the social incentive to respond ents by stressing the importance of the data collections as part of a study of innovative practices that will provide much

- needed information to districts and schools. AIR’s experience in past evaluations has demonstrated the importance and value of buildi ng a consensus of support with participating districts. This leads to districts and schools that have the capacity, willingness, and commitment to cooperate fully with the research and data collection responsibilities. Investing in site development at the front end reduces problems in the back end, helping to ensure smooth implementation of the evaluation.

### 4. **Expert Review and Piloting Procedures** [Note to reviewer: this paragraph reflects our intent for piloting activities, although we have not piloted instrum ents yet, we are retaining this text as a placeholder.] Following publication of this OMB submission for public comment, the study team conducted piloting interviews with district officials, principals, and teachers in three states
. Study team members aske d these educators and administrators to react to (1) the overall organization, flow, and length of the interview, (2) the clarity of the interview wording and language, (3) specific questions that were unclear or difficult to answer, and (4) any recommende d changes. The study team then used the responses from these pilots to modify the protocols to make them as suitable as possible for the actual site visits. 

### 5. **Individuals and Organizations Involved in Project** AIR is the contractor for the Study of Promoting Student Success in Algebra I. The project director is Dr. Kirk Walters, who is supported by an experienced team of educators leading the major tasks of the study (see Exhibit B.9 for a list of key staff and responsibilities). During data collection and pa rticularly during the initial phase of analysis, the contractors will draw on the cross

-staffing of some key members of the study, including the project director, deputy project director, and team leaders, to combine findings across these data sources to c reate the synergies that are at the core of the case study design. Contact information for the individuals and organizations involved in the study is presented in Exhibit B.9
.

Exhibit B.9 . Organizations, Individuals Involved in Project Responsibility Organ ization Contact Name Telephone Number Project Director AIR Dr. Kirk Walters (202) 403

-5838 Deputy Director AIR Dr. Nicholas Sorensen (312) 283

-2318 Task Lead: TWG Meeting and Senior Math Consultant AIR Dr. Toni Smith (202) 403

-5926 Senior Math Content Advisor AIR Dr. Steven Leinwand (202) 403

-6926 Task Lead: Case Studies AIR Dr. Kerstin Carlson Le Floch (202) 403

-5649 Task Lead: Qualitative Analysis AIR Andrea Boyle (650) 843

-8294 Windwalker Jennifer Grossman (703) 970

-3518 Windwalker Meschelle Linjean (703) 970

-3554 Windwalker Lauren Malone (703) 970

-3542 Windwalker Alison McNeil (703) 970

-3528 Windwalker Samantha Schwartz (703) 970

-3519