# OMB Package **Part A :**  **Evaluation of a District** **- Wide Implementation of a**  **Professional Learning Community Initiative**  **September 2013** OMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative ii 

# Table of Contents PART A. JUSTIFICATIO N 1 1. Circumstances Necessitating the Collection of Information 1 a. Statement of Need to Evaluate District -Wide Imple mentation of a Professional Learning Community Initiative 1 b. Research Questions 2 c. Study Overview 3 2. Purposes and Uses of Data 6 3. Use of Technology to Reduce Burden 6 4. Efforts to Avoid Duplication 7 5. Methods to Minimize Burden on Small Entities 7 6. Consequences of Not Collecting Data 7 7. Special Circumstances 7 8. Federal Register Announcement and Consultation 8 9. Consultations Outside the Agency 8 10. Payments or Gifts 8 11. Assurances of Confidentiality 8 12. Justification for Sensitive Questions Hours 10 13. Estimates of Burden Hours 10 14. Estimates of Cost Burden to Respondents 10 15. Annualized Cost to the Federal Government 11 16. Reasons for Program Changes or Adjustments 11 17. Plans for Tabulation and Publication of Results 11 a. Tabulation Plans 11 b. Publication Plans 13 18. Approval Not to Display the Expiration Date for OMB Approval 14 19. Exception to the Certification Statement 14 References 15OMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 1 

## PA**RT A.**  **JUSTIFICA**

**TION** This submission is a request for approval of data collection activities that will be used to support the Mid -Atlantic R egional Educational Laboratory (REL MA) project, Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative . The study is being funded by the Institute of Education Sciences (IES) of the U.S. Department of Education (ED), and is being implemented by ICF International and its subcontractor, Rutgers University’s Center for Effective School Practices. This study aims to address the need for systematic information about district -wide implementation of professional learning communities as a critical element in improving teacher quality and instruction, thereby contributing to increased student achievement. The study will survey online a population of teacher participants in school - based professional learning communities and interview principals face to face about the context and their perceptions of the initiative , pre - and po st - implementation. Data collection from teachers will focus on what the professional learning communities do, how they operate, and to what extent they produce the outcomes expected of them as framed by six conceptual attributes of professional learning co mmunities and five specific tasks. Data collection from principals will focus on contextual information about school culture and conditions such as resources that support implementation. Teachers and principals will also provide their reflections on the ch allenges of implementing professional learning communities and their suggestions for improvement. The analysis will enable comparisons among professional learning communities within and across schools, and between teachers’ pre - implementation expectations and post - implementation experiences. Study findings are expected to inform both theory and practice related to implementation of professional learning communities. This submission requests approval to recruit principals and teachers for the study and conduct in - person interviews with principals and online surveys with teachers in the study district. 

## 1. **Circumstances Necessitating the Collection of Information a.**  **Statement of Need to Evaluate District** **- Wide**  **Implementation of a Professional Learning Community I**  **nitiative** The specific legislation authorizing this data collection is specified in Part D, Section 174 (20 U.S.C. 9564) of the Education Sciences and Reform Act (ESRA) of 2002. Part D, Section 174 provides for ED to enter into five -year contracts with entities to establish a networked system of ten regional laboratories that serve the needs of each region of the United States. The Regional Educational Laboratories (RELs) are to carry out a range of activiti es to serve the needs of each region in the United States, including applied research, and development, dissemination, training, and technical assistance activities that focus on how to use data and analysis . The primary mission of the RELs is to help stat es and districts systematically use data and analysis to answer important issues of policy and practice with the goal of improving student outcomes. Accordingly, t his project was conceived in response to a specific request made of the REL MA from a superin tendent of a large school district in Pennsylvania (West Chester Area School District, abbreviated as WCASD) to assist in the evaluation of the implementation of professional learning communities in the district during the 2013 - 2014 school year.OMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 2

Practitio ners in the education field have widely embraced professional learning communities (PLCs) because they are aligned, at least in theory, with several features of effective professional development, such as active learning and teacher collaboration (for a re view, see Stoll, Bolam, McMahon, Wallace, & Thomas, 2006). These collaborative teams are believed to be effective because they give teachers the opportunity to be exposed to new ideas and practices , and improve their pedagogy via a process of critical refl ection (Hord, 1997; Wood, 2007). O perationally speaking, PLCs are teams of educators (most commonly, teachers) that meet regularly, either in - person or virtually, often but not always during scheduled school time, to develop lesson plans, examine student w ork, monitor student progress, assess the effectiveness of instruction, and identify needs for professional learning. These teams operate with a commitment to the norms of continuous improvement and experimentation , and engage their members in improving th eir daily work to advance the achievement of districts’ and schools’ goals for student learning (National Staff Development Council, 2001). Ideal ly, PLCs should be characterized by a set of dimensions or attributes such as shared beliefs, values, and visio n; shared and supportive leadership; supportive structural conditions; supportive relational conditions; collective learning; and peer sharing (DuFour, 2005; Hord, 1997). Further, it is generally acknowledged that strong PLCs are not only those in which ne w knowledge regarding content and pedagogy is acquired, but also those in which existing assumptions about teaching and learning are challenged and critiqued (Little, 2003). The increasing interest of education stakeholders in PLCs also appears to reflect a belief in the effectiveness of these learning communities. Recognizing the value of building systems for peer-to - peer professional learning among teachers and other education leaders, the U.S. Department of Education is investing in and supporting PLCs a s part of several key initiatives, including Investing in Innovation ( i3 ), the Office of Special Education Programs’ technical assistance centers, Promise Neighborhoods, and Race to the Top . Thus, there is strong support among school administrators and pol icymakers alike for expanding this practice (Talbert, 2010). Unfortunately, studies that evaluate the effects of PLCs on teacher professional learning and/or student learning outcomes are limited in number. A recent review (Vescio et al., 2008) identified a mere 11 studies that examined the effectiveness of PLCs and observed that few studies have moved beyond teachers’ self- reports. However, the researchers found cumulative evidence that is consistent with the hypothesized effects of PLCs . They concluded th at participation in PLCs influences teaching practice as teachers move toward student - centered instruction , and that when teachers participate in a learning community, students benefit as well, as indicated by improved achievement scores over time. Similarly, a recent meta - analysis found that PLCs have a small but statistically significant positive effect on student achi evement (Lomos, Hofman, & Bosker, 2011) . Thus, while the majority of studies that evaluated the efficacy of PLCs have important methodological limitations, the limited evidence from programs that were more systematically evaluated seems to support the conn ection between teachers’ conscious efforts to improve instructional practice through group/peer study and changes in classroom practice and student achievement ( Vescio et al., 2008 ). We expect that in addition to serving WCASD - specific information needs, t his study will help close the important theoretical and empirical gap in the current literature by addressing the question of how and under which conditions (e.g., leadership style, group norms, available resources, and work arrangements) PLCs get in a pos ition to achieve key desired attributes of collaborative work and meet standards of productivity that have the potential to enhance student learning and achievement. 

### b. **Research Questions** The primary research questions to be addressed in the study are listed below. The questions were formulated with direct input from the leadership of WCASD to ensure that the study meets their specificOMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 3

needs. However, these questions are also informed by the existing gaps in the current literature regarding the “ ho w and under what conditions” focus discussed in the previous section.
 Research Question 1 : How do PLC teams implement the study district’s PLC tasks? That is, what collaborative teamwork routines do they develop to achieve the key attributes of PLCs and t he key tasks listed in the study district’s PLC materials?  Research Question 2 : How do the study district’s teachers and principals evaluate their experience with the program? Is their actual PLC experience the same or different from what they expected it to be before the implementation of the program? How do teachers’ evaluation of the program and of the implementation compare with those of principals?  Research Question 3 : What specific artifacts (e.g., essential learning targets, standardized common asse ssments, and systematic interventions) are produced by each PLC team in the district ? 

### c. **Study Overview** I n response to a specific request from the superintendent, REL MA will evaluate the implementation of WCASD’s district -wide PLC initiative during the 20 13 - 2014 school year . As a result of its planning process, WCASD produced the WCASD PLC Protocol that explains what PLCs are, how and why they will be integrated into the district, and the specific sequence of five tasks that each PLC team should accomplish over time, regardless of grade level and content area. The protocol was shared with all educational staff in the district but does not specify how PLC teams should be established or what work routines they should adopt. Rather, this decision is left up to the principal and teachers of each school in order to accommodate the unique set of circumstances and resources that exist within each school and to leverage more effectively the expertise and experience of current teachers and administrators. Because PLC teams in the district will not follow a uniform, fully structured implementation design, t he proposed study has two main goals: to describe the process of implementing PLCs in WCASD and to assess and compare the progress that these PLC teams make on sp ecific tasks related to collaborative professional learning and instruction during the first year of implementation. Specifically, t he study will involve collecting survey data from WCASD’s teachers and interview data from WCASD’s principals to : describe a nd compare (a) the work routines of teachers in PLC teams within and across schools in the district and (b) the specific outcomes (or products) of their collaborative work. REL MA will also ask teachers and principals to evaluate both their experience with specific aspects of the implementation – such as team work, productivity, and professional development – and lessons they learned from that experience. This information will permit WCASD to identify evidence - based best practices that can support the ongoi ng implementation of the district’s PLC program and its impact. This information will also permit WCASD to identify actual and/or perceived barriers to implementation during the first full year of implementation that the district may want to address in sub sequent years of the implementation. To be efficient, REL MA proposes to use the same survey data collected from individual teachers, at the aggregate, to assess and compare the progress that PLC teams in the district make on achieving the stated goals o f the program during the first full year of implementation. First, the data collected from individual teachers who are members of the same PLC team will be aggregated to represent measures of PLC team - level variables of interest such as communication and l eadership style in the group. Next, the aggregated data (with PLC teams being the units of analysis) will be analyzed in relation to a logic model (see Figure 1) which REL MA developed with input from the district based on a scheme proposed byOMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 4

Nelson and c olleagues (Nelson, Cordray, Hulleman, Darrow, & Sommer, 2012) that inquires about the extent to which core components of the program have been implemented. The mode l is designed to benchmark the “PLC status” ( as beginning, evolving, or mature: see Grossman et al., 2000 ) of each PLC team in the district based on a measurement scale that is theoretically derived from the six key desired attributes of effective PLCs (i.e., shared mission, collaborative culture, collective inquiry, action orientation, commitmen t to continuous improvement, and results orientation ) as identified in the extant literature on the topic ( see DuFour et al. , 20 10; Hord, 1997) and as also referenced in WCASD’s PLC protocol. The short -, medium -, and long-term outcomes listed in the logic model specify the characteristics PLC teams should exhibit as they evolve over time from a beginning status in the short term to a mature status in the long term. The model also specifies the five sequential tasks that PLC teams in the district were asked to complete. These are: (a) formulation of essential learning targets, (b) d evelopment of common formative assessments that measure proficiency on essential learning targets, (c) m odifications of the core instructional program to allow differentiated instr uction by degree of student proficiency on one or more learning targets, (d) d evelopment of supplemental interventions for students who do not demonstrate proficiency on one or more learning targets through the core instructional program , and (e) formation of child study teams to coordinate more intensive and more frequent individualized interventions for students who do not meet proficiency on one or more learning targets through the core instructional program and sup plemental interventions. In all, using the logic model as an analytical tool will allow the study team to answer the research questions while providing WCASD with valuable insights about the progress of the PLC initiative district -wide. Data for the study will be collected through a post - imple mentation online survey of all teachers (N= 930 ) in all of WCASD’s 16 schools, and through semi -structured interviews with all principals in the district (N=16) post - implementation. REL MA will design and administer the post - implementation survey of teachers and will also design and collect the post - implementation interview data from principals. All WCASD’s teachers and principals will be involved in data collection activities because PLCs reside in individual schools and depend to a large extent on school - l evel supporting conditions. Teachers are the members of PLCs; it is their expectations and perceptions we wish especially to discern. This study will enable a fine -grained assessment of their experiences post - implementation as well as how their experience compares to that of other team members ( t o this end we will use a unique identifier to link individual teachers to PLC teams). Principals are included because they are on -the - scene observers and gatekeepers who control access to critical resources for impl ementation, and because they are key stakeholders who are deeply invested in the outcomes of teacher quality and student achievement. Importantly, because PLCs in WCASD will be given complete freedom to organize, it is very likely that teams’ composition a nd work routines (and therefore the PLC experience of individual teachers and principals ) will vary within and among schools in the district. Therefore, it is not feasible to limit data collection to a random sample of teachers and administrators in light of the goal of provid ing adequate representation of their experiences with implementation in the absence of a well - defined sampling frame . The specific data collection plan with the timeline for all data collection activities is included in Table 3.OMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 5

**Figure 1 : Logic Model**  **for Evaluation of a District -** **Wide Implementation of a Professiona l Learning Community Initiative** 

**Inputs Outputs**  Outcomes Activities Participation Short- term Medium - term Long - term Principals facilitate the process of determining grade - and subject- specific PLC teams within each building, such that all members on any given PLC team share learning outcomes (that is, teachers must teach the same academic standards in order to be on the same PLC team). Each PLC team is provided with a copy of the district- wide PLC protocols. Each PLC team meets for approximately 45–60 consecutive minutes weekly . PLC teams have access to expertise from external sources, such as program supervisors, co lleagues from other schools, and university professors. PLC teams organize independently to achieve key attributes of PLCs (shared mission, collaborative culture, collective inquiry, action orientation, commitment to continuous improvement, and resul ts orientation) that are described in the PLC protocols . Task 1: Collaborate to determine 3–5 essential learning targets for each unit taught as part of the core instructional program. Task 2: Design two versions of a common formative assessment for each unit taught to assess students’ proficiency in the essential learning targets. Task 3: Modify core instructional program based on assessment results to allow differentiated instru ction . Task 4: Develop supplemental interventions for nonproficient students.

Task 5: Form child study teams to coordinate more intensive and more frequent interventions for struggling students. All WCASD teachers in their respective PLCs

All WCASD teachers of a given subject and their program supervisor PLC teams working independently to create the assessments and then reaching consensus with other subjectspecific PLC teams PLC teams

PLC teams

PLC teams (potentially, multiple teams involved)  Regular and active participation in PLC activities  Positive evaluation of the PLC experience  Positive beliefs about the professional development (PD) value of PLCs  Positive beliefs about the impact of PLCs on studen ts

 Teachers positively evaluate and agree with the essential learning targets.

 Teachers positively evaluate the quality of the common formative assessment created by the team.

 Teachers analyze student assessment on formative common assessment to i dentify need in differentiated instruction.

 Teachers analyze student assessment on formative common assessment to identify nonproficient students.

 Teachers analyze student assessment on formative common assessment to identify struggling students.  Continued regular and active participation in PLC activities  Personal commitment to collaborative work  Personal commitment to PD and learning collabora tively  Translation of (or intention to translate) knowledge acquired to actual classroom practice

 Teachers use essential learning targets to guide instruction activities.

 All teachers make standard and consistent use of the common formative assessment .

 Teachers assign students to receive differentiated instruction based on analysis.

 Teachers assign students to receive supplemental interventions based on analysis.

 Teachers form child study teams to design individualized interventions for struggling students.  Sustainability of PLCs in WCASD  Shared professional culture that values collaboration and learning in WCASD  WCASD instrumental in diffusing the practice to other educational entities in the region and beyond

 As a matter of routine, teachers revise and update essential learning targets as necessary.

 As a matter of routine, teachers revise and update the common formative assessment as necessary.

 As a matter of routine, teachers review student progress and reconsider differentiated instruction assignment.

 As a matter of routine, teachers review student progress and reassess need for intervention (as well as the intervention itself).

 As a matter of routine, teachers review student progress and reassess need for intervention (as well as the intervention itself).OMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 6



## 2. **Purposes and Uses of Data** Rutgers University’s Center for Effective School Practices will analyze all project data for the REL MA Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative under contract number ED - IES - 12 - C- 0006 . The primary purpose of the study is to evaluate the implementation of the district -wide PLC initiative in the WCASD in response to a specific request from the superintendent. The proposed study has two main goals: to describe the process of implementing PLCs and to assess and compare the progress that PLC teams in the district make on specific tasks related to collaborative professional learning and instruction during the first year of implementation against the desired progress as per the program’s logic model . Findings from this study will be used by WCASD to formulate an operational definition of each attribute of PLCs and assess PLC teams’ progress toward achieving each one of these attributes, as well as progress toward completing the tasks identified in the district’s PLC protocol . The insights generated by the study will also assist the district’s leadership in considering programmatic modifications that can enhance the productivity and sustai nability of PLCs and provide baseline data for continuing efforts to monitor their functioning. The close collaboration between the district and study team is also expected to help build the district’s internal capacity to collect, manage, and use program data more efficiently based on the instruments and methodology used in the study . It is also reasonable to expect that study findings w ill inform similar efforts by other districts that are members of the Professional Learning Research Alliance (including the West Amwell Township School District, the South Hunterdon School District, the Monroe Township School District), as well as other stakeholders across the mid -Atlantic region which are not as far along as WCASD in planning and implementing PLCs or whic h are implementing but lack evidence - based information for adjusting their PLC implementation. The data will also inform efforts by state departments of education (starting with the Departments of Education in New Jersey and Pennsylvania, which are also me mbers of the Professional Learning Research Alliance) to support the planning and implementation of PLCs statewide. REL MA has already prepared a review of the literature on online and hybrid PLCs in response to strong interest from research alliance membe rs. Finally, it is likely that the findings of this study will be used by other federal agencies and a national audience of practitioners and scholars who are interested in PLCs . The description of how and under which conditions (e.g., leadership style, g roup norms, available resources, and work arrangements) will shed light on positioning PLCs to achieve key desired attributes of collaborative work, as well as to meet standards of productivity that have the potential of enhancing student learning and achi evement, and will narrow a conceptual and theoretical gap in the field. 

## 3. **Use of Technology to Reduce Burden**  Teacher d ata will be collected online through Survey Monkey. All teachers at WCASD have email and internet access through their district -supported accounts. Teachers will use these accounts to receive email communication about the study, access the surveys, and submit their electronic responses. Data from the semi - structured interviews of principals cannot be collected through such methods as web surveys. Computer- assisted telephone interviews are not optimal for collecting these data. The proposed face -to -face interviews will be necessary to allow in - depth, conversational exchange s with respondents, and to obtain answers to both open - ended and detailed questions. However, if repeatedOMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 7

attempts at scheduling these two - hour interviews during regular work hours fail, the REL MA study team will consider using telephone and/or computer
- a ssisted telephone interviews (e.g., via Skype) , providing that these can be audio - recorded for the purpose of transcribing the interviews. 

## 4. **Efforts to Avoid Duplication**  Prior to designing this study, REL M A produced a literature review on online and hybrid PLCs . REL M A has continued to search for relevant literature and to augment its literature base in the course of designing the study and gaining requisite approvals from OMB and the IRB at Rutgers University. Previous reviews of the literature on PLCs (e. g., Hord, 1997; Stoll & Louis, 2007; Wood, 2007) are based on findings from case studies. A recent review (Vescio et al., 2008) identified a mere 11 studies that examined the effectiveness of PLCs and observed that few studies have moved beyond teachers’ s elfreports. REL M A’s ongoing review of the literature reveals that no in - depth study has been conducted of a single district’s implementation of PLCs that details the experience of teachers with qualitative and quantitative measures of attitudes and produ ctivity, compares pre - implementation expectations with post - implementation experience , and provides in - depth analysis of differences among PLCs in similar contexts. Thus, t h e type of PLCs implementation evaluation proposed for this study has not been condu cted t o date . 

## 5. **Methods to Minimize Burden on Small Entities**  The study will utilize validated survey instruments and efficient interview protocols that focus on the data of most interest to the district and other audiences. WCASD will enable teachers to respond to the survey during periods that are already dedicated to school -wide and district -wide work. In terms of principal interviews, principals will be offered flexible scheduling to avoid potential conflicts or disruptions to their routine work. 

## 6. **Cons equences**  **of Not Collecting Data** This is one -time collection. WCASD has no means other than the proposed data collection to track and assess the implementation of a major initiative that has the potential to deliver effective professional development for al l teachers in the district and improve learning outcomes for all students in all of the district’s schools. The rich data collected through this project will generate valuable insights about the process of implementing PLCs in the district (including poten tial and perceived barriers to implementation) and will provide the baseline for assessing the impact of this program both in the short and long run. There has been no previous comprehensive study of PLCs in WCASD so t here is no alternative source for the information to be collected. Moreover, the implementation of PLCs in WCASD presents a rare research opportunity (a natural experiment of a sort) to study in depth the dynamic nature of PLCs and the implementation process, thus making a significant contribu tion to current theory and evidence - based practice regarding PLCs and their impact on teachers’ professional development and students’ learning outcomes. 

## 7. **Special Circumstances**  There are no special circumstances associated with this data collection.OMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 8



## 8. **Federal Register**  **Announcement and Consultation a.**  **Federal Register Announcement and Consultation**  A 60 - day notice to solicit public comments will be published in the Federal Register by February 1, 2014 . If and when public comments are received by April 1, 2014 , the study team will inform IES of the comments and ways i n which they will be addressed by April 7, 2014 . 

## b. **Consultations Outside the Agency** The study team has worked with IES to identify the five members listed below to serve as the Technical Working Group (TWG) for this study. 1. Robert Boruch University of Pennsylvania P: 215 - 898 0409 2. Laura Hamilton Senior Behavioral Scientist, RAND P: 412 - 683 - 2300, x4403 3. Chris Hulleman James Madison University P: 540 - 568 - 2516 4. Andrew Porter University of Pennsylvania P: 215 - 898 - 7014 5. Christopher Rhoads University of Connecticut P: 860 -486 - 3321 These experts on research methods, data analysis, implementation, and teaching and learning have provided input on the study’s design. Based on their input, the original emphasis on conducting process and outcomes evaluation of WCASD’s PLC initiative has been replaced with a focus on the process of implementation and the degree to which core elements of the program are implemented. This approach has informed the logic model of the evaluation as well as the overall research methodology which combines quantitative and qualitative data collection. 

## 9. **Payments or Gifts** The study will not provide respondents with incentives for participation and we will not give payments or gifts to schools or to the district for providing access and study data . The superintendent of WCASD requested this study. The district will administ er the pre - implementation online survey as well as t he recruitment of all study participants overall . The superintendent has enlisted the support of principals in making time available for teachers to participate during the regular school hours; we plan to interview principals during regular school hours as well, unless they re quest otherwise. . 

## 10. **Assurances of Confidentiality**  The data collection efforts that are the focus of this clearance package will be conducted in accordance with all relevant regulations and requirements, including the following :

 The Privacy Act of 1974, P. L. 93 - 579 (5 U.S.C. 552a) .  The Family Educational and Rights and Privacy Act (FERPA) (20 U.S.C. 1232g; 34 CFR Part 99) .  The Education Sciences Reform Act, P.L. 107 - 279 (20 U.S.C. 9573) .  The Freedom of Information Act (5 U. S.C. 552) .OMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 9

REL MA (specifically, Rutgers University’s Center for Effective School Practices, a subcontractor to ICF International) will protect the confidentiality of information for the study and will use it for research purposes only. The project director, Dr. Blitz, will ensure that i ndividually identifiable information about study members remain s confidential . When reporting the results, data will be present ed in aggregate form only so that individuals and schools will not be identified . The following safeguards, which are routinely employed by Rutgers University’s Center for Effective School Practices to carry out confidentiality assurances, will be applied consistently during the study :

 A list of study team members will be forwarded to NCEE.  All employees sign a confidentiality pledge ( Appendix A ) , which describes both the importance of and the employee’s obligation to discretion. All study team members who will have access to study participants or data will sign the pledge and will obtain appropriate clearance.  The plan for mai ntaining confidentiality also includes that all personnel with access to individual identifiers provide notarized nondisclosure affidavits and undergo training regarding (1) the meaning of confidentiality, particularly as it relates to handling requests fo r information and providing assurance to respondents about the protection of their responses; (2) controlled and protected access to computer files under the control of a single database manager; (3) built - in safeguards concerning status monitoring and rec eipt control systems; and (4) a secured and operator- controlled, in - house computing facility.  Procedures will be followed to revoke clearance in a timely fashion from members who leave the study team.  REL MA staff have obtained clearance through NCEE ’ s se curity clearance office.  Respondents’ personally identifiable information is maintaine d on separate forms and files, which are linked by sample identification number only .  Access to hard copy documents is strictly limited. Documents are stored in locked files and cabinets, and d iscarded materials are shredded.  Access to computer data files is protected by secure user names and passwords, which are available to specific users only. REL MA will also include appropriate statements about protecting confidentiality in the requests for data collection from participants and other materials generated by the study team . Those documents include:

 Post - I mplementation Interviews of P rincipals ( see Part B, Appendix D )  Post - Implementation Online Survey for Teachers (see Part B, Appendix E ) The following statements will appear in these documents to assure respondents about preserving confidentiality:

 “ Responses to the data collection activities will be used for research purposes only . The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific school or individual . We will not provide information that identifies you or your school to anyone outside the study team, except as required by law” .  “The contractor follows the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). The contractor will protect the co nfidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released. Information onOMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 10

respondents will be linked to their institution but not to any indivi dually identifiable information. No individually identifiable information will be maintained by the study team ” . 

## 11. **Justification for Sensitive Questions**  There are no questions of a sensitive nature in the district interview s . 

## 12. **Estimates of**  **Burden Hours** Table 2 shows the estimated burden hours for principals and teachers who will participate in data collection. These estimates are based on REL MA’s experience collecting such data from principals and teachers for similar studies, and on pre -testing in othe r school districts. REL MA pre -tested the interview protocols with three principals and the online survey with four t eachers from a school district in NJ represented in the PLRA . The average completion time for both online surveys was about 30 minutes. **Ta ble 2. Estimated Response Time for Data Collection**  **Respondent/Data Request**  **Number of**

**Targeted Respondents**  **Expected**

**Response Rate (%)**  **Number of**

**Respondents Unit** 

**Response Time**  **(Hours) Total** 

**Response Time**  **(Hours) Total Cost** 

**(Based on hourly**  **wage rate)**  **Principals** Post - implementation In - person interview 16 100% 16 2 32 $1,400 **Annualized Subtotal 32**  **$ 1,400**  **Teachers** Post - implementation Online survey 930 85% 79 0 0. 5 395 $1 0, 752 **Annualized Subtotal 395**  **$ 10**  **, 752**  **TOTAL Annualized ALL $**  **12,152** Total cost estimates based on hourly wage rates for the Bureau of Labor Statistics’ May 2012 National Industry- Specific Occupational Employment and Wage Estimates, NAICS 61100: Educational Services ( http://www.bls.gov/oes/current/naics3_611000.htm ). Principal burden estimates based on a mean annual salary of $90,980 and 2,080 annual hours, or $43.74 per hour, for Educational Administrators in Elementary and Secondary Schools (11 - 9032). Teacher burden estimates based on a mean annual salary of $56,6 20 and 2,080 annual hours, or $27.22 per hour, for Preschool, Primary, Secondary, and Special Education Teachers (25
- 2000). Total costs rounded to the nearest dollar. 

## 13. **Estimates of Cost Burden to Respondents**  There are no capital or start - up costs for resp ondents. There will also be no additional operational, maintenance, or purchase of service costs for respondents. WCASD already has computers for staff use and has established email accounts and Internet access for all teachers and principals. The district also has included an annual account for Survey Monkey in its budget.OMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 11



## 14. **Annualized Cost to the Federal Government**  The estimated cost of the study to the federal government is $ 280,722 over two years, representing an average annual cost of $140,361 . The esti mated total cost of the REL MA five -year contract is $32,353,087, of which this study accounts for $ 280,722 . Study costs were estimated using fullyburdened (“loaded”) hourly rates for staff, multiplied by the estimated number of hours of labor, and adding other direct costs. Multipliers for indirect rates were applied to direct costs. 

## 15. **Reasons for Program Changes or Adjustments**  This is a new data collection. 

## 16. **Plans for Tabulation and Publication of Results Table 3. Timeline**  **for Data Collection and Analysis 2014**  **1 2**  **3 4**  **5 6**  **7 8**  **9 10**  **11 12**  Administer p ost - implementation online survey of teachers   Analyze post - implementation online teacher survey data    Conduct Principal post implementation interviews   Analyze of principal interviews    Draft report   Final report and WCASD debrief  



### a. **Tabulation Plans** REL MA will use the information gathered from the teacher survey and principal interviews to examine and evaluate district -wide implementation of a PLC initiative. O nce the teacher survey dataset is ready for analysis, the study team will explore the distribution of individual responses on all variables of interest for irregular patterns (outliers, missing values, and deviation from normality for continuous variables) and make adjustments as needed. Next, the study team will compare responses across subgroups of teachers who teach at different levels (elementary, middle or high school), different content area, as well as teachers from different departments and schools, to identify potential sources of bias in our data. This is done for data diagnostic purposes and will not be included in public reports. As a rule, d ata will be reported at the aggregate to protect individuals’ identity. Audio recordings of principal interviews will be transcribed and the analyzed using Dedoose. Special attention will be given to extracting insights into barriers and facilitators to implementation. The content of teachers’ responses to the open - ended questions in post - implementation online survey will be downloaded and then content analyzed. W e will use D edoose to identify and extract teachers’ reports of most positive and most negative PLC experiences, and to synthesize their recommendations for improving the implementation of PLCs.OMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 12

Research Question 1 (how do PLC teams implement the study district’s charge?) will be answered primarily through analyzing the post - implementation survey data collected from teachers in Year 2 of the project. The survey will include several items ( see Part B, Appendix E ) that ask teache rs to report retrospectively on aspects of the work routine of the PLC in which they are a member (e.g., degree to which the team meets regularly, level of active participation in the group, group dynamics, and leadership style within the group).
 Responses (on a Likert scale) provided by teachers who are members of a specific PLC team (as ascertained by the team’s unique study identifier) will be summed and averaged to produce team - level measures of these variables.
 These variables will be analyze d next using cluster analysis to develop taxonomies of work routines as they emerge from this exploratory procedure. The study team will determine the specific cluster analysis algorithm to be used once there is an opportunity to assess the specific properties of the data collected from teachers.
 Next, the study team will use findings from content analyzing teachers’ responses to the open ended questions on the post - implementation online survey (where they describe the most positive and the most negative P LC experience they had and recommend possible improvements) to delineate common challenges/barriers that PLC teams encounter as members collaborate to establish work routines. The data analysis approach to answering Research Question 2 (how do WCASD teach ers and principals evaluate their experience with the program compared to their initial expectations ) will also utilize teachers’ responses to the online survey, specifcaly the variables measuring expectations, attitudes, evaluations, and self- reported beh aviors.
 Findings from teachers’ responses to the open - ended reflection questions on the post implementation survey will also be used to answer this question, either by corroborating and/or clarifying themes that emerge from the teacher surveys or adding n ew insights that were not captured through these surveys. To answer this research question regarding the principals, the study team will analyze qualitative findings from the post - implementation interviews with principals.
 Audio recordings of the intervi ews will be transcribed verbatim and checked against the original audio recording for accuracy.
 Dedoose, a web - based application for managing, integrating, and analyzing qualitative and mixed methods data, will be used for data analysis. A grounded theory approach will be employed to explore a priori and emergent themes regarding principals’ evaluation of the program. The a priori themes will be defined according to the explicit themes explored in the interview (e.g., role of principal in implementation an d allocation and use of school resources to support PLC work). It is anticipated that emergent themes will be focused on facilitators and barriers to implementation a s well as on the impact, if any, PLCs had on teachers, students, and/or the school in the course of the first year of the program.
 An extensive list of codes and their definitions that relate to each theme will be derived from the interview data. Two independent coders will use the coding scheme to analyze transcripts of four randomly selected interviews and agreement among coders (intercoder reliability) will be assessed using Cohen’s Kappa. The coding scheme will be adjusted to eliminate disagreements among coders and improve overall reliability (Cohen’s k > .85).OMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 13

 All codes will be entered i nto Dedoose and tagged to their associated segments of text for all interviews. Text segments will then be sorted by codes and reviewed by the coding team to identify emergent themes and to identify recurring patterns of responses to assess prevalence of t hemes.
 In the final step of the analysis, the study team will examine relationships among themes by looking at associations between differentially coded content. Lastly, the study team will compare key findings about teachers’ perspective on the PLC init iative and its implementation to key findings that emerge from the analysis of the interview data obtained from principals . The study team will note similarities and differences in perspectives as a way of assessing possible changes to school culture post - implementation. To answer Research Question 3 (specific products produced by PLC teams), the study team will analyze the survey data for teachers’ reports about artifacts produced by their team.
 Specifically, teachers will be asked to indicate the number of instructional units for which the team designed (1) essential learning targets, (2) core formative assessments, and (3) assessment rubrics. They will also be asked to indicate if they shared these artifacts with stakeholders (students, parents, a nd principal) and whether these artifacts were actually used with students in the course of the first year of implementation. The study team will also reference principals’ assessment of artifacts produced by PLC teams in their school based on analyzing th e interview data collected from all principals in the district at the end of the first year of full implementation of the PLC initiative. 

### b. **Publication Plans** REL MA will produce a “What’s Happening” report of 15 to 30 pages at the conclusion of the project . The study team will present findings in complementary narrative and tabular formats, and will include user-friendly graphic depictions (i.e., charts and graphs) to highlight and contextualize descriptive findings. The study findings will be presented with out reference to any names of individuals or specific schools, in order to avoid deductive disclosure of study respondents. The final report, scheduled for release in November, 2014, on the IES and REL M A websites, will be based on data from the pre imple mentation and post - implementation teacher surveys and principal interviews. We anticipate that the “What’s Happening” report will include:

 A brief description of the study district’s PLC initiative and implementation objectives  A discussion of the evaluati on approach and logic model  P resentation of key findings from the analysis of quantitative and qualitative data  D iscussion of study limitations, the appropriate interpretation of findings, and implications for current knowledge about PLCs, as well as for planning, implementation, and evaluation of PLCs  A study methodology appendix. REL MA will conduct a one - day debriefing to share the findings with the district’s leadership and to facilitate a discussion on the potential programmatic implications of these findings. REL MA will also conduct a separate webinar to disseminate the study results with a broader audience and will use its dissemination network to alert regional stakeholders about the final report and webinar opportunities. RE L MA will also work with members of the Professional Learning Research Alliance to identify ways in which they may use this report to guide their current and future professional learning activities. The study team will present this work at academic and pro fessional conferences and at meetings involving Research Alliance members and other regional stakeholders. Finally, the study teamOMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 14

anticipates producing several peer- reviewed journal articles based on this project, including a paper to present the logic mo del used in the project, a paper that will discuss the validation of the survey instrument used in this project, and a paper that introduces the overall methodological approach to the evaluation of the implementation, in addition to papers that describe an d discuss the key findings of the evaluation. 

## 17. **Approval Not to Display the**  **Expiration Date for OMB Approval** The study will display the OMB expiration date. 

## 18. **Exception to the Certification Statement**  ED is not seeking an exception.OMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 15 

## Re **fer ences**  Baenen, N., & Jackl, A. (2010, June). Evaluation of central services professional learning teams as of spring 2010. Eye on Evaluation . E&R Research Report No. 10.06. Raleigh, NC: Wake County Public School System. Delaware Department of Education. (2012). DDOE/Data coach project. Retrieved from www.doc.k12.de.us/tleu_files/PLCSurveyReport_2012.pdf. Drummond, K., Chinen, M., Duncan, T.G., Miller, H.R., Fryer, L., Zmach, C., & Culp, K. (2011). Impact of the Thinking Reader® software program on grade 6 reading vocabulary, comprehension, strategies, and motivation (NCEE 2010-4035). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. DuFour, R., Eaker, R., & DuFour, R. (2005). Recurring themes of professional learning communities and the assumptions they challenge. In R. DuFour, R. Eaker, & R. DuFour (E ds.), On common ground: The power of professional learning communities (pp. 7–30). Bloomington, IN: National Educational Services. DuFour, R., DuFour, R., Eaker, R., & Many, T. (2010). Learning by doing: A handbook for professional learning communities at work (2 d ed.). Bloomington, IN: Solution Tree. Grossman, P., Wineburg, S., & Woolworth, S. (2000). What makes teacher community different from a gathering of teachers? Seattle, WA: Center for the Study of Teaching and Policy. Hall, G., & Loucks, S. (1979). Implementing innovations in schools: A concerns - based approach. Austin, TX: Research and Development Center for Teacher Education. Hemphill, J. (1956). Group dimensions: A manual for their measurement. Research Monograph No. 87. Columbus, OH: Ohio State University, Bureau of Business Research. Heppen, J.B., Walters, K., Clements, M., Faria, A., Tobey, C., Sorensen, N., and Culp, K. (2012). Access to Algebra I: The Effects of Online Mathematics for Grade 8 Students. (NCEE 2012–4021). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Hord, S. M. (1997). Professional Learning Communities: Communities of Continuous Inquiry and Improvement. Austin, TX: Sou thwest Educational Development Lab. Killion, J. (2006). Collaborative professional learning in school and beyond: A tool kit for New Jersey educators . New Jersey Department of Education Office of Academic and Professional Standards and the New Jersey Professional Teaching Standards Board in cooperation with the National Staff Development Council. Little, J. (2003). Inside teacher communities: Representations of classroom practice. Teachers College Record, 105(6), 913–945. Lomos, C., Hofman, R. H., & Bo sker, R. J. (2011). Professional communities and student achievement: A meta - analysis. School Effectiveness and School Improvement, 22(2), 121– 148. Martin, T., Brasiel, S. J., Turner, H., & Wise, J.C. (2012). Effects of the Connected Mathematics Project 2 (CMP2) on the Mathematics Achievement of Grade 6 Students in the Mid-Atlantic Region (NCEEOMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 16

2012 -4017). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. National Staff Development Council. (2001). Standards for staff development . Oxford, OH: Author. Retrieved from http://www.nsdc.org/standards/learningcommunities.cfm Nelson, M. C., Cordray, D. S ., Hulleman, C. S., Darrow, C. L., & Sommer, E. C. (2012). A procedure for assessing intervention fidelity in experiments testing educational and behavioral interventions. Journal of Behavioral Health Services Research , 39(4): 374–96. Office of Management and Budget (2006). Questions and answers when designing surveys for information collection . Washington, DC: Author. Olivier, D.F. & Hipp, K.K. (2010). Demystifying professional learning communities: School leadership at its best. Lanham, MD: Rowman & Littl efield Education. Stoll, L., Bolam, R., McMahon, A., Wallace, M., & Thomas, S. (2006). Professional learning communities: A review of the literature. Journal of Educational Change, 7 , 221–258. Stoll, L., & Louis, K. S. (Ed.) (2007). Professional learning c ommunities: Divergence, depth and dilemmas. Berkshire, UK: Open University Press. Talbert, J. E. (2010). Professional Learning Communities at the Crossroads: How Systems Hinder or Engender Change. In Hargreaves, A., et al. (eds.). The Second International Handbook of Educational Change, Springer International Handbooks of Education , Volume 23, Dordrecht, Springer, 555–571. Vescio, V., Ross, D., & Adams, A. (2008). A review of research on the impact of professional learning communities on teaching practice a nd student learning. Teaching and Teacher Education, 24(1) , 80– 91. Wijekumar, K., Hitchcock, J., Turner, H., Lei, PW., and Peck, K. (2009). A Multisite Cluster Randomized Trial of the Effects of CompassLearning Odyssey® Math on the Math Achievement of Selected Grade 4 Students in the Mid-Atlantic Region (NCEE 2009 -4068). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Wood, D. R. (2007). Professional learning co mmunities: Teachers, knowledge, and knowing. Theory Into Practice, 46(4), 281–290.OMB Package: Evaluation of a District- Wide Implementation of a Professional Learning Community Initiative 17 

## Appendix **A :	Confidentiality	Pledge**  I understand that the names, and any other identifying facts or information, of individuals, businesses, organizations, and families participating in projects conducted by the Center for Effective School Practices at Rutgers University or its subsidiaries are confidential information. I agree that I will not reveal such confidential information, regardless of how or where I acquired it, to any person unless such person has been authorized by the Director or Project Manager at Center for Effective School Pra ctices at Rutgers University to have access to the information. I further understand that the unauthorized access to, use, or disclosure of any confidential information is a breach of the terms of my employment, or my consultant agreement with Center for Effective School Practices at Rutgers University and may subject me to court action by any interested party or to other sanctions by Center for Effective School Practices at Rutgers University. I acknowledge that this agreement shall continue to bind me ev en after the project(s) is (are) completed and/or even though my employment or my consultant agreement with Center for Effective School Practices at Rutgers University has terminated. In addition, in the course of my employment I may have access to person al information, electronic and otherwise, about fellow employees. I agree that I will treat that information as having the highest confidentiality, and not communicate it to fellow employees or others outside Center for Effective School Practices at Rutgers University. Final determination of whether or not there is a business purpose requiring that I access a fellow employees’ records will be made in consultation with the Director of Human Resources. Failure to uphold this standard is a breach of trust and may subject me to disciplinary action, including termination of employment. Other than in the course of my authorized employment or my consultant agreement, I further agree that I will not use, nor facilitate the use by any third party, in any way any in formation deemed confidential by the terms of any contract or other written agreement between Center for Effective School Practices at Rutgers University and any other organization, except by written authorization by both parties. It is my understanding th at Center for Effective School Practices at Rutgers University and the contracting organization(s) have the exclusive right to all information acquired or developed under such a contract or other written agreement. I acknowledge that I acquire no right, ti tle, or interest in and to any data or information to which I have access by reason of my employment or my consultant agreement and that I may not remove such data from my assigned work location without prior authorization. Name: Signature: Date: