# OMB Package Part A: Evaluation of a **District -** **Wide Implementation of a Professional Learning Community**  **Initiative**

## September 2013 

# Table of ContentsPART A. JUSTIFICATIO N 1

1. Circumstances Necessitating the Collection of Information 1 a. Statement of Need to Evaluate District

-Wide Implementation of a Professional Learning Community Initiative 1 b. Research Questions 2 c. Study Overview 3 2. Purposes and Uses of Data 6 3. Use of Technology to Reduce Burden 6 4. Efforts to Avoid Duplication 7 5. Methods to Minimize Burden on Small Entities 7 6. Consequences of Not Collecting Data 7 7. Special Circumstances 7 8. Federal Register Announcement and Consultation 8 9. Consultations Outs ide the Agency 8 10. Payments or Gifts 8 11. Assurances of Confidentiality 8 12. Justification for Sensitive Questions Hours 10 13. Estimates of Burden Hours 10 14. Estimates of Cost Burden to Respondents 11 15. Annualized Cost to the Federal Government 11 16. Reasons for Program Changes or Adjustments 11 17. Plans for Tabulation and Publication of Results 11 a. Tabulation Plans 12 b. Publication Plans 14 18. Approval Not to Display the Expiration Date for OMB Approval 14 19. Exception to the Certification Statement 14 References 15

## PART	A.	JUSTIFICATIO **N**

This submission is a request for approval of data collection activities that will be used to support the Mid

-Atlantic Regional Educational Laboratory (REL MA) project, Evaluation of a District

- Wide Implementation of a Professional Learning Community Initiative . The study is being funded by the Institute of Education Sciences (IES) of the U.S. Department of Education (ED), and is being implemented by ICF International and its subcontractor, Rutgers University’s Center for Effective School Practices. This study aims to address the need for systematic information about district

-wide implementation of professional learning communities as a critical element in improving teach er quality and instruction, thereby contributing to increased student achievement. The study will survey online a population of teacher participants in school

- based professional learning communities and interview principals face to face about the context a nd their perceptions of the initiative, pre

- and post

- implementation. Data collection from teachers will focus on what the professional learning communities do, how they operate, and to what extent they produce the outcomes expected of them as framed by si x conceptual attributes of professional learning communities and five specific tasks. Data collection from principals will focus on contextual information about school culture and conditions such as resources that support implementation. Teachers and princ ipals will also provide their reflections on the challenges of implementing professional learning communities and their suggestions for improvement. The analysis will enable comparisons among professional learning communities within and across schools, and between teachers’ pre

- implementation expectations and post

- implementation experiences. Study findings are expected to inform both theory and practice related to implementation of professional learning communities. This submission requests approval to recr uit principals and teachers for the study and conduct in

- person interviews with principals and online surveys with teachers in the study district.



## • **Circumstances Necessitating the Collection of Information**

• **Statement of Need to Evaluate District -** **Wide Impleme ntation of a Professional**  **Learning Community Initiative** The specific legislation authorizing this data collection is specified in Part D, Section 174 (20 U.S.C. 9564) of the Education Sciences and Reform Act (ESRA) of 2002. Part D, Section 174 provides for ED to enter into five

-year contracts with entities to e stablish a networked system of ten regional laboratories that serve the needs of each region of the United States. The Regional Educational Laboratories (RELs) are to carry out a range of activities to serve the needs of each region in the United States, i ncluding applied research, and development, dissemination, training, and technical assistance activities that focus on how to use data and analysis. The primary mission of the RELs is to help states and districts systematically use data and analysis to ans wer important issues of policy and practice with the goal of improving student outcomes. Accordingly, t his project was conceived in response to a specific request made of the REL MA from a superintendent of a large school district in Pennsylvania (West Che ster Area School District, abbreviated as WCASD) to assist in the evaluation of the implementation of professional learning communities in the district during the 2013

- 2014 school year. Practitioners in the education field have widely embraced professional learning communities (PLCs) because they are aligned, at least in theory, with several features of effective professional development, such as active learning and teacher collaboration (for a review, see Stoll, Bolam, McMahon, Wallace, &Thoma s, 2006). These collaborative teams are believed to be effective because they give teachers the

opportunity to be exposed to new ideas and practices, and improve their pedagogy via a process of critical reflection (Hord, 1997; Wood, 2007). O perationally sp eaking, PLCs are teams of educators (most commonly, teachers) that meet regularly, either in

- person or virtually, often but not always during scheduled school time, to develop lesson plans, examine student work, monitor student progress, assess the effecti veness of instruction, and identify needs for professional learning. These teams operate with a commitment to the norms of continuous improvement and experimentation, and engage their members in improving their daily work to advance the achievement of dist ricts’ and schools’ goals for student learning (National Staff Development Council, 2001). Ideally, PLCs should be characterized by a set of dimensions or attributes such as shared beliefs, values, and vision; shared and supportive leadership; supportive s tructural conditions; supportive relational conditions; collective learning; and peer sharing (DuFour, 2005; Hord, 1997). Further, it is generally acknowledged that strong PLCs are not only those in which new knowledge regarding content and pedagogy is acq uired, but also those in which existing assumptions about teaching and learning are challenged and critiqued (Little, 2003). The increasing interest of education stakeholders in PLCs also appears to reflect a belief in the effectiveness of these learning c ommunities. Recognizing the value of building systems for peer

-to
- peer professional learning among teachers and other education leaders, the U.S. Department of Education is investing in and supporting PLCs as part of several key initiatives, including Investing in Innovation ( i3 ), the Office of Special Education Programs’ technical assistance centers, Promise Neighborhoods, and Race to the Top . Thus, there is strong support among school administrators and policymakers alike for expanding this practice (Talb ert, 2010). Unfortunately, studies that evaluate the effects of PLCs on teacher professional learning and/or student learning outcomes are limited in number. A recent review (Vescio et al., 2008) identified a mere 11 studies that examined the effectiveness of PLCs and observed that few studies have moved beyond teachers’ self

- reports. However, the researchers found cumulative evidence that is consistent with the hypothesized effects of PLCs . They concluded that participation in PLCs influences teaching practice as teachers move toward student

- centered instruction, and that when teachers participate in a learning community, students benefit as well, as indicated by improved achievement scores over time. Similarly, a recent meta
- analysis found that PLCs have a small but statistically significant positive effect on student achievement (Lomos, Hofman, & Bosker, 2011). Thus, while the majority of studies that evaluated the efficacy of PLCs have important methodological limitations, the limited evidence from progra ms that were more systematically evaluated seems to support the connection between teachers’ conscious efforts to improve instructional practice through group/peer study and changes in classroom practice and student achievement ( Vescio et al ., 2008). We ex pect that in addition to serving WCASD

- specific information needs, this study will help close the important theoretical and empirical gap in the current literature by addressing the question of how and under which conditions (e.g., leadership style, group norms, available resources, and work arrangements) PLCs get in a position to achieve key desired attributes of collaborative work and meet standards of productivity that have the potential to enhance student learning and achievement.



### • **Research Questions** The primary research questions to be addressed in the study are listed below. The questions were formulated with direct input from the leadership of WCASD to ensure that the study meets their specific needs. However, these questions are also informed by th e existing gaps in the current literature regarding the “how and under what conditions” focus discussed in the previous section.• Research Question 1 : How do PLC teams implement the study district’s PLC tasks? That is, what

collaborative teamwork routines do they develop to achieve the key attributes of PLCs and the key tasks listed in the study district’s PLC materials?

• Research Question 2 : How do the study district’s teachers and principals evaluate their experience with the program? Is their actual PLC e xperience the same or different from what they expected it to be before the implementation of the program? How do teachers’ evaluation of the program and of the implementation compare with those of principals?

• Research Question 3 : What specific artifacts (e.g., essential learning targets, standardized common assessments, and systematic interventions) are produced by each PLC team in the district?


### c. **Study Overview** In response to a specific request from the superintendent, REL MA will evaluate the implement ation of WCASD’s district

-wide PLC initiative during the 2013

- 2014 school year. As a result of its planning process, WCASD produced the WCASD PLC Protocol that explains what PLCs are, how and why they will be integrated into the district, and the specific sequence of five tasks that each PLC team should accomplish over time, regardless of grade level and content area. The protocol was shared with all educational staff in the district but does not specify how PLC teams should be established or what work rout ines they should adopt. Rather, this decision is left up to the principal and teachers of each school in order to accommodate the unique set of circumstances and resources that exist within each school and to leverage more effectively the expertise and exp erience of current teachers and administrators. Because PLC teams in the district will not follow a uniform, fully structured implementation design, the proposed study has two main goals: to describe the process of implementing PLCs in WCASD and to assess and compare the progress that these PLC teams make on specific tasks related to collaborative professional learning and instruction during the first year of implementation. Specifically, the study will involve collecting survey data from WCASD’s teachers and interview data from WCASD’s principals to: describe and compare (a) the work routines of teachers in PLC teams within and across schools in the district and (b) the specific outcomes (or products) of their collaborative work. REL MA will also ask teac hers and principals to evaluate both their experience with specific aspects of the implementation
– such as team work, productivity, and professional development
– and lessons they learned from that experience. This information will permit WCASD to identify evidence

- based best practices that can support the ongoing implementation of the district’s PLC program and its impact. This information will also permit WCASD to identify actual and/or perceived barriers to implementation during the first full year of i mplementation that the district may want to address in subsequent years of the implementation. To be efficient, REL MA proposes to use the same survey data collected from individual teachers, at the aggregate, to assess and compare the progress that PLC teams in the district make on achieving the stated goals of the program during the first full year of implementation. First, the data collected from individual teachers who are members of the same PLC team will be aggregated to represent measures of PLC te am - level variables of interest such as communication and leadership style in the group. Next, the aggregated data (with PLC teams being the units of analysis) will be analyzed in relation to a logic model(see Figure 1) which REL MA developed with input from the district based on a scheme proposed by

Nelson and colleagues (Nelson, Cordray, Hulleman, Darrow, & Sommer, 2012) that inquires about the extent to which core components of the program have been implemented. The model is designed to benchmark the “PL C status” (as beginning, evolving, or mature: see Grossman et al., 2000) of each PLC team in the district based on a measurement scale that is theoretically derived from the six key desired attributes of effective PLCs (i.e., shared mission, collaborative culture, collective inquiry, action orientation, commitment to continuous improvement, and results orientation) as identified in the extant literature on the topic (see DuFour et al., 2010; Hord, 1997) and as also referenced in WCASD’s PLC protocol. The sh ort
-, medium

-, and long

-term outcomes listed in the logic model specify the characteristics PLC teams should exhibit as they evolve over time from a beginning status in the short term to a mature status in the long term. The model also specifies the five s equential tasks that PLC teams in the district were asked to complete. These are: (a) formulation of essential learning targets, (b) development of common formative assessments that measure proficiency on essential learning targets, (c) modifications of th e core instructional program to allow differentiated instruction by degree of student proficiency on one or more learning targets, (d) development of supplemental interventions for students who do not demonstrate proficiency on one or more learning targets through the core instructional program, and (e) formation of child study teams to coordinate more intensive and more frequent individualized interventions for students who do not meet proficiency on one or more learning targets through the core instructio nal program and supplemental interventions. In all, using the logic model as an analytical tool will allow the study team to answer the research questions while providing WCASD with valuable insights about the progress of the PLC initiative district

-wide. Data for the study will be collected through a post

- implementation online survey of all teachers (N=930) in all of WCASD’s 16 schools, and through semi

- structured interviews with all principals in the district (N=16) post

- implementation. REL MA will desig n and administer the post

- implementation survey of teachers and will also design and collect the post

- implementation interview data from principals. All WCASD’s teachers and principals will be involved in data collection activities because PLCs reside in i ndividual schools and depend to a large extent on school

- level supporting conditions. Teachers are the members of PLCs; it is their expectations and perceptions we wish especially to discern. This study will enable a fine

- grained assessment of their experi ences post
- implementation as well as how their experience compares to that of other team members (to this end we will use a unique identifier to link individual teachers to PLC teams). Principals are included because they are on

-the

- scene observers and gat ekeepers who control access to critical resources for implementation, and because they are key stakeholders who are deeply invested in the outcomes of teacher quality and student achievement. Importantly, because PLCs in WCASD will be given complete freedo m to organize, it is very likely that teams’ composition and work routines (and therefore the PLC experience of individual teachers and principals) will vary within and among schools in the district. Therefore, it is not feasible to limit data collection t o a random sample of teachers and administrators in light of the goal of providing adequate representation of their experiences with implementation in the absence of a well

- defined sampling frame. The specific data collection plan with the timeline for all data collection activities is included in Table 3.

**Figure 1: Logic Model for Evaluation of a District -** **Wide Implementation of a Professional Learning Community Initiative** **Inputs Outputs**  Outcomes Activities Participation Short

- term Medium

- term Long
- term Principals facilitate the process of determining grade

and subject

specific PLC teams within each building, such that all members on any given PLC team share learning outcomes (that is, teachers must teach the same academic standards in order to be on the same PLC team). Each PLC team is provided with a copy of the district

wide PLC protocols. Each PLC team meets for approximately 45
– 60 consecutive minutes weekly. PLC teams have access to expertise from external sources, such as program supervisors, co lleagues from other schools, and university professors. PLC teams organize independently to achieve key attributes of PLCs (shared mission, collaborative culture, collective inquiry, action orientation, commitment to continuous improvement, and results orientation) that are described in the PLC protocols.

Ta sk 1: Collaborate to determine 3
–5 essential learning targets for each unit taught as part of the core instructional program. Task 2: Design two versions of a common formative assessment for each unit taught to assess students’ proficiency in the essentia l learning targets. Task 3: Modify core instructional program based on assessment results to allow differentiated instruction.
Task 4: Develop supplemental interventions for nonproficient students.

Task 5: Form child study teams to coordinate more intensive and more frequent interventions for All WCASD

teachers in their respective PLCs

All WCASD teachers of a given subject and their program supervisor

PLC teams working independently to create the assessments and then reaching consensus with other subject

specific PLC teams

PLC teams

PLC teams

PLC teams (potentially, multiple teams involved) • Regular and active participation in PLC activities
• Positive evaluation of the PLC experience
• Positive beliefs about the professional development (PD) value of PLCs
• Positive beliefs about the impact of PLCs on students

• Teachers positively evaluate and ag ree with the essential learning targets.

• Teachers positively evaluate the quality of the common formative assessment created by the team.

• Teachers analyze student assessment on formative common assessment to identify need in differentiated instruction.

• Teachers analyze student assessment on formative common assessment to identify nonproficient students. • Continued

regular and active participation in PLC activities
• Personal commitment to collaborative work
• Personal commitment to PD and learning collaboratively
• Translation of (or inte ntion to translate) knowledge acquired to actual classroom practice

• Teachers use essential learning targets to guide instruction activities.

• All teachers make standard and consistent use of the common formative assessment.

• Teachers assign students to receive differentiated instruction based on analysis.

• Teachers assign students to receive supplemental interventions based on analysis.

• Teachers form child study • Sustainability of

PLCs in WCASD
• Shared professional culture that values collaboration and learning in WCASD
• WCASD instrumental in diffusing the practice to o ther educational entities in the region and beyond

• As a matter of routine, teachers revise and update essential learning targets as necessary.

• As a matter of routine, teachers revise and update the common formative assessment as necessary.

• As a matter of routine, teachers review student progress and reconsider differentiated instruction assignment.

• As a matter of routine, teachers review student progress and reassess need for intervention (as well as the intervention itself).

• As a matter of routine, teachers review student progress andstruggling students. • Teachers analyze student asses sment on formative common assessment to identify struggling students. teams to design

individualized interventions for struggling students. reassess need for

intervention (as well as the intervention itself).



## • **Purposes and Uses of Data** Rutgers University’s Center for Effective School Practices will analyze all project data for the REL MA Evaluation of a District

- Wide Implementation of a Professional Learning Community Initiative under contract number ED

- IES
- 12
- C
- 0006. The primary purpos e of the study is to evaluate the implementation of the district

- wide PLC initiative in the WCASD in response to a specific request from the superintendent. The proposed study has two main goals: to describe the process of implementing PLCs and to assess a nd compare the progress that PLC teams in the district make on specific tasks related to collaborative professional learning and instruction during the first year of implementation against the desired progress as per the program’s logic model. Findings f rom this study will be used by WCASD to formulate an operational definition of each attribute of PLCs and assess PLC teams’ progress toward achieving each one of these attributes, as well as progress toward completing the tasks identified in the district’s PLC protocol. The insights generated by the study will also assist the district’s leadership in considering programmatic modifications that can enhance the productivity and sustainability of PLCs and provide baseline data for continuing efforts to monitor their functioning. The close collaboration between the district and study team is also expected to help build the district’s internal capacity to collect, manage, and use program data more efficiently based on the instruments and methodology used in the s tudy. It is also reasonable to expect that study findings will inform similar efforts by other districts that are members of the Professional Learning Research Alliance (including the West Amwell Township School District, the South Hunterdon School District, the Monroe Township School District), as well as other stakeholders across the mid

-Atlantic region which are not as far along as WCASD in planning and implementing PLCs or which are implementing but lack evidence

- based information for adjusting their PLC impl ementation. The data will also inform efforts by state departments of education (starting with the Departments of Education in New Jersey and Pennsylvania, which are also members of the Professional Learning Research Alliance) to support the planning and i mplementation of PLCs statewide. REL MA has already prepared a review of the literature on online and hybrid PLCs in response to strong interest from research alliance members. Finally, it is likely that the findings of this study will be used by other fe deral agencies and a national audience of practitioners and scholars who are interested in PLCs. The description of how and under which conditions (e.g., leadership style, group norms, available resources, and work arrangements) will shed light on positioning PLCs to achieve key desired attributes of collaborative work, as well as to meet standards of productivity that have the potential of enhancing student learning and achievement, and will narrow a conceptual and theoretical gap in the field.

## • **Use of Technology to Reduce Burden**

Teacher data will be collected online through Survey Monkey. All teachers at WCASD have email and internet access through their district

- supported accounts. Teachers will use these accounts to receive email communication about the study, access the surveys, and submit their electronic responses. Data from the semi

- structured interviews of principals cannot be collected throu gh such methods as web surveys. Computer

- assisted telephone interviews are not optimal for collecting these data. The proposed face

-to
-face interviews will be necessary to allow in

- depth, conversational exchanges with respondents, and to obtain answers to both open

- ended and detailed questions. However, if repeated attempts at scheduling these two

- hour interviews during regular work hours fail, the REL MA study team will consider using telephone and/or computer

- assisted telephone interviews (e.g., via Skype ), providing that these can be audio

- recorded for the purpose of transcribing the interviews.



## • **Efforts to Avoid Duplication** Prior to designing this study, REL MA produced a literature review on online and hybrid PLCs. REL MA has continued to search for rel evant literature and to augment its literature base in the course of designing the study and gaining requisite approvals from OMB and the IRB at Rutgers University. Previous reviews of the literature on PLCs (e.g., Hord, 1997; Stoll & Louis, 2007; Wood, 20 07) are based on findings from case studies. A recent review (Vescio et al ., 2008) identified a mere 11 studies that examined the effectiveness of PLCs and observed that few studies have moved beyond teachers’ self

reports. REL MA’s ongoing review of the l iterature reveals that no in

- depth study has been conducted of a single district’s implementation of PLCs that details the experience of teachers with qualitative and quantitative measures of attitudes and productivity, compares pre

- implementation expectat ions with post
- implementation experience, and provides in

- depth analysis of differences among PLCs in similar contexts. Thus, the type of PLCs implementation evaluation proposed for this study has not been conducted to date.



## • **Methods to Minimize Burden on Small Entities**  The study will utilize validated survey instruments and efficient interview protocols that focus on the data of most interest to the district and other audiences. WCASD will enable teachers to respond to the survey during periods that are al ready dedicated to school

-wide and district

- wide work. In terms of principal interviews, principals will be offered flexible scheduling to avoid potential conflicts or disruptions to their routine work.



## • **Consequences of Not Collecting Data** This is one

-time collection. WCASD has no means other than the proposed data collection to track and assess the implementation of a major initiative that has the potential to deliver effective professional development for all teachers in the district and improve learning outcomes for all students in all of the district’s schools. The rich data collected through this project will generate valuable insights about the process of implementing PLCs in the district (including potential and perceived barriers to implementation) a nd will provide the baseline for assessing the impact of this program both in the short and long run. There has been no previous comprehensive study of PLCs in WCASD so there is no alternative source for the information to be collected. Moreover, the imple mentation of PLCs in WCASDpresents a rare research opportunity (a natural experiment of a sort) to study in depth the dynamic

nature of PLCs and the implementation process, thus making a significant contribution to current theory and evidence

- based practi ce regarding PLCs and their impact on teachers’ professional development and students’ learning outcomes.



## • **Special Circumstances** There are no special circumstances associated with this data collection.



## • **Federal Register Announcement and Consultation**

• **Federal Register Announcement and Consultation** A 60
- day notice to solicit public comments was published on October 31, 2013. There are no public comments received to date.



## • **Consultations Outside the Agency** The study team has worked with IES to identify the five members listed below to serve as the Technical Working Group (TWG) for this study.

• Robert Boruch University of Pennsylvania P: 215
- 898 0409
• Laura Hamilton Senior Behavioral Scientist, RAND P: 412
- 683
- 2300, x4403
• Chris Hulleman James Madison University P: 540
- 568
- 2516
• Andrew Porter University of Pennsylvania P: 215
- 898
- 7014
• Christopher Rhoads University of Connecticut P: 860
- 486
- 3321
These experts on research methods, data analysis, implementation, and teaching and learning have provided input on the study’s design. Based on their input, the original emphasis on conducting process and outcomes evaluation of WCASD’s PLC initiative has b een replaced with a focus on the process of implementation and the degree to which core elements of the program are implemented. This approach has informed the logic model of the evaluation as well as the overall research methodology which combines quantit ative and qualitative data collection.



## • **Payments or Gifts** The study will not provide respondents with incentives for participation and we will not give payments or gifts to schools or to the district for providing access and study data. The superintendent of WCASD requested this study. The district will administe r the pre

- implementation online survey as well as the recruitment of all study participants overall. The superintendent has enlisted the support of principals in making time available for teachers to participate during the regular school hours; we plan to interview principals during regular school hours as well, unless they request otherwise. .

## • **Assurances of Confidentiality**

The data collection efforts that are the focus of this clearance package will be conducted in accordance with all relevant regulations and requirements, including the following:

• The Privacy Act of 1974, P.L. 93

- 579 (5 U.S.C. 552a).
• The Family Educational and Rights and Privacy Act (FERPA) (20 U.S.C. 1232g; 34 CFR Part 99).
• The Education Sciences Reform Act, P.L. 107

- 279 (20 U.S.C. 9573). Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for statistical purposes. The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific district, school, or individual. We will not provide information that identifies you, your district or your school to anyone outside the study team, except as permitted by law. REL MA (specifically, Rutgers Univ ersity’s Center for Effective School Practices, a subcontractor to ICF International) will protect the confidentiality of information for the study and will use it for research purposes only. The project director, Dr. Blitz, will ensure that individually i dentifiable information about study members remains confidential. When reporting the results, data will be presented in aggregate form only so that individuals and schools will not be identified. The following safeguards, which are routinely employed by Rutgers University’s Center for Effective School Practices to carry out confidentiality assurances, will be applied consistently during the study:

• A list of study team members will be forwarded to NCEE.
• All employees sign a confidentiality pledge (Appen dix A), which describes both the importance of and the employee’s obligation to discretion. All study team members who will have access to study participants or data will sign the pledge and will obtain appropriate clearance.
• The plan for maintaining conf identiality also includes that all personnel with access to individual identifiers provide notarized nondisclosure affidavits and undergo training regarding (1) the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses; (2) controlled and protected access to computer files under the control of a single database manager; (3) built

- in safeguards concerning status monitoring and receipt control systems; and (4) a secured and operator
- controlled, in

- house computing facility.
• Procedures will be followed to revoke clearance in a timely fashion from members who leave the study team.
• REL MA staff have obtained clearance through NCEE’s security clearance office.
• Respondents’ personally identifiable information is maintained on separate forms and files, which are linked by sample identification number only.
• Access to hard copy documents is strictly limited. Documents are stored in locked files and cabinets, and discarded materials are shredded.
• Access to computer data files is protected by secure user names and passwords, which are available to specific users only. REL MA will also incl ude appropriate statements about protecting confidentiality in the requests for data collection from participants and other materials generated by the study team. Those documents include:• Post
- Implementation Interviews of Principals (see Part B, Appendix D)

• Post
- Implementation Online Survey for Teachers (see Part B, Appendix E) The following statements will appear in these documents to assure respondents about preserving confidentiality:

• “Responses to the data collection activities will be used for rese arch purposes only. The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific school or individual. We will not provide information that identifies you or your school to anyone outside the study team, except as required by law”.
• “The contractor follows the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). The contractor will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released. Information on respondents will be linked to their institution but not to any individually identifiable i nformation. No individually identifiable information will be maintained by the study team”.



## • **Justification for Sensitive Questions** There are no questions of a sensitive nature in the district interviews.



## • **Estimates of Burden Hours** Table 2 shows the estimated burden hours for principals and teachers who will participate in data collection. These estimates are based on REL MA’s experience collecting such data from principals and teachers for similar studies, and on pre

-testing in othe r school districts. REL MA pre

-tested the interview protocols with three principals and the online survey with four teachers from a school district in NJ represented in the PLRA. The average completion time for both online surveys was about 30 minutes. Ann ualized burden estimates over two years are 403 respondents (790 + 16 = 803 ÷ 2 = 403) and 214 minutes (395 + 32 = 427 ÷ 2 = 214). **Table 2. Estimated Response Time for Data Collection**

Respondent/Data Request Number of Targeted Respondents Expected

Response Rate (%) Number of

Respondents Unit

Response Time (Hours) Total

Response Time (Hours) **Principals** Post
- implementation In

person interview 16 100% 16 2 32 **Teachers** Post
- implementation Online survey 930 85% 790 0.5 395 **Annualized Totals 403**  **214**

## • **Estimates of Cost Burden to Respondents**

There are no capital or start

- up costs for respondents. There will also be no additional operational, maintenance, or purchase of service costs for respondents. WCASD already has computers for staff use and has established email accounts and Internet acces s for all teachers and principals. The district also has included an annual account for Survey Monkey in its budget.



## • **Annualized Cost to the Federal Government** The estimated cost of the study to the federal government is $280,722 over two years, representi ng an average annual cost of $140,361. The estimated total cost of the REL MA five

-year contract is $32,353,087, of which this study accounts for $280,722. Study costs were estimated using fully

burdened (“loaded”) hourly rates for staff, multiplied by the estimated number of hours of labor, and adding other direct costs. Multipliers for indirect rates were applied to direct costs.



## • **Reasons for Program Changes or Adjustments** This is a new data collection.



## • **Plans for Tabulation and Publication of Results Tabl**  **e 3. Timeline for Data Collection and Analysis 2014**  **1 2**  **3 4**  **5 6**  **7 8**  **9 10**  **11 12**  Administer post

- implementation online survey of teachers Analyze post

- implementation online teacher survey data Conduct Principal post

implementation interviews Analyze of principal interviews Draft report Final report and WCASD debrief



### • **Tabulation Plans** REL MA will use the information gathered from the teacher survey and principal interviews to examine and evaluate district

- wide implementation of a PLC initiative. Once the teacher survey dataset is ready for analysis, the study team will explore the distr ibution of individual responses on all variables of interest for irregular patterns (outliers, missing values, and deviation from normality for continuous variables) and make adjustments as needed. Next, the study team will compare responses across subgrou ps of teachers who teach at different levels (elementary, middle or high school), different content area, as well as teachers from different departments and schools, to identify potential sources of bias in our data. This is done for data diagnostic purpos es and will not be included in public reports. As a rule, data will be reported at the aggregate to protect individuals’ identity.Audio recordings of principal interviews will be transcribed and the analyzed using Dedoose. Special

attention will be given to extracting insights into barriers and facilitators to implementation. The content of teachers’ responses to the open

- ended questions in post

- implementation online survey will be downloaded and then content analyzed. We will use Dedoose to identify and extract teachers’ reports of most positive and most negative PLC experiences, and to synthesize their recommendations for improving the implementation of PLCs. Research Question 1 (how do PLC teams implement the study district’s charge?) will be answered p rimarily through analyzing the post

- implementation survey data collected from teachers in Year 2 of the project. The survey will include several items (see Part B, Appendix E) that ask teachers to report retrospectively on aspects of the work routine of th e PLC in which they are a member (e.g., degree to which the team meets regularly, level of active participation in the group, group dynamics, and leadership style within the group).

• Responses (on a Likert scale) provided by teachers who are members of a s pecific PLC team (as ascertained by the team’s unique study identifier) will be summed and averaged to produce team
- level measures of these variables.

• These variables will be analyzed next using cluster analysis to develop taxonomies of work routines as t hey emerge from this exploratory procedure. The study team will determine the specific cluster analysis algorithm to be used once there is an opportunity to assess the specific properties of the data collected from teachers.

• Next, the study team will use findings from content analyzing teachers’ responses to the open

ended questions on the post

- implementation online survey (where they describe the most positive and the most negative PLC experience they had and recommend possible improvements) to delineate common challenges/barriers that PLC teams encounter as members collaborate to establish work routines. The data analysis approach to answering Research Question 2 (how do WCASD teachers and principals evaluate their experience with the program compared to their initial expectations ) will also utilize teachers’ responses to the online survey, specifcaly the variables measuring expectations, attitudes, evaluations, and self

- reported behaviors.

• Findings from teachers’ responses to the open

- ended reflection q uestions on the post

implementation survey will also be used to answer this question, either by corroborating and/or clarifying themes that emerge from the teacher surveys or adding new insights that were not captured through these surveys. To answer this research question regarding the principals, the study team will analyze qualitative findings from the post

- implementation interviews with principals.

• Audio recordings of the interviews will be transcribed verbatim and checked against the original audio recording for accuracy.

• Dedoose, a web

- based application for managing, integrating, and analyzing qualitative and mixed methods data, will be used for data analysis. A grounded theory approach will be employed to explore a priori and emergent themes regard ing principals’ evaluation of the program. The a priori themes will be defined according to the explicit themes explored in the interview (e.g., role of principal in implementation and allocation and use of school resources to support PLC work). It is anti cipated that emergent themes will be focused on facilitators and barriers to implementation as well as on the impact, if any, PLCs had on teachers, students, and/or the school in the course of the first year of the program.• An extensive list of codes and their definitions that relate to each theme will be derived from the interview data. Two independent coders will use the coding scheme to analyze transcripts of four randomly selected interviews and agreement among coders (intercoder reliability) will be a ssessed using Cohen’s Kappa. The coding scheme will be adjusted to eliminate disagreements among coders and improve overall reliability (Cohen’s k > .85).

• All codes will be entered into Dedoose and tagged to their associated segments of text for all interviews. Text segments will then be sorted by codes and reviewed by the coding team to identify emergent themes and to identify recurring patterns of responses to assess prevalence of themes.

• In the final step of the analysis, the study team will examine re lationships among themes by looking at associations between differentially coded content. Lastly, the study team will compare key findings about teachers’ perspective on the PLC initiative and its implementation to key findings that emerge from the analys is of the interview data obtained from principals. The study team will note similarities and differences in perspectives as a way of assessing possible changes to school culture post

- implementation. To answer Research Question 3 (specific products produc ed by PLC teams), the study team will analyze the survey data for teachers’ reports about artifacts produced by their team.

• Specifically, teachers will be asked to indicate the number of instructional units for which the team designed (1) essential learni ng targets, (2) core formative assessments, and (3) assessment rubrics. They will also be asked to indicate if they shared these artifacts with stakeholders (students, parents, and principal) and whether these artifacts were actually used with students in the course of the first year of implementation. The study team will also reference principals’ assessment of artifacts produced by PLC teams in their school based on analyzing the interview data collected from all principals in the district at the end of t he first year of full implementation of the PLC initiative.



### • **Publication Plans** REL MA will produce a “What’s Happening” report of 15 to 30 pages at the conclusion of the project
. The study team will present findings in complementary narrative and tabular formats, and will include user-friendly graphic depictions (i.e., charts and graphs) to highlight and contextualize descriptive findings. The study findings will be presented wit hout reference to any names of individuals or specific schools, in order to avoid deductive disclosure of study respondents. The final report, scheduled for release in November, 2014, on the IES and REL MA websites, will be based on data from the pre

impl ementation and post

- implementation teacher surveys and principal interviews. We anticipate that the “What’s Happening” report will include:

• A brief description of the study district’s PLC initiative and implementation objectives

• A discussion of the evaluat ion approach and logic model

• Presentation of key findings from the analysis of quantitative and qualitative data

• Discussion of study limitations, the appropriate interpretation of findings, and implications for current knowledge about PLCs, as well as for planning, implementation, and evaluation of PLCs

• A study methodology appendix.REL MA will conduct a one - day debriefing to share the findings with the district’s leadership and to

facilitate a discussion on the potential programmatic implications of these findings. REL MA will also conduct a separate webinar to disseminate the study results with a broader audience and will use its dissemination network to alert regional stakeholders about the final report and webinar opportunities. REL MA will also work wi th members of the Professional Learning Research Alliance to identify ways in which they may use this report to guide their current and future professional learning activities. The study team will present this work at academic and professional conferences and at meetings involving Research Alliance members and other regional stakeholders. Finally, the study team anticipates producing several peer

- reviewed journal articles based on this project, including a paper to present the logic model used in the project, a paper that will discuss the validation of the survey instrument used in this project, and a paper that introduces the overall methodological approach to the evaluation of the implementation, in addition to papers that describe and discuss the key find ings of the evaluation.



## • **Approval Not to Display the Expiration Date for OMB Approval** The study will display the OMB expiration date.



## • **Exception to the Certification Statement** ED is not seeking an exception.


## References Baenen, N., & Jackl, A. (2010, June). Evaluation of central services professional learning teams as of spring 2010. Eye on Evaluation . E&R Research Report No. 10.06. Raleigh, NC: Wake County Public School System. Delaware Department of Education. (2012). DDOE/Data coac h project. Retrieved from www.doc.k12.de.us/tleu_files/PLCSurveyReport_2012.pdf

. Drummond, K., Chinen, M., Duncan, T.G., Miller, H.R., Fryer, L., Zmach, C., & Culp, K. (2011). Impact of the Thinking Reader® software program on grade 6 reading vocabulary, comprehension, strategies, and motivation (NCEE 2010

-4035). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U. S. Department of Education. DuFour, R., Eaker, R., & DuFour, R. (2005). Recurring themes of professional learning communities and the assumptions they challenge. In R. DuFour, R. Eaker, & R. DuFour (Eds.), On common ground: The power of professional learning communities (pp. 7
–30). Bloomington, IN: National Educational Services. DuFour, R., DuFour, R., Eaker, R., & Many, T. (2010). Learning by doing: A handbook for professional learning communities at work (2 d ed.). Bloomington, IN: Solution Tree. Grossman, P., Wineburg, S., & Woolworth, S. (2000). What makes teacher community different from a gathering of teachers? Seattle, WA: Center for the Study of Teaching and Policy.Hall, G., & Loucks, S. (1979). Implementing innovations in schools: A concern s - based approach. Austin,

TX: Research and Development Center for Teacher Education. Hemphill, J. (1956). Group dimensions: A manual for their measurement. Research Monograph No. 87. Columbus, OH: Ohio State University, Bureau of Business Research. Heppen, J.B., Walters, K., Clements, M., Faria, A., Tobey, C., Sorensen, N., and Culp, K. (2012). Access to Algebra I: The Effects of Online Mathematics for Grade 8 Students. (NCEE 2012
–4021). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Hord, S. M. (1997). Professional Learning Communities: Communities of Continuous Inquiry and Improvement. Austin, TX: Southwest Educational Development Lab. Killion, J. (2006). Collaborative professional learning in school and beyond: A tool kit for New Jersey educators . New Jersey Department of Education Office of Academic and Professional Standards and the New Jersey Professional Teaching Standards Board in cooperation with the Na tional Staff Development Council. Little, J. (2003). Inside teacher communities: Representations of classroom practice. Teachers College Record, 105(6), 913
–945. Lomos, C., Hofman, R. H., & Bosker, R. J. (2011). Professional communities and student achieve ment: A meta
- analysis. School Effectiveness and School Improvement, 22(2), 121
– 148. Martin, T., Brasiel, S. J., Turner, H., & Wise, J.C. (2012). Effects of the Connected Mathematics Project 2 (CMP2) on the Mathematics Achievement of Grade 6 Students in the Mid

-Atlantic Region (NCEE 2012
-4017). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. National Staff Development Council. (2001). Standards for staff development . Oxford, OH: Author. Retrieved from http://www.nsdc.org/standards/learningcommunities.cfm Nelson, M. C., Cordray, D. S., Hulleman, C. S., Darrow, C. L. , & Sommer, E. C. (2012). A procedure for assessing intervention fidelity in experiments testing educational and behavioral interventions. Journal of Behavioral Health Services Research , 39(4): 374
–96. Office of Management and Budget (2006). Questions and answers when designing surveys for information collection . Washington, DC: Author. Olivier, D.F. & Hipp, K.K. (2010). Demystifying professional learning communities: School leadership at its best . Lanham, MD: Rowman & Littlefield Education. Stoll, L., Bola m, R., McMahon, A., Wallace, M., & Thomas, S. (2006). Professional learning communities: A review of the literature. Journal of Educational Change, 7 , 221
–258. Stoll, L., & Louis, K. S. (Ed.) (2007). Professional learning communities: Divergence, depth and dilemmas. Berkshire, UK: Open University Press. Talbert, J. E. (2010). Professional Learning Communities at the Crossroads: How Systems Hinder or Engender Change. In Hargreaves, A., et al. (eds.). The Second International Handbook of Educational Change, Springer International Handbooks of Education , Volume 23, Dordrecht, Springer, 555
–571. Vescio, V., Ross, D., & Adams, A. (2008). A review of research on the impact of professional learning communities on teaching practice and student learning. Teaching and Teacher Education, 24(1) , 80
– 91.Wijekumar, K., Hitchcock, J., Turner, H., Lei, PW., and Peck, K. (2009). A Multisite Cluster Randomized

Trial of the Effects of CompassLearning Odyssey® Math on the Math Achievement of Selected Grade 4 Students in the Mid

-Atlantic Region (NCEE 2009

-4068). Washington, DC: National Center for Education Evaluation and Regional Ass istance, Institute of Education Sciences, U.S. Department of Education. Wood, D. R. (2007). Professional learning communities: Teachers, knowledge, and knowing. Theory Into Practice, 46(4), 281
–290.



## Appendix	A:	Confidentiality	Pledge I understand that the names, and any other identifying facts or information, of individuals, businesses, organizations, and families participating in projects conducted by the Center for Effective School Practices at Rutgers University or its subsidiaries are confidential information. I agree that I will not reveal such confidential information, regardless of how or where I acquired it, to any person unless such person has been authorized by the Director or Project Manager at Center for Effective School Pra ctices at Rutgers University to have access to the information. I further understand that the unauthorized access to, use, or disclosure of any confidential information is a breach of the terms of my employment, or my consultant agreement with Center for Effective School Practices at Rutgers University and may subject me to court action by any interested party or to other sanctions by Center for Effective School Practices at Rutgers University. I acknowledge that this agreement shall continue to bind me ev en after the project(s) is (are) completed and/or even though my employment or my consultant agreement with Center for Effective School Practices at Rutgers University has terminated. In addition, in the course of my employment I may have access to person al information, electronic and otherwise, about fellow employees. I agree that I will treat that information as having the highest confidentiality, and not communicate it to fellow employees or others outside Center for Effective School Practices at Rutger s University. Final determination of whether or not there is a business purpose requiring that I access a fellow employees’ records will be made in consultation with the Director of Human Resources. Failure to uphold this standard is a breach of trust and may subject me to disciplinary action, including termination of employment. Other than in the course of my authorized employment or my consultant agreement, I further agree that I will not use, nor facilitate the use by any third party, in any way any in formation deemed confidential by the terms of any contract or other written agreement between Center for Effective School Practices at Rutgers University and any other organization, except by written authorization by both parties. It is my understanding th at Center for Effective School Practices at Rutgers University and the contracting organization(s) have the exclusive right to all information acquired or developed under such a contract or other written agreement. I acknowledge that I acquire no right, ti tle, or interest in and to any data or information to which I have access by reason of my employment or my consultant agreement and that I may not remove such data from my assigned work location without prior authorization.Name: Signature: Date: