OMB Package Part A: Evaluation of a 

District-Wide Implementation of a 

Professional Learning Community 

Initiative

September 2013
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative ii

Table of Contents

PART A. JUSTIFICATION 1

1. Circumstances Necessitating the Collection of Information 1

a. Statement of Need to Evaluate District-Wide Implementation of a Professional Learning 

Community Initiative 1

b. Research Questions 2

c. Study Overview 3

2. Purposes and Uses of Data 6

3. Use of Technology to Reduce Burden 6

4. Efforts to Avoid Duplication 7

5. Methods to Minimize Burden on Small Entities 7

6. Consequences of Not Collecting Data 7

7. Special Circumstances 7

8. Federal Register Announcement and Consultation 8

9. Consultations Outside the Agency 8

10. Payments or Gifts 8

11. Assurances of Confidentiality 8

12. Justification for Sensitive Questions Hours 10

13. Estimates of Burden Hours 10

14. Estimates of Cost Burden to Respondents 10

15. Annualized Cost to the Federal Government 11

16. Reasons for Program Changes or Adjustments 11

17. Plans for Tabulation and Publication of Results 11

a. Tabulation Plans 11

b. Publication Plans 13

18. Approval Not to Display the Expiration Date for OMB Approval 14

19. Exception to the Certification Statement 14

References 15
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 1

PART	A.	JUSTIFICATION

This submission is a request for approval of data collection activities that will be used to support the 

Mid-Atlantic Regional Educational Laboratory (REL MA) project, Evaluation of a District-Wide 

Implementation of a Professional Learning Community Initiative. The study is being funded by the 

Institute of Education Sciences (IES) of the U.S. Department of Education (ED), and is being implemented 

by ICF International and its subcontractor, Rutgers University’s Center for Effective School Practices. 

This study aims to address the need for systematic information about district-wide implementation of 

professional learning communities as a critical element in improving teacher quality and instruction, 

thereby contributing to increased student achievement. The study will survey online a population of 

teacher participants in school-based professional learning communities and interview principals face to 

face about the context and their perceptions of the initiative, pre- and post-implementation. Data 

collection from teachers will focus on what the professional learning communities do, how they operate, 

and to what extent they produce the outcomes expected of them as framed by six conceptual attributes 

of professional learning communities and five specific tasks. Data collection from principals will focus on 

contextual information about school culture and conditions such as resources that support 

implementation. Teachers and principals will also provide their reflections on the challenges of 

implementing professional learning communities and their suggestions for improvement. The analysis 

will enable comparisons among professional learning communities within and across schools, and 

between teachers’ pre-implementation expectations and post-implementation experiences. Study 

findings are expected to inform both theory and practice related to implementation of professional 

learning communities.

This submission requests approval to recruit principals and teachers for the study and conduct in-person 

interviews with principals and online surveys with teachers in the study district.

1. Circumstances Necessitating the Collection of Information

a. Statement of Need to Evaluate District-Wide Implementation of a Professional 

Learning Community Initiative

The specific legislation authorizing this data collection is specified in Part D, Section 174 (20 U.S.C. 9564) 

of the Education Sciences and Reform Act (ESRA) of 2002. Part D, Section 174 provides for ED to enter 

into five-year contracts with entities to establish a networked system of ten regional laboratories that 

serve the needs of each region of the United States. The Regional Educational Laboratories (RELs) are to 

carry out a range of activities to serve the needs of each region in the United States, including applied 

research, and development, dissemination, training, and technical assistance activities that focus on 

how to use data and analysis. The primary mission of the RELs is to help states and districts 

systematically use data and analysis to answer important issues of policy and practice with the goal of 

improving student outcomes. Accordingly, this project was conceived in response to a specific request 

made of the REL MA from a superintendent of a large school district in Pennsylvania (West Chester Area 

School District, abbreviated as WCASD) to assist in the evaluation of the implementation of professional 

learning communities in the district during the 2013-2014 school year. 
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 2

Practitioners in the education field have widely embraced professional learning communities (PLCs)

because they are aligned, at least in theory, with several features of effective professional development, 

such as active learning and teacher collaboration (for a review, see Stoll, Bolam, McMahon, Wallace, & 

Thomas, 2006). These collaborative teams are believed to be effective because they give teachers the 

opportunity to be exposed to new ideas and practices, and improve their pedagogy via a process of 

critical reflection (Hord, 1997; Wood, 2007). Operationally speaking, PLCs are teams of educators (most 

commonly, teachers) that meet regularly, either in-person or virtually, often but not always during 

scheduled school time, to develop lesson plans, examine student work, monitor student progress, assess 

the effectiveness of instruction, and identify needs for professional learning. These teams operate with a 

commitment to the norms of continuous improvement and experimentation, and engage their members 

in improving their daily work to advance the achievement of districts’ and schools’ goals for student 

learning (National Staff Development Council, 2001). Ideally, PLCs should be characterized by a set of 

dimensions or attributes such as shared beliefs, values, and vision; shared and supportive leadership; 

supportive structural conditions; supportive relational conditions; collective learning; and peer sharing 

(DuFour, 2005; Hord, 1997). Further, it is generally acknowledged that strong PLCs are not only those in 

which new knowledge regarding content and pedagogy is acquired, but also those in which existing 

assumptions about teaching and learning are challenged and critiqued (Little, 2003).

The increasing interest of education stakeholders in PLCs also appears to reflect a belief in the 

effectiveness of these learning communities. Recognizing the value of building systems for peer-to-peer 

professional learning among teachers and other education leaders, the U.S. Department of Education is 

investing in and supporting PLCs as part of several key initiatives, including Investing in Innovation (i3), 

the Office of Special Education Programs’ technical assistance centers, Promise Neighborhoods, and 

Race to the Top. Thus, there is strong support among school administrators and policymakers alike for 

expanding this practice (Talbert, 2010).

Unfortunately, studies that evaluate the effects of PLCs on teacher professional learning and/or student 

learning outcomes are limited in number. A recent review (Vescio et al., 2008) identified a mere 11 

studies that examined the effectiveness of PLCs and observed that few studies have moved beyond 

teachers’ self-reports. However, the researchers found cumulative evidence that is consistent with the 

hypothesized effects of PLCs. They concluded that participation in PLCs influences teaching practice as 

teachers move toward student-centered instruction, and that when teachers participate in a learning 

community, students benefit as well, as indicated by improved achievement scores over time. Similarly, 

a recent meta-analysis found that PLCs have a small but statistically significant positive effect on student 

achievement (Lomos, Hofman, & Bosker, 2011). Thus, while the majority of studies that evaluated the 

efficacy of PLCs have important methodological limitations, the limited evidence from programs that 

were more systematically evaluated seems to support the connection between teachers’ conscious 

efforts to improve instructional practice through group/peer study and changes in classroom practice 

and student achievement (Vescio et al., 2008). We expect that in addition to serving WCASD-specific 

information needs, this study will help close the important theoretical and empirical gap in the current 

literature by addressing the question of how and under which conditions (e.g., leadership style, group 

norms, available resources, and work arrangements) PLCs get in a position to achieve key desired 

attributes of collaborative work and meet standards of productivity that have the potential to enhance 

student learning and achievement. 

b. Research Questions

The primary research questions to be addressed in the study are listed below. The questions were 

formulated with direct input from the leadership of WCASD to ensure that the study meets their specific 
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 3

needs. However, these questions are also informed by the existing gaps in the current literature 

regarding the “how and under what conditions” focus discussed in the previous section.

 Research Question 1: How do PLC teams implement the study district’s PLC tasks? That is, what 

collaborative teamwork routines do they develop to achieve the key attributes of PLCs and the 

key tasks listed in the study district’s PLC materials?

 Research Question 2: How do the study district’s teachers and principals evaluate their 

experience with the program? Is their actual PLC experience the same or different from what 

they expected it to be before the implementation of the program? How do teachers’ evaluation 

of the program and of the implementation compare with those of principals? 

 Research Question 3: What specific artifacts (e.g., essential learning targets, standardized 

common assessments, and systematic interventions) are produced by each PLC team in the 

district?

c. Study Overview

In response to a specific request from the superintendent, REL MA will evaluate the implementation of 

WCASD’s district-wide PLC initiative during the 2013-2014 school year. As a result of its planning 

process, WCASD produced the WCASD PLC Protocol that explains what PLCs are, how and why they will 

be integrated into the district, and the specific sequence of five tasks that each PLC team should 

accomplish over time, regardless of grade level and content area. The protocol was shared with all 

educational staff in the district but does not specify how PLC teams should be established or what work 

routines they should adopt. Rather, this decision is left up to the principal and teachers of each school in 

order to accommodate the unique set of circumstances and resources that exist within each school and 

to leverage more effectively the expertise and experience of current teachers and administrators. 

Because PLC teams in the district will not follow a uniform, fully structured implementation design, the 

proposed study has two main goals: to describe the process of implementing PLCs in WCASD and to 

assess and compare the progress that these PLC teams make on specific tasks related to collaborative 

professional learning and instruction during the first year of implementation. Specifically, the study will 

involve collecting survey data from WCASD’s teachers and interview data from WCASD’s principals to: 

describe and compare (a) the work routines of teachers in PLC teams within and across schools in the 

district and (b) the specific outcomes (or products) of their collaborative work. REL MA will also ask 

teachers and principals to evaluate both their experience with specific aspects of the implementation –

such as team work, productivity, and professional development – and lessons they learned from that 

experience. This information will permit WCASD to identify evidence-based best practices that can 

support the ongoing implementation of the district’s PLC program and its impact. This information will 

also permit WCASD to identify actual and/or perceived barriers to implementation during the first full 

year of implementation that the district may want to address in subsequent years of the 

implementation. 

To be efficient, REL MA proposes to use the same survey data collected from individual teachers, at the 

aggregate, to assess and compare the progress that PLC teams in the district make on achieving the 

stated goals of the program during the first full year of implementation. First, the data collected from 

individual teachers who are members of the same PLC team will be aggregated to represent measures of 

PLC team-level variables of interest such as communication and leadership style in the group. Next, the 

aggregated data (with PLC teams being the units of analysis) will be analyzed in relation to a logic model 

(see Figure 1) which REL MA developed with input from the district based on a scheme proposed by 
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 4

Nelson and colleagues (Nelson, Cordray, Hulleman, Darrow, & Sommer, 2012) that inquires about the 

extent to which core components of the program have been implemented. The model is designed to 

benchmark the “PLC status” (as beginning, evolving, or mature: see Grossman et al., 2000) of each PLC 

team in the district based on a measurement scale that is theoretically derived from the six key desired 

attributes of effective PLCs (i.e., shared mission, collaborative culture, collective inquiry, action 

orientation, commitment to continuous improvement, and results orientation) as identified in the 

extant literature on the topic (see DuFour et al., 2010; Hord, 1997) and as also referenced in WCASD’s 

PLC protocol. The short-, medium-, and long-term outcomes listed in the logic model specify the 

characteristics PLC teams should exhibit as they evolve over time from a beginning status in the short 

term to a mature status in the long term. The model also specifies the five sequential tasks that PLC 

teams in the district were asked to complete. These are: (a) formulation of essential learning targets, (b) 

development of common formative assessments that measure proficiency on essential learning targets, 

(c) modifications of the core instructional program to allow differentiated instruction by degree of 

student proficiency on one or more learning targets, (d) development of supplemental interventions for 

students who do not demonstrate proficiency on one or more learning targets through the core 

instructional program, and (e) formation of child study teams to coordinate more intensive and more 

frequent individualized interventions for students who do not meet proficiency on one or more learning 

targets through the core instructional program and supplemental interventions. In all, using the logic 

model as an analytical tool will allow the study team to answer the research questions while providing 

WCASD with valuable insights about the progress of the PLC initiative district-wide.

Data for the study will be collected through a post-implementation online survey of all teachers (N=930) 

in all of WCASD’s 16 schools, and through semi-structured interviews with all principals in the district 

(N=16) post-implementation. REL MA will design and administer the post-implementation survey of 

teachers and will also design and collect the post-implementation interview data from principals. All 

WCASD’s teachers and principals will be involved in data collection activities because PLCs reside in 

individual schools and depend to a large extent on school-level supporting conditions. Teachers are the 

members of PLCs; it is their expectations and perceptions we wish especially to discern. This study will 

enable a fine-grained assessment of their experiences post-implementation as well as how their 

experience compares to that of other team members (to this end we will use a unique identifier to link 

individual teachers to PLC teams). Principals are included because they are on-the-scene observers and 

gatekeepers who control access to critical resources for implementation, and because they are key 

stakeholders who are deeply invested in the outcomes of teacher quality and student achievement. 

Importantly, because PLCs in WCASD will be given complete freedom to organize, it is very likely that 

teams’ composition and work routines (and therefore the PLC experience of individual teachers and 

principals) will vary within and among schools in the district. Therefore, it is not feasible to limit data 

collection to a random sample of teachers and administrators in light of the goal of providing adequate 

representation of their experiences with implementation in the absence of a well-defined sampling 

frame. The specific data collection plan with the timeline for all data collection activities is included in 

Table 3. 
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 5

Figure 1: Logic Model for Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative

Inputs Outputs Outcomes

Activities Participation Short-term Medium-term Long-term

Principals facilitate the process 

of determining grade- and 

subject-specific PLC teams 

within each building, such that 

all members on any given PLC 

team share learning outcomes 

(that is, teachers must teach the 

same academic standards in 

order to be on the same PLC 

team). 

Each PLC team is provided with 

a copy of the district-wide PLC 

protocols.

Each PLC team meets for 

approximately 45–60 

consecutive minutes weekly.

PLC teams have access to 

expertise from external sources, 

such as program supervisors, 

colleagues from other schools, 

and university professors. 

PLC teams organize 

independently to achieve key 

attributes of PLCs (shared 

mission, collaborative culture, 

collective inquiry, action 

orientation, commitment to 

continuous improvement, and 

results orientation) that are 

described in the PLC protocols. 

Task 1: Collaborate to 

determine 3–5 essential 

learning targets for each unit 

taught as part of the core 

instructional program.

Task 2: Design two versions of a 

common formative assessment 

for each unit taught to assess 

students’ proficiency in the 

essential learning targets. 

Task 3: Modify core 

instructional program based on 

assessment results to allow 

differentiated instruction.

Task 4: Develop supplemental 

interventions for nonproficient 

students.

Task 5: Form child study teams 

to coordinate more intensive 

and more frequent 

interventions for struggling 

students. 

All WCASD teachers in their 

respective PLCs

All WCASD teachers of a given 

subject and their program 

supervisor

PLC teams working 

independently to create the 

assessments and then reaching 

consensus with other subjectspecific PLC teams 

PLC teams

PLC teams

PLC teams (potentially, multiple 

teams involved)

 Regular and active 

participation in PLC activities

 Positive evaluation of the PLC 

experience

 Positive beliefs about the 

professional development 

(PD) value of PLCs

 Positive beliefs about the 

impact of PLCs on students 

 Teachers positively evaluate 

and agree with the essential 

learning targets.

 Teachers positively evaluate 

the quality of the common 

formative assessment created 

by the team.

 Teachers analyze student 

assessment on formative 

common assessment to 

identify need in differentiated 

instruction. 

 Teachers analyze student 

assessment on formative 

common assessment to 

identify nonproficient 

students. 

 Teachers analyze student 

assessment on formative 

common assessment to 

identify struggling students. 

 Continued regular and active 

participation in PLC activities

 Personal commitment to 

collaborative work

 Personal commitment to PD 

and learning collaboratively

 Translation of (or intention 

to translate) knowledge 

acquired to actual classroom 

practice

 Teachers use essential 

learning targets to guide 

instruction activities.

 All teachers make standard 

and consistent use of the 

common formative 

assessment. 

 Teachers assign students to 

receive differentiated 

instruction based on analysis. 

 Teachers assign students to 

receive supplemental 

interventions based on 

analysis. 

 Teachers form child study 

teams to design 

individualized interventions 

for struggling students. 

 Sustainability of PLCs in 

WCASD

 Shared professional culture 

that values collaboration 

and learning in WCASD

 WCASD instrumental in 

diffusing the practice to 

other educational entities in 

the region and beyond

 As a matter of routine, 

teachers revise and update 

essential learning targets as 

necessary. 

 As a matter of routine, 

teachers revise and update 

the common formative 

assessment as necessary.

 As a matter of routine, 

teachers review student 

progress and reconsider 

differentiated instruction 

assignment. 

 As a matter of routine, 

teachers review student 

progress and reassess need 

for intervention (as well as 

the intervention itself). 

 As a matter of routine, 

teachers review student 

progress and reassess need 

for intervention (as well as 

the intervention itself). 
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 6

2. Purposes and Uses of Data

Rutgers University’s Center for Effective School Practices will analyze all project data for the REL MA

Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative under 

contract number ED-IES-12-C-0006. 

The primary purpose of the study is to evaluate the implementation of the district-wide PLC initiative in 

the WCASD in response to a specific request from the superintendent. The proposed study has two main 

goals: to describe the process of implementing PLCs and to assess and compare the progress that PLC

teams in the district make on specific tasks related to collaborative professional learning and instruction 

during the first year of implementation against the desired progress as per the program’s logic model. 

Findings from this study will be used by WCASD to formulate an operational definition of each attribute 

of PLCs and assess PLC teams’ progress toward achieving each one of these attributes, as well as 

progress toward completing the tasks identified in the district’s PLC protocol. The insights generated by 

the study will also assist the district’s leadership in considering programmatic modifications that can 

enhance the productivity and sustainability of PLCs and provide baseline data for continuing efforts to 

monitor their functioning. The close collaboration between the district and study team is also expected 

to help build the district’s internal capacity to collect, manage, and use program data more efficiently 

based on the instruments and methodology used in the study. 

It is also reasonable to expect that study findings will inform similar efforts by other districts that are 

members of the Professional Learning Research Alliance (including the West Amwell Township School 

District, the South Hunterdon School District, the Monroe Township School District), as well as other 

stakeholders across the mid-Atlantic region which are not as far along as WCASD in planning and 

implementing PLCs or which are implementing but lack evidence-based information for adjusting their 

PLC implementation. The data will also inform efforts by state departments of education (starting with 

the Departments of Education in New Jersey and Pennsylvania, which are also members of the 

Professional Learning Research Alliance) to support the planning and implementation of PLCs statewide. 

REL MA has already prepared a review of the literature on online and hybrid PLCs in response to strong 

interest from research alliance members. 

Finally, it is likely that the findings of this study will be used by other federal agencies and a national 

audience of practitioners and scholars who are interested in PLCs. The description of how and under 

which conditions (e.g., leadership style, group norms, available resources, and work arrangements) will 

shed light on positioning PLCs to achieve key desired attributes of collaborative work, as well as to meet 

standards of productivity that have the potential of enhancing student learning and achievement, and 

will narrow a conceptual and theoretical gap in the field. 

3. Use of Technology to Reduce Burden

Teacher data will be collected online through Survey Monkey. All teachers at WCASD have email and 

internet access through their district-supported accounts. Teachers will use these accounts to receive 

email communication about the study, access the surveys, and submit their electronic responses. 

Data from the semi-structured interviews of principals cannot be collected through such methods as 

web surveys. Computer-assisted telephone interviews are not optimal for collecting these data. The 

proposed face-to-face interviews will be necessary to allow in-depth, conversational exchanges with 

respondents, and to obtain answers to both open-ended and detailed questions. However, if repeated 
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 7

attempts at scheduling these two-hour interviews during regular work hours fail, the REL MA study team 

will consider using telephone and/or computer-assisted telephone interviews (e.g., via Skype), providing 

that these can be audio-recorded for the purpose of transcribing the interviews. 

4. Efforts to Avoid Duplication

Prior to designing this study, REL MA produced a literature review on online and hybrid PLCs. REL MA

has continued to search for relevant literature and to augment its literature base in the course of 

designing the study and gaining requisite approvals from OMB and the IRB at Rutgers University. 

Previous reviews of the literature on PLCs (e.g., Hord, 1997; Stoll & Louis, 2007; Wood, 2007) are based 

on findings from case studies. A recent review (Vescio et al., 2008) identified a mere 11 studies that 

examined the effectiveness of PLCs and observed that few studies have moved beyond teachers’ selfreports. REL MA’s ongoing review of the literature reveals that no in-depth study has been conducted of 

a single district’s implementation of PLCs that details the experience of teachers with qualitative and 

quantitative measures of attitudes and productivity, compares pre-implementation expectations with 

post-implementation experience, and provides in-depth analysis of differences among PLCs in similar 

contexts. Thus, the type of PLCs implementation evaluation proposed for this study has not been 

conducted to date. 

5. Methods to Minimize Burden on Small Entities

The study will utilize validated survey instruments and efficient interview protocols that focus on the 

data of most interest to the district and other audiences. WCASD will enable teachers to respond to the 

survey during periods that are already dedicated to school-wide and district-wide work. In terms of 

principal interviews, principals will be offered flexible scheduling to avoid potential conflicts or 

disruptions to their routine work. 

6. Consequences of Not Collecting Data

This is one-time collection. WCASD has no means other than the proposed data collection to track and 

assess the implementation of a major initiative that has the potential to deliver effective professional 

development for all teachers in the district and improve learning outcomes for all students in all of the 

district’s schools. The rich data collected through this project will generate valuable insights about the 

process of implementing PLCs in the district (including potential and perceived barriers to 

implementation) and will provide the baseline for assessing the impact of this program both in the short 

and long run. There has been no previous comprehensive study of PLCs in WCASD so there is no 

alternative source for the information to be collected. Moreover, the implementation of PLCs in WCASD 

presents a rare research opportunity (a natural experiment of a sort) to study in depth the dynamic 

nature of PLCs and the implementation process, thus making a significant contribution to current theory 

and evidence-based practice regarding PLCs and their impact on teachers’ professional development 

and students’ learning outcomes.

7. Special Circumstances

There are no special circumstances associated with this data collection.
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 8

8. Federal Register Announcement and Consultation

a. Federal Register Announcement and Consultation

A 60-day notice to solicit public comments will be published in the Federal Register by February 1, 2014. 

If and when public comments are received by April 1, 2014 , the study team will inform IES of the 

comments and ways in which they will be addressed by April 7, 2014.

b. Consultations Outside the Agency

The study team has worked with IES to identify the five members listed below to serve as the Technical 

Working Group (TWG) for this study. 

1. Robert Boruch University of Pennsylvania P: 215-898 0409

2. Laura Hamilton Senior Behavioral Scientist, RAND P: 412-683-2300, x4403

3. Chris Hulleman James Madison University P: 540-568-2516

4. Andrew Porter University of Pennsylvania P: 215-898-7014

5. Christopher Rhoads University of Connecticut P: 860-486-3321

These experts on research methods, data analysis, implementation, and teaching and learning have 

provided input on the study’s design. Based on their input, the original emphasis on conducting process 

and outcomes evaluation of WCASD’s PLC initiative has been replaced with a focus on the process of 

implementation and the degree to which core elements of the program are implemented. This approach 

has informed the logic model of the evaluation as well as the overall research methodology which 

combines quantitative and qualitative data collection.

9. Payments or Gifts 

The study will not provide respondents with incentives for participation and we will not give payments 

or gifts to schools or to the district for providing access and study data. The superintendent of WCASD 

requested this study. The district will administer the pre-implementation online survey as well as the 

recruitment of all study participants overall. The superintendent has enlisted the support of principals in 

making time available for teachers to participate during the regular school hours; we plan to interview 

principals during regular school hours as well, unless they request otherwise. .

10. Assurances of Confidentiality

The data collection efforts that are the focus of this clearance package will be conducted in accordance 

with all relevant regulations and requirements, including the following:

 The Privacy Act of 1974, P.L. 93-579 (5 U.S.C. 552a).

 The Family Educational and Rights and Privacy Act (FERPA) (20 U.S.C. 1232g; 34 CFR Part 99).

 The Education Sciences Reform Act, P.L. 107-279 (20 U.S.C. 9573).

 The Freedom of Information Act (5 U.S.C. 552).
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 9

REL MA (specifically, Rutgers University’s Center for Effective School Practices, a subcontractor to ICF 

International) will protect the confidentiality of information for the study and will use it for research 

purposes only. The project director, Dr. Blitz, will ensure that individually identifiable information about 

study members remains confidential. When reporting the results, data will be presented in aggregate 

form only so that individuals and schools will not be identified.

The following safeguards, which are routinely employed by Rutgers University’s Center for Effective 

School Practices to carry out confidentiality assurances, will be applied consistently during the study: 

 A list of study team members will be forwarded to NCEE. 

 All employees sign a confidentiality pledge (Appendix A), which describes both the importance 

of and the employee’s obligation to discretion. All study team members who will have access to 

study participants or data will sign the pledge and will obtain appropriate clearance. 

 The plan for maintaining confidentiality also includes that all personnel with access to individual 

identifiers provide notarized nondisclosure affidavits and undergo training regarding (1) the 

meaning of confidentiality, particularly as it relates to handling requests for information and 

providing assurance to respondents about the protection of their responses; (2) controlled and 

protected access to computer files under the control of a single database manager; (3) built-in 

safeguards concerning status monitoring and receipt control systems; and (4) a secured and 

operator-controlled, in-house computing facility.

 Procedures will be followed to revoke clearance in a timely fashion from members who leave 

the study team. 

 REL MA staff have obtained clearance through NCEE’s security clearance office.

 Respondents’ personally identifiable information is maintained on separate forms and files, 

which are linked by sample identification number only.

 Access to hard copy documents is strictly limited. Documents are stored in locked files and 

cabinets, and discarded materials are shredded.

 Access to computer data files is protected by secure user names and passwords, which are 

available to specific users only.

REL MA will also include appropriate statements about protecting confidentiality in the requests for data 

collection from participants and other materials generated by the study team. Those documents 

include:

 Post-Implementation Interviews of Principals (see Part B, Appendix D)

 Post-Implementation Online Survey for Teachers (see Part B, Appendix E)

The following statements will appear in these documents to assure respondents about preserving 

confidentiality: 

 “Responses to the data collection activities will be used for research purposes only. The reports 

prepared for the study will summarize findings across the sample and will not associate 

responses with a specific school or individual. We will not provide information that identifies 

you or your school to anyone outside the study team, except as required by law”. 

 “The contractor follows the confidentiality and data protection requirements of IES (The 

Education Sciences Reform Act of 2002, Title I, Part E, Section 183). The contractor will protect 

the confidentiality of all information collected for the study and will use it for research purposes 

only. No information that identifies any study participant will be released. Information on 
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 10

respondents will be linked to their institution but not to any individually identifiable 

information. No individually identifiable information will be maintained by the study team”. 

11. Justification for Sensitive Questions

There are no questions of a sensitive nature in the district interviews.

12. Estimates of Burden Hours

Table 2 shows the estimated burden hours for principals and teachers who will participate in data 

collection. These estimates are based on REL MA’s experience collecting such data from principals and 

teachers for similar studies, and on pre-testing in other school districts. REL MA pre-tested the interview 

protocols with three principals and the online survey with four teachers from a school district in NJ 

represented in the PLRA. The average completion time for both online surveys was about 30 minutes.

Table 2. Estimated Response Time for Data Collection

Respondent/Data 

Request

Number of 

Targeted 

Respondents

Expected 

Response 

Rate (%)

Number of 

Respondents

Unit 

Response 

Time 

(Hours)

Total

Response 

Time 

(Hours)

Total Cost 

(Based on 

hourly 

wage 

rate)

Principals

Post-implementation 

In-person interview 16 100% 16 2 32 $1,400

Annualized Subtotal 32 $1,400

Teachers

Post-implementation

Online survey 930 85% 790 0.5 395 $10,752

Annualized Subtotal 395 $10,752

TOTAL Annualized ALL $12,152

Total cost estimates based on hourly wage rates for the Bureau of Labor Statistics’ May 2012 National Industry-Specific 

Occupational Employment and Wage Estimates, NAICS 61100: Educational Services 

(http://www.bls.gov/oes/current/naics3_611000.htm). Principal burden estimates based on a mean annual salary of 

$90,980 and 2,080 annual hours, or $43.74 per hour, for Educational Administrators in Elementary and Secondary 

Schools (11-9032). Teacher burden estimates based on a mean annual salary of $56,620 and 2,080 annual hours, or 

$27.22 per hour, for Preschool, Primary, Secondary, and Special Education Teachers (25-2000). Total costs rounded to 

the nearest dollar. 

13. Estimates of Cost Burden to Respondents

There are no capital or start-up costs for respondents. There will also be no additional operational, 

maintenance, or purchase of service costs for respondents. WCASD already has computers for staff use 

and has established email accounts and Internet access for all teachers and principals. The district also 

has included an annual account for Survey Monkey in its budget. 
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 11

14. Annualized Cost to the Federal Government

The estimated cost of the study to the federal government is $280,722 over two years, representing an 

average annual cost of $140,361. The estimated total cost of the REL MA five-year contract is 

$32,353,087, of which this study accounts for $280,722. Study costs were estimated using fullyburdened (“loaded”) hourly rates for staff, multiplied by the estimated number of hours of labor, and 

adding other direct costs. Multipliers for indirect rates were applied to direct costs. 

15. Reasons for Program Changes or Adjustments

This is a new data collection.

16. Plans for Tabulation and Publication of Results

Table 3. Timeline for Data Collection and Analysis

2014

1 2 3 4 5 6 7 8 9 10 11 12

Administer post-implementation 

online survey of teachers  

Analyze post-implementation 

online teacher survey data   

Conduct Principal postimplementation interviews  

Analyze of principal interviews   

Draft report  

Final report and WCASD debrief  

a. Tabulation Plans

REL MA will use the information gathered from the teacher survey and principal interviews to examine 

and evaluate district-wide implementation of a PLC initiative. Once the teacher survey dataset is ready 

for analysis, the study team will explore the distribution of individual responses on all variables of 

interest for irregular patterns (outliers, missing values, and deviation from normality for continuous 

variables) and make adjustments as needed. Next, the study team will compare responses across 

subgroups of teachers who teach at different levels (elementary, middle or high school), different 

content area, as well as teachers from different departments and schools, to identify potential sources 

of bias in our data. This is done for data diagnostic purposes and will not be included in public reports. 

As a rule, data will be reported at the aggregate to protect individuals’ identity. 

Audio recordings of principal interviews will be transcribed and the analyzed using Dedoose. Special 

attention will be given to extracting insights into barriers and facilitators to implementation. The 

content of teachers’ responses to the open-ended questions in post-implementation online survey will 

be downloaded and then content analyzed. We will use Dedoose to identify and extract teachers’ 

reports of most positive and most negative PLC experiences, and to synthesize their recommendations 

for improving the implementation of PLCs.
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 12

Research Question 1 (how do PLC teams implement the study district’s charge?) will be answered 

primarily through analyzing the post-implementation survey data collected from teachers in Year 2 of 

the project. The survey will include several items (see Part B, Appendix E) that ask teachers to report 

retrospectively on aspects of the work routine of the PLC in which they are a member (e.g., degree to 

which the team meets regularly, level of active participation in the group, group dynamics, and 

leadership style within the group). 

 Responses (on a Likert scale) provided by teachers who are members of a specific PLC team (as 

ascertained by the team’s unique study identifier) will be summed and averaged to produce 

team-level measures of these variables. 

 These variables will be analyzed next using cluster analysis to develop taxonomies of work 

routines as they emerge from this exploratory procedure. The study team will determine the 

specific cluster analysis algorithm to be used once there is an opportunity to assess the specific 

properties of the data collected from teachers. 

 Next, the study team will use findings from content analyzing teachers’ responses to the openended questions on the post-implementation online survey (where they describe the most 

positive and the most negative PLC experience they had and recommend possible 

improvements) to delineate common challenges/barriers that PLC teams encounter as members 

collaborate to establish work routines. 

The data analysis approach to answering Research Question 2 (how do WCASD teachers and principals 

evaluate their experience with the program compared to their initial expectations) will also utilize 

teachers’ responses to the online survey, specifcaly the variables measuring expectations, attitudes, 

evaluations, and self-reported behaviors. 

 Findings from teachers’ responses to the open-ended reflection questions on the postimplementation survey will also be used to answer this question, either by corroborating 

and/or clarifying themes that emerge from the teacher surveys or adding new insights that 

were not captured through these surveys. To answer this research question regarding the 

principals, the study team will analyze qualitative findings from the post-implementation 

interviews with principals. 

 Audio recordings of the interviews will be transcribed verbatim and checked against the 

original audio recording for accuracy. 

 Dedoose, a web-based application for managing, integrating, and analyzing qualitative and 

mixed methods data, will be used for data analysis. A grounded theory approach will be 

employed to explore a priori and emergent themes regarding principals’ evaluation of the 

program. The a priori themes will be defined according to the explicit themes explored in 

the interview (e.g., role of principal in implementation and allocation and use of school 

resources to support PLC work). It is anticipated that emergent themes will be focused on 

facilitators and barriers to implementation as well as on the impact, if any, PLCs had on 

teachers, students, and/or the school in the course of the first year of the program. 

 An extensive list of codes and their definitions that relate to each theme will be derived 

from the interview data. Two independent coders will use the coding scheme to analyze 

transcripts of four randomly selected interviews and agreement among coders (intercoder 

reliability) will be assessed using Cohen’s Kappa. The coding scheme will be adjusted to 

eliminate disagreements among coders and improve overall reliability (Cohen’s k > .85). 
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 13

 All codes will be entered into Dedoose and tagged to their associated segments of text for 

all interviews. Text segments will then be sorted by codes and reviewed by the coding team 

to identify emergent themes and to identify recurring patterns of responses to assess 

prevalence of themes. 

 In the final step of the analysis, the study team will examine relationships among themes by 

looking at associations between differentially coded content. 

Lastly, the study team will compare key findings about teachers’ perspective on the PLC initiative and its 

implementation to key findings that emerge from the analysis of the interview data obtained from 

principals. The study team will note similarities and differences in perspectives as a way of assessing 

possible changes to school culture post-implementation. 

To answer Research Question 3 (specific products produced by PLC teams), the study team will analyze 

the survey data for teachers’ reports about artifacts produced by their team. 

 Specifically, teachers will be asked to indicate the number of instructional units for which the 

team designed (1) essential learning targets, (2) core formative assessments, and (3) assessment 

rubrics. They will also be asked to indicate if they shared these artifacts with stakeholders 

(students, parents, and principal) and whether these artifacts were actually used with students 

in the course of the first year of implementation. The study team will also reference principals’ 

assessment of artifacts produced by PLC teams in their school based on analyzing the interview 

data collected from all principals in the district at the end of the first year of full implementation 

of the PLC initiative. 

b. Publication Plans

REL MA will produce a “What’s Happening” report of 15 to 30 pages at the conclusion of the project.

The study team will present findings in complementary narrative and tabular formats, and will include 

user-friendly graphic depictions (i.e., charts and graphs) to highlight and contextualize descriptive 

findings. The study findings will be presented without reference to any names of individuals or specific 

schools, in order to avoid deductive disclosure of study respondents. The final report, scheduled for 

release in November, 2014, on the IES and REL MA websites, will be based on data from the preimplementation and post-implementation teacher surveys and principal interviews. We anticipate that 

the “What’s Happening” report will include:

 A brief description of the study district’s PLC initiative and implementation objectives

 A discussion of the evaluation approach and logic model

 Presentation of key findings from the analysis of quantitative and qualitative data

 Discussion of study limitations, the appropriate interpretation of findings, and implications for 

current knowledge about PLCs, as well as for planning, implementation, and evaluation of PLCs

 A study methodology appendix.

REL MA will conduct a one-day debriefing to share the findings with the district’s leadership and to 

facilitate a discussion on the potential programmatic implications of these findings.

REL MA will also conduct a separate webinar to disseminate the study results with a broader audience

and will use its dissemination network to alert regional stakeholders about the final report and webinar 

opportunities. REL MA will also work with members of the Professional Learning Research Alliance to 

identify ways in which they may use this report to guide their current and future professional learning 

activities. The study team will present this work at academic and professional conferences and at 

meetings involving Research Alliance members and other regional stakeholders. Finally, the study team 
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 14

anticipates producing several peer-reviewed journal articles based on this project, including a paper to 

present the logic model used in the project, a paper that will discuss the validation of the survey 

instrument used in this project, and a paper that introduces the overall methodological approach to the 

evaluation of the implementation, in addition to papers that describe and discuss the key findings of the 

evaluation. 

17. Approval Not to Display the Expiration Date for OMB Approval

The study will display the OMB expiration date.

18. Exception to the Certification Statement

ED is not seeking an exception.
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 15

References

Baenen, N., & Jackl, A. (2010, June). Evaluation of central services professional learning teams as of 

spring 2010. Eye on Evaluation. E&R Research Report No. 10.06. Raleigh, NC: Wake County Public 

School System.

Delaware Department of Education. (2012). DDOE/Data coach project. Retrieved from 

www.doc.k12.de.us/tleu_files/PLCSurveyReport_2012.pdf.

Drummond, K., Chinen, M., Duncan, T.G., Miller, H.R., Fryer, L., Zmach, C., & Culp, K. (2011). Impact of 

the Thinking Reader® software program on grade 6 reading vocabulary, comprehension, strategies, 

and motivation (NCEE 2010-4035). Washington, DC: National Center for Education Evaluation and 

Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

DuFour, R., Eaker, R., & DuFour, R. (2005). Recurring themes of professional learning communities and 

the assumptions they challenge. In R. DuFour, R. Eaker, & R. DuFour (Eds.), On common ground: The 

power of professional learning communities (pp. 7–30). Bloomington, IN: National Educational 

Services.

DuFour, R., DuFour, R., Eaker, R., & Many, T. (2010). Learning by doing: A handbook for professional 

learning communities at work (2ded.). Bloomington, IN: Solution Tree.

Grossman, P., Wineburg, S., & Woolworth, S. (2000). What makes teacher community different from a 

gathering of teachers? Seattle, WA: Center for the Study of Teaching and Policy.

Hall, G., & Loucks, S. (1979). Implementing innovations in schools: A concerns-based approach. Austin, 

TX: Research and Development Center for Teacher Education.

Hemphill, J. (1956). Group dimensions: A manual for their measurement. Research Monograph No. 87. 

Columbus, OH: Ohio State University, Bureau of Business Research.

Heppen, J.B., Walters, K., Clements, M., Faria, A., Tobey, C., Sorensen, N., and Culp, K. (2012). Access to 

Algebra I: The Effects of Online Mathematics for Grade 8 Students. (NCEE 2012–4021). Washington,

DC: National Center for Education Evaluation and Regional Assistance, Institute of Education 

Sciences, U.S. Department of Education. 

Hord, S. M. (1997). Professional Learning Communities: Communities of Continuous Inquiry and 

Improvement. Austin, TX: Southwest Educational Development Lab.

Killion, J. (2006). Collaborative professional learning in school and beyond: A tool kit for New Jersey 

educators. New Jersey Department of Education Office of Academic and Professional Standards and 

the New Jersey Professional Teaching Standards Board in cooperation with the National Staff 

Development Council.

Little, J. (2003). Inside teacher communities: Representations of classroom practice. Teachers College 

Record, 105(6), 913–945.

Lomos, C., Hofman, R. H., & Bosker, R. J. (2011). Professional communities and student achievement: A 

meta-analysis. School Effectiveness and School Improvement, 22(2), 121–148.

Martin, T., Brasiel, S. J., Turner, H., & Wise, J.C. (2012). Effects of the Connected Mathematics Project 2 

(CMP2) on the Mathematics Achievement of Grade 6 Students in the Mid-Atlantic Region (NCEE 
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 16

2012-4017). Washington, DC: National Center for Education Evaluation and Regional Assistance, 

Institute of Education Sciences, U.S. Department of Education.

National Staff Development Council. (2001). Standards for staff development. Oxford, OH: Author. 

Retrieved from http://www.nsdc.org/standards/learningcommunities.cfm

Nelson, M. C., Cordray, D. S., Hulleman, C. S., Darrow, C. L., & Sommer, E. C. (2012). A procedure for 

assessing intervention fidelity in experiments testing educational and behavioral interventions. 

Journal of Behavioral Health Services Research, 39(4): 374–96.

Office of Management and Budget (2006). Questions and answers when designing surveys for 

information collection. Washington, DC: Author.

Olivier, D.F. & Hipp, K.K. (2010). Demystifying professional learning communities: School leadership at its 

best. Lanham, MD: Rowman & Littlefield Education.

Stoll, L., Bolam, R., McMahon, A., Wallace, M., & Thomas, S. (2006). Professional learning communities: 

A review of the literature. Journal of Educational Change, 7, 221–258.

Stoll, L., & Louis, K. S. (Ed.) (2007). Professional learning communities: Divergence, depth and dilemmas. 

Berkshire, UK: Open University Press.

Talbert, J. E. (2010). Professional Learning Communities at the Crossroads: How Systems Hinder or 

Engender Change. In Hargreaves, A., et al. (eds.). The Second International Handbook of Educational 

Change, Springer International Handbooks of Education, Volume 23, Dordrecht, Springer, 555–571.

Vescio, V., Ross, D., & Adams, A. (2008). A review of research on the impact of professional learning 

communities on teaching practice and student learning. Teaching and Teacher Education, 24(1), 80–

91.

Wijekumar, K., Hitchcock, J., Turner, H., Lei, PW., and Peck, K. (2009). A Multisite Cluster Randomized 

Trial of the Effects of CompassLearning Odyssey® Math on the Math Achievement of Selected Grade 

4 Students in the Mid-Atlantic Region (NCEE 2009-4068). Washington, DC: National Center for 

Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of 

Education.

Wood, D. R. (2007). Professional learning communities: Teachers, knowledge, and knowing. Theory Into 

Practice, 46(4), 281–290.
OMB Package: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative 17

Appendix	A:	Confidentiality	Pledge

I understand that the names, and any other identifying facts or information, of individuals, businesses, 

organizations, and families participating in projects conducted by the Center for Effective School 

Practices at Rutgers University or its subsidiaries are confidential information. I agree that I will not 

reveal such confidential information, regardless of how or where I acquired it, to any person unless such 

person has been authorized by the Director or Project Manager at Center for Effective School Practices 

at Rutgers University to have access to the information.

I further understand that the unauthorized access to, use, or disclosure of any confidential information 

is a breach of the terms of my employment, or my consultant agreement with Center for Effective 

School Practices at Rutgers University and may subject me to court action by any interested party or to 

other sanctions by Center for Effective School Practices at Rutgers University. I acknowledge that this 

agreement shall continue to bind me even after the project(s) is (are) completed and/or even though my 

employment or my consultant agreement with Center for Effective School Practices at Rutgers 

University has terminated.

In addition, in the course of my employment I may have access to personal information, electronic and 

otherwise, about fellow employees. I agree that I will treat that information as having the highest 

confidentiality, and not communicate it to fellow employees or others outside Center for Effective 

School Practices at Rutgers University. Final determination of whether or not there is a business 

purpose requiring that I access a fellow employees’ records will be made in consultation with the 

Director of Human Resources. Failure to uphold this standard is a breach of trust and may subject me to 

disciplinary action, including termination of employment.

Other than in the course of my authorized employment or my consultant agreement, I further agree 

that I will not use, nor facilitate the use by any third party, in any way any information deemed 

confidential by the terms of any contract or other written agreement between Center for Effective 

School Practices at Rutgers University and any other organization, except by written authorization by 

both parties. It is my understanding that Center for Effective School Practices at Rutgers University and 

the contracting organization(s) have the exclusive right to all information acquired or developed under 

such a contract or other written agreement. I acknowledge that I acquire no right, title, or interest in 

and to any data or information to which I have access by reason of my employment or my consultant 

agreement and that I may not remove such data from my assigned work location without prior 

authorization.

Name: Signature:

Date: