Contract No: ED
- IES
- 1 2
- C
-0007



### Section	B June 14, 2013

**Submitted to: Submitted**  **by:** Institutes of Education Sciences Marzano Research Laboratory U.S. Department of Education 9000 E. Nichols Ave., Ste. 210 555 New Jersey Ave., NW Centennial , CO 80112 Washington, DC 20208 (303) 766

-9199 (202) 219

- 1385 Project Officer: Project Director: Ok

- Choon Park Trudy L. Cherasaro

## Table	of	Contents

Section B. Data Collection Procedures and Statistical Methods
................................ ................................ ... 2 B1. Respondent Universe and Sampling Methods
................................ ................................ ................... 2 Pilot Study
................................ ................................ ................................ ................................ ............. 2 Full Study
................................ ................................ ................................ ................................ ............... 3 B2. Procedures for the Collection of Information
................................ ................................ ..................... 6 Statistical Power
................................ ................................ ................................ ................................ .... 6 Data Collection
................................ ................................ ................................ ................................ ...... 6 B3. Methods to Maximize Response Rates and To Deal With Non

- Response
................................ ......... 7 B4. Tests of Procedures or Methods to be U ndertaken
................................ ................................ ........... 8 Analysis
................................ ................................ ................................ ................................ ................. 9 B5. Individuals Consulted on Statistical Aspects of the Design
................................ .............................. 10 References
................................ ................................ ................................ ................................ .................. 10

## Section	B.	Data	Collection	Procedures	and	Statistical	Methods

The purpose of this research study is to examine a model of performance feedback to gain a deeper understanding of feedback characteristics that may be necessary for teachers to move from receiving feedback to improved performance in the context of teacher evaluation systems. Specifically, the study will examine the relationship among the usefulness of feedback, accuracy of feedback, credibility of the person providing feedback, learning opportunities related to the feedback, responsiveness to feedback, and teacher performance. The results of the study can be used to inform future training and guidance on providing feedback in teacher evaluation systems by considering the importance of different feedback characteristics (usefulness, accuracy, and credibility) and learning opportunities in the use of feedback. This information may be helpful for prioritiz ing needs both at the state and the district level for training and guidance on providing feedback in teacher evaluation systems, and also for informing the state of additional data collection needed to further understand feedback characteristics. **B1. Resp**  **ondent	Universe	and	Sampling	Methods Pilot Study**  **Sample** The pilot test is intended to test the survey prior to full study implementation on a small sample of teachers in districts that are implementing a teacher evaluation system that has the intention of informing teacher development through feedback. The pilot sample will include teachers from four districts in Colorado that are implementing four different teacher evaluation systems. These four districts served as partner districts in the development of the Colorado Department of Education ( CDE ) model evaluation system during the initial development of the CDE system
. These partner districts have been implementing locally

- developed teacher evaluation systems for a few years, and they shared their lessons learned with CDE to inform the development of the model evaluation system. Based on conversations with CDE, it is believed that these districts are further along in the teacher evaluation process and would be likely to have valuable information about the t eacher feedback included in their evaluation systems. Table 1 presents demographic information for these districts. Table 2 provides a brief description of the district evaluation systems. A random sample of 900 teachers would be drawn from these districts to represent various grade levels and subject areas. The 9 00 teachers is a sufficiently large enough sample to conduct the planned analysis, including confirmatory factor analysis (CFA) w hich would require approximately 265 to 530 participants given the proposed CFA model. **Table 1**  **: Description of**  **p artner**  **d istricts**  **Partner District Locale**  **Number of Teachers** Brighton School District Rural: Fringe 776 Denver Public Schools City: Large 4960 Eagle County School Districts Town: Remote 455 Harrison School District 2 City: Large 696**Table 2**  **: Brief d**  **escription of**  **p artner**  **d istrict**  **e valuation**  **s ystems** 

**Partner District Description of Evaluation System**  Brighton School District

- Coaches and principals are trained on what good teaching looks like.

- Student achievement data is also used to inform evaluation ratings.

- Observations are conducted by the principal. Denver Public Schools

- Locally developed rubric with 12 indicato rs of effective teaching

- Student growth is 50% of performance criteria.

- Observations conducted by peer observers and principal Eagle County School Districts - Rubric developed based on TAP system with 19 indicators of effective teaching

- Student growth measure included

- Observations conducted by master teachers and the principal Harrison School District 2

- Locally

- developed rubric with seven standards of effective teaching

- Student growth is 50% of performance criteria.

- Four observations are conducted each year to inform evaluation ratings. **Full Study Sample**  Within each state, researchers will identify a sample of teachers, large enough to meet the requirements described above, using a single sampling framework. In each state, researchers first identified the districts who are participating in the pilot of the state’s model evaluation system. Each state sought volunteer districts to participate in the pilot of the statewide evaluation system. These districts were ide ntified to participate in the proposed study because within each state these districts will all be implementing the same evaluation system (the state system) , and these districts will have the most experience with implementing this system because they have participated in the pilot and therefore have implemented the state system for one to three years prior to data collection in this study. Using the districts who were involved in the pilot will allow us to identify a sample within each state that is using the same system and has the most experience with implementing the system. Additionally, the districts that are involved in piloting the teacher evaluation system in each state proportionally represent locales across the state in a manner similar to that of the statewide distribution (see Table s 3, 4, and 5 ). Researchers seek to obtain a sample of teachers that represent the various locale/size of the districts and teacher types participating in each state’s evaluation system pilot study, which in each state continues through 2013
– 14, the year planned for data collection for the proposed study and the year before each state plans to offer their model for adoption by any district. Within each state, we will use both stratified random sampling and quota samplin g in order to obtain representation of teachers from all districts and from all grade level and subject area combinations. Both sampling methods are required to create a sample that accurately represents the few larger districts and the additional smaller districts that are part of the pilot. In both the larger and smaller districts, teachers will be sampled to represent six subject areas (language arts, math, science, social studies, non

- core subjects, and ELL/SPED/intervention) and three school levels (el ementary, middle, and high school). Stratified randomsampling will be used in the larger districts to proportionally represent each subgroup, because there

are enough teachers to allow for this type of sampling. In smaller districts, where there are too few teachers to use stratified random sampling, proportional quota sampling will be used to ensure that subgroups are accurately represented in the sample. The use of both methods is preferred over random sampling to ensure that the larger districts are not overrepresented in the sample, because the majority of the teachers will be from these larger districts. For each state, we will obtain a list of all teachers with information on their grade and subject area from each of the participating districts to det ermine the total number of teachers in each subgroup and the number needed to proportionally represent the overall number of teachers in each subgroup. In la rger districts, t he appropriate number of teachers in each subgroup will then be randomly selected using the random

- number

-generator feature in Microsoft Excel
. Information about the pilot districts and the expected sample for each state is described below.
**Colorado Sampling** There are 25 districts who are participating in the pilot of the Colorado mode l teacher evaluation system during the 2012
– 13 school year. Ten of these districts are also participating in the pilot of the new Colorado Academic Standards. This study will include the 15 districts who are participating only in the pilot of the model tea cher evaluation system. The 15 districts participating in the pilot represent various locales (Table 3 ). In the 15 pilot districts, a pool of approximately 7,200 full

-time
- equivalent (FTE) teachers are available for potential participation in this study. T hirteen of the districts are rural or town and therefore have a small teaching population ranging from 18 to 156 FTE teachers . The two remaining districts are larger suburban districts with approximately 1500 and 4900 FTE teachers
. Approximately 88 percent of teachers come from the two larger districts and 22 percent come from a smaller district. Employing the sampling methods described above to obtain representation from locales/sizes and subgroups a sample of 20 percent of teachers in the larger districts ( n = 1280) and a sample of approximately 200 teachers in the smaller districts will be recruited. This yields a total sample of approximately 1,480 teachers that represents the proportion of teachers in the smaller and larger districts (88% to 22% respectively) and allows for representation across the subgroups. The final count of teachers by subgroup will not be known until we obtain the list o f teachers in participating districts for the 2014
– 15 school year. **Table 3**  **: Percent of**  **p ilot**  **d istricts in Colorado by**  **l ocale**  **Locale Pilot Districts**  **All Districts**  % City 0% 7% % Suburb 13% 7% % Town 20% 21% % Rural 67% 65%**Kansas Sampling**

There are 22 districts participating in the pilot of the Kansas Educator Evaluation Protocol ( KEEP ) during the 2012
– 13 school year. The 22 districts in the pilot represent various locale (Table 4 ). In the 22 pilot districts, a pool of approximately 8,900 FTE teachers are available for potential participation in this study. Fifteen of the districts have a small teaching population ranging from 14 to 200 FTE teachers. The remaining seven districts have between 320 and 3,100 FTE teachers. Approximately 84 perc ent of teachers come from the 7 larger districts and 16 percent come from the 15 smaller districts. Employing the sampling methods described above to obtain representation from locales and subgroups, a sample of 20 percent of teachers in the larger districts ( n = 1512) and a sample of approximately 300 teachers in the smaller districts will be recruited. This yields a total sample of approximately 1,812 teachers that approximately represents the proportion of teachers in the smaller and larger districts (84 % to 16%, respectively) and allows for representation across the subgroups. The final count of teachers by subgroup will not be known until we obtain the list of teachers in participating districts for the 2014
– 15 school year. **Table 4**  **: Percent of**  **p ilot**  **d is** **tricts in Kansas**  **by l**  **ocale Locale**  **Pilot Districts All Districts**  % City 13% 2% % Rural 50% 71% % Suburb 5% 3% % Town 32% 24% **Missouri Sampling** There are 97 school districts participating in the 2012
– 13 pilot of Missouri’s Educator Evaluation System representing various locales (Table 5 ). In the 97 pilot districts, a pool of approximately 16,400 FTE teachers is available for potential participation in this study. Eighty

-two of the districts have a smal l teaching population ranging from 6 to 200 FTE teachers. The 15 remaining districts have between 300 and 1600 FTE teachers
. Approximately 68 percent of teachers come from the 15 larger districts and 32 percent come from the 82 smaller districts. Employing the sampling methods described above to obtain representation from locales and subgroups, a sample of approximately 10 percent of teachers in the larger districts ( n = 1100 ) and a sample of approximately 500 teachers in the smaller districts will be recru ited. This yields a total sample of approximately 1600 teachers that represents the proportion of teachers in the smaller and larger districts ( 68 % to 32 % respectively) and allows for representation across the subgroups. The final count of teachers by subgroup will not be known until we obtain the list of teachers in participating districts for the 2014
– 15 school year. **Table 5**  **: Percent of**  **p ilot**  **d istricts in**  **Missouri by**  **l ocale**  **Locale Pilot Districts**  **All Districts** % City 3% 3% % Rural 70% 74%% Suburb 10% 7%

% Town 17% 16% **B2. Procedures	for	the	Collection	of	I**  **nformation Statistical Power**  Structural equation modeling requires a minimum of 100 to 150 subjects (Loehlin, 1992), or between 5 and 10 subjects per parameter estimated , depending on the distribution of the variables (Tabachnick & Fidell, 2001 ). We can estimate the number of parameters and degrees of freedom using the proposed models, which include the latent variables and the indicators based on the potential survey item s and teacher evaluation rating data. The latent variables are the same in all of the states, and all indicators are the same except the indicators for teacher performance. Because of the difference in indicators of teacher performance, the model will be t ested for each state. Table 6 identifies information about the model for each state and presents the sample size needed based on Tabachnick & Fidell’s recommendations and also based on the methods for calculating power estimates provided by MacCallum, Brow ne, and Sugawara (1996). For a test of close fit based on the following assumptions: 

### α = 0.05 , 

### ε 0 = 0 .05 and 

### ε 1 = 0 .08 , where 

### ε 0 is the null value of the root

- mean - square error of approximation, and 

### ε 1 is the alternative value. **Table 6:**  **Sample size s**  **need to test the model in each state**  **State Number of** 

**p arameters in**  **p roposed**  **m odel**  **Degrees of**

**freedom in proposed**  **model Sample size** 

**(Tabachnick & Fidell recommendation)**  **Desired**

**p ower**  **Sample size**

**based on desired power**  Colorado 69 366 345 to 690 90% 71 Kansas 65 313 325 to 650 90% 78 Missouri 63 288 315 to 630 90% 84
**Data Collection** Prior to data collection, teachers will be asked to sign a consent form to participate in the study, which will provide them with information on the procedures for collecting data, including information about how the researchers will protect the confidential data. Researchers plan to hold on

- site meetings with teachers to provide information about the study and to collect consent forms. States and districts participa ting in the study will be members of the research alliance. At the start of the study, researchers will identify a point of contact within each district who will provide the evaluation rating data, including a unique teacher identifier, and teacher email a ddresses for invitation. **Pilot	Tests	Data	Collection	Procedures** The pilot test includes two data collection instruments: a researcher

- developed survey and a teacher interview protocol. Researchers will obtain a list of email addresses for teachers who agre ed to participate from the districts and will also collect email addresses on the consent forms to verifyaddresses as necessary. The email addresses will be associated with a unique identification number and

uploaded into an online survey software system such as SurveyMonkey. An online survey will be administered to teachers in the pilot sample near the end of the 2013
– 14 school year. The survey will take approximately 30 minutes to complete. Shortly after the survey is administered, a subset of approximat ely 20 teachers will also be asked to participate in an interview in which they will be asked to provide feedback on the clarity of the survey. The 20 teachers will be purposively selected to represent various response patterns on the survey (such as level s of agreement or disagreement)
. The in
- person interview wil l last approximately one hour. **Full	Study	Data	Collection	Procedures** The study involves two data collection instruments: a teacher evaluation instrument already in use in the districts and a researcher

- developed teacher survey. Teacher Evaluation Instrument. Within each of the states, teachers currently receive a performan ce rating based on the ir state’s teacher evaluation instrument. At the start of the study, researchers will identify a point of contact within each district who will provide the teacher evaluation rating data, including a unique teacher identifier. Prior t o study implementation, researchers will work with their contact in the district to set up protocols for collecting this data to ensure that th e data is in the needed format. A copy of the current draft of the teacher evaluation protocol for each state is provided in Attachment H (Colorado), Attachment I (Missouri) and Attachment J (Kansas). Teacher Survey. Researchers will obtain a list of email addresses for teachers who agreed to participate from the district contact and will also collect email addresses on the consent forms to use to verify email addresses as necessary. The email addresses will be associated with a unique identification number and uploaded into an online survey software system such as SurveyMonkey. Online surveys will be administered to teachers who agree to participate in the study. The survey takes approximately 30 minutes to complete. In order to maximize response rates, pre

- invitation an d reminder emails will be sent. Researchers will combine evaluation data from each district into one data set for each state. Teacher survey data and evaluation rating data will be combined using a relational database that links the data based on the teacher’s assigned identification num ber. **B3. M**  **ethods	to	Maximize	Response	Rates	and	To	Deal	W ith**  **Non -**  **R esponse**  In order to maximize the response rate, we will employ techniques identified in the tailored design survey methods (Dillman, 2000). The tailored design method emphasizes developing easily

- understood , brief surveys, and using personalized and repeated c ommunication with participants as methods for increasing response rates. We will aim to minimize the burden on the participants by designing a survey that is easily understood and takes a minimal amount of time to complete, and by working with educators to determine a window of time within the school year to administer the online survey that would allow for greater response. We will involve members of our research alliance, including key personnel in the state anddistricts, early in the research process. The involvement of key personnel will help with appropriate

messaging about the study, and communication about the importance of the study to states and districts. I n order to obtain an 80% or higher response rate, as suggested by the Office of Management and Budget (2006), we will focus on personalized and repeated communication. W e will first meet in

- person with teachers, followed by personalized email invitations, and targeted email re minders. In addition, we will offer incentives for participation. Because we are collecting teacher evaluation rating in addition to the survey data, and teacher evaluation rating data is somewhat sensitive information, it will be important for the partici pants to have an understanding of how this data will be used. Teachers who are identified in the sample will be invited to attend an in
- person meeting in which they will receive information on the study and procedures for collecting data, including informa tion about how the researchers will protect the confidential data. Consent forms will be collected during this meeting; therefore, teachers will provide consent to participate prior to receiving the survey email invitation. We will communicate with our di strict contacts to coordinate the in

- person meeting, so that the invitat ion will come from the district and teachers will meet in a central location in the district in which the district typically convenes teachers. The district contact will send a letter about the study with the invitation to the in

- person meeting. In this letter, the district contact will show their support and endorsement of the study and explain to teachers the benefits of their participation. Researchers will also send up to three targeted email reminders in order to increase the response rate (see Attachment G for the email invitation and reminders)
. If teachers do not respond after two email reminders, we will also send a reminder in the mail to the teacher’s school address. Teach ers will be provided with the option to request a paper survey rather than completing an online survey in the initial invitation. Additionally, t eachers will be offered an incentive of a $30 V isa gift card upon completion of the survey as part of the study
. As part of the Reading First Impact study, researchers conducted a study to examine the effects of monetary incentives on survey response rates for teachers and found that incentives significantly increased response rates (Games, Bloom, Kemple & Jacob, 2 008). This incentive was determined using NCEE guidance, which suggests a $30 incentive for a 30

- minute survey about interventions (NCEE, 2005). The NCEE guidance noted that collective bargaining agreements in districts may not allow teachers to complete surveys during school time , which may result in a decreased response rate without the offer of an incentive. In order to address the potential of incorrect email addresses in the list provided by the district, we will collect email addresses on the consen t forms as well. If an email is “bounced back” as undeliverable, we will verify the email address against the address provided on the consent form and make necessary changes. If the problem cannot be resolved by checking the email address on the consent form, we will contact the district to identify a correct email address and/or the reason why the email address is not working. **B4. Tests	of	Procedures	or	Methods	to	be	U**  **ndertaken**During the 2013– 14 school year, researchers will conduct a pilot test of the survey instrument. The pilot

test will be used to gather data to refine the survey and assess reliability and validity. This refined survey w ill then be used to conduct the full study in the 2014– 15 school year. Information about the sample, data collection, and analysis for the pilot test is presented below. The online survey will be administered to teachers in the pilot sample near the end of the 2013
– 14 school year. Shortly after the surv ey is administered, a subset of approximately 20 teachers will also be asked to participate in an interview in which they will be asked to provide feedback on the clarity of the survey. The interviews will be transcribed. Teachers will be provided with a c opy of the survey and asked to review it and respond to questions such as those proposed by the National Service Knowledge Network (2008):

 What problems, if any, did you have completing the survey?  Are the directions clear?  Are there any words or language in the instrument that teachers might not understand?  Did you find any of the questions to be unnecessary or redundant?  Were any questions difficult to answer?  What did you think this question was asking? How would you phrase it in your own words ?  Did the answer choices allow you to answer as you intended?  Is there anything you would change about the instrument? **Analysis** Researchers will examine the clarity and psychometric properties of the survey. Data from the interviews will be analyzed to he lp determine if the respondents interpreted the directions and questions as the researchers intended. Interview transcripts will be coded using MAX QDA, a qualitative analysis program , to identify patterns in responses to the interview questions. Researchers will modify the survey based on common responses about the clarity of the survey. Researchers will conduct item analysis to identify problematic items and to examine internal consistency of the survey. Cronbach’s alpha coefficients will be computed for e ach scale and correlation analysis will be used to determine the extent to which the items correlate with the scales and the effect on Cronbach’s alpha if the item were removed. Items will be deleted if removal of the items would increase Cronbach’s alpha and increase the average inter

- item correlation . Following the criterion proposed by Nunnally & Bernstein (1994), alpha coefficients of 0 .70 or higher will be considered as reliable. Following the item analysis, researchers will analyze the proposed meas urement model in Figure 3 , Section A, using confirmatory factor analysis (CFA). The factors in this measurement model will be assumed to covary, with no hypothesis that any one factor causes the other. The methods for CFA proposed by Kline (2005) will be followed. First, researchers will test for a single factor. This step allows researchers to examine the feasibility of testing a more complex model, and to determine if the fit of the single

-factor model is comparable. Selected fit indexes will be examined to determine if the results indicate a poor fit, which would substantiate the need for a more complex model. Next, the five

-factor model will be examined
. Researchers will examine the model fit indices, the indicator loadings, and thefactor correlations t o determine not only if the model fits the data, but also if the indicators have

significant loadings on the associated factors, and if the estimated correlations between the factors are not “excessively high (e.g., > .85)”(Kline, 2005 ; p. 73). If the mode l does not fit the data, the model will be respecified based on an examination of modification indices, correlation residuals, and practical and theoretical considerations. To determine if the fit of the five

-factor significantly reduces model fit, Chi

sq uare difference tests will be performed . These results could inform changes to the final study, such as determining that certain indicators or survey items are not good indicators of the factors, or determining that there are more or fewer factors than ori ginally conceived. **B5. Individuals	Consulted	on	Statistical	A** **spects of	the	D**  **esign** The following individuals were consulted on the statistical, data collection, and analytic aspects of this study: Name Title Organization Contact Information Dr. Steve Meyer Senior Researcher RMC Research Corporation meyer@rmcdenver.com

303
- 305
-4261 Dr. Fatih Unlu Senior Scientist Abt Associates Fatih_Unlu@abtassoc.com 617
- 520
- 2528 The following individuals will be involved in the study implementation: Name Role Organization Contact Information Dr. Trudy Cherasaro Principal

Investigator Marzano Research

Laboratory trudy.cherasaro@marzanoresearch.com

303
- 766
-9199 Dr. Helen Apthorp Co
- Principal

Investigator Marzano Research

Laboratory helen.apthorp@marzanoresearch.com

303
- 766
-9199 Dr. Kerry Englert Technical Working

Group Member Seneca Consulting kenglert@senecaconsulting.org



## References Dillman, D. A., (2000). Mail and internet surveys: The tailored design method (2nd ed
. ). New York: Wiley. Gamse, B.C., Bloom, H.S., Kemple, J.J., Jacob, R.T., (2008). Reading First impact study: Interim report (NCEE 2008

-4016). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Kline, R.B. (2005). Principles and practices of structural equation modeling (2nd ed.). New York: Guilford. Loehlin, J. C. (1992). Latent variable models; An introduction to factor, path, and structural analysis. Mahwah, NJ: Erlbaum. MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1 , 130– 149.National Center for Education Evaluation (2005). Guidelines for incentives for NCEE impact evaluations
.

Washington, DC: Author. National Service Knowledge Network (2008). Planning and conducting a pilot test. Scotts Valley, CA: Author. Retrieved July 18, 2012 from http://www.nationalserviceresources.org/practices/19498
. Nunnally, J. C., & Bern stein, I. H. (1994). Psychometric theory (3 rd ed.). New York: McGraw

- Hill. Office of Management and Budget (2006). Guidance on agency survey and statistical information collections. Washington , DC: Author. Retrieved July 18, 2012 from: http://www.whitehous e.gov/sites/default/files/omb/inforeg/pmc_survey_guidance_2006.pdf
. Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics (4th ed.). Boston: Allyn and Bacon.