Contract No: ED - IES - 1 2 - C - 0007



## Supporting	Justification	for	OMB	Clearance	of	a	Study	of	Clinical	Practice	in

Traditional	Teacher	Preparation	Programs	in	Missouri Section	B December 5 , 2013

**Submitted to: Submitted**  **by:** Institute of Education Sciences REL Central at Marzano Research Laboratory U.S. Department of Education 555 New Jersey Ave., NW 9000 E. Nichols Ave., Ste. 112 Washington, DC 20208 Centennial , CO 80112 (202) 219 - 1385 (303) 766 - 9199 Project Officer: Project Director: Sandra Garcia Stephen Meyer

## Table of Contents

Section B. Data Collection Procedures and Statistical Methods ................................ ................................ ... 2 B1. Respondent Universe and Sampling Methods ................................ ................................ ................... 2 B2. Procedures for the Collection of Information ................................ ................................ ..................... 4 B3. Methods to Maximize Response Rates and To Deal With Non - Response ................................ ......... 5 B4. Tests of Procedures or Methods to be Undertaken ................................ ................................ ........... 6 B5. Individuals Consulted on Statistical Aspects of the Design ................................ ................................ 7 References ................................ ................................ ................................ ................................ ................ 7 Attachment A: Public Law 107 - 279 - Nov . 5, 2002 Education Sciences Reform Attachment B: Printed Copy of Online Survey Attachment C: Affidavit of Nondisclosure Attachment D: Text for Invitations and Reminders

## Section	B.	Data	Collection	Procedures	and	Statistical	Methods

This study is designed to support needs in the Central R egion by providing descriptive information about clinical practice in Teacher Preparation Programs (TPPs). The study purpose is to document current practice and provide a data collection tool and an approach for collecting data from TPP graduates that can be ado pted and adapted as part of future research to be conducted by the Educator Effectiveness Research Alliance ( Teacher Preparation Workgroup) . Data from the study may be used to understand the extent to which clinical practice components vary within and acro ss programs. This will inform conversations about emerging state standards for TPPs, such as what constitutes minimum implementation of priority components. These data will also inform conversations about TPP reform or redesign by identifying areas in whic h programs have less or greater emphasis. **B1. Respondent	Universe	and	Sampling	Methods**  **State Selection** . Data to address the research questions will be collected from first -year teachers teaching in public schools in Missouri. The sample is focused on one state to provide a feasible scope of data collection, with the potential to apply the study methodology in other Central Region states based on their needs and priorities in subsequent years. Missouri was chosen as the Central Region state in which to col lect data for the following reasons: 1. Missouri prepares more teachers each year than any other state in the region; 2. The Missouri Department of Elementary and Secondary Education (MO DESE) has a data system with the capacity to link value - added student achie vement to TPPs and has collaborated with external researchers to analyze these data and publish results about the effectiveness of TPPs ( for example , Ehlert, M. et al. , 2011; Koedel, Parsons, Podgursky, & Ehlert , 2012). These data could be used in future w ork linking TPP characteristics to student achievement outcome data; 3. The MO DESE Office of Educator Quality is actively developing program standards for a comprehensive TPP assessment system with an annual program - level reporting component and is exploring new ways to collect data about TPPs; 4. Missouri is one of two Central Region states pursuing reportedly “innovative” models for clinical preparation as part of its participation in the NCATE State Alliance for Clinically - Based Teacher Preparation 1 ; and 5. Administrators in the MO DESE Office of Educator Quality (who are also members of REL Central’s Educator Effectiveness Research Alliance) have relationships with TPP administrators and have agreed to provide contact information for all first -year teachers in the state. **Focus on Teachers Prepared in Traditional Programs.** Missouri offers six “routes to certification 2 .” The study will include teachers prepared through Missouri’s “traditional route” for two primary reasons. First, most graduates of Missouri T PPs are trained through this route. Missouri’s 2012 Title II State Report 3 , shows that that 6,226 teacher candidates (8 6 %) completed the “traditional” route at one of 38

1The NCATE State Alliance for Clinically - Based Teacher Preparation creates partnerships between states and national experts to pilot diverse approaches to implementation and to bring new models of clinical preparation to scale . Th is alliance works with organizations such as the America n Association of Colleges of Teacher Education, the Association of Teacher Educators, and teacher unions to share information among entities working to transform teacher education (NCATE, 2010). 2 See http://dese.mo.gov/eq/cert/routes -to - certification.htm for additional information. 3 Available here: http://title2.ed.gov/Title2STRC/DataTools/2011/Missouri.xlsinstitutions of higher education in the state during the 20 1 0– 1 1 academic year. Second , in traditional

programs there is a clear distinction between teacher preparation and teaching — unlike in alternative certification routes in which many teacher candidates serve as the classroom teacher of record prior to completing their preparation progr am and becoming a certified teacher. Such a distinction is needed because a retrospective survey of clinical practice experiences cannot be meaningfully completed by practicing teachers who have not completed their preparation program. Only teachers who ha ve completed their preparation can provide accurate and complete reports of their clinical experience. Further, a survey of the clinical experiences of teachers trained through alternative routes requires a different set of questions to address fundamental differences in definitions of terms such as “field experience” and “student teaching experience” which would provide information that could not be directly compared to that collected from students in traditional programs. In this first effort to collect c omprehensive and individual - level data about clinical experience in teacher preparation, REL Central believes that study resources are most efficiently deployed through a study that focuses on the pre service preparation activities of teachers in tradition al programs. The study will generate findings that are relevant to the route through which most teachers are prepared in Missouri, providing information that can be meaningfully compared across programs. **Identification of Recent Teacher Preparation Program Graduates**  . To identify individuals who recently completed teacher preparation programs in Missouri, REL Central will work with MO DESE to collect names and contact information of all first -year public school teachers in the state who were prepared in traditional prog rams. Since 2007, MO DESE has administered an online First- Year Teacher Survey to all first -year teachers in the state. Each year, the state works with the Office of Social and Economic Analysis (OSEDA) at the University of Missouri to develop a comprehens ive list of teacher names and contact information for this purpose. OSEDA has developed protocols to ensure inclusion only of first year teachers, excluding those teachers who are new to a particular district or new to the state, for example. Use of MO DES E contact information for practicing first -year teachers was chosen in favor of an alternative approach in which program graduate contact information would be collected from each of the nearly 40 teacher preparation programs in the state. While collecting contact information from each Missouri TPP would help to ensure representation of all TPP graduates regardless of whether they chose to teach in the state, consideration of the following factors led REL Central to choose the former approach:  Collection of graduate contact information from multiple TPPs — some of which might have additional requirements for data release or human subjects review — would require OMB approval which would require additional effort and lengthen the project timeline;  Data collection f rom multiple TPPs would require contact with administrators at each TPP, increasing the overall data collection burden for respondents;  The quality and consistency of data collected from multiple TPPs would likely vary, creating potential challenges for su ccessfully contacting graduates; and  Collection of information about the pre - service clinical practice experiences of first -year teachers in Missouri is potentially more useful in that: (1) state administrators and policymakers may be most interested in th e experiences of teachers who actually go on to practice in the state and, (2) the data collected via REL Central’s survey can be more easily linked to data maintained by MO DESE in future research. For example, reports of teacher preparedness collected on the First- Year Teacher Survey could potentially be linked to reports of clinical practice experience in a future research project.**Representativeness of the Sampling Frame** . Data from the prior-year state Title II report will be used to

provide an indicat ion of the percentage of all traditional TPP graduates who are included in the study sampling frame. Using the count of teachers trained during the 2011 – 12 academic year (from Missouri’s 2013 Title II State Report) and the number of first -year public schoo l teachers who were trained in traditional Missouri TPPs (those in the study sampling frame), REL Central will compute the estimated percentage of TPP graduates included in the sampling frame. **Sampling Plan** . REL Central used the Guidance on Agency Survey a nd Statistical Information Collections (Office of Management and Budget, 2006) to design the sampling plan for the study. Based on Missouri’ s 2012 Title II State Report, eighteen teacher preparation programs had fewer than 100 graduates; twelve programs ha d between 100–200 graduates; and eight programs had more than 300 graduates. Among these programs, there are only four that are likely to have had more than 300 graduates who go on to teach in public schools in Missouri. Based on OMB guidance, and given a confidence interval level of 95 percent, the minimum proportion of respondents needed (to achieve estimates with a 95 percent confidence interval no larger than plus or minus five percent) is listed in Table 1 for several teacher preparation program sizes . “Adjusted minimum” proportions are presented to account for a nonresponse rate of up to 20 percent. 

### Table **1 :**  **Minimum Proportion of**  **Respondents N eeded**  **by Program S ize** 

As Table 1 shows, OMB guidance suggests that all graduates of programs with 300 or fewer graduates should be included in the study sample to provide estimates with a 95 percent confidence interval of no larger than plus or minus five percent — accounting for an estimated survey nonresponse rate of up to 20%. Therefore, REL Central will include all first -year teachers who graduated from tradit ional teacher preparation programs in the study sample, resulting in an estimated 3,500 first year teachers in the sampling frame . **B2. Procedures	for	the	Collection	of	I**  **nformation** The study involves one data collection instrument: a researcher- developed teacher survey, which will be administered online during a 12 - week period beginning either in April or September 2014 4 . REL Central will work with MO DESE to collect contact informat ion for first -year teachers who teach during spring 2014 or fall 2014 . Staff in the Educator Preparation Section at MO DESE have agreed to share the contact information for first -year teachers that is used for administration of the states’ First- Year Teach er Survey . MO DESE has agreed to provide the following data elements to REL Central: teacher identifier, teacher name, teacher email address, name of school in which they are teaching and physical address, current teaching assignment, certification route, name of preparation program, teaching certification(s), gender, and race/ethnicity.

4 Timing is contingent on OMB approval. **Program Size Minimum** 

**Proportion Adjusted Minimum** 

**Proportion** 100 95% 100% 200 90% 100% 300 80% 100% 400 60% 75% 500 50% 62.5%Using the data provided by MO DESE , an invitation to complete an online survey will be sent via email to

each first -year teacher who was prepared in a traditional program. The invitation will be sent by REL Central, describing the purpose of the study, the voluntary nature of participation, intended use of the data collected, methods for maintaining confidentiality, and the estimated amount of time needed to complet e t he survey (estimated to be 2 5 minutes). Customized invitations will be used with respondent identifiers embedded in the emailed survey links to track survey response. If email addresses are not available, the study team will communicate with respondents via U.S. Postal Service mail sent to their school address. Paper surveys will be mailed to respondents via USPS will contain unique identifiers for the same purpose. Text for invitations and reminders is presented in Attachment D. **B3. M**  **ethods	to	Maximize Response	Rates	and	To	Deal	W** **ith Non**  **- R** **esponse** MO DESE has offered to endorse the survey and to help REL Central solicit the support of state teacher associations. REL Central will seek endorsement of the study by the Missouri National Education Association , the Missouri State Teachers Association, and the American Federation of Teachers in St. Louis. The survey will include information about the purpose of the study, including its potential utility for improving teacher preparation in the state. All endorse ments will be included in the study invitation sent to respondents. Potential respondents will also be informed of the importance of their feedback to understanding the range of experiences among first -year teachers. MO DESE has been able to achieve a 55 to 60 percent response rate for the First- Year Teacher Survey by soliciting respondents by email only and sending two or three follow - up email messages to non respondents. REL Central will use the same initial approach to collect data from first -year teachers — contacting the first -year teachers via email using contact information from MO DESE. Researchers will employ strategies identified in Dillman (2000) to seek response rates (for the entire sample and for each TPP) of a minimum of 80 percent, using the following approach.

 Initial email messages will be sent by REL Central, with information about the study purpose and organizational endorsements, as described above.  Email messages will contain links to the online survey with embedded respondent identifiers that allow REL Central to track survey response and foll ow up with non - respondents. Paper surveys mailed to respondents via USPS will contain unique identifiers for the same purpose.  REL Central will follow up with non - respondents for a total of up to seven occasions using three different modes: email, hardcop y via US Postal Service (USPS), and telephone. The follow - up sequence comprises the following contacts: Contact 1: A postcard thank -you note and reminder will be sent to all participants after the first week of the email survey implementation. Contacts 2–4: Survey non - respondents will receive another three emails 5 that contain the survey at approximately 10 - day intervals after the first contact.

5 In the case that email addresses are not available or generate “bounce - back” messages , the study team will communicate with respondents via U.S. Postal Service mail sent to their school address. MO DESE indicated that email address information is typically missing for up to 5% of first - year t eachers and that physical address information is available for all teachers in the sample.Contact 5: A USPS package will be sent to each non - respondent. The package will contain the paper survey and a cover letter encouraging participation. Contacts 6 & 7: REL Central will make the last two follow - up contacts via telephone. The researcher will call non - respondents during school hours, informing them that a survey was sent to them previously, asking if they have questions about the study, and encouraging them to complete the survey. If a non - respondent is not available for the phone call, the researcher will leave a voice message and make a follow - up phone call within two work days. During the calls, the researcher will give non - respondents the opportunity to complete the survey on the telephone or send another paper copy of the survey. **B4. Tests	of	Procedures	or	Methods	to	be	U**  **ndertaken Survey Pretest.**  Cognitive interviews 6 were conducted by teleph one with nine first -year teachers in Missouri in summer 2013. REL Central collected contact information for a random sample of potential respondents, who we re graduates of nine randomly - sampled teacher preparation programs. Respondent participation was solicited via email by providing information about the survey and cognitive interview process, and the study for which the survey was developed. Participants in t he survey pretest will be excluded from the sampling frame for the full study. Cognitive interviews were used to improve the relevance and clarity of survey items and to minimize response error (Tourangeau, 1984; Willis, 1999). The Theory of Response Proce ss proposed by Tourangeau, Rips, and Rasinski (2000) was used as a basis for plans to understand how survey respondents answer questions and to identify sources of survey error. Respondents were asked to first complete the survey online and then contacted for a telephone interview. Retrospective verbal probing was used to review the draft survey content with each respondent and collect feedback to clarify and improve item wording (Willis, 1999). Information collected via cognitive interviews was used to cre ate a final version of the instrument. **Survey Data Analysis** **.** REL Central will present descriptive statistics for all respondents in the sample. To provide context for interpreting findings, respondent counts, survey response rates, and descriptive informa tion about survey respondents will be presented (name of teacher preparation program, certification(s), current teaching assignment, gender and race/ethnicity). These variables will also be used to assess nonresponse bias. For continuous variables ( such as number of hours or weeks), means and standard deviations will be presented. For categorical variables, the proportion of respondents selecting each response category will be presented. For “check all that apply” responses ( such as subject areas taught), m eans and standard deviations based on the total number of responses selected will be reported , a long with the proportion of respondents selecting each response category. Frequencies (expressed as percentages) of response categories will be presented in tab les with categorical response options. **Nonresponse Analysis .**  As one of the four sources of error in sample survey research, nonresponse bias is a concern for response rates as high as 90 percent (Dillman, 2000; Miller & Smith, 1983). This study will assess the presence of nonresponse bias by comparing respondent and non - respondent

6 Cognitive interviews were chosen as the approach to inform survey design in favor of a traditional pilot test to increase efficiency in terms of time and cost. RE L Central expects that each cognitive interview will last approximately 60 minutes.characteristics based on available data for the entire sample (current teaching assignment, certification

route, name of preparation program, teaching certification(s), gend er, and race/ethnicity). Characteristics of respondents and non - respondents will be analyzed to identify significant differences. If nonresponse bias is identified, sample weights will be developed and incorporated into analysis plans with under the guidan ce of a sampling statistician. Either post - stratification weighting ( for example, see Taylor, 2005; Oh and Scheuren, 1983) or propensity score adjustment ( for example, see Duffy, Smith, Terhanian, & Bremer

, 2005; Lee, 2006; Loosveldt & Sonck, 2008) will be used to develop sample weights. **B5. Individuals	Consulted	on	Statistical	A**  **spects of	the	D**  **esign** The following individuals were consulted on the statistical, data collection, and analytic aspects of this study: **Name Title**  **Organization Contact**  **Information** Bruce Randel President Century Analytics bruce.randel@centuryanalytics.com Trudy Cherasaro Senior Researcher Marzano Research Laboratory trudy.cherasaro@marzanoresearch.com The following individuals will be involved in the study implementation: **Name Role**  **Organization Contact Information**  Stephen Meyer Principal Investigator RMC Research Corporation meyer@rmcdenver.com 303 - 825 - 3636 Xin Wang Research Analyst RMC Research Corporation wang@rmcdenver.com 303 - 82 5 - 3636 Matt Linick Research Analyst RMC Research Corporation linick@rmcdenver.com 303 - 825 - 3636 Brandie Ward Data Collection Oversight RMC Research Corporation ward@rmcdenver.com 303 - 825 - 3636 Gale Hairston SEA Study Liaison Missouri Department of Elementary and Secondary Education (MO DESE) Gale.Hairston@dese.mo.gov **References** 

### Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method ( 2nd ed. ). New York: Wiley. Duffy, B., Smith, K., Terhanian, G., & Bremer, J. (2005). Comparing data from online and face - to face surveys. International Journal of Market Research, 47(6), 615 – 639. Ehlert, M., Podgursky, M., Harrington, J. Xiang, P., Parsons, E., Frazelle, S., & Taylor Knight, L. (2011). Teacher and Leade r Preparation in Relationship to Student Achievement in the Kansas City Region . Kansas City, MO: Kansas City Area Education Research Consortium.

### Koedel, C., Parsons, E., Podgursky, M., & Ehlert, M. (2012). Teacher preparation programs and

teacher quality: Are there real differences across programs? Washington, DC: CALDER/American Institutes for Research. Lee, S. (2006). Propensity score adjustment as a weighting scheme for volunteer panel web surveys. Journal of Official Statistics 22 (2), 329–349. Loosveldt, G & Sonck, N. (2008). An evaluation of the weighting procedures for an online access panel survey. Survey Research Methods 2 (2): 93 – 105. Miller L. E., & Smith, K. (1983). Handling nonresponse issues. Journal of Extension On - line, 21 (5). Retrieve d on Apr 15, 2013 from http://www.joe.org/joe/1983september/83 - 5 - a7.pdf . National Council for Accreditation of Teacher Education. (2010, November). Transforming teacher education through clinical practice: A national strategy to prepare effective teachers . Washington, DC: Author. Office of Management and Budget . (2006). Questions and answers when designing surveys for information collections . Washington, DC: Office of Information and Regulatory Affairs. Oh, H.L. and Scheuren, F.J. (1983). Weighting adjustme nt for unit nonresponse. I n W.G. Madow, I. Olkin , & D.B. Rubin (E ds. ) , Incomplete data in s ample surveys, Volume 2: Theory and Bibliography (pp. 143 – 184) . New York: Academic Press. Taylor, H. (2005). Does internet research “work”? Comparing online survey results with telephone surveys. International Journal of Market Research, 42 (1), 51– 63. Tourangeau, R. (1984). Cognitive sciences and survey methods. In T. Jabine, M. Straf, J. Tanur, & R. Tourangeau ( Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines, pp. 73 - 100. Washington, DC: National Academy Press. Tourangeau, R., Rips, L.J., & Rasinski, K. (2000). The Psychology of Survey Response. Cambridge: Cambridge University Press. Willis, G.B. (1999). Cognitive interviewing: A “how- to” guide . Presentation at the 1999 Meeting of the American Statistical Association.