Contract No: ED-IES-12-C-0007

Supporting	Justification	for	OMB	Clearance	of	a	Study	of	Clinical	Practice	in	

Traditional	Teacher	Preparation	Programs	in	Missouri

Section	B

December 5, 2013

Submitted to: Submitted by:

Institute of Education Sciences REL Central at Marzano Research Laboratory

U.S. Department of Education 

555 New Jersey Ave., NW 9000 E. Nichols Ave., Ste. 112

Washington, DC 20208 Centennial, CO 80112

(202) 219-1385 (303) 766-9199

Project Officer: Project Director:

Sandra Garcia Stephen Meyer
Table of Contents

Section B. Data Collection Procedures and Statistical Methods...................................................................2

B1. Respondent Universe and Sampling Methods ...................................................................................2

B2. Procedures for the Collection of Information.....................................................................................4

B3. Methods to Maximize Response Rates and To Deal With Non-Response .........................................5

B4. Tests of Procedures or Methods to be Undertaken ...........................................................................6

B5. Individuals Consulted on Statistical Aspects of the Design ................................................................ 7

References ................................................................................................................................................7

Attachment A: Public Law 107-279-Nov. 5, 2002 Education Sciences Reform

Attachment B: Printed Copy of Online Survey

Attachment C: Affidavit of Nondisclosure

Attachment D: Text for Invitations and Reminders
Section	B.	Data	Collection	Procedures	and	Statistical	Methods

This study is designed to support needs in the Central Region by providing descriptive information about 

clinical practice in Teacher Preparation Programs (TPPs). The study purpose is to document current 

practice and provide a data collection tool and an approach for collecting data from TPP graduates that 

can be adopted and adapted as part of future research to be conducted by the Educator Effectiveness 

Research Alliance (Teacher Preparation Workgroup). Data from the study may be used to understand 

the extent to which clinical practice components vary within and across programs. This will inform 

conversations about emerging state standards for TPPs, such as what constitutes minimum 

implementation of priority components. These data will also inform conversations about TPP reform or 

redesign by identifying areas in which programs have less or greater emphasis.

B1.	Respondent	Universe	and	Sampling	Methods

State Selection. Data to address the research questions will be collected from first-year teachers 

teaching in public schools in Missouri. The sample is focused on one state to provide a feasible scope of 

data collection, with the potential to apply the study methodology in other Central Region states based 

on their needs and priorities in subsequent years. Missouri was chosen as the Central Region state in 

which to collect data for the following reasons:

1. Missouri prepares more teachers each year than any other state in the region;

2. The Missouri Department of Elementary and Secondary Education (MO DESE) has a data system 

with the capacity to link value-added student achievement to TPPs and has collaborated with 

external researchers to analyze these data and publish results about the effectiveness of TPPs 

(for example, Ehlert, M. et al., 2011; Koedel, Parsons, Podgursky, & Ehlert, 2012). These data 

could be used in future work linking TPP characteristics to student achievement outcome data;

3. The MO DESE Office of Educator Quality is actively developing program standards for a 

comprehensive TPP assessment system with an annual program-level reporting component and 

is exploring new ways to collect data about TPPs;

4. Missouri is one of two Central Region states pursuing reportedly “innovative” models for clinical 

preparation as part of its participation in the NCATE State Alliance for Clinically-Based Teacher 

Preparation1; and

5. Administrators in the MO DESE Office of Educator Quality (who are also members of REL 

Central’s Educator Effectiveness Research Alliance) have relationships with TPP administrators 

and have agreed to provide contact information for all first-year teachers in the state.

Focus on Teachers Prepared in Traditional Programs. Missouri offers six “routes to certification2.” The 

study will include teachers prepared through Missouri’s “traditional route” for two primary reasons.

First, most graduates of Missouri TPPs are trained through this route. Missouri’s 2012 Title II State 

Report3, shows that that 6,226 teacher candidates (86%) completed the “traditional” route at one of 38 

 

1

The NCATE State Alliance for Clinically-Based Teacher Preparation creates partnerships between states and national experts to 

pilot diverse approaches to implementation and to bring new models of clinical preparation to scale. This alliance works with 

organizations such as the American Association of Colleges of Teacher Education, the Association of Teacher Educators, and 

teacher unions to share information among entities working to transform teacher education (NCATE, 2010).

2

See http://dese.mo.gov/eq/cert/routes-to-certification.htm for additional information.

3

Available here: http://title2.ed.gov/Title2STRC/DataTools/2011/Missouri.xls
institutions of higher education in the state during the 2010–11 academic year. Second, in traditional 

programs there is a clear distinction between teacher preparation and teaching—unlike in alternative 

certification routes in which many teacher candidates serve as the classroom teacher of record prior to 

completing their preparation program and becoming a certified teacher. Such a distinction is needed 

because a retrospective survey of clinical practice experiences cannot be meaningfully completed by 

practicing teachers who have not completed their preparation program. Only teachers who have 

completed their preparation can provide accurate and complete reports of their clinical experience.

Further, a survey of the clinical experiences of teachers trained through alternative routes requires a 

different set of questions to address fundamental differences in definitions of terms such as “field 

experience” and “student teaching experience” which would provide information that could not be 

directly compared to that collected from students in traditional programs. In this first effort to collect 

comprehensive and individual-level data about clinical experience in teacher preparation, REL Central 

believes that study resources are most efficiently deployed through a study that focuses on the preservice preparation activities of teachers in traditional programs. The study will generate findings that 

are relevant to the route through which most teachers are prepared in Missouri, providing information 

that can be meaningfully compared across programs.

Identification of Recent Teacher Preparation Program Graduates. To identify individuals who recently 

completed teacher preparation programs in Missouri, REL Central will work with MO DESE to collect 

names and contact information of all first-year public school teachers in the state who were prepared in 

traditional programs. Since 2007, MO DESE has administered an online First-Year Teacher Survey to all 

first-year teachers in the state. Each year, the state works with the Office of Social and Economic 

Analysis (OSEDA) at the University of Missouri to develop a comprehensive list of teacher names and 

contact information for this purpose. OSEDA has developed protocols to ensure inclusion only of firstyear teachers, excluding those teachers who are new to a particular district or new to the state, for 

example. Use of MO DESE contact information for practicing first-year teachers was chosen in favor of 

an alternative approach in which program graduate contact information would be collected from each 

of the nearly 40 teacher preparation programs in the state. While collecting contact information from 

each Missouri TPP would help to ensure representation of all TPP graduates regardless of whether they 

chose to teach in the state, consideration of the following factors led REL Central to choose the former 

approach:

 Collection of graduate contact information from multiple TPPs—some of which might have 

additional requirements for data release or human subjects review—would require OMB 

approval which would require additional effort and lengthen the project timeline;

 Data collection from multiple TPPs would require contact with administrators at each TPP, 

increasing the overall data collection burden for respondents;

 The quality and consistency of data collected from multiple TPPs would likely vary, creating 

potential challenges for successfully contacting graduates; and

 Collection of information about the pre-service clinical practice experiences of first-year 

teachers in Missouri is potentially more useful in that: (1) state administrators and policymakers 

may be most interested in the experiences of teachers who actually go on to practice in the 

state and, (2) the data collected via REL Central’s survey can be more easily linked to data 

maintained by MO DESE in future research. For example, reports of teacher preparedness 

collected on the First-Year Teacher Survey could potentially be linked to reports of clinical 

practice experience in a future research project. 
Representativeness of the Sampling Frame. Data from the prior-year state Title II report will be used to 

provide an indication of the percentage of all traditional TPP graduates who are included in the study 

sampling frame. Using the count of teachers trained during the 2011–12 academic year (from Missouri’s 

2013 Title II State Report) and the number of first-year public school teachers who were trained in 

traditional Missouri TPPs (those in the study sampling frame), REL Central will compute the estimated 

percentage of TPP graduates included in the sampling frame.

Sampling Plan. REL Central used the Guidance on Agency Survey and Statistical Information Collections

(Office of Management and Budget, 2006) to design the sampling plan for the study. Based on 

Missouri’s 2012 Title II State Report, eighteen teacher preparation programs had fewer than 100 

graduates; twelve programs had between 100–200 graduates; and eight programs had more than 300 

graduates. Among these programs, there are only four that are likely to have had more than 300 

graduates who go on to teach in public schools in Missouri. 

Based on OMB guidance, and given a confidence interval level of 95 percent, the minimum proportion of 

respondents needed (to achieve estimates with a 95 percent confidence interval no larger than plus or 

minus five percent) is listed in Table 1 for several teacher preparation program sizes. “Adjusted 

minimum” proportions are presented to account for a nonresponse rate of up to 20 percent. 

Table 1: Minimum Proportion of Respondents Needed by Program Size

As Table 1 shows, OMB guidance suggests that all graduates of programs with 300 or fewer graduates 

should be included in the study sample to provide estimates with a 95 percent confidence interval of no 

larger than plus or minus five percent—accounting for an estimated survey nonresponse rate of up to 

20%. Therefore, REL Central will include all first-year teachers who graduated from traditional teacher 

preparation programs in the study sample, resulting in an estimated 3,500 first year teachers in the 

sampling frame.

B2.	Procedures	for	the	Collection	of	Information

The study involves one data collection instrument: a researcher-developed teacher survey, which will be 

administered online during a 12-week period beginning either in April or September 20144. REL Central 

will work with MO DESE to collect contact information for first-year teachers who teach during spring 

2014 or fall 2014. Staff in the Educator Preparation Section at MO DESE have agreed to share the 

contact information for first-year teachers that is used for administration of the states’ First-Year 

Teacher Survey. MO DESE has agreed to provide the following data elements to REL Central: teacher 

identifier, teacher name, teacher email address, name of school in which they are teaching and physical 

address, current teaching assignment, certification route, name of preparation program, teaching 

certification(s), gender, and race/ethnicity.

 

4

Timing is contingent on OMB approval.

Program Size

Minimum 

Proportion

Adjusted Minimum 

Proportion

100 95% 100%

200 90% 100%

300 80% 100%

400 60% 75%

500 50% 62.5%
Using the data provided by MO DESE, an invitation to complete an online survey will be sent via email to 

each first-year teacher who was prepared in a traditional program. The invitation will be sent by REL 

Central, describing the purpose of the study, the voluntary nature of participation, intended use of the 

data collected, methods for maintaining confidentiality, and the estimated amount of time needed to 

complete the survey (estimated to be 25 minutes). Customized invitations will be used with respondent 

identifiers embedded in the emailed survey links to track survey response. If email addresses are not 

available, the study team will communicate with respondents via U.S. Postal Service mail sent to their 

school address. Paper surveys will be mailed to respondents via USPS will contain unique identifiers for 

the same purpose. Text for invitations and reminders is presented in Attachment D.

B3.	Methods	to	Maximize Response	Rates	and	To	Deal	With	Non-Response

MO DESE has offered to endorse the survey and to help REL Central solicit the support of state teacher 

associations. REL Central will seek endorsement of the study by the Missouri National Education 

Association, the Missouri State Teachers Association, and the American Federation of Teachers in St. 

Louis. The survey will include information about the purpose of the study, including its potential utility 

for improving teacher preparation in the state. All endorsements will be included in the study invitation 

sent to respondents. Potential respondents will also be informed of the importance of their feedback to 

understanding the range of experiences among first-year teachers.

MO DESE has been able to achieve a 55 to 60 percent response rate for the First-Year Teacher Survey by 

soliciting respondents by email only and sending two or three follow-up email messages to nonrespondents. REL Central will use the same initial approach to collect data from first-year teachers—

contacting the first-year teachers via email using contact information from MO DESE. Researchers will 

employ strategies identified in Dillman (2000) to seek response rates (for the entire sample and for each 

TPP) of a minimum of 80 percent, using the following approach.

 Initial email messages will be sent by REL Central, with information about the study purpose and 

organizational endorsements, as described above.

 Email messages will contain links to the online survey with embedded respondent identifiers 

that allow REL Central to track survey response and follow up with non-respondents. Paper 

surveys mailed to respondents via USPS will contain unique identifiers for the same purpose. 

 REL Central will follow up with non-respondents for a total of up to seven occasions using three 

different modes: email, hardcopy via US Postal Service (USPS), and telephone. The follow-up 

sequence comprises the following contacts:

Contact 1: A postcard thank-you note and reminder will be sent to all participants after the 

first week of the email survey implementation. 

Contacts 2–4: Survey non-respondents will receive another three emails5that contain the 

survey at approximately 10-day intervals after the first contact. 

 

5

In the case that email addresses are not available or generate “bounce-back” messages, the study team will communicate 

with respondents via U.S. Postal Service mail sent to their school address. MO DESE indicated that email address information is 

typically missing for up to 5% of first-year teachers and that physical address information is available for all teachers in the 

sample. 
Contact 5: A USPS package will be sent to each non-respondent. The package will contain 

the paper survey and a cover letter encouraging participation.

Contacts 6 & 7: REL Central will make the last two follow-up contacts via telephone. The 

researcher will call non-respondents during school hours, informing them that a survey was 

sent to them previously, asking if they have questions about the study, and encouraging 

them to complete the survey. If a non-respondent is not available for the phone call, the 

researcher will leave a voice message and make a follow-up phone call within two work 

days. During the calls, the researcher will give non-respondents the opportunity to complete 

the survey on the telephone or send another paper copy of the survey.

B4.	Tests	of	Procedures	or	Methods	to	be	Undertaken

Survey Pretest. Cognitive interviews6 were conducted by telephone with nine first-year teachers in 

Missouri in summer 2013. REL Central collected contact information for a random sample of potential 

respondents, who were graduates of nine randomly-sampled teacher preparation programs. 

Respondent participation was solicited via email by providing information about the survey and 

cognitive interview process, and the study for which the survey was developed. Participants in the 

survey pretest will be excluded from the sampling frame for the full study. Cognitive interviews were 

used to improve the relevance and clarity of survey items and to minimize response error (Tourangeau, 

1984; Willis, 1999). The Theory of Response Process proposed by Tourangeau, Rips, and Rasinski (2000)

was used as a basis for plans to understand how survey respondents answer questions and to identify 

sources of survey error. Respondents were asked to first complete the survey online and then contacted 

for a telephone interview. Retrospective verbal probing was used to review the draft survey content 

with each respondent and collect feedback to clarify and improve item wording (Willis, 1999). 

Information collected via cognitive interviews was used to create a final version of the instrument.

Survey Data Analysis. REL Central will present descriptive statistics for all respondents in the sample. To 

provide context for interpreting findings, respondent counts, survey response rates, and descriptive 

information about survey respondents will be presented (name of teacher preparation program, 

certification(s), current teaching assignment, gender and race/ethnicity). These variables will also be 

used to assess nonresponse bias. For continuous variables (such as number of hours or weeks), means 

and standard deviations will be presented. For categorical variables, the proportion of respondents 

selecting each response category will be presented. For “check all that apply” responses (such as subject 

areas taught), means and standard deviations based on the total number of responses selected will be 

reported, along with the proportion of respondents selecting each response category. Frequencies 

(expressed as percentages) of response categories will be presented in tables with categorical response 

options. 

Nonresponse Analysis. As one of the four sources of error in sample survey research, nonresponse bias 

is a concern for response rates as high as 90 percent (Dillman, 2000; Miller & Smith, 1983). This study 

will assess the presence of nonresponse bias by comparing respondent and non-respondent 

 

6

Cognitive interviews were chosen as the approach to inform survey design in favor of a traditional pilot test to increase 

efficiency in terms of time and cost. REL Central expects that each cognitive interview will last approximately 60 minutes.
characteristics based on available data for the entire sample (current teaching assignment, certification 

route, name of preparation program, teaching certification(s), gender, and race/ethnicity). 

Characteristics of respondents and non-respondents will be analyzed to identify significant differences.

If nonresponse bias is identified, sample weights will be developed and incorporated into analysis plans 

with under the guidance of a sampling statistician. Either post-stratification weighting (for example, see

Taylor, 2005; Oh and Scheuren, 1983) or propensity score adjustment (for example, see Duffy, Smith, 

Terhanian, & Bremer, 2005; Lee, 2006; Loosveldt & Sonck, 2008) will be used to develop sample 

weights.

B5.	Individuals	Consulted	on	Statistical	Aspects	of	the	Design	

The following individuals were consulted on the statistical, data collection, and analytic aspects of this 

study:

Name Title Organization Contact Information

Bruce Randel President Century Analytics bruce.randel@centuryanalytics.com

Trudy Cherasaro Senior Researcher Marzano Research 

Laboratory

trudy.cherasaro@marzanoresearch.com

The following individuals will be involved in the study implementation:

Name Role Organization Contact Information

Stephen Meyer Principal Investigator RMC Research 

Corporation

meyer@rmcdenver.com

303-825-3636

Xin Wang Research Analyst RMC Research 

Corporation

wang@rmcdenver.com

303-825-3636

Matt Linick Research Analyst RMC Research 

Corporation

linick@rmcdenver.com

303-825-3636

Brandie Ward Data Collection 

Oversight

RMC Research 

Corporation

ward@rmcdenver.com

303-825-3636

Gale Hairston SEA Study Liaison Missouri Department 

of Elementary and 

Secondary Education 

(MO DESE)

Gale.Hairston@dese.mo.gov

References

Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method (2nd ed.). New 

York: Wiley. 

Duffy, B., Smith, K., Terhanian, G., & Bremer, J. (2005). Comparing data from online and face-toface surveys. International Journal of Market Research, 47(6), 615–639.

Ehlert, M., Podgursky, M., Harrington, J. Xiang, P., Parsons, E., Frazelle, S., & Taylor Knight, L. 

(2011). Teacher and Leader Preparation in Relationship to Student Achievement in the 

Kansas City Region. Kansas City, MO: Kansas City Area Education Research Consortium.
Koedel, C., Parsons, E., Podgursky, M., & Ehlert, M. (2012). Teacher preparation programs and 

teacher quality: Are there real differences across programs? Washington, DC: 

CALDER/American Institutes for Research.

Lee, S. (2006). Propensity score adjustment as a weighting scheme for volunteer panel web 

surveys. Journal of Official Statistics 22(2), 329–349.

Loosveldt, G & Sonck, N. (2008). An evaluation of the weighting procedures for an online access 

panel survey. Survey Research Methods 2(2): 93–105.

Miller L. E., & Smith, K. (1983). Handling nonresponse issues. Journal of Extension On-line, 21(5). 

Retrieved on Apr 15, 2013 from http://www.joe.org/joe/1983september/83-5-a7.pdf.

National Council for Accreditation of Teacher Education. (2010, November). Transforming 

teacher education through clinical practice: A national strategy to prepare effective 

teachers. Washington, DC: Author.

Office of Management and Budget. (2006). Questions and answers when designing surveys for 

information collections. Washington, DC: Office of Information and Regulatory Affairs.

Oh, H.L. and Scheuren, F.J. (1983). Weighting adjustment for unit nonresponse. In W.G. Madow, 

I. Olkin, & D.B. Rubin (Eds.), Incomplete data in sample surveys, Volume 2: Theory and 

Bibliography (pp. 143–184). New York: Academic Press.

Taylor, H. (2005). Does internet research “work”? Comparing online survey results with 

telephone surveys. International Journal of Market Research, 42(1), 51–63.

Tourangeau, R. (1984). Cognitive sciences and survey methods. In T. Jabine, M. Straf, J. Tanur, & 

R. Tourangeau (Eds.), Cognitive aspects of survey methodology: Building a bridge 

between disciplines, pp. 73-100. Washington, DC: National Academy Press.

Tourangeau, R., Rips, L.J., & Rasinski, K. (2000). The Psychology of Survey Response. Cambridge: 

Cambridge University Press.

Willis, G.B. (1999). Cognitive interviewing: A “how-to” guide. Presentation at the 1999 Meeting 

of the American Statistical Association.