Assessing Mathematical Knowledge for Teaching 1 Running head : Examination of the LMT Assessing Mathematical Knowledge for Teaching : An Examination of the Learning Mathematics for Teaching Instrument

This study examines the factor structure of the Learning Mathematics for Teaching (LMT) instrument when used for program assessment with undergraduate pre-service teachers. While the one-factor and three-factor models previously identified for in-service teachers did not fit when examined with pre-service teachers, there was reason to believe that a bi-factor model, which specifies all items to load on a general “math ability” factor and subsets of items to load onto a content specific factor, may be more appropriate. However, subsequent analysis of the bi-factor model was not able to converge to a solution. Assessing Mathematical Knowledge for Teaching 3 Assessing Mathematical Knowledge for Teaching: An Examination of the Learning Mathematics for Teaching Instrument As a response to increased calls for accountability in teacher training programs, there has been a growing interest in the assessment of knowledge required for teaching mathematics. As the knowledge requirements for teaching evolve, our tools for assessing that knowledge should change as well. One particular instrument currently being used in research studies on the assessment of knowledge for teaching mathematics is the Learning Mathematics for Teaching (LMT) instrument (Hill, Schilling, &Ball, 2004). The knowledge domains the LMT purports to measure have specific implications for teacher training programs. By identifying and developing items to assess the knowledge domains that are important to teaching mathematics, Hill et al. (2004) have provided teacher training programs an opportunity to assess learning outcomes that relate to mathematical knowledge domains. The inferences from these assessments can assist training programs in identifying ways to increase teacher knowledge for teaching mathematics. The LMT has primarily been used for in-service teacher training program assessment. In-service teachers would typically be administered this instrument during teacher development workshops to assess their growth in mathematical knowledge as a result of those workshops. However, its current use in undergraduate pre-service programs has been sparked by their interest in outcomes assessment related to the mathematical knowledge domains measured by the LMT. Most undergraduate teacher preparation programs are held accountable for their ability to prepare pre-service teachers for the classroom. To this end, programs need to be aware of their students’ knowledge, skills, and abilities related to teaching. Program assessment is a means to this end. The Learning Mathematics for Teaching Instrument (LMT) The Learning Mathematics for Teaching instrument (LMT) emerged from a project that focused on developing test items measuring the knowledge required for teaching mathematics. Instruments Assessing Mathematical Knowledge for Teaching 4 such as the Praxis already exist to measure mathematical content knowledge; however, Hill et al. (2004) were more interested in developing items that would assist in identifying additional types of knowledge. These types of knowledge are Knowledge of Content (CK) and Knowledge of Student and Content (KCS). Knowledge of content has two subcategories: common content knowledge and specialized content knowledge. Common content knowledge consists of mathematic knowledge expectedly held by an average adult. Specialized content knowledge consists of an understanding of mathematical concepts derived from experience in teaching. This includes being able to present the same concept in different ways and understanding different methods of deriving answers to problems. Knowledge of student and content consists of the ability to identify common mistakes students make and how, as well as indentifying students’ problem solving strategies. The conceptualization of these knowledge domains was generated through an integration of the pedagogical knowledge theories produced by Ball & Bass (2000), Grossman (1990), and Shulman (1987). An in-depth analysis of the nomological network comprising their conceptualization of pedagogical knowledge can be found in Schilling, Hill, & Ball (2004). The items for the LMT were written using three math content areas: numbers and operations, geometry, and patterns, functions, and algebra. These content areas were chosen because they represent a large portion of the mathematical content taught in K-6. Hill et al. (2004) began their research for item writing by analyzing student course work, analyzing qualitative data on teacher experiences, and reviewing curricula of the chosen content areas. The developers also ensured the items were devoid of references to any pedagogical technique that would advantage one teacher over another. This prevented the authors from confounding their inferences about teachers’ mathematical knowledge for teaching with teachers’ mastery of pedagogical technique. These stipulations formed their model for item development. As a result, the authors developed 138 selected response items. Assessing Mathematical Knowledge for Teaching 5 Assessing Factor Structure Hill et al. (2004) developed and piloted three forms of selected response items written to represent the aforementioned knowledge domains and content areas. An exploratory factor analysis (EFA) was conducted using ORDFAC, a factor analysis software program, to identify the factor structure underlying the measure. This program was chosen for its ability to perform factor analysis on data sets that include subgroups of items corresponding to a common stimulus, called testlets. Although the factors did not align as theorized due to the multidimensional items, the developers concluded that a three factor model was adequate for explaining the data. They labeled the domains knowledge of content in numbers and operations, knowledge of student and content in numbers and operations, and knowledge of content in patterns function and algebra. In the same study, the authors also conducted a five-factor bi-factor analysis in which they found that a substantial number of items loaded onto a general math knowledge factor. Items also loaded differentially onto four factors representing items written in a combination of content (e.g. number concepts) and knowledge domains. Multidimensionality was found, with no firm patterns of loadings. Though some evidence of the knowledge domains was found, the authors concluded that more studies should be conducted with more items representing the three content areas and two knowledge domains. Hill et al. (2004) further investigated the properties of each item using IRT methods. The items were split into the four areas observed in the previous factor analysis results. A total of 61 numbers and operations CK items, 48 numbers and operations KSC items, 43 geometry items, and 29 patterns, functions and algebra items were piloted on a non-random sample of teachers. The developers found adequate reliabilities across all forms. However, multidimensionality amongst several of the items was apparent. Again, this finding underscores the need for more psychometric work with the instrument. In another study, Hill and Ball (2004) used the items from the numbers and operations scale to assess the development of content knowledge for in-service teachers participating for 1 to 3 weeks in Assessing Mathematical Knowledge for Teaching 6 California’s Mathematics Professional Development Institutes. Through the use of three parallel forms they were able to obtain pre-test and posttest data on 398 teachers. Reliabilities for these forms ranged from . 71 to . 78. Through the use of mixed models in Hierarchical Linear Models (HLM), the authors were able to identify significant growth in their sample. They also found that the duration of the development institute contributed to more growth amongst the teachers. As expected, teachers participating for three weeks showed more growth in numbers and operations content knowledge than those who participated for less than three weeks. This further demonstrates the usefulness of the instrument in measuring content knowledge. In their 2005 study, Hill, Rowan, and Ball attempted to provide validity evidence for the purported link between teacher content knowledge and student achievement in mathematics. They chose a sample of 334 first grade and 365 third grade teachers from 115 schools participating in a comprehensive school reform program. From the classrooms of participating teachers, the study obtained a first grade cohort of 1330 students and a third grade cohort of 1773 students. Several instruments were used to gain information on student achievement, student demographics, teacher background, and classroom characteristics. Numbers and operations and algebra scales were used to capture content knowledge for teaching mathematics. Using IRT methods for analyses, a reliability estimate of . 88 was found for the mathematics knowledge items. Through the use of linear mixture modeling analyses the authors found that mathematical knowledge for teaching, as measured by the LMT items in numbers and operations and algebra, predicted student achievement in the first and third grades. Need For Additional Measurement Studies Though these studies have provided insight into the instrument’s usefulness with in-service teachers, there have been very few inquiries into its use with other samples, such as undergraduate teacher preparation programs (i.e. pre-service teacher development). Recent investments into the use Assessing Mathematical Knowledge for Teaching 7 of the LMT for program evaluation have been made by several undergraduate teacher preparation programs. However, before inferences can be made about pre-service teacher scores on the LMT, some psychometric issues must be resolved. Purpose The responsibility of gathering validity evidence for an instrument rests with the user when the instrument is being used with a population other than the population for which it was created (American Educational Res