Testing a TPACK-Based Technology Integration Assessment Rubric

Although there is ever-increasing emphasis on integrating technology in teaching, there are few well-tested and refined assessments to measure the quality of this integration. The few measures that are available tend to favor constructivist approaches to teaching, and thus do not accurately assess the quality of technology integration across a range of different teaching approaches. We have developed a more “pedagogically inclusive” instrument that reflects key TPACK concepts and that has proven to be both reliable and valid in two successive rounds of testing. The instrument’s interrater reliability coefficient (.857) was computed using both Intraclass Correlation and a score agreement (84.1%) procedure. Internal consistency (using Cronbach’s Alpha) was . 911. Test-retest reliability (score agreement) was 87.0%. Five TPACK experts also confirmed the instrument’s construct and face validities. We offer this new rubric to help teacher educators to more accurately assess the quality of technology integration in lesson plans, and suggest exploring its use in project and unit plans. Developing and Assessing TPACK New understanding of the complex, situated, and interdependent nature of teachers’ technology integration knowledge—termed “technological pedagogical content knowledge,” or TPACK (Mishra & Koehler, 2006; Koehler & Mishra, 2008)—has led to inevitable questions about how this knowledge can be both developed and assessed. As the summaries below demonstrate, there is considerably more variety at present among TPACK development approaches than among assessment strategies. Koehler & Mishra (2005) recommend a learning-by-design approach to TPACK development in which educators, content experts, and technology specialists design instruction collaboratively, building TPACK as they do so (Koehler, Mishra & Yahya 2007). Other researchers promote immersive, content-based approaches, such as instructional modeling (Niess, 2005), collaborative lesson study with university researchers (Groth, Spickler, Bergner, & Bardzell, 2009), and meta-cognitive exploration of “deictic” TPACK that emerges as curricula and technologies change (Hughes & Scharber, 2008). Still others promote active, professional reflection and inquiry. Dawson (2007) and Pierson (2008), for example, suggest TPACK as a focus for teachers’ action research. Mouza & Wong (2009) propose a TPACK-based case development strategy in which teachers learn from their practice. Two approaches focus TPACK development within teachers’ planning. Roblyer & Doering (2010) recommend TPACK self-assessment as the first step in each stage of instructional decision-making. Harris and Hofer (2006; 2009) draw upon research about teachers’ planning practices to suggest a learning activities-based approach to selecting and combining curriculum-keyed teaching/learning strategies and complementary educational technologies. By contrast, published instruments that assess TPACK development and that have been tested for reliability and validity are of one type only: the self-report survey. Schmidt, Baran, Thompson, Koehler, Shin, & Mishra (2009) and Archambault & Crippen (2009) developed self-report instruments with multiple items keyed to each of the seven types of knowledge represented in the TPACK construct: technological (T), pedagogical (P), content (C), technological pedagogical (TP), technological content (TC), pedagogical content (PC), and technological pedagogical content knowledge (TPACK). Schmidt et al.’s survey was designed for repeated use by preservice teachers as they progress through their teacher education programs. It was also found to be reliable and valid for use at the beginning and end of shorter-duration summer courses in technology integration. Archambault and Crippen’s survey instrument was designed to be used by inservice instructors, and was found to be reliable and valid with a nationally representative sample of approximately 600 K-12 online teachers. Though the testing of these two instruments proved them to be quite robust measures, the challenges inherent in accurately estimating teachers’ knowledge via self-reports—in particular, that of inexperienced teachers—are well-documented. Unfortunately, research has shown that measured gains in teachers’ self-assessed knowledge over time are more reflective of their increased confidence regarding a particular professional development topic than their actual increased knowledge in practice (Lawless & Pellegrino, 2007; Schrader & Lawless, 2004). Self-report data should therefore be triangulated with external assessments of teachers’ TPACK knowledge. Since no instrument had been developed and published to date (to our knowledge) that supported this type of performance-based evaluation of TPACK, we decided to create and test one.