An analysis of foreign language achievement test drafts

Abstract: In order to obtain concrete information to improve test writing, this study analyzes 13 achievement test drafts written by graduate teaching assistants, lecturers, and adjunct faculty teaching introductory and intermediate college Spanish. The analysis focuses on factors of test method and content likely to create variance in test performance unrelated to learners' abilities. Bachman and Palmer's (1996) rubric for describing target language use and test tasks functions as an organizational tool to describe and analyze the drafts. Results reveal patterns of inappropriate input and inadequately specified procedures, tasks, scoring criteria, and expected responses. Findings suggest common pitfalls for the novice test writer to avoid in test development, and provide a starting point for teachers of any language who want to improve their own achievement tests. Introduction Foreign language teachers both want and need information about their students' current level of ability, progress, and readiness for subsequent levels of instruction. One common source of such information is an achievement test. Achievement tests are foreign language assessment tools directly tied to a particular curriculum and are used to evaluate student progress toward or mastery of course objectives (Hughes, 2002). As a result, such tests are necessarily context specific, and are influenced by the course materials and syllabus, the instructor's preferred teaching methodology, the student population, local administrative factors, and so forth. Many teachers choose or modify commercially produced tests distributed by their textbook publisher; others create their own. In each of these cases, teachers need adequate knowledge of language testing to be able to provide a reliable and valid instrument to measure their students' level of achievement. In many cases, however, teacher certification programs and university foreign language departments are unable to provide adequate professional development in language testing due to time constraints and competing priorities. Nonetheless, teachers often create their own tests and quizzes with which to make decisions about their students' achievement. In these circumstances, measurement error may arise due to the use of inappropriate test content or method that produces construct-irrelevant variance (i.e., variance unrelated to students' language abilities), potentially reducing the validity and reliability of the test. The purpose of this study was to determine whether common sources of construct-irrelevant variance related to test content and method can be identified and targeted for improvement during the test writing process. This study analyzed 13 achievement test drafts for introductory and intermediate Spanish written by university graduate teaching assistants, lecturers, and adjunct faculty beginning professional development in test writing. The study identified several common types of problems related to construct-irrelevant variance that may reduce the validity or reliability of a test. These results point to aspects of test writing that can be improved through professional development for teachers. An awareness of the types of content and method problems found in test drafts written by these teachers-intraining may help other test writers to avoid common pitfalls during the test writing process. Two research questions guided this investigation: (1) Which aspects of the test items written by instructors would become a source of construct-irrelevant variance if this test were administered? (2) Which sources of construct-irrelevant variance have the greatest impact in terms of either frequency or global effect? To answer these questions, the present study utilized Bachman and Palmer's (1996) "Task Characteristics" and their model of language test usefulness as tools to describe problems of test content and method identified in the test drafts. The Task Characteristics allow for a structured evaluation of the drafts' content and test method, which in turn serve as the basis for the suggestions regarding professional development in test writing that conclude this study. …