Problems with Assessment Validity and Reliability in Web-Based Distance Learning Environments and Solutions

Assessment of learning is critical to the learners, teachers, and designers of learning environments. Current assessment techniques in web-based distance learning apply age-old techniques to a new medium and are not adequate for web-based distance learning environments (WBDLE). The goals of this article are to extend existing critiques of assessment validity to web-based distance learning environments, define the problems with current WBDLE assessment methods using a learner, task, inference framework, and present three unique solutions to the problems. We describe our solutions, present validity and reliability issues for each method, and report instructor and student experiences with these solutions. ********** Web-based distance learning environments (WBDLE) are growing exponentially and appear to be driven by a technology focus and not a learning focus. WBDLEs are being used with students from K-12 to post-graduate levels and the best thing people say about them is that they are no worse than traditional classrooms! No significant difference is the WBDLE buzzword. There are many challenges in WBDLEs. They include the validity and reliability of our measures, quality of assessments, and the myth that web-based course management systems provide an all inclusive learning package. This article focuses on validity and reliability, evaluation of current assessment practices, challenges that must be addressed (including web course management systems), and solutions to the problems. "Educational assessment seeks to determine how well students are learning and is an integral part of the quest for improved education. It provides feedback to students, educators, parents, policy makers, and the public about the effectiveness of educational services" (National Research Council, 2001, p. 1). Measurement theories on reliability and validity are critical components in any assessments that lend credibility to the inferences drawn. WBDLEs present unique challenges to assessment of learning and are in desperate need of information that can give credibility to its assessment practices. Brookhart (2003) suggested that classroom assessment measurement theory is significantly different from its parent large scale assessment theory because of differences in the context, integration of assessment and instruction, and formative-summative assessment purposes. Since web-based distance learning environments are significantly different from traditional classrooms (Suen & Parkes, 1996) Brookhart's line of thinking must be extended further to rethink the measurement theories for WBDLEs. The first goal of this paper is to extend the measurement theories on reliability and validity to include WBDLEs. Suen and Parkes, (1996) and Reeves, (2000) have called for improved practices in assessing web-based learning environments. For example Reeves suggested that online learning environments have to concentrate on improving assessment practices by concentrating on alternative assessments such as cognitive, performance, and portfolio assessments. He also stated that "colleges and universities are increasingly expected to help their student develop positive habits of mind such as commitment, motivation, and ethics as well as higher order outcomes such as problem solving, intellectual curiosity, and critical analysis. These new outcomes are much harder to assess than discrete knowledge and skills" (p. 104). Suen and Parkes (1996) suggested that distance learning "has a myriad of implications for the learning and assessment processes. Among these implications are (a) the isolation of the learner from resources, support, and peers; (b) the lack of face-to-face interaction with instructors; and (c) delayed feedback. These factors have, in part, necessitated a larger quantity and diversity of media and technology." The second goal of this article is to use the extended reliability and validity issues to evaluate current assessments in WBDLEs and see whether current assessments use new and diverse media or have just re-invented the wheel (and rather poorly) by converting methods used in traditional classroom settings to web-based learning environments. …