Iterative Usability Evaluation Methods Applied to Learning Technology Development

The goals of this project were to: a) carry out the evaluation of a learning system for teaching Civil Engineering students to use Geographic Information Systems software within the context of relevant problems and b) explore the efficacy of iterative usability testing as a tool for development of learning technologies. A series of three usability tests were conducted based on an initial proposed design of the learning system. In general, the results pointed to the role of the learning system as a support tool, and a consequent design that focuses on progressive scaffolding, flexibility, and logical consistency. The results also supported the efficacy of usability testing, in that iterations of the system improved as development progressed. Finally, the importance of scarce resources and education of the development team on the role of usability testing, were identified as central factors in determining the effectiveness of usability testing. Usability Testing and Prototyping as Formative Evaluation Methods Usability testing is the process of observing users interacting with and using a product to ascertain how well users function with the product. The iterative nature of usability testing comes when the testing is integrated into the design and development cycle resulting in incremental improvements. Characteristics that are generally found in usability testing (Dumas & Redish, 1999) are: 1) Each test has particular goals and concerns stated when designing the test; 2) The participants are representative of real users; 3) Participants work on real tasks; 4) Participants actions and words are recorded and observed; 5) Data is analyzed to diagnose problems and propose recommendations. Thus, in usability testing there is a controlled situation, recognizable with explicit goals, and nearly everything (times, comments, pauses, expressions, etc) is recorded. Usability testing is generally considered to be different than a traditional experiment; the latter consisting of formal hypothesis testing, the manipulation of an independent variable, and the random assignment of participants to experimental groups while usability testing is closer in nature to observational or exploratory research, where variables are not manipulated, and hypotheses are not required. Questionnaires and interviews are often administered to determine prior knowledge as well as satisfaction with the product. Users are often asked to adhere to a “talk-aloud” protocol, the process of explicitly stating what one is thinking during a given task, to provide rich data about ongoing thought processes. Usability testing typically employs a small number of users, a phenomenon referred to as “discount usability testing,” because this results in a discount in both cost and time (Nielsen, 1994). Due to the low number of participants, there is generally an emphasis on qualitative, rather than quantitative analyses, and inferential statistics with an emphasis on probability levels are not usually applied. Usability Testing in Education Some researchers who work within both the Human-Computer Interaction (HCI) and Learning Technology communities have suggested that the communication between the two is less than optimal, despite the fact that they share common subject matter (Dillon, 2000). Though usability testing of web-based learning technologies has become more common in recent years (Dean, 1999; Richardson, Swan, & Newman, 2000), it is still relatively rare. There are a number of reasons why usability testing has not become a standard tool for learning technology researchers. First, usability testing does not fit well within the most common conceptions of qualitative research in education, which tend to focus more on naturalistic observation, with intense prolonged contact in a field setting (Miles & Huberman, 1994). Second, the small number of participants typically used in usability studies does not allow for much generalization of results, particularly if the technology being studied has a wide and diverse effect on users. Third and conversely, the use of single-subject observation and intensive data collection and analysis does not easily allow for the large number of participants required for any sort of inferential statistics to be applied. Fourth, and most fundamentally, usability testing is usually implemented with well-defined tasks and well-defined performance measures. Despite these issues, there are some specific cases where traditional usability testing can be very beneficial for the evaluation of educational technologies. In particular, when used as a supplement to more traditional quantitative experimentation, a detailed analysis of a small number of typical users interacting with instructional software can provide invaluable insights and explanations with respect to the quantitative results (Hall, Philpot, Hubing, Flori, & Yellamraju, 2004).