DRAFT Comparative Cognitive Task Analysis

Cognitive Task Analysis (CTA) is a recognized technique for analyzing how people operate within a specific task domain and with a specific toolset. In this paper we introduce an improved variant, Comparative CTA (C2TA) that allows the separation of effects due to basic human cognition from those due to the toolset. An example from weather forecasting is provided, including sample results and recommendations that would not have emerged from traditional CTA. Intro Weather forecasting is a complex process. The supporting information is multidimensional, distributed, and often uncertain. It includes both “raw” observations (e.g., current temperature, winds, pressure, clouds, precipitation, radar returns, satellite pictures, etc.) and analytic weather models, at varying scales predicting future conditions. How does the designer incorporate User-Centered Design and Human Centered Computing into this complex and specialized domain? How does the designer gain enough knowledge of the users’ tasks and processes to provide useful assistance? And how does the designer disentangle the effects of task, training, teamwork arrangements, and basic human cognition from those of the design of the tools? The traditional way human factors engineers approach this problem is to perform a task analyses to determine how people operate in a specific domain on a specific task. Cognitive Task Analysis (CTA) is a set of methods that takes into account the perception (i.e., vision), cognition (i.e., decision making), and motor actions (i.e., mouse movements) to accomplish a task. Running Head: Comparative Cognitive Task Analysis 3 In this paper, we build on traditional CTA methods by suggesting comparative cognitive task analysis can help solve the above problems. Comparative CTA (C2TA) is based on replication studies conducted in different environments. Because it derives data from more than one environment, C2TA provides insight into interface design that single site studies and single CTA methods cannot. There are many versions of task analysis ranging from time and motion study [1] to GOMS (Goals, Operators, Methods, Selection rules) analysis [2] to Ecological Interface Design (EID) [3]. Each is best suited to a subset of design problems. For example, GOMS analysis is a keystroke-level process for describing human-computer interactions (e.g., mouse and keyboard interactions). EID focuses on how the operator interacts with indicators of physical functioning such as in a power plant or manufacturing control room. CTA is especially useful in situations where the task is heavily dependant on human interpretation and integration of dynamic and highly uncertain data [4]. Weather forecasters typically deal with large amounts of data over time and space. Additionally, the information they examine is uncertain on several dimensions (i.e., the predictive weather models that are run may be based on a small number of data points in some areas (like in the middle of the ocean), which necessitates interpolating from the current data, which may cause the final output to be more uncertain). The dependence on a meteorologist interpreting the weather models, the dynamic nature of weather, and the uncertainty in the weather models makes weather forecasting an excellent candidate for CTA. However, most of the data analyzed by CTA methods come from a single source (i.e., most CTA studies have been performed on a single system and/or a small group of people). While the single approach is adequate in many situations, it may not be as generalizable as it Running Head: Comparative Cognitive Task Analysis 4 could be. That is, any problems can typically be traced to the interaction between the person and the system. You may discover, for example, that a specific pointing device is not very effective on a particular system, but you do not know if that is a limitation of the pointing device or the way people think; you only know that the combination of people and pointing device on the task you are examining is not very effective. By examining radically different tools (i.e., different types of pointing devices on similar tasks, you can start to dissociate the effects of cognition and tool. For example, the pen, the typewriter, and the computer keyboard are all tools that can be used for writing a document. The writing process consists of planning, composing, editing, and production (writing / typing). The quantity and sequence of these processes is differentially supported by the three tools. For example, the computer supports longer compositions, however, the writer plans longer before editing with a pen [5]. This may be because editing with a pen includes crossing out, rewriting, cutting pages apart and taping them back together, arrows for inserts, etc. and then repeating the production process (re-writing) on clean paper. Editing on a typewriter uses similar cross out, cut, glue, and retype processes. With both of these tools, the re-write (production) process is effortful. However, writers using a computer edit more as they write and new versions do not require re-doing the physical production [6]. The data for the two analyses reported here were collected during two studies in two very different locations, a United States Navy (USN) Meteorology and Oceanography (METOC) center in California and a Royal Australian Navy (RAN) METOC facility. These studies employed the methods of both naturalistic field study and quasi-naturalistic observation in a laboratory field study. The studies were part of a project to provide improved tools for Navy Running Head: Comparative Cognitive Task Analysis 5 weather forecasting. Only by understanding current practices and forecasting tools could improvements be suggested that would make the process more efficient. The two studies allowed us to map the information usage of decision maker to information visualization tools and to compare the mappings of USN and RAN forecasters in order to distinguish between effects that are dictated by the tools and training of these specialists and those due to basic human cognition. In the remainder of this paper we will first briefly describe the data collection at the two sites. Then we will review the results of the C2TA and show how suggestions for the design or redesign of tools flow from the C2TA results. More detailed results from both studies can be found in [7, 8]. Two Studies USN (2000) The first of the two studies took place in San Diego, CA at a Naval meteorological and oceanographic location. This laboratory field study provided naval forecasters in a simulated METOC center with computer access to their usual forecasting tools. Most of the forecasting information came from numerous meteorological web sites including military, non-military government, and university sites. Three pairs, consisting of a forecaster and technician took part in the study. Each pair developed a forecast and prepared a forecast briefing for an air strike to take place 12 hours in the future on Whidbey Island, WA. All actions were video taped and the participants were requested to “talk aloud” so as to produce a verbal protocol. Running Head: Comparative Cognitive Task Analysis 6 RAN (2001) The second study was a naturalistic observation of RAN forecasters working at a Weather and Oceanography Centre at an airbase in eastern Australia. They too were forecasting for 12 hour (and 24 to 72 hour) air operations, prepared forecasts and forecast briefings, and used computer-based tools. As with the USN forecasters, most of the forecasting information came from numerous meteorological web sites. Also, like the USN forecasters, they were video taped and instructed to “talk aloud” to produce verbal protocols. By retaining the task (forecasting) and moving to another group of practitioners with different tools, training, and teamwork practices, we disentangle the effects due to human cognition versus effects due to the tools used. There-by replicating basic results, extending findings, and drawing conclusions about how to better support the common forecasting tasks for both groups. Results Information Usage Comparative CTA can tell two kinds of stories. Similarities in classes of information usage that are independent of the tools, training, and teamwork patterns imply basic processes of human cognition. In contrast, we can impute differences in information usage patterns as being due to the impact of differences in tools, training and teamwork. To find either, we must code the verbal protocols to capture the way the forecasters use information. To analyze these data we selected usage encodings that capture what the forecaster did with the information. In other Running Head: Comparative Cognitive Task Analysis 7 reports we have examined the format of the information (text, graph, animation, etc.) or the form of the information (qualitative or quantitative) [7, 8]. The major encoding categories are extracting information, comparing information from two or more visualizations, deriving information by combining what was available in the visualization with the forecaster’s knowledge, and recording information. Extracting information occurs when a forecaster examines a visualization and extracts some sort of local or global features that are explicitly represented in the visualization. Comparing information occurs when a forecaster compares two or more different visualizations. Deriving information occurs when a forecaster goes beyond the information on the visualization and makes inferences or combines the displayed information with their background knowledge. Deriving implies that some of the information comes from the forecasters’ general domain knowledge rather than the information about the specific conditions for the current forecast. Recording information occurs when the forecaster writes down or copies information for their weather prediction. Table 1 defines each and gives an example. Note that, in terms of exp