Validation of Workload Algorithms Within the Integrated Performance Modeling Environment

The current research provides a review and assessment of implemented models of workload within the Integrated Performance Modeling Environment (IPME) to determine their level of maturity and validity at predicting cognitive workload. To focus the design and conduct of future validation efforts, a series of studies are being conducted within a combined Air Traffic Control (ATC) and Bakan task paradigm. This approach provides a framework necessary to begin the validation of the constructs underlying the workload models in a controlled manner. 1 Background Current trends in the Human Factors Engineering (HFE) community are towards the development of computational models of human behavior that can be incorporated within complex systems to compare systems performance, evaluate design alternatives for immersive and real system simulations, and predict human performance and workload prior to virtual and field-based trials of real systems (Allender, 2000). Additional research is being conducted on the efficacy of replacing human operators with Human Behavior Models (HBMs) in virtual simulations (Pew and Mavor, 1998). In particular, the application of HBMs to the analysis and assessment of cognitive workload is seeing a significant increase in the HFE community. To date there has been a lack of extensive validation of the computational models within the various HBM technologies used to predict cognitive workload. The majority of validation work that has been conducted by the community is notably performed by the developers of the toolset in question (Glenn, Neville, Stokes and Ryder 2004). This raises concerns within the HFE and modeling community that the workload predictions from HBMs may not be representative of human performance. The lack of objective validation becomes increasingly problematic as traditionally separate workload algorithms are now being combined in a single approach which increases the probability that errors in combined models may be compounded. Further, the source of the workload data integrated within each algorithm is also a concern as their derivation is associated with cognitive workload theories of questionable validity. The objective of the current research is to review and assess implemented models of workload to determine their level of maturity and validity at predicting cognitive workload. To focus the design and conduct of future validation efforts, a series of studies are being conducted within a combined Air Traffic Control and Bakan task paradigm. This approach provides a framework necessary to begin the validation of the constructs underlying the workload models in a controlled manner. 1.1 Information Processing and Cognitive Workload Human information processing has been studied in relation to theoretical models that describe the manner in which information is processed and the constraints that are imposed by the limitations of the cognitive system. In general, these theories are concerned with the human’s ability to encode, process, and respond to external stimuli within the environment. Research efforts have resulted in the development of several different models, each of which has focused on a particular aspect of information processing (e.g., vision, audition, interference, etc.) and the manner in which it is facilitated or inhibited. The Multiple Resource Theory (MRT) proposed by Wickens (1987) argues that incoming stimuli are processed by a set of dichotomous dimensions that can be differentiated on the basis of their particular functionality. Further, this four-dimensional model suggests that information is processed differentially depending upon the following characteristics: modality (e.g., visual vs. auditory encoding), processing codes (e.g., spatial vs. verbal), visual channels (e.g. focal vs. ambient), and stage (e.g., early vs. late processes). Interference between tasks is predicted to occur as a result of the tasks sharing similar content (e.g., two spatial tasks) and therefore drawing upon a common resource pool or structure. Cross-modal processing is predicted to facilitate dual-task performance by enhancing time-sharing, and by minimizing performance decrements due to the influence of task difficulty (Wickens, 1984). In practice, the identification of the resources responsible for task completion is of great importance for the measurement of performance and the development of a workload construct. The following section defines a workload construct founded in theories of information processing that is amenable to review and supports the development of various tools to facilitate its analysis and prediction. 1.2 Definition and underlying factors of Cognitive Workload To date, the concept of cognitive workload has been difficult to define due to the significant number of factors that can influence both its management and outcomes. The issue becomes further compounded when additional factors such as stress, fatigue, arousal, and emotions are considered as potential influencers of resource capacity (Kahneman, 1973). Clearly, the construct of cognitive workload must therefore draw from a multi-dimensional space in order to be clearly understood. For the purposes of the present discussion, the attentional resource theory reviewed provides a useful basis from which to form a foundation for the cognitive workload construct (Young & Stanton, 2002). As discussed, these theories assume that task performance is moderated by a finite attentional capacity that may be allocated to one or more tasks to achieve a desired level of performance. The ratio of available attentional capacity to task demands determines the degree of performance achievable within a given setting. As task demands increase beyond the point of currently allocated resources, task performance is maintained at a static level by allocating further resources, or degrades if resources are insufficient to meet task demands (Xie & Salvendy, 2000; Young & Stanton, 2002). The factors underlying cognitive workload may also be shown to have both static and dynamic attributes that vary according to a timeline (North & Riley, 1989; Hendy & Farrell, 1997). Measures of processing demand can then reflect the state of cognitive workload across a specified time interval, as well as an indice of current demands at a specific moment in time. In conjunction with the attentional resource foundation, the time factor allows the measurement of the amount of mental activity, or effort, necessary for an individual to conduct a specified task over a given period of time. This clearly suggests that workload should be assessed through a dynamic simulation rather than considering only the characteristic demands imposed by tasks. For the present discussion, cognitive workload is therefore understood to refer to a multi-dimensional rather than a uni-dimensional construct (Reid & Nygren, 1988; Xie & Salvendy, 2000). The structure and interaction of these multiple dimensions in turn determines an operator’s ability to conduct task management and task completion within a complex work environment. This includes the management of multiple tasks either simultaneously by timesharing or sequentially by taskscheduling up to the point of task overload. Further, cognitive workload also refers to the resolution of interference effects between tasks that may, but not necessarily, result in degradations of human performance. 1.3 Workload Assessment Methods The utility of the cognitive workload construct has seen fruition in two major approaches to the evaluation of human performance. These approaches involve the measurement of cognitive workload from human subjects in real and simulated environments as well as studies using workload models in simulated environments to predict cognitive workload and performance. In contrast to empirical measurement techniques using human subjects, analytical measurements use mathematical, algorithmic, task analysis and computer simulation techniques to predict estimates of cognitive workload without the presence of a human-in-the-loop. They provide both a predictive and evaluative capability, and the ability to conduct workload assessments without the costs associated with using a physical system, or human operators, makes the analytic approach desirable. However, until recently, they have been the least common method employed to assess cognitive workload and have not been evaluated to the extent of the empirical approaches (Cain & Hendy, 1998; Xie & Salvendy, 2002). The use of Task Network Models (TNM) provides a software infrastructure from which to develop a model of a real or proposed system under a specific set of conditions. An advantage of this approach is the ability to study the behavior of a system as time progresses. These models are typically used to generate estimates of mission or composite task completion times, predictions of cognitive workload, and occasionally task accuracy or performance. A number of commercial and open-source TNM packages are available; however, two that cater to human performance modeling are IMPRINT and IPME. The current study is focused on the IPME due to its broader use within Canada and the United Kingdom. A core assumption of the TNM paradigm is that human behavior can be modeled as a set of interrelated tasks. The data used to drive the performance of the model (e.g., time, cognitive workload values) is assigned by a human factors analyst and based on a task analysis of the interaction between the operator and a specific system component or a unique task. As such, the TNM modeling environment does not generate time constraints on human behavior based on fundamental neuropsychological data (Craig, et al., 2002), but is constrained to behave within the context of the data input by the analyst. These models are then executed using a discrete event simulation engine that models the behavior of state variables at a discrete set of points in time (Banks, Carson, Nelson & Nicol, 2001)