THE ROLE OF MULTIMODAL IN THE LEARNING SENSORIMOTOR INVOLVEMENT OF COGNITIVE SKILLS

The existence and extent of effects of small-scale haptic involvement on recall performance in an explicit memory task were investigated in two related experiments involving a total of 40 subjects. The memory task was structured in such a way that it abstracts two of the fundamental activities encountered when interacting with realworld equipment through physical control interfaces: 1) the identification of the state of the system to be acted on; and 2) the recall of the series of actions which are to be taken when the system is in that state. Furthermore, the experiments were designed as training experiments. Haptic sensorimotor involvement is evaluated as an aid to training -it is used during practice and not used during performance. Results from Memory Experiment 1 show (i.e., Ho: p < 0.01 for 5 minute, 30 minute, and 24 hour examinations) that experimental group subjects performed approximately 2.5 times better when they trained with both visual stimuli (i.e., a keypad graphic) and haptic involvement than when they trained without these visual and haptic stimuli. Memory Experiment 2 shows (i.e., Ho: p < 0.05 for 5 and 30 minute examinations) that experimental group subjects performed approximately 1.5 times better when they trained with the keypad graphic and haptic interaction than when they trained with the keypad graphic only. These results were obtained even though the design of the experiments was such that independent variables affected the training of only the action part of sequences (i.e., the training of the sequence label was not affected by the independent variables) and experimental group examinations required recall of sequences memorized under two different conditions. The results of the two memory experiments suggest that: 1) small-scale haptic involvement can act as a training aid for a purely cognitive task; and 2) small-scale haptic involvement can compliment visual stimuli as an aid to training in a purely cognitive task. Thesis Supervisor : Nathaniel I. Durlach Title: Senior Scientist, Department of Electrical Engineering and Computer 1.0 Introduction The increased commercial availability of interface and computational devices supporting the development of multimodal, spatially-oriented, interactive human/machine interfaces has contributed to a burgeoning interest in the field of human/computer interaction (Biocca, 1992). Devices such as binocular stereoscopic helmet mounted displays, binaural stereophonic spatialization devices, haptic interaction devices, and unobtrusive position/orientation sensors are enabling the creation of human/machine interfaces for interaction with telerobots and computational models of a form and type which were previously found only in specialized research and military efforts (e.g., Freund, 1986; Shaker and Wise, 1988; Tachi, Arai, and Maeda, 1989; Aviles, Hughes, Everett, et. al., 1990). In their most extreme manifestations, systems utilizing these technologies are attempting to remove the boundaries associated with devices mediating interaction between humans and remote task environments (physical and computer generated) and, in some sense, project the human into these remote or virtual environments. Virtual environment (VE) systems in particular are experiencing a high degree of interest from the military, commercial, and private sectors. Current and foreseen application domains for VE systems include: (1) design, manufacturing, and marketing; (2) medicine and health care; (3) entertainment; and (4) training (Durlach and Mavor, 1995). This discussion will focus on research issues associated with using VE systems and technology for training. From a training perspective, some key features of virtual environments are that they are: 1) interactive and adaptive; 2) reconfigurable in software; 3) multimodal; and 4) can generate supernormal situations (Durlach, Pew, Aviles, DiZio, and Zeltzer, 1992). These features allow VE-based training systems to be developed which can be precisely tailored to specific tasks and individuals. In many ways, however, VE-based training systems may be viewed as natural outgrowths of previous simulation-based training systems (e.g., flight trainers). In some form, all four aforementioned key VE features apply to these "classical" trainers. In what ways, therefore, are VE-based systems unique? First of all, in contrast to classical simulation systems, VE-based systems can be highly reconfigurable for BOTH the far-field (i.e., objects and interactions out of reach) and the near-field (i.e., objects and interactions within reach). Classical flight simulators, for example, only reconfigure the scenery and environment outside of the airplane (i.e., the far-field) and do not easily allow flexibility in the airplane display and control layout (i.e., the near-field). In contrast, a VE-based training system can simulate a variety of nearand far-field configurations while utilizing the same human/machine interface devices. The second key characteristic of VE-based training systems is the potential for allowing multimodal (i.e., visual, auditory, and haptic) interactions. Unlike classical computer-based simulation systems, near-field simulations can be generated which may seen, felt, and heard in a spatially-oriented manner. Virtual environment systems, therefore, are unique in that they can generate reconfigurable multimodal sensorimotor near-field interactions. A large and growing percentage of the tasks encountered in military, commercial, and private enterprises, however, are predominantly cognitive in nature. For example, many of these tasks require mainly that the human operator control and monitor a system through the use of a control panel. Although the main interaction with such control panels is through the sensorimotor system, the requirement to know what controls to operate and when to operate them is largely a cognitive task. One viewpoint is that, given the aforementioned trends, training for manipulative skills is unimportant. Strong proponents of this view argue that the majority of IMPORTANT skills, independent of frequency of occurrence, are largely cognitive in nature (Welford, 1976). By extension, it is argued that, since sensorimotor skills are not the main skills which must be trained, sensorimotor involvement in VE-based training systems is not required or is of secondary importance. In other words, it is assumed that sensorimotor involvement is not needed for the training of cognitive skills. In the research described in this thesis, these assumptions were tested and the effect of multimodal sensorimotor involvement in the training of predominantly cognitive tasks was examined in a task which requires subjects to: 1) identify a discrete system state, and 2) identify a series of discrete actions which must be taken in response to that state. Training effectiveness is measured and compared between 1) "classical" textbased training methods and 2) training methods which provide visual stimuli and which require haptic sensorimotor involvement. 2.0 Background Intellectual or cognitive skills "link perception and action and are concerned with translating perceptual input into a skilled response by using appropriate decisions. " Colley and Beach, 1989, p. 2 From a training research viewpoint, VE-based systems are interesting only to the extent that training efficiency, skill performance, or skill retention are influenced and/or psychophysical insight may be gained. From a practical viewpoint, issues of cost (e.g., training system cost, time/cost required to train to a certain level of proficiency, and resource use costs), safety (i.e., limiting trainee exposure to threatening situations), portability (i.e., ability to effect training at a variety of sites) and reconfigurability (i.e., ability to use the same system to train a variety of skills) are of import. Lower training effectiveness measures may be tolerable in certain situations if the system's practical factors evaluate positively for a given training system. Nonetheless, a crucial factor in the evaluation of any training methodology or system is its training effectiveness. To date, however, research relevant to assessing the quantitative impact of multimodal sensorimotor involvement in the training of predominantly cognitive tasks has been minimal. Current models that are directed towards the learning of cognitive skills often fail to consider the possible effects of sensorimotor involvement in the learning of these cognitive skills (e.g., Anderson 1990, Posner, 1989; Preece, Rogers, Sharp, Benyon, Holland, and Carey, 1994; Benyon and Murray, 1993; Frederiksen and White, 1993; Roberts, 1993). Most relevant research has occurred in the use of subject performed tasks (SPTs) and experimenter performed tasks (EPTs) in event/action recall memory experiments (e.g., Cohen, 1981; Cohen, Peterson, and Mantini-Atkinson, 1987; Koriat, Ben-Zur, and Nussbaum, 1990; McAndrews and Milner, 1991). These experiments, however, explore the effects of multimodal sensorimotor involvement on the recall of tasks which are inherently sensorimotor in nature. To the knowledge of the author, principled examination of the impact of multimodal sensorimotor involvement on the training of predominantly cognitive skills is non-existent. Why pursue work in this area? First of all, insight into the role of multimodal sensorimotor involvement in training will have strong implications for the practical design of VE-based training systems (e.g., assessing if the added complexity and cost of haptic and auditory components is worthwhile for a given training task). Second, experimental results may aid in extending current cognitive training theory to encompass sensorimotor involvement. The question still remains, however, as to what evidence or plausible mechanisms exist which indicate that multimodal sensorimotor involvement may have an effect on the training of cognitive tasks? Changes in training perform