Using speech interfaces to support human task performance in complex training environments
Creators: name: Vukovic, Maria
Publisher: name: RMIT University
Year: 2024
Subjects: classificationCode: 460202
subject: Autonomous agents and multiagent systems
classificationCode: 460806
subject: Human-computer interaction

description: In a complex and dynamic environment in which human-machine teaming is enhanced through the use of natural language communications, there is a need for workload-aware teams. Providing the machine with operator state information, would enable the machine to adjust its level of automation to maintain or improve task performance as the human becomes overloaded. Complex dynamic environments of this nature include large-scale military simulation training events, search and rescue, and any scenario in which elements of a team are remote such as in space. 

Adaptive systems are defined by Feigh et al. [43] as ``the technological component of joint human–machine systems that can change their behaviour to meet the changing needs of their users, often without explicit instructions from their users'. This research hypothesises that if simulated semi-autonomous aircraft can adjust their behaviour when they detect that the operator is experiencing high cognitive load, performance will be improved and the human will be afforded cognitive respite.

Three areas of research are covered in this thesis:  speech as a control modality in human-machine teaming; cognitive load measurement from speech and adaptive automation. The first stage of this research assesses whether the addition of speech as a control modality could improve task performance in complex and highly-dynamic environments. Subsequently, this research investigates how cognitive load could be measured from speech recorded during large-scale simulation exercises, and then how it could be measured from speech commands to aircraft agents. The final stage of this research combines previous findings, merging the control of simulated semi-autonomous aircraft using speech, with cognitive load estimation from that speech. The estimates of cognitive load are used to trigger adaptation of levels of automation for the semi-autonomous aircraft.

The initial research setting of Air Battle Management accessed through Defence Science and Technology Group, provides a unique opportunity to explore real problems and potential solutions in an operational setting. A typical scenario that exemplifies this research in the military air domain consists of Simulation Pilots controlling aircraft in a virtual battlespace to train Air Battle Managers (ABMs). ABMs report positioning information to pilots (in a similar way to Air Traffic Controllers in civilian air space). The research setting progresses to the development of a civilian drone package-delivery scenario in simulated airspace. ABMs, Simulator Pilots and drone operators share common tasks; they each control air assets in highly dynamic and complex 3-D space while ensuring safety, task performance and efficiency of action are maintained. They also all communicate with external players using radio communications. Together these required tasks reflect a more cognitive than physical workload on operators. It is this environment that motivated the research conducted and presented in this thesis.

This research shows that the addition of a speech interface to traditional keyboard and mouse control tasks, in the context of Simulator Pilot tasking under increasing workload, does not lead to a degradation in performance as much as using the single modalities on their own. 

This finding complements recent literature which found multi-modal interfaces of speech and touch improved task performance under increasing workload. It also exemplifies Wickens [151] Multiple Resource Theory;
speech affords participants more resources with which to action any increased workload by giving participants the capability to multi-task and use each modality independently. In this research, speech collected through the speech interface is also utilised for its paralinguistic properties. This research proposes and evaluates an approach to train a cognitive load classifier from a real-world speech dataset which contains unbalanced and sparse self-labelled data. A Partially Speaker Dependent model is derived which includes some data from the speaker being tested. This model demonstrates better estimation results than Speaker Dependent and Speaker Independent models. Speaker Dependent models repeatedly show improved estimation accuracy in reported literature however they rely on speech from one speaker. A Partially Speaker Dependent model is an additional training technique that maximises the use of real-world data when there is limited speaker-dependent speech data collected. Cognitive load class determination from self-labelled speech data is also investigated in this research. Self-labelled speech data provides some boundary determination challenges due to the non-linear nature of self-assessed workload. A sigmoidal curve boundary class division technique is proposed and evaluated. This class boundary division technique results in superior estimation of cognitive load over linear boundaries, thus contributing another technique to the literature on speech labelling using subjective labels.

The application of the above research findings is combined in a final empirical adaptive automation control scheme assessment in which simulated drones are controlled by speech and mouse to deliver packages. The data shows that adaptive automation triggered by cognitive load estimates (AA-Cognitive Load) improves task performance over both, adaptive automation triggered by task performance (AA-Performance) and No Automation control schemes. Self-assessed workload is shown to remain constant across both adaptive automation control schemes as tasking load increases. A key contribution to the literature on adaptive automation in human-machine teaming is the use of cognitive load estimates from speech commands to adapt levels of automation in the AA-Cognitive Load control scheme. Cognitive load estimates from electroencephalography have been used to adapt human-machine interfaces in previous research, however the use of cognitive load estimates from speech as a trigger is novel, and not yet reported in literature.

Cognitive load estimation from speech can be used in adaptive automation to facilitate task performance improvements of human-machine teams while maintaining an acceptable operator workload.