# Deep Neural Network-Based Modeling of Multimodal Human-Computer Interaction in Aircraft Cockpits
## 1 Introduction  
With the rapid development of aviation industry technology, various airplane cockpits widely use human-computer interaction (HCI) technology. In flight, the operator needs to deal with all kinds of flight information and complete human-computer interaction through a large number of manual operations. A large number of manual operations reduce the efficiency of human-computer interaction and easily cause operator errors. The gaze interaction [1, 2, 3] can effectively solve this problem. The gaze interaction defines the position and trajectory of the operator's gaze as computer interaction commands. The gaze interaction can replace the traditional peripheral interaction methods of mouse or keystrokes and reduce the operator's manual operation. However, the \"Midas touch [4]\" problem limits the application of gaze interaction techniques. \"The Midas touch\" problem makes it difficult for the interaction system to recognize the operator's intention. When the operator looks at the target, the gaze interaction system cannot tell whether the operator is viewing information or triggering an action.  
Many researchers have added speech, gesture, and EEG modalities to the gaze interaction to establish a multimodal interaction approach to solving the Midas touch problem. In contrast to gaze interaction, multimodal interaction methods that combine gaze andother modalities can discern the intent of the operator's gaze behavior. In addition, multimodal interaction enables computers to have human-like perceptual functions. It makes the human-computer interaction process closer to exchanging information between humans [5]. Among the gaze-based multimodal interactions, gaze+EEG has more potential applications than gaze+speech and gaze+posture. This is because gaze+EEG is neither disturbed by other users' speech nor requires additional physical movements. However, in the existing studies on gaze+EEG for multimodal human-computer interaction, the interaction objects are mostly larger-sized, stationary targets. They do not apply to selecting smaller-sized, moving aircraft targets on the cockpit radar page. Therefore, this paper proposes to build a multimodal human-computer interaction model that uses a long short-term memory network (LSTM) and transformer architecture for the operator's target selection in the cockpit radar page. The model can deeply extract features from eye movement and EEG data, thus effectively improving the accuracy of moving and small radar target selection.  
A brilliant, comfortable, personalized driving experience is the future direction of intelligent cockpit development. Multimodal human-computer interaction technology is one of the core driving forces for developing intelligent cockpits. This study explores the application of multimodal human-computer interaction in future intelligent cockpits. In a gaze+EEG multimodal interaction, the gaze provides a clear visual focus, and the EEG reflects the cognitive state of the manipulator. This interaction method can understand the user's intention more comprehensively and improve the accuracy of the interaction. However, in current practical applications, we must address issues such as integrating eye tracking with brain-computer interface devices, system latency, and user adaptability. Nonetheless, the multimodal human-computer interaction technology of gaze+EEG still has a lot of potential for development and application in the future because it can enhance the level of cockpit intelligence and user experience.