# Use of a DNN in Recording and Analysis of Operator Attention in Advanced HMI Systems
## 3 Operator Attention Tracking System Using a DNN  
Figure 3 presents the modular structure of a designed workstation for recording and analysing the attention of operators of advanced systems using an HMI. Within Module A are sets of infrared sensors to detect the position and orientation of the observer's pupil (1) and scene observation cameras (2). The module is responsible for acquiring the video stream of the observed scene and recording the spatial position of the observer's pupils. Module B performs the tasks of attentional coordinate detection (3) and fixation detection (5). Module C performs AOI detection tasks in the video stream with the trained DNN [26; 27; 28; 29; 30]. Its main component (4) performs the task of detecting in the video stream the presence of instruments that are part of the HMI in use. This is completed by using the trained weight set obtained from the set of interface components (4a). The direct instrument recognition block (4b) in the video stream uses a deep neural network (4 bb). The authors assumed that a set of pre-trained architectures provided by the DNN Toolbox Matlab(r) library environment would be used in the DNN simulation studies. A modified KLT algorithm (see Algorithm 1) was implemented to prepare a training set that would contain a series of video frames of the operator cockpit instruments of a given  
Figure 3: Schematic diagram of a modular workstation structure for recording and analysing operator attention in advanced HMI systems. (**A–D**) denotes consecutive modules for signal processing in the system. station. Its task is to match the template image \(T(x)\) with the input image \(I(x)\). The vector \(x\) contains the coordinates of the image \([x,y]T\). ```
Input: \(I(x),\epsilon\) Output: \(p\) begin do{  warp \(I(x)\) with \(W(x,p)\to I(W(x,p))\)  the error \(T(x)\to I(W(x,p))\)  warped gradients \(\triangledown I=[I_{x},I_{y}]\) evaluated at \(W(x,p)\)  the Jacobian of the warping \(\frac{\delta W}{\partial p}\)  steepest descent \(\triangledown I\frac{\partial W}{\partial p}\)  inverse Hessian \(H^{-1}=\left[\sum_{x\in T}\left[\triangledown I\frac{\partial W}{\partial p }\right]^{T}\left[\triangledown I\frac{\partial W}{\partial p}\right]\right]^{-1}\)  multiply steepest descend with error:  compute \(\Delta p\)  update parameters \(p\gets p+\Delta p\) ]while\((\Delta p<\epsilon)\) end;
```  
**Algorithm 1** Image characteristic point detector tracker-KLT algorithm  
In matrix notation form, it can be represented accordingly:  
\[W(x,p)= \begin{bmatrix}x+p_{1}\\ x+p_{2}\end{bmatrix}  \tag{1}\]  
\[\Delta p=H^{-1}\sum_{x\in T}\left[\triangledown I\frac{\partial W}{\partial p }\right]^{T}\left[T(x)-I(W(x,p))\right] \tag{2}\]  
\[H=\sum_{x\in T}\left[\triangledown I\frac{\partial W}{\partial p}\right]^{T} \left[\triangledown I\frac{\partial W}{\partial p}\right] \tag{3}\]  
In this form, the algorithm was introduced as a tracker function into VideoLabeler, which allowed the generation of a sequence of images into the GroundTruth variable that constitutes the training set for each simulator. It should be noted that if tracked points are lost from the field of view (see Figure 4), it is unable to re-determine their location when they reappear--hence the need to use machine learning for object recognition. An example of video recording showing the loss of ROI3 when the operator performs hand movements--the lost Region of Interest (ROI) is marked with a dashed line (left bottom corner of the cockpit image on Figure 4). The data obtained in the marking session exported to the GroundTruth object (see Table 1, Figure 5) contain, among other things, information about the time (in seconds since the beginning of the video) and the position of rectangular object envelopes in the form of centre coordinates and dimensions (in pixels): \([x,y,W,H]\)As we already know, only a few layers in a deep neural network are responsible for selecting image features.