# A Real-Time Detection of Pilot Workload Using Low-Interference Devices
## 3 Methods
### Ensemble Learning
#### 3.2.3 FCN-Transformer  
The FCN-Transformer model serves as a pivotal base learner, incorporating a self-attention mechanism to conduct an in-depth analysis of inputs. It combines the advantages of fully connected networks (FCN) and Transformers, effectively extracting sequential features from complex data and particularly excelling at capturing long-range dependencies and intricate patterns within sequence data. In practice, feature data are continuously input in 10 s time windows, embodying certain time-series characteristics, which leverages the strengths of the Transformer model to enhance prediction accuracy. * Input Linear Transformation LayerThe input first passes through a linear transformation layer, denoted as fc1. The purpose of this layer is to transform the dimensions of the input data from the original feature space to an intermediate representation space. The linear transformation is defined by the following formula:  
\[x^{\prime}=ReLU(W_{1}x+b_{1}),\]  
where \(x^{\prime}\) represents the input feature vector, and \(W_{1}\) and \(b_{1}\) are the weights and biases of the linear layer, respectively. ReLU is applied to introduce non-linearity, aiding the model in learning more complex features. * Transformer Encoder Layer  
A multi-head self-attention mechanism is at the core of the Transformer architecture, aimed at enabling the model to capture information from different representation subspaces simultaneously. The input sequence X first undergoes three sets of linear transformations to generate the query (Q), key (K), and value (V) matrices:  
\[Q=XW^{Q},K=XW^{K},V=XW^{V},\]  
where \(W^{Q},W^{K}\), and \(W^{V}\) are learnable weight matrices. We utilize the dot-product attention function to calculate the similarity between queries and all keys, followed by normalization through the softmax function to compute the attention scores:  
\[Attention(Q,K,V)=softmax\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}}\bigg{)}V,\]  
where \(d_{k}\) denotes the dimensionality of the key vectors, which serves to stabilize gradients. The softmax function ensures that all attention scores are positive and add up to 1. In order to enhance the expressive power of the model, the aforementioned process is executed in parallel across multiple heads, with each head corresponding to a different weight matrix. The output is then integrated through a linear transformation after concatenating the results from all heads. \[MultiHead(Q,K,V)=Concat(head_{1},\dots,head_{n})W^{O},\]  
where each \(head_{i}\) is an independent self-attention operation result. Through iterative parameter tuning, this study employs an 8-head self-attention mechanism. \(W^{O}\) represents another learnable weight matrix. The Transformer model adopts residual connections to help alleviate the gradient vanishing problem in deep networks. The output of each sublayer, whether it be a self-attention layer or a feed-forward network, is added to the input of that sublayer and then normalized within the layer:  
\[LayerNorm(x+Sublayer(x)),\]  
where Sublayer(x) represents the output of the sublayer itself. This design enhances the stability and efficiency of the training process. Each encoder layer also incorporates a feed-forward network, which independently applies the same operation to the output at each position. This network consists of two linear transformations and a ReLU activation function:  
\[FFN(x)=\max(0,xW_{1}+b_{1})W_{2}+b_{2}.\]  
Here, \(W_{1},W_{2}\), \(b_{1}\), and \(b_{2}\) are network parameters. The feed-forward network further enhances the model's ability to handle non-linear problems. * Output Linear Transformation LayerAfter being processed by the Transformer encoder layer, the output y is then passed through another linear transformation layer. The purpose of this layer is to map the output of the encoder layer to the final class space. The formula for this step is as follows:  
\[z=W_{2}y+b_{2},\]  
where \(W_{2}\) and \(b_{2}\) denote the weights and biases of this layer.