## 3 Materials and Methods
### Models Overview
#### 3.7.9 Transformer  
The Transformer model [29] is a groundbreaking architecture that relies solely on attention mechanisms to capture dependencies in sequential data, eliminating the need for recurrence. This self-attention mechanism allows the Transformer to focus on relevant parts of the input sequence, making it highly effective for handling long-term dependencies in HAR.  
The core of the Transformer is the multi-head attention mechanism, which computes attention weights for each input token across multiple heads, allowing the model to focus on various aspects of the input sequence. The attention mechanism is defined as follows:  
\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Sensor Configuration** & **Layers** & **Filter Sizes** & **Kernel Sizes** & **Optimizer/Learning Rate** \\ \hline
2 Sensors—Non-overlapping & 2 & [96; 128; 96] & [2; 3; 4] & Adamax/0.0043 \\
2 Sensors—50\% Overlapping & 2 & [96; 32; 128] & [3; 2; 2] & Adam/0.0006 \\
4 Sensors—Non-overlapping & 2 & [96; 64; 32] & [3; 3; 2] & Adam/0.0011 \\
4 Sensors—50\% Overlapping & 1 & [32; 128; 32] & [4; 3; 3] & RMSProp/0.0025 \\ \hline \hline \end{tabular}
\end{table}
Table 14: Optimal Hyperparameters for TCN in Different Sensor Configurations.  
Figure 8: Architecture of a Temporal Convolutional Network (TCN) for Activity Recognition, adapted from [28]. Dilated causal convolutions capture long-term dependencies, with dropout layers to prevent overfitting.  
\[Attention(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{softmax}\Big{(}\frac{\mathbf{Q} \mathbf{K}^{T}}{\sqrt{d_{k}}}\Big{)}\mathbf{V}, \tag{27}\]  
where the query \(\mathbf{Q}\), key \(\mathbf{K}\), and value \(\mathbf{V}\) are matrices derived from the input embeddings, with \(d_{k}\) representing the dimension of the queries and keys. This formulation allows the model to simultaneously attend to various parts of the sequence in a parallel manner. As illustrated in Figure 9, the multi-head attention mechanism allows the model to attend to various parts of the sequence in parallel, supported by positional encoding to handle sequential dependencies effectively.  
The hyperparameters for the Transformer model were optimized using nested cross-validation (outer = 5, inner = 3), as outlined in Table 15. Temporary values include four attention heads, two layers, and an Adam optimizer with a learning rate of 0.0008.  
The Transformer's self-attention mechanism, combined with its ability to process sequences in parallel, makes it highly efficient and powerful for HAR tasks, particularly in capturing complex temporal relationships within sensor data.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Sensor Configuration** & **Layers** & **Head Size/Numbers** & **MLP Layers/MLP Units** & **Optimizer/Learning Rate** \\ \hline
2 Sensors—Non-overlapping & 6 & 168/4 & 2/[224; 128; 224] & Adamax/0.0060 \\
2 Sensors—50% Overlapping & 4 & 40/8 & 1/[96; 224; 128] & AdaMax/0.0066 \\
4 Sensors—Non-overlapping & 4 & 8/16 & 2/[224; 256; 192] & Adamax/0.0055 \\
4 Sensors—50\% Overlapping & 2 & 8/6 & 2/[96; 96; 64] & Adam/0.0015 \\ \hline \hline \end{tabular}
\end{table}
Table 15: Optimal Hyperparameters for Transformer in Different Sensor Configurations.  
Figure 9: Architecture of the Transformer Model used in Activity Recognition, illustrating the multi-head attention and feed-forward layers, adapted from [29]. The positional encoding enables handling of sequential data without recurrence.