# On the Development of a Neural Network Architecture for Magnetometer-Based UXO Classification
## 4 Experimental Setup
### Neural Network Design  
The second phase of experimentation focused on neural network architectures. Starting from the best-performing classical neural network model, we developed several neural networks, with an emphasis on Convolutional Neural Networks (CNNs) for effective temporal pattern extraction. In our evaluation we ignored transformer-based models, since they require very large datasets and, as presented in [20], often other models outperforms transformer architectures, which, by employing positional encoding and using tokens to embed parts of the inputs, may cause information loss.  
The input data for the models was formatted as tensors of shape \(1\times 101\times 3\), representing 101 temporal samples from three parallel sensors. As only the magnetometer magnitude was available, a single measurement of a sensor was represented as a single value (magnitude without three spatial components of the magnetic field vector). This data was fed into a CNN comprising multiple convolutional blocks.  
Inspired by ResNet [21], our architecture adopted a modular design wherein each building block consists of two convolutional layers, element-wise batch normalization layers, and a skip connection that facilitates gradient flow and mitigates vanishing gradients. Unlike traditional batch normalization [22], element-wise normalization was applied independently to each element of an input tensor. The element-wise normalization was implemented as follows:  
\[y_{ij}=\frac{x_{i,j}-E[x_{i,j}]}{\sqrt{Var[x_{i,j}]+\epsilon}}\cdot\gamma+\beta \tag{1}\]  
Here, the input data \(X\) is assumed to be of size \([B\times H\times W]\), where \(B\) denotes the batch size and \(H\times W\) represents the dimensions of a single input tensor (in our specific case, \(H=3\) and \(W=101\)). The normalization is applied independently for each \((i,j)\) coordinate across the batch size dimension. \(\gamma\) and \(\beta\) are learnable parameters that allow for better adjustment to the input data's profile, and \(\epsilon\) is a small constant to avoid dividing by zero.  
An overview of the network architecture is provided in Figure 2. It presents (a) the design of a single building block and (b) the full network architecture. The network begins with an initial convolutional layer, followed by one or more building blocks, and concludes with an adaptive max-pooling layer of size \((1,1)\) to reduce overfitting by condensing each filter to a single maximum activation. A fully connected layer follows the convolutional part, integrating meta-features such as seabed distance and sensor orientation.  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Model** & **Params** \\ \hline kNN & \(k=\{3,5,7,9,13\}\) \\ \hline RF & \(\#trees=\{50,100,300,500\}\) \\ \hline MLP & \(\dfrac{struct}{truct}=\{(10,10),(100),(200),(10,10),(200,100),(500,100)\}\) \\ \cline{2-3}  & \(lr=0.01\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Hyperparameter of the classical machine learning models used in the experiments.  
Meta-features are processed via an additional fully connected layer with four neurons before being concatenated with the output of the CNN.  
The network was implemented in PyTorch (version 2.5.1+cu121). Hyperparameters, summarized in Table 3, were selected based on convergence behavior. Early stopping was applied after six iterations without training loss improvement. Learning rate was reduced by a factor of 0.1 if no improvement was observed for three consecutive iterations. A batch size of 512 was used to stabilize the batch normalization estimates, as smaller batches yielded unstable mean and variance estimates due to high signal variance. The influence of the network hyperparameters on prediction performance are shown in Figure 3.  
All networks in this stage were evaluated using five-fold cross-validation.