# Transmission Line Fault Classification Using Conformer Convolution-Augmented Transformer Model
## 2 Methodology
### Machine Learning-Based Model: Conformer: Convolution-Augmented Transformer Model  
This study uses an ML-based method, employing the Conformer model [21]. The Conformer model is a state-of-the-art architecture commonly used in speech recognition tasks. It combines the Transformer model with the architecture of a CNN. The model architecture is illustrated in Figure 8.  
A Conformer encoder model consists of four modules stacked together in sequence. These modules are as follows:  
* Feed-forward module: This module processes the input data by applying a set of linear transformations and activation functions to capture relevant local features. * Self-attention module [24]: The self-attention mechanism allows the Conformer to capture global dependencies and relationships within the input data. * Convolution module: While the convolution module effectively captures local patterns, it requires many parameters and significant depth to capture global dependencies [25]. Figure 8: Conformer encoder model architecture [21]. Figure 7: Resampled voltage waveform. (**a**) Resample to 16 samples per cycle; (**b**) resample to 4 samples per cycle. The parameters refer to the weights and bias in a neural network, while the depth refers to the number of layers in a neural network. To address this, batch normalization is applied immediately after the convolutional layers in deep neural networks. Batch normalization, a technique used in neural networks, normalizes each mini-batch of data during training. This normalization process transforms the data into a mean of 0 and a standard deviation of 1, following a standard normal distribution. This helps to alleviate the vanishing gradient problem and mitigate the effects of Internal Covariate Shift, which is the change in the distribution of the network's internal activations during training. Additionally, batch normalization leverages the parallelism offered by modern computing platforms to significantly improve training efficiency [26]. To ensure the effective application of batch normalization, maintaining a uniform data length for each batch is essential. Resampling the data can help achieve this uniformity, ensuring that the input data have a consistent length, which is crucial for batch normalization to work optimally. * Second feed-forward module: Similar to the first feed-forward module, this module further processes the output from the previous modules, refining and extracting additional features from the data. By combining these four modules, the Conformer block effectively integrates local and global information, allowing it to model dependencies within the audio sequence in a parameter-efficient manner. The Conformer has the ability to outperform LSTM and other RNNs, which are common models used for handling time-relevant input data, in terms of classification accuracy with the same set of parameters [20]. This means that Conformer achieves higher accuracy in a similar training time compared to LSTM and other RNNs [20]. Furthermore, the Conformer model exhibits outstanding performance within a short training period by carefully designing the batch size in batch normalization. Model hyper-parameters for Conformer are detailed in Table 2. The optimal Conformer model for fault classification purposes is determined by sweeping various combinations and selecting the best performing models within the parameter constraints. Our model, designed with 68,791 parameters and featuring three encoder layers with an encoder dimension of 128, requires significantly fewer parameters compared to the Conformer model for ASR purposes outlined in [21]. We observed that increasing the number of attention heads to three enhances accuracy, as each head focuses on different aspects of the input, thereby improving predictions beyond a simple weighted average. Furthermore, enlarging the convolution kernel size to five leads to better accuracy.