# HCTC: Hybrid Convolutional Transformer Classifier for Automatic Modulation Recognition
## 2 Received Signal and Proposed HCTC Model
### Proposed HCTC Model
#### 2.2.2 Stage B--Transformer Layer  
The output of the convolutional layer after reshaping was fed to stage B, the transformer layer. In the proposed model, a transformer encoder layer, as shown in Figure 2, was added, which was motivated by the work presented in [60]. Stage B of the proposed method extracted the temporal features from the input by applying positional encoding.  
The following are the details of each block.  
**Input Sequence:** This block receives processed signals from the convolutional layer, formatted as a tensor. It is a feature-rich representation of the original I/Q signals, ready for further processing. The input signal fed to the encoder step is often called input token embedding, which assigns a number of tokens to the input sequence.  
**Positional Encoding**: This layer adds positional information to the input sequences. Because transformers do not inherently understand the sequence order, positional encoding helps the model to capture the temporal relationships between the elements. To adjust the positional encoding, multiplication of the range tensor was performed, followed by computation of the sine and cosine of the angle in radians. The formulae for the position, often referred to as _pos_ in positional encoding (PE), are given in Equations (3) and (4).  
\[PE(pos,2i)=\sin(\frac{pos}{10000^{2i/d}}) \tag{3}\]  
\[PE(pos,2i+1)=\cos(\frac{pos}{10000^{2i/d}}) \tag{4}\]  
where \(d\) is the depth of the embedding.  
**Multihead Attention**: This mechanism allows for the model to focus on different parts of the input simultaneously. It computes attention scores for the input sequences, enabling the model to weigh the importance of various signals in the context. In the proposed method, two heads were used, which utilized the concept of multiple attention heads to capture the aspect of the relationship between tokens from different parts of the sequence simultaneously. The results for all the heads were concatenated and linearly transformed.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Layer (Name)** & **Output Shape** & **Description** \\ \hline gaussian\_dropout\_1 (GaussianDropout) & (2, 128, 64) & Dropout rate: 0.1 \\  & & Number of filters: 32 \\  & & Kernel size: (2, 5) \\ conv4 (Conv2D) & (1, 124, 32) & Activation function: ReLU \\  & & Kernel initializer: Glorot uniform \\  & & Kernel regularizer: L2 (1 \(\times\) 10\({}^{-4}\)) \\ max\_pooling2d\_2 ( MaxPooling2D) & (1, 124, 32) & Pool size: (2, 2); Stride: 1; Padding: Same \\ gaussian\_dropout\_2 ( GaussianDropout) & (1, 124, 32) & Dropout rate: 0.1 \\ bn1 (BatchNorm Normalization) & (1, 124, 32) & Batch normalization \\ \hline \hline \end{tabular}
\end{table}
Table 1: _Cont._  
Figure 2: Block diagram of the transformer layer (stage B) of the proposed HCTC model.  
**Add and Norm (Post-Attention)**: This step implements a residual connection [61], where the output of the multihead attention is added back to its original input.  
\[Output_{Attention}=Attention(x)+(x) \tag{5}\]  
This preserves the initial signal information and mitigates potential issues related to vanishing gradients in deeper networks. Following the addition, layer normalization [62] is applied to stabilize the output, and it is given by \(LayerNorm(x+Sublayer(x))\).  
**Feed-Forward Network (FFN)**: This component consists of two dense layers with a ReLU activation function. It enhances the model's capacity to learn complex transformations of token embedding.  
**Add and Norm (Post-FFN)**: Similar to the previous block, this step maintains a residual connection; that is, it adds the input from the FFN back to its output, maintaining a residual connection. Following the addition, layer normalization is again applied to stabilize the output.  
The details of the transformer encoder configuration are listed in Table 2.