# HCTC: Hybrid Convolutional Transformer Classifier for Automatic Modulation Recognition
## 5 Experiments on Variations in the Model Configuration and Batch Size  
The configuration of the transformer encoder was varied by altering the number of attention heads and the model depth. The proposed HCTC model employed two attention heads, a deeper model with 64 layers, a low feed-forward dimension, and a hidden layer size of 256. HCTC-A included two attention heads, 128 layers of depth, and a feed-forward dimension of 256. These models were labeled as HCTC (Head 2, Depth 32), HCTC (Head 4, Depth 32), and HCTC (Head 4, Depth 64). They were used to compare the performances of the proposed HCTC model. The increased depth allowed for more complex feature extraction and representation, although the reduced number of dimensions may have limited the capacity of the model to encode detailed information. In the second experiment, the batch size was varied among 32, 64, and 128, which allowed for the investigation of the influences of the batch size on the training and performance of the proposed HCTC model. It is seen in the literature that if the dataset is small, the small batch size will improve the model training, and if the dataset is large, the large batch size will improve the speed of the training. The batch size is typically considered as a power of two, e.g., 2, 4, 8, 16, 32, 64, and so on. As we are dealing with the RML2016A dataset, which has 220,000 samples (large), we are experimenting with batch sizes of 32, 64, 128, and 400. According to the literature, MCLDNN [31] uses a batch size of 400 and shows a competitive performance; we did an ablation experiment and decided to use a batch size of 400 for the proposed method. By systematically varying the batch size and other architectural parameters, the aim was to optimize the training process and performance of the transformer encoder across different configurations. Figure 12 shows the accuracy curves of the experiment, where the proposed HCTC model had effective modeling parameters, such as two attention heads, a depth of 64, and a feed-forward dimension of 256, with a batch size of 400. In order to consider two attention heads and a depth of 64, we experimented with the number of heads and the depth of the model. The choices of the two attention heads and the depth of the model as 64 show the balance between the number of parameters and the accuracy. Two heads allow for the model to capture different types of relationships within the data while keeping the architecture manageable. As we increase the depth, there is an increase in the computational cost, as the number of parameters is increased. The results of this experiment are shown in Table 7. A feed-forward dimension of 256 was chosen to provide sufficient capacity for non-linear transformations between the attention layers. Figure 11: Numbers of FLOPs of various models with the RML2016a dataset. In the proposed HCTC model, we combine a convolutional network architecture with a transformer encoder architecture. Now, we experiment with each individual architecture as (A) a purely convolutional architecture and (B) a pure transformer encoder architecture. The experiments were performed on the RML2016.10a dataset. We considered the recognition accuracy for each of the model architectures and compared it with that for the proposed HCTC model. The recognition accuracy per SNR is presented in Table 8.  
This table clearly demonstrates that the proposed HCTC model outperforms the other models. It achieves the highest accuracy among all the sub-models. Notably, the transformer-only model displays a lower accuracy compared to that of the convolution-only model, indicating that the convolutional layer effectively extracts the most significant features from the input dataset. This highlights the importance for combining the convolutional model with the transformer encoder model in enhancing the recognition accuracy. Table 9 compares the three models--purely convolutional, pure transformer encoder, and the proposed HCTC--across the total number of parameters and average accuracies.