# Enhancing Automatic Modulation Recognition for IoT Applications Using Transformers
## 3 Transformer Architectures  
The Transformer model, presented in [5], marked a significant advancement in the field of sequence-to-sequence processing. Comprising an encoder and a decoder, each with multiple layers, this model efficiently maps an input sequence of symbols (words) to a sequence of continuous representations. The decoder then transforms these representations to produce an output sequence. This architecture has demonstrated remarkable accuracy across a variety of sequence-to-sequence tasks, including machine translation and text summarization. The Transformer architecture, initially designed for language tasks, has been effectively adapted for image processing in the ViT paper [13]. Unlike the original Transformer model, which includes both an encoder and decoder for sequence-to-sequence tasks, ViT adapts the Transformer's encoder to process images. ViT treats an image as a sequence of patches and applies the Transformer encoder to these patches to perform image classification and process images as sequences of flattened patches. The encoder consists of several identical layers, each comprising two sub-layers: a multi-head self-attention mechanism and a feed-forward neural network. Additionally, it has residual connections and layer normalization to mitigate the vanishing gradient issue that arises with deep models. The feed-forward neural network consists of a fully connected layer architecture. In the self-attention mechanism, each element of the input sequence interacts with all other elements to calculate the attention weights, highlighting the importance of the relationships among \begin{table}
\begin{tabular}{c c c} \hline \hline  & **RadioML2016.10b [16]** & **CSPB.ML.2018+** \\ \hline Number of Modulation Types & 10 (8 digital and 2 analog modulations) & 8 digital modulations \\ \hline \multirow{4}{*}{Modulation Pool} & BPSK, QPSK, 8 PSK, QAM16, QAM64, & BPSK, QPSK, 8 PSK, DQPSK, MSK, \\  & BPSK, CPPSK, PAM4, WBFM, AM-DSB & 16-QAM, 64-QAM, 256-QAM \\ \hline Signal Length & 128 & 1024 \\ \hline SNR Range & \(-20\) dB to +18 dB & \(-19\) dB to +40 dB \\ \hline Number of Samples & 1,200,000 & 3,584,000 \\ \hline Sample Distribution across SNR Range & log-uniform distribution & log-normal distribution \\ \hline \multirow{4}{*}{Channel Effects} & \(\bullet\) & thermal noise & \(\bullet\) \\  & \(\bullet\) & simple multi-path fading & \(\bullet\) \\ \cline{1-1}  & \(\bullet\) & center-frequency and sample rate & slow and fast multi-path fading \\ \cline{1-1}  & & offset & \(\bullet\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Dataset characteristics: RadioML2016.10b [16] vs. CSPB.ML.2018+different positions within the sequence. The multi-head attention mechanism splits the attention calculation across several heads, with each head independently performing the attention computation. This allows each head to concentrate on distinct features of the input sequence, providing a better understanding of the sequence. This shows how the Transformer can be used not just for text, but also for images, demonstrating its versatility. In this section, we introduce different methods for recognizing signal modulation types using a Transformer-based architecture. Figure 1 illustrates the overall architecture of our Transformer-based approaches, comprising three key components: The tokenization module, the Transformer-encoder module, and the classifier module. The raw _IQ_ data, composed of in-phase (\(I\)) and quadrature (\(Q\)) components, forms a two-channel input. However, Transformer networks require their inputs to be in the form of tokens, which is achieved through the tokenization module. Various strategies for tokenization are employed in each Transformer model discussed in subsequent sections. In all these architectures, the input undergoes tokenization to generate tokens before being fed into the Transformer-encoder module for capturing relevant features from the data, and the classification task concludes with the output from the classifier module, which is a fully connected neural network.