# Enhancing Automatic Modulation Recognition for IoT Applications Using Transformers
## 3 Transformer Architectures
### TransIQ  
In the TransIQ design, as illustrated in the Figure 4, we enhanced the model's feature-extraction capacity by expanding its receptive field in the tokenization phase. Following the tokenization approach of earlier techniques, we first slice the \(IQ\) sequence into shorter segments using a sliding window technique in the tokenization module. These segments are then reorganized into a matrix of signal sequences, and one-dimensional convolutional encoding is performed.  
The introduction of a CNN layer between the tokenization module and the Transformer-encoder plays a crucial role in enhancing the feature representation of the input data and facilitates the extraction of detailed features from each token. The CNN layer, through its convolutional filters (kernels), is inherently designed to extract spatial features from the data. Each filter specializes in identifying specific patterns within the input. By applying these filters across the signal sequences, the CNN layer can highlight important features from the input data. This process effectively prepares a rich, detailed feature map for each token. On the other hand, the Transformer-encoder utilizes a self-attention mechanism to weigh the importance of different features within the input. To elaborate more, after the CNN layer enriches the feature representation, the Transformer encoder assesses these features, determining their relevance to the task of modulation classification. The self-attention mechanism allows the model to focus on the most relevant features by assigning higher weights to them. This capability stems from the Transformer's design to dynamically allocate attention based on the input's context, enabling it to detect which features (enhanced by the CNN layer) are crucial for understanding the input sequence's characteristics. So, the CNN layer ensures that the Transformer encoder receives a high-resolution feature map, where important characteristics are already highlighted. Subsequently, the Transformer can efficiently evaluate these features' relevance, adjusting its internal weights to prioritize tokens crucial for accurate modulation classification. So, leveraging CNNs for improved  
Figure 3: Tokenization module of TransDirect-Overlapping architecture, which divides \(IQ\) samples into tokens, each with a length \(l\), where each token overlaps the preceding one by \(l/2\).  
feature representation and Transformers to dynamically adjust the importance of those features, TransIQ efficiently allocates suitable weights to each feature segment, focusing on key positions in the input temporal data.  
So, the tokens generated from the sliding window are fed into a convolutional layer with a kernel size of \(k\) with the same padding and \(N_{c}\) number of output channels. The output of the convolution layer is fed to a ReLU activation function. Up to this stage, the model transforms the two-dimensional vector at each time sample into an \(N_{c}\) (output channel) -dimensional feature representation using different kernels in the convolutional layer. This matrix is then processed through \(N_{l}\) Transformer encoder layers, which operates on an \((N+1)\times d\) matrix using the self-attention mechanism. Here, \(N+1\) represents the total number of tokens, and \(d\) represents the embedding dimension of the Transformer. In this scenario, \(d\) equals the product of the token size and the number of output channels of the CNN layer. The first token from the output of the Transformer encoder is then fed into the classifier module to perform the modulation classification.