# Enhancing Automatic Modulation Recognition for IoT Applications Using Transformers
## 3 Transformer Architectures
### TransDirect  
The detailed architecture considered for TransDirect is shown in Figure 2. The process starts with the tokenization of data, a crucial step in implementing the Transformer network. This approach is similar to ViT, where image patches are generated from segments of the input as tokens.  
In the context of TransDirect, tokenization involves dividing the _IQ_ samples, which have a sequence length of \(n\), and consists of two channels into multiple shorter subsequences that are called tokens. The \(I\) and \(Q\) samples from the signal sequence can be expressed as \(I=[i_{1},i_{2},\ldots,i_{n}]\) and \(Q=[q_{1},q_{2},\ldots,q_{n}]\), with \(n\) denoting the length of the signal. In this arrangement, both the \(I\) and \(Q\) sequences are divided into shorter segments of length \(l\), using a sliding window technique with a step size of \(w\). As the tokens (segments) are designed to be non-overlapping, the stride length \(w\) is set equal to the segment length  
Figure 1: The overall architecture of our proposed Transformer-based model for the AMR task includes three main components: a tokenization module that converts the signal into tokens, a Transformer-encoder module, which captures information and extracts relevant features through self-attention mechanism, and a classifier module for the final classification step.  
(token size) \(l\), ensuring distinct and consecutive tokens. Consequently, the segmented subsequences for the \(l\)s are given by \(I_{1}=[i_{1},i_{2},\ldots,i_{l}]\), with subsequent sequences like \(I_{2}=[i_{1+w},i_{2+w},\ldots,i_{l+w}]\), leading up to \(I_{N}=[i_{1+(N-1)w},i_{2+(N-1)w},\ldots,i_{l+(N-1)w}]\). A similar segmentation applies to the \(Q\) sequence. As a result, the overall count of tokens using this method is determined by \(N=\frac{n}{l}\).  
Until this point, the model converts the two-dimensional vector at each time step into \(N\) tokens with two channels. These tokens are then processed through the Transformer-encoder module, according to Figure 1, where they first get transformed into linear sequences by the Linear Projection Layer, acting as the starting point of the Transformer-encoder's processing. Moreover, an additional trainable \"class token\" is also included with the tokens, increasing the total count of tokens to \(N+1\). Positional encoding is then applied to the tokens before they are input into the Transformer-encoder, enabling the model to understand the relative positioning of the tokens. The tokens then pass through \(N_{l}\) Transformer encoder layers, which utilize the self-attention, with a hidden size (embedding size) of \(m\) and attention heads of \(N_{\text{head}}\). As the core part of the Transformer, the attention mechanism maps a query and a set of key-value pairs to the output, establishing connections between various positions within a sequence to integrate information across the entire input data. After passing through \(N_{l}\) Transformer encoder layers, the output is an \((N+1)\times d\) matrix, where \(d\) is the embedded dimension, calculated as twice the number of samples per token \(l\), reflecting the dual-channel structure of the input data. The first token from this output is then extracted and fed into the classifier module. This module consists of fully connected layers with a set number of hidden layers of \(N_{\text{hidden}}\). The final result of the classification comes from the output of the classifier module, generating a c-dimensional vector, where each represents the probability of a particular modulation type.