# Adaptive Modem Based on LSTM-AutoEncoder with Vector Quantization
## 4 Modulation and Demodulation Model Based on a Vector Quantization LSTM Autoencoder
### LSTM-AE-Based Modulation and Demodulation Framework  
LSTM-AE plays an important role in the task of time series anomaly detection [12]. The core of this task is the reconstruction of sequence data, which aligns with our objectives to some extent. Therefore, LSTM-AE was employed in the system model depicted in Figure 1 to perform the information compression and recovery processes in the communication system. This results in a modulation and demodulation framework based on LSTM-AE, as illustrated in Figure 2.  
The general framework of the improved LSTM-AE is shown in Figure 2. This section will describe the composition of the architecture in detail. The input of the network is the one-hot code matrix \(\mathbf{O}_{e}\) of the symbol string \(S\). The encoder consists of an LSTM hidden layer, a normalization layer, and an output layer, according to the processing flow, and the output is a two-dimensional real vector sequence \(\mathbf{O}_{SN}\). The first layer accepts data input, which can be simply expressed as Formula (2), where \(X\) represents the input data, \(y_{0}\) represents the initialized hidden information, \(c_{0}\) represents the initialized cell information, and \(\Theta_{1}\) is the parameter of this layer. \[y_{1}=\text{LSTM}(X,y_{0},c_{0};\Theta_{1}) \tag{2}\]  
After the input layer, the data are processed by BatchNorm and compressed by \(K-1\) layers of LSTM. The output of the \(j\) layer can be expressed as Formula (3), where \(Y^{j-1}\) represents the output hidden state information from the previous layer, \(\mathbf{C}^{j-1}\) comes from the output cell information of the previous layer, \(\Theta_{j}\) is the parameter of this layer, and \(j\in[1,k-1]\). \[y_{j}=\text{LSTM}\Big{(}Y^{j-1},C^{j-1};\Theta_{j}\Big{)} \tag{3}\]  
The output layer of the encoder is the \(K\)th LSTM layer. While compressing the information, the potential representation is fixed to the form of a two-dimensional real  
Figure 2: Architecture of LSTM-AE-based modulation. vector sequence. Similar to the previous two formulas, the output can be expressed as follows:  
\[y_{K}=\text{LSTM}\Big{(}Y^{K-1},C^{K-1};\Theta_{K}\Big{)} \tag{4}\]  
The input of the decoder is the output of the channel, denoted by \(\mathbf{I}_{RN}\). The decoder is designed symmetrically with the encoder. It consists of an input layer, a normalization layer, and a \(K-1\) LSTM hidden layer according to the processing flow. The output is a matrix \(\mathbf{O}_{RN}\). We used the LSTM block for a single signal rather than a time series, so we set the timestamp to one. The structure of the LSTM is illustrated in Figure 3. The introduction of LSTM has addressed the issues of gradient vanishing and gradient exploding that were prevalent in earlier recurrent neural networks (RNNs) to a considerable extent. LSTM incorporates long-term memory while blending the output with short-term memory, thereby enhancing the network's ability to capture temporal dependencies. An LSTM unit contains a forget gate, an input gate, and an output gate, which can be expressed by Equations (5)-(10) [15]. Formula (5) gives the processing method of input data \(x_{t}\) and the previous round output \(y_{t-1}\), where \(W_{z}\) and \(R_{z}\) represent the weight matrices about \(x_{t}\) and \(y_{t-1}\), respectively, and \(b_{z}\) is the bias. \(g()\) is the activation function. \[z_{t}=g(W_{z}x_{t}+R_{z}y_{t-1}+b_{z}) \tag{5}\]  
Formula (6) gives the process of updating the input gate, \(W_{i}\) and \(R_{i}\) represent the weight matrices about \(x_{t}\) and \(y_{t-1}\), respectively, \(b_{i}\) is the bias, and \(\sigma(x)\) is the activation function. \[i_{t}=\sigma(W_{i}x_{t}+R_{i}y_{t-1}+b_{i}) \tag{6}\]  
Formula (7) gives the process of updating the forget gate, which is the key to information transmission and determines the degree of information retention, where \(W_{f}\) and \(R_{f}\) represent the weight matrices of \(x_{t}\) and \(y_{t-1}\), respectively, and \(b_{f}\) is the bias.