# Adaptive Modem Based on LSTM-AutoEncoder with Vector Quantization
## 4 Modulation and Demodulation Model Based on a Vector Quantization LSTM Autoencoder
### LSTM-AE-Based Modulation and Demodulation Framework
\[f_{t}=\sigma\Big{(}W_{f}x_{t}+R_{f}y_{t-1}+b_{f}\Big{)} \tag{7}\]  
Formula (8) is the calculation method of the cell value. The forget gate selects the cell value of the previous round, and the input gate processes the input data, and the cell value of this round is calculated in combination; \(\odot\) is the vector dot product symbol. Figure 2.0  
Figure 2.1 Architecture of a typical LSTM block. Formula (9) gives the process of updating the output gate. The output gate considers the current round input, the previous round output, and the cell value together and performs nonlinear transformation through the activation function \(\sigma(x)\). Here \(W_{o}\) and \(R_{o}\) represent the weight matrices about \(x_{t}\) and \(y_{t-1}\), respectively, and \(b_{o}\) is the bias. Figure 2.2  
Finally, we can obtain the output \(y_{t}\) of the LSTM unit in this round from Formula (10), which is calculated by combining the cell value in this round with the output gate result. Figure 2.3  
Generally speaking, we use the logistic sigmoid function for the activation function \(\sigma(x)=\frac{1}{1+e^{1-x}}\) and the \(tanh(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}\) function for \(g(x)\). LSTM-AE uses a large number of LSTM blocks as hidden layer units, which play the role of compressing information to the latent space in the encoder and decompressing and restoring the signal in the decoder. In Figure 4, the encoder and decoder can be considered as utilizing the connection and stacking of multiple LSTM layers, with the detailed structure shown in Figure 2.4 The horizontal transmission is time information; the LSTM layer will be reused many times and finally leave the filtered information. The spatial information is transmitted vertically. The hidden state Y output by the previous layer is used as the input information of the next layer. Increasing the depth can provide stronger fitting ability. After the one-hot code matrix \(\mathbf{O}_{e}\) is input to the encoder, it first passes through \(k-1\) LSTM layers. Each layer consists of multiple LSTM units. The compression ratio of each layer can be controlled by setting the number of LSTM layer output information units. Due to the LSTM cells' ability to store long-term information, the network can decouple the input data dimension N from the output data dimension of each layer by adjusting the hidden layer size, thus avoiding the need for exponential decay patterns. The last layer of the encoder is the output layer. When the hidden layer dimension is set to two, the purpose of outputting a fixed range of feature two-dimensional real vectors can be achieved. The two dimensions represent the I/Q signals. The output of LSTM is shown in Formula (10), which is obtained by multiplying the activation value of the cell state \(c^{(t)}\) and the result of the output gate \(o^{(t)}\). The activation function generally uses the tanh function, so it can be ensured that the requirement is met within the interval (\(-1\), \(1\)). Before the final output, a normalization operation is added, which helps to speed up the convergence of the network and prevents the gradient from disappearing. As mentioned above, the decoder part is the reverse operation of the encoder part that decompresses the received signal containing channel noise into a tensor of the same shape as the input data and then uses the argmax function to restore it to the output data in the one-hot encoding format. The received signal can be represented as \(\mathbf{I}_{\text{RN}}=\mathbf{O}_{\text{SN}}+\mathbf{\mathrm{n}}\), where each element of \(\mathbf{\mathrm{n}}\) follows a complex Gaussian distribution with zero mean and variance \(\sqrt{\frac{P_{s}}{SNR}}\). \(P_{s}\) is the signal power, and SNR is the signal-to-noise ratio specified during training or testing. According to the conclusion of [6], during training we set the loss function to the mean square error function.