# Adaptive Modem Based on LSTM-AutoEncoder with Vector Quantization
## 4 Modulation and Demodulation Model Based on a Vector Quantization LSTM Autoencoder
### Vector Quantization  
In the application of classic autoencoder networks, the input data usually have high-dimensional features, leading to significant training costs in terms of computational resources and substantial storage space for a large number of model parameters. Vector quantization technology effectively mitigates these resource requirements for model training and storage, thereby reducing the overall cost. The DeepMind team [13] successfully integrated the idea of vector quantization with the autoencoder model, proving that discrete variables can also fully represent information. Given the discrete nature of modulated signals, using discrete variables as compressed feature vectors is naturally appropriate. Vector quantization technology can better capture the categorical properties in the data and assist the model in generating clearer outputs.  
As shown in Figure 5, a codebook structure was added based on Figure 2, the main purpose of which was to discretize the output of the encoder into the closest vector in the codebook and convert the continuous encoded output into a discrete representation. First, we needed to define a potential embedding space \(e\in R^{d\times D}\), that is, the codebook. The codebook contains \(d\) vectors as quantization centers, each with a dimension of \(D=2\). The codebook is not fixed. After the initial random initialization, it will gradually approach the distribution after compression of different signals during training. The output of the encoder \(\mathbf{O}_{SN}\) searches for the vector in the potential embedding space according to the nearest neighbor principle and performs one-hot coding so that only the most similar quantization center index \(d\) matched is 1, and the rest are 0. The discrete latent variable index \(z\) distribution is shown in Formula (12).  
\[\mathbf{Q}\;(z=d\mid\mathbf{O}_{SN})=\left\{\begin{array}{ll}1,&\text{for }d= argmin_{j}\big{\|}\mathbf{O}_{SN}-e_{j}\big{\|}_{2}\\ 0,&\text{otherwise}\end{array}\right. \tag{12}\]  
Figure 4: Working structural diagram of the encoder.  
After completing the index search for the encoder output \(\mathbf{O}_{SN}\), we passed the corresponding discrete signal into the AWGN channel, and the channel output \(\mathbf{y}_{R}\) is the input of the decoder. In this step, the use of codebook quantization interrupts the gradient propagation of the model, but because the expectation of the noise signal in the AWGN channel is 0, it is a random variable unrelated to the transmitted signal, so the additive noise has no effect on the gradient transfer. By directly assigning the gradient of the encoder output to the decoder input, the gradient discontinuity problem can be avoided.  
Although the gradient replication method connects the transmission and reception ends, it is still impossible to complete the learning of the codebook. To achieve this goal, it is necessary to modify the loss function of Formula (11) to indicate the iteration direction of the codebook. Specifically, in addition to the reconstruction loss, a part that measures the difference between the compressed potential representation and the predefined codebook and a part that measures the difference between the encoder's fixed output and the codebook quantization center were added. The former constrains the encoder output to be close to the codebook, and the latter constrains the codebook to learn the encoder output. The complete loss function is shown in Formula (13):  
\[L=L_{Re}+\parallel\text{sg}[\mathbf{O}_{SN}]-e\parallel_{2}^{2}+\beta\parallel \mathbf{O}_{SN}-\text{sg}[e]\parallel_{2}^{2} \tag{13}\]  
In Equation (13), \(L_{Re}\) represents the reconstruction loss, which is consistent with Formula (13); \(e\) represents the quantization center selected in the codebook; sg means stopping the gradient update and fixing variables; and \(\beta\) is a hyperparameter used to control the weight of this item in the overall loss function--its value is generally 0.25.