# Training Autoencoders Using Relative Entropy Constraints
## 2 Background  
The autoencoder is a special kind of neural network that consists of two parts: an encoder and a decoder. The encoder converts the input data into a different representation, and the decoder converts this new representation back to the original format. The model parameters of the autoencoder can be trained by either a backpropagation algorithm or a forward propagation algorithm. The autoencoder forward propagation algorithm is summarized below. The training process of the autoencoder forward propagation algorithm is shown in Figure 1.  
From the perspective of the autoencoder implementation, the feature mapping parameters mentioned in the first step of Algorithm 1 are the parameters of the weights from the input layer to the hidden layer of the autoencoder. The feature mapping parameters are only used to map the input data to the hidden layer space of the autoencoder and do not encode the input data. To avoid confusion with the model parameters of the autoencoder, they are named feature mapping parameters in this paper. In the third step, the decoder model parameters do not include bias terms, so the encoder model parameters obtained through weight bundling also do not include the bias term. Figure 1: The training process of the autoencoder forward propagation algorithm. ```
1:Solve for feature mapping parameters \(\textbf{W}_{p}\) and \(\textbf{b}_{p}\). 2:Forward propagate the input data, and use the model parameters \(\textbf{W}_{p}\) and \(\textbf{b}_{p}\) to map the input data to the hidden layer space. 3:Solve for the decoder model parameters \(\textbf{W}_{d}\). 4:Use weight bundling to calculate encoder model parameters \(\textbf{W}_{e}\). ```  
**Algorithm 1** Autoencoder forward propagation algorithm. When designing an autoencoder, it is sometimes necessary to control the parameter scale of the autoencoder according to application requirements. The commonly used method is to use weight bundling in the autoencoder, that is, make the following constraints on the autoencoder's encoder weight parameters \(\textbf{W}_{e}\) and decoder weight parameters \(\textbf{W}_{d}\)[19; 20]:  
\[\textbf{W}_{e}^{T}=\textbf{W}_{d}. \tag{1}\]  
The difference between different autoencoder forward propagation training algorithms is how to solve for the feature mapping parameters. [PERSON] et al. [19] solve for the feature map parameters by calculating the low-rank approximation of the pseudo-inverse of the input matrix. The advantage of this method is its excellent stability. Since there is no random initialization, the model parameters obtained for a given data set are also the same each time. [PERSON] et al. [20] solve for feature mapping parameters by randomly initializing weights. Random initialization does not require additional calculations, so the advantage of this method is that the parameters can be solved quickly. The existing autoencoder forward training algorithms do not fully consider the characteristics of the data itself. The algorithm proposed by [PERSON] et al. [19] uses the low-rank approximation of the pseudo-inverse of the input data as the feature mapping parameter, but its main purpose is to obtain stable feature map parameters. The algorithms proposed by [PERSON] et al. [20] are data-independent. To make full use of the characteristics of the data itself, this paper proposes to train autoencoders using relative entropy as a constraint. When REAE solves for the feature mapping parameters of the autoencoder, it restricts the average activation value of the hidden layer outputs according to the characteristics of the dataset. It should be noted that REAE is different from the classic sparse autoencoder. There are two main differences between them: (1) The constrained objects are different. The sparse autoencoder imposes constraints on the hidden layer outputs generated by the encoder. REAE imposes constraints on the hidden layer outputs generated by the feature mapping; (2) different constraint requirements.