# Training Autoencoders Using Relative Entropy Constraints
## 1 Introduction  
The autoencoder is a hot research topic in the field of computer vision [1; 2; 3; 4; 5; 6; 7; 8; 9; 10]. It can be used for feature extraction and data dimensionality reduction. In the deep learning framework, autoencoders can be used as the basic modules to form deep and complex networks [11; 12; 13; 14; 15; 16]. A deep autoencoder composed of stacking multiple autoencoders can extract more complex and abstract features than a shallow autoencoder. Inspired by the hierarchical information processing mechanism of the human visual system in cognitive neuroscience, [PERSON] and [PERSON] propose an autoencoder-based deep neural network model [17]. Through layer-by-layer greedy pre-training and fine-tuning, this autoencoder-based deep neural network model achieves significantly better classification correctness than traditional methods on image classification datasets. During the pre-training process, the autoencoder is used to initialize the weights of the deep neural network. This highlights the importance of training autoencoders.  
The classic autoencoder training algorithm is the backpropagation algorithm [18]. However, this algorithm has the problem of slow convergence. In addition, the backpropagation algorithm requires the user to specify parameters such as step size, momentum factor and maximum number of iterations when optimizing the parameters of the neural network model. The setting of these parameters has an important impact on the training effect of the neural network, but there is no clear theoretical guidance on how to set these parameters. For a specific problem, the user needs to repeatedly debug based on experience to achieve a good result. When the dataset size is large, debugging the parameters based on iterative trial and error can be an extremely time-consuming process.  
To solve the problems in the backpropagation algorithm, researchers propose to train the autoencoder in a forward way [19; 20]. These methods first map the input data into the hidden layer space, then solve for the model parameters of the decoder, and finally derive the model parameters of the encoder using weight bundling. Different datasets havedifferent characteristics. In the case of image datasets, for example, the type and complexity of images contained in different datasets also vary. However, existing forward propagation algorithms do not fully consider the characteristics of the data itself.  
This paper proposes an autoencoder training algorithm based on relative entropy constraints, called relative entropy autoencoder (REAE). It uses relative entropy to constrain the average activation value of the feature mapping on the hidden nodes. When REAE solves for the feature map parameters, different constraints are imposed on the average activation value of the hidden layer outputs obtained by the feature map for different data sets. To verify the validity of REAE, it is applied to the image classification task. The autoencoder cannot be used directly for classification, so it is common practice to combine it with a Softmax classifier to form a new classification model. The experimental results on image classification datasets show that the classification performance of the classification model constructed by REAE is better than that of the classification model constructed by other forward propagation algorithms.