# Training Autoencoders Using Relative Entropy Constraints
## 4 Experiments
### Analysis of Factors Affecting Classification Performance  
#### 4.2.1 Average Activation Value of the Hidden Layer Outputs  
The purpose of training an autoencoder is to extract effective features for application services. This subsection takes the classification application as an example, and analyzes the influence of the average hidden layer activation value \(\rho\) on the classification performance of the REAN autoencoder network under different settings of the number of hidden nodes. The number of hidden nodes is set to 350, 550, and 750 on the MNIST dataset, and 450, 700, and 950 on the CIFAR-10 dataset, respectively. The average hidden layer activation value \(\rho\) is taken on the grid \(\{0.01,0.02,0.04,0.06,0.08,0.1,0.2,0.4,0.6,0.8\}\). In the REAN autoencoder network, the search range of the regularization parameter \(\lambda\) of the autoencoder and the regularization parameter \(\tau\) of the Softmax classifier is the grid \(\{10^{-5},10^{-4},\cdots,10^{2}\}\). The specific parameter values are determined by cross-validation.  
The experimental results are shown in Figure 2. The blue polyline, red polyline, and magenta polyline in Figure 2a correspond to the experimental results when the number of hidden nodes on the MNIST data set is 350, 550, and 750, respectively. The blue polyline, red polyline, and magenta polyline in Figure 2b correspond to the experimental results when the number of hidden nodes on the CIFAR-10 data set is 450, 700, and 950, respectively.  
Figure 2 shows that given the number of hidden nodes, the average hidden layer activation value \(\rho\) has a direct impact on the classification performance of the REAN autoencoder network. It can be seen from Figure 2a that, on the MNIST data set, when the average hidden layer activation value \(\rho\) is in the interval \([0.1,0.2]\), the classification performance of the REAN autoencoder network is the best. When the average hidden layer activation value \(\rho\) is greater than 0.2, the classification performance of the REAN autoencoder network generally declines. When the number of hidden nodes is 350, the classification performance of the REAN autoencoder network is the best at \(\rho=0.1\), and the classification accuracy rate is 96.7%. When the number of hidden layer nodes is 550 and 750, the classification performance of REAN autoencoder network is the best at \(\rho=0.2\), and the classification accuracies are 97.3% and 97.5%, respectively. Observing Figure 2b, it can be found that the classification performance of the REAN autoencoder network shows an obvious trend of improvement with the increase in the average hidden layer activation value \(\rho\). The best classification performance of the REAN autoencoder network is obtained when the average hidden layer activation value \(\rho\) takes a larger value (0.6 or 0.8). The interpretation of this paper is that the data complexity of CIFAR-10 is much greater than that of MNIST. When the number of hidden nodes is 700, the classification performance of the REAN autoencoder network is the best at \(\rho=0.6\), and the classification accuracy rate is 40.0%. When the number of hidden layer nodes is 450 and 950, the classification performance of the REAN autoencoder network is the best at \(\rho=0.8\), and the classification accuracies are 39.5% and 39.8%, respectively.  
It can be seen from the value of the optimal average hidden layer activation value \(\rho\) on the MNIST dataset and the CIFAR-10 dataset that the selection of \(\rho\) is closely related to the dataset where the REAE algorithm is applied. This is the desired result of this paper, i.e., to represent the characteristics of different datasets by means of a hidden layer average activation value constraint.