# Training Autoencoders Using Relative Entropy Constraints
## 5 Discussion and Conclusions  
This paper presents a new autoencoder forward training algorithm, which is named REAE. Existing autoencoder forward propagation algorithms do not take into account the properties of the data itself when solving for the feature mapping parameters. According to the data set, REAE uses relative entropy to constrain the average activation value of hidden \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Name**} & \multicolumn{3}{c}{**MNIST**} & \multicolumn{3}{c}{**CIFAR-10**} & \multicolumn{3}{c}{**SVHN**} \\ \cline{2-9}  & \multirow{2}{*}{**Accuracy**} & \multicolumn{2}{c}{**Parameter**} & \multicolumn{2}{c}{**Parameter**} & \multirow{2}{*}{**Accuracy**} & \multicolumn{2}{c}{**Parameter**} \\ \cline{3-4} \cline{6-9}  & & \(\lambda\) & \(\tau\) & & \(\lambda\) & \(\tau\) & & \(\lambda\) & \(\tau\) \\ \hline REAN & **97.3** & \(10^{-2}\) & \(10^{-1}\) & **40.0** & \(10^{-2}\) & \(10^{-2}\) & **62.4** & \(10^{-3}\) & \(10^{-3}\) \\ PIAN & 97.0 & \(10^{-5}\) & \(10^{0}\) & 37.0 & \(10^{-4}\) & \(10^{-1}\) & 56.5 & \(10^{-5}\) & \(10^{-1}\) \\ RAN & 96.9 & \(10^{-2}\) & \(10^{-1}\) & 39.3 & \(10^{-2}\) & \(10^{-1}\) & 60.2 & \(10^{-3}\) & \(10^{-5}\) \\ ROAN & 96.8 & \(10^{-2}\) & \(10^{-1}\) & 39.5 & \(10^{-2}\) & \(10^{0}\) & 60.7 & \(10^{-3}\) & \(10^{-2}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparative experiment results on MNIST, CIFAR-10 and SVHN datasets. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Name**} & \multicolumn{2}{c}{**REAN vs. PIAN**} & \multicolumn{2}{c}{**REAN vs. RAN**} & \multicolumn{2}{c}{**REAN vs. ROAN**} \\ \cline{2-7}  & **H** & **P** & **H** & **P** & **H** & **P** \\ \hline MNIST & 1 & 1.34e-05 & 1 & 4.12e-07 & 1 & 5.03e-06 \\ CIFAR-10 & 1 & 3.34e-09 & 1 & 2.13e-04 & 1 & 2.69e-02 \\ SVHN & 1 & 3.54e-10 & 1 & 5.55e-06 & 1 & 4.42e-05 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Paired \(t\)-tests on MNIST, CIFAR-10 and SVHN datasets. layer outputs. The REAE algorithm consists of four steps; firstly, solving for the feature mapping parameters, then mapping the input data into the hidden space, then solving for the decoder model parameters, and finally, obtaining the encoder model parameters by weight binding. There is no direct performance comparison between autoencoders trained by different forward algorithms. Autoencoders are mainly used for data feature extraction, so they are applied to image classification tasks, and the classification accuracy is used as a criterion to evaluate the performance of the algorithm. The autoencoder itself does not have a classification function. In this paper, the decoder of the autoencoder is removed and the output of the encoder is used as the input features of the Softmax classifier to build a classification model. The experiments consist of two parts: analysis of factors affecting classification performance and comparison of algorithm performance. The average activation value of the hidden layer outputs and the number of hidden nodes affect the classification performance. The experimental results on the average activation value of the hidden layer outputs show that different datasets correspond to different optimal hidden layer activation values. This is exactly what is expected in this paper, i.e., to characterize the different datasets by means of a hidden layer average activation value constraint. The experimental results on hidden nodes show that the classification performance of the REAN autoencoder network improves as the number of hidden nodes increases. This is due to the fact that, as the number of hidden nodes increases, REAE can extract a richer set of features. It should be noted that there is no clear functional relationship between the average activation value of the hidden layer and the number of hidden nodes, and cross-validation is required to determine the values of these two hyperparameters in specific applications.