# Deep Error-Correcting Output Codes
## 3 Deep Error-Correcting Output Codes (DeepECOCs)
### DeepECOCs
Firstly, we compared DeepECOCs with two traditional feature learning models (including a principal component analysis (PCA) [51] and marginal Fisher analysis (MFA) [52]), AE, DAE and single-layer ECOC framework. For MFA, the number of nearest neighbors for constructing the intrinsic graph was set to 5, while that for constructing the penalty graph was set to 15. For DeepECOCs, we also used 6 coding design methods, as in the previous experiments. We used a batch gradient descent for the fine-tuning process; the batch size was set to 100, the learning rate was set to 1, the number of epoch was set to 40,000 and the denoising rate and dropout rate were set to 0.1. We also used the incremental SVMs with an RBF kernel as base classifiers. For the single-layer ECOC framework, we adopted ECOCONE (initializing by one-versus-one) as the coding design method and linear loss-weighted (LLW) decoding strategy. The experimental results are shown in Figure 3.

From Figure 3, we can observe that DeepECOCs with the ECOCONE (initializing by one-versus-one) coding strategy achieved the best result among other methods, including

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Problem** & **Single** & **AE** & **DAE** & **V1** & **V2** & **V3** & **V4** \\ \hline Dermatology & 0.9513 & 0.9429 \(\pm\) 0.0671 & 0.9674 \(\pm\) 0.0312 & **0.9731**\(\pm\) **0.0314** & 0.8852 \(\pm\) 0.0561 & 0.9722 \(\pm\) 0.0286 & 0.8834 \(\pm\) 0.0349 \\ Iris & 0.9600 & 0.9600 \(\pm\) 0.0562 & 0.9333 \(\pm\) 0.0889 & 0.8818 \(\pm\) 0.0695 & 0.8137 \(\pm\) 0.0768 & **0.9667**\(\pm\) **0.0471** & **0.9667**\(\pm\) **0.0471** \\ Ecoli & 0.8147 & 0.7725 \(\pm\) 0.0608 & 0.8000 \(\pm\) 0.0362 & 0.8275 \(\pm\) 0.0427 & 0.7868 \(\pm\) 0.0633 & 0.9102 \(\pm\) 0.0701 & **0.9183**\(\pm\) **0.0611** \\ Wine & 0.9605 & 0.9765 \(\pm\) 0.0264 & 0.9563 \(\pm\) 0.0422 & 0.9063 \(\pm\) 0.0793 & 0.9625 \(\pm\) 0.0604 & **0.9875**\(\pm\) **0.0264** & 0.9625 \(\pm\) 0.0437 \\ Glass & 0.6762 & 0.6669 \(\pm\) 0.1032 & 0.6669 \(\pm\) 0.0715 & 0.7563 \(\pm\) 0.0653 & 0.6875 \(\pm\) 0.0877 & **0.8013**\(\pm\) **0.0476** & 0.7830 \(\pm\) 0.0893 \\ Thyroid & 0.9210 & 0.9513 \(\pm\) 0.0614 & 0.9599 \(\pm\) 0.0567 & 0.8901 \(\pm\) 0.1177 & **0.9703**\(\pm\) **0.0540** & 0.9647 \(\pm\) 0.0431 & 0.9560 \(\pm\) 0.0633 \\ Vowel & 0.7177 & 0.6985 \(\pm\) 0.0745 & 0.7101 \(\pm\) 0.0756 & 0.7020 \(\pm\) 0.0529 & 0.6563 \(\pm\) 0.0721 & **0.7628**\(\pm\) **0.0716** & 0.6874 \(\pm\) 0.0438 \\ Balance & 0.8222 & 0.8036 \(\pm\) 0.0320 & 0.8268 \(\pm\) 0.0548 & 0.8528 \(\pm\) 0.0534 & 0.8108 \(\pm\) 0.0634 & 0.8879 \(\pm\) 0.1331 & **0.9090**\(\pm\) **0.0438** \\ Yeast & 0.5217 & 0.5641 \(\pm\) 0.0346 & 0.5891 \(\pm\) 0.0272 & 0.5861 \(\pm\) 0.0318 & 0.5368 \(\pm\) 0.0582 & **0.6080**\(\pm\) **0.0385** & 0.5968 \(\pm\) 0.0378 \\ Satimage & 0.8537 & 0.8675 \(\pm\) 0.0528 & 0.8897 \(\pm\) 0.0304 & 0.8763 \(\pm\) 0.0576 & 0.8238 \(



traditional feature learning models, existing deep learning methods and a single-layer ECOC framework. However, if we used other coding design strategies (except ECOCONE initialized by one-versus-one), the DeepECOCs did not outperform an autoencoder and denoising autoencoder. In the second part, we evaluated DeepECOCs with a different number of hidden layers, and we used 2 to 6 hidden layers in our experiments. The parameter settings were the same as in the first part. Figure 4 shows the experimental results. We can observe that DeepECOCs obtained the best result when adopting 3 hidden layers. When the number of hidden layers was less than 3, the accuracies obtained by DeepECOCs increased, and the number of the hidden layers increased. When with the number of the hidden layers was more than 3, the performance of DeepECOCs decreased. ### Classification on the MNIST Data Set

MNIST includes a training set of 60,000 samples and a test set of 10,000 samples with 784 dimensional features. We designed an architecture for the autoencoder, denoising autoencoder and DeepECOCs. The architecture is 784-\(Z_{1}\)-\(Z_{2}\)-\(Z_{3}\)-10.