# Deep Error-Correcting Output Codes
## 3 Deep Error-Correcting Output Codes (DeepECOCs)
### DeepECOCs
We built our DeepECOCs with different coding design methods. They were one-versus-all, one-versus-one, discriminant ECOC (DECOC) and ECOC optimizing node embedding (ECOCONE). Here, because the ECOCONE's initial codings were different (initialized by one-versus-all, one-versus-one and DECOC), the coding strategies of DeepECOCs had 6 types. In our experiments, we adopted 3 hidden layers DeepECOCs with 0.1 denoising rate and 0.1 dropout rate,

\[\mathbf{x}\xrightarrow{dD}\mathbf{\tilde{x}}\xrightarrow{\mathbf{W}^{1}} \mathbf{z}^{1}\xrightarrow{\mathbf{W}^{2}}\mathbf{z}^{2}\xrightarrow{\mathbf{W }^{3}}\mathbf{z}^{3}\xrightarrow{softmax}\mathbf{y}. \tag{11}\]

For the fine-tuning process, we used the stochastic gradient descent algorithm. The learning rate and epoches for different data sets were given in Table 3. The autoencoder and denoising autoencoder's architectures were same as the DeepECOCs with ECOCONE (initializing by one-versus-one) coding design method. For the single-layer ECOC framework, we chose the best results from [9] as the compared results. The average classification accuracy and standard deviation based on a 10-fold cross validation were reported. Table 4 shows the average classification accuracy and standard deviation on the 16 UCI data sets. Except for the OptDigits data set, DeepECOCs consistently achieved the best results compared with autoencoder, denoising autoencoder and single-layer ECOC framework. On the OptDigits data set, indeed, DeepECOCs achieved a comparative result with the single-layer ECOC framework. In addition, DeepECOCs with an ECOCONE coding strategy achieved the best results in most cases. From the mean rank value, we can observe that DeepECOC with an ECOCONE strategy (initialized by one-versus-one and DECOC) far surpasses other methods. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Problem**} & \multicolumn{2}{c}{**OneVsOne**} & \multicolumn{2}{c}{**OneVsAll**} & \multicolumn{2}{c}{**DECOC**} & \multicolumn{2}{c}{**ECOCONE**} \\ \cline{2-9}  & **LibSVM** & **I-SVM** & **LibSVM** & **I-SVM** & **LibSVM** & **I-SVM** & **LibSVM** & **I-SVM** \\ \hline Vowel & 0.4187 & **0.4956** & 0.3953 & **0.6591** & 0.2736 & **0.3278** & 0.4476 & **0.4956** \\ Balance & 0.8927 & 0.8927 & 0.9042 & 0.9042 & 0.8927 & 0.8927 & 0.8927 & 0.8927 \\ Yeast & 0.4741 & **0.5744** & 0.2383 & **0.2760** & 0.3751 & **0.5693** & 0.4147 & **0.6000** \\ Satimage & 0.8361 & **0.8692** & 0.7620 & **0.8334** & 0.7411 & 0.8532 & 0.8305 & **0.8604** \\ Letter & 0.7443 & **0.8216** & 0.2166 & **0.4536** & 0.7530 & **0.8375** & 0.7440 & **0.8120** \\ Pendigits & 0.9688 & **0.9859** & 0.9021 & **0.9773** & 0.9065 & **0.9758** & 0.9691 & **0.9850** \\ Segmentation & 0.8595 & **0.8872** & 0.6187 & **0.6759** & **0.7969** & 0.6706 & 0.8601 & **0.8872** \\ OptDigits & 0.9492 & **0.9860** & 0.8866 & **0.9649** & 0.8793 & **0.9808** & 0.9492 & **0.9860** \\ Shuttle & 0.9178 & **0.9475** & 0.8312 & **0.9034** & 0.8352 & **0.9407** & 0.9179 & **0.9532** \\ Vehicle & 0.6324 & **0.7595** & **0.7378** & 0.4892 & 0.6054 & **0.7243** & 0.6324 & **0.7595** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: _Cont._ \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Problem** & \(\eta\) & **Epoch** & **Problem** & \(\eta\) & **Epoch** \\ \hline Dermatology & 0.1 & 2000 & Yeast & 0.01 & 4000 \\ Iris & 0.1 & 400 & Satimage & 0.01 & 4000 \\ Ecoli & 0.1 & 2000 & Letter & 0.01 & 8000 \\ Wine & 0.1 & 2000 & Pendigits & 0.01 & 2000 \\ Glass & 0.01 & 4000 & Segmentation & 0.01 & 8000 \\ Thyroid & 0.1 & 800 & Optdigits & 0.01 & 2000 \\ Vowel & 0.1 & 4000 & Shuttle & 0.1 & 2000 \\ Balance & 0.1 & 4000 & Vehicle & 0.1 & 4000 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Details of the learning rate \(\eta\) and epoch on the UCI data sets. ### Classification on the USPS Data Set

The USPS handwritten digits data set includes 7291 training samples and 2007 test samples from 10 classes with 256 dimensional features. Our experiments on this data set were divided into 2 parts.