# Deep Error-Correcting Output Codes
## 3 Deep Error-Correcting Output Codes (DeepECOCs)
### DeepECOCs
```
1:\(\mathbf{x}_{i}\), \(\mathbf{y}_{i}\), \(\mathbf{z}_{i}\


```
1:The set of training samples \(\mathbf{X}=\{\mathbf{x}_{1},\dots,\mathbf{x}_{j},\dots,\mathbf{x}_{n}\}\);
2:The labels corresponding to training samples \(\mathbf{y}=\{y_{1},\dots,y_{j},\dots,y_{n}\}\). 3:Parameters \(\mathbf{W}\) and \(\mathbf{b}\). 4:Initialize the first layer input \(\mathbf{Z}^{1}=\mathbf{X}\);
5:for\(i=1\) to \(L-1\)do
6:Initialize the weights and bias of \(i\)-th ECOC module;
7:\(\mathbf{W}^{i}=\mathbf{0},\mathbf{b}^{i}=\mathbf{0}\);
8:Pre-train process (ECOC coding step):
9:(1) Learn the ECOC matrix with a coding strategy, and obtain \(\mathbf{M}\in\{-1,1\}^{C\times P}\) (binary case) or \(\mathbf{M}\in\{-1,0,1\}^{C\times P}\) (ternary case);
10:(2) Train the incremental SVM binary classifiers according to \(\mathbf{M}\):
11:for\(k=1\) to \(P\)do
12:\((a_{k},b_{k}^{i})=\text{I-SVM}(\mathbf{Z}^{i},\mathbf{y})\);
13:\(\mathbf{w}_{k}^{i}=\sum_{j=1}^{N}\alpha_{j}y_{j}\mathbf{Z}_{j}^{i}\);
14:endfor
15:\(\mathbf{W}^{i}=\{\mathbf{w}_{1}^{i},\dots,\mathbf{w}_{k}^{i},\dots,\mathbf{w} _{p}^{i}\}\);
16:\(\mathbf{b}^{i}=\{b_{1}^{i},\dots,b_{k}^{i},\dots,b_{p}^{i}\}\);
17:\(\mathbf{z}^{i+1}=s((\mathbf{W}^{i})^{T}\mathbf{z}^{i}+\mathbf{b}^{i})\);
18:endfor
19:\(\mathbf{z}^{L}=\text{softmax}(\mathbf{z}^{L-1})\). 20:Use the back-propagation algorithm for fine-tuning. 21:return\(\mathbf{W}=\{\mathbf{W}^{i}\}\) and \(\mathbf{b}=\{\mathbf{b}^{i}\}\), \(i=1,\dots,L-1\). ```

**Algorithm 1** The training procedure of DeepECOCs; \(L\) is the number of layers, \(\text{I-SVM}(\mathbf{x},\mathbf{y})\) is the incremental SVM binary classifier, \(s(x)\) is the sigmoid function, and \(\text{softmax}(\mathbf{x})\) is the softmax function

### Combining Convolutional Neural Networks (CNNs) with DeepECOCs

CNNs can effectively extract features from images and videos [48; 49], while DeepECOCs can be used as a strong and flexible classifier. Here, we construct a new deep architecture to integrate the advantages of CNNs and DeepECOCs. Figure 2 illustrates such a deep architecture. In the next section, we demonstrate that this architecture can improve the accuracy of the original CNNs. The training process of this combined neural architecture consists of three steps. Firstly, we train the original CNNs as usual. Secondly, we take the output of the fully connected layer before the classifier as the input of DeepECOCs, and train DeepECOCs. Finally, we fine-tune the whole network by training it for a few epochs. Figure 2: A deep architecture combining CNNs and DeepECOCs. The CNN module produces a vector presentation of data as shown with the long black bar. DeepECOC takes this data representation as input and performs the classification task. This deep architecture can be trained in an end-to-end manner. ## 4 Experiments

To evaluate the effectiveness of the proposed method, DeepECOCs, we conducted 6 experiments. In the first one, we compared DeepECOCs with some deep learning models and a single-layer ECOC framework on 16 small and medium data sets from the UCI machine learning repository ([[http://archive.ics.uci.edu/ml/](http://archive.ics.uci.edu/ml/)]([http://archive.ics.uci.edu/ml/](http://archive.ics.uci.edu/ml/)) (accessed on 19 October 2023)). In the second one, we used the USPS handwritten digits data set ([[https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass.html#usps](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass.html#usps)]([https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass.html#usps](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass.html#usps)) (accessed on 19 October 2023)) and tested DeepECOCs with a different number of hidden layers. In the third one, we used the MNIST data set ([[http://yann.lecun.com/exdb/mnist/](http://yann.lecun.com/exdb/mnist/)]([http://yann.lecun.com/exdb/mnist/](http://yann.lecun.com/exdb/mnist/)) (accessed on 19 October 2023)) to further demonstrate the effectiveness of DeepECOCs on handwritten digits recognition tasks.