# Deep Error-Correcting Output Codes
## 3 Deep Error-Correcting Output Codes (DeepECOCs)
### DeepECOCs  
To combine the advantages of ECOC and deep learning algorithms, we build the DeepECOC architecture as follows  
\[\mathbf{x}\xrightarrow{q_{D}}\tilde{\mathbf{x}}\xrightarrow{\mathbf{W}^{1}} \mathbf{z}^{1}\xrightarrow{\mathbf{W}^{2}}\cdots\xrightarrow{\mathbf{W}^{n-1 }}\mathbf{z}^{n-1}\xrightarrow{softmax}\mathbf{y}, \tag{5}\]  
where the first step makes the clean input \(\mathbf{x}\in[0,1]^{d}\) partially destroyed by means of a stochastic mapping \(\tilde{\mathbf{x}}\sim q_{D}(\tilde{\mathbf{x}}\mid\mathbf{x})\). In the corrupting process, we set a parameter called the denoising rate \(\
u\). For each input \(\mathbf{x}\), a fixed number \(vd\) of components are chosen at random, and their value is forced to \(0\), while the others are left untouched. This operation makes the model more robust and prevents the overfitting problem in most cases [27]. Subsequently, the \"corrupted\" data are taken as inputs for the DeepECOCs model. \(\mathbf{W}^{1}\) and \(\mathbf{b}^{1}\) are the weight matrix and bias vector learned from the first ECOC module. The output of the first hidden layer is denoted as  
\[\mathbf{z}^{1}=s((\mathbf{W}^{1})^{\mathrm{T}}\mathbf{x}+\mathbf{b}^{1}), \tag{6}\]where \(s(\cdot)\) is the sigmoid activation function \(s(x)=\frac{1}{1+e^{-x}}\). From the second layer to the \((n-1)\)-th layer, we use the stacked ECOC modules to learn the weight matrices and biases, which can be considered as weights between two successive layers. Similarly, we use the output of the \(k-1\)-th layer as the input of the \(k\)-th layer,  
\[\mathbf{z}^{k}=s((\mathbf{W}^{k})^{T}\mathbf{z}^{k-1}+\mathbf{b}^{k}). \tag{7}\]  
Here, \(\mathbf{z}^{k}\) can be viewed as an approximate posterior probability and new representation of the input datum \(\mathbf{x}\). For example, if we adopt the OneVsAll coding strategy for the stacked ECOC modules, we first define the coding matrix \(\mathbf{M}^{C\times C}\), where \(C\) is the number of classes. Then, we can train \(C\) incremental SVM classifiers to obtain the weight matrix \(\mathbf{W}=\{\mathbf{w}_{1},\ldots,\mathbf{w}_{i},\ldots,\mathbf{w}_{C}\}\) and the bias \(\mathbf{b}=\{b_{1},\ldots,b_{i},\ldots,b_{C}\}\). Next, we calculate the output of the first layer by using Equation (6). Subsequently, we repeat this process layer by layer to build the DeepECOCs model. It is obvious that, if we adopt different coding strategies, we can obtain different kinds of DeepECOC architectures. For the last layer of DeepECOCs, we employ the softmax regression for the multi-class learning. Its cost function is defined as  
\[J(\mathbf{w})=-\frac{1}{N}(\sum_{i=1}^{N}\sum_{j=1}^{K}\mathbf{I}(y_{i}=j)\log \frac{\exp(\mathbf{w}_{j}^{T}\mathbf{z}_{i}^{n-1})}{\sum_{l=1}^{K}\exp( \mathbf{w}_{l}^{T}\mathbf{z}_{i}^{n-1})}), \tag{8}\]  
where \(\mathbf{I}(x)\) is the indicator function, and \(\mathbf{I}(x)=1\) if \(x\) is true, else \(\mathbf{I}(x)=0\). \(y_{i}\) is the label corresponding to \(\mathbf{x}_{i}\). It is easy to compute the probability that \(\mathbf{x}_{i}\) is classified to class \(j\),  
\[p(y_{i}=j|\mathbf{x}_{i},\mathbf{w})=\frac{\exp(\mathbf{w}_{j}^{T}\mathbf{z}_{ i}^{n-1})}{\sum_{l=1}^{K}\exp(\mathbf{w}_{l}^{T}\mathbf{z}_{i}^{n-1})}. \tag{9}\]  
Taking derivatives, one can show that the gradient of \(J(\mathbf{w})\) with respect to \(\mathbf{w}\) is,  
\[\
abla J(\mathbf{w})=-\frac{1}{N}\sum_{i=1}^{N}[\mathbf{x}_{i}(\mathbf{I}(y_{ i}=j)-p(y_{i}=j|\mathbf{x}_{i},\mathbf{w}))]. \tag{10}\]  
After the initialization step, we use back propagation [46] to fine-tune the whole architecture. In the fine-tuning process, we apply the dropout technique for regularization [47]. It is a very efficient way to perform model averaging with neural networks. Through the above processes, we can finally obtain the DeepECOCs model, which is robust and easy to be applied to multi-class learning tasks. In summary, we show the training process of our DeepECOCs framework in Algorithm 1.