# Deep Neural Networks Classification via Binary Error-Detecting Output Codes
## 2 Methods and Materials
### Decision Method  
How can each training pattern be connected with a suitable class? While the neural network itself implements a nonlinear mapping \(\mathcal{R}^{m}\rightarrow\mathcal{R}^{n}\), \(m>n\), the classification's goal is to assign a class from the finite set (expressed by the codeword in our case) to each input sample. The essential question to be answered is, which codeword is the most similar to the obtained output? Even though the codeword is binary, we can assume it as a point in the Euclid space. We can find the closest codeword to the network output vector regarding a given distance (e.g., Euclidian). This distance-based approach is sufficient if the most \begin{table}
\begin{tabular}{c c c c c} \hline \hline CRC1: 100,0011 & CRC2: 101,1011 & CRC3: 110,0001 & CRC4: 110,0111 & CRC5: 111,0011 \\ \hline CRC6: 100,1001 & CRC7: 101,0111 & CRC8: 110,1101 & CRC9: 111,0101 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The binary form of the generating polynomials of used Cyclic Redundancy Checks (CRCs). similar codeword finding is a final step in the classification chain. However, the output assignment to the codeword is a logical statement, and thus we can apply logical rules to find new properties. We can even interpret binary \(n\)-tuples as vectors that, together with Galois-field based scalars, create a finite vector space and derive additional properties using the coding theory. This rich source of logical and algebraic tools is available only after assigning neural network outputs to the codewords. Unfortunately, after this flip from the continuous space into the binary space, we lose all similarity information. To tie the benefits of both worlds, we look at the result of an algebraic operation as a statement, and we evaluate the result by the truth value, applying the fuzzy logic. For example, this approach will assess the syndromes' truth values and prove that the truest codeword gives the truest syndrome. As explained later, we decided to use the Zadeh (standard) fuzzy logic and \(t\in[0,1]\) as the truth-values. Therefore, the first step is the normalization of the individual neural network outputs. Let the response of the neural network to the given sample be \(\boldsymbol{y}~{}=~{}(y_{0},\ldots,y_{n-1})\in\mathcal{R}^{n}\), and \(\widetilde{\boldsymbol{y}}~{}=~{}(\widetilde{y}_{0},\ldots,\widetilde{y}_{n -1})\in[0,1]^{n}\) after normalization, where \(n\) is the number of output bits (\(n=10\) in our case). We use three types of normalization for \(i=0,\ldots,n-1\):  
\[\text{sigmoid}\;\widetilde{y}_{i}~{}=~{}\sigma(y_{i})~{}=~{}\frac{1}{1+e^{-y _{i}}} \tag{2}\]  
\[\text{linear}\;\widetilde{y}_{i}~{}=~{}\frac{y_{i}-y_{min}}{y_{max}-y_{min}},~{}y_{min}~{}=~{}min\{y_{0},\ldots,y_{n-1}\},y_{max}~{}=~{}max\{y_{0},\ldots, y_{n-1}\} \tag{3}\]  
\[\text{vector}\;\widetilde{\boldsymbol{y}}~{}=~{}\frac{\boldsymbol{y}}{|| \boldsymbol{y}||_{2}} \tag{4}\]  
After normalization, we have to map the individual outputs to a codeword. The simplest way (that is an extension of the one-dimensional task) is to decide for each output independently. \[c_{i}~{}=~{}\left\{ \begin{array}{ll}0,~{}\widetilde{y}_{i}<0.5\\ 1,~{}\widetilde{y}_{i}\geq 0.5\end{array} ,i~{}=~{}0,1,\ldots,n-1.\right. \tag{5}\]  
We call this method a \"bit threshold method\". From the other perspective, we can interpret the output value as a truth value of the fuzzy statement that the output gives as a specific bit (see Equation (9) for details). The more complex approaches take the output vector \(\widetilde{\boldsymbol{y}}\) as a whole; the most popular is the softmax method. This approach is based on one-hot encoding, where the bit dependence is based on the fact that all bits are equal to zero except one that is equal to one. After an exponential transformation,  
\[\widetilde{y}_{i}\leftarrow\frac{e^{\widetilde{y}_{i}}}{\sum_{j =~{}0}^{n-1}e^{\widetilde{y}_{j}}}~{}~{}. \tag{6}\]  
applying a probability distribution scale, the expected value one is assigned to the maximum value and the expected zeros, to the others.