# Deep Neural Networks Classification via Binary Error-Detecting Output Codes
## 2 Methods and Materials
### Output Coding  
Categorical data at the neural network outputs can divide the space in different ways. Nowadays, most of the classifiers use the one-hot (one-per-class) output coding. Applying the softmax layer, the winning class gives the maximum output, and the output space is divided by hyperplanes into class subspaces. One-hot codewords create an orthogonal standard codespace basis. Bits of the codewords are not independent because giving a value to the one bit changes the possibility of the rest of the bits' values in the codeword. This property provides an advantage to such a holistic approach embodied into one-hot encoding by softmax. The disadvantage of one-hot encoding is a small pairwise Hamming distance equal to two bits.  
Another approach is to use a nonstandard orthogonal codespace basis. One possibility is given by Hadamard vectors [25]. The Hadamard matrix provides the following codewords:  
\[\mathbf{H}_{i+1}\ =\ \left(\begin{array}{cc}\mathbf{H}_{i}&\mathbf{H}_{i}\\ \mathbf{H}_{i}&-\mathbf{H}_{i}\end{array}\right)\text{, }\mathbf{H}_{0}\ =\ (1), \tag{1}\]  
where the value \"\(-1\)\" is replaced by \"0\". Except for the first matrix row, in which all elements are equal to one, other rows contain bits of \"1\" in the half of bits and \"0\" in the rest of the codeword. A helpful feature of the Hadamard matrix is an equidistant distribution of the codewords with \(2n-1\) bits Hamming distance, where \(2n\) is the number of neural network outputs. Let us notice that number of classes has to be less than or equal to the number of network outputs. Further, ref [26] also argues for the same number of \"0\" and \"1\" in the matrix columns (except the first one). They try to preserve this property by using fewer codewords than giving the full codespace (the first bit in all matrix rows is excluded from the codeword).  
Binary coding can give a bigger output capacity. While outputs in one-hot or Hadamard coding are limited to basic vectors, binary coding outputs can include all vectors generated by this basis. While the one-hot encoding is vulnerable to adversarial samples [27], the binary coding, e.g., multi-way coding [28], is more resistant.  
Assigning codewords to the categories in binary coding is not as direct as in one-hot encoding. A well-studied example is the Error Correcting Output Code (ECOC) introduced in [8]. Recognition of each bit of the output binary code (given by a code matrix) is trained separately on the divided binary pattern or feature training set. The aim is to have the bits independent. Reaching a suitable code matrix remains the main challenge [29; 30; 31; 32]. We can interpret the ECOC codeword as an ensemble of independent bits where each bit indicates the appearance of a corresponding feature in the pattern. Our paper focuses on multi-class patterns, where the label is assigned to the whole binary codeword. The dependency of bits allows using inter-bit information in a way similar to softmax.  
The main advantage of the binary codes is the excellent knowledge acquired in the coding theory. We use linear block codes for their possibility of the separation class information from the checksum part of the codeword. To compare one-hot versus binary encoding at the 10-class database CIFAR-10, we choose 10-bit binary codes. The essential decision on codeword acceptance is based on syndrome calculation. The error is detected [15] if the syndrome is a non-zero vector or the zero vector (16 codewords), but the codeword is not assigned to any class (e.g., 6 of the 16 codewords). When the rules detect the codeword as an error, we consider the sample untrustworthy, and we do not assign any class to it (we abstain from classification). We use (10, 4) block systematic Cyclic Redundancy Check (CRC) codes as an example. Finding an optimal code from the classification accuracy point of view remains a challenge for the future as the neural network errors are correlated. Table 1 gives generating polynomials [15] of these codes. All of them are irreducible, and the first five are also primitive.