# Improving Convolutional Neural Networks' Accuracy in Noisy Environments Using k-Nearest Neighbors
## 2 Materials and Methods  
The presented work has an eminently experimental nature. Our hypothesis is that a hybrid CNN + kNN can outperform the predictive power of the widely-considered CNN + softmax in noisy environments. To assess this, we compare the results of a CNN trained for a classification task with the learned neural codes of the same CNN, but applying a kNN classifier to the last hidden layer output. In addition, we consider the \(\ell_{2}\) norm for the normalization of the neural codes [13; 19]. Let \(x\) be a vector of size \(n\), which represents the neural codes; the \(\ell_{2}\) norm is defined as:  
\[|x|=\sqrt{\sum_{k=1}^{n}|x_{k}|^{2}}\]  
After training a CNN with standard backpropagation, four configurations are evaluated in the inference stage (see Figure 1):  
1. Use of the CNN softmax layer. Use of the last hidden CNN layer (before the softmax) to get the neural codes, which are fed to a kNN that compares them to the training set prototypes using Euclidean distance in order to get the most likely class. 3. Similar to the previous configuration, but normalizing with \(\ell_{2}\) the neural codes before the kNN. 4. Using directly the kNN on the raw data without any representation learning. The presented configurations are evaluated with different datasets selected to depict different numbers of features and samples. Specifically, our evaluation comprises the following seven datasets of images (summarized in Table 1):  
* United States Postal Office (USPS) [20] and MNIST [21] are datasets of handwritten digits with binary images. * Landsat is a dataset from the UCI repository [22]. It comprises \(3\times 3\) windows in the four spectral bands depicting images of satellites. Each window is labeled as regards the information of the central pixel of the window. * Handwritten Online Musical Symbol (HOMUS) [23] depicts binary images of isolated handwritten music symbols collected from 100 different musicians. * NIST Special Database 19 (NIST) of the National Institute of Standards and Technology [24] consists of a huge dataset of isolated characters. For this work, a subset of the upper case characters was selected. * CIFAR-10 and CIFAR-100 [25] are standard object recognition datasets for the computer vision community. They consist of \(32\times 32\) color images extracted from the 80 million tiny images dataset [26] and containing 10 and 100 different categories, respectively. As regards preprocessing of the input data, the values of the pixels from MNIST, HOMUS and NIST images are divided by 255 for normalization, whereas the mean image is subtracted from CIFAR images. The USPS and Landsat datasets are already normalized in origin. Furthermore, Table 2 shows the evaluated network topologies for each dataset. The selected CNN architectures are designed to obtain close to state-of-the-art results on the target task. Custom models \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Name** & **Instances** & **Classes** & **Shape** \\ \hline USPS & 9298 & 10 & \(1\times 16\times 16\) \\ MNIST & 70,000 & 10 & \(1\times 28\times 28\) \\ Landsat & 6435 & 6 & \(4\times 3\times 3\) \\ HOMUS & 15,200 & 32 & \(1\times 40\times 40\) \\ NIST & 44,951 & 26 & \(1\times 32\times 32\) \\ CIFAR10 & 60,000 & 10 & \(3\times 32\times 32\) \\ CIFAR100 & 60,000 & 100 & \(3\times 32\times 32\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Description of the image datasets used in the experimentation. HOMUS, Handwritten Online Musical Symbol. Figure 1: Scheme of the four configurations evaluated. have been designed for USPS, MNIST, Landsat, HOMUS and NIST, whereas the CIFAR datasets were trained using the well-known VGG16 [27] and the 50-layered ResNet [28] models. The CNN training is carried out by means of stochastic gradient descent [29] considering the adaptive learning rate method proposed by [PERSON] [30]. A 128 mini-batch size is used, and the networks have been trained with a maximum of 200 epochs, with early stopping if the loss does not decrease during 10 epochs.