Nearest Neighbour Noise (NNN) as Regularization Method for Neural Networks
By: Denis Baskan
Tags: Convolutional Network, Neural Networks, Regularization, Manifold, Noise

During my research about Noise Injection (NI) as a regularization method, an idea came into my mind. The usual NI works as follows:

The most common approach is to take a Gaussian Normal Distribution and simply add it to the Input of your data set during each epoch. Other variants have been proposed, e.g. multiplicating instead of adding noise and injecting noise at the second or third hidden layer of a neural network given the fact that the network is deep (some more layers follow after the injection).

Your data set probably lies on a so-called manifold as Ian Goodfellow and colleagues have described it in their book “Deep Learning”. You can think of a line in a k-dimensional space where all your data sit on. k is hereby smaller than n, which is the dimension of your data set. Since the NI is applied on all dimensions, you can think of a circle around a data point. The radius of the circle is determined by the variance of your distribution. The higher the variance, the bigger the circle. So, you have to choose an appropriate amount of noise. Your model learns to map the corrupted data (noisy points) back to the manifold. This is illustrated below. The orange point got some noise and will be mapped back to the manifold.

Manifold Representation

The corrupted point is mapped towards the closest point of the manifold because the model doesn’t know where the original point came from. Points of one class are assumed to lie closely together. A problem can now rise up. Imagine the points of different classes are close to each other (the borders of the classes). Applying noise to these points shift them towards the area of other classes. This makes it difficult to classify them correctly.

Another possible fact to mention is that, imagine you have two points of one class, and they are already close together. One point is now shifted even closer to its neighbour. You model sees two nearly identical points coming from the same class. This is not a bad thing, but your model won’t learn much from it since it even decreases the variance of your data set. The variance of your data set can be interpreted as how different your samples are. I’ll now describe my idea.

Nearest Neighbour Noise (NNN), adds the same amount of noise to a point, but slightly shifted. A point gets shifted away from its nearest neighbour. This means the points get shifted to an area where fewer points are. I’ll define the direction of the shift as ‘r’.

Example Data

We can see that the data distributions have changed and they look somehow distorted. Let’s investigate how this approach affects the accuracy of a CNN trained on the Fashion MNIST data set.

I’ll use one type of CNN architecture and will run the following experiments.

No noise

Standard Gaussian noise

NNN with r=r*0.1

NNN with r=r*0.2

NNN with r=r*0.35

NNN with r=r*0.5

To speed up training, I’ll use a fourth of the training set. This has also a technical reason. A distance matrix has to be calculated containing lots of entries. The matrix has n rows and n columns. Distances on the diagonal represent the distance from a sample with itself, which is of course 0. Further, the matrix is symmetric, meaning the distance from sample 1 to sample 2 is equivalent to the distance from sample 2 to 1. I’ve found some code snippets where the calculations can be done on the GPU:

Because the entire training set doesn’t fit into 8GB of VRAM, I had to shrink the training set.

Results

I’ve always taken the results of the best test accuracy.

No noise:

train Loss: 0.3182 Acc: 0.8813

test Loss: 0.3968 Acc: 0.8581

Gaussian noise:

train Loss: 0.3364 Acc: 0.8709

test Loss: 0.3921 Acc: 0.8603

NNN with r=0.1

train2 Loss: 0.3360 Acc: 0.8711

test Loss: 0.3909 Acc: 0.8604

NNN with r=0.2

train2 Loss: 0.3365 Acc: 0.8713

test Loss: 0.3859 Acc: 0.8645

NNN with r=0.35

train2 Loss: 0.3401 Acc: 0.8708

test Loss: 0.3862 Acc: 0.8628

NNN with r=0.5

train2 Loss: 0.3452 Acc: 0.8680

test Loss: 0.3868 Acc: 0.8607

Two things can be noticed. Once, all achieved accuracies are already quite high. Second, NNN delivered in all experiments the best results!

Another more advanced approach is to take the nearest neighbour of a different class. This separates points at borders less than points which are far away. The shift, therefore, has to be scaled.

I’ve used r=r/norm(sample)*0.4.