Title: How 1 pixel shifts in images can kill your RBM performance - PyImageSearch

URL Source: https://pyimagesearch.com/2014/06/16/1-pixel-shifts-images-can-kill-rbm-performance/

Published Time: 2014-06-16T14:00:51+00:00

Markdown Content:
[![Image 1: Can you tell the difference between these two images?](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20202'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2014/06/raptor_translate.jpg)

Can you tell the difference between the two images above?

Probably not.

The one on the right has been shifted one pixel down.

And while it still looks like the same image to us, to a Restricted Boltzmann Machine, this translation could spell trouble.

Raw Pixel Intensities as Feature Vectors
----------------------------------------

Did you know that there is a subtle, _but critical_ issue when using raw pixels as feature vectors, which is commonly done in image-based deep learning classification tasks?

If you’re not careful and don’t take the appropriate precautions, _small, 1 pixel shifts_ in your input image can _dramatically_ hurt the performance of your classifier.

And we’re only talking about a _one pixel_ shift. A shift this small is barely (if at all) noticeable to the human eye.

But if you’ve trained your Restricted Boltzmann Machine on raw pixel features, you might be in for a surprise.

Sure, convolutional neural networks help alleviate this translation issue. Actually, _alleviate_ is too strong of a word. They are able to _tolerate_ translations in the image. The convolutional nets (in general) are still susceptible to this translation issue.

And as one of Google’s most recent papers, _Intriguing properties of neural nets_ suggests, small changes in the input image can dramatically alter the overall classification of the network.

The authors call these types of images “adversarial images” do to the fact that they are, for all intents and purposes, identical to their original images according to the human eye.

_Note: If you’re interested in reading more about Google’s paper (along with my criticism of machine learning fads), definitely check out my previous post,_ [Get off the deep learning band wagon and get some perspective](https://pyimagesearch.com/2014/06/09/get-deep-learning-bandwagon-get-perspective/)_._

However, these adversarial images were constructed in a far more complex method than simple one pixel translations of the input image. In fact, these images were constructed by manipulating pixel values by a very small amount in order to maximize the error of the deep learning network.

But it does demonstrate how subtle changes in an image, which are totally undetectable to the human eye, can lead to a misclassification when using raw pixel features.

In order to demonstrate (on a substantially smaller scale) some of the issues associated with deep learning nets based on raw pixel feature vectors, I’ve decided to conduct a little experiment: _**If I take my testing set of images and shift each image one pixel up, down, left, and right, will performance decrease?**_

Again, for all intents and purposes, these one pixel shifts will be unnoticeable to the human eye…but what about a Restricted Boltzmann Machine?

**So here’s what we’re going to do:**

*   Construct a training and testing split using the raw pixel features of a sample of the [MNIST dataset](http://yann.lecun.com/exdb/mnist/).
*   Apply a single Restricted Boltzmann Machine to learn an unsupervised feature representation from the MNIST sample.
*   Train a Logistic Regression classifier on top of the learned features.
*   Evaluate the classifier using the test set to obtain a baseline.
*   Perturb the testing set by shifting the images one pixel up, down, left, and right.
*   Re-evaluate our classification pipeline and see if accuracy decreases. Again, these one pixel shifts are virtually unnoticeable to the human eye — _but they will end up hurting the overall performance of the system._

**And here’s what we’re _NOT_ going to do:**

*   Replicate Google’s results using adversarial images.
*   Claim that these results are incriminating of all raw pixel based approaches. They’re not. I’m only going to use a single RBM here. So there won’t be any stacking — and thus there won’t be any deep learning.
*   Claim that researchers and developers should abandon raw pixel based approaches. There are ways to fix the problems I am suggesting in this post. The most common way is to apply deformations to the images at training time (i.e. generating more training data by artificially transforming the image) to make the neural net more robust. The second way is to sub-sample regions of the input image.

But _what I am going to do_ is show you how _small, one pixel shifts in images_ can **_dramatically_** decrease the accuracy of your RBM if the appropriate precautions aren’t taken.

Hopefully this series of blog posts, with Python and scikit-learn code included, will aide some students and researchers who are just exploring neural nets and mapping raw pixel feature vectors to outputs.

Preliminary Results
-------------------

The code to generate my results isn’t quite presentable yet (that will come next post), but I wanted to show some preliminary results:

RBM + LOGISTIC REGRESSION ON ORIGINAL DATASET
             precision    recall  f1-score   support

          0       0.95      0.98      0.97       196
          1       0.97      0.96      0.97       245
          2       0.92      0.95      0.94       197
          3       0.93      0.91      0.92       202
          4       0.92      0.95      0.94       193
          5       0.95      0.86      0.90       183
          6       0.95      0.95      0.95       194
          7       0.93      0.91      0.92       212
          8       0.91      0.90      0.91       186
          9       0.86      0.90      0.88       192

avg / total       0.93      0.93      0.93      2000

The first thing I did was take a sample of the MNIST dataset (2000 data points; roughly uniformly distributed per class label) and constructed a 60/40 split — 60% of the data for training and 40% for validation.

Then I used Bernoulli Restricted Boltzmann Machine to learn an unsupervised feature representation from the training data, which was then fed into a Logistic Regression classifier.

Finally, this RBM + Logistic Regression pipeline was evaluated using the testing data, obtaining 93% accuracy.

All relevant parameters were grid-searched and cross-validated to help ensure optimal values.

Then, I decided to “nudge” the testing test, by shifting each image in the testing set one pixel up, down, left, and right, yielding a testing set four times larger than the original.

These shifted images, while nearly identical to the human eye, provided to be a challenge for the pipeline as accuracy dropped 5%.

RBM + LOGISTIC REGRESSION ON NUDGED DATASET
             precision    recall  f1-score   support

          0       0.94      0.93      0.94       784
          1       0.96      0.89      0.93       980
          2       0.87      0.91      0.89       788
          3       0.85      0.85      0.85       808
          4       0.88      0.92      0.90       772
          5       0.86      0.80      0.83       732
          6       0.90      0.91      0.90       776
          7       0.86      0.90      0.88       848
          8       0.80      0.85      0.82       744
          9       0.84      0.79      0.81       768

avg / total       0.88      0.88      0.88      8000

On a small scale, we can see the issue with using raw pixels as feature vectors. Slight translations in the image, tiny rotations, and even noise during the image capturing process can reduce accuracy when fed into the net.

While these results are by no means conclusive, they at least demonstrate the general intuition that using raw pixel feature vectors can be prone to error without significant preprocessing beforehand.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

In this blog post I introduced the notion that small, one pixel shifts in images can kill your Restricted Boltzmann Machine performance if you can’t careful.

I then provided a “teaser” set of results to demonstrate that the one pixel translations in the images, while nearly identical to the human eye, can lead to a reduction in accuracy.

Next Up:
--------

In my next blog post, I’ll show you my Python code to apply deep learning and a Restricted Boltzmann Machine to the MNIST dataset.

Be sure to signup for the newsletter below to receive an update when the post goes live! You won’t want to miss it…

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.