Retinal Inspired Neural Network Structure
By: Jeremiah Johnson
Tags: Neural Networks, Artificial Intelligence, Image Classification, Generative Adversarial, AI

If you’ve ever trained a GAN(Generative Adversarial Network) or an image classification neural network, you know just how data hungry it can be. Especially when you want to work with higher resolution images. The other day, I tried running a GAN on a couple hundred images. Just on a resolution of 256x256 and the compute time was close to an hour. And I’ve got a fairly decent graphics card: a GeForce RTX 2070 Super.

So that got me thinking: How in the world do animals process so much visual information on the go? Think about your own eyes for a minute.

Photo by Paul Skorupskas on Unsplash

We focus on a point in center, while everything else is out of focus. If you could look at the back of your eyeball under a microscope, you’d see around 130 million rods(think light/dark) and 7 million cones(think color). But the bulk of those rods and cones are smack dab in the center of your eye.

Photo by salvatore ventura on Unsplash

Having all of those sensors in the center allows you to see high definition at the focal point, while still being able to see if there is anything of interest is in the periphery.

So I wondered if a model based on nature could help reduce the high GPU requirements in any way. And it turns out, yes it can, and in a much more significant way than I could have imagined.

The Problem

To start, let’s consider a typical GAN and, say we were to feed it 10,000 images at 4,096 x 4,096 resolution. If you pull out your handy calculator, you’ll see that is shy of 17 million points of data in gray scale per image, and 3 times that if in RGB. Compare that to the 256x256 images I was processing earlier, which total around 65kb per picture. So we’re talking about an increase of 256 times the size for the larger images.

But that doesn’t take into consideration that training a neural network requires performing operations on humongous matrices. The time it takes to calculate the product of two matrices grows exponentially at just a measly* n³!!! (https://en.wikipedia.org/wiki/Matrix_multiplication_algorithm)

*dash of sarcasm

That turns our 256 times increase in data into nearly 17 million times the processing time! See the problem? So any way to reduce the amount of data needing to be processed will yield dividends.

Plus, it’s not like every part of an image is important for determining what you are looking at. With a quick glance at a dog, you know you’re looking at your furry companion and not a hungry crocodile.

Photo by Roberto Nickson on Unsplash

But you didn’t have to focus on the wall, floor, fur, etc. It was all there, but just not as clear.

So how could we turn a simple glance into digital code? I developed a method to do just this.

The RNN-i Structure

Meet the Retinal Inspired Neural Network Input method, or RNN-i, for short.

First, we want to choose a focal area size. For this example, we’ll use 4x4. That will be at the center of our focus. We will pass all of those data points into our first row of “glance” data. Since it is 4x4, that will be 16 data points in our first row.

Second, we will want to take an area of 8x8(two times the width and height), centered on the same focal point. But we will process each 2x2 part to take the mean average. The mean average is just (x1+x2+x3+x4)/4. That turns those four pixels into one averaged pixel. Then we continue doing this with each 2x2 subsection in the 8x8 section. This gives us, again, 16 points of data, which we now enter into our second row.

Third, we continue with this process, but with 16x16 as our area and 4x4 subsections to be averaged.

RNN-i Grid

Fourth, we can keep doing this as many times as we like, until our entire image is “in view”. Each time we double the width and height, we only add one row of 16 pieces of data.

You might be asking now, “Wouldn’t that be confusing since each row of data is starting with a new top left corner?” Maybe for us. But the cool thing about neural networks is that they can adapt to whatever way you feed the data, as long as you stay consistent. The order doesn’t matter if you just make sure you always send it in the same order every time.

The Result

So just how much data can this method save? Taking our 4,096 x 4,096 pixel image example earlier, instead of entering close to 17 million data points, we will be entering only 176 points of data per “look” with this method.

Photo by Ben White on Unsplash

Game-changer.

Now, with so few data points, you would expect to lose a lot of accuracy. But I tested this method out on the MNIST fashion set here. And, in this experiment, the accuracy lost was only from 88.3% accuracy for whole images down to 81.4% accuracy with a single glance at center, using this approach.

Here is a link to the GitHub paper and code: https://github.com/therealjjj77/RNNi

How can we increase accuracy?

With just one look, we are already doing a fantastic job of categorizing the images. But suppose we have a self driving car and 80% survival is just not the kind of odds we want to leave in the hands of AI. Then we will need a more comprehensive solution.

RNN-Brn

So you’re driving on the road and, out of the corner of your eye, you see a truck plowing toward you. Within a few short instances, your eyes dart to see the massive object coming your way and determine how you can avoid it or if you have time to stop before entering its path.

Photo by Ernesto Leon on Unsplash

In this case, you needed multiple looks to understand the object and make some determination about it. A certain part of your brain (think neural network), that has been trained to identify peripheral threats, gave you a “heads up” that you should look that direction and see what’s going on.

Likewise, an RNN-i will need a processing neural network layered to analyze the output, certainty of classification, and decide below what threshold requires a second or third look at the object. And, lastly, where the new focal center should be relative to the first.

Utilizing a second look at the object may greatly increase accuracy, but only requires an additional 176 points of data. We’re still not even scratching the surface of what a whole 4,096 x 4,096 image view would cost in time and data!

So that is where a Retinal Inspired Neural Network Bidirectional Relay Node (or RNN-Brn for short) comes in. This will allow us to, if necessary, loop back through our trained RNN-i and see if a second look in another part of the image improves the certainty of classification.

I might come back to continue this at a later date with a follow-up article, depending on the how interested people are in this topic. Leave a comment if you have any questions or thoughts you’d like to share.