Rorschach Tests for Deep Learning Image Classifiers | by Mathieu Lemay | Towards Data Science

What do they see, when there is nothing to see?
With a lot of exploration in image classification and labeling, we tend to play with various libraries to quickly assess and label pictures. But other than memory use and ease of retraining layers, how do they behave when presented with unclear, grainy pictures?
In order to get a better feel for the nuances and trends of each ConvNet at a practical level, I ran the most popular ones, in series, on all of the cards from the Rorschach Test.
The goal was simple: set my expectations and clear up my understanding for each one, so I can use the right tool for the job moving forward.
Invented (or more accurately, modified) by Hermann Rorschach, the Rorschach Test is the most famous of the inkblot tests. An inkblot test is a simple conversation starter with your therapist, that goes “What do you see?” after being shown an intentionally abstract (and usually symmetrical) drop of ink squeezed in a sheet of paper. Rorschach took it one step further, and created a standard set. This standard set could now we used between patients and therapists.
The goal of this test is to evoke a subconscious reaction to the abstract art, overlaying it with your own biases and deepest thoughts, so that whatever you see first is what you’re thinking about. Although it has fallen out of favor in the last few years (“pseudo-science” has been used to describe it), it’s still a very useful trope to describe the field of psychology.
In this case, I’ve used it to force various libraries to classify this innocuous dataset into pre-defined labels.
In order to classify the pictures, I looked at the following libraries:
ResNet50
VGG16
VGG19
InceptionV3
InceptionResNetV2
Xception
MobileNet
MobileNetV2
DenseNet
NASNet
(These are all of the native application libraries available with Keras 2.2.0.)
Each one of these classifiers has a particular structure and weights associated with it.
Besides for the memory usage and trainable parameters, there is a lot of differences in the details of the implementation of each one. Instead of digging in the particularities of each structure, let’s look at how they view the same confusing data.
(If you want to learn more about each structure individually, here are the links if you want to learn more about ResNet, Inception, DenseNet, NASNet, VGG16/19, and MobileNet.)
Overall, the goal is to get a quick sense of the prediction and intensity of the belief behind the prediction. In order to do this, I’ve grouped the Top-1 predictions by name, and added their score together. This gives a sense of how much each classifier is ready to gamble on a particular card. Doing this for each label gives us a good proxy for the belief of each classifier, and gives us a sense of their relative prediction confidence for each card.
As an example, InceptionResNetV2, NASNetLarge, and DensetNet201 believed that Card 1 was a warplane (with scores of 88.3%, 46.2%, 18.6%, respectively). I’ve then added it up to a dimensionless score of 153.1. Now, I can compare this score in between the classifiers to see which one performs the best.
Card 1:warplane 153.05wall_clock 75.41letter_opener 47.29lampshade 33.13buckeye 28.95jigsaw_puzzle 23.19paper_towel 22.51birdhouse 11.50Card 2:cock 72.84        # <--- Rooster.  They mean rooster.lampshade 59.99brassiere 59.47velvet 43.84earthstar 41.42mask 29.46candle 21.84tray 19.30wall_clock 18.41hair_slide 17.84vase 11.44Card 3:book_jacket 325.35  # <--- No idea what's going on here.coffee_mug 62.61chocolate_sauce 45.00candle 32.68ant 25.81matchstick 24.02jersey 16.94
Cards 4 through 7 elicit a bit more smoke, spaceship, and bugs for me. Which brings me back to my rural childhood, and playing with the best LEGO sets of all time. (For which I obviously mean M-Tron, the evil BlackTron, and Ice Planet.)
Card 4:volcano 101.76fountain 72.11space_shuttle 32.72hen-of-the-woods 29.40pitcher 28.28vase 25.00king_crab 23.99wall_clock 18.25triumphal_arch 11.04Card 5:            # <--- Bugs. Definitely bugs.isopod 106.54king_crab 83.67ant 61.58long-horned_beetle 32.23tick 30.05hip 26.32rhinoceros_beetle 14.52Card 6:space_shuttle 174.48warplane 155.58conch 104.73missile 63.71airship 57.73fountain 11.57Card 7:missile 195.66parachute 52.52projectile 42.31space_shuttle 31.18hip 29.89geyser 20.92warplane 17.50
Card 8:fountain 189.59parachute 98.27umbrella 94.61pencil_sharpener 63.27spoonbill 51.08poncho 45.19coral_fungus 12.05shovel 10.12Card 9:missile 238.45fountain 64.82parachute 48.21volcano 44.77paper_towel 41.59lampshade 13.48Card 10:tray 229.22handkerchief 151.63conch 77.94feather_boa 60.34rapeseed 15.95
Of course, we can go a lot deeper to see the average confidence of each one of the predictions. Below is some sample data for Card 1. All of the results are available on the project’s GitHub page.
In order to label each picture, we start off by loading Pandas, NumPy, and the Keras image pre-processing libraries:
from keras.preprocessing import imagefrom keras.preprocessing.image import load_imgfrom keras.preprocessing.image import img_to_arrayfrom keras.models import Modelimport keras.backend as Kimport numpy as npimport pandas as pdimport json
We then create a helper function to return a dataframe containing the score of the top 10 results for each library, in order to quickly assemble each one of the images’ score:
def getLabels(model, dims, pi, dp):"""Returns the top 10 labels, given a model, image dimensions, preprocess_input() function, and the labels inside decode_predictions()."""df = pd.DataFrame()        for img_path in images:# Resize image array        image = load_img(img_path, target_size=(dims, dims))        image = img_to_array(image)        image = image.reshape((1, image.shape[0], image.shape[1], image.shape[2]))        image = pi(image)                # Predict what is in the image        yhat = model.predict(image)                # return the top 10 labels        labels = dp(yhat, top=10)[0]# create empty list and counter        labellist = []        counter = 0                # get labels for each image        for label in labels:            # Display the score of the label            print('%s (%.2f%%)' % (label[1], label[2]*100))                        # Add image results to list            labellist.append(                {"labelnumber":counter,                  "labelname"  :label[1],                  "pct"        :'%.2f%%' % (label[2]*100),                  "image"      :img_path})            counter = counter + 1                    # Add to dataframe        df_row = pd.Series()        df_row = pd.read_json(json.dumps(labellist), typ='frame', orient='records')        df = df.append(df_row,ignore_index=True)        print("------------")        return df
Now, we have a method for evaluating each library very quickly.
# ...# ResNet50 Rorschach assessmentfrom keras.applications.resnet50 import ResNet50from keras.applications.resnet50 import preprocess_inputfrom keras.applications.resnet50 import decode_predictionsmodel = ResNet50()df = getLabels(model, 224, preprocess_input, decode_predictions)df.to_csv('results_resnet50.csv', index=False)# ----------# VGG16 Rorschach assessmentfrom keras.applications.vgg16 import VGG16from keras.applications.vgg16 import preprocess_inputfrom keras.applications.vgg16 import decode_predictionsmodel = VGG16()df = getLabels(model, 224, preprocess_input, decode_predictions)df.to_csv('results_vgg16.csv', index=False)# ...
...and so forth.
What’s fun about this approach is that it makes for a very functional notebook when playing with different images sets:
This notebook structure allows you to quickly explore the different data sets, and see what’s the best classifiers for your product or project. Are you looking for a reliable top-1 accuracy on a fuzzy picture? What about identifying all of the different contents of a picture? your mileage may vary, but you can now start exploring on your own.
You can get the full code here.
-matt.matt@lemay.ai
Lemay.ai1 (855) LEMAY-AI
Other articles you may enjoy:
Hubris, laziness and playing with toys: the winning attitudes of a startup engineer
AI Prototypes for Clients as a Marketing Strategy
Other articles you may enjoy from Daniel Shapiro, my CTO:
Artificial Intelligence and Bad Data
Artificial Intelligence: Hyperparameters
Artificial Intelligence: Get your users to label your data
P.S. It’s pronounced roar-shawk.