Comparison of Explainable AI for Image Classification to Human Perception: A Case Study of Threaded Fasteners
Authors: Gill Amaninder Singh , Agarwal Ankit , Tummala Vijayanth , Lee Seung-Jin , Mears Laine 
Published: January 2025

Machine vision in quality control and sorting applications enhance organizational efficiency. Machine vision systems are coupled with a pre-trained Convolutional Neural Network (CNN) to enhance the capability of the system for classification and identification tasks. The overarching research goal of this study is a) to understand how a CNN decides on classifying threaded fasteners, and b) how well does the CNN’s decision making compare with that of a human. In order to answer the first research question, an image-based fastener identification model augmented with a pre-trained CNN was deployed. The CNN used is called Efficient-Net-b0, that can perform a wide range of image classification tasks. The training set provided to the Efficient-Net-b0 model consisted of labeled images of 12 types of threaded fasteners. The data set was enlarged by using image augmentation techniques such as varying the brightness, contrast, and orientation of the captured images. The results produced by the CNN classifier were then parsed through Gradient-weighted Class Activation Mapping (Grad-CAM). This technique produce visual explanations of the decisions made by CNNs. This is the XAI component of this research. It provides transparency in the reasons for the identification and classification done by the Efficient-Net-b0, thereby providing context for the key feature of the threaded fastener that was used to classify and identify it.

In order to answer the second research question, a user study was conducted. The participants of this study are novice and experienced mechanical engineers enrolled in a Bachelor’s and a Master’s program at two universities in the United States. The aim of this study was to answer three research sub-questions, each of which was compared to the results from the Efficient-Net-b0 as explained by Grad-CAM. The three questions are: i) Can human subjects distinguish between fasteners within the same category?, ii) What features do human subjects look at when distinguishing between fasteners in the same category as compared to XAI?, and iii) Can human subjects identify the same fastener when it is placed in different orientations? The study was conducted via an online Google form that shows the participant two pictures of threaded fasteners simultaneously. There is a total of 40 pairs of images. Each set of pictures comes with two questions. The first question is a multiple-choice question that asks the user whether the image they are looking at represents the same component. The second question is a short answer question asking the user to describe their choice for the multiple-choice question. Each user answers a total of 80 questions, and the total number of participants 43, resulting in a sizable dataset. In addition to providing a comparison between the accuracy rates of both human and AI in classifying fastners the results of the experiments also gave us a better insights into the features/categories utilized by both XAI and participants to achieve the goal of fastner classification.