Identifying Nendoroids using Convolutional Neural Networks
By: Clavin Dsouza
Tags: Image Classification, Artificial Intelligence, Nendoroid, Deep Learning, Neural Networks

Nendoroids are a brand of action figures owned by the Good Smile Company. They are usually short in size or chibi-sized and cover a lot of characters from all sorts of mediums such as movies, games TV shows, animations, books, etc. They are very popular among the anime and manga fandom due to most of their figures being based on most popular anime/manga/light novels. They consist of around 30–40 movable parts and some of them are known to be hand-made to this day.

If you want to know more about their production, I recommend watching this 30 min video of how they are made.

In this blog, we’re going to classify pics of Nendoroids with other action figure brands such as Figmas, Gunplas, etc. using deep learning. I’ll try my best to explain these concepts in layman terms. The reason for doing so is that making an image classification project using cats, dogs, fruits doesn’t sound so exciting.

Creating an image dataset

Collecting the images was sort of easy. I downloaded (manually or through a script or software) the Nendoroid and Non-Nendoroid images from Google Images, Pinterest, and online stores that sell these figures and place them into their respective folders (Nendoroid and non-Nendoroids). There’s no set amount for the number of images in a dataset, but the rule of thumb is around a total of 1000 images.

One thing that I had to keep in mind is that the images only contained one or 2 figures and are facing the camera.

For labeling, I renamed the file names in both folder to something like nendo_(insert number) and nonendo_(insert number) using the Python OS library.

A much better way to label the images is using an annotator like Pidgeon (https://github.com/agermanidis/pigeon) thanks to this blog on collecting images for the dataset.

Here are some examples of Nendoroids. They have a very miniature build that is more expressive and their poses can be customized.

Photo by Tengyart on Unsplash

Photo by Daniel Lee on Unsplash

Here are other examples of action figures that are not Nendoroids like Gunplas (owned by Bandai Hobby Center) and Figmas (which are also owned by the Good Smile Company).

Photo by Daily Choice on Unsplash

Photo by Ryan Yao on Unsplash

Components of an image

Before we get into preprocessing the data, we need to understand what an image consists of. A digital image consists of pixels that are arranged in a given length and width.

An example of this would be an image of a resolution of 1920 x 1080. This contains 816,000 pixels, where the pixels are arranged within the length of 1920 pixels horizontally and 1080 pixels vertically.

Every pixel contains a value. For grayscaled images, those values range between 0(black) to 255(white) and each of them represent the different shades of the 2 colors.

What gives the color to an image is the RGB (Red, Green, Blue) channels. These channels consist of their own pixel value from 0 to 255, each representing a different shade of that color channel. The pixel value of that color image is a vector of all the 3-pixel values from each channel.

Preprocessing the images

For preprocessing our data, there are 3 steps to be followed :

Read our image and convert it to greyscale. This is because we are more focused on the structures of the figures and not the color. Resize it to a certain resolution because all the images must have a uniform resolution size. Convert the image to an n-dimensional array and flatten it to a 1-dimensional array.

Do that for both image classes, append the labels and we have our dataset to feed the model with.

os.chdir("Insert file link to dataset") files = os.listdir() data = [] for i in files:



#Read and convert the images to greyscale

ig = cv2.imread(os.path.join(os.getcwd(),i),cv2.IMREAD_GRAYSCALE)



#Resize the image to a uniform dimension

new_ig = cv2.resize(ig,dsize=(img_size,img_size))



#append converted data to label

data.append([new_ig,'Nendoroid'])

Now to split our dataset into training, validation, and testing datasets.

x_train,x_test,y_train,y_test = train_test_split(x,y,test_size=0.2,random_state=42) x_train,x_val,y_train,y_val = train_test_split(x_train,y_train,test_size=0.2,random_state=42)

First thing is to normalize our images by dividing it with the value 255, which is the highest number that represents a shade of color from 0–255. The reason for doing so is that improve our Convolutional Neural Network during classification since the values will be at a range between 0–1.

x_train = np.array(x_train)/255

x_test = np.array(x_test)/255

x_val = np.array(x_val)/255

Next is to resize them back to their image size and send it as input to the Convolutional Neural Network.

x_train = x_train.reshape(-1,img_size,img_size,1)

y_train = np.array(y_train) x_test = x_test.reshape(-1,img_size,img_size,1)

y_test = np.array(y_test) x_val = x_val.reshape(-1,img_size,img_size,1)

y_val = np.array(y_val)

Convolutional Neural Network

To put it simply, the convolutional neural network (CNN) involves breaking down the images into the smallest representation, before it is sent as input to the neural networks. How it breaks down the image? It goes like this.

Step 1: A matrix of a given dimension aka a kernel goes through the image step by step from up to down, left to right, and multiplies. The result is a value that represents that kernel at that position. This is known as a convolution.

Convolution process

How many steps it needs to take is determined by a value called stride.

Step 2: That convoluted matrix then gets broken down even further through a process called pooling, which is similar to convolution but it returns a matrix containing the max or average value of that kernel in the pooling phase.

Pooling process

Step 3: This goes on until it returns the smallest possible image dimensions.

How does CNN find out any patterns and classify the images?

In the convolution layer, there exists filters, which are a bunch of weights as a vector. During the training of the model, it multiplies with the output of the convolution layer and as it goes through all the images, the weights keep on changing. When the model predicts an image, if the image has a very similar pattern to one of the images in the training dataset, those weights return high values.

Building the Convolutional Neural Network

Now it’s time to build the Convolutional Neural Network using Keras via Tensorflow. Start by calling the Sequential() method to stack our layers and then add the convolutional layer (Conv2D) with a given number of nodes and the size of the kernel to use. After that add a max-pooling layer of given kernel size and keep on doing this until the dimension of the images through the layers are near zero or one. Once that’s done, the images are flattened to a 1-dimensional array and it goes through the dense neural network at the last stages.

cnn = Sequential()

cnn.add(Conv2D(32,(3,3),activation='relu',input_shape=(img_size,img_size,1)))

cnn.add(MaxPooling2D((2,2)))

cnn.add(Dropout(0.1))

cnn.add(Conv2D(64,(3,3),activation='relu'))

cnn.add(MaxPooling2D((2,2)))

cnn.add(Dropout(0.1))

cnn.add(Conv2D(64,(3,3),activation='relu'))

cnn.add(MaxPooling2D((2,2)))

cnn.add(Dropout(0.1))

cnn.add(Conv2D(64,(3,3),activation='relu'))

cnn.add(MaxPooling2D((2,2)))

cnn.add(Dropout(0.1))

cnn.add(Conv2D(128,(3,3),activation='relu'))

cnn.add(MaxPooling2D((2,2)))

cnn.add(Dropout(0.1))

cnn.add(Conv2D(128,(3,3),activation='relu'))

cnn.add(MaxPooling2D((2,2)))

cnn.add(Dropout(0.1))

cnn.add(Flatten())

cnn.add(Dense(128,activation='relu'))

cnn.add(Dropout(0.1))

cnn.add(Dense(1,activation='sigmoid'))

Activation functions are what convert the output of one node to be sent as input to the corresponding node. Except for the last layer, the ReLu activation function is used since it takes fewer resources and it doesn’t fall victim to the ‘vanishing gradient’ problem, unlike the Sigmoid function.

The last layer has ‘Sigmoid’ as the activation function and it outputs a value between 0 to 1.

Dropouts are added to lessen the chances of overfitting the model. Here’s a visual representation of the Dropout process.

Meme representation of Dropouts in Neural Networks made by me

For compiling the neural network, the optimizer (which changes the weights and biases in the Neural Network at a given learning rate) chosen is the RMSprop, the loss function is binary cross-entropy because we have only 2 classes to classify from. The Convolutional Neural Network is trained for about 35 rounds or epochs, with the validation data used to make sure that it isn’t ‘memorizing’ anything and can classify unseen data well.

cnn.compile(optimizer='rmsprop',loss='binary_crossentropy',metrics=['accuracy']) history = cnn.fit(x_train,y_train,epochs=35,validation_data=(x_val,y_val))

Comparing the accuracy values of both the training and validation data for 20 epochs and the test data accuracy, there seems to be overfitting. This can be changed through trial and error by changing the learning rate, dropout value, removing layers, etc. but I’ll stop it from here.

Comparing between training and validation data accuracy

Looking at the confusion matrix for the test data, the model did its job of classifying the test data quite well, albeit some false negatives.

Confusion matrix of test dataset

The moment of truth

Now for the moment of truth. It’s time to use our classifier against images outside of the dataset we used. What’s a classifier if it doesn’t classify anything outside of the used dataset ?. There are 40 images (20 Nendoroid and non-Nendoroid images) in this set and we’re going to see how well it was able to classify our images.

Like before, we preprocess the images and then use the model.predict_classes() function to predict the class for each image.

Looking at the accuracy and the confusion matrix, there are a lot less false positives and false negatives found, so I’d say it did its job.

Confusion matrix of images in the Test data folder

Here are some of the images classified correctly and not correctly classified (0 indicates it’s not a Nendoroid and 1 indicates it is a Nendoroid).

Images correctly classified as non-Nendoroids

Images correctly classified as a Nendoroid

Images incorrectly classified as either Nendoroids or non-Nendoroids

Conclusion

In short, we were able to make a classifier that was able to differentiate and classify between Nendoroids and Non-Nendoroid images using a Convolutional Neural Network. Even though this could have been done better with some slight tuning and modifications, we now know that it was effective in doing so.

Overall, this was a fun experiment to learn about image preprocessing and classification as well as the inner workings of the Convolutional Neural Network. Here’s a reward for going through all of this.

If you like the Nendoroids, Gunplas, and figmas in this article and want to buy for yourself, you can go to their official sites to buy them.

If you like this article, please share it with others. I’ll be grateful if you do so. Feel free to give feedback about this and the notebook for this project will be on my Github page.