Board Game Image Recognition using Neural Networks | by Andrew Underwood | Towards Data Science

Utilizing computer vision techniques and convolutional neural networks (CNN), the algorithms created for this project classify chess pieces and identify their location on a chessboard. The final application saves images throughout to visualize the performance and outputs a 2D image of the chessboard to see the results (see below). The purpose of this article is to step through this project so that it can be used as the “foundation” for new iterations. See the code for this project on GitHub.
I had high requirements for this project’s dataset because I knew it would ultimately drive my results. The chess datasets I found online were created with either a different chess set (Chess Vision), a different camera setup (Chess ID Public Data), or both (Raspberry Turk Project), which led me to create my own. I generated a custom dataset using my chess set and camera setup (GoPro Hero6 Black at a “first-person view” angle), which allowed my model to be more accurate. The dataset contained 2,406 images classified into 13 categories (see below). Take-away: This took the majority of my time, but ultimately you want to train your model on images that are as close as possible to the ones it will see when used in your application.
To build this dataset, I first created capture_data.py, which takes a frame from a video stream when the S-key is clicked and saves it to the director. This program allowed me to change the pieces on the chessboard seamlessly and capture an image of the board over and over until I had built up a large number of different board configurations. Next, I created create_data.py to crop the frames into separate pieces by using the board detection techniques discussed in the next section. Finally, I classified the cropped images by separating them into labeled folders. And... Voila!
For chessboard detection, I wanted to do something more involved than using findChessboardCorners (OpenCV), but not as high-level as a CNN. Low and mid-level computer vision techniques were used to find features of the chessboard and then turn those features into coordinates of the outer border and 64 individual squares. The process was centered around implementing Canny edge detection and Hough Transform to generate intersecting horizontal and vertical lines. Hierarchical clustering was used to group the intersections by distance and the groups were averaged to create the final coordinates (see below).
When I started this project, I knew I wanted to use Keras/TensorFlow to create a CNN model and classify the chess pieces. However, after creating my dataset, a CNN alone would not give me the results I wanted given the size of my dataset. To overcome this obstacle, I utilized ImageDataGenerator and transfer learning, which augmented my data and used other pre-trained models as a base.
I created and trained the CNN model in the cloud in order to use a GPU, which significantly decreased the training time. Quick Tip: Google Colab is a quick and easy way to get started using a GPU. To increase my data’s effectiveness, I used ImageDataGenerator to augment my original images and expose my model to different versions of the data. The function ImageDataGenerator randomly rotated, rescaled, and flipped (horizontal) my training data for each epoch, essentially creating more data (see below). While there are many more transformation options, I found that these were the most effective for this project.
from keras.preprocessing.image import ImageDataGeneratordatagen = ImageDataGenerator(        rotation_range=5,        rescale=1./255,        horizontal_flip=True,        fill_mode='nearest')test_datagen = ImageDataGenerator(rescale=1./255)train_gen = datagen.flow_from_directory(    folder + '/train',    target_size = image_size,    batch_size = batch_size,    class_mode = 'categorical',    color_mode = 'rgb',    shuffle=True)test_gen = test_datagen.flow_from_directory(    folder + '/test',    target_size = image_size,    batch_size = batch_size,    class_mode = 'categorical',    color_mode = 'rgb',    shuffle=False)
Instead of training a full-scale model from scratch, I implemented transfer learning by leveraging a pre-trained model and added a top-layer model, trained using my custom dataset. I followed the typical transfer learning workflow:
1. Took layers from a previously trained model (VGG16).
from keras.applications.vgg16 import VGG16model = VGG16(weights='imagenet')model.summary()
2. Froze them to avoid destroying any of the information they contained during training rounds.
3. Added new, trainable layers on top of the frozen layers.
from keras.models import Sequentialfrom keras.layers import Dense, Conv2D, MaxPooling2D, Flattenfrom keras.models import Modelbase_model = VGG16(weights='imagenet', include_top=False, input_shape=(224,224,3))  # Freeze convolutional layers from VGG16for layer in base_model.layers:    layer.trainable = False# Establish new fully connected blockx = base_model.outputx = Flatten()(x) x = Dense(500, activation='relu')(x) x = Dense(500, activation='relu')(x)predictions = Dense(13, activation='softmax')(x)# This is the model we will trainmodel = Model(inputs=base_model.input, outputs=predictions)model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['categorical_accuracy'])
4. Trained the new layers on the custom dataset.
epochs = 10history = model.fit(    train_gen,     epochs=epochs,    verbose = 1,    validation_data=test_gen)model.save_weights('model_VGG16.h5')
While I created models using VGG16 or VGG19 as my pre-trained model, I chose the model using VGG16 due to better validation accuracy. Also, I found that the optimal number of epochs was 10. Any number above ten did not cause an increase in validation accuracy and increased the difference between the train and validation accuracy, which hints at overfitting. Take-away: Transfer learning allowed me to take full advantage of deep learning’s strengths in image classification without requiring a large dataset.
To better visualize the validation accuracy, I created a confusion matrix of the model’s predictions. From this chart, it’s easy to evaluate the model’s strengths and weaknesses. Strengths: Empty — precision was 99% and recall was 100%; White and Black Pawns (WP and BP) — had an F1-score of ~95%. Weaknesses: White Knight (WN) — had high recall (98%) but very low precision (65%); White Bishop (WB) — had the lowest recall at 74%.
The goal for the application was to use the CNN model and visualize the performance at each step. I created cv_chess.py (see excerpt below), which clearly shows the steps, and cv_chess_functions.py, which shows each step’s details. This application saves the original frame from the live video stream, the 64 cropped images for each square, and the final 2D image of the chessboard. I built the application this way to quickly identify areas of strength and weakness for future improvements.
print('Working...')        # Save the frame to be analyzed        cv2.imwrite('frame.jpeg', frame)        # Low-level CV techniques (grayscale & blur)        img, gray_blur = read_img('frame.jpeg')        # Canny algorithm        edges = canny_edge(gray_blur)        # Hough Transform        lines = hough_line(edges)        # Separate the lines into vertical and horizontal lines        h_lines, v_lines = h_v_lines(lines)        # Find and cluster the intersecting        intersection_points = line_intersections(h_lines, v_lines)        points = cluster_points(intersection_points)        # Final coordinates of the board        points = augment_points(points)        # Crop the squares of the board a organize into a sorted list        x_list = write_crop_images(img, points, 0)        img_filename_list = grab_cell_files()        img_filename_list.sort(key=natural_keys)        # Classify each square and output the board in Forsyth-Edwards Notation (FEN)        fen = classify_cells(model, img_filename_list)        # Create and save the board image from the FEN        board = fen_to_image(fen)        # Display the board in ASCII        print(board)# Display and save the chessboard image        board_image = cv2.imread('current_board.png')        cv2.imshow('current board', board_image)        print('Completed!')
I hope you enjoyed the demo! For more of my projects, see my website and GitHub. Please feel free to contact me through LinkedIn with ideas for future iterations of this project. Cheers!