Solving Sudoku with Convolution Neural Network | Keras | by Shiva Verma | Towards Data Science

I used to solve sudoku a long time ago. A few days back I was wondering if I can solve it with Convolution Neural Network(CNN). I knew Sudoku has spatial features since it has a particular arrangement of numbers and CNNs are good at extracting spatial features. Let’s see how this experiment goes.
I found the following data on Kaggle, which contains 1 million unsolved and solved Sudoku games. Please take a look at the data below.
The dataset contains 2 columns. The column quizzes has the unsolved games and the column solutions has respective solved games. Each game is represented by a string of 81 numbers. Following is a 9x9 sudoku converted from the string. The number 0 represents the blank position in unsolved games.
[[0 0 4 3 0 0 2 0 9] [0 0 5 0 0 9 0 0 1] [0 7 0 0 6 0 0 4 3] [0 0 6 0 0 2 0 8 7] [1 9 0 0 0 7 4 0 0] [0 5 0 0 8 3 0 0 0] [6 0 0 0 0 0 1 0 5] [0 0 3 5 0 8 6 9 0] [0 4 2 9 1 0 3 0 0]]
Our task is to feed the unsolved sudoku to a neural network and get the solved sudoku out of it. This means we have to feed 81 numbers to the network and need to have 81 output numbers from it.
We have to convert the input data(unsolved games) into a 3D array since we have to feed it to the CNN. I have converted each string of 81 numbers in a shape of (9,9,1). Then I normalized the input data by dividing it with 9 and subtracting 0.5. By doing so data becomes zero mean-centred and in the range of (-0.5 – 0.5). Neural networks generally perform better with zero centred normalized data.
In a typical multi-class classification, the neural network outputs scores for each class. Then we apply softmax function on the final scores to convert them into probabilities. And the data is classified into a class that has the highest probability value(refer to the following image).
But in sudoku, the scenario is different. We have to get 81 numbers for each position in the sudoku game, not just one. And we have a total of 9 classes for each number because a number can fall in a range of 1 to 9.
To comply with this design, our network should output 81x9 numbers. Where each row represents one of the 81 numbers, and each column represents one of 9 classes. Then we can apply softmax and take the maximum along with each row so that we have 81 numbers classified into one of the 9 classes.
I created the following simple network for this task. The network consists of 3 Convolution layers and one Dense layer on top for classification.
Note I am reshaping the output of the Dense layers in a shape of (81, 9) then adding a softmax layer on it. I compiled the model with sparse_categorical_crossentropy loss and adam optimizer.
Since we are using SCC loss, we don't need to provide a one-hot encoded target vector. Our target vectors shape is(81,1) where the vector elements represent the true class of 81 numbers.
I trained the network for 2 epochs, with batch size 64. The learning rate for the first epoch was 0.001 and for second epochs I reduced it to 0.0001. The final training loss settled down to 0.34. I tried a few different network architecture and strategies but could not reduce the loss further so I went ahead with this network. Its time to test the network.
Now, I tried to solve the game using our trained network. I saw that the network always predict few values wrong. Following is a game predicted by the network. You can see a few numbers repeating in rows and columns.
> Input (Unsolved)[[0 1 6 9 0 4 0 0 7] [0 0 4 0 3 0 0 8 0] [0 0 3 0 6 1 9 2 0] [5 0 9 1 4 0 8 0 0] [1 7 0 0 0 0 0 0 0] [0 0 8 7 0 0 0 6 5] [6 0 0 0 0 2 0 4 0] [0 2 0 8 0 5 3 1 0] [0 3 0 0 0 0 0 0 9]]> Output[[2 1 6 9 8 4 5 3 7] [2 9 4 2 3 7 6 8 1] [7 8 3 5 6 1 9 2 4] [5 6 9 1 4 6 8 7 3] [1 7 2 5 5 8 4 9 4] [4 4 8 7 2 9 1 6 5] [6 9 1 3 1 2 5 4 8] [9 2 7 8 9 5 3 1 6] [4 3 1 4 1 6 2 5 9]]
I had to try something else instead of changing network architecture to solve the game since training loss was not going below a certain number.
As humans when we solve Sudoku, we fill numbers one by one. We do not simply look at the sudoku once and fill all the numbers. The advantage of filling the numbers one by one is that each time we fill a number we keep getting a better idea about the next move.
I implemented the same approach while solving the sudoku now. Instead of predicting all 81 numbers at once, I am picking just one number among all blank position that has the highest probability value, and filling that number in the sudoku. After filling one number we again feed this puzzle to the network and make a prediction. We keep repeating this and filling the blank positions one by one with the highest probability number until we are left with no blank positions.
This approach boosted the performance and the network was able to solve almost all the games in this dataset. Test accuracy on 1000 games was 0.99.
Following is a sudoku game picked from the web, and the model solved it correctly. You can replace the following game string with your sudoku game in my jupyter notebook and check the results.
However I need to check my model on more authentic games picked form the web, since those games can be different from randomly generated ones. It takes a lot of time to manually copy the sudoku in string format, so I have left that part for the future.
Following is the GitHub link to this project with the saved model. I will keep posting more such projects.
github.com
If you enjoyed this article, then you should also check out the following article.