Handwritten Equation Solver in Python

26 Jun, 2019
Acquiring Training Data
Downloading DatasetDownload the dataset from this link. Extract the zip file. There will be different folders containing images for different maths symbol. For simplicity, use 0–9 digits, +, ?-?and, times images in our equation solver. On observing the dataset, we can see that it is biased for some of the digits/symbols, as it contains 12000 images for some symbol and 3000 images for others. To remove this bias, reduce the number of images in each folder to approx. 4000.
Download the dataset from this link. Extract the zip file. There will be different folders containing images for different maths symbol. For simplicity, use 0–9 digits, +, ?-?and, times images in our equation solver. On observing the dataset, we can see that it is biased for some of the digits/symbols, as it contains 12000 images for some symbol and 3000 images for others. To remove this bias, reduce the number of images in each folder to approx. 4000.
Extracting FeaturesWe can use contour extraction to obtain features.Invert the image and then convert it to a binary image because contour extraction gives the best result when the object is white, and surrounding is black.To find contours use ‘findContour’ function. For features, obtain the bounding rectangle of contour using ‘boundingRect’ function (Bounding rectangle is the smallest horizontal rectangle enclosing the entire contour).Since each image in our dataset contains only one symbol/digit, we only need the bounding rectangle of maximum size. For this purpose, we calculate the area of the bounding rectangle of each contour and select the rectangle with maximum area.Now, resize the maximum area bounding rectangle to 28 by 28. Reshape it to 784 by 1. So there will be now 784-pixel values or features. Now, give the corresponding label to it (For e.g., for 0–9 images same label as their digit, for – assign label 10, for + assign label 11, for times assign label 12). So now our dataset contains 784 features column and one label column. After extracting features, save the data to a CSV file.
We can use contour extraction to obtain features.Invert the image and then convert it to a binary image because contour extraction gives the best result when the object is white, and surrounding is black.To find contours use ‘findContour’ function. For features, obtain the bounding rectangle of contour using ‘boundingRect’ function (Bounding rectangle is the smallest horizontal rectangle enclosing the entire contour).Since each image in our dataset contains only one symbol/digit, we only need the bounding rectangle of maximum size. For this purpose, we calculate the area of the bounding rectangle of each contour and select the rectangle with maximum area.Now, resize the maximum area bounding rectangle to 28 by 28. Reshape it to 784 by 1. So there will be now 784-pixel values or features. Now, give the corresponding label to it (For e.g., for 0–9 images same label as their digit, for – assign label 10, for + assign label 11, for times assign label 12). So now our dataset contains 784 features column and one label column. After extracting features, save the data to a CSV file.
Invert the image and then convert it to a binary image because contour extraction gives the best result when the object is white, and surrounding is black.
To find contours use ‘findContour’ function. For features, obtain the bounding rectangle of contour using ‘boundingRect’ function (Bounding rectangle is the smallest horizontal rectangle enclosing the entire contour).
Since each image in our dataset contains only one symbol/digit, we only need the bounding rectangle of maximum size. For this purpose, we calculate the area of the bounding rectangle of each contour and select the rectangle with maximum area.
Now, resize the maximum area bounding rectangle to 28 by 28. Reshape it to 784 by 1. So there will be now 784-pixel values or features. Now, give the corresponding label to it (For e.g., for 0–9 images same label as their digit, for – assign label 10, for + assign label 11, for times assign label 12). So now our dataset contains 784 features column and one label column. After extracting features, save the data to a CSV file.
Training Data using Convolutional Neural Network
Since convolutional neural network works on two-dimensional data and our dataset is in the form of 785 by 1. Therefore, we need to reshape it. Firstly, assign the labels column in our dataset to variable y_train. Then drop the labels column from the dataset and then reshape the dataset to 28 by 28. Now, our dataset is ready for CNN.
Building Convolutional Neural NetworkFor making CNN, import all the necessary libraries.import pandas as pdimport numpy as npimport picklenp.random.seed(1212)import kerasfrom keras.models import Modelfrom keras.layers import * from keras import optimizersfrom keras.layers import Input, Densefrom keras.models import Sequentialfrom keras.layers import Densefrom keras.layers import Dropoutfrom keras.layers import Flattenfrom keras.layers.convolutional import Conv2Dfrom keras.layers.convolutional import MaxPooling2Dfrom keras.utils import np_utilsfrom keras import backend as KK.set_image_dim_ordering('th')from keras.utils.np_utils import to_categoricalfrom keras.models import model_from_jsonConvert the y_train data to categorical data using ‘to_categorical’ function. For making model, use the following line of code.model = Sequential()model.add(Conv2D(30, (5, 5), input_shape =(1, 28, 28), activation ='relu'))model.add(MaxPooling2D(pool_size =(2, 2)))model.add(Conv2D(15, (3, 3), activation ='relu'))model.add(MaxPooling2D(pool_size =(2, 2)))model.add(Dropout(0.2))model.add(Flatten())model.add(Dense(128, activation ='relu'))model.add(Dense(50, activation ='relu'))model.add(Dense(13, activation ='softmax'))# Compile modelmodel.compile(loss ='categorical_crossentropy',               optimizer ='adam', metrics =['accuracy'])
For making CNN, import all the necessary libraries.
import pandas as pdimport numpy as npimport picklenp.random.seed(1212)import kerasfrom keras.models import Modelfrom keras.layers import * from keras import optimizersfrom keras.layers import Input, Densefrom keras.models import Sequentialfrom keras.layers import Densefrom keras.layers import Dropoutfrom keras.layers import Flattenfrom keras.layers.convolutional import Conv2Dfrom keras.layers.convolutional import MaxPooling2Dfrom keras.utils import np_utilsfrom keras import backend as KK.set_image_dim_ordering('th')from keras.utils.np_utils import to_categoricalfrom keras.models import model_from_json
Convert the y_train data to categorical data using ‘to_categorical’ function. For making model, use the following line of code.
model = Sequential()model.add(Conv2D(30, (5, 5), input_shape =(1, 28, 28), activation ='relu'))model.add(MaxPooling2D(pool_size =(2, 2)))model.add(Conv2D(15, (3, 3), activation ='relu'))model.add(MaxPooling2D(pool_size =(2, 2)))model.add(Dropout(0.2))model.add(Flatten())model.add(Dense(128, activation ='relu'))model.add(Dense(50, activation ='relu'))model.add(Dense(13, activation ='softmax'))# Compile modelmodel.compile(loss ='categorical_crossentropy',               optimizer ='adam', metrics =['accuracy'])
Fitting Model to DataFor fitting CNN to data use the following lines of code.model.fit(np.array(l), cat, epochs = 10, batch_size = 200,           shuffle = True, verbose = 1)It will take around three hours to train our model with an accuracy of 98.46%. After training, we can save our model as json file for future use, So that we don’t have to train our model and wait for three hours every time. To save our model, we can use the following line of codes.model_json = model.to_json()with open("model_final.json", "w") as json_file:    json_file.write(model_json)# serialize weights to HDF5model.save_weights("model_final.h5")
For fitting CNN to data use the following lines of code.
model.fit(np.array(l), cat, epochs = 10, batch_size = 200,           shuffle = True, verbose = 1)
It will take around three hours to train our model with an accuracy of 98.46%. After training, we can save our model as json file for future use, So that we don’t have to train our model and wait for three hours every time. To save our model, we can use the following line of codes.
model_json = model.to_json()with open("model_final.json", "w") as json_file:    json_file.write(model_json)# serialize weights to HDF5model.save_weights("model_final.h5")
Testing our Model or Solving Equation using it
Firstly, import our saved model using the following line of codes.
json_file = open('model_final.json', 'r')loaded_model_json = json_file.read()json_file.close()loaded_model = model_from_json(loaded_model_json)# load weights into new modelloaded_model.load_weights("model_final.h5")
Now, input an image containing a handwritten equation. Convert the image to a binary image and then invert the image(if digits/symbols are in black).
Now obtain contours of the image, by default, it will obtain contours from left to right.
Obtain bounding rectangle for each contour.
Sometimes, it will result in two or more contours for the same digit/symbol. To avoid that, check if the bounding rectangle of those two contours overlaps or not. If they overlap, then discard the smaller rectangle.
Now, resize all the remaining bounding rectangle to 28 by 28.
Using the model, predict the corresponding digit/symbol for each bounding rectangle and store it in a string.
After that use ‘eval’ function on the string to solve the equation.
Download the full code for Handwritten equation solver from here.My Personal Notes
arrow_drop_upSave
Articles
Machine Learning
Project
Python
Python Programs
Machine Learning
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.