Computer Vision — Auto grading Handwritten Mathematical Answersheets | by Divyaprabha M | Towards Data Science

Grading is an essential part of education. Assessing each answer sheet manually, offering fair, unbiased and valid grade is difficult most of the time. This article is about my internship project with my mentor Bijon Guha on building a computer vision model that will automatically evaluate the answer sheets thereby ensuring that the grades are based solely on the student’s performance.
Below are the sample worksheets we are going to examine and grade.
Each of these worksheets was written by different people. There will be variations in line width, character style in the same page, pen nib width and character spacing, etc.
The idea is to correct each line in the worksheet and mark the lines with boxes. Where the green box represents that the line is correct and the red box represents the line is in-correct.
( Note: I have not added all the code here, if you want to check you can visit my GitHub where I have the tutorial in a ipynb notebook )
There are two modules in the workflow Workspace detection module and Analysis Module. Workspace Detection module is responsible for detecting multiple work spaces in a given sheet of paper.
Analysis module is responsible for detecting and localizing characters in lines in any given single workspace, and mathematically analyzing them and then drawing red, green boxes depending upon their correctness.
Workspace detection module assumes that there are valid rectangular boxes in the given scanned worksheet. This image below shows the worksheet design. Three largest rectangular boxes in this worksheet are the work-spaces.
Workspace detection module is done using openCV. We will first find the rectangular boxes, then sort them based on their positions in the worksheet. Since there are many rectangles in the worksheet, we will have to select the valid work-spaces among the other rectangles. Let’s see how each step is done
Rectangles are formed by two horizontal and vertical lines. So the first step is to find all the horizontal and vertical lines ignoring digits, symbols or anything that is written on the worksheet.
This code below will first create a binary image called “vertical_lines_img” which contains all the vertical lines that are present in the worksheet, then another binary image called “horizontal_lines_img” which contains all horizontal lines that are present in the worksheet.
Next, we have to add image “vertical_lines_img” with “horizontal_lines_img” to get the final image.
Contours are defined as the line joining all the points along the boundary of an image that are having the same intensity.
OpenCV has findContour() function that helps in extracting the contours from the image. Each individual contour is a Numpy array of (x,y) coordinates of boundary points of the object. We can use that to find all the objects in the final image (Only objects in the final image are the rectangles).
Since final image is just the binary version of the original image coordinates of the rectangles in the final image is equal to the coordinates of the rectangles in the original image.
Now we know the coordinates lets draw them on the original image using openCV’s drawContours() function.
Now we have found all the rectangles, its time to sort them top-to-bottom based on their coordinates. This code below will do that for us.
sort_contours function will return contours and bounding boxes(top-left and bottom-right coordinates) sorted in the method we have given. In this case method is top to bottom.
There are many rectangles, but we only need the three largest ones. How can we select the three largest rectangles?.... One answer is to find the area of the rectangles, then choose the top 3 rectangles which have the maximum area.
These selected rectangles are the work-spaces that are then extracted from the worksheet and sent to the Analysis Module.
Analysis module as explained above will first detect the lines, predict the characters in each line and finally forms an equation with the predicted characters then evaluate them by marking boxes.
Detecting the lines is the tricky part, everyone has their way of solving equations some solve step by step, some can solve in just one line, some might write steps for pages and some writes exponents way away from the equation confusing the module to treat those exponents as a separate line.
Our line detection module assumes that there is a sufficient gap between lines and there is some intersection between exponential characters and line. First, the detected work-spaces are converted to binary images then compressed in a single array to take the forward derivative. Wherever there is a line there will be a change in the derivative.
The above code is just a glimpse of how line extraction works. To see complete code click extract_line.
After detecting all the lines we have to send the extracted line images to the text_segment function which will use openCV’s find contours to segment the characters and sort them using the function sort_contours described above where method is now set to left-to-right.
It’s easy for us to say whether the given number is an exponent or not but for the model it’s not that easy. Assuming that the exponents are at-least above half of the line, we can drew a baseline at the center of the image any character which is above the baseline is considered as an exponent.
We can use MNIST dataset for digits (28*28 pixels) and Kaggle’s Handwritten Mathematical symbols dataset for symbols(45*45 pixels) to train the model.
How MNIST is actually created?
Handwritten digits of 128 * 128 pixels collected from 500 different writers.A Gaussian filter is applied to the image to soften the edgesThe digit is then placed and centered into a square image by preserving the aspect ratio.The image is then down-sampled to 28 × 28 pixels using bi-cubic interpolation
Handwritten digits of 128 * 128 pixels collected from 500 different writers.
A Gaussian filter is applied to the image to soften the edges
The digit is then placed and centered into a square image by preserving the aspect ratio.
The image is then down-sampled to 28 × 28 pixels using bi-cubic interpolation
Images of symbols are preprocessed in the same way as MNIST digits before training. The reason for preprocessing is that the two data-sets we have chosen have different characteristics like dimensions, thickness and line width this makes hard for the deep learning model to find the patterns. Preprocessing helps us reduce the variations among digits and symbols.
Almost 60,000 images of digits and preprocessed symbols were trained on Deep Columnar Convolutional Neural Network (DCCNN) a single deep and wide neural network architecture that offers near state-of-the-art performance like ensemble models on various image classification challenges, such as MNIST, CIFAR-10, and CIFAR-100 datasets. This model achieved atmost 96 % accuracy.
To see training code click DCCNN_training.ipynb
Evaluation is the last and the most important part. To solve any equation we can use python’s eval method.
The eval method parses the expression passed to it and runs python expression(code) within the program
This is an example of how eval works
>>Enter the function y(in terms of x):‘x*(x+1)*(x+2)’>>Enter the value of x:3>>print(y)60
Steps involved in evaluation process are,
Solve the given math question and save the answerSolve each handwritten lines and compare its derived value with the answer stored.Draw a green bounding box if the line correct and red if the line is wrong.
Solve the given math question and save the answer
Solve each handwritten lines and compare its derived value with the answer stored.
Draw a green bounding box if the line correct and red if the line is wrong.
Let’s take an example, say the question is to solve the equation A*x2 + B*y
where, A=56, B=7, x=3 and y=13 and the answer to this equation is 595 (56*32 +7*13 = 595).
A green box indicates the line is correct, whereas red indicates the line is wrong.
The first and the last line is correct, on solving these lines we get 595 which matches with the actual answer.
The second line (56*7 + 92) is wrong. 32 is 9 but it’s written as 7 and on solving we get 584 which is not equal to 595
The third line (595 + 92) is also wrong, on solving this line we get 684 which again is not equal to 595.
Let’s summarize, the scanned worksheet is sent to the work-spaces detection module it will return all the rectangular work-spaces in the given worksheet, then the detected work-spaces are passed to line extraction module to extract all the lines. The extracted lines are then sent to character segmentation module it will segment the character and the deep learning model DCCNN will predict the digit/symbol. Finally, evaluation module will assess the line and draw red/green bounding box.
Automating grading process not only helps the teachers and also creates a comfortable learning experience for the students. This solution can be made even cooler by recognizing more complex mathematical equations like differential integral equations, recognition of cursive handwriting where the character is not separated, detecting plagiarism and recognizing chemical equations.
Thanks for reading till the end !!... If you have enjoyed this post let me know by clapping and I’d be very grateful if you’d help it spread by sharing with your friends :). ✌️