A Simple Plagiarism Rate Checker in Python and Golang
By: Akinola Ayodeji
Tags: Programming, Python, Computer Science, Rabin Karp, Algorithms

A Simple Plagiarism Rate Checker in Python and Golang

Using The Rabin-Karp String Matching Algorithm

Plagiarism checker is the process of locating instances of plagiarism in a document. According to the Oxford Dictionary of English, plagiarism is “the practice of taking someone else’s work or ideas and passing them off as one’s own” without proper citations. The widespread use of computers and the advent of the Internet have made it easier to plagiarize the work of others and get away with it, but with the help of a computer-aided plagiarism detector, we can determine the plagiarism level of two documents using the Rabin Karp algorithm.

What is Rabin Karp?

Rabin Karp algorithm is a string matching algorithm and this algorithm makes use of hash functions and the rolling hash technique, A hash function is essentially a function that map data of arbitrary size to a value of fixed size while A rolling hash allows an algorithm to calculate a hash value without having to rehash the entire string, this allows a new hash to be computed very quickly. We need to take note of two sets of data which are pattern and text, the pattern represents the string we are trying to match in the text.

The Rabin-Karp algorithm is an improved version of the brute force approach which compares each character of pattern with each character of text and has a time complexity of O(mn)

Major features

uses a hashing function techniques and rolling hash;

preprocessing phase in O (m) time complexity and constant space;

(m) time complexity and constant space; searching phase in O (nm) time complexity;

(nm) time complexity; O(n+m) expected running time.

How it works

Given a pattern “cc” and a text “abdcceag”, I will demonstrate how the Rabin Karp algorithm works

Figure 1

As we iterate through the text we will generate the hash value for the current window which is of length pattern and compare it with the hash value of the pattern, if they don’t match, we move to the next window to generate the next hash value using the rolling hash technique. NB: subsequent hashes are computed from the previous hash, E.G “BD” is computed from “AB” E.T.C.

and finally, we found a match in the 4th window. So how can this simple technique be used in plagiarism detection in documents? well, I found two interesting research papers that did justice to that and this article further explains that with code implementations.

Research paper 1: https://osf.io/8ju3h/download/ Research paper 2: https://osf.io/yxjnp/download

Hash Functions

The hash function is a function that determines the feature value of a particular syllable fraction or word in the document. It converts each string into a number, called a hash value. Rabin-Karp algorithm determines hash value based on the same word K-Gram.

K-grams are k-length subsequences of a string, it can be 1,2,3,4 E.T.C. These are mostly used for spelling correction, but in this case, we are using k-gram to determine plagiarism pattern between two documents. K-gram is going to replace our pattern length in figure 1. A good hash function is important in order to prevent multiple words from having the same hash value which is popularly known as spurious hit and a bad hashing will invalidate our solution in the long run.

We will be using a modular arithmetic hash function with a technique called rolling hash where the input is hashed in a window that moves or iterate through the input in this article.

Modular arithmetic hash function:

We compute our string using a modular arithmetic hash function using the formula below

Where:

Given that every character has ASCII code that maps to it E.G A => 1, B => 2

A => 1, B => 2 c1, c2, c3 ….. ck represents the ASCII code of each character in a string

….. represents the ASCII code of each character in a string m represents the k-gram or number of characters there are in the string we are comparing

represents the k-gram or number of characters there are in the string we are comparing where b is a constant

is a constant where Q is modulo.

Example:

Given a pattern “cc” and a text “abdcceag”, let us find the hash of pattern “cc”

Where: b = 26, m = 2, Q = 5807 H("cc")= ((3×26**1 ) % 5807 + (3×26**0) ) % 5807

H("cc") = 81

Rolling Hash

As discussed earlier, the rolling hash allows us to compute a new hash rapidly using the previous hash. We will compute our new hash using the formula below:

rolling hash formula

Example:

Given the text “abdcceag”, let’s find the hash of window “ab” and “bd” using the Modular arithmetic hash function and rolling hash technique.

Where: b = 26, m = 2, Q = 5807 H(“ab”)= ((1×26**1 ) % 5807 + (2×26**0) ) % 5807

H(“ab”) = 28 H(“bd”) = ((H(“ab”) — H(“a”) )* b + h(“d”)) % Q H(“bd”) = ((28-26) * 26 + 4) % 5807 H(“bd”) = 56

Solution

We understand how Rabin Karp works, now it’s time to move on to the actual implementation, this process is going to involve four(4) different steps which include:

step 1: Tokenizing

Tokenizing is essentially splitting a phrase, sentence, paragraph, or an entire text document into smaller units, such as individual words or terms. Each of these smaller units are called tokens.

step 2: Stopword Removal

The process of removing basic words that always exist in the document such as: because, with, and, or, not, and others.

step 3: Stemming

Stemming is the process of reducing a word to its word stem that affixes to suffixes and prefixes or to the roots of words

known as a lemma in NLP. E.g car, cars, car’s, cars’ has the stem car

Code Implementation

Python

The prepare_content() handles stemming, tokenizing, and stopword removal, using natural language tool kit library (nltk)

Golang

Golang is a new programming language that has less support for natural language processing but I found some interesting libraries that implements stemming and tokenization pretty well. Check the code below but please endeavor to check the github repo for better context

step 4: Hashing

This step generates the hash value for the content in our document using the Rabin Karp algorithm with a specified K-gram value that determines the length of the string we need for each window after it has undergone the 3 steps listed above.

Implementation of Rabin Karp algorithm

Using the rolling hash technique and modular hash function

Python

Golang

Calculate document hash

Python

The below code makes use of the above-implemented Rabin Karp algorithm to calculate and generate the hash for our document content. Take note of calculate_hash() function and k_gram value, in this case, we are using 5 as our k-gram

Golang

Take note of the CalculateHash() function and KGram value

Rate Calculation

We compute Plagiarism Rate using the formula below

Plagiarism Rate formula

Where:

P is the Plagiarism Rate

is the Plagiarism Rate SH is the Identical Hash

is the Identical Hash THA is the Total Hash in Document A

is the Total Hash in Document A THB is the Total Hash in Document B

Final Code (Python)

Take note of the calculate_plagiarism_rate() function, we computed this function using the formula given above and we calculated our identical hash using numpy intersect1d() function

Final code (Golang)

Take note of the CalculatePlagiarismRate() function.

NB: “The length of K-Gram is the determinant of the plagiarism level. Determining the proper length of K-Gram, improves our plagiarism level results. The results will vary foreach K-Gram value.”

Demo

Given document A

…and document B

….after the computation of both documents in python, The percentage of plagiarism held by both documents is 50.76142131979695% while in golang i got 53.731343%.

Hash values and percentage calculation

Time Complexity

The Rabin-Karp algorithm has the complexity of O(nm) for the worst case, O(n + m) for the best case where n is the length of the text, while m is the length of the pattern. So where is it compared to brute-force matching? Well, brute force matching complexity is O(nm), so as it seems there’s not much of a gain in performance. However, it’s considered that Rabin-Karp’s complexity is O(n+m) in practice with an efficient hash function and that makes it a bit faster.

Conclusion

Detecting the rate of plagiarism in a document is dependent on the value of k-gram and the calculation of hash value greatly affects the result of this algorithm. Higher K-Gram values tend to have a low level of similarity, while lower K-Gram values increase the percentage of similarity, and also , agood hash function is crucial in order to prevent spurious hit, to be honest, the modular hash function is not the best, check out this article for other interesting techniques to improve our hash function. This solution is not the best but I believe we have been able to demonstrate an interesting solution on how this can work in real-life applications

References

There is so much to learn about string matching that we didn't cover in this article. The resources below made this article possible.