Plagiarism Detection in Online Exams using Machine Learning | by Christopher Gerling | Towards Data Science

During the Covid-19 pandemic, many educational institutions have been forced to provide home schooling. This includes alternative examination methods for schools and universities. A traditional exam is written in person under permanent surveillance of educational staff. However, online teaching and examinations can fail to ensure a proper surveillance of students. Different online proctoring methods are already used to make plagiarism or examination fraud more difficult. Typical examples are individual video surveillance via webcam or technical user monitoring via mouse and keyboard tracking.Although these techniques already provide a proper solution to make plagiarism in online exams more difficult, the act of surveillance infringes the personal privacy [1]. Furthermore, these surveillance tools do not completely prevent students from multilateral communication among each other.
A recent approach to prevent examination fraud is the analytical identification of similarities between exams. This article proposes a machine learning-based analytical pipeline in order to tackle the problem of collusion from a data-driven perspective. It can replace privacy-infringing surveillance methods during online exams and raises honesty awareness among students.
Related Works
Committing plagiarism can have different forms in exams. Therefore, take-home exams are often designed in a way that all lecture materials and the internet can be used. The act of copying knowledge from external sources is either already expected or irrelevant for answering the questions. However, students are still able to cheat during take-home exams: Collusion is the “active, intentional and obvious act of cheating done in co-operation with others” [2].
Different strategies are used for preventing students’ collusion during exams. One way is to generate student-individual assignments. For instance, in data analytics courses, each student can be given an individual sample drawn from an original data set [3]. However, the individualisation of exams is not possible in every field. Considering a huge number of participants in an exam, the individualisation of questions makes comparability almost impossible. Hence, other strategies are also discussed in literature.
The estimation of similarities between assignments or exams is widely examined: One central problem in detecting collusion is that “no algorithm can detect plagiarism among students who have provided the same correct response where only one correct response is possible” [4]. For that reason, the exact opposite is analysed. A numeric representation of exam is required to put more emphasis on incorrect or rare answers. A specific weirdness vector “highlights suspicious behavior by detecting pairs of exams that contain identical incorrect answers, yet it is also worthwhile to identify pairs of exams that have identical correct answers” [4].
The estimation of an exam vector is dependent on the underlying structure of the questions. Multiple-choice answers, written explanation texts, numeric answers or programming code is evaluated in different ways. Very often, exams include a variety of different question types. This renders automated plagiarism detection difficult using one single detection model. For instance, textual answers can be compared using NLP models (e.g. bag-of-words, tf-idf, n-grams, LSA) [4] [5].
Text mining methods are a presented by multiple authors.However, “no simple process of comparing documents character by character will detect a sophisticated plagiarism” [6]. Other features need to be taken into consideration as well, such as event mining for analysing and comparing the timing of submissions [7].
A powerful method of plagiarism detection are rubric-based tools [1]. This method is applicable on any kind of exam or assignment. The examiner defines rubric-specific items for evaluating a student’s solution. The individual response is described by these multiple items, which are analysed for similarities [1]. Another advantage of this strategy is objectivity and transparency in grading.
Methodology & Results
This article analysis a take-home exam for assessing students’ programming skills. The data set contains 88 written exams and each exam consists of the same 8 tasks with multiple subtasks. During grading, the exam responses are evaluated using rubric-specific items.
The complete process of collusion identification is depicted in figure 1. Initially, the students’ identities are anonymised using randomised chiffre encryption. This step is only necessary for presentation purposes of the results.
import randomimport stringimport numpy as npimport pandas as pdmapping = dict()for c in string.ascii_uppercase:    mapping[c] = random.sample(string.ascii_uppercase,1)for c in string.ascii_lowercase:    mapping[c] = random.sample(string.ascii_lowercase,1)mapping[" "] = " "df_exam.columns = ["".join([mapping[c][0] for c in x if c in mapping.keys()]) for x in df_exam.columns]
The chiffre data frame has 85 columns with student names and 33 descriptive items (rows).
After loading and anonymising the data, the exam vectors are created. Initially, each column contains 33 numeric items. Without any transformation, these could already be used as exam vectors with 33 dimensions. However, there exist many items which have the same value for the majority of students. Specifically, if (almost) all students were able to solve one task and present the same solution, then the information gain of this item is redundant. Items, which add not much information about the differences (variances) within the data set can hence be omitted.
The formulation of a set of items with much variety of different answers is done by a simple machine learning algorithm for dimensionality reduction. A principal component analysis defines a certain number of dimensions which can represent the variance in the data without redundancies.
# one row for each studentX = np.array(df_exam).transpose()# pcafrom sklearn.decomposition import PCApca = PCA(n_components=25)pca.fit(X)# get explained varianceround(sum(list(pca.explained_variance_ratio_))*100, 2)# apply PCAX_new = pca.fit_transform(X)
By setting the number of components (dimensions) to 25, the explained variance is marginally decreased to 98.76% of the original variance. That is, a reduction of 8 items has only marginal impact on modelling the individual differences between students. Moreover, this reduction of dimensionality helps to interpret the “distances” between exams properly. The result is a more compact vector for each exam.
The typical distance measure for vectors is the cosine distance (resp. cosine similarity):
For all 85 exams, the pairwise cosine distances are calculated and stored in a data frame matrix (Figure 3). The values range between -1 for unsimilar exams and +1 for identical exams.
# define cosine distance as similarity measuredef cosine(a, b):     return np.dot(a, b)/(np.linalg.norm(a)*np.linalg.norm(b))# create similarity matrixsims = np.empty([len(X_new), len(X_new)])for stud in range(len(X_new)):    for peer in range(len(X_new)):        sims[stud][peer] = cosine(X_new[stud], X_new[peer])df_sims = pd.DataFrame(sims)df_sims.columns = list(df_exam.columns)df_sims.index = list(df_exam.columns)
The final step for identifying collusive behaviour is sorting the pairwise exam similarities decreasingly. Remember that this approach depends on errors. Specifically, the “weirdness” of two students committing the same errors. Exams without errors will be identical per definition. Therefore, the best exams are ignored in this ordering.
pairwise = df_sims.stack().sort_values(ascending=False).reset_index()pairwise = pairwise[pairwise["level_0"]!=pairwise["level_1"]]# add points to DataFramepoints = pd.DataFrame(df_exam.sum().sort_values(ascending=False))points["name"] = points.index# join informationresult = pd.merge(pairwise, points, left_on="level_0", right_on="name", how= "inner")result = pd.merge(result, points, left_on="level_1", right_on="name", how= "inner")result = result.rename(columns={"level_0":"student_1", "level_1":"student_2", "0_x":"similarity", "0_y":"score_1", 0:"score_2"})result = result[["student_1", "student_2", "similarity","score_1", "score_2"]].reset_index(drop=True)result = result.sort_values(by="similarity", ascending=False)# show only those with mistakesresult[(result["score_1"]<31) & (result["score_2"]<31) & (result["student_1"]<result["student_2"])][:40]
The most similar exams (only those with errors) have a similarity score of 1 (completely identical) or close to 1 (very similar). Figure 4 demonstrates, that this algorithm is able to identify a couple of student-pairs with very similar exam vectors. The student-pair in the first row has the same exam vector over 25 dimensions, which results in a cosine similarity of +1. Considering that both students achieved only average results, the model strongly indicates collusive behaviour.
The similarity score in the following rows remains significantly high, which can also be a hint for collusion. Using this model, the examiners are able to identify suspicious structures easily. A manual comparison of these pairs of exams is then recommended.
The pairwise similarities can be divided into three groups:
The best 20% of exams compared to the other 80% (top_20 vs. others)The best 20% of exams in comparison (top_20 vs. top_20)The lower 80% of exams in comparison (others vs. others)
The best 20% of exams compared to the other 80% (top_20 vs. others)
The best 20% of exams in comparison (top_20 vs. top_20)
The lower 80% of exams in comparison (others vs. others)
Starting with (1), the distribution of the similarity scores seems to be almost evenly distributed. A very good exam compared to a very bad exam results in a highly negative similarity score. Likewise, in comparison to a good exam, the similarity score tends to be high.
Focussing only on the best 20% of exams (2), the similarity scores are all quite high. This underlines the hypothesis that two very good exams share a similarity score close to one. Hence, the identification of collusion among these students is almost impossible per definition.
When excluding the best students and only analysing the “worst” 80% of exams (3), the plot seems to follow a normal distribution with an expected similarity score of 0 on average.
For identifying plagiarism, (1) and (3) are the two important groups. The case of two bad/average exams with a similarity score of almost +1 is considered highly suspicious. Figure 5 shows a couple of these cases. The comparison of bad and a good exams is also worth checking, since many pairwise similarities are close to 1.
Reducing the number of dimensions even further to only two dimensions allows to plot the exams in a coordinate system. This visualisation helps to further understand the distribution of exams based on the grade. The students were segmented into three groups using k-means clustering. These three groups represent “bad/worst grades”, “good/best grades” and “average grades”. Figure 6 shows the distribution of the exams using two latent variables. Here, the latent dimension 1 can be interpreted as the final score of a student. The latent dimension 2 is necessary to visualise the similarities/distances between exams.
Once more, this plot shows the high similarity between very good exams. A proper distinction for collusion identification seems impossible. However, the data points of those students with average or bad grades are much more scattered. The expected similarity between these exams should be around 0. Thus, two bad/average exams with a higher similarity score are worth a manual investigation by the examiner.
From an ethical perspective, the data-driven approach of plagiarism detection opens up new opportunities: It can prevent collusive behaviour without permanent and privacy-infringing surveillance during online-exams. The possibility of being caught cheating by an algorithm after the evaluation is a mild but effective threat. In this scenario, the students themselves are neither observed nor analysed during the exam. Only the given responses are digitally processed. However, as a matter of fairness, this approach fully relies on the students’ awareness that all results will be analysed for collusion. Making the process of plagiarism detection transparent will hence prevent students from cheating without drastic privacy infringement.
Christopher Gerling
Follow me on: www.linkedin.com/in/christopher-gerling/
www.mission-digital.comgithub.com/christopher3996
References
A. M. Edward F. Gehringer and G. Wang, “Tools for detecting plagiarism in online exams,” in 2021 ASEE Virtual Annual Conference Content Access. Virtual Conference: ASEE Conferences, 2021, https://peer.asee.org/37915.D. Dobrovska and A. Pokorny, “Avoiding plagiarism and collusion.”, 2007.J. Coakley and C. K. Tyran, “Can e-cheating be prevented?: An approach to detect plagiarism in computer skills courses,” 2001.E. F. Gehringer, X. Liu, A. D. Kariya, and G. Wang, “Comparing and combining tests for plagiarism detection in online exams,” in EDM, 2020.J. Lemantara, M. Dewiyani Sunarto, B. Hariadi, T. Sagirani, and T. Amelia, “Prototype of online examination on molearn applications using text similarity to detect plagiarism,” in 2018 5th International Conference on Information Technology, Computer, and Electrical Engineering (ICITACEE), 2018, pp. 131–136.J. Leskovec, A. Rajaraman, and J. D. Ullman, Mining of Massive Datasets, 2nd ed. Cambridge University Press, 2014. [Online]. Available: http://mmds.org.C. Cleophas, C. Honnige, F. Meisel, and P. Meyer, “Who’s cheating?  ̈ mining patterns of collusion from text and events in online exams,” INFORMS Transactions on Education, 2021. [Online]. Available: https://doi.org/10.1287/ited.2021.0260.
A. M. Edward F. Gehringer and G. Wang, “Tools for detecting plagiarism in online exams,” in 2021 ASEE Virtual Annual Conference Content Access. Virtual Conference: ASEE Conferences, 2021, https://peer.asee.org/37915.
D. Dobrovska and A. Pokorny, “Avoiding plagiarism and collusion.”, 2007.
J. Coakley and C. K. Tyran, “Can e-cheating be prevented?: An approach to detect plagiarism in computer skills courses,” 2001.
E. F. Gehringer, X. Liu, A. D. Kariya, and G. Wang, “Comparing and combining tests for plagiarism detection in online exams,” in EDM, 2020.
J. Lemantara, M. Dewiyani Sunarto, B. Hariadi, T. Sagirani, and T. Amelia, “Prototype of online examination on molearn applications using text similarity to detect plagiarism,” in 2018 5th International Conference on Information Technology, Computer, and Electrical Engineering (ICITACEE), 2018, pp. 131–136.
J. Leskovec, A. Rajaraman, and J. D. Ullman, Mining of Massive Datasets, 2nd ed. Cambridge University Press, 2014. [Online]. Available: http://mmds.org.
C. Cleophas, C. Honnige, F. Meisel, and P. Meyer, “Who’s cheating?  ̈ mining patterns of collusion from text and events in online exams,” INFORMS Transactions on Education, 2021. [Online]. Available: https://doi.org/10.1287/ited.2021.0260.
The code and the data were both created by the author and published on Github.