Package: pymousse
mousse: plagiarism detection using MOSS with post-processingmousseis a command-line tool to submit programming assignments toMOSSplagiarism checker.
Beyond submitting, it also performs the following:download the report generated by MOSSextract amounts of copied lines from itcompute distances between pairs of projectsgenerate heatmaps to visually identify clusters of similar projectsThe generated heatmaps are clickable, allowing one to directly look at MOSS report for a pair of projects.
Before submission, code is cleaned a little bit to ease reading reports, in particular:comments are removedmultiple files are grouped into a unique filemoussemay also be used as a Python library to use its features separately.(By the way,mousseif the French word formoss.)InstallationRunpip install pymousse.moussedepends on:Python 3 (developed with 3.9, may work with earlier versions)requestsbeautifulsoup4chardettqdmmosspythefuzzpython-Levenshteinfastclusterpandasnumpyscipyseabornmatplotlib(By the way,pymousseif pronounced in French as a famous brand of candies calledPimousse, and as namemoussewas already used on PyPi, it was natural to name the project like this.)UsingmousseOverviewmousseoperates on a single directoryroot(specified on the command line) that should be organised as follows:root/
+- projects/
|  +- student_1/
|  +- student_2/
|  +- ...
+- base/Directoryprojects(specified with option-s/--source) holds one directory for each student, where source code will be searched for to be sent to MOSS.
Directorybase(specified with-b/--base) is optional, and if used must contain the code that has been provided to students and will not be taken into account for plagiarism detection.Assuming current working directory is whererootis located,moussecan by run aspython -m mousse -b base root, or, if nobaseis provided, just aspython -m mousse root.
After runningmousse, directoryroothas been populated with new content:root/
+- projects/  (untouched)
+- base/      (untouched)
+- moss/
|  +- index.html
|  +- match-0.html
|  +- ...
+- dists.csv
+- dists.pdf
+- dists-pruned.pdfDirectorymossis where MOSS report has been downloaded.
Filedists.csvis a CSV file with the distances computed from MOSS' matches: the more source code two projects have in common, the closer they are with respect to this distance.
Filedists.pdfis the heatmap computed to visually show the distances and cluster similar projects, anddists-pruned.pdfis a reduced version of the heatmap where only the most significant projects have been kept.A heatmap is a matrix showing the distance between every pairs of projects.
It is clustered in such a way that close projects (ie, those that share more source code) are displayed next to each other, forming visual cluster of similar projects.
Each pair of project is displayed as a square whose colour scales from red (very similar) to blue (very distinct).
Each such square is clickable which allows to open directly the corresponding page in MOSS' report.
Above the heatmap (and on its right), a dendogram is displayed to show how the clustering is organised, just like inphylogenetic trees.
Note that the diagonal of such a heatmap is red by definition because it compares a project with itself.Command line optionsmoussesupports the following options:-u USERID,--userid USERIDuseUSERIDto authenticate with MOSS-c CONFIG,--config CONFIGpath of configuration file (see below)-i,--incremental(default: no)
do not submit source and dowload MOSS report is it exists-s SOURCE,--source SOURCE(default:projects)
the directory withinrootwhere projects are located-b BASE,--base BASE(default: none)
the directory withinrootwhere base source code is located-r REPORT,--report REPORT(default:moss)
the directory where the downloaded MOSS report is saved-R,--rawuse raw distances without cleaning (see below)-p PRUNE,--prune PRUNE(default: 0.5)
how full heatmap is pruned (see below)-a,--absolute(default: relative)
use absolute scale for heatmap coloring (see below)-l LANG,--lang LANG(default:c)
programming language used for projects-L,--list-langlist supported programming languages and exitThen, except is-Lwas used,mousseexpects then a path forroot.Finally, any remaining option has to be formatted asKEY=VALUEand can be used to control heatmap rendering:lnk_OPT=VALUEOPT=VALUEis passed toscipy.cluster.hierarchy.linkagewhen computing the clusteringsns_OPT=VALUEOPT=VALUEis passed toseaborn.clustermapwhen drawing the heatmapplt_OPT=VALUEOPT=VALUEis passed tomatplotlib.pylab.savefigwhen saving the heatmapConfiguration fileA configuration file~/.config/mousse.inican be used to store MOSS user id, which avoids having to pass it with option-uevery times:[moss]
userid = 123456789Additionally, this file may contain definitions for new languages (or redefinition of existing languages), for instance C is defined as:[lang:c]
suffix = .c
search = *.c *.h
comment = ["//", ("/*", "*/")]
moss_lang = cIn this definition:lang:cmeans we (re)define a language calledcsuffixis the extension to use when creating a single file gathering all the source code from one projectsearchis the list of files patterns searched in a project directory, all other files will be ignored and not submitted to MOSScommentis a Python list of either strings or pairs of strings that define the comments supported by the programming language.
A string defines a single-line comment starting with the given marker, a pair defines a single- or multi-line enclosed into the two given markers.
If the language has only single-line comments with just one marker, it may be given directly instead of a Python list, eg, for Python one may usecomment = #moss_langis the language name as MOSS knows itCleaning and pruning heatmaps, absolute/relative colour scaleWhen distance between projects are computed from MOSS reports, a heuristic is used to filter matches that are not relevant.
For each pair of projectsP1,P2, and for each each matchM1,M2(ie, two fragments of source code that MOSS reported as similar), a score is computed as:the length of the match (maximum of the length ofM1andM2)divided by theLevenshtein distancebetweenM1andM2multiplied by the number of lines copied betweenP1andP2as reported by MOSS
If all the matches forP1,P2score below the average, then this pair of project is assigned distance 1 (maximal distance).
This result in practice in hiding pairs of projects with only irrelevant matches, that is, following how scores are computed, with:short matcheswith large edit distancesfrom projects with few similaritiesAs this is only a heuristic, it may be disabled using option-R, in which case the distance as directly computed from MOSS matches is used.
Note that even if the heuristic is applied, the heatmap remains clickable and MOSS reports can be looked at, but the corresponding square is just drawn blue.Using option-p PRUNEallows one to control how the full heatmap can be pruned in order to keep only the pairs of most similar projects.
First,PRUNE < 0cancels pruning, only the full heatmap is computed
Then, ifPRUNE > 1, at mostPRUNEprojects are kept, all the other are removed.
This is performed by descending the dendogram on the heatmap progressively and removing isolated projects until we reach the expected count.
Finally, if0 <= PRUNE <= 1, projects whose distance to any other is more thanMEAN - STDEV * PRUNEare discarded, whereMEANis the average distance between every pair of projects andSTDEVis the standard deviation.Heatmap are colored by default with a relative color scale, which means that the top of the dendogram is blue and the bottom red.
When option-ais used, blue is for distance 1 and red is for 0, regardless of how the distances are actually distributed.
Relative heatmaps exhibit more contrasted colours but tend to suggest closer proximity between projects.APImoussemay be used from Python programs.TODO: write some documentation, including docstrings