Package: linguado
LinguadoLinguado is a tool which compares the abstract syntax trees (AST) of two or more scripts to measure the similarity. The
main goal intended for this tool is to detect two variants of the same malware.BackgroundThis tool was developed by Guzmán Cernadas Pérez (@DonCaralludo) working for BE:SEC (@BESEC_byEmetel). It was shown by
Marcos Carro Fernández and Guzmán Cernadas Pérez at theVICONin april 2023.InstallationFrom pypi:pip3 install linguadoFrom repo:git clone https://github.com/caralludo/linguado.git
cd linguado
pip3 install .UsageIn order to execute this tool, you have to have two or more source codes in different files, and you have to know in
which language they are made.The help is as follows:usage: main.py [-h] [-p PAR] [-o OUTPUT] Files [Files ...] Language

Linguado is a tool which compares the AST of two or more files. Created by Guzmán Cernadas Pérez (@DonCaralludo)
working for BE:SEC (@BESEC_byEmetel)

positional arguments:
  Files                 Files to analyze
  Language              Language of the files. Options: c, javascript, nasm, php, python2, python3, vba

options:
  -h, --help            show this help message and exit
  -p PAR, --par PAR     Changes the number of iterations of the Weisfeiler-Lehman algorithm (default: 3)
  -o OUTPUT, --output OUTPUT
                        Changes the base name of the output files (default: result.csv)ExamplesCompare two source codes made in python3:linguado source1.py source2.py python3Compare two or more files made in python3:linguado source* python3Compare two or more files made in python3 and change the number of iterations of the Weisfeiler-Lehman algorithm:linguado source* python3 -p 10Compare two source codes made in python3 and changing the output name:linguado source1.py source2.py python3 -o output.csvAvailable languagesFor the moment, the tool can compare the following programming languages:CJavaScriptNASMPHPPython2Python3VBAAdding new languagesTo add a new language you have to do the following steps:Install ANTLRCreate or obtain a grammar in ANTLR4 format.Generate the files with the following command:antlr4 -Dlanguage=Python3 *.g4Save the files in a new folder in the path ./linguado/[language name]Import the Lexer and Parser in the file linguado/main.pyfrommygrammar.MyGrammarLexerimportMyGrammarLexerfrommygrammar.MyGrammarParserimportMyGrammarParserfromlinguado.c.CLexerimportCLexerfromlinguado.c.CParserimportCParserfromjavascript.JavaScriptLexerimportJavaScriptLexerfromjavascript.JavaScriptParserimportJavaScriptParserfromphp.PhpLexerimportPhpLexerfromphp.PhpParserimportPhpParserfrompython2.Python2LexerimportPython2Lexerfrompython2.Python2ParserimportPython2Parserfrompython3.Python3LexerimportPython3Lexerfrompython3.Python3ParserimportPython3Parserfromvba.vbaLexerimportvbaLexerfromvba.vbaParserimportvbaParserModify the dictionary in the file linguado/main.py putting the lerxer, the parser and the first rule of the grammarlanguage_functions={"c":[CLexer,CParser,"translationUnit"],"javascript":[JavaScriptLexer,JavaScriptParser,"program"],"mygrammar":[MyGrammarLexer,MyGrammarParser,"first_rule"],"php":[PhpLexer,PhpParser,"htmlDocument"],"python2":[Python2Lexer,Python2Parser,"file_input"],"python3":[Python3Lexer,Python3Parser,"file_input"],"vba":[vbaLexer,vbaParser,"startRule"]}OutputA possible output example could be:Generating AST's
100%|██████████| 4/4 [00:03<00:00,  1.16it/s]
Calculating Weisfeiler-Lehman matrix
100%|██████████| 3/3 [00:00<00:00, 28.09it/s]
Checking isomorphism (igraph)
100%|██████████| 4/4 [00:02<00:00,  1.98it/s]
Weisfeiler-Lehman:
[[58162880. 58162880. 58162880. 58162880.]
 [58162880. 58162880. 58162880. 58162880.]
 [58162880. 58162880. 58162880. 58162880.]
 [58162880. 58162880. 58162880. 58162880.]]
Weisfeiler-Lehman (%):
[[100. 100. 100. 100.]
 [100. 100. 100. 100.]
 [100. 100. 100. 100.]
 [100. 100. 100. 100.]]
Mean: 58162880.0 , Standard deviation: +- 0.0 ,  0.0
Isomorphism test (igraph):
[[ True  True  True  True]
 [ True  True  True  True]
 [ True  True  True  True]
 [ True  True  True  True]]In each matrix, the columns represents each source code file ordered by name, and each row represents the source code
file ordered by name. So, in each intersection is represented the comparation between the two files.source1.py  source2.py  source3.py  source4.py
source1.py [[ 58162880.   58162880.   58162880.   58162880.]
source2.py  [ 58162880.   58162880.   58162880.   58162880.]
source3.py  [ 58162880.   58162880.   58162880.   58162880.]
source4.py  [ 58162880.   58162880.   58162880.   58162880.]]source1.py  source2.py  source3.py  source4.py
source1.py [[ True        True        True        True]
source2.py  [ True        True        True        True]
source3.py  [ True        True        True        True]
source4.py  [ True        True        True        True]]Also, the tool creates two csv files with the same information in the terminal.Measuring similarityTwo codes will have the same abstract syntax tree if:The isomorphism test matrix has a True in the intersection of the two sources.Two codes will not have the same abstract syntax tree if:The Weisfeiler-Lehman matrix has different values.If the sources do not have the same abstract syntax tree, we can use the standard deviation to know if they are similar:If the standard deviation is close to zero (less than 5%), then the sources will be very similar.If the standard deviation is around the 20%, then could be a chance that the sources are sharing some code.If the standard deviation is more than 50%, then the sources will not be the same.BehaviorGenerates the abstract syntax tree with ANTLR4.From the abstract syntax tree generates a graph which we can work with.Calculates the Weisfeiler-Lehman matrix.Performs the isomorphism test (igraph).Prints on the screen and writes in a CSV the results of the Weisfeiler-Lehman algorithm and the isomorphism test.Other usesThis tool can be used to look for plagiarism in academic environments.Links of interestANTLRGrammars for ANTLRVX-Underground Malware RepositoryJavaScript Malware RepositorySPTH Repository