Package: picblocks
PicBlocksAn experimental project using position-independent code hashing over basic blocks for code similarity estimation.UsageBoth module files in./picblocksand in./utilsare runnable and contain examples of their usage:$ python -m picblocks.blockhasher <target_binary_path>- produces ablock-reportfor a single binary.$ python -m picblocks.blockhashmatcher <block_reports_path>- creates a new./db/picblocksdb.jsonfrom theblock-reportslocated in<block_reports_path>$ python -m blocks.blockhashmatcher <block_reports_path> <target_binary_path>- matches a binary against data stored in./db/picblocksdb.jsonif it exists, or otherwise creates./db/picblocksdb.jsonfrom theblock-reportslocated in<block_reports_path>$ python -m utils.import_picblocksdb_to_mongo.pyassumes some mongodb configurations (please check inside the file to adapt to yours) it merely takes the json generated DB into a most easy to manage (and query)  mongodb.$ python -m utils.make_stats.pyit assumes a mongodb connection (please check inside the file to adapt to yours), the generated json db intodb/picblocksdb.json(you can change it directly in the relative varible) and the generated blocks report into./block-reports/folder. It builds up some statistics about detections and DB composition. The results would be available in a dedicated (and very simple) stats web ui.Creating a DatabaseThe scripthash_malpedia.pyis an example of how to process a collection of binaries into./block-reports, which will then be aggreated into a./db/picblocksdb.json.Database EvaulationIn oder to quantify and to measure the quality of your detection rate you should check some basic informations about tests run against your db.
The simple (and preliminary) script namedmake_stats.pywould build up some initial stats for you about detection rates.
It assumes to have a mongodb connection, the generated json db intodb/picblocksdb.json(you can change it directly in the relative varible) and the generated blocks reports into./block-reports/folder (you can change it directly on the specified variable).
Once you run it, it takes every single block report and check it against the json database.
It build some stats and saves all the matching results into db.
A dedicated web page (and a relative API) is built to show the detection rates and some more interesting statistics on your database.Running as a ServiceIf a./db/picblocksdb.jsonexists, you can run$ python app.pyto spawn a local demo server (https://127.0.0.1:9001) to query against.ScreenshotsJust few screenshots about the initial stage of web user interface
The submit form. Once you have a given database (./db/picblocksdb.json) you can check matching from samples by submitting your samples from
this form.If the submitted sample gets some matches against the given database you should see a block similarity matrix (still under development for a better visualization)Finally the matching database statistics generated by the script intoutils/make_stats.pywhich takes all the generated block_reports (block-reports/) and check them against the generated databases (./db/picblocksdb.json) in order to estimate the detection rate on a given database.ContributorsDaniel PlohmannMarco RamilliDaniele BellavistaVersion History2023-11-24: v2.0.1 - SMDA pinned to 1.12.7 before our bigger fix for PIC calculation2022-09-08: v2.0.0 - (BREAKING CHANGE) now intraprocedural control flow transfers are wildcarded by default, which should improve matching2022-08-04: v1.1.3 - extended format for blockhash representation of functions2021-10-01: v1.1.1 - added script to check detection rates and relative web interface page2021-09-28: v1.1.0 - added simple web user interface and a db connection2021-09-12: v1.0.6 - added submission form fields for bitness and base address to force overrides for those values.2021-08-24: v1.0.5 - improved parsing of bitness from submission filenames.2021-08-20: v1.0.4 - Tweaked result visualization, now showing all unique matches beyond the first 20.