Package: sailreval
SAILR Evaluation PipelineThe SAILR evaluation pipeline,sailreval, is a tool for measuring various aspects of decompilation quality.
This evaluation pipeline was originally developed for the USENIX 2024 paper"Ahoy SAILR! There is No Need to DREAM of C:
A Compiler-Aware Structuring Algorithm for Binary Decompilation". It supports 26 different C packages from Debian,
for compiling, decompiling, and measuring. Currently, angr, Hex-Rays (IDA Pro), and Ghidra are supported as decompilers.If you are only looking to use the SAILR version of angr, then jump to theusing SAILR on angrsection.Table of ContentsOverviewInstallationUsageCompilationDecompilationMeasuringAggregationSAILR Paper ArtifactsUsing SAILR on angr decompilerSAILR evaluation results filesReproducing SAILR paper resultsMiscellaneousCompiling Windows TargetsOverview:This repo contains thesailrevalPython package and information about the SAILR paper artifacts.sailrevalis the Python package that contains all the code for running the evaluation pipeline.sailrevalevaluates the quality of decompilation by comparing it to the original source code.
This evaluation is done in four phases:Compilation: a project described in thetargetsdirectory is downloaded, preprocessed, and compiled into object files.Decompilation: decompilerssupported in sailrevalare used to decompile the object files into C source files.Measurement: the preprocessed source and decompiled source are compared usingmetricsinsailreval.Aggregation: the results from the measurement are normalized for functions that had a metric on all decompilers.Each phase requires the phase directly before it runs; however, you can skip stages if you manually provide the
required files. For example, you can skip the decompilation phase if you already have the object files and preprocessed source.InstallationThesailrevalpackage can be used in two ways: locally or in a docker container.
If you plan on reproducing the results of the SAILR paper or using some pre-packaged decompiler like Ghidra, then you
will need both. Below are two methods for installing: one is heavy (docker and local), and one is light (only local).
Make sure you have Docker installed on your system.Install Script (Recommended)On Linux and MacOS:./setup.shThis will build the Docker container, install system dependencies, and install the Python package locally.Only Python PackageIf you want to use only local decompilers and you have the build dependencies installed for your compiled project, you
can install the Python package without the Docker container. For an example of this use case, see
ourCI runner.pip3insatll-e.Note: you will need to install the system dependencies for the Python project yourself, listed [here](CI runner.
The package is also available on PyPi, so remote installation works as well.Install VerificationVerify the installation by running:./scripts/verify_pipeline.shThis will use both the Docker container and your local install to run the Pipeline.
If you installed it correctly, you should see some final output like:# Evaluation Data## StatsLayout: ('sum', 'mean', 'median')### O2Metric     | source      | angr_sailr  | angr_dream
---------- | ----------- | ----------- | -----------
gotos      | 1/0.12/0.0  | 1/0.12/0.0     | 0/0/0.0
...UsageAfter installation, if you used the script normally (i.e. the docker install), than you can use thedocker-eval.shscript
which is a proxy to theeval.pyscript, but inside the container.
As an example you can use:./docker-eval.sh--help
./eval.py--helpThey should both produce the same result.Using the steps below, you can run the entire pipeline stage-by-stage. In each evaluated target intargetsyou will
be able to find asailr_compiled,sailr_decompiled, andsailr_measuredfolder in the package folder.
Each folder will contain the results of the respective stage. All targets are places in theresultsdirectory under
their respective optimization.
For coreutils compiled with O2, you'll seeresults/O2/coreutils.CompilingTo compile a package it must be described in thetargetsfolder by atarget.toml. Here is coreutils:package_name="coreutils"source_remote="git://git.sv.gnu.org/coreutils.git"remote_type="git"download=truepost_download_cmds=["./bootstrap"]version="v9.1"package_dir="coreutils"pre_make_cmds=["./configure --quiet"]make_cmd="make"post_make_cmds=[]source_dir="src"There are many flags that you can set which are defined in thesailr_targetclass.We compile just coreutils using the docker wrapper:docker-eval.sh--compilecoreutils--cores8--opt-levelsO2After compiling is done, you can find the results in theresults/O2/coreutilsdirectory.
In thesailr_compiledfolder located incoreutilsyou will find all the object files, preprocessed source, and normal source.
The next phase will destroy the normal source and replace it with the preprocessed source.
It's critical that you do not edit the preprocessed source in any way.DecompilingThe target must contain thesailr_compiledfolder with.ofiles in it. In the case of coreutils that would be:./results/O2/coreutils/sailr_compiled/. The source must also be present in that folder.For the very first time you decompile a target, you must "decompile" the source, which creates normalized preprocessed source.
Do it like so:./eval.py --decompile coreutils --use-dec source --cores 20 --opt-levels O2Highly recommend to run locally for speed.
After this is done, you don't need to do it again even if you re-decompile for other decompilers.Next, you decompile all the decompilers you want:./docker-eval.sh --decompile coreutils --use-dec ghidra angr_sailr angr_phoenix --cores 20 --opt-levels O2All the decompilation files, including the preprocessed source, will be found inside thesailr_decompiledfolder.
For coreutils that would be:./results/O2/coreutils/sailr_decompiled/.
You will find the preprocessed source assource_*.cand the decompilation as<decompiler>_*.c.
You will also notice files likeangr_sailr_mv.linemaps,angr_sailr_mv.toml, andmv.dwarf.linemaps.
These files contain the line mappings for decompiled source to original source and pre-computed metrics likegotocounts.If you plan on using IDA Pro, you must mount it into the container.
Please mount theidat64binary directly into the container at/tools/.
To do that, add-v /path/to/idat64_folder/:/tools/to thedocker runcommand in thedocker-eval.shscript.MeasuringLike the decompilation phase, this phase requires thesailr_decompiledto exist with.oand.cfiles in it.
If you plan on usingcfged, then thesailr_decompildfolder must contain the linemaps, toml, and dwarf files for each targeted object file.If you ran the decompilation step above, you should automatically have that. Measure with:./eval.py--measurecoreutils--use-metricgotoscfged--use-decsourceangr_sailr--cores15NOTE: you must putsourceas one of the targeted decompilers if you are usingcfged.After runing, you will find files in thesailr_measuredfolder.
For coreutils that would be:./results/O2/coreutils/sailr_measured/.
In the folder you will find varioustomlfiles that look like the following:binary="mv"total_time=231.44658088684082timeout=false[source.cfged]main="0.0"[source.gotos]main=0[angr_sailr.cfged]main="309.0"[angr_sailr.gotos]main=11# ...You can use thetomllibrary in Python to load these files into a dictionary.
The dictionary is keyed by[decompiler][metric][function]and the value is the metric value.AggregatingAfter measuring, you can aggregate the results like so:./eval.py--summarize-targetscoreutils--use-decsourceangr_sailr--use-metricgotoscfgedThe results will look something like, which is all sums:# Evaluation Data## Stats### O2Decompiler | gotos_sum | cfged_sum
---------- | --------- | ---------
source     | 46        | 0
angr_sailr | 668       | 39701## Metadatatotal_unique_functions_in_src | total_unique_functions_in_all_metrics
----------------------------- | -------------------------------------
1152                          | 918Only the last printed table matters. Tables printed before that are intermediate results.
You can also showSum/Average/Medianby using the--show-statsarg.The above summarization is the normalized results where each count is based on functions that successfully decompiled and measured
on all decompilers specified in the command. You can also do multiple targets at once:./eval.py--summarize-targetscoreutilsdiffutils...There is a special case summarization for projects that are the same but may have different names.
This happens in the case ofcoreutilsandcoreutils_gcc5. Both are Coreutils compiled with different decompilers.
You can normalize across both projects for binaries and functions that only exist across both projects with:./eval.py--merge-results./results/O2/coreutils*/sailr_measured--use-decsourceangr_sailr--use-metricgotoscfgedSAILR Paper ArtifactsTheSAILR paperintroduced four artifacts:Theangr decompiler, found in theangrrepo.TheSAILRalgorithm,
built on the angr decompiler as optimization passes.The SAILR evaluation pipeline, found in thesailrevalPython packageThe results ofsailrevalfor the paper (tomls and decompilation outputs)Below you will find instructions for using each of these artifacts.Using SAILR on angr decompilerCurrently, SAILR is being slowlyintegrated into the angr master branch.
Until then, you can use theangr-sailrfork of angr inside our provided stripped down Dockerfile found inmisc/angr_sailr_dec/Dockerfile.
You can also use the pre-built docker image found onDockerhub(~3.5gb).
Note, this fork will not receive updates and is the exact version used in the paper.
The commit isbe3855762a84983137696aa14efe2431a86a7e97.To build the decompiler docker image, run, from the root of the repo:dockerbuild-tangr-sailr-dec-fmisc/angr_sailr_dec/Dockerfile.You could now run the image withdocker run --rm -it angr-sailr-dec, but we recommend using the wrapper script.
You can use the wrapper script that will run this image for you:./scripts/docker-angr-sailr-dec.sh--helpNOTE: this mounts the current directory into the container so the decompiler can access the binary.Verify your version works by running it on themotivating_examplebinary in the root of the repo:./scripts/docker-angr-sailr-dec.sh./tests/binaries/motivating_exampleschedule_job--structuring-algosailrIf working correctly, you should see the following output, which matches the papers example:longlongschedule_job(unsignedlonga0,unsignedlonglonga1,unsignedlonga2){if(a0&&a1){complete_job();if(EARLY_EXIT==a2)gotoLABEL_40126b;next_job();}refresh_jobs();if(a1||a1)fast_unlock();LABEL_40126b:complete_job();log_workers();returnjob_status(a1);}SAILR evaluation results filesIn the SAILR paper, we run an evaluation on all 26 packages in thetargetsdirectory.
We generated data for optimization levelsO0,O1,O2, andO3.
We also recorded the decompilation on all these targets with the following decompilers:SAILR,IDA Pro 8.0,Ghidra 10.1,Phoenix,DREAM, andrev.ng. In eachsailr_decompiledfolder,
you will find files likes so<decompiler>_<binary_name>.c. For example,sailr_decompiled/ida_mv.cis the decompilation
ofmvfrom coreutils with IDA Pro 8.0. angr based decompilation starts withangr_and then the structuring algorithm.All the files, which are about 11gbs in total, can be downloaded fromthis Dropbox link.After downloading, you can extract the files with:tarxfresults.tar.gz--use-compress-program=pigzThe will looks like the following (but with more files):results/
├── O0
├── O1
├── O2
│   └── coreutils
│       ├── sailr_compiled
│       ├── sailr_decompiled
│       └── sailr_measured
└── O3To further understand what is contained in thesailr_*folders, see theusagesection above.
You can now use thesailrevalpackage to aggregate the results like so to get the results from the paper:./eval.py--summarize-targets./eval.py--measurebashlibselinuxshadowlibeditbase-passwdopenssh-portable\dpkgdashgrepdiffutilsfindutilsgnutlsiproute2gzipsysvinitbzip2libacllibexpatlibbsdtarrsyslog\croniezlibe2fsprogscoreutils\--use-decsourceidaghidraangr_sailrangr_phoenixangr_dreamangr_comb\--use-metricgotoscfgedboolsfunc_calls\--opt-levelsO0O1O2O3\--show-statsReproducing SAILR paper resultsWe ran the entire pipeline of SAILR on an Ubuntu 22.04 machine that had 40 logical cores and 64 GB of RAM.
With these specs, it took about 8 hours to run the entire pipline for all 26 packages on the O2 optimization level.
If you intend to reproduce the results as they were in the paper, checkout this repo to commit8442959e99c9d386c2cdfaf11346bf0f56e959eb,
which was the last version with minor fixes to the pipeline, but not edits to CFGED.
If you plan on evaluating modernly, use the latest commit, since it will have stability, speed, and other fixes to components of SAILR.Due to slowness in processing of source with Joern, we recommend running the Joern stageLOCALLYand not in the
container. Here is an example run of only coreutils:./docker-eval.sh--compilecoreutils--cores20&&\./eval.py--decompilecoreutils--use-decsource--cores20&&\./docker-eval.sh--decompilecoreutils--use-decghidraangr_sailrangr_phoenixangr_dreamangr_comb--cores20&&\./eval.py--measurecoreutils--use-metricgotoscfgedboolsfunc_calls--use-decsourceghidraangr_sailrangr_phoenixangr_dreamangr_comb--cores20&&\./eval.py--summarize-targetscoreutils--use-decsourceghidraangr_sailrangr_phoenixangr_dreamangr_comb--use-metricgotoscfgedboolsfunc_calls--show-statsTake note of wheneval.pyis run instead of the docker version.To reproduce the results from the paper, you run the following evaluation scripts that will run the entire pipeline for you:./scripts/evaluation/all_packages_o2_table3.sh
./scripts/evaluation/coreutils_o2_table4.shRun them one at a time to observe their output.Note, you will likelynotget the exact numbers shown in the paper, but the final conclusions on the numbers (i.e. the relative distance of each score) should be the same.
This is due to a fundamental limitation in CFGED, which relies on GED to compute the edit distance between two CFGs.
Since we never know if GED will conclude, we must use a timeout, which can be affected by the machine you run on.
However, for most cases the timeout should not be triggered.MiscellaneousCompiling Windows TargetsWindows targets, likelibz_windows, will not be compiled by this pipeline, so you must compile them yourself.
Follow the following the steps to compile a windows target:Download the source code for the target specified in the targets toml fileMake a new configuration in MSVCProject->Properties->Configuration Manager->Active Solution Configuration->NewName isSAILRGo toProject->Properties->C/C++->Preprocessorand enable Preprocessor Definitions to FileHit compile with SAILR config, the copy all*.i,*.c, and*.objfiles into thesrcfolder you need to makeRename the*.objto*.oIf step5failed, then just remove the preprocessor option after running onceTo run the full pipeline for Widnows targets, you must havellvm-pdbutilinstalled on the system.