Package: polytracker
PolyTrackerPolyTracker is a tool originally created for theAutomated Lexical Annotation
and Navigation of Parsers, a backronym devised solely for the purpose of
referring to it asThe ALAN Parsers Project. However, it has evolved into a
general purpose tool for efficiently performing data-flow and control-flow
analysis of programs. PolyTracker is an LLVM pass that instruments programs to
track which bytes of an input file are operated on by which functions. It
outputs a database containing the data-flow information, as well as a runtime
trace. PolyTracker also provides a Python library for interacting with and
analyzing its output, as well as an interactive Python REPL.PolyTracker can be used in conjunction withPolyFileto automatically determine
the semantic purpose of the functions in a parser. It also has an experimental
feature capable of generating a context free grammar representing the language
accepted by a parser.Unlike dynamic instrumentation alternatives likeTaintgrind, PolyTracker imposes
negligible performance overhead for almost all inputs, and is capable of
tracking every byte of input at once. PolyTracker started as a fork of the LLVM
DataFlowSanitizer and takes much inspiration from theAngora Fuzzer. However, unlike the
Angora system, PolyTracker is able to track the entireprovenanceof a taint.
In February of 2021, the LLVM DataFlowSanitizer added a new feature for tracking
taint provenance calledorigin tracking.
However, it is only able to track at most 16 taints at once, while PolyTracker
can track up to 231-1.This README serves as the general usage guide for installing PolyTracker and
compiling/instrumenting binaries. For programmatically interacting with or
extending PolyTracker through its Python API, as well as for interacting with
runtime traces produced from instrumented code,consult the Python documentation.QuickstartPolyTracker is controlled via a Python script calledpolytracker. You can
install it by runningpip3 install polytrackerPolyTracker requires a very particular system environment to run, so almost all
users are likely to run it in a containerized environment. Luckily,polytrackermakes this easy. All you need to do is havedockerinstalled,
then run:polytracker docker pullandpolytracker docker runThe latter command will mount the current working directory into the PolyTracker
Docker container, and allow you to build and run instrumented programs.Thepolytrackercontrol script—which you can run from either your host system
or from inside the Docker container—has a variety of commands, both for
instrumenting programs as well as analyzing the resulting artifacts. For
example, you can explore the dataflows in the execution, reconstruct the
instrumented program's control flow graph, and even extract a context free
grammar matching the inputs accepted by the program. You can explore these
commands by runningpolytracker --helpThepolytrackerscript is also a REPL, if run with no command line arguments:$polytrackerPolyTracker(4.0.0)https://github.com/trailofbits/polytrackerType"help"or"commands">>>commandsInstrumenting a simple C/C++ programPolyTracker also comes with abuildcommand. This command allows the user to
run any build command in aBlightinstrumented environment. This will produce ablight_journal.jsonlfile that
records all commands run during the build. If you have a C/C++ target, you can
instrument it by invokingpolytracker buildand passing your build command:polytrackerbuildgcc-g-omy_binarymy_source.cTo instrument a build target, use theinstrument-targetscommand. By default
the command will use the ablight_journal.jsonlin your current working
directory to build an instrumented version of your build target. The
instrumented build target will be built using the same flags as the original
build target.polytrackerinstrument-targetsmy_binarybuildalso supports more complex programs that use a build system like
autotiools or CMake:polytrackerbuildcmake..-DCMAKE_BUILD_TYPE=Release
polytrackerbuildninja# orpolytrackerbuild./configure
polytrackerbuildmakeThen runinstrument-targetson any targets of the build:$polytrackerinstrument-targetsa.binb.soThena.instrumented.binandb.instrumented.sowill be the instrumented
versions. See the Dockerfiles in theexamplesdirectory for examples of how real-world programs can be instrumented.Running and Analyzing an Instrumented ProgramThe instrumented software will write its output to the path specified inPOLYDB, orpolytracker.tdagif omitted. This is a binary file that can be
operated on by running:frompolytrackerimportPolyTrackerTrace,taint_dagtrace=PolyTrackerTrace.load("polytracker.tdag")tdfile=trace.tdfilefirst_node=list(tdfile.nodes)[0]print(f"First node affects control flow:{first_node.affects_control_flow}")# Operate on all Range nodesforindex,nodeinenumerate(tdfile.nodes):ifisinstance(node,taint_dag.TDRangeNode):print(f"Node{index}: first{node.first}, last{node.last}")# Access taint foresttdforest=trace.taint_forestn1=tdforest.get_node(1)print(f"Forest node{n1.label}. Parent labels:{n1.parent_labels}, "f"source:{n1.source.pathifn1.sourceisnotNoneelseNone}, "f"affects control flow:{n1.affected_control_flow}")You can also run an instrumented binary directly from the REPL:$polytrackerPolyTracker(4.0.0)https://github.com/trailofbits/polytrackerType"help"or"commands">>>trace=run_trace("path_to_binary","path_to_input_file")This will automatically run the instrumented binary in a Docker container, if
necessary.:warning:If running PolyTracker inside Docker or a VM: PolyTracker can be
very slow if running in a virtualized environment and either the input file
or, especially, the output database are located in a directory mapped or
mounted from the host OS. This is particularly true when running PolyTracker
in Docker from a macOS host. The solution is to write the database to a path
inside of the container/VM and then copy it out to the host system at the very
end.The Python API documentation is availablehere.Runtime Parameters and Instrumentation TuningAt runtime, PolyTracker instrumentation looks for a number of configuration
parameters specified through environment variables. This allows one to modify
instrumentation parameters without needing to recompile the binary.Environment VariablesPolyTracker accepts configuration parameters in the form of environment
variables to avoid recompiling target programs. The current environment
variables PolyTracker supports is:POLYDB: A path to which to save the output database (default is polytracker.tdag)

WLLVM_ARTIFACT_STORE: Provides a path to an existing directory to store artifact/manifest for all build targets

POLYTRACKER_TAINT_ARGV: Set to '1' to use argv as a taint source.Polytracker will set its configuration parameters in the following order:If a parameter is specified via an environment variable, use that valueElse if a default value for the parameter exists, use the defaultElse throw an errorABI ListsDFSan uses ABI lists to determine what functions it should automatically
instrument, what functions it should ignore, and what custom function wrappers
exist. See thedfsan documentationfor
more information.Creating custom ignore lists from pre-built librariesAttempting to build large software projects can be time consuming, especially
older/unsupported ones. It's even more time consuming to try and modify the
build system such that it supports changes, like dfsan's/our instrumentation.There is a script located inpolytracker/scriptsthat you can run on any ELF
library and it will output a list of functions to ignore. We use this when we do
not want to track information going through a specific library like libpng, or
other sub components of a program. TheDockerfile-listgen.demoexists to build
common open source libraries so we can create these lists.This script is a slightly tweaked version of what DataFlowSanitizer has, which
focuses on ignoring system libraries. The original script can be found indfsan_rt.Building the ExamplesCheck out this Git repository. From the root, either build the base PolyTracker
Docker image:pip3 install -e ".[dev]" && polytracker docker rebuildor pull the latest prebuilt version from DockerHub:docker pull trailofbits/polytracker:latestFor a demo of PolyTracker running on theMuPDFparser run
this command:docker build -t trailofbits/polytracker-demo-mupdf -f examples/pdf/Dockerfile-mupdf.demo .mutool_trackwill be build in/polytracker/the_klondike/mupdf/build/debug.
Runningmutool_trackwill outputpolytracker.tdagwhich contains the
information provided by the taint analysis.For a demo of PolyTracker running on Poppler utils version 0.84.0 run this
command:docker build -t trailofbits/polytracker-demo-poppler -f examples/pdf/Dockerfile-poppler.demo .All the poppler utils will be located in/polytracker/the_klondike/poppler-0.84.0/build/utils.$ cd /polytracker/the_klondike/poppler-0.84.0/build/utils
$ ./pdfinfo_track some_pdf.pdfBuilding PolyTracker from SourceThe compilation process for both PolyTracker LLVM and PolyTracker is rather
fickle, since it involves juggling both instrumented and non-instrumented
versions of standard library bitcode. We highly recommend using our pre-built
and tested Docker container if at all possible. Installing the PolyTracker
Python package on your host system will allow you to seamlessly interact with
the prebuilt Docker container. Otherwise, to install PolyTracker natively, we
recommend first replicating the install process from thepolytracker-llvmDockerfile,
followed by replicating the install process from thePolyTracker Dockerfile.Build DependenciesPolyTracker LLVM.
PolyTracker is built atop its own fork of LLVM,polytracker-llvm. This
fork modifies theDataFlow Sanitizerto
use increased label sizes (to allow for tracking orders of magnitude more
taints), as well as alternative data structures to store them. We have
investigated up-streaming our changes into LLVM proper, but there has been
little interest.CMakeNinja(ninja-buildon Ubuntu)Runtime DependenciesThe following tools are required to test and run PolyTracker:Python 3.7+ andpip(apt-get -y install python3.7 python3-pip). These are
used for both seamlessly interacting with the Docker container (if necessary),
as well as post-processing and analyzing the artifacts produced from runtime
traces.gllvm(go get github.com/SRI-CSL/gllvm/cmd/...) is used to create whole program
bitcode archives and to extract bitcode from targets.Building on Apple silicon:Prebuilt Docker images forpolytracker-llvmare only available foramd64.
Users witharm64systems will have to build the image locally and then changepolytracker's Dockerfile to point to it:$ mkdir repos && cd repos
$ git clone https://github.com/trailofbits/polytracker
$ git clone https://github.com/trailofbits/polytracker-llvm
$ cd polytracker-llvm
$ DOCKER_BUILDKIT=1 docker build -t trailofbits/polytracker-llvm .
$ cd ../polytracker
$ ## Replace the first line of the Dockerfile with "FROM trailofbits/polytracker-llvm:latest" (no quotes)
$ docker build -t trailofbits/polytracker .Current Status and Known IssuesPolyTracker currently only runs on Linux, because that is the only system
supported by the DataFlow Santizer. This limitation is just due to a lack of
support for semantics for other OSes system calls, which could be added in the
future. However, this means that running PolyTracker on a non-Linux system will
require Docker to be installed.Taints will not propagate through dynamically loaded libraries unless those
libraries were compiled from source using PolyTracker,orthere is specific
support for the library calls implemented in PolyTracker. Thereiscurrently
support for propagating taint through the majority of uninstrumented C standard
library calls. To be clear, programs that use uninstrumented functions will
still run normally, however, operations performed by unsupported library calls
will not propagate taint. We are currently working on adding robust support for
C++ programs, but currently the best results will be from C programs.If there are issues with Docker, try performing a system prune and build with--no-cachefor both PolyTracker and whatever demo you are trying to run.The worst case performance of PolyTracker is exercised when a single byte in
memory is simultaneously tainted by a large number of input bytes from the
source file. This is most common when instrumenting compression and
cryptographic algorithms that have large block sizes. There are a number of
mitigations for this behavior currently being researched and developed.License and AcknowledgementsThis research was developed byTrail of Bitswith funding from the Defense Advanced Research Projects Agency (DARPA) under
the SafeDocs program as a subcontractor toGalois. It is
licensed under theApache 2.0 license. © 2019, Trail of Bits.MaintainersEvan SultanikHenrik BrodinMarek SurovičFacundo Tuescafirstname.lastname@trailofbits.com