Package: polyfile
PolyFileA utility to identify and map the semantic and syntactic structure of files,
including polyglots, chimeras, and schizophrenic files. It hasa pure-Python implementation of libmagicand can act as a drop-in replacement for thefilecommand. However, unlikefile, PolyFile can recursively identify embedded files, likebinwalk.PolyFile can be used in conjunction with its sister toolPolyTrackerforAutomated Lexical Annotation and Navigation of Parsers, a backronym
devised solely for the purpose of collectively referring to the tools
asThe ALAN Parsers Project.QuickstartYou can install the latest stable version of PolyFile from PyPI:pip3 install polyfileTo install PolyFile from source, in the same directory as this README, run:pip3 install .Important: Before installing from source, make sure Java is installed. Java is used to
run the Kaitai Struct compiler, which compiles the file format definitions.This will automatically install thepolyfileandpolymergeexecutables in your path.UsageRunningpolyfileon a file with no arguments will mimic the behavior offile --keep-going:$polyfilepng-polyglot.pngPNG image data, 256 x 144, 8-bit/color RGB, non-interlacedBrainfu** ProgramMalformed PDFPDF document, version 1.3,  1 pagesZIP end of central directory record Java JAR archiveTo generate an interactive hex viewer for the file, use the--htmloption:$polyfile--htmloutput.htmlpng-polyglot.pngFound a file of type application/pdf at byte offset 0Found a file of type application/x-brainfuck at byte offset 0Found a file of type image/png at byte offset 0Found a file of type application/zip at byte offset 0Found a file of type application/java-archive at byte offset 0Saved HTML output to output.htmlRunpolyfile --helpfor full usage instructions.Interactive DebuggerPolyFile has an interactive debugger both for its file matching and parsing. It can be used to debug a libmagic pattern
definition, determine why a specific file fails to be classified as the expected MIME type, or step through a parser.
You can run PolyFile with the debugger enabled using the-dboption.File SupportPolyFile has a cleanroom,pure Python implementation of the libmagic file classifier, and supports all 263 MIME types that it can identify.It currently has support for parsing and semantically mapping the following formats:PDF, using an instrumented version ofDidier Stevens' public domain, permissive, forensic parserZIP, including recursive identification of all ZIP contentsJPEG/JFIF, using itsKaitai Struct grammariNESAny other formatspecified in aKSY grammarFor an example that exercises all of these file formats, run:curl-v--silenthttps://www.sultanik.com/files/ESultanikResume.pdf|polyfile--htmlESultanikResume.html-Prior to PolyFile version 0.3.0, it used theTrID databasefor file
identification rather than the libmagic file definitions. This proved to be very slow (since TrID has many duplicate
entries) and prone to false positives (since TrID's file definitions are much simpler than libmagic's). The original
TrID matching code is still shipped with PolyFile and can be invoked programmatically, but it is not used by default.Output FormatPolyFile has several options for outputting its results, specified by its--formatoption. For computer-readable output, PolyFile has an extension of theSBuDJSON format describedin the documentation. Prior to version 0.5.0 this was the default output format of PolyFile. However, now the default output format is to mimic the behavior of thefilecommand. To maintain the original behavior, use the--format sbudoption.libmagic ImplementationPolyFile has a cleanroom implementation oflibmagic (used in thefilecommand).
It can be invoked programmatically by running:frompolyfile.magicimportMagicMatcherwithopen("file_to_test","rb")asf:# the default instance automatically loads all file definitionsformatchinMagicMatcher.DEFAULT_INSTANCE.match(f.read()):formimetypeinmatch.mimetypes:print(f"Matched MIME:{mimetype}")print(f"Match string:{match!s}")To load a specific or custom file definition:list_of_paths_to_definitions=["def1","def2"]matcher=MagicMatcher.parse(*list_of_paths_to_definitions)withopen("file_to_test","rb")asf:formatchinmatcher.match(f.read()):...Extending PolyFileInstructions on extending PolyFile to support more file formats with new matchers and parsers is described [in the documentation](in the documentation).License and AcknowledgementsThis research was developed byTrail of
Bitswith funding from the Defense
Advanced Research Projects Agency (DARPA) under the SafeDocs program
as a subcontractor toGalois. It is licensed under theApache 2.0 license.
© 2019, Trail of Bits.