Package: parse-binary-file
Parse Binary FileInstall withpython -m pip install parse_binary_fileParse binary files by describing their structure.Description fileTheparse_binary_filepackage is based on the idea that binary file formats
should be parsed from a description of their format.If the description file is a list,pbfassumes that the elemnts are fields.
One can also pass a dictionary, though, to describe more details about the file
format.There are three top-level items that can be specified:info:Describes information about the file as a whole (e.g. endianess).default_options:Describes defaults options for each field type.fields:Fields of the file format.InfoDescribes information about the file.Propertiesbyte_order:Byte order of the file.
Valid values: ['little', 'big', 'native']nativebyte ordering uses the byte ordering of the current machine.
[Default: 'native']encoding:File encoding.
Used to encode the description file's values to bytes when matching against
the file being parsed.
See Python'sstandard encodingsfor valid values.
[Default: 'utf-8']word_size:Bytes per word of the file. [Default: 4 bytes]
[Inactive]Default OptionsDescribes parsing information for each type of field.
All types can specify default values for a particular field property (discussed below).
To specify a default field property first specify the type of the field, then the property.e.g.
To specify strings are null terminated (\x00) by defaultdefault_options:str:terminator:"\x00"FieldsDescribes the fields of the file format.
Each field must one and only one termination condition.
Termination conditions are determined by thesize,value, andterminatorfields.
for some types these can be interpreted. For instance, anintis assumed to
have a size of 4, so a termination condition does not need to be explicitly stated.All fields can have the properties:type:Data type of the field.
If a type is enclosed in brackets ([]) it marks the field as an array-like field.
Array like fields are assumed to contain multiple instances of the specified
type. For more information see theArray Fieldssection.size:Sets the size of the field in bytes. This may be inferred
depending on the field'stype.
A size of-1indicated the field extends until the end of the file.
Only the final field may have a length of-1.value:Sets the size of the fiel based on the expected value.format:How to format the field. This is type dependent.
Forstrfields, this represents the encoding of the string and defaults to
the file's encoding provided in theinfosection.
For other types this represents how Python'sstruct.unpackshould
interpret the field.terminator:Termination string.is_null:Used forbytesdata type to indicate all bytes should be null.
Should be used in conjunction with thesizeproperty to indicate the
termination condition.name:Name of the field. Used for accessing the data by name.description:Description of the field.fields:Describes the subfields of the field.
If a field is made up of subfields, its type must bebytesor[bytes].exec:Execution hooks for logical processing. [Inactive]TypeIf a field is not provided a type it defaults tobytes.Valid types are:bytes:Interpret the field as raw bytes. [Default]char:Single character. (1 byte)str:String data.bool:Boolean. (1 byte)short:Short integer. (2 bytes)u_short:Unsigned short integer. (2 bytes)int:Integer.(4 bytes)u_int:Unisigned integer. (4 bytes)long:Long integer. (1 word)u_long:Unsigned long integer. (1 word)long_long:Double long integer. (2 words)u_long_long:Unsigned double long integer. (2 words)float:Floating point number. (4 bytes)double:Double precision floating point number. (2 words)Any type may be wrapped in brackets[]to indicate the field is an array
of that value type. Details of the elements are set using theelementsproperty.For more information read about Pythonstruct's formatting strings.Termination conditionThe termination condition of a field tells the parser how far to read.
There are three explicit and one implit termination condition.Explicitsize:Sets the size of the field in bytes.
This may be inferred depending on the field'stype.terminator:Termination string as bytes.value:Sets the size of the field based on the expected value.Implicitsubfields:[Inactive] The parent field can be terminated when the last child field is terminated.SubfieldsEach field can be made up of subfields. Each subfield follows the same pattern
as top-level fields. This is a recursive concept, so fields can be nested as
deeply as needed.Fields that are made up of subfields must have their type set tobytesor[bytes].
If the type is[bytes]it is assumed that the subfields repeat in a array-like
manner with all sufields being repeated an equal number of times.
i.e. If the data for the parent field ends before all subfields have been read
an exception is raised.Array FieldsA field may consist of multiple elements. This is where array fields come into
play. For instance, imagine a field that consists of a list of floats. To
indicate a field is array-like, enclose its type in square brackets ('[]')
(e.g.[bytes],[float]).Execution Hooks:warning: These fields allow arbitrary Python code to be executed.It may be necessary to add some basic logic to your description.
For instance, if a field nameddatamay have a variable length offset buffer
specified by a field nameddata_offset.
Theexecfield allows you to specify code before (pre) or after (post)
data for the field has been loaded and has access to all previous fields.
Each of these fields should contain a lambda function with signature(data, fields)wheredatais byte string of the current field andfieldsis aparse_binary_file.Dataobject representing the file with
all previous fields loaded.ComponentsFieldDescriptionDescribes a field.Propertiestype:The type of the field being described.
When loading data into the field it will be parsed based on the type.[Default: 'bytes']name:Name of the field.
This is used to label the field.value:The expected value of the field.
This can be usedto indicate a specific value is expected for the field.size:Number of bytes in the field.
The last field can have a size of -1 which indicates the field continues to
the end of the file.terminator:Sequence of values indicating the end of the field.is_null:Indicates the field only contains null bytes.
Settings to true is equivalent to settingvalueequal to all null bytes.
Must setsizeifTrue.description:Description of the field.fields:Used to specifiy the structure of the elements of an array.MethodsFieldDescription(**properties):Initializes a newFieldDescriptionwith
the provided properties.FileFormatContainer forFieldDescriptions to describe the structure of a file.Propertiesfields:List ofFieldDescriptions.info:Dictionary of information on the file.Fields can be accessed by name or index using brackets ([])Methodsfrom_dicts(desc, info, defaults):@staticmethodConverts a list of dictionaries into aFileFormat.keys():Returns a list of the keys of named fields.ParserUsed for parsing files in a given format.Propertiesformat:AFileFormatused to parse files.Methodsparse(stream):Returns aDataobject representing the data fromstream.FieldContains information about a field, including its description and loaded value.Propertiesdesc:FieldDescriptionthat was used to create theField.value:Parsed value.expected_value:self.desc.value. Provided for convenience.fields:Tuple of childrenFields.size:If theField's value is set, returns its size.
If theField's value has not yet been set return theField's expected size if available.data:Original data as bytes.value_is_valid:Whether the parsed value matches the expected value. If a specificvaluewas not specified by theFieldDescriptionthis will always returnTrue.Values of theField'sFieldDescriptionare accessible as properties, as well.MethodsField(desc: FieldDescription, fields: tuple[Field]):Initializes a newField.descshould describe the field, andfieldsare the childrenFields.from_data(data: bytes, desc: FieldDescription):Creates a newFieldusingdescas its description to parsedata.parse_data(val: bytes, **options):Sets the value of theFieldbased on its description and the provided value.DataRepresents data from a file.Propertiesfields:Fields containing the parsed data.is_loaded:If data has been loaded yet.value:Values of the parsed data.named_field_values:Returns a dictionary of name-value pairs for namedFields.
If multiple fields have the same name a tuple is returned with the values in
order.Fields are accessible by name and index using brackets ([]). If multipleFields have the same name, they are returned as a tuple in order.UseThis library is intended to be used by describing the struture of a binary file
format in a configuration file. That file is then loaded and used to create aDatainstance. TheDatainstance can thenload()data from a file and
assort it into the correct fields.ExampleWe would like to describe a new binary file format of.msg.OffsetSizeNameDescription0x000x02head0xff 0x00Indicates the file is a.msgfile.0x020x04sizeTotal size of the file.0x060x04--Null bytes to separate header from body.0x0a--greetingMessage greeting, null terminated.----messageMessage content.We can describe this structure in a YAML file.msg.yamlinfo:byte_order:'little'fields:-name:'head'value:"\xff\x00"-type:'int'name:'file_size'description:'Sizeoffileinbytes'-size:4is_null:true-type:'str'terminator:"\x00"name:'greeting'-type:'str'size:-1name:'message'To read a.msgfile we can use the description of it.importosimportyaml# use pyyaml to load `msg.yaml`importparse_binary_fileaspbf# load descriptionwithopen('msg.yaml')asf:msg_desc=yaml.safe_load(f)# create file formatmsg_format=pbf.FieldFormat.from_dicts(msg_desc['fields'],info=msg_desc['info'])# create parserparser=pbf.Parser(msg_format)# create datawithopen('my_message.msg','rb')asf:data=parser.parse(f)is_corrupt=(data['file_size']!=os.path.getsize('my_message.msg'))greeting=data['greeting'].valuemessage=data[-1].value