Package: cs.binary
Facilities associated with binary data parsing and transcription.
The classes in this module support easy parsing of binary data
structures,
returning instances with the binary data decoded into attributes
and capable of transcribing themselves in binary form
(trivially viabytes(instance)and also otherwise).Latest release 20240201:
BREAKING CHANGE: drop the long deprecated PacketField related classes.Note: this module requires Python 3.6+ because various default
behaviours rely ondicts preserving their insert order.Seecs.iso14496for an ISO 14496 (eg MPEG4) parser
built using this module.Terminology used below:buffer:
an instance ofcs.buffer.CornuCopyBuffer,
which presents an iterable of bytes-like values
via various useful methods;
it also has a few factory methods to make one from a variety of sources
such as bytes, iterables, binary files,mmapped files,
TCP data streams, etc.chunk:
a piece of binary data obeying the buffer protocol,
almost always abytesinstance or amemoryview,
but in principle also things likebytearray.There are 5 main classes on which an implementor should base their data structures:BinarySingleStruct: a factory for classes based
on astruct.structformat string with a single value;
this builds anamedtuplesubclassBinaryMultiStruct: a factory for classes based
on astruct.structformat string with multiple values;
this also builds anamedtuplesubclassBinarySingleValue: a base class for subclasses
parsing and transcribing a single valueBinaryMultiValue: a base class for subclasses
parsing and transcribing multiple values
with no variationSimpleBinary: a base class for subclasses
with custom.parseand.transcribemethods,
for structures with variable fieldsAll the classes derived from the above inherit all the methods
ofBinaryMixin.
Amongst other things, this means that the binary transcription
can be had simply frombytes(instance),
although there are more transcription methods provided
for when greater flexibility is desired.
It also means that all classes haveparse* andscan* methods
for parsing binary data streams.You can also instantiate objects directly;
there's no requirement for the source information to be binary.There are several presupplied subclasses for common basic types
such asUInt32BE(an unsigned 32 bit big endian integer).ClassAbstractBinary(BinaryMixin)Abstract class for allBinary* implementations,
specifying theparseandtranscribemethods
and providing the methods fromBinaryMixin.ClassBinaryByteses(AbstractBinary, BinaryMixin)A list ofbytesparsed directly from the native iteration of the buffer.FunctionBinaryFixedBytes(class_name, length: int)Factory for anAbstractBinarysubclass matchinglengthbytes of data.
The bytes are saved as the attribute.data.ClassBinaryListValues(AbstractBinary, BinaryMixin)A list of values with a common parse specification,
such as sample or Boxes in an ISO14496 Box structure.ClassBinaryMixinPresupplied helper methods for binary objects.Naming conventions:parse* methods parse a single instance from a bufferscan* methods are generators yielding successive instances from a bufferFunctionBinaryMultiStruct(class_name: str, struct_format: str, field_names: Union[str, List[str]])A class factory forAbstractBinarynamedtuplesubclasses
built around complexstructformats.Parameters:class_name: name for the generated classstruct_format: thestructformat stringfield_names: field name list,
a space separated string or an interable of stringsExample:# an "access point" record from the .ap file
Enigma2APInfo = BinaryMultiStruct('Enigma2APInfo', '>QQ', 'pts offset')

# a "cut" record from the .cuts file
Enigma2Cut = BinaryMultiStruct('Enigma2Cut', '>QL', 'pts type')FunctionBinaryMultiValue(class_name, field_map, field_order=None)Construct aSimpleBinarysubclass namedclass_namewhose fields are specified by the mappingfield_map.Thefield_mapis a mapping of field name to buffer parsers and transcribers.Note:
iffield_orderis not specified
it is constructed by iterating overfield_map.
Prior to Python 3.6,dicts do not provide a reliable order
and should be accompanied by an explicitfield_order.
From 3.6 onward adictis enough and its insertion order
will dictate the defaultfield_order.For a fixed record structure
the default.parseand.transcribemethods will suffice;
they parse or transcribe each field in turn.
Subclasses with variable records should override
the.parseand.transcribemethods
accordingly.Thefield_mapis a mapping of field name
to a class returned by thept_spec()function.If the class has bothparse_valueandtranscribe_valuemethods
then the value itself will be directly stored.
Otherwise the class it presumed to be more complex subclass
ofAbstractBinaryand the instance is stored.Here is an example exhibiting various ways of defining each field:n1: defined with the *_valuemethods ofUInt8,
which return or transcribe theintfrom an unsigned 8 bit value;
this stores aBinarySingleValuewhose.valueis anintn2: defined from theUInt8class,
which parses an unsigned 8 bit value;
this stores anUInt8instance
(also aBinarySingleValuewhole.valueis anint)n3: liken2data1: defined with the *_valuemethods ofBSData,
which return or transcribe the databytesfrom a run length encoded data chunk;
this stores aBinarySingleValuewhose.valueis abytesdata2: defined from theBSDataclass
which parses a run length encoded data chunk;
this is aBinarySingleValueso we store itsbytesvalue directly.>>> class BMV(BinaryMultiValue("BMV", {
...         'n1': (UInt8.parse_value, UInt8.transcribe_value),
...         'n2': UInt8,
...         'n3': UInt8,
...         'nd': ('>H4s', 'short bs'),
...         'data1': (
...             BSData.parse_value,
...             BSData.transcribe_value,
...         ),
...         'data2': BSData,
... })):
...     pass
>>> BMV.FIELD_ORDER
['n1', 'n2', 'n3', 'nd', 'data1', 'data2']
>>> bmv = BMV.from_bytes(b'\x11\x22\x77\x81\x82zyxw\x02AB\x04DEFG')
>>> bmv.n1  #doctest: +ELLIPSIS
17
>>> bmv.n2
34
>>> bmv  #doctest: +ELLIPSIS
BMV(n1=17, n2=34, n3=119, nd=nd_1_short__bs(short=33154, bs=b'zyxw'), data1=b'AB', data2=b'DEFG')
>>> bmv.nd  #doctest: +ELLIPSIS
nd_1_short__bs(short=33154, bs=b'zyxw')
>>> bmv.nd.bs
b'zyxw'
>>> bytes(bmv.nd)
b'zyxw'
>>> bmv.data1
b'AB'
>>> bmv.data2
b'DEFG'
>>> bytes(bmv)
b'\x11"w\x81\x82zyxw\x02AB\x04DEFG'
>>> list(bmv.transcribe_flat())
[b'\x11', b'"', b'w', b'\x81\x82zyxw', b'\x02', b'AB', b'\x04', b'DEFG']FunctionBinarySingleStruct(class_name, struct_format, field_name=None)A convenience wrapper forBinaryMultiStructforstruct_formats with a single field.Parameters:class_name: the class name for the generated classstruct_format: the struct format string, specifying a
single struct fieldfield_name: optional field name for the value,
default'value'Example:>>> UInt16BE = BinarySingleStruct('UInt16BE', '>H')
>>> UInt16BE.__name__
'UInt16BE'
>>> UInt16BE.format
'>H'
>>> UInt16BE.struct   #doctest: +ELLIPSIS
<_struct.Struct object at ...>
>>> field = UInt16BE.from_bytes(bytes((2,3)))
>>> field
UInt16BE(value=515)
>>> field.value
515ClassBinarySingleValue(AbstractBinary, BinaryMixin)A representation of a single value as the attribute.value.Subclasses must implement:parseorparse_valuetranscribeortranscribe_valueClassBinaryUTF16NUL(BinarySingleValue, AbstractBinary, BinaryMixin)A NUL terminated UTF-16 string.MethodBinaryUTF16NUL.__init__(self, value, *, encoding):
pylint: disable=super-init-not-calledClassBinaryUTF8NUL(BinarySingleValue, AbstractBinary, BinaryMixin)A NUL terminated UTF-8 string.ClassBSData(BinarySingleValue, AbstractBinary, BinaryMixin)A run length encoded data chunk, with the length encoded as aBSUInt.ClassBSSFloat(BinarySingleValue, AbstractBinary, BinaryMixin)A float transcribed as a BSString of str(float).ClassBSString(BinarySingleValue, AbstractBinary, BinaryMixin)A run length encoded string, with the length encoded as a BSUInt.ClassBSUInt(BinarySingleValue, AbstractBinary, BinaryMixin)A binary serialised unsignedint.This uses a big endian byte encoding where continuation octets
have their high bit set. The bits contributing to the value
are in the low order 7 bits.Functionflatten(chunks)Flattenchunksinto an iterable ofbytesinstances.This exists to allow subclass methods to easily return
transcribeable things (having a.transcribemethod), ASCII
strings or bytes or iterables or evenNone, in turn allowing
them simply to return their superclass' chunks iterators
directly instead of having to unpack them.An example from thecs.iso14496.METABoxBodyclass:def transcribe(self):
    yield super().transcribe()
    yield self.theHandler
    yield self.boxesThe binary classesflattenthe result of the.transcribemethod to obtainbytesinsteances for the object's bnary
transcription.ClassFloat64BE(Float64BE, builtins.tuple, AbstractBinary, BinaryMixin)AnAbstractBinarynamedtuplewhich parses and transcribes
the struct format'>d'and presents the attributes ('value',).ClassFloat64LE(Float64LE, builtins.tuple, AbstractBinary, BinaryMixin)AnAbstractBinarynamedtuplewhich parses and transcribes
the struct format'<d'and presents the attributes ('value',).ClassInt16BE(Int16BE, builtins.tuple, AbstractBinary, BinaryMixin)AnAbstractBinarynamedtuplewhich parses and transcribes
the struct format'>h'and presents the attributes ('value',).ClassInt16LE(Int16LE, builtins.tuple, AbstractBinary, BinaryMixin)AnAbstractBinarynamedtuplewhich parses and transcribes
the struct format'<h'and presents the attributes ('value',).ClassInt32BE(Int32BE, builtins.tuple, AbstractBinary, BinaryMixin)AnAbstractBinarynamedtuplewhich parses and transcribes
the struct format'>l'and presents the attributes ('value',).ClassInt32LE(Int32LE, builtins.tuple, AbstractBinary, BinaryMixin)AnAbstractBinarynamedtuplewhich parses and transcribes
the struct format'<l'and presents the attributes ('value',).Functionpt_spec(pt, name=None)Convert a parse/transcribe specificationptinto anAbstractBinarysubclass.This is largely used to provide flexibility
in the specifications for theBinaryMultiValuefactory
but can be used as a factory for other simple classes.If the specificationptis a subclass ofAbstractBinarythis is returned directly.Ifptis a 2-tuple ofstrthe values are presumed to be a format string forstruct.structand filed names separated by spaces;
a newBinaryMultiStructclass is created from these and returned.Otherwise two functionsf_parse_value(bfr)andf_transcribe_value(value)are obtained and used to construct a newBinarySingleValueclass
as follows:Ifpthas.parse_valueand.transcribe_valuecallable attributes,
use those forf_parse_valueandf_transcribe_valuerespectively.Otherwise, ifptis anintdefinef_parse_valueto obtain exactly that many bytes from a buffer
andf_transcribe_valueto return those bytes directly.Otherwise presumeptis a 2-tuple of(f_parse_value,f_transcribe_value).ClassSimpleBinary(types.SimpleNamespace, AbstractBinary, BinaryMixin)Abstract binary class based on aSimpleNamespace,
thus providing a nice__str__and a keyword based__init__.
Implementors must still define.parseand.transcribe.To constrain the arguments passed to__init__,
define an__init__which accepts specific keyword arguments
and pass through tosuper().__init__(). Example:def __init__(self, *, field1=None, field2):
    """ Accept only `field1` (optional)
        and `field2` (mandatory).
    """
    super().__init__(field1=field1, field2=field2)ClassUInt16BE(UInt16BE, builtins.tuple, AbstractBinary, BinaryMixin)AnAbstractBinarynamedtuplewhich parses and transcribes
the struct format'>H'and presents the attributes ('value',).ClassUInt16LE(UInt16LE, builtins.tuple, AbstractBinary, BinaryMixin)AnAbstractBinarynamedtuplewhich parses and transcribes
the struct format'<H'and presents the attributes ('value',).ClassUInt32BE(UInt32BE, builtins.tuple, AbstractBinary, BinaryMixin)AnAbstractBinarynamedtuplewhich parses and transcribes
the struct format'>L'and presents the attributes ('value',).ClassUInt32LE(UInt32LE, builtins.tuple, AbstractBinary, BinaryMixin)AnAbstractBinarynamedtuplewhich parses and transcribes
the struct format'<L'and presents the attributes ('value',).ClassUInt64BE(UInt64BE, builtins.tuple, AbstractBinary, BinaryMixin)AnAbstractBinarynamedtuplewhich parses and transcribes
the struct format'>Q'and presents the attributes ('value',).ClassUInt64LE(UInt64LE, builtins.tuple, AbstractBinary, BinaryMixin)AnAbstractBinarynamedtuplewhich parses and transcribes
the struct format'<Q'and presents the attributes ('value',).ClassUInt8(UInt8, builtins.tuple, AbstractBinary, BinaryMixin)AnAbstractBinarynamedtuplewhich parses and transcribes
the struct format'B'and presents the attributes ('value',).Release LogRelease 20240201:
BREAKING CHANGE: drop the long deprecated PacketField related classes.Release 20231129:
BinaryMultiStruct.parse: promote the buffer arguments to a CornuCopyBuffer.Release 20230401:BinaryMixin.scan:bfrparameter may be any object acceptable to CornuCopyBuffer.promote.BinaryMixin.scan: accept new optional with_offsets parameter; deprecate scan_with_offsets and scan_fspathi in favour of scan.Release 20230212:BinaryMixin: new load(file) and save(file) methods.BinaryMixin.scan: promote the bfr argument.Release 20221206:
Documentation fix.Release 20220605:
BinaryMixin: replace scan_file with scan_fspath, as the former left uncertainty about the amount of the file consumed.Release 20210316:BSUInt: rename parse_bytes to decode_bytes, the former name conflicted with BinaryMixin.parse_bytes and broken the semantics.Minor refactors.Release 20210306:
MAJOR RELEASE: The PacketField classes and friends were hard to use; this release supplied a suite of easier to use and more consistent Binary* classes, and ports most of those things based on the old scheme to the new scheme.Release 20200229:ListField: replace transcribe method with transcribe_value method, aids external use.Add.lengthattribute to struct based packet classes providing the data length of the structure (struct.Struct.size).Packet: newadd_deferred_fieldmethod to consume the raw data for a field for parsing later (done automatically if the attribute is accessed).New@deferred_fielddecorator for the parser for that stashed data.Release 20191230.3:
Docstring tweak.Release 20191230.2:
Documentation updates.Release 20191230.1:
Docstring updates. Semantic changes were in the previous release.Release 20191230:ListField: newitermethod.Packet:str: accept optionalskip_fieldsparameter to omit some field names.Packet: new .add_from_value method to add a named field with a presupplied value.Packet: new remove_field(field_name) and pop_field() methods to remove fields.BytesesField:iteryields the bytes values, transcribe=iter.PacketField: propagate keyword arguments through various methods, required for parameterised PacketFields.New UTF16NULField, a NUL terminated UTF16 string.PacketField: provide a default.transcribe_valuemethod which makes a new instance and calls its.transcribemethod.Documentation update and several minor changes.Release 20190220:Packet.self_check: fields without a sanity check cause a warning, not a ValueError.New Float64BE, Float64LE and BSSFloat classes for IEEE floats and floats-as-strings.Additional module docstringage on subclassing Packet and PacketField.BSString: drop redundant from_buffer class method.PacketField.init: default to value=None if omitted.Release 20181231:
flatten: do not yield zero length bytelike objects, can be misread as EOF on some streams.Release 20181108:New PacketField.transcribe_value_flat convenience method to return a flat iterable of bytes-like objects.New PacketField.parse_buffer generator method to parse instances of the PacketField from a buffer until end of input.New PacketField.parse_buffer_values generator method to parse instances of the PacketField from a buffer and yield the.valueattribute until end of input.Release 20180823:Some bugfixes.Define PacketField.eq.BSUInt, BSData and BSString classes implementing the serialisations from cs.serialise.New PacketField.value_from_bytes class method.New PacketField.value_from_buffer method.Release 20180810.2:
Documentation improvements.Release 20180810.1:
Improve module description.Release 20180810:
BytesesField.from_buffer: make use of the buffer's skipto method if discard_data is true.Release 20180805:Packet: now an abstract class, new self_check method initially checking thePACKET_FIELDS class attribute against the instance, new methods get_fieldand set_field to fetch or replace existing fields, allow keyword argumentsto initialise the Packet fields and document the dependency on keywordargument ordering.PacketField:lencomputed directory from a transcribe, drop otherlenmethods.EmptyField singleton to use as a placeholder for missing optional fields.BytesField: implement value_s and from_buffer.multi_struct_field: implementlenfor generated class.flatten: treat memoryviews like bytes.Assorted docstrings and fixes.Release 20180801:
Initial PyPI release.